WO2025123765A1 - 数据处理、智能交互、模型训练和开发方法、设备和介质 - Google Patents

数据处理、智能交互、模型训练和开发方法、设备和介质 Download PDF

Info

Publication number
WO2025123765A1
WO2025123765A1 PCT/CN2024/113737 CN2024113737W WO2025123765A1 WO 2025123765 A1 WO2025123765 A1 WO 2025123765A1 CN 2024113737 W CN2024113737 W CN 2024113737W WO 2025123765 A1 WO2025123765 A1 WO 2025123765A1
Authority
WO
WIPO (PCT)
Prior art keywords
tuning
network
data
subnetwork
input
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2024/113737
Other languages
English (en)
French (fr)
Inventor
江泽胤子
毛超杰
潘玉霖
韩振
张靖峰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba China Co Ltd
Original Assignee
Alibaba China Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba China Co Ltd filed Critical Alibaba China Co Ltd
Priority to KR1020267008759A priority Critical patent/KR20260052168A/ko
Publication of WO2025123765A1 publication Critical patent/WO2025123765A1/zh
Anticipated expiration legal-status Critical
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0475—Generative networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/096—Transfer learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/10—Interfaces, programming languages or software development kits, e.g. for simulating neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models

Definitions

  • the embodiments of the present application relate to the field of computer technology, and in particular to a method, device and medium for data processing, intelligent interaction, model training and development.
  • generative models such as large language models are constructed through generative networks.
  • diffusion generative networks have effective network structures and can show excellent generalization capabilities after sample training.
  • Basic generative models pre-trained on large-scale data can be fine-tuned for a series of downstream tasks and applications such as image generation, image editing, and image transformation.
  • embodiments of the present application provide a data processing, intelligent interaction, model training and development method, device and medium to at least partially solve the above problems.
  • a data processing method comprising: obtaining a pre-trained generative model, the generative model comprising a residual structure, the residual structure comprising a jump connection between an encoding layer and a decoding layer, and at least one tuning subnetwork is formed between the encoding layer and the decoding layer; at least descriptive data is input into the generative model to obtain generated data corresponding to the descriptive data.
  • an intelligent interaction method comprising: obtaining first description data input in an interactive interface; at least inputting the first description data into a pre-trained generative model to obtain first generated data corresponding to the first description data, wherein the first generated data includes at least one of an image, a video frame, and a video, the generative model includes a residual structure, the residual structure includes a jump connection between a coding layer and a decoding layer, and at least one tuning subnetwork is formed between the coding layer and the decoding layer.
  • the first generated data is displayed in the question-and-answer interface.
  • a model training method comprising: obtaining an initially trained generative network, the generative network comprising an encoder and a decoder, wherein a jump connection is formed between an encoding layer of the encoder and a decoding layer of the decoder; connecting at least one tuning subnetwork of the jump connection between the encoding layer and the decoding layer of the initially trained generative network to obtain a generative network to be fine-tuned, wherein an output of the encoding layer is connected to an input of the at least one tuning subnetwork, and an output of the at least one tuning subnetwork is connected to an input of the decoding layer; and fine-tuning the generative network to be fine-tuned based on training samples to obtain a generative model.
  • an application development method comprising: creating a user interface module of an application, the user interface module being configured to generate description data based at least on user operation data, and returning presentation data based on the generated data of the description data; obtaining a calling interface of a generative model, the generative model being obtained according to the model training method described in the third aspect, the calling interface being configured to return the generated data when called; and embedding at least the calling interface of the generative model into the user interface module.
  • a data processing device including: an acquisition module, configured to acquire a pre-trained generative model, the generative model including a residual structure, the residual structure including a jump connection between a coding layer and a decoding layer, and at least one tuning sub-network is formed between the coding layer and the decoding layer; a generation module, configured to at least input description data into the generative model to obtain generation data corresponding to the description data.
  • an intelligent interaction device including: an acquisition module, configured to acquire first description data input in an interaction interface; a generation module, configured to at least input the first description data into a pre-trained generative model to obtain first generated data corresponding to the first description data, wherein the first generated data includes at least one of an image, a video frame, and a video, and the generative model includes a residual structure, and the residual structure includes a jump connection between a coding layer and a decoding layer, and at least one tuning sub-network is formed between the coding layer and the decoding layer; a display module, configured to display the first generated data in the question-and-answer interface.
  • a model training device comprising: a module acquisition module, configured to acquire an initially trained generative network, the generative network comprising an encoder and a decoder, a jump connection being formed between an encoding layer of the encoder and a decoding layer of the decoder; a model adjustment module, configured to connect at least one tuning subnetwork of the jump connection between the encoding layer and the decoding layer of the initially trained generative network, to obtain a generative network to be fine-tuned, wherein an output of the encoding layer is connected to an input of the at least one tuning subnetwork, and an output of the at least one tuning subnetwork is connected to an input of the decoding layer; a model training module, configured to perform fine-tuning training on the generative network to be fine-tuned based on training samples, to obtain a generative model.
  • an application development device comprising: a creation module, configured to create a user interface module of an application, the user interface module being configured to generate description data based at least on user operation data, and to return presentation data based on the generated data of the description data; an acquisition module, configured to acquire a calling interface of a generative model, the generative model being obtained according to the model training method described in the third aspect, the calling interface being configured to return the generated data when called; and an embedding module, configured to embed at least the calling interface of the generative model into the user interface module.
  • an electronic device comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the first aspect.
  • a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in the first aspect is implemented.
  • the pre-trained generative model includes a residual structure
  • the residual structure includes a jump connection between the encoding layer and the decoding layer.
  • At least one tuning sub-network is formed between the encoding layer and the decoding layer. Therefore, at least one tuning sub-network can efficiently train the residual structure by connecting to the encoding layer and the decoding layer of the jump connection, thereby obtaining a generative model with higher prediction reliability, so that more reliable data processing can be performed using the generative model.
  • FIG1 is a schematic diagram of a structure of a generative network according to some examples.
  • FIG. 2A is a flowchart of steps of a data processing method according to other embodiments of the present application.
  • FIG. 2B is a schematic diagram of a partial structure of a generative network of some examples of the embodiment of FIG. 2 .
  • FIG. 2C is a schematic diagram of a partial structure of a generative network of other examples of the embodiment of FIG. 2 .
  • FIG. 2D is a flowchart of the steps of a data processing method according to other embodiments of the present application.
  • FIG. 2E is a schematic diagram of a partial structure of a generative network of other examples of the embodiment of FIG. 2 .
  • FIG3 is a flowchart of steps of an intelligent interaction method according to other embodiments of the present application.
  • FIG4A is a flowchart of the steps of a model training method according to other embodiments of the present application.
  • FIG4B is a flowchart of the steps of a model training method according to other embodiments of the present application.
  • FIG5 is a flowchart of the steps of an application development method according to other embodiments of the present application.
  • FIG. 6 is a schematic block diagram of a data processing device according to some other embodiments of the present application.
  • FIG. 7 is a schematic block diagram of an intelligent interaction device according to some other embodiments of the present application.
  • FIG8 is a schematic block diagram of a model training device according to other embodiments of the present application.
  • FIG. 9 is a schematic block diagram of an application development device according to other embodiments of the present application.
  • FIG. 10 is a schematic structural diagram of an electronic device according to other embodiments of the present application.
  • Generative models such as large language models are constructed through generative networks.
  • the generative network shown in FIG1 includes an encoder 110 and a decoder 120.
  • the output of the encoder 110 can be directly connected to the input of the decoder 120.
  • the output of the encoder 110 can also be connected to the input of the decoder 120 through a context alignment layer (not shown) using an attention mechanism.
  • the generative network of FIG1 can be a generative diffusion network, which has a skip connection (SC) structure between the encoding layer and the decoding layer.
  • the skip connection can be implemented as one or more, for example, skip connection #1, skip connection #2, ..., skip connection #N shown in FIG1.
  • Each skip connection forms a residual structure between the encoder 110 and the decoder 120, so that the generative network obtains a better generalization ability.
  • the cost of the method for adapting such a model to various tasks becomes quite expensive.
  • the various embodiments of the present application provide a series of technical solutions that can improve the efficiency of fine-tuning training and reduce the fine-tuning training overhead of the generative network.
  • FIG2A is a flowchart of the steps of a data processing method according to other embodiments of the present application.
  • the scheme of this embodiment can be applied to any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs, etc.
  • a computing device for example, a data center
  • a CPU an example of a processing unit
  • GPU an example of an acceleration unit
  • Computing devices such as data centers can be deployed in cloud servers such as proprietary clouds, private clouds, or hybrid clouds.
  • a computing device configured with a CPU (an example of a processing unit) + GPU (an example of an acceleration unit) architecture can also be used to perform inference operations.
  • the data processing method of FIG2A includes:
  • S210 Acquire a pre-trained generative model, where the generative model includes a residual structure, where the residual structure includes a skip connection between a coding layer and a decoding layer, and at least one tuning sub-network is formed between the coding layer and the decoding layer.
  • the generative model can be obtained by training a generative network, and the generative network can be a generative diffusion network.
  • the generative model can be a generative model that generates images based on descriptive text, or a text processing model that generates text based on text, such as a knowledge question answering model or a translation model.
  • the generative model may include an encoder and a decoder, and the encoder and the decoder may have a residual structure with one or more skip connections (SC), such as a U-Net structure.
  • SC skip connections
  • the network layer in the encoder is the encoding layer
  • the network layer in the decoder is the decoding layer
  • the encoding layer and the decoding layer connected by the skip connection correspond to each other.
  • the encoder and the decoder may use an attention mechanism or the like to form a context alignment structure including one or more network layers (e.g., a fully connected layer).
  • S220 at least input the description data into the generative model to obtain generated data corresponding to the description data.
  • the description data can describe text or can be converted into description data that describes text.
  • the voice description data can be converted into corresponding description text and input into the generative model.
  • the voice description data can be input into a pre-trained text conversion model to obtain the description text, or the text conversion model can be connected to the input side of the encoder of the generative model.
  • the generative model can also be trained to receive voice description data as input, that is, when the generative model is trained end-to-end, the input side sample of the training sample is a voice sample.
  • the generated data can be a descriptive text (for example, in a knowledge question answering scenario or a translation scenario); the generated data can also be a picture or a video frame, or a video based on a video frame sequence. This depends on the sample type of the end-to-end training used in the training phase of the generative model. If the output side sample is a video frame sequence, the generated data in the inference phase is a video; if the output side sample is a picture or a video frame, the generated data in the inference phase is a picture or a video frame.
  • the pre-trained generative model includes a residual structure
  • the residual structure includes a jump connection between the encoding layer and the decoding layer.
  • At least one tuning sub-network is formed between the encoding layer and the decoding layer. Therefore, at least one tuning sub-network can efficiently train the residual structure by connecting to the encoding layer and the decoding layer of the jump connection, thereby obtaining a generative model with higher prediction reliability, so that more reliable data processing can be performed using the generative model.
  • conditional image data are input into each conditional sub-network respectively.
  • the conditional image can be a description text and multiple additional conditions. Different image conditions are used to describe different types of image features.
  • the multiple types of image feature data include but are not limited to different image features such as image edge detection features, image depth features, image segmentation features, image color features, image extension features, and image restoration features.
  • the description text is input into the encoder of the generative model, and each additional condition is input into the corresponding tuning sub-network of multiple jump connections to obtain the output generated image.
  • the output of the encoding layer is connected to the input of each tuning sub-network 30, and the output of each tuning sub-network is connected to the input of the decoding layer.
  • the output of each tuning sub-network is fused with the input of the encoding layer through the jump connection as the input of the decoding layer of the jump connection. That is:
  • Oj SC is the output of the jth layer in the residual structure
  • X Nj is the input of the jump connection
  • Tj is the output of the tuning sub-network
  • N is the total number of encoder layers in the residual structure.
  • the residual structure further includes at least one conditional subnetwork, and the at least one conditional subnetwork corresponds to at least one tuning subnetwork.
  • the description data can be input into the encoding layer of the generative model
  • the conditional data can be input into at least one conditional subnetwork of the generative model
  • the generated data corresponding to the description data can be obtained from the decoding layer of the generative model.
  • At least one tuning subnetwork corresponds to at least one conditional subnetwork
  • the first output of each conditional subnetwork is connected between the input of the corresponding tuning subnetwork and the input of the encoding layer
  • the second output of each conditional subnetwork is connected between the output of the corresponding tuning subnetwork and the input of the decoding layer.
  • each conditional sub-network is fused with the output of the corresponding tuning sub-network, it is processed through the conditional weights of the conditional sub-network in multiple conditional sub-networks and output to the input of the decoding layer.
  • the description data is a description text
  • the generated data includes a plurality of video frames.
  • the data processing method further includes: performing time-series coding processing on the plurality of video frames to obtain a generated video of the description text.
  • FIG. 2D shows a data processing method according to some other embodiments of the present application.
  • the generative model of this embodiment may include a tuning subnetwork with a conditional subnetwork.
  • at least one tuning subnetwork 30 corresponds to at least one conditional subnetwork 40
  • the first output of each conditional subnetwork 40 is connected between the input of the corresponding tuning subnetwork 30 and the input of the encoding layer
  • the second output of each conditional subnetwork 40 is connected between the output of the corresponding tuning subnetwork 30 and the input of the decoding layer.
  • Oj CSC is the output processed by the tuning subnetwork of the j-th layer in the residual structure; m is the m-th condition, traversing the value between 1 and M; X Nj is the input; Tj m is the tuning subnetwork of the m-th condition of the j-th layer, ⁇ m is the conditional weight coefficient of the m-th condition; Cj is the input after encoding the m-th condition of the j-th layer, and Cj is the condition set of ⁇ Cj0...Cjm ⁇ .
  • the M conditions are conditional data that supplement the input data of the encoder during fine-tuning training.
  • the M conditions may be conditional images that supplement the description text.
  • the number of conditional subnetworks may be consistent with the number of tuning subnetworks, that is, the mth conditional subnetwork corresponds to the mth tuning subnetwork.
  • N jump connections are formed between the multiple encoding layers of the encoder and the multiple decoding layers of the decoder, each jump connection corresponds to M tuning subnetworks and M conditional subnetworks, and the mth conditional subnetwork corresponds to the mth tuning subnetwork.
  • the mth conditional subnetwork includes N network layers arranged sequentially from the input side to the output side, the nth network layer corresponds to the nth jump connection, and the output of the nth network layer is connected to the input of the mth tuning subnetwork of the nth jump connection.
  • N is an integer greater than 1
  • M is an integer greater than 1 ⁇ n ⁇ N, 1 ⁇ m ⁇ M.
  • each network layer can be implemented by a linear layer such as a downsampling layer and an activation layer, and the data dimension of the output of the nth network layer is aligned or consistent with the data dimension of the encoding layer of the nth jump connection, thereby improving the reliability of the training process of the generative model and the reliability of data processing using the generative model.
  • a generated image of a generative model including a tuning subnetwork without a conditional subnetwork can be used as a conditional image of a generative model including a tuning subnetwork with a conditional subnetwork, thereby generating a conditional image through a descriptive text, and further adopting the generative model of the tuning subnetwork with a conditional subnetwork to obtain a generated image by combining both the conditional image and the further descriptive text.
  • the generated image of the generative model of the tuned subnetwork without the conditional subnetwork can be used as input data of the encoder of the generative model of the tuned subnetwork including the conditional subnetwork, so as to further obtain the generated image that better matches the description text by using the attached conditional subnetwork.
  • the intelligent interaction method of FIG3 includes:
  • S320 Input at least the first description data into a pre-trained generative model to obtain first generated data corresponding to the first description data, wherein the first generated data includes at least one of an image, a video frame, and a video, and the generative model includes a residual structure, the residual structure includes a jump connection between the encoding layer and the decoding layer, and at least one tuning subnetwork is formed between the encoding layer and the decoding layer.
  • the description data input by the user is received through the user interface and the data processing results are displayed to the user, thereby realizing quick interaction.
  • this embodiment does not limit the type of description data input by the user, so that the user can input various types of description data such as text description data or voice description data.
  • the generated data presented by the user interface includes but is not limited to text, pictures or videos. Therefore, this embodiment provides users with intelligent information services by combining the convenient interaction method of the user interface and the data processing capability of the generative model. For example, it can be applied to intelligent products such as intelligent assistants and virtual experts.
  • the intelligent interaction method also includes: obtaining second description data that is modified for the first generated data from the interaction interface; at least inputting the second description data into a pre-trained generative model to generate second generated data corresponding to the second description data as a modification result of the first generated data.
  • the first description data is used as context data and based on the second description data as modified description data, the first generated data is adjusted to obtain the second generated description.
  • the types of the first generated data and the second generated data here can be consistent.
  • the voice data is modified to obtain modified voice data
  • the text data is modified to obtain modified text data
  • the video data is modified to obtain modified video data.
  • the types of the first description data and the second description data may be the same or different.
  • the first generated data is a picture or a video frame.
  • Obtaining the second description data modified for the first generated data from the interactive interface includes: obtaining a modification mark for the picture or video from the interactive interface; identifying the modification mark to generate the second description data, or combining the modification mark and the modification description text of the picture or video to generate the second description data.
  • the second description data of the same type as the first description data can be used as a supplement and modification prompt for the first description data
  • the second description data of a different type from the first description data can also be used as a supplement and modification prompt for the first description data.
  • the first description data is text data
  • the text data can be further supplemented to modify the first generated data
  • voice data/operation data different from the text data can be used as a supplement or modification prompt for the text data.
  • the operation data can be the target area marked in the above-mentioned picture or video frame, that is, the target area is the area to be modified or the area with a larger modification weight.
  • FIG4A shows a model training method according to some embodiments of the present application.
  • the model training method of FIG4A includes:
  • S410 Obtaining an initially trained generative network, where the generative network includes an encoder and a decoder, and a skip connection is formed between an encoding layer of the encoder and a decoding layer of the decoder.
  • the initially trained generative network may be a generative network obtained through pre-training, for example, a generative diffusion network.
  • S420 Connect at least one tuning subnetwork with a jump connection between the encoding layer and the decoding layer of the initially trained generative network to obtain a generative network to be fine-tuned, wherein the output of the encoding layer is connected to the input of the at least one tuning subnetwork, and the output of the at least one tuning subnetwork is connected to the input of the decoding layer.
  • the tuning subnetwork can be connected between some or all of the skip-connected encoding layers and decoding layers, and the structure formed by the tuning subnetwork can be called SC-Tuner.
  • the tuning subnetwork includes at least a linear layer and an activation layer.
  • the tuning subnetwork can be formed by a first linear layer, an activation layer, and a second linear layer, and the activation layer is arranged between the first linear layer and the second linear layer.
  • other activation layers can be added to the input side of the first thread layer and/or the output side of the second linear layer, and other linear layers can be added to any position of the tuning subnetwork.
  • the above-mentioned linear layer can be a fully connected layer or a convolutional layer.
  • the linear layer in the tuning subnetwork can be two-dimensional data.
  • the linear layer in the tuning subnetwork can be one-dimensional data. This is because the tuning subnetwork is connected between the encoding layer and the decoding layer of the skip connection, and only needs to be the same dimension as the output data of the decoder.
  • the fine-tuning training of the fine-tuning generative network can be supervised training, and the training samples can include input side samples and output side samples.
  • the input side samples can be text data injected into a specific encoding layer of the encoder, and the input side samples can also include image samples of the encoder input.
  • the output side samples can be text data or image data such as pictures or video frames.
  • At least one tuning subnetwork of the generative network to be fine-tuned is connected between the encoding layer and the decoding layer connected to the jump structure, the output of the encoding layer is connected to the input of at least one tuning subnetwork, and the output of at least one tuning subnetwork is connected to the input of the decoding layer, so that when the generative network is trained by fine-tuning, the jump structure is further optimized by at least one tuning subnetwork, thereby improving the efficiency of fine-tuning training.
  • at least one tuning subnetwork is not in the encoder or the decoder, and there is no need to perform excessive parameter adjustments in the fine-tuning training, thereby saving the computational overhead of the fine-tuning training.
  • the input side sample may be a descriptive text
  • the output side sample may be a generated image.
  • the descriptive text is used to describe the generated image, so that the trained generative model can more effectively adapt to the downstream tasks of the initially trained generative network.
  • each conditional subnetwork is fused with the output of the corresponding tuning subnetwork, it is processed by the conditional weights of the conditional subnetwork in multiple conditional subnetworks and output to the input of the decoding layer.
  • the conditional weights of a conditional subnetwork in multiple conditional subnetworks indicate the importance of the conditional subnetwork.
  • the conditional weights of the conditional subnetworks can also be implemented as linear layers such as fully connected layers, connected between each tuning subnetwork and the corresponding encoding layer. At least one tuning subnetwork and its conditional subnetwork correspond to a jump connection, and the tuning subnetwork with the conditional subnetwork attached can also be referred to as a CSC-Tuner.
  • conditional sub-networks have different functions, that is, multiple conditional sub-networks are used to output various image feature data respectively.
  • the various image feature data include but are not limited to different image features such as image edge detection features, image depth features, image segmentation features, image color features, image expansion features, and image restoration features.
  • a component for fine-tuning the training residual structure can be configured by inputting the skip connection and the related conditions into the SC-Tuner (i.e., the tuning subnetwork of the subnetwork without the condition) and the CSC-Tuner (i.e., the tuning subnetwork of the subnetwork with the condition), and concatenating them with the input in the decoding layer of the original corresponding residual structure and sending them to the next decoding layer.
  • SC-Tuner i.e., the tuning subnetwork of the subnetwork without the condition
  • CSC-Tuner i.e., the tuning subnetwork of the subnetwork with the condition
  • gj +1 Gj [ Oj (XNj -1 + Cj ); gj ], where gj is the input of the decoding layer of the j-th residual structure; Gj is the output of the decoding layer of the j-th residual structure after operation; Oj is the output of the j-th layer in the residual structure after processing by the tuning subnetwork.
  • Multiple skip connections 130 are respectively formed between the multiple encoding layers of the encoder 120 and the multiple decoding layers of the decoder 120.
  • the conditional sub-network 40 includes multiple network layers arranged in sequence from the input side to the output side, and the multiple network layers correspond to the multiple skip connections respectively.
  • the number of conditional subnetworks may be consistent with the number of tuning subnetworks, that is, the mth conditional subnetwork corresponds to the mth tuning subnetwork.
  • N jump connections are formed between the multiple encoding layers of the encoder and the multiple decoding layers of the decoder, each jump connection corresponds to M tuning subnetworks and M conditional subnetworks, and the mth conditional subnetwork corresponds to the mth tuning subnetwork.
  • the mth conditional subnetwork includes N network layers arranged sequentially from the input side to the output side, the nth network layer corresponds to the nth jump connection, and the output of the nth network layer is connected to the input of the mth tuning subnetwork of the nth jump connection.
  • N is an integer greater than 1
  • M is an integer greater than 1 ⁇ n ⁇ N, 1 ⁇ m ⁇ M.
  • each network layer can be implemented by a linear layer such as a downsampling layer and an activation layer, and the data dimension of the output of the nth network layer is aligned or consistent with the data dimension of the encoding layer of the nth jump connection, thereby improving the reliability of the training process of the generative model.
  • the output of the mth conditional subnetwork of the n+1th jump connection is connected to the input of the n+1th network layer, and the mth conditional subnetwork of the nth jump connection is obtained. That is, the output of the n+1th network layer is used as the output of the mth conditional subnetwork of the nth jump connection.
  • FIG4B shows a model training method according to some other embodiments of the present application, including:
  • S470 Obtain an initially trained generative network, where the generative network includes an encoder and a decoder, and a skip connection is formed between an encoding layer of the encoder and a decoding layer of the decoder.
  • S480 Connect at least one tuning subnetwork and at least one conditional subnetwork with jump connections between the encoding layer and the decoding layer of the initially trained generative network to obtain a generative network to be fine-tuned, wherein the output of the encoding layer is connected to the input of at least one tuning subnetwork, and the output of at least one tuning subnetwork is connected to the input of the decoding layer.
  • At least one tuning subnetwork corresponds to at least one conditional subnetwork, and the first output of each conditional subnetwork is connected between the input of the corresponding tuning subnetwork and the input of the encoding layer, and the second output of each conditional subnetwork is connected between the output of the corresponding tuning subnetwork and the input of the decoding layer.
  • S490 Using the input side samples as the input of the encoder, using the output side samples as the output of the decoder, using the conditional samples as the input of at least one conditional sub-network, adjusting the network parameters in at least one tuning sub-network and at least one conditional sub-network, and obtaining a generative model.
  • step S470 is similar to step S410.
  • the tuning subnetwork when the tuning subnetwork is attached with a conditional subnetwork (i.e., CSC-Tuner), as an example of fine-tuning the generative network to be fine-tuned, at least one tuning subnetwork corresponds to at least one conditional subnetwork respectively, and the input side sample can be used as the input of the encoder, the output side sample can be used as the output of the decoder, and the conditional sample can be used as the input of at least one conditional subnetwork, and the network parameters in at least one tuning subnetwork and at least one conditional subnetwork are adjusted to obtain a generative model.
  • CSC-Tuner conditional subnetwork
  • conditional control generative image processing task the condition is first encoded through the cascaded network layer (i.e., Cj), as an input of the tuning subnetwork, and is independently applied to the tuning subnetwork with another input of the tuning subnetwork from the encoding layer, thereby improving the training effect of the tuning subnetwork.
  • Cj cascaded network layer
  • the application development method according to other embodiments of the present application will be described in detail below in conjunction with Figure 5.
  • the application development method of Figure 5 can be applied to PAAS cloud services or IAAS cloud services, that is, the generative model performs the reasoning process on the server side of the PAAS cloud service or IAAS cloud service (its training process can be executed on the server side or on different servers before being deployed to the server side), and the calling interface of the generative model is opened to the tenants of the PAAS cloud service or IAAS cloud service in the form of a virtual machine, that is, the calling interface of the generative model is provided to one or more virtual machines, and the tenant can perform application development in the virtual based on the calling interface.
  • application development methods include:
  • S510 Create a user interface module of the application, where the user interface module is configured to generate description data based at least on user operation data, and return presentation data based on the generated data of the description data.
  • S520 Acquire a calling interface of a generative model, where the generative model is obtained according to a model training method, and the calling interface is configured to return generated data when called;
  • the user interface module is further configured to call the service module to return presentation data
  • the service module is configured to process the generated data to obtain presentation data
  • the calling interface is configured to return the generated data when called by the service module.
  • the processing process of the service module can be partially executed by the generative model, that is, the data processing capability of the service module is enhanced based on the reasoning capability of the generative model.
  • the function function of the service module is configured to call the calling interface of the generative model, or the output data returned by the generative model through the calling interface is used as the input of the function function of the service module.
  • the data processing apparatus according to some other embodiments of the present application will be described below in conjunction with FIG6 .
  • the data processing apparatus in FIG6 corresponds to the data processing method, and the data processing apparatus includes:
  • An acquisition module 610 is configured to acquire a pre-trained generative model, wherein the generative model includes a residual structure, wherein the residual structure includes a skip connection between a coding layer and a decoding layer, and at least one tuning subnetwork is formed between the coding layer and the decoding layer;
  • the generation module 620 is configured to at least input the description data into the generative model to obtain the generated data corresponding to the description data.
  • the residual structure also includes at least one conditional sub-network, which corresponds to at least one tuning sub-network;
  • the generation module is configured to: input the description data into the encoding layer of the generative model, input the conditional data into at least one conditional sub-network of the generative model, and obtain the generated data corresponding to the description data from the decoding layer of the generative model.
  • the at least one tuning subnetwork corresponds to at least one conditional subnetwork, respectively, the first output of each conditional subnetwork is connected between the input of the corresponding tuning subnetwork and the input of the encoding layer, and the second output of each conditional subnetwork is connected between the output of the corresponding tuning subnetwork and the input of the decoding layer.
  • the second output of each conditional sub-network is fused with the output of the corresponding tuning sub-network, processed by the conditional weights of the conditional sub-network in the multiple conditional sub-networks, and output to the input of the decoding layer.
  • the description data is a description text
  • the generated data includes multiple video frames
  • the data processing device also includes: a video processing module, configured to perform time-series encoding processing on the multiple video frames to obtain a generated video of the description text.
  • the intelligent interaction device corresponds to the intelligent interaction method, and specifically, the intelligent interaction device includes:
  • An acquisition module 710 is configured to acquire first description data input in the interactive interface
  • the generation module 720 is configured to input at least the first description data into a pre-trained generative model to obtain first generated data corresponding to the first description data, wherein the first generated data includes at least one of an image, a video frame, and a video, and the generative model includes a residual structure, and the residual structure includes a jump connection between the encoding layer and the decoding layer, and at least one tuning subnetwork is formed between the encoding layer and the decoding layer.
  • the display module 730 is configured to display the first generated data in the question-and-answer interface.
  • the acquisition module is further configured to: acquire second description data modified for the first generated data from the interactive interface.
  • the generation module is further configured to: input at least the second description data into a pre-trained generative model to generate second generated data corresponding to the second description data as a modification result of the first generated data.
  • the first generated data is a picture or a video frame.
  • the acquisition module is further configured to: acquire a modification mark for the picture or video from the interactive interface, identify the modification mark, and generate the second description data, or generate the second description data by combining the modification mark and the modification description text of the picture or video.
  • the model training device can be described in detail below in conjunction with Figure 8.
  • the solution of this embodiment can be applied to any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs, etc.
  • a computing device for example, a data center
  • a CPU an example of a processing unit
  • GPU an example of an acceleration unit
  • Computing devices such as data centers can be deployed in cloud servers such as proprietary clouds, private clouds, or hybrid clouds.
  • a computing device configured with a CPU (an example of a processing unit) + GPU (an example of an acceleration unit) architecture can also be used to perform inference operations.
  • model training device of FIG8 corresponds to the model training method, and the model training device includes:
  • the module acquisition module 810 is configured to acquire an initially trained generative network, where the generative network includes an encoder and a decoder, and a jump connection is formed between the encoding layer of the encoder and the decoding layer of the decoder.
  • the model adjustment module 820 is configured to connect at least one tuning subnetwork of the jump connection between the encoding layer and the decoding layer of the initially trained generative network to obtain a generative network to be fine-tuned, wherein the output of the encoding layer is connected to the input of the at least one tuning subnetwork, and the output of the at least one tuning subnetwork is connected to the input of the decoding layer.
  • the model training module 830 is configured to perform fine-tuning training on the generative network to be fine-tuned based on the training samples to obtain a generative model.
  • At least one tuning sub-network of the generative network to be fine-tuned is connected between the encoding layer and the decoding layer connected to the jump structure, the output of the encoding layer is connected to the input of at least one tuning sub-network, and the output of at least one tuning sub-network is connected to the input of the decoding layer, so that when the generative network is trained by fine-tuning, the jump structure is further optimized by at least one tuning sub-network, thereby improving the efficiency of fine-tuning training.
  • at least one tuning sub-network is not in the encoder or the decoder, and there is no need to perform excessive parameter adjustments in the fine-tuning training, thereby saving the computational overhead of the fine-tuning training.
  • fine-tuning training is performed on the generative network to be fine-tuned based on the training samples, including: performing multiple iterative training on the generative network to be fine-tuned based on the training samples to obtain a generative model, wherein in each iterative training, the difference between the input side sample in the training sample after the forward propagation output of the generative network to be fine-tuned and the input side sample in the training sample is determined, and while keeping the various parameters of the encoder, the difference is back-propagated through the gradients of the decoder and the at least one tuning sub-network to adjust the various parameters in the decoder and the at least one tuning sub-network.
  • the generative network to be fine-tuned is fine-tuned to obtain a generative model, including: taking input side samples as input of the encoder, taking output side samples as output of the decoder, and adjusting network parameters in at least one tuning subnetwork to obtain a generative model.
  • At least one tuning subnetwork corresponds to at least one conditional subnetwork
  • the first output of each conditional subnetwork is connected between the input of the corresponding tuning subnetwork and the input of the encoding layer
  • the second output of each conditional subnetwork is connected between the output of the corresponding tuning subnetwork and the input of the decoding layer
  • the generative network to be fine-tuned is fine-tuned to obtain a generative model, including: taking the input side sample as the input of the encoder, taking the output side sample as the output of the decoder, taking the conditional sample as the input of at least one conditional subnetwork, and adjusting the network parameters in at least one tuning subnetwork and the at least one conditional subnetwork to obtain a generative model.
  • the second output of each conditional sub-network is fused with the output of the corresponding tuning sub-network, processed by the conditional weights of the conditional sub-network in the multiple conditional sub-networks, and output to the input of the decoding layer.
  • N skip connections are formed between the multiple encoding layers of the encoder and the multiple decoding layers of the decoder, each skip connection corresponds to M tuning subnetworks and M conditional subnetworks, and the mth conditional subnetwork corresponds to the mth tuning subnetwork.
  • the mth conditional subnetwork includes N network layers arranged sequentially from the input side to the output side, the nth network layer corresponds to the nth skip connection, and the output of the nth network layer is connected to the input of the mth tuning subnetwork of the nth skip connection.
  • the application development device corresponds to the application development method.
  • the application development device includes:
  • a creation module 910 configured to create a user interface module of an application, wherein the user interface module is configured to generate description data based at least on user operation data, and return presentation data based on the generated data of the description data;
  • An acquisition module 920 is configured to acquire a calling interface of a generative model, wherein the generative model is obtained according to a model training method, and the calling interface is configured to return the generated data when being called;
  • the embedding module 930 is configured to embed at least the calling interface of the generative model into the user interface module.
  • the user interface module is also configured to call a service module to return the presentation data
  • the service module is configured to perform data processing on the generated data to obtain the presentation data
  • the calling interface is configured to return the generated data when called by the service module.
  • each module in the above-mentioned model training device or data processing device can refer to the corresponding description of the corresponding steps in the above-mentioned method embodiment, and has corresponding beneficial effects, which will not be repeated here.
  • the specific working process of the above-described device and module can refer to the corresponding process description in the above-mentioned method embodiment, which will not be repeated here.
  • FIG. 10 there is shown a schematic structural diagram of an electronic device according to another embodiment of the present application.
  • the specific embodiment of the present application does not limit the specific implementation of the electronic device.
  • the electronic device may include: a processor (processor) 1002 for executing a program 1010, a communication interface (Communications Interface) 1004, a memory (memory) 1006, and a communication bus 1008.
  • processor processor
  • Communication interface Communication Interface
  • memory memory
  • the processor, the communication interface, and the memory communicate with each other via a communication bus.
  • Communication interface used to communicate with other electronic devices or servers.
  • the processor is used to execute the program, and specifically can execute the relevant steps in the above method embodiment.
  • the program may include program codes including computer operation instructions.
  • the processor may be a CPU, or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.
  • the one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.
  • the memory is used to store programs.
  • the memory may include high-speed RAM memory and may also include non-volatile memory, such as at least one disk storage.
  • the program may include multiple computer instructions. Specifically, the program may enable the processor to execute operations corresponding to the model training method or data processing method described in any of the aforementioned method embodiments through multiple computer instructions.
  • each step in the program can refer to the corresponding description of the corresponding steps and units in the above method embodiment, and has corresponding beneficial effects, which will not be repeated here.
  • Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiment, which will not be repeated here.
  • the user-related information including but not limited to user device information, user personal information, etc.
  • data including but not limited to sample data used to train the model, data used for analysis, stored data, displayed data, etc.
  • the collection, use and processing of relevant data need to comply with relevant regulations and standards, and corresponding operation entrances are provided for users to choose to authorize or refuse.
  • the above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA)).
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., a random access memory (RAM), a read-only memory (ROM), a flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented.
  • a general purpose computer accesses code for implementing the methods shown herein, the execution of the code transforms the general purpose computer into a special purpose computer for executing the methods shown herein.
  • the pre-trained generative model provided in the embodiment of the present application includes a residual structure, and the residual structure includes a jump connection between the encoding layer and the decoding layer. At least one tuning sub-network is formed between the encoding layer and the decoding layer. Therefore, at least one tuning sub-network can efficiently train the residual structure by connecting to the encoding layer and the decoding layer of the jump connection, thereby obtaining a generative model with higher prediction reliability, so that more reliable data processing can be performed using the generative model.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)

Abstract

一种数据处理、智能交互、模型训练和开发方法、设备和介质。数据处理方法包括：获取预先训练的生成式模型，生成式模型包括残差结构，残差结构包括编码层与解码层之间的跳跃连接，编码层与解码层之间形成至少一个调优子网络；至少将描述数据输入到生成式模型，得到描述数据对应的生成数据。残差结构包括编码层与解码层之间的跳跃连接，编码层与解码层之间形成至少一个调优子网络，因此，至少一个调优子网络能够通过连接到跳跃连接的编码层与解码层之间，高效地训练残差结构，进而得到预测可靠性更高的生成式模型，从而采用生成式模型执行更可靠的数据处理。

Description

数据处理、智能交互、模型训练和开发方法、设备和介质
本申请要求于2023年12月15日提交中国专利局、申请号为202311736419.9、申请名称“数据处理、智能交互、模型训练和开发方法、设备和介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及计算机技术领域，尤其涉及一种数据处理、智能交互、模型训练和开发方法、设备和介质。
 背景技术
在人工智能领域中，诸如大语言模型等生成式模型通过生成式网络构建，诸如扩散生成式网络具有有效的网络结构，通过样本训练之后能够表现出极优的泛化能力。基于大规模数据预训练下的基础生成式模型能够针对诸如图像生成、图像编辑、图像变换等一系列下游任务及应用进行微调训练。
随着生成式模型规模的参数数量不断增长，对这样的模型进行各项任务适配的方法的成本变得相当昂贵，而如果采用低成本的训练方式，则训练效果较差，相应地采用训练后的生成式模型进行数据处理时，可靠性较差。因此，生成式模型的训练效率以及采用生成式模型进行数据处理的可靠性均存在提升的空间。
发明内容
有鉴于此，本申请实施例提供一种数据处理、智能交互、模型训练和开发方法、设备和介质，以至少部分解决上述问题。
根据本申请实施例的第一方面，提供了一种数据处理方法，包括：获取预先训练的生成式模型，所述生成式模型包括残差结构，所述残差结构包括编码层与解码层之间的跳跃连接，所述编码层与所述解码层之间形成至少一个调优子网络；至少将描述数据输入到所述生成式模型，得到所述描述数据对应的生成数据。
根据本申请实施例的第二方面，提供了一种智能交互方法，包括：获取交互界面中输入的第一描述数据；至少将所述第一描述数据输入到预先训练的生成式模型，得到所述第一描述数据对应的第一生成数据，其中，所述第一生成数据包括图像、视频帧、视频中的至少一者，所述生成式模型包括残差结构，所述残差结构包括编码层与解码层之间的跳跃连接，所述编码层与所述解码层之间形成至少一个调优子网络。在所述问答界面中展示所述第一生成数据。
根据本申请实施例的第三方面，提供了一种模型训练方法，包括：获取初始训练的生成式网络，所述生成式网络包括编码器和解码器，所述编码器的编码层与所述解码器的解码层之间形成跳跃连接；在所述初始训练的生成式网络的所述编码层与所述解码层之间连接所述跳跃连接的至少一个调优子网络，得到待微调生成式网络，其中，所述编码层的输出连接到所述至少一个调优子网络的输入，所述至少一个调优子网络的输出连接到所述解码层的输入；基于训练样本，对所述待微调生成式网络进行微调训练，得到生成式模型。
根据本申请实施例的第四方面，提供了一种应用开发方法，包括：创建应用程序的用户界面模块，所述用户界面模块被配置成至少基于用户操作数据生成描述数据，并且基于所述描述数据的生成数据返回呈现数据；获取生成式模型的调用接口，所述生成式模型根据第三方面所述的模型训练方法得到，所述调用接口被配置成在被调用时返回所述生成数据；至少将所述生成式模型的调用接口嵌入到所述用户界面模块中。
根据本申请实施例的第五方面，提供了一种数据处理装置包括：获取模块，设置为获取预先训练的生成式模型，所述生成式模型包括残差结构，所述残差结构包括编码层与解码层之间的跳跃连接，所述编码层与所述解码层之间形成至少一个调优子网络；生成模块，设置为至少将描述数据输入到所述生成式模型，得到所述描述数据对应的生成数据。
根据本申请实施例的第六方面，提供了一种智能交互装置包括：获取模块，设置为获取交互界面中输入的第一描述数据；生成模块，设置为至少将所述第一描述数据输入到预先训练的生成式模型，得到所述第一描述数据对应的第一生成数据，其中，所述第一生成数据包括图像、视频帧、视频中的至少一者，所述生成式模型包括残差结构，所述残差结构包括编码层与解码层之间的跳跃连接，所述编码层与所述解码层之间形成至少一个调优子网络；展示模块，设置为在所述问答界面中展示所述第一生成数据。
根据本申请实施例的第七方面，提供了一种模型训练装置包括：模块获取模块，设置为获取初始训练的生成式网络，生成式网络包括编码器和解码器，编码器的编码层与解码器的解码层之间形成跳跃连接；模型调整模块，设置为在所述初始训练的生成式网络的所述编码层与所述解码层之间连接所述跳跃连接的至少一个调优子网络，得到待微调生成式网络，其中，所述编码层的输出连接到所述至少一个调优子网络的输入，所述至少一个调优子网络的输出连接到所述解码层的输入；模型训练模块，设置为基于训练样本，对所述待微调生成式网络进行微调训练，得到生成式模型。
根据本申请实施例的第八方面，提供了一种应用开发装置包括：创建模块，设置为创建应用程序的用户界面模块，所述用户界面模块被配置成至少基于用户操作数据生成描述数据，并且基于所述描述数据的生成数据返回呈现数据；获取模块，设置为获取生成式模型的调用接口，所述生成式模型根据第三方面所述的模型训练方法得到，所述调用接口被配置成在被调用时返回所述生成数据；嵌入模块，设置为至少将所述生成式模型的调用接口嵌入到所述用户界面模块中。
根据本申请实施例的第九方面，提供了一种电子设备，包括：处理器、存储器、通信接口和通信总线，所述处理器、所述存储器和所述通信接口通过所述通信总线完成相互间的通信；所述存储器用于存放至少一可执行指令，所述可执行指令使所述处理器执行如第一方面所述方法对应的操作。
根据本申请实施例的第十方面，提供了一种计算机存储介质，其上存储有计算机程序，该程序被处理器执行时实现如第一方面所述的方法。
在本实施例中，预先训练的生成式模型包括残差结构，残差结构包括编码层与解码层之间的跳跃连接，编码层与解码层之间形成至少一个调优子网络，因此，至少一个调优子网络能够通过连接到跳跃连接的编码层与解码层之间，高效地训练残差结构，进而得到预测可靠性更高的生成式模型，从而采用生成式模型能够执行更可靠的数据处理。
 附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请实施例中记载的一些实施例，对于本领域普通技术人员来讲，还可以根据这些附图获得其他的附图。
图1为根据一些示例的生成式网络的结构示意图。
图2A为根据本申请的另一些实施例的数据处理方法的步骤流程图。
图2B为图2实施例的一些示例的生成式网络的部分结构示意图。
图2C为图2实施例的另一些示例的生成式网络的部分结构示意图。
图2D为根据本申请的另一些实施例的数据处理方法的步骤流程图。
图2E为图2实施例的另一些示例的生成式网络的部分结构示意图。
图3为根据本申请的另一些实施例的智能交互方法的步骤流程图。
图4A为根据本申请的另一些实施例的模型训练方法的步骤流程图。
图4B为根据本申请的另一些实施例的模型训练方法的步骤流程图。
图5为根据本申请的另一些实施例的应用开发方法的步骤流程图。
图6为根据本申请的另一些实施例的数据处理装置的示意性框图。
图7为根据本申请的另一些实施例的智能交互装置的示意性框图。
图8为根据本申请的另一些实施例的模型训练装置的示意性框图。
图9为根据本申请的另一些实施例的应用开发装置的示意性框图。
图10为根据本申请的另一些实施例的电子设备的结构示意图。
 具体实施方式
为了使本领域的人员更好地理解本申请实施例中的技术方案，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅是本申请实施例一部分实施例，而不是全部的实施例。基于本申请实施例中的实施例，本领域普通技术人员所获得的所有其他实施例，都应当属于本申请实施例保护的范围。
下面结合本申请实施例附图进一步说明本申请实施例具体实现。
诸如大语言模型等生成式模型通过生成式网络构建，例如，图1所示的生成式网络包括编码器110和解码器120，编码器110的输出可以直接连接到解码器120的输入，编码器110的输出也可以通过采用注意力机制的上下文对齐层(未示出)连接到解码器120的输入。图1的生成式网络可以为生成式扩散网络，其具有编码层与解码层之间的跳跃连接(Skip Connection，SC)的结构。跳跃连接可以实现为一个或多个，例如，图1所示的跳跃连接#1、跳跃连接#2、…、跳跃连接#N。各个跳跃连接在编码器110与解码器120之间形成了残差结构，从而使生成式网络获得了更优良的泛化能力。然而，随着生成式模型规模的参数数量不断增长对这样的模型进行各项任务适配的方法的成本变得相当昂贵，本申请的各个实施例提供了一系列的技术方案能够提高微调训练的效率并且减少生成式网络的微调训练开销。
图2A为根据本申请的另一些实施例的数据处理方法的步骤流程图。本实施例的方案可以适用于任意适当的具有数据处理能力的电子设备，包括但不限于：服务器、移动终端(如手机、PAD等)和PC机等。例如，在模型训练(training)阶段，可以利用配置有CPU(处理单元的示例)+GPU(加速单元的示例)架构的计算设备(例如，数据中心)基于训练样本对编码器解码器模型进行训练。诸如数据中心的计算设备可以部署在诸如专有云、私有云、或混合云的云服务器中。相应地，在推理(inference)阶段，也可以利用配置有CPU(处理单元的示例)+GPU(加速单元的示例)架构的计算设备进行推理运算。具体地，图2A的数据处理方法，包括：
S210：获取预先训练的生成式模型，生成式模型包括残差结构，残差结构包括编码层与解码层之间的跳跃连接，编码层与解码层之间形成至少一个调优子网络。
应理解，生成式模型可以通过生成式网络训练得到，生成式网络可以是生成扩散网络。生成式模型可以是根据描述文本来生成图像的生成式模型，也可以是根据文本来生成文本的文本处理模型，例如，知识问答模型或翻译模型。
还应理解，生成式模型可以包括编码器和解码器，编码器与解码器之间可以具有一个或多个跳跃连接(Skip Connection，SC)的残差结构，例如，U-Net结构。编码器中的网络层为编码层解码器中的网络层为解码层，跳跃连接所连接的编码层和解码层对应。编码器与解码器之间可以采用注意力机制等形成包括一个或多个网络层(例如，全连接层)的上下文对齐结构。
S220：至少将描述数据输入到生成式模型，得到描述数据对应的生成数据。
应理解，描述数据可以描述文本或者能够转换成描述文本的描述数据，例如，可以将语音描述数据转换成对应的描述文本，输入到生成式模型中。对于具体的转换方式，可以将语音描述数据输入到预先训练的文本转换模型中，得到描述文本，也可以将文本转换模型连接到生成式模型的编码器的输入侧。可替代地，还可以将生成式模型训练成接收语音描述数据作为输入，也就是说，在对生成式模型进行端到端训练时，训练样本的输入侧样本为语音样本。
还应理解，生成数据可以为描述文本(例如，在知识问答场景或翻译场景中)；生成数据也可以为图片或视频帧，或者基于视频帧序列的视频。这取决于生成式模型的训练阶段所采用的端到端训练的样本类型，如果输出侧样本为视频帧序列，则推理阶段的生成数据为视频，如果输出侧样本为图片或视频帧，则推理阶段的生成数据为图片或视频帧。
在本实施例中，预先训练的生成式模型包括残差结构，残差结构包括编码层与解码层之间的跳跃连接，编码层与解码层之间形成至少一个调优子网络，因此，至少一个调优子网络能够通过连接到跳跃连接的编码层与解码层之间，高效地训练残差结构，进而得到预测可靠性更高的生成式模型，从而采用生成式模型能够执行更可靠的数据处理。
在另一些示例中，各种条件图像数据分别输入到各个条件子网络。条件图像可以为描述文本和多个附加条件，不同图像条件用于描述不同类型的图像特征，多种类型的图像特征数据包括但不限于指示图像边缘检测特征，图像深度特征、图像分割特征、图像颜色特征、图像扩展特征、图像修复特征等不同图像特征。具体地，将描述文本输入到生成式模型的编码器，将每个附加条件分别输入到多个跳跃连接的对应调优子网络中，得到输出的生成图像。
具体而言，在如图2B所示的残差结构(U-Net结构)中，编码层的输出连接到每个调优子网络30的输入，每个调优子网络的输出连接到解码层的输入。也就是说，每个调优子网络的输出与编码层通过跳跃连接的输入进行融合作为跳跃连接的解码层的输入。即：
其中，OjSC为残差结构中的第j层的输出，XN-j为跳跃连接的输入，Tj为调优子网络的输出，N为残差结构中编码器的总层数。
在另一些示例中，残差结构还包括至少一个条件子网络，至少一个条件子网络对应于至少一个调优子网络。作为至少将描述数据输入到生成式模型，得到描述数据对应的生成数据的示例，可以将描述数据输入到生成式模型的编码层，将条件数据输入到生成式模型的至少一个条件子网络，从生成式模型的解码层获得描述数据对应的生成数据。
具体地，至少一个调优子网络分别对应至少一个条件子网络，每个条件子网络的第一输出连接到对应的调优子网络的输入与编码层的输入之间，每个条件子网络的第二输出连接在对应的调优子网络的输出与解码层的输入之间。
更具体地，每个条件子网络的第二输出与对应的调优子网络的输出融合后，经由该条件子网络在多个条件子网络中的条件权重进行处理，输出到解码层的输入。
在另一些示例中，描述数据为描述文本，生成数据包括多个视频帧。数据处理方法还包括：将多个视频帧进行时序编码处理，得到描述文本的生成视频。
图2D示出了根据本申请的另一些实施例的数据处理方法。
S280：获取预先训练的生成式模型。
S290：将描述数据输入到生成式模型的编码层，将条件数据输入到生成式模型的至少一个条件子网络，从生成式模型的解码层获得描述数据对应的生成数据。
应理解，图2D的实施例与图2A的实施例相同或类似的部分，此处不再赘述。例如，步骤S280与步骤S210类似。
进一步地，本实施例的生成式模型可以包括附带条件子网络的调优子网络。例如，如图2C所示，至少一个调优子网络30分别对应至少一个条件子网络40，每个条件子网络40的第一输出连接在对应的调优子网络30的输入与编码层的输入之间，每个条件子网络40的第二输出连接在对应的调优子网络30的输出与解码层的输入之间。
即：其中，OjCSC为残差结构中的第j层的调优子网络处理后的输出；m为第m个条件，遍历1到M之间的值；XN-j为输入；Tjm为第j层第m个条件的调优子网络，αm为第m个条件的条件权重系数；Cj为第j层第m个条件编码后的输入，Cj为{Cj0…Cjm}的条件集合。
其中，M个条件为在微调训练时与编码器的输入数据的补充条件数据。例如，在编码器的输入为描述文本时，M个条件可以为对描述文本进行补充的条件图像。
更进一步地，如图2E所示，对于各个条件子网络(例如，M个条件子网络)，条件子网络的数目可以与调优子网络的数目一致，也就是说，第m条件子网络对应于第m调优子网络。
更具体地，编码器的多个编码层与解码器的多个解码层之间分别形成N个跳跃连接，每个跳跃连接对应M个调优子网络和M个条件子网络，第m个条件子网络对应于第m个调优子网络。第m个条件子网络包括从输入侧到输出侧依次设置的N个网络层，第n个网络层对应于第n跳跃连接，第n个网络层的输出连接到第n跳跃连接的第m个调优子网络的输入。其中，N为大于1的整数，M为大于1的整数，1≤n≤N，1≤m≤M。
进一步地，第n网络层的输出连接到第n跳跃连接的编码层。应理解，每个网络层可以由诸如下采样层的线性层和激活层等实现，第n网络层的输出的数据维度与第n跳跃连接的编码层的数据维度对齐或一致，从而提高了生成式模型的训练过程的可靠性，也提高了采用生成式模型进行数据处理的可靠性。
在数据处理方法的一些变型例中，包括未附带条件子网络的调优子网络的生成式模型的生成图像可以作为包括附带条件子网络的调优子网络的生成式模型的条件图像，从而通过描述文本生成了条件图像，进一步采用附带条件子网络的调优子网络的生成式模型，将条件图像与进一步的描述文本两者得到生成图像。
在数据处理方法的另一些变型例中，未附带条件子网络的调优子网络的生成式模型的生成图像可以作为包括附带条件子网络的调优子网络的生成式模型的编码器的输入数据，从而采用所附带的条件子网络进一步得到更加匹配描述文本的生成图像。
下面将结合图3详细描述本申请的另一些实施例的基于生成式模型的智能交互方法的步骤流程图。图3的智能交互方法包括：
S310：获取交互界面中输入的第一描述数据。
S320：至少将第一描述数据输入到预先训练的生成式模型，得到第一描述数据对应的第一生成数据，其中，第一生成数据包括图像、视频帧、视频中的至少一者，生成式模型包括残差结构，残差结构包括编码层与解码层之间的跳跃连接，编码层与解码层之间形成至少一个调优子网络。
S330：在问答界面中展示第一生成数据。
在本实施例中，通过用户界面来接收用户输入的描述数据并且向用户展示数据处理结果，实现了快捷的交互，此外，本实施例对用户输入的描述数据的类型不作限定，方便用户能够输入诸如文本描述数据或语音描述数据等各种类型的描述数据。此外，用户界面所呈现的生成数据包括不限于文本、图片或视频。因此，本实施例通过结合用户界面的便捷交互方式以及生成式模型的数据处理能力，为用户提供了智能信息服务，例如，可以应用于智能助手、虚拟专家等智能产品中。
在一些示例中，智能交互方法还包括：从交互界面中获取针对第一生成数据进行修改的第二描述数据；至少将第二描述数据输入到预先训练的生成式模型，生成第二描述数据对应的第二生成数据，作为第一生成数据的修改结果。
通过上述数据修改流程，在第一生成数据的初始生成流程之后，以第一描述数据作为上下文数据，基于第二描述数据作为修改描述数据，对第一生成数据进行了调整，得到了第二生成描述。
应理解，此处的第一生成数据和第二生成数据的类型可以一致，例如，对语音数据进行修改得到了修改后的语音数据，对文本数据进行修改得到了修改后的文本数据，对视频数据进行修改得到了修改后的视频数据。
进一步地，第一描述数据和第二描述数据的类型可以相同，也可以不同。例如，在另一些示例中，第一生成数据为图片或视频帧。从交互界面中获取针对第一生成数据进行修改的第二描述数据，包括：从交互界面中获取针对图片或视频的修改标记；识别修改标记，生成第二描述数据，或者，结合修改标记以及图片或视频的修改描述文本，生成第二描述数据。
也就是说，可以采用与第一描述数据的相同类型的第二描述数据，作为第一描述数据的补充和修改提示，也可以采用与第一描述数据的不同类型的第二描述数据，作为第一描述数据的补充和修改提示。例如，在第一描述数据为文本数据时，可以进一步补充文本数据来修改第一生成数据，也可以采用与文本数据不同的语音数据/操作数据来作为文本数据的补充或修改提示，操作数据可以是对上述图片或视频帧中所标记的目标区域，也就是说，目标区域作为待修改的区域或者修改权重较大的区域，通过这样的操作，结合了操作数据的第二描述数据由于提供了更多了修改权重的信息，能够进一步提高修改的精度和可靠性。
具体地，图4A示出了根据本申请的一些实施例的模型训练方法。图4A的模型训练方法包括：
S410：获取初始训练的生成式网络，生成式网络包括编码器和解码器，编码器的编码层与解码器的解码层之间形成跳跃连接。
应理解，初始训练的生成式网络可以是经由预训练得到的生成式网络，例如，生成式扩散网络。
S420：在初始训练的生成式网络的编码层与解码层之间连接跳跃连接的至少一个调优子网络，得到待微调生成式网络，其中，编码层的输出连接到至少一个调优子网络的输入，至少一个调优子网络的输出连接到解码层的输入。
应理解，部分或全部跳跃连接的编码层与解码层之间可以连接至少一个调优子网络，调优子网络形成的结构可以被称为SC-Tuner。调优子网络至少包括线性层和激活层，例如，调优子网络可以形成有第一线性层、激活层和第二线性层形成，激活层设置在第一线性层与第二线性层之间。此外，还可以在第一线程层的输入侧和/或第二线性层的输出侧增加其他激活层，也可以调优子网的任意位置增加其他线性层。上述的线性层可以为全连接层、也可以为卷积层。在待微调生成式网络的输出为图像数据的情况下，调优子网络中的线性层可以是二维数据，在待微调生成式网络的输出为文本数据的情况下，调优子网络中的线性层可以是一维数据，这是因为，调优子网络连接在跳跃连接的编码层与解码层之间，只需与解码器的输出数据的维度相同。
S430：基于训练样本，对待微调生成式网络进行微调训练，得到生成式模型。
应理解，对待微调生成式网络的微调训练可以是监督训练，训练样本可以包括输入侧样本和输出侧样本，输入侧样本可以为注入到编码器中的特定编码层的文本数据，输入侧样本还可以包括编码器的输入的图像样本，输出侧样本可以为文本数据，也可以为诸如图片或视频帧的图像数据。
在本申请实施例的方案中，待微调生成式网络的至少一个调优子网络连接在跳跃结构所连接的编码层与解码层之间，编码层的输出连接到至少一个调优子网络的输入，至少一个调优子网络的输出连接到解码层的输入，使得在通过微调训练生成式网络时，通过至少一个调优子网络进一步地优化跳跃结构，提高了微调训练的效率。此外，至少一个调优子网络未处于编码器中或解码器中，无需在微调训练中过多地进行参数调整，节省了微调训练的计算开销。
进一步地，作为基于训练样本，对待微调生成式网络进行微调训练的示例，可以基于训练样本，对待微调生成式网络进行多次迭代训练，得到生成式模型，其中，在每次迭代训练中，确定训练样本中的输入侧样本经过待微调生成式网络的正向传播输出与训练样本中的输入侧样本之间的差异，并且在保持编码器的各个参数的情况下，通过差异在解码器和至少一个调优子网络中的梯度反向传播，调整解码器和至少一个调优子网络中的各个参数，即，在每次迭代训练中，无需对编码器中的参数进行训练，对编码器与至少一个调优子网络的训练过程进行了解耦，从而显著地减小了微调训练中的计算开销。
具体地，输入侧样本可以为描述文本，输出侧样本可以为生成图像，例如，描述文本用于描述生成图像，从而能够使训练后的生成式模型能够更有效地适配初始训练的生成式网络的下游任务。
例如，以稳定扩散生成式网络为例，诸如U-Net等残差结构中的往往包含12层跳跃连接，在每一层的跳跃连接上加上调优子网络，可针对不同的任务进行相应高效训练。相比于诸如LoRA等其他微调训练方法，能够以更小的调优参数和内存消耗进行训练。也就是说，在高效的生成式调优或少样本任务调优上的应用各个调优子网络，可以适用到少量样本的条件下进行快速定制化的迁移训练，诸如学习到特定的人物特征、特定的图像风格等图像特征。
在调优子网络未附带条件子网络(即，SC-Tuner)的情况下，作为对待微调生成式网络进行微调训练的示例，可以将输入侧样本作为编码器的输入，将输出侧样本作为解码器的输出，对至少一个调优子网络中的网络参数进行调整，得到生成式模型，从而可靠地训练了未附带条件子网络的调优子网络。也就是说，调优子网络的输入侧的约束和输出侧的约束来自于调优子网络对应的跳跃连接的编码层和解码层，相应地，输入侧样本至少包括描述文本，输出侧样本至少包括图像样本，图像样本包括但不限于图片和视频帧。
进一步地，每个条件子网络的第二输出与对应的调优子网络的输出融合后，经由该条件子网络在多个条件子网络中的条件权重进行处理，输出到解码层的输入。应理解，由于多个条件子网络对应的条件的重要性可以是不同，一条件子网络在多个条件子网络中的条件权重指示该条件子网络中的重要性。此外，条件子网络的条件权重也可以实现为诸如全连接层的线性层，连接在各个调优子网络与对应的编码层之间。至少一个调优子网络及其条件子网络对应于跳跃连接，附带条件子网络的调优子网络也可以被称为CSC-Tuner。
进一步地，不同的条件子网络的功能不同，即，多个条件子网络分别用于输出多种图像特征数据，例如，多种图像特征数据包括但不限于指示图像边缘检测特征，图像深度特征、图像分割特征、图像颜色特征、图像扩展特征、图像修复特征等不同图像特征。
不失一般性地，在模型训练框架中，可以配置用于微调训练残差结构的组件，通过将跳跃连接和相关的条件输入到SC-Tuner(即，不附带条件子网络的调优子网络)和CSC-Tuner(即，附带条件子网络的调优子网络)中，并和原始的相应残差结构的解码层中的输入进行拼接送入下一解码层中。
即，gj+1＝Gj[Oj(XN-j-1+Cj)；gj]，其中，gj为第j层的残差结构的解码层的输入；Gj为第j层的残差结构的解码层操作后的输出；Oj为残差结构中的第j层的调优子网络处理后的输出。
编码器120的多个编码层与解码器120的多个解码层之间分别形成多个跳跃连接130，条件子网络40包括从输入侧到输出侧依次设置的多个网络层，多个网络层分别对应于多个跳跃连接。
具体而言，如图2E所示，对于各个条件子网络(例如，M个条件子网络)，条件子网络的数目可以与调优子网络的数目一致，也就是说，第m条件子网络对应于第m调优子网络。
更具体地，编码器的多个编码层与解码器的多个解码层之间分别形成N个跳跃连接，每个跳跃连接对应M个调优子网络和M个条件子网络，第m个条件子网络对应于第m个调优子网络。第m个条件子网络包括从输入侧到输出侧依次设置的N个网络层，第n个网络层对应于第n跳跃连接，第n个网络层的输出连接到第n跳跃连接的第m个调优子网络的输入。其中，N为大于1的整数，M为大于1的整数，1≤n≤N，1≤m≤M。
进一步地，第n网络层的输出连接到第n跳跃连接的编码层。应理解，每个网络层可以由诸如下采样层的线性层和激活层等实现，第n网络层的输出的数据维度与第n跳跃连接的编码层的数据维度对齐或一致，从而提高了生成式模型的训练过程的可靠性。
也就是说，第n+1跳跃连接的第m条件子网络输出连接到第n+1网络层的输入，得到第n跳跃连接的第m条件子网络。即，第n+1网络层的输出作为第n跳跃连接的第m条件子网络的输出。
图4B示出了根据本申请的另一些实施例的模型训练方法，包括：
S470：获取初始训练的生成式网络，生成式网络包括编码器和解码器，编码器的编码层与解码器的解码层之间形成跳跃连接。
S480：在初始训练的生成式网络的编码层与解码层之间连接跳跃连接的至少一个调优子网络和至少一个条件子网络，得到待微调生成式网络，其中，编码层的输出连接到至少一个调优子网络的输入，至少一个调优子网络的输出连接到解码层的输入。至少一个调优子网络分别对应至少一个条件子网络，每个条件子网络的第一输出连接到对应的调优子网络的输入与编码层的输入之间，每个条件子网络的第二输出连接在对应的调优子网络的输出与解码层的输入之间。
S490：将输入侧样本作为编码器的输入，将输出侧样本作为解码器的输出，将条件样本作为至少一个条件子网络的输入，对至少一个调优子网络和至少一个条件子网络中的网络参数进行调整，得到生成式模型。
[根据细则91更正 11.09.2024]
在本实施例中，待微调生成式网络的网络结构可以参考图2C和图2E，此处不再赘述。例如，步骤S470与步骤S410类似。
也就是说，在本实施例中，在调优子网络附带条件子网络(即，CSC-Tuner)的情况下，作为对待微调生成式网络进行微调训练的示例，至少一个调优子网络分别对应至少一个条件子网络，可以将输入侧样本作为编码器的输入，将输出侧样本作为解码器的输出，将条件样本作为至少一个条件子网络的输入，对至少一个调优子网络和至少一个条件子网络中的网络参数进行调整，得到生成式模型。也就是说，在条件控制生成式图像处理任务中，将条件先通过级联的网络层进行编码(即，Cj)，作为调优子网络的一个输入，与调优子网络从编码层的另一输入独立地应用于调优子网络，从而提升了调优子网络的训练效果。
下面将结合图5来详细描述根据本申请的另一些实施例的应用开发方法。图5的应用开发方法可以应用于PAAS云服务或IAAS云服务，即，生成式模型在PAAS云服务或IAAS云服务的服务端执行推理过程(其训练过程可以在该服务端执行或在不同的服务端执行后再部署到该服务端)，并且向PAAS云服务或IAAS云服务的租户以虚拟机的方式开放生成式模型的调用接口，即，生成式模型的调用接口提供给一个虚拟机或多个虚拟机，租户可以基于调用接口在虚拟中执行应用程序的开发。
具体地，应用开发方法包括：
S510：创建应用程序的用户界面模块，用户界面模块被配置成至少基于用户操作数据生成描述数据，并且基于描述数据的生成数据返回呈现数据。
S520：获取生成式模型的调用接口，生成式模型根据模型训练方法得到，调用接口被配置成在被调用时返回生成数据；
S530：至少将生成式模型的调用接口嵌入到用户界面模块中。
在另一些示例中，用户界面模块还被配置成调用服务模块返回呈现数据，服务模块被配置成对生成数据进行数据处理，得到呈现数据，调用接口被配置成在被服务模块调用时返回生成数据。
具体地，用户界面模块用于实现应用程序的与用户交互的界面，可以用于接收用户的输入操作并且向用户提供输出结果，应用程序还包括至少一个服务模块，服务模块与用户界面模块对应，能够执行基于用户的输入进行数据处理，并且得到处理结果。服务模块的处理过程可以完全由生成式模型执行，即，服务模块的输入直接或通过简单预处理作为生成式模型的输入，生成式模块的输出直接或者通过简单的后续处理作为服务模块的输出。可替代地，服务模块的处理过程可以部分有生成式模型执行，即，基于生成式模型的推理能力增强服务模块的数据处理能力，这时，服务模块的功能函数被配置为调用生成式模型的调用接口，或者，生成式模型通过调用接口返回的输出数据作为服务模块的功能函数的输入。
例如，应用程序为内容编辑类应用程序，其中包括诸如字幕生成、文案生成、配音生成等服务模块。相应地，用户界面模块中可以提供内容编辑界面。进一步地，通过嵌入生产式模型的调用接口，可以从调用接口返回生成的字幕数据、文案数据或配音数据等，服务模块中的功能函数能够进一步对字幕数据、文案数据或配音数据等进行基于时间戳的对齐处理，并且将对齐处理结果整体作为服务模块的输出，返回到用户界面模块将对齐处理结果以可视化的方式提供给内容编辑界面，使用户实现了应用程序的高效开发。此外，将虚拟机与生成式模型的调用接口打包提供给租户，在提高应用程序的开发效率的同时还降低了本地开发的资源要求，并且提高了开发的灵活性。
下面将结合图6描述根据本申请的另一些实施例的数据处理装置。图6的数据处理装置与数据处理方法对应，数据处理装置包括：
获取模块610，设置为获取预先训练的生成式模型，所述生成式模型包括残差结构，所述残差结构包括编码层与解码层之间的跳跃连接，所述编码层与所述解码层之间形成至少一个调优子网络；
生成模块620，设置为至少将描述数据输入到所述生成式模型，得到所述描述数据对应的生成数据。
在另一些示例中，所述残差结构还包括至少一个条件子网络，所述至少一个条件子网络对应于至少一个调优子网络；生成模块设置为：将描述数据输入到所述生成式模型的编码层，将条件数据输入到所述生成式模型的至少一个条件子网络，从所述生成式模型的解码层获得所述描述数据对应的生成数据。
在另一些示例中，所述至少一个调优子网络分别对应至少一个条件子网络，每个条件子网络的第一输出连接到对应的调优子网络的输入与所述编码层的输入之间，每个条件子网络的第二输出连接在对应的调优子网络的输出与所述解码层的输入之间。
在另一些示例中，每个条件子网络的第二输出与对应的调优子网络的输出融合后，经由该条件子网络在所述多个条件子网络中的条件权重进行处理，输出到所述解码层的输入。
在另一些示例中，所述描述数据为描述文本，所述生成数据包括多个视频帧，数据处理装置还包括：视频处理模块，设置为将所述多个视频帧进行时序编码处理，得到所述描述文本的生成视频。
下面将结合图7详细描述根据本申请的另一些实施例的智能交互装置。智能交互装置与智能交互方法对应，具体地，智能交互装置包括：
获取模块710，设置为获取交互界面中输入的第一描述数据；
生成模块720，设置为至少将所述第一描述数据输入到预先训练的生成式模型，得到所述第一描述数据对应的第一生成数据，其中，所述第一生成数据包括图像、视频帧、视频中的至少一者，所述生成式模型包括残差结构，所述残差结构包括编码层与解码层之间的跳跃连接，所述编码层与所述解码层之间形成至少一个调优子网络。
展示模块730，设置为在所述问答界面中展示所述第一生成数据。
在另一些示例中，获取模块还设置为：从所述交互界面中获取针对所述第一生成数据进行修改的第二描述数据。生成模块还设置为：至少将所述第二描述数据输入到预先训练的生成式模型，生成所述第二描述数据对应的第二生成数据，作为所述第一生成数据的修改结果。
在另一些示例中，所述第一生成数据为图片或视频帧。获取模块还设置为：从所述交互界面中获取针对所述图片或视频的修改标记，识别所述修改标记，生成所述第二描述数据，或者，结合所述修改标记以及所述图片或视频的修改描述文本，生成所述第二描述数据。
下面将结合图8详细描述根据本申请的另一些实施例的模型训练装置。本实施例的方案可以适用于任意适当的具有数据处理能力的电子设备，包括但不限于：服务器、移动终端(如手机、PAD等)和PC机等。例如，在模型训练(training)阶段，可以利用配置有CPU(处理单元的示例)+GPU(加速单元的示例)架构的计算设备(例如，数据中心)基于训练样本对编码器解码器模型进行训练。诸如数据中心的计算设备可以部署在诸如专有云、私有云、或混合云的云服务器中。相应地，在推理(inference)阶段，也可以利用配置有CPU(处理单元的示例)+GPU(加速单元的示例)架构的计算设备进行推理运算。
具体地，图8的模型训练装置对应模型训练方法对应，模型训练装置包括：
模块获取模块810，设置为获取初始训练的生成式网络，生成式网络包括编码器和解码器，编码器的编码层与解码器的解码层之间形成跳跃连接。
模型调整模块820，设置为在所述初始训练的生成式网络的所述编码层与所述解码层之间连接所述跳跃连接的至少一个调优子网络，得到待微调生成式网络，其中，所述编码层的输出连接到所述至少一个调优子网络的输入，所述至少一个调优子网络的输出连接到所述解码层的输入。
模型训练模块830，设置为基于训练样本，对所述待微调生成式网络进行微调训练，得到生成式模型。
在本申请实施例的方案中，待微调生成式网络的至少一个调优子网络连接在跳跃结构所连接的编码层与解码层之间，编码层的输出连接到至少一个调优子网络的输入，至少一个调优子网络的输出连接到解码层的输入，使得在通过微调训练生成式网络时，通过至少一个调优子网络进一步地优化跳跃结构，提高了微调训练的效率，此外，至少一个调优子网络未处于编码器中或解码器中，无需在微调训练中过多地进行参数调整，节省了微调训练的计算开销。
在另一些示例中，基于训练样本，对所述待微调生成式网络进行微调训练，包括：基于训练样本，对所述待微调生成式网络进行多次迭代训练，得到生成式模型，其中，在每次迭代训练中，确定所述训练样本中的输入侧样本经过所述待微调生成式网络的正向传播输出与所述训练样本中的输入侧样本之间的差异，并且在保持编码器的各个参数的情况下，通过所述差异在所述解码器和所述至少一个调优子网络中的梯度反向传播，调整所述解码器和所述至少一个调优子网络中的各个参数。
在另一些示例中，基于训练样本，对所述待微调生成式网络进行微调训练，得到生成式模型，包括：将输入侧样本作为所述编码器的输入，将输出侧样本作为所述解码器的输出，对至少一个调优子网络中的网络参数进行调整，得到生成式模型。
在另一些示例中，至少一个调优子网络分别对应至少一个条件子网络，每个条件子网络的第一输出连接在对应的调优子网络的输入与所述编码层的输入之间，每个条件子网络的第二输出连接在对应的调优子网络的输出与所述解码层的输入之间；基于训练样本，对所述待微调生成式网络进行微调训练，得到生成式模型，包括：将输入侧样本作为所述编码器的输入，将输出侧样本作为所述解码器的输出，将条件样本作为至少一个条件子网络的输入，对至少一个调优子网络和所述至少一个条件子网络中的网络参数进行调整，得到生成式模型。
在另一些示例中，每个条件子网络的第二输出与对应的调优子网络的输出融合后，经由该条件子网络在所述多个条件子网络中的条件权重进行处理，输出到所述解码层的输入。
在另一些示例中，编码器的多个编码层与所述解码器的多个解码层之间分别形成N个跳跃连接，每个跳跃连接对应M个调优子网络和M个条件子网络，第m个条件子网络对应于第m个调优子网络。第m个条件子网络包括从输入侧到输出侧依次设置的N个网络层，第n个网络层对应于第n跳跃连接，第n个网络层的输出连接到第n跳跃连接的第m个调优子网络的输入。
下面将结合图9描述根据本申请的另一些实施例的应用开发装置。应用开发装置与应用开发方法对应，具体地，应用开发装置包括：
创建模块910，设置为创建应用程序的用户界面模块，所述用户界面模块被配置成至少基于用户操作数据生成描述数据，并且基于所述描述数据的生成数据返回呈现数据；
获取模块920，设置为获取生成式模型的调用接口，所述生成式模型根据模型训练方法得到，所述调用接口被配置成在被调用时返回所述生成数据；
嵌入模块930，设置为至少将所述生成式模型的调用接口嵌入到所述用户界面模块中。
在另一些示例中，所述用户界面模块还被配置成调用服务模块返回所述呈现数据，所述服务模块被配置成对所述生成数据进行数据处理，得到所述呈现数据，所述调用接口被配置成在被所述服务模块调用时返回所述生成数据。
上述的模型训练装置或数据处理装置中各模块的具体实现可以参见上述方法实施例中的相应步骤对应的描述，并具有相应的有益效果，在此不赘述。所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的装置和模块的具体工作过程，可以参考前述方法实施例中的对应过程描述，在此不再赘述。
参照图10，示出了根据本申请的另一实施例的一种电子设备的结构示意图，本申请具体实施例并不对电子设备的具体实现做限定。
如图10所示，该电子设备可以包括：用于执行程序1010的处理器(processor)1002、通信接口(Communications Interface)1004、存储器(memory)1006、以及通信总线1008。
处理器、通信接口、以及存储器通过通信总线完成相互间的通信。
通信接口，用于与其它电子设备或服务器进行通信。
处理器，用于执行程序，具体可以执行上述方法实施例中的相关步骤。
具体地，程序可以包括程序代码，该程序代码包括计算机操作指令。
处理器可能是CPU，或者是特定集成电路ASIC(Application Specific Integrated Circuit)，或者是被配置成实施本申请实施例的一个或多个集成电路。智能设备包括的一个或多个处理器，可以是同一类型的处理器，如一个或多个CPU；也可以是不同类型的处理器，如一个或多个CPU以及一个或多个ASIC。
存储器，用于存放程序。存储器可能包含高速RAM存储器，也可能还包括非易失性存储器(non-volatile memory)，例如至少一个磁盘存储器。
程序可包括多条计算机指令，程序具体可以通过多条计算机指令使得处理器执行前述多个方法实施例中任一实施例所描述的模型训练方法或数据处理方法对应的操作。
程序中各步骤的具体实现可以参见上述方法实施例中的相应步骤和单元中对应的描述，并具有相应的有益效果，在此不赘述。所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的设备和模块的具体工作过程，可以参考前述方法实施例中的对应过程描述，在此不再赘述。
本申请实施例还提供了一种计算机存储介质，其上存储有计算机程序，该程序被处理器执行时实现前述多个方法实施例中任一实施例所描述的方法。该计算机存储介质包括但不限于：只读光盘(Compact Disc Read-Only Memory，CD-ROM)、随机存储器(Random Access Memory，RAM)、软盘、硬盘或磁光盘等。
本申请实施例还提供了一种计算机程序产品，包括计算机指令，该计算机指令指示计算设备执行上述多个方法实施例中的模型训练方法或数据处理方法对应的操作。
此外，需要说明的是，本申请实施例所涉及到的与用户有关的信息(包括但不限于用户设备信息、用户个人信息等)和数据(包括但不限于用于对模型进行训练的样本数据、用于分析的数据、存储的数据、展示的数据等)，均为经用户授权或者经过各方充分授权的信息和数据，并且相关数据的收集、使用和处理需要遵守相关规定和标准，并提供有相应的操作入口，供用户选择授权或者拒绝。
需要指出，根据实施的需要，可将本申请实施例中描述的各个部件/步骤拆分为更多部件/步骤，也可将两个或多个部件/步骤或者部件/步骤的部分操作组合成新的部件/步骤，以实现本申请实施例的目的。
上述根据本申请实施例的方法可在硬件、固件中实现，或者被实现为可存储在记录介质(诸如CD-ROM、RAM、软盘、硬盘或磁光盘)中的软件或计算机代码，或者被实现通过网络下载的原始存储在远程记录介质或非暂时机器可读介质中并将被存储在本地记录介质中的计算机代码，从而在此描述的方法可被存储在使用通用计算机、专用处理器或者可编程或专用硬件(诸如专用集成电路(Application Specific Integrated Circuit，ASIC)或现场可编辑门阵列(Field Programmable Gate Array，FPGA))的记录介质上的这样的软件处理。可以理解，计算机、处理器、微处理器控制器或可编程硬件包括可存储或接收软件或计算机代码的存储组件(例如，随机存储器(Random Access Memory，RAM)、只读存储器(Read-Only Memory，ROM)、闪存等)，当所述软件或计算机代码被计算机、处理器或硬件访问且执行时，实现在此描述的方法。此外，当通用计算机访问用于实现在此示出的方法的代码时，代码的执行将通用计算机转换为用于执行在此示出的方法的专用计算机。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及方法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请实施例的范围。
以上实施方式仅用于说明本申请实施例，而并非对本申请实施例的限制，有关技术领域的普通技术人员，在不脱离本申请实施例的精神和范围的情况下，还可以做出各种变化和变型，因此所有等同的技术方案也属于本申请实施例的范畴，本申请实施例的专利保护范围应由权利要求限定。
 工业实用性
本申请实施例所提供的预先训练的生成式模型包括残差结构，残差结构包括编码层与解码层之间的跳跃连接，编码层与解码层之间形成至少一个调优子网络，因此，至少一个调优子网络能够通过连接到跳跃连接的编码层与解码层之间，高效地训练残差结构，进而得到预测可靠性更高的生成式模型，从而采用生成式模型能够执行更可靠的数据处理。

Claims (17)

  1. 一种数据处理方法，包括：
    获取预先训练的生成式模型，所述生成式模型包括残差结构，所述残差结构包括编码层与解码层之间的跳跃连接，所述编码层与所述解码层之间形成至少一个调优子网络；
    至少将描述数据输入到所述生成式模型，得到所述描述数据对应的生成数据。
  2. 根据权利要求1所述的数据处理方法，其中，所述残差结构还包括至少一个条件子网络，所述至少一个条件子网络对应于至少一个调优子网络；
    至少将描述数据输入到所述生成式模型，得到所述描述数据对应的生成数据，包括：
    将描述数据输入到所述生成式模型的编码层，将条件数据输入到所述生成式模型的至少一个条件子网络，从所述生成式模型的解码层获得所述描述数据对应的生成数据。
  3. 根据权利要求2所述的数据处理方法，其中，所述至少一个调优子网络分别对应至少一个条件子网络，每个条件子网络的第一输出连接到对应的调优子网络的输入与所述编码层的输入之间，每个条件子网络的第二输出连接在对应的调优子网络的输出与所述解码层的输入之间。
  4. 根据权利要求3所述的数据处理方法，其中，每个条件子网络的第二输出与对应的调优子网络的输出融合后，经由该条件子网络在所述多个条件子网络中的条件权重进行处理，输出到所述解码层的输入。
  5. 根据权利要求1所述的数据处理方法，其中，所述描述数据为描述文本，所述生成数据包括多个视频帧，
    所述方法还包括：
    将所述多个视频帧进行时序编码处理，得到所述描述文本的生成视频。
  6. 一种智能交互方法，包括：
    获取交互界面中输入的第一描述数据；
    至少将所述第一描述数据输入到预先训练的生成式模型，得到所述第一描述数据对应的第一生成数据，其中，所述第一生成数据包括图像、视频帧、视频中的至少一者，所述生成式模型包括残差结构，所述残差结构包括编码层与解码层之间的跳跃连接，所述编码层与所述解码层之间形成至少一个调优子网络；
    在所述问答界面中展示所述第一生成数据。
  7. 根据权利要求6所述的智能交互方法，其中，所述方法还包括：
    从所述交互界面中获取针对所述第一生成数据进行修改的第二描述数据；
    至少将所述第二描述数据输入到预先训练的生成式模型，生成所述第二描述数据对应的第二生成数据，作为所述第一生成数据的修改结果。
  8. 根据权利要求6所述的智能交互方法，其中，所述第一生成数据为图片或视频帧，
    从所述交互界面中获取针对所述第一生成数据进行修改的第二描述数据，包括：
    从所述交互界面中获取针对所述图片或视频的修改标记；
    识别所述修改标记，生成所述第二描述数据，或者，结合所述修改标记以及所述图片或视频的修改描述文本，生成所述第二描述数据。
  9. 一种模型训练方法，包括：
    获取初始训练的生成式网络，所述生成式网络包括编码器和解码器，所述编码器的编码层与所述解码器的解码层之间形成跳跃连接；
    在所述初始训练的生成式网络的所述编码层与所述解码层之间连接所述跳跃连接的至少一个调优子网络，得到待微调生成式网络，其中，所述编码层的输出连接到所述至少一个调优子网络的输入，所述至少一个调优子网络的输出连接到所述解码层的输入；
    基于训练样本，对所述待微调生成式网络进行微调训练，得到生成式模型。
  10. 根据权利要求9所述的训练方法，其中，基于训练样本，对所述待微调生成式网络进行微调训练，包括：
    基于训练样本，对所述待微调生成式网络进行多次迭代训练，得到生成式模型，其中，在每次迭代训练中，确定所述训练样本中的输入侧样本经过所述待微调生成式网络的正向传播输出与所述训练样本中的输入侧样本之间的差异，并且在保持编码器的各个参数的情况下，通过所述差异在所述解码器和所述至少一个调优子网络中的梯度反向传播，调整所述解码器和所述至少一个调优子网络中的各个参数。
  11. 根据权利要求9所述的训练方法，其中，基于训练样本，对所述待微调生成式网络进行微调训练，得到生成式模型，包括：
    将输入侧样本作为所述编码器的输入，将输出侧样本作为所述解码器的输出，对所述至少一个调优子网络中的网络参数进行调整，得到生成式模型。
  12. 根据权利要求9所述的训练方法，其中，所述至少一个调优子网络分别对应至少一个条件子网络，每个条件子网络的第一输出连接到对应的调优子网络的输入与所述编码层的输入之间，每个条件子网络的第二输出连接在对应的调优子网络的输出与所述解码层的输入之间；
    基于训练样本，对所述待微调生成式网络进行微调训练，得到生成式模型，包括：
    将输入侧样本作为所述编码器的输入，将输出侧样本作为所述解码器的输出，将条件样本作为至少一个条件子网络的输入，对所述至少一个调优子网络和所述至少一个条件子网络中的网络参数进行调整，得到生成式模型。
  13. 根据权利要求12所述的训练方法，其中，每个条件子网络的第二输出与对应的调优子网络的输出融合后，经由该条件子网络在所述多个条件子网络中的条件权重进行处理，输出到所述解码层的输入。
  14. 一种应用开发方法，包括：
    创建应用程序的用户界面模块，所述用户界面模块被配置成至少基于用户操作数据生成描述数据，并且基于所述描述数据的生成数据返回呈现数据；
    获取生成式模型的调用接口，所述生成式模型根据权利要求9-13中任一项所述的模型训练方法得到，所述调用接口被配置成在被调用时返回所述生成数据；
    至少将所述生成式模型的调用接口嵌入到所述用户界面模块中。
  15. 根据权利要求14所述的方法，其中，所述用户界面模块还被配置成调用服务模块返回所述呈现数据，所述服务模块被配置成对所述生成数据进行数据处理，得到所述呈现数据，所述调用接口被配置成在被所述服务模块调用时返回所述生成数据。
  16. 一种电子设备，包括：处理器、存储器、通信接口和通信总线，所述处理器、所述存储器和所述通信接口通过所述通信总线完成相互间的通信；
    所述存储器用于存放至少一可执行指令，所述可执行指令使所述处理器执行根据权利要求1-15中任一项所述的方法对应的操作。
  17. 一种计算机存储介质，其上存储有计算机程序，该程序被处理器执行时实现根据权利要求1-15中任一项所述的方法。
PCT/CN2024/113737 2023-12-15 2024-08-21 数据处理、智能交互、模型训练和开发方法、设备和介质 Pending WO2025123765A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020267008759A KR20260052168A (ko) 2023-12-15 2024-08-21 데이터 처리, 지능형 인터랙션, 모델 훈련 및 개발 방법, 기기 및 매체

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202311736419.9 2023-12-15
CN202311736419.9A CN120197650A (zh) 2023-12-15 2023-12-15 数据处理、智能交互、模型训练和开发方法、设备和介质

Publications (1)

Publication Number Publication Date
WO2025123765A1 true WO2025123765A1 (zh) 2025-06-19

Family

ID=96056379

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/113737 Pending WO2025123765A1 (zh) 2023-12-15 2024-08-21 数据处理、智能交互、模型训练和开发方法、设备和介质

Country Status (3)

Country Link
KR (1) KR20260052168A (zh)
CN (1) CN120197650A (zh)
WO (1) WO2025123765A1 (zh)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111612790A (zh) * 2020-04-29 2020-09-01 杭州电子科技大学 一种基于t型注意力结构的医学图像分割方法
CN115482242A (zh) * 2021-06-16 2022-12-16 西南科技大学 基于改进U-Net网络的肝脏分割方法
CN117114063A (zh) * 2023-08-30 2023-11-24 北京百度网讯科技有限公司 用于训练生成式大语言模型和用于处理图像任务的方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111612790A (zh) * 2020-04-29 2020-09-01 杭州电子科技大学 一种基于t型注意力结构的医学图像分割方法
CN115482242A (zh) * 2021-06-16 2022-12-16 西南科技大学 基于改进U-Net网络的肝脏分割方法
CN117114063A (zh) * 2023-08-30 2023-11-24 北京百度网讯科技有限公司 用于训练生成式大语言模型和用于处理图像任务的方法

Also Published As

Publication number Publication date
KR20260052168A (ko) 2026-04-17
CN120197650A (zh) 2025-06-24

Similar Documents

Publication Publication Date Title
CN113344794B (zh) 一种图像处理方法、装置、计算机设备及存储介质
CN118042246A (zh) 视频生成方法、装置、电子设备及可读存储介质
CN115134658B (zh) 视频处理方法、装置、设备及存储介质
WO2025060293A1 (zh) 彩色视频生成方法、装置、电子设备及存储介质
WO2024253580A1 (zh) 任务处理方法、系统、平台及自动问答方法
CN116010899A (zh) 多模态数据处理及预训练模型的预训练方法、电子设备
CN110717421A (zh) 一种基于生成对抗网络的视频内容理解方法及装置
CN117835001A (zh) 视频编辑方法、装置、设备和介质
CN120111318A (zh) 视频生成方法及相关设备
CN110166850A (zh) 多重cnn网络预测全景视频观看位置的方法和系统
CN110602163B (zh) 文件上传方法及装置
CN120568108A (zh) 处理音频数据的方法和装置
WO2025152543A1 (zh) 一种模型训练和信息检索的方法及装置
CN117312855B (zh) 用于选择训练数据的方法、装置、电子设备和介质
CN117710527B (zh) 基于人工智能大模型的图像处理方法、装置及产品
CN116939130A (zh) 一种视频生成方法、装置、电子设备和存储介质
CN113992866B (zh) 视频制作方法及装置
CN113778608A (zh) 开发、容器部署、识别、运行方法、装置、电子设备和存储介质
CN114781622A (zh) 图像处理方法和装置、视频处理方法和装置
CN110415318A (zh) 图像处理方法和装置
WO2022227689A1 (zh) 视频处理方法及装置
US12561829B2 (en) Method, computer device, and computer program for providing high-quality image of region of interest by using single stream
WO2025060754A1 (zh) 数据处理方法、电子设备以及计算机可读存储介质
CN113256765A (zh) Ai主播视频的生成方法、装置、电子设备及存储介质
WO2024222206A1 (zh) 音频处理方法、装置、电子设备、计算机可读存储介质及计算机程序产品

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24902191

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2026517552

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 2026517552

Country of ref document: JP

ENP Entry into the national phase

Ref document number: 1020267008759

Country of ref document: KR

Free format text: ST27 STATUS EVENT CODE: A-0-1-A10-A15-NAP-PA0105 (AS PROVIDED BY THE NATIONAL OFFICE)

WWE Wipo information: entry into national phase

Ref document number: 1020267008759

Country of ref document: KR

WWP Wipo information: published in national office

Ref document number: 1020267008759

Country of ref document: KR