CN116757178A - Information processing method and device - Google Patents
Information processing method and device Download PDFInfo
- Publication number
- CN116757178A CN116757178A CN202310538258.6A CN202310538258A CN116757178A CN 116757178 A CN116757178 A CN 116757178A CN 202310538258 A CN202310538258 A CN 202310538258A CN 116757178 A CN116757178 A CN 116757178A
- Authority
- CN
- China
- Prior art keywords
- information
- training
- decoder
- encoder
- processed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000010365 information processing Effects 0.000 title claims abstract description 34
- 238000003672 processing method Methods 0.000 title claims abstract description 23
- 238000012549 training Methods 0.000 claims abstract description 172
- 238000000034 method Methods 0.000 claims abstract description 38
- 230000008569 process Effects 0.000 claims abstract description 28
- 238000004458 analytical method Methods 0.000 claims abstract description 22
- 239000013598 vector Substances 0.000 claims description 88
- 230000006870 function Effects 0.000 claims description 34
- 230000000873 masking effect Effects 0.000 claims description 11
- 238000012545 processing Methods 0.000 claims description 11
- 238000006243 chemical reaction Methods 0.000 claims description 9
- 230000036961 partial effect Effects 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 8
- 230000002829 reductive effect Effects 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 13
- 238000004891 communication Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 239000012634 fragment Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000005057 refrigeration Methods 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 239000000126 substance Substances 0.000 description 2
- 238000010276 construction Methods 0.000 description 1
- 230000001351 cycling effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008451 emotion Effects 0.000 description 1
- 230000008014 freezing Effects 0.000 description 1
- 238000007710 freezing Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
- G06N3/0455—Auto-encoder networks; Encoder-decoder networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/094—Adversarial learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Machine Translation (AREA)
Abstract
The invention discloses an information processing method and device, and relates to the technical field of intelligent information processing. The method embodiment may include: acquiring information to be processed received by a terminal device, and converting the information to be processed into a standard text; analyzing the standard text by using a pre-training language model comprising an encoder and a decoder, wherein the pre-training language model comprising the encoder and the decoder is obtained by training the encoder and the decoder, and noise is added to the encoded information for training the decoder in the process of training the decoder; and generating feedback information corresponding to the information to be processed according to the analysis result, and providing the feedback information to the terminal equipment. According to the embodiment, noise interference can be reduced, so that the user demand can be accurately analyzed, and the user experience of intelligent business service is improved.
Description
Technical Field
The present invention relates to the field of intelligent information processing technologies, and in particular, to an information processing method and apparatus.
Background
In order to enable various intelligent business services such as intelligent customer service, robot service and the like to accurately analyze the demands of users so that the users obtain satisfactory services, relatively accurate natural language models need to be trained, and the trained natural language models are applied to the intelligent business services.
At present, a pre-training language model including an encoder-decoder is commonly used as a natural language model for intelligent business service, however, the existing training mode can make the pre-training language model including the encoder-decoder worse in denoising capability, and in many cases, due to noise interference, the user demand cannot be accurately analyzed, so that the user experience on the intelligent business service is worse.
Disclosure of Invention
In view of this, embodiments of the present invention provide an information processing method and apparatus, which can reduce noise interference, so as to accurately analyze user requirements, thereby improving user experience of intelligent business services.
To achieve the above object, in a first aspect, an embodiment of the present invention provides an information processing method, including:
acquiring information to be processed received by a terminal device, and converting the information to be processed into a standard text;
analyzing the standard text using a pre-training language model comprising an encoder and a decoder, wherein the pre-training language model comprising an encoder and a decoder is obtained by training the encoder and the decoder, and noise is added to the encoded information used for training the decoder during the training of the decoder;
and generating feedback information corresponding to the information to be processed according to the analysis result, and providing the feedback information to the terminal equipment.
Optionally, the converting the information to be processed into standard text includes:
identifying the information type of the information to be processed;
under the condition that the information type of the information to be processed is identified as a picture type, identifying first text information from the information to be processed, and extracting entity information from the first text information;
under the condition that the information type of the information to be processed is recognized as a voice type, converting the information to be processed into second text information, and extracting entity information from the second text information;
and under the condition that the information type of the information to be processed is identified as the text type, directly extracting entity information from the information to be processed.
Optionally, the training the encoder and the decoder includes:
training the encoder by using training data, and acquiring coding information output by the encoder;
adding noise to the encoded information, and inputting the noise-added encoded information to a decoder, so that the decoder is de-noised to train the decoder.
Optionally, the training the encoder with training data includes:
determining training data, and masking part of text included in the training data;
and encoding the training data after the mask processing by using the encoder.
Optionally, the coding information output by the coder comprises a coding vector and a position vector of characters contained in the training data;
the adding noise to the encoded information includes:
randomly adjusting the position vector corresponding to any one or more characters.
Optionally, the training the encoder and the decoder further comprises:
constructing a loss function of the pre-training language model based on a predictive probability function corresponding to each character of the modified position vector;
and adjusting the encoder and the decoder according to the prediction result output by the loss function.
Optionally, the randomly modifying the position vector corresponding to any one or more characters includes:
for the case where the encoded information includes partial encoded information corresponding to one or more masked characters,
and adjusting position vectors in the part of the coded information corresponding to any one or more characters processed by the mask.
Optionally, the adjusting any one or more position vectors in the partial code information corresponding to the mask includes:
the following is performed for each training period:
determining a historical prediction text of a previous training period corresponding to the current training period;
and modifying the position vector in the coding information corresponding to any one or more characters predicted in the historical prediction text, wherein the any one or more characters predicted in the historical prediction text are obtained by predicting the mask.
In a second aspect, an embodiment of the present invention provides an information processing apparatus including: an information conversion module, an information analysis module and an information feedback module, wherein,
the information conversion module is used for acquiring information to be processed received by the terminal equipment and converting the information to be processed into a standard text;
the information analysis module is used for analyzing the standard text by utilizing a pre-training language model comprising an encoder and a decoder, wherein the pre-training language model comprising the encoder and the decoder is obtained by training the encoder and the decoder, and noise is added to the coding information for training the decoder in the process of training the decoder;
the information feedback module is used for generating feedback information corresponding to the information to be processed according to the analysis result and providing the feedback information for the terminal equipment.
One embodiment of the above invention has the following advantages or benefits: the scheme provided by the embodiment of the invention analyzes the standard text converted from the information to be processed by using the pre-training language model comprising the encoder and the decoder, and the pre-training language model comprising the encoder and the decoder is obtained by training the encoder and the decoder, wherein noise is added to the encoded information for training the decoder in the process of training the decoder, so that the decoder can recognize the noise, the noise can be filtered as far as possible, and the feedback information corresponding to the information to be processed, which is generated according to the analysis result, can be compared and attached to the information to be processed, so that noise interference can be reduced, and the user requirement can be accurately analyzed, thereby improving the user experience of intelligent business service.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
fig. 2 is a schematic diagram of a main flow of an information processing method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of the main flow of training an encoder and decoder according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of the main flow involved in training an encoder and decoder process according to the present invention;
fig. 5 is a schematic diagram of a main flow of an information processing method according to an embodiment of the present invention;
fig. 6 is a schematic diagram of main modules of an information processing apparatus according to an embodiment of the present invention;
fig. 7 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
The intelligent scene aimed at by the embodiment of the invention can be used for carrying out targeted push information generation, emotion analysis, intention recognition and the like for intelligent robots, intelligent customer service and the like so as to realize intelligent analysis of user demands and provide feedback information matched with the user demands.
Fig. 1 shows an exemplary system architecture 100 to which an information processing method or an information processing apparatus of an embodiment of the present invention can be applied.
As shown in fig. 1, a system architecture 100 may include terminal devices 101, 102, a network 103, and a server 104. The network 103 is the medium used to provide communication links between the terminal devices 101, 102 and the server 104. The network 1003 may include various connection types such as wired, wireless communication links, or fiber optic cables, among others.
The user can input voice, text, picture waiting process information using the terminal devices 101, 102 and interact with the server 104 through the network 103 so that the terminal devices 101, 102 transmit the voice, text, picture waiting process information input by the user to the server 104. Various applications may be installed on the terminal devices 101, 102, such as instant messaging clients, browsers, mailbox clients, social platform software, etc. (for example only).
The server 104 may be a server that provides various services, such as a background management server (by way of example only) that provides support for information processing. The background management server may set a pre-training language model including an encoder and a decoder, analyze the standard text converted from the information to be processed through the pre-training language model including an encoder and a decoder, generate feedback information matching the information to be processed according to the analysis result, and provide the processing result (e.g., information of an intelligent customer service reply user—only as an example) to the terminal devices 101, 102.
The terminal devices 101, 102 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart robots, desktops, smartphones, tablets, and the like.
It should be noted that, the information processing method provided in the embodiment of the present invention is generally completed by the server 104, and accordingly, each module of the information processing apparatus may be separately provided in the server 104.
It should be understood that the number of development terminals, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Fig. 2 is a schematic flow chart of an information processing method according to an embodiment of the present invention. As shown in fig. 2, the information processing method may include the steps of:
step S201: acquiring information to be processed received by a terminal device, and converting the information to be processed into a standard text;
step S202: analyzing the standard text by using a pre-training language model comprising an encoder and a decoder, wherein the pre-training language model comprising the encoder and the decoder is obtained by training the encoder and the decoder, and noise is added to the encoded information for training the decoder in the process of training the decoder;
step S203: and generating feedback information corresponding to the information to be processed according to the analysis result, and providing the feedback information to the terminal equipment.
The information to be processed in step S201 mainly refers to text, voice, picture, etc. information input by the user through the terminal device. For example, text information related to information consultation, voice information related to information consultation, picture information of an item to be purchased, etc. which are input to the intelligent customer service by the user in the instant messaging client or page.
The standard text related to step S201 mainly refers to a text after deleting virtual words such as "ground", "get", "have", or the like, or a text retaining only entity information. For example, what are the information to be processed for "popular ball games? "its corresponding standard text" popular ball games ". For another example, what is the price of commodity a for the information to be processed? What is the function of commodity a? "its corresponding standard text" commodity A price, commodity A function ".
The pre-training language model comprising an encoder and a decoder is obtained by training the encoder and the decoder, and is mainly obtained by inputting training text samples into the encoder to obtain codes corresponding to the training text samples, and then inputting the codes of the training text samples into the decoder to realize the training of the encoder and the decoder.
The purpose of adding noise is mainly that the training decoder can accurately generate corresponding feedback information for the standard text under the condition that noise exists, so that the accuracy of the pre-training language model is improved.
In the embodiment shown in fig. 2, the pre-training language model including the encoder and the decoder is used to analyze the standard text converted from the information to be processed, and since the pre-training language model including the encoder and the decoder is obtained by training the encoder and the decoder, noise is added to the encoded information for training the decoder in the process of training the decoder, so that the decoder can recognize the noise, and the decoder can filter the noise as much as possible, the feedback information corresponding to the information to be processed generated according to the analysis result can be compared and attached to the information to be processed, noise interference can be reduced, and user requirements can be accurately analyzed, thereby improving user experience of intelligent business services.
The specific embodiment of converting the information to be processed into the standard text in the step S201 may include: identifying the information type of the information to be processed;
under the condition that the information type of the information to be processed is identified as the picture type, identifying first text information from the information to be processed, and extracting entity information from the first text information;
under the condition that the information type of the information to be processed is recognized as the voice type, converting the information to be processed into second text information, and extracting entity information from the second text information;
and under the condition that the information type of the information to be processed is identified as the text type, directly extracting the entity information from the information to be processed.
The information types mainly comprise picture types, voice types, text types, web page links and the like.
The method comprises the steps of identifying information type of information to be processed as picture type, identifying first text information from the information to be processed mainly by adopting the existing picture text identification technology, and the existing technology for identifying text from pictures is not repeated here.
The method mainly adopts the existing technology of converting the information to be processed into the second text information by adopting the existing technology of converting the voice into the text, and the existing technology of converting the voice into the text is not repeated here.
Further, if the information type is a web page link, acquiring information from a page corresponding to the web page link.
Through the above process of processing the information to be processed with different information types, the scheme provided by the embodiment of the invention can feed back the corresponding feedback information aiming at the information to be processed with different information types, thereby meeting the requirements of various information scenes such as task execution instructions issued by users for intelligent robots and the like through voices so as to better meet the user demands of various information scenes
Further, the above information processing method may further include: training the encoder and decoder.
The process for training the encoder and decoder includes the existing training encoder and decoder (e.g., the existing part: the encoder maps the natural language sequence, i.e., the training sample text, to the hidden layer (i.e., the code, including the mathematical expression of the natural language sequence), and then the decoder remaps the hidden layer to the natural language sequence) and the innovative part involved in the scheme provided by the embodiments of the present invention.
Wherein the innovative parts involved in the scheme provided by the embodiments of the present invention are directed to the step of training the encoder and decoder. As shown in fig. 3, the training encoder and decoder embodiments may include the steps of:
step S301: training the encoder by using training data, and acquiring encoding information output by the encoder;
step S302: noise is added to the encoded information, and the noise-added encoded information is input to a decoder to denoise the decoder to train the decoder.
The step S301 may be implemented by using an existing encoder training method, or may add noise to the training data during the process of training the encoder (the process of adding noise to the training data during the process of training the encoder is described in detail in the following embodiments), so as to improve the ability of the encoder to output accurate codes of the information to be processed.
The method and the device are realized by adding noise to the decoder training process, so that the trained decoder can identify the noise or accurately give feedback information corresponding to the information to be processed even under noise interference, and the accuracy of the trained pre-training language model is improved.
Further, an implementation of training an encoder with training data may include: determining training data, and masking part of text included in the training data; the training data after the masking process is encoded by an encoder. For example, the training data is a segment of standard text x= (x) 1 ,x 2 ,…,x n ) Wherein x is i Is the i-th character in a standard text x. After the masking process, the text input into the encoder becomes:
x mask ＝(x 1 ,x 2 ,…,x k ,…,x l ,…,x n ) Wherein (x) k ,…,x l ) I.e. a masked text fragment.
Noise is introduced into the process of training the encoder through the mask processing time limit, so that the noise interference eliminating capacity of the trained pre-training language model is improved, and the accuracy of the pre-training language model is improved.
The coding information output by the coder can comprise coding vectors and position vectors of characters contained in training data; on this basis, adding noise to the encoded information may include: randomly adjusting the position vector corresponding to any one or more characters.
I.e. the input encoder and decoder are both vector information. That is, the substance of the input encoder is a vector of standard text or training data, and the substance of the input decoder is a vector corresponding to the output result of the encoder. Wherein, whether the vector is input into the encoder or the vector is input into the decoder is generally composed of two parts, namely, the code vector corresponding to the character and the position vector corresponding to the character, which are searched from the code table, namely:
the vector input into the encoder or the vector input into the decoder may be as shown in the following formula (1).
Wherein the method comprises the steps ofA vector representing an i-th character corresponding to the text x in the input encoder or the input decoder;a code vector representing an ith character corresponding to text x; />Representing a position vector corresponding to the ith character of text x.
The vector in the input encoder is text x mask ＝(x 1 ,x 2 ,…,x k ,…,x l ,…,x n ) Corresponding vector, where (x k ,…,x l ) I.e. a masked text fragment. In training the encoder and decoder, the masked text segment (x k ,…,x l ) Noise is added to the vector input to the decoder during training, e.g. for character x for which a mask has been predicted k The corresponding real character is retrained to the character x in the retrained process k Corresponding trueThe position vector corresponding to the real character adds noise. The following describes a specific example. For example, training data "Chinese national ball table tennis" is processed by the masking process to obtain masked text "Chinese national ball [ MASK ]][MASK][MASK]The vector corresponding to the text after the MASK is input into an encoder, and the encoder generates the code corresponding to' Chinese ball ping [ MASK ]][MASK]"encoded vector or" Chinese national table tennis [ MASK ]]"encoded vector or encoded vector corresponding to" chinese ball table tennis ". The encoded vector is a character vector and a position vector for each character and mask.
Randomly adjusting the position vector corresponding to any one or more characters, e.g. for "Chinese ball ping [ MASK ]][MASK]"the encoded vector comprises: each character "middle", "country", "ball", "ping", "[ MASK"]"and" [ MASK ]]"corresponding vector. Wherein the vector of the character 'ping' is E Table tennis ＝T Table tennis +P 5 I.e. the vector of character "ping" is the coded vector T of the character Table tennis The sum character "ping" is the position vector P of the 5 th bit character 5 . Adding noise to it, the vector of "ping" can be adjusted to E Table tennis ＝T Table tennis +P 9 I.e., randomly adjusting the position vector of the "ping" to a vector corresponding to position 9. It should be noted that the adjusted character may be any other character, such as the 3 rd character, "country", so that the decoder training process reduces the dependence on the previous word, to further improve the accuracy of the decoder.
Further, to further increase the effectiveness of noise introduction, embodiments of the randomly modifying the position vector corresponding to any one or more characters may include: for the case where the encoded information includes partial encoded information corresponding to one or more masked characters, the position vector in the partial encoded information corresponding to any one or more masked characters is adjusted. For example, for a vector whose code information is "chinese ball ping [ MASK ]" where the character "ping" is a character processed by a MASK in "chinese ball [ MASK ]," the position vector in the vector corresponding to the character "ping" can be added to noise. For another example, for a vector encoded with "chinese ball ping-pong [ MASK ]" as the encoding information, where the characters "ping" and "pong" are the characters processed by the MASK in "chinese ball [ MASK ]," the position vectors in the vector corresponding to the characters "ping" and/or "pong" may be added with noise. For another example, for a vector encoded with "chinese ball ping-pong" as the encoding information, where the characters "ping", "pong" and "ball" are the characters processed by the MASK in "chinese ball [ MASK ], the position vectors in the vectors corresponding to the characters" ping "and/or" pong "and/or" ball "may be added with noise.
The noise can be added in a targeted manner by adjusting the position vector in the part of the coding information corresponding to any one or more characters processed by the mask, so that training is more targeted, and the accuracy of the trained decoder is effectively improved.
Additionally, embodiments of adjusting the position vector in any one or more of the partial encoded information corresponding to the mask may include: the following is performed for each training period:
determining a historical prediction text of a previous training period corresponding to the current training period; and modifying the position vector in the coding information corresponding to any one or more characters predicted in the historical prediction text, wherein the any one or more characters predicted in the historical prediction text are obtained through mask prediction.
For example, the vector input to the decoder in the first training period is the vector corresponding to "chinese ball [ MASK ]" and the obtained prediction text "chinese ball ping [ MASK ]", the position vector of "ping" in the vector corresponding to "chinese ball ping [ MASK ]" input in the second training period can be added with noise, so that the second training period is used to obtain the text "chinese ball ping [ MASK ]", which can still be obtained without the prompt of "ping". The noise can be added to the position vector of ping-pong and/or ping-pong in the vector corresponding to the vector ' Chinese national ball ping-pong ' input in the third training period, so that the text ' Chinese national ball ping-pong ' can be still obtained under the prompt of not obtaining ping-pong ' and/or ping-pong in the third training period, and the accuracy of training out the decoder is effectively improved.
Further, as shown in fig. 4, the training encoder and decoder may further include the steps of:
step S401: constructing a loss function of the pre-training language model based on the predictive probability function corresponding to each character of the modified position vector;
wherein, the loss function constructed in this step is as follows formula (2):
wherein L is t Representing a target loss value of the trained pre-training language model; p (x) t |x ＜k-1 ,x′ k ,…,x′ ＜t-1 ) Representing the predicted character x at time t t Probability of x ＜k-1 Representing unmasked text segments in the text sequence; x's' k ，...，x′ ＜t-1 A text segment representing a masked text segment in the text sequence input to the encoder and having been decoded by the decoder before time t and subsequently rejoined with position noise. For example, "Chinese ball table tennis", when t=7, P (x t |x ＜k-1 ,x′ k ,…,x′ ＜t-1 ) =p ("ball" | "chinese ball×"), i.e. the probability that the 7 th character is a "ball" after the 5 th and 6 th characters are noisy.
Step S402: and adjusting the encoder and the decoder according to the prediction result output by the loss function.
Through the loss function, the anti-noise interference capability of the decoder can be reflected more truly, and the encoder and the decoder are adjusted so as to further enhance the anti-noise interference capability of the decoder, thereby improving the accuracy of feedback information.
The specific implementation steps of the information processing method provided by the implementation of the invention in the intelligent customer service scene are described in detail below by taking the intelligent customer service scene as an example. As shown in fig. 5, the information processing method in the intelligent customer service scenario may include the following steps:
step S501: acquiring a training data sample, and converting the training data sample into a training standard text;
for example, one sample included in the training data sample is "the function of the refrigerator of model a is mainly refrigerated, preserved, frozen", and the corresponding standard text is "the function of the refrigerator of model a is refrigerated, preserved and frozen".
Step S502: masking a part of text included in the training data;
for example, MASK processing is performed on the "model A refrigerator function refrigeration fresh-keeping freezing" to obtain the "model A refrigerator function [ MASK ] [ MASK ] [ MASK ] [ MASK ] [ MASK ]".
Step S503: encoding the training data after mask processing by an encoder;
i.e. the vector corresponding to the training data after masking is encoded, which step mainly trains the encoder.
The coding information output by the coder comprises coding vectors and position vectors of characters contained in training data;
step S504: acquiring coding information output by an encoder, and randomly adjusting position vectors corresponding to any one or more characters in the coding information;
in this step, for the case where the encoded information includes partial encoded information corresponding to one or more mask-processed characters, the position vector in the partial encoded information corresponding to any one or more mask-processed characters is adjusted.
For example, for the vector of the input encoder sample "model A refrigerator function [ MASK ] [ MASK ] [ MASK ] [ MASK ] [ MASK ]," the encoding information output by the encoder is the encoding information corresponding to "model A refrigerator function refrigeration [ MASK ] [ MASK ] [ MASK ] [ MASK ]", the 'cold storage' is a MASK in the process of inputting the encoder, and the position vector of any one or two characters in the 'cold storage' in the 'model A refrigerator function cold storage [ MASK ] [ MASK ] [ MASK ] [ MASK ]' can be added with noise.
The above description is made with only one sample, and the sample input in the training process is a large amount of training data.
Step S505: inputting the noise-added encoded information into a decoder to make the decoder denoise so as to train the decoder;
step S506: constructing a loss function of the pre-training language model based on the predictive probability function corresponding to each character of the modified position vector;
step S507: adjusting the encoder and the decoder according to the prediction result output by the loss function to obtain a pre-training language model;
the pre-training language model is obtained by cycling through steps S501 to S507 in a plurality of training periods.
Step S508: acquiring information to be processed received by a terminal device, and converting the information to be processed into a standard text;
for example, a function of a refrigerator of a model B is received, wherein the function is input by a user through an intelligent customer service interactive interface to inquire or inquire. And the standard text converted from the information to be processed is used for realizing the model B refrigerator performance.
Step S509: analyzing the standard text using a pre-trained language model comprising an encoder and a decoder;
step S510: and generating feedback information corresponding to the information to be processed according to the analysis result, and providing the feedback information to the terminal equipment.
The feedback information is obtained by searching information related to the model B refrigerator performance after the pre-training language model recognizes that the user inquires or inquires the model B refrigerator performance, and the feedback information is fed back to the terminal equipment.
Fig. 6 is a schematic structural diagram of an information processing apparatus according to an embodiment of the present invention. As shown in fig. 6, the information processing apparatus 600 may include: an information conversion module 601, an information analysis module 602, and an information feedback module 603, wherein,
the information conversion module 601 is configured to obtain information to be processed received by the terminal device, and convert the information to be processed into a standard text;
an information analysis module 602, configured to analyze a standard text using a pre-training language model including an encoder and a decoder, where the pre-training language model including the encoder and the decoder is obtained by training the encoder and the decoder, and add noise to the encoded information for training the decoder during the training of the decoder;
the information feedback module 603 is configured to generate feedback information corresponding to the information to be processed according to the analysis result, and provide the feedback information to the terminal device.
In the embodiment of the present invention, the information conversion module 601 is further configured to identify an information type of the information to be processed; under the condition that the information type of the information to be processed is identified as the picture type, identifying first text information from the information to be processed, and extracting entity information from the first text information; under the condition that the information type of the information to be processed is recognized as the voice type, converting the information to be processed into second text information, and extracting entity information from the second text information; and under the condition that the information type of the information to be processed is identified as the text type, directly extracting the entity information from the information to be processed.
In an embodiment of the present invention, as shown in fig. 6, the information processing apparatus 600 may further include: the training module 604, wherein,
training module 604, which trains the encoder by using training data and obtains the encoding information output by the encoder; noise is added to the encoded information, and the noise-added encoded information is input to a decoder to denoise the decoder to train the decoder.
In the embodiment of the present invention, the training module 604 is further configured to determine training data, and perform mask processing on a portion of text included in the training data; the training data after the masking process is encoded by an encoder.
In the embodiment of the invention, the coding information output by the coder comprises coding vectors and position vectors of characters contained in training data; training module 604 is further configured to randomly adjust a position vector corresponding to any one or more characters.
In the embodiment of the present invention, the training module 604 is further configured to construct a loss function of the pre-training language model based on the predictive probability function corresponding to each character of the modified position vector; and adjusting the encoder and the decoder according to the prediction result output by the loss function.
In the embodiment of the present invention, the training module 604 is further configured to adjust, for a case where the encoded information includes part of the encoded information corresponding to the one or more characters subjected to masking, a position vector in the part of the encoded information corresponding to the one or more characters subjected to masking.
In an embodiment of the present invention, the training module 604 is further configured to perform the following operations for each training period: determining a historical prediction text of a previous training period corresponding to the current training period; and modifying the position vector in the coding information corresponding to any one or more characters predicted in the historical prediction text, wherein the any one or more characters predicted in the historical prediction text are obtained through mask prediction.
Referring now to FIG. 7, there is illustrated a schematic diagram of a computer system 700 suitable for use in implementing a terminal device or server in accordance with an embodiment of the present invention. The terminal device or server shown in fig. 7 is only an example, and should not impose any limitation on the functions and scope of use of the embodiments of the present invention.
As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, ROM 702, and RAM 703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
The following components are connected to the I/O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output portion 707 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 708 including a hard disk or the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. The drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read therefrom is mounted into the storage section 708 as necessary.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 709, and/or installed from the removable medium 711. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 701.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor, for example, as: a processor comprises an information conversion module, an information analysis module and an information feedback module. The names of these modules do not constitute limitations on the module itself in some cases, and for example, the information conversion module may also be described as "a module that converts information to be processed into standard text".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to include: acquiring information to be processed received by a terminal device, and converting the information to be processed into a standard text; analyzing the standard text by using a pre-training language model comprising an encoder and a decoder, wherein the pre-training language model comprising the encoder and the decoder is obtained by training the encoder and the decoder, and noise is added to the encoded information for training the decoder in the process of training the decoder; and generating feedback information corresponding to the information to be processed according to the analysis result, and providing the feedback information to the terminal equipment.
According to the technical scheme, the pre-training language model comprising the encoder and the decoder is utilized to analyze the standard text converted from the information to be processed, and the pre-training language model comprising the encoder and the decoder is obtained by training the encoder and the decoder, wherein noise is added to the encoded information for training the decoder in the process of training the decoder, so that the decoder can recognize the noise as far as possible, the noise can be filtered out, and the feedback information corresponding to the information to be processed, which is generated according to the analysis result, can be compared and attached to the information to be processed, so that noise interference can be reduced, and user requirements can be accurately analyzed, and therefore user experience of intelligent business service is improved.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (11)
1. An information processing method, characterized by comprising:
acquiring information to be processed received by a terminal device, and converting the information to be processed into a standard text;
analyzing the standard text using a pre-training language model comprising an encoder and a decoder, wherein the pre-training language model comprising an encoder and a decoder is obtained by training the encoder and the decoder, and noise is added to the encoded information used for training the decoder during the training of the decoder;
and generating feedback information corresponding to the information to be processed according to the analysis result, and providing the feedback information to the terminal equipment.
2. The information processing method according to claim 1, wherein the converting the information to be processed into standard text includes:
identifying the information type of the information to be processed;
under the condition that the information type of the information to be processed is identified as a picture type, identifying first text information from the information to be processed, and extracting entity information from the first text information;
under the condition that the information type of the information to be processed is recognized as a voice type, converting the information to be processed into second text information, and extracting entity information from the second text information;
and under the condition that the information type of the information to be processed is identified as the text type, directly extracting entity information from the information to be processed.
3. The information processing method according to claim 1, wherein the training of the encoder and the decoder includes:
training the encoder by using training data, and acquiring coding information output by the encoder;
adding noise to the encoded information, and inputting the noise-added encoded information to a decoder, so that the decoder is de-noised to train the decoder.
4. The information processing method according to claim 3, wherein the training the encoder using training data includes:
determining training data, and masking part of text included in the training data;
and encoding the training data after the mask processing by using the encoder.
5. The information processing method according to claim 3 or 4, wherein,
the coding information output by the coder comprises coding vectors and position vectors of characters contained in the training data;
the adding noise to the encoded information includes:
randomly adjusting the position vector corresponding to any one or more characters.
6. The information processing method according to claim 5, wherein the training the encoder and the decoder further comprises:
constructing a loss function of the pre-training language model based on a predictive probability function corresponding to each character of the modified position vector;
and adjusting the encoder and the decoder according to the prediction result output by the loss function.
7. The information processing method according to claim 5, wherein randomly modifying the position vector corresponding to any one or more of the characters comprises:
for the case where the encoded information includes partial encoded information corresponding to one or more masked characters,
and adjusting position vectors in the part of the coded information corresponding to any one or more characters processed by the mask.
8. The information processing method according to claim 7, wherein the adjusting of the position vector in any one or more pieces of the partial code information corresponding to the mask includes:
the following is performed for each training period:
determining a historical prediction text of a previous training period corresponding to the current training period;
and modifying the position vector in the coding information corresponding to any one or more characters predicted in the historical prediction text, wherein the any one or more characters predicted in the historical prediction text are obtained by predicting the mask.
9. An information processing apparatus, characterized by comprising: an information conversion module, an information analysis module and an information feedback module, wherein,
the information conversion module is used for acquiring information to be processed received by the terminal equipment and converting the information to be processed into a standard text;
the information analysis module is used for analyzing the standard text by utilizing a pre-training language model comprising an encoder and a decoder, wherein the pre-training language model comprising the encoder and the decoder is obtained by training the encoder and the decoder, and noise is added to the coding information for training the decoder in the process of training the decoder;
the information feedback module is used for generating feedback information corresponding to the information to be processed according to the analysis result and providing the feedback information for the terminal equipment.
10. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs,
when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-8.
11. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310538258.6A CN116757178A (en) | 2023-05-12 | 2023-05-12 | Information processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310538258.6A CN116757178A (en) | 2023-05-12 | 2023-05-12 | Information processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116757178A true CN116757178A (en) | 2023-09-15 |
Family
ID=87950378
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310538258.6A Pending CN116757178A (en) | 2023-05-12 | 2023-05-12 | Information processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116757178A (en) |
-
2023
- 2023-05-12 CN CN202310538258.6A patent/CN116757178A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10762305B2 (en) | Method for generating chatting data based on artificial intelligence, computer device and computer-readable storage medium | |
| CN108419094B (en) | Video processing method, video retrieval method, device, medium and server | |
| CN110309275B (en) | Dialog generation method and device | |
| CN112183084B (en) | Audio and video data processing method, device and equipment | |
| CN109920431B (en) | Method and apparatus for outputting information | |
| CN113362811B (en) | Training method of voice recognition model, voice recognition method and device | |
| CN112650841A (en) | Information processing method and device and electronic equipment | |
| CN113327599B (en) | Voice recognition method, device, medium and electronic equipment | |
| CN113436620A (en) | Model training method, speech recognition method, device, medium and equipment | |
| CN114724168A (en) | Training method of deep learning model, text recognition method, text recognition device and text recognition equipment | |
| CN115309877A (en) | Dialog generation method, dialog model training method and device | |
| CN114495977B (en) | Speech translation and model training method, device, electronic equipment and storage medium | |
| CN111368551A (en) | Method and device for determining event subject | |
| CN114841142A (en) | Text generation method and device, electronic equipment and storage medium | |
| US20220059107A1 (en) | Method, apparatus and system for hybrid speech synthesis | |
| CN113571044A (en) | Voice information processing method and device and electronic equipment | |
| CN110852801B (en) | Information processing method, device and equipment | |
| CN116958738A (en) | Training method and device of picture recognition model, storage medium and electronic equipment | |
| CN111027495A (en) | Method and device for detecting key points of human body | |
| CN116757178A (en) | Information processing method and device | |
| CN115619142A (en) | Method, device, equipment and computer readable medium for matching data | |
| CN115129845A (en) | Text information processing method and device and electronic equipment | |
| CN112149426B (en) | Reading task processing method and related equipment | |
| CN113076932A (en) | Method for training audio language recognition model, video detection method and device thereof | |
| CN110209878B (en) | Video processing method and device, computer readable medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |