WO2025000910A1 - 识别关键词类别的方法、装置及电子设备 - Google Patents

识别关键词类别的方法、装置及电子设备 Download PDF

Info

Publication number
WO2025000910A1
WO2025000910A1 PCT/CN2023/136792 CN2023136792W WO2025000910A1 WO 2025000910 A1 WO2025000910 A1 WO 2025000910A1 CN 2023136792 W CN2023136792 W CN 2023136792W WO 2025000910 A1 WO2025000910 A1 WO 2025000910A1
Authority
WO
WIPO (PCT)
Prior art keywords
text information
processing layer
layer
text
keywords
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/136792
Other languages
English (en)
French (fr)
Inventor
付薇薇
刘欣璋
陈诣文
方瑞玉
宋双永
张寅�
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Telecom Corp Ltd
Original Assignee
China Telecom Corp Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Telecom Corp Ltd filed Critical China Telecom Corp Ltd
Publication of WO2025000910A1 publication Critical patent/WO2025000910A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/35—Clustering; Classification
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis

Definitions

  • the present disclosure relates to the field of communication technology, and in particular to a method, device and electronic device for identifying keyword categories.
  • Communication Assistant is a value-added service application developed for users of the Communication Assistant business in the mobile Internet era. It provides functions such as missed call inquiry, address book backup, SMS backup, schedule reminders, private space, black and white lists, etc.
  • an embodiment of the present disclosure provides a method for identifying keyword categories, the method comprising: obtaining text information to be identified; and inputting the text information into a target model, and outputting the keyword category to which each element in the text information belongs; wherein the target model comprises a first processing layer for extracting semantic features of the text information, a second processing layer for extracting target keywords existing in a business dictionary from the text information, and a third processing layer for determining the keyword category to which each element in the text information belongs based on the semantic features and the target keywords, the first processing layer and the second processing layer being arranged in parallel; the business dictionary comprises a plurality of keywords that have been identified in a business process.
  • an embodiment of the present disclosure provides a device for identifying keyword categories, the device comprising: a first acquisition module, used to acquire text information to be identified; and a recognition module, used to input the text information into a target model and output the keyword category to which each element in the text information belongs; wherein the target model
  • the type includes a first processing layer for extracting semantic features of the text information, a second processing layer for extracting target keywords in the text information that exist in a business dictionary, and a third processing layer for determining the keyword category to which each element in the text information belongs based on the semantic features and the target keywords, wherein the first processing layer is arranged in parallel with the second processing layer; the business dictionary includes multiple keywords that have been identified in the business process.
  • an embodiment of the present disclosure provides an electronic device comprising a memory, a transceiver and a processor: the memory is used to store a computer program; the transceiver is used to send and receive data under the control of the processor; the processor is used to read the computer program in the memory and execute the method for identifying keyword categories described in the first aspect above.
  • an embodiment of the present disclosure provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, wherein the computer program is used to enable a processor to execute the method for identifying keyword categories described in the first aspect.
  • an embodiment of the present disclosure provides a computer program, comprising: instructions, which, when executed by a processor, cause the processor to execute the method for identifying keyword categories described in the first aspect.
  • FIG1 is a flow chart of a method for identifying keyword categories provided by an embodiment of the present disclosure
  • FIG2 is a schematic diagram of a mask operation during pre-training in the related art
  • FIG3 is a schematic diagram of a MASK operation during pre-training in an embodiment of the present disclosure
  • FIG4 is a schematic diagram of the architecture of a target model in a specific implementation of the method for identifying keyword categories according to an embodiment of the present disclosure
  • FIG5 is a structural block diagram of an apparatus for identifying keyword categories provided by an embodiment of the present disclosure.
  • FIG6 is a structural block diagram of an electronic device provided in an embodiment of the present disclosure.
  • the term "and/or” describes the association relationship of associated objects, indicating that there may be three types of relationships, for example: For example, A and/or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone.
  • the character "/" generally indicates that the related objects are in an "or” relationship.
  • the term “plurality” refers to two or more than two, and other quantifiers are similar.
  • the inventors of the present disclosure have discovered that, in related technologies, the categories to which each element belongs are identified only based on the semantic features of text information, which may lead to misidentification in different business scenarios of communication assistants, thereby causing incorrect extraction of keywords and reducing the business experience.
  • an embodiment of the present disclosure provides a method for identifying keyword categories to improve the accuracy of extracting keywords from text information.
  • an embodiment of the present disclosure provides a method for identifying keyword categories. As shown in FIG1 , the method includes the following steps 101 to 102 .
  • Step 101 Obtain text information to be recognized.
  • the text information may be text information corresponding to the call content, such as text content converted from the call content; or, the text information may be text information extracted from a video file or other audio files other than the call content; or, the text information may be directly acquired text content, such as content in a document.
  • Step 102 Input the text information into a target model, and output the keyword category to which each element in the text information belongs.
  • the target model includes a first processing layer for extracting semantic features of the text information, a second processing layer for extracting target keywords existing in the business dictionary from the text information, and a third processing layer for determining the keyword category to which each element in the text information belongs based on the semantic features and the target keywords.
  • the first processing layer is arranged in parallel with the second processing layer; the business dictionary includes multiple keywords that have been identified in the business process.
  • text information is input into the target model, the semantic features of the text information are obtained through the first processing layer, the target keywords existing in the business dictionary are extracted from the text information through the second processing layer, and then the obtained semantic features and target keywords are respectively input into the third processing layer, so that the category of the keywords to which each element in the text information belongs is output through the third processing layer.
  • the keywords identified in the business process can be stored in the business dictionary, so that the target model can extract keywords from the text information based on the business dictionary.
  • the keyword category to which each element in the text information belongs can be more accurately identified.
  • the keyword category includes at least one of a time keyword, a location keyword, a person keyword, and an event keyword. It is understandable that the keyword category may also include other categories besides the categories listed here.
  • keywords can be extracted from the text information according to the keyword category to which each element belongs, so that the extracted keywords are more accurate.
  • the above business dictionary can be provided by the business party, or it can be composed of keywords with relatively high confidence and support collected by data backflow after the target model is actually deployed and launched.
  • the target model includes a first processing layer for extracting semantic features of the text information, a second processing layer for extracting target keywords existing in the business dictionary from the text information, and a third processing layer for determining the keyword category to which each element in the text information belongs based on the semantic features and the target keywords, and the first processing layer is arranged in parallel with the second processing layer;
  • the business dictionary includes multiple keywords that have been identified in the business process.
  • the above-mentioned target model adds a second processing layer for extracting keywords in the business dictionary on the basis of the first processing layer for extracting semantic features of text information.
  • the target model can combine the semantic features and the keywords that have been identified in the business process to determine the keyword category to which each element in the text information belongs, making full use of the existing keyword business knowledge, thereby improving the accuracy of determining the keyword category to which each element in the text information belongs, and further improving the accuracy of extracting keywords from the text information.
  • the first processing layer includes a word representation layer, a segment representation layer, a position representation layer and a multi-attention network layer, the word representation layer, the segment representation layer and the position representation layer are arranged in parallel, and the outputs of the word representation layer, the segment representation layer and the position representation layer are respectively input into the multi-attention network layer.
  • the text information is input into the first processing layer, it is processed by the word representation layer, segment representation layer and position representation layer respectively to obtain word vectors, segment vectors and position vectors respectively. Then the word vectors, segment vectors and position vectors are input into the multi-attention network layer for processing and the text semantic vector is output.
  • the token embedding layer converts the elements of the input text information into a fixed-dimensional word vector.
  • a word is broken down into roots and affixes, and then the roots and affixes are converted into fixed-dimensional vectors respectively;
  • each character is converted into a fixed-dimensional vector; the vector obtained by conversion here is the word vector.
  • the Segment Embedding layer is used to distinguish the order of sentences in the text and convert them into corresponding vectors.
  • the vector obtained by conversion here is the segment vector.
  • the Position Embedding layer is used to encode the position information of elements in text information into a feature vector.
  • the vector obtained by conversion here is the position vector.
  • the multi-attention network layer can fuse multiple features.
  • the three semantic features extracted by the word representation layer, the segment representation layer and the position representation layer can be fused to obtain a text semantic vector that can more accurately represent the semantic features of the text information.
  • the dimension of the second processing layer can be the same as that of the Token Embedding layer, and the initialization parameters are also the same as those of the Token Embedding layer; and the parameters of the Keyword Embedding layer can also be set to be trainable.
  • the method further includes the following steps A1 to A3.
  • Step A1 Obtain pre-training text information.
  • Step A2 replace the keywords in the business dictionary in the pre-trained text information with masks (ie, perform a MASK operation) to obtain pre-trained input text.
  • Step A3 pre-training the first processing layer according to the pre-training input text.
  • the above pre-trained input text may be used to pre-train the first processing layer.
  • the pre-trained text information is "Hello, I am AB takeaway, please come down to pick up your express”
  • the keyword hitting the business dictionary is "AB takeaway”
  • "AB takeaway” can be replaced with the mask ⁇ MASK>, that is, "Hello, I am ⁇ MASK> ⁇ MASK> ⁇ MASK> ⁇ MASK>, please come down to pick up your express”
  • "Hello, I am ⁇ MASK> ⁇ MASK> ⁇ MASK> ⁇ MASK> ⁇ MASK>, please come down to pick up your express” is the pre-trained input text.
  • the MASK matrix of the self-attention layer of the Multi-attention setting based on the bidirectional encoder representation transformer (BERT) is randomly generated, as shown in Figure 2. That is, when the BERT model in the related art is doing MLM pre-training, some words are randomly masked (replaced with a unified marker [MASK]), and then these masked words are predicted to train the bidirectional language model, and each word is Representing reference context information will make the model unable to improve the keyword sequence prediction task in a targeted manner.
  • the pre-training method in the embodiment of the present disclosure makes special restrictions on the MASK matrix of the first processing layer, and only performs MASK operations on the keyword part of the pre-training input text that hits the business dictionary, as shown in Figure 3, so that the first processing layer can make predictions for the keyword sequence.
  • the pre-training method in the embodiment of the present disclosure can be used to process more pre-training text information, thereby obtaining more pre-training input text, thereby further improving the accuracy of the first processing layer.
  • step A3 pre-training the first processing layer according to the pre-training input text” includes the following steps B1 to B3.
  • Step B1 input the pre-trained input text into the first processing layer, and output the prediction result of the mask.
  • Step B2 Calculate the cross entropy between the prediction results corresponding to the same pre-trained input text and the replaced keywords.
  • Step B3 Adjust the parameters of the first processing layer according to the cross entropy.
  • the pre-trained texts can be input into the above first processing layer to obtain the output of the first processing layer (i.e., the content after the mask in the pre-trained input text is replaced by the prediction result).
  • the parameters of the first processing layer can be adjusted according to the output of the first processing layer and the pre-trained text information before the MASK operation. For example, the cross entropy of the two is calculated, and the parameters of the first processing layer are adjusted according to the cross entropy.
  • the output of the first processing layer here (i.e., the content after the mask in the pre-trained input text is replaced by the predicted result) has repeated content with the pre-trained text information before the MASK operation (such as the aforementioned example: Hello, I am, please come down to pick up your express delivery).
  • the pre-trained text information before the MASK operation such as the aforementioned example: Hello, I am, please come down to pick up your express delivery.
  • the cross entropy between the prediction result of the mask output by the first processing layer and the keyword replaced by the mask is calculated, that is, the loss function value of the two is calculated and the gradient is returned, so that the output of the first processing layer at the mask position can predict the "keyword replaced by the mask".
  • the loss function value of the two is calculated and the gradient is returned, so that the output of the first processing layer at the mask position can predict the "keyword replaced by the mask”.
  • step 102 inputs the text information into the target model, outputs each The keyword category to which the element belongs” includes the following steps C1 to C3.
  • Step C1 input the text information into the first processing layer, and output a text semantic vector for representing the semantic features of the text information.
  • Step C2 input the text information into the second processing layer, extract the target keywords through the second processing layer, concatenate the target keywords into a target length to obtain target information, and convert the target information into an embedding vector, wherein the target length is the length of the text information.
  • Step C3 input the text semantic vector and the embedding vector into the third processing layer respectively, and output the keyword category to which each element in the text information belongs.
  • step C2 for example, the text information is "Hello, I am AB takeaway, please come down to pick up your express.”
  • the target keyword that hits the business dictionary is "AB takeaway”
  • the target keyword can be padded to the same length as the text information, that is: AB takeaway [PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD]; and then the target information obtained after filling is converted into an Embeding vector.
  • the third processing layer includes at least one standard fully connected layer and a normalization processing layer
  • Step C3 inputting the text semantic vector and the embedding vector into the third processing layer respectively, and outputting the keyword category to which each element in the text information belongs” includes the following steps D1 to D4.
  • Step D1 Add the text semantic vector to the embedding vector to obtain a first vector
  • Step D2 inputting the first vector into the at least one standard fully connected layer, and outputting a second vector
  • Step D3 inputting the second vector into the normalization processing layer, and outputting the probability (probs) distribution of each element in the text information belonging to a plurality of predetermined keyword categories;
  • Step D4 outputting the keyword category to which each element in the text information belongs according to the probability distribution.
  • step D4 the argmax operation can be performed on the probs distribution of each element in the text information belonging to a predetermined plurality of keyword categories (i.e., in the probability distribution, the maximum probability value corresponding to each element is determined), so that the keyword category to which each element in the text information belongs can be obtained.
  • argmax is a function that finds the parameter (set) of a function.
  • the standard fully connected layer plays the role of a classifier here, that is, after the first vector after the addition of the text semantic vector and the embedding vector is input into the standard fully connected layer, the various elements of the text information are divided into multiple keyword categories through the processing of the standard fully connected layer, and then through the processing of the normalization layer, the probability distribution of each element in the text information belonging to a predetermined multiple keyword categories can be obtained.
  • the keyword category to which the maximum probability value corresponding to each element belongs is the keyword category to which the element belongs.
  • the at least one standard fully connected layer may include, for example, two first standard fully connected layers and one second standard fully connected layer, the output dimension (outdim) of the first standard fully connected layer may be set to be the same as the hidden layer size (hidden size) in the word representation layer described above, and the remaining parameters may be defaulted; the input dimension of the second standard fully connected layer may be the same as the hidden layer size (hidden size) in the word representation layer described above, and the output dimension is k*2+1, where k represents the number of predetermined keyword categories.
  • the "*2" in "k*2+1” is because the BIO structure is adopted, and each category has a difference between begin and I; "+1" is because non-keyword text is marked as O.
  • the method before inputting the text information into the target model, the method further includes the following steps E1 to E2.
  • Step E1 Obtain the field information to which the text information belongs.
  • Step E2 Add the domain information to the text information.
  • any method in the relevant technology can be used to obtain the field information to which the text information belongs.
  • the domain information can be added to the front end of the text information, and the two can be connected using a preset character (such as a period ".”).
  • field information can be integrated into text information, so that when identifying the keyword category to which the elements in the text information belong, the field information can be combined to reduce the probability of misjudgment and improve the accuracy of keyword extraction.
  • domain information may also be incorporated into the pre-training process described above. For example, after step A1 and before step A2, domain information of the pre-trained text information is obtained and added to the pre-trained text information.
  • the method further includes the following steps F1 to F2.
  • Step F1 Use the sequence labeling method to label the elements in the training text information to obtain each of the Labels for training text information.
  • Step F2 training the target model according to the training text information and the label of the training text information.
  • the training process of the target model is different from the pre-training process described above. That is, the pre-training is performed on the first processing layer. After the pre-training of the first processing layer is completed, the second processing layer and the third processing layer are added on the basis of the first processing layer, and then the target model is constructed and then trained.
  • the sequence labeling method is: given a sequence, mark each element in the sequence, or give each element a label.
  • the labeling uses the standard sequence labeling BIO structure (i.e. B-begin, I-inside, O-outside). For example, if the training text information is "Hello, I am AB takeaway, please come down to pick up your delivery.”, the label is: O O O O O O B-takeaway name I-takeaway name I-takeaway name O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O.
  • domain information of the training text information can also be added to the above training text information. For example, if the above training text information is "Hello, I am AB takeaway. Please come down to pick up your express.” After adding the domain information "takeaway", the corresponding label is: O O O O O O O O O B-takeaway name I-takeaway name I-takeaway name O O O O O O O O O O O O.
  • a training sample includes training text information, domain information of the training text information, and a label.
  • step F2 "training the target model according to the training text information and the label of the training text information" can be specifically:
  • the training text information is input into the target model to obtain the corresponding output, and then the output of the model is compared with the label corresponding to the training text to determine whether the keyword category to which each element in the training text information predicted by the target model belongs is accurate. Based on this, the parameters of the target model can be adjusted until the accuracy of the result output by the target model reaches a certain requirement, and the training of the target model is completed.
  • a Keyword Embedding layer i.e., the second processing layer
  • two first standard fully connected layers one second standard fully connected layer
  • a normalization processing layer and a result output layer are further added to obtain the target model shown in Figure 4.
  • the constructed target model is trained, and the training process is described in steps 2.1 to 2.3 below.
  • the following steps 3.1 to 3.5 are performed to obtain the text information.
  • the text information after integrating the domain information is input into the Token Embedding layer, Segment Embedding layer, Position Embedding layer, and Keyword Embedding layer of the target model respectively to obtain the word vector, segment vector, and position vector respectively. Then these three vectors are input into the Multi-attention layer respectively to output the text semantic vector.
  • the Keyword Embedding layer extracts keywords that hit the business dictionary from the text information, then concatenates them to the same length as the text information, and then converts them into embedding vectors.
  • the first vector is obtained.
  • the first vector input is sequentially input into two first standard fully connected layers and one second standard fully connected layer, and the second vector is output.
  • the second vector is input to the normalization processing layer, and the probability distribution of each element in the text information belonging to a plurality of predetermined keyword categories is output.
  • the result output layer outputs the keyword category to which each element in the text information belongs according to the probability distribution.
  • each input token i.e., word embedding
  • the representation of each input token is generated by adding the corresponding word vector, segment vector, position vector, and embedding vector of the hit business dictionary.
  • the calculation method of the vector fusion output by the Keyword Embedding and Multi-attention layers is given, realizing the combination of the Keyword Embedding layer and the first processing layer mentioned above.
  • a Keyword Embedding layer is set in the input layer, which can make full use of the existing keyword business knowledge, so that the existing knowledge can be maximized in training and subsequent operations.
  • the MAS operation is performed on keywords, so that the semantic knowledge embedding training of special keywords can be performed on a large amount of data, and these keywords are given special embedding parameter values to help improve the accuracy of the model. That is, in the embodiment of the present disclosure, through the unique MASK method, certain specific keyword information can be integrated into the dialogue corpus, thereby improving the generalization of the model and improving the model prediction accuracy.
  • an embodiment of the present disclosure further provides a device for identifying keyword categories.
  • the device includes: a first acquisition module 501 and an identification module 502 .
  • the first acquisition module 501 is used to acquire text information to be recognized.
  • the recognition module 502 is used to input the text information into the target model and output the keyword category to which each element in the text information belongs;
  • the target model includes a first processing layer for extracting semantic features of the text information, a second processing layer for extracting target keywords existing in the business dictionary from the text information, and a third processing layer for determining the keyword category to which each element in the text information belongs based on the semantic features and the target keywords.
  • the first processing layer and the second processing layer are set in parallel.
  • the business dictionary includes a plurality of keywords that have been identified in the business process.
  • the first processing layer includes a word representation layer, a segment representation layer, a position representation layer and a multi-attention network layer, the word representation layer, the segment representation layer and the position representation layer are arranged in parallel, and the outputs of the word representation layer, the segment representation layer and the position representation layer are respectively input into the multi-attention network layer.
  • the device also includes: a second acquisition module for acquiring pre-trained text information; a mask processing module for replacing keywords in the business dictionary in the pre-trained text information with masks to obtain pre-trained input text; and a pre-training module for pre-training the first processing layer based on the pre-trained input text.
  • the pre-training module is specifically used to: input the pre-training input text into the first processing layer and output the prediction result of the mask; calculate the cross entropy between the prediction result corresponding to the same pre-training input text and the replaced keyword; and adjust the parameters of the first processing layer according to the cross entropy.
  • the recognition module 502 includes: a first processing submodule, which is used to input the text information into the first processing layer and output a text semantic vector for characterizing the semantic features of the text information; a second processing submodule, which is used to input the text information into the second processing layer, extract the target keywords through the second processing layer, concatenate the target keywords into a target length to obtain target information, and convert the target information into is an embedding vector, wherein the target length is the length of the text information; and a third processing submodule, which is used to input the text semantic vector and the embedding vector into the third processing layer respectively, and output the keyword category to which each element in the text information belongs.
  • a first processing submodule which is used to input the text information into the first processing layer and output a text semantic vector for characterizing the semantic features of the text information
  • a second processing submodule which is used to input the text information into the second processing layer, extract the target keywords through the second processing layer, concatenate the target keywords into a target length to obtain target
  • the third processing layer includes at least one standard fully connected layer and a normalization processing layer; the third processing submodule is specifically used to: add the text semantic vector to the embedding vector to obtain a first vector; input the first vector to the at least one standard fully connected layer and output a second vector; input the second vector to the normalization processing layer and output the probability distribution that each element in the text information belongs to a predetermined plurality of keyword categories; and output the keyword category to which each element in the text information belongs based on the probability distribution.
  • the device further includes: a third acquisition module, used to acquire the domain information to which the text information belongs; and a domain fusion module, used to add the domain information to the text information.
  • the apparatus further comprises: a labeling module, configured to label elements in the training text information using a sequence labeling method to obtain a label for each of the training text information; and
  • the training module is used to train the target model according to the training text information and the label of the training text information.
  • the keyword category includes at least one of a time keyword, a location keyword, a person keyword, and an event keyword.
  • the method and the device are based on the same application concept. Since the method and the device solve the problem in a similar principle, the implementation of the device and the method can refer to each other, and the repeated parts will not be repeated.
  • each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
  • the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium.
  • the technical solution of the present application can essentially or partly contribute to the prior art or all or part of the technical solution can be embodied in the form of a software product.
  • the computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application.
  • the aforementioned storage media include: USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (Random Access Memory, RAM), magnetic disks, or optical disks, etc.
  • An embodiment of the present disclosure further provides an electronic device, as shown in FIG6 , which includes a memory 620 , a transceiver 610 , and a processor 600 .
  • the memory 620 is used to store computer programs.
  • the transceiver 610 is configured to receive and send data under the control of the processor 600 .
  • the processor 600 is configured to read the computer program in the memory 620 and execute the method for identifying keyword categories described in the first aspect.
  • the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 600 and various circuits of memory represented by memory 620 are linked together.
  • the bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein.
  • the bus interface provides an interface.
  • the transceiver 610 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium, which transmission medium includes a wireless channel, a wired channel, an optical cable, and other transmission media.
  • the processor 600 is responsible for managing the bus architecture and general processing, and the memory 620 may store data used by the processor 600 when performing operations.
  • the processor 600 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD).
  • the processor 600 may also adopt a multi-core architecture.
  • a computer program comprising: instructions, which, when executed by a processor, cause the processor to execute the above-mentioned method for identifying keyword categories.
  • the embodiment of the present disclosure further provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, wherein the computer program is used to enable the processor to execute the above-mentioned method for identifying keyword categories.
  • the processor-readable storage medium is a non-transitory processor-readable storage medium.
  • the processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.
  • magnetic storage such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.
  • optical storage such as CD, DVD, BD, HVD, etc.
  • semiconductor storage such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)
  • the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.
  • a computer-usable storage media including but not limited to disk storage and optical storage, etc.
  • each process and/or box in the flowchart and/or block diagram, and the combination of the process and/or box in the flowchart and/or block diagram can be implemented by computer executable instructions.
  • These computer executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the function specified in one process or multiple processes in the flowchart and/or one box or multiple boxes in the block diagram.
  • processor-executable instructions may also be stored in a processor-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the processor-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.
  • processor-executable instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Machine Translation (AREA)

Abstract

本公开提供了一种识别关键词类别的方法、装置及电子设备。该方法包括：获取待识别的文本信息；将文本信息输入至目标模型，输出文本信息中的各个元素所属的关键词类别；其中，目标模型包括用于提取文本信息的语义特征的第一处理层、用于提取文本信息中存在于业务词典中的目标关键词的第二处理层、以及用于基于语义特征和目标关键词确定文本信息中的各个元素所属的关键词类别的第三处理层，第一处理层与第二处理层并行设置；业务词典包括在业务过程中已识别出的多个关键词。

Description

识别关键词类别的方法、装置及电子设备
相关申请的交叉引用
本申请是以CN申请号为202310800741.7，申请日为2023年6月30日的申请为基础，并主张其优先权，该CN申请的公开内容在此作为整体引入本申请中。
技术领域
本公开涉及通信技术领域，尤其涉及一种识别关键词类别的方法、装置及电子设备。
背景技术
通信助理是移动互联网时代，为通信助理业务用户开发的一款增值服务的应用程序，提供漏话查询、通信录备份、短信备份、日程提醒、私密空间、黑白名单等功能。
目前，各个运营商的通信助理业务已日趋成熟，市场竞争也日趋激烈,因此，通信助理需要在业务上做到语义理解，同时帮助用户提取重要信息，并进行提醒。
其中，在通信助理业务中，从通话中提取关键信息至关重要。如果在从通话内容中提取关键词之前，能够确定该通话内容对应的文本信息中各个元素所属的类别，则可以更加准确地提取文本信息中的关键词。
发明内容
第一方面，本公开实施例提供了一种识别关键词类别的方法，所述方法包括：获取待识别的文本信息；和将所述文本信息输入至目标模型，输出所述文本信息中的各个元素所属的关键词类别；其中，所述目标模型包括用于提取所述文本信息的语义特征的第一处理层、用于提取所述文本信息中存在于业务词典中的目标关键词的第二处理层、以及用于基于所述语义特征和所述目标关键词确定所述文本信息中的各个元素所属的关键词类别的第三处理层，所述第一处理层与所述第二处理层并行设置；所述业务词典包括在业务过程中已识别出的多个关键词。
第二方面，本公开的实施例提供了一种识别关键词类别的装置，所述装置包括：第一获取模块，用于获取待识别的文本信息；和识别模块，用于将所述文本信息输入至目标模型，输出所述文本信息中的各个元素所属的关键词类别；其中，所述目标模 型包括用于提取所述文本信息的语义特征的第一处理层、用于提取所述文本信息中存在于业务词典中的目标关键词的第二处理层、以及用于基于所述语义特征和所述目标关键词确定所述文本信息中的各个元素所属的关键词类别的第三处理层，所述第一处理层与所述第二处理层并行设置；所述业务词典包括在业务过程中已识别出的多个关键词。
第三方面，本公开的实施例提供了一种电子设备，包括存储器、收发机和处理器：所述存储器用于存储计算机程序；所述收发机用于在所述处理器的控制下收发数据；所述处理器用于读取所述存储器中的计算机程序并执行上述第一方面所述的识别关键词类别的方法。
第四方面，本公开的实施例提供了一种处理器可读存储介质，所述处理器可读存储介质存储有计算机程序，所述计算机程序用于使处理器执行上述第一方面所述的识别关键词类别的方法。
第五方面，本公开的实施例提供了一种计算机程序，包括：指令，所述指令当由处理器执行时使所述处理器执行上述第一方面所述的识别关键词类别的方法。
附图说明
为了更清楚地说明本公开实施例的技术方案，下面将对本公开实施例的描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本公开的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为本公开实施例提供的识别关键词类别的方法的流程图；
图2为相关技术中进行预训练时的掩码(MASK)操作的示意图；
图3为本公开实施例中进行预训练时的MASK操作的示意图；
图4为本公开实施例的识别关键词类别的方法的具体实施方式中目标模型的架构示意图；
图5为本公开实施例提供的识别关键词类别的装置的结构框图；
图6为本公开实施例提供的电子设备的结构框图。
具体实施方式
本公开实施例中术语“和/或”，描述关联对象的关联关系，表示可以存在三种关系，例 如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。字符“/”一般表示前后关联对象是一种“或”的关系。
本申请实施例中术语“多个”是指两个或两个以上，其它量词与之类似。
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，并不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本公开的发明人发现，在相关技术中，仅基于文本信息的语义特征来识别各个元素所属的类别，在通信助理的不同业务场景下可能会误识别，从而导致关键词的误提取，进而降低业务体验。
鉴于此，本公开的实施例提供了一种识别关键词类别的方法，以提升从文本信息中提取关键词的准确度。
第一方面，本公开的实施例提供了一种识别关键词类别的方法，如图1所示，该方法包括如下步骤101至102。
步骤101：获取待识别的文本信息。
在一些实施例中，所述文本信息可以为通话内容对应的文本信息，例如将通话内容转换为的文本内容；或者，所述文本信息也可以为从视频文件或者除通话内容之外的其他音频文件中提取的文本信息；或者所述文本信息也可以为直接获取的文本内容，例如文档中的内容。
步骤102：将所述文本信息输入至目标模型，输出所述文本信息中的各个元素所属的关键词类别。
其中，所述目标模型包括用于提取所述文本信息的语义特征的第一处理层、用于提取所述文本信息中存在于业务词典中的目标关键词的第二处理层、以及用于基于所述语义特征和所述目标关键词确定所述文本信息中的各个元素所属的关键词类别的第三处理层，所述第一处理层与所述第二处理层并行设置；所述业务词典包括在业务过程中已识别出的多个关键词。
即文本信息输入至目标模型中，通过第一处理层获取该文本信息的语义特征，通过第二处理层从文本信息中提取存在于业务词典中的目标关键词，然后将获取的语义特征以及目标关键词，分别输入至第三处理层，从而通过第三处理层输出文本信息中各个元素所属的关键词的类别。
由此可知，在本公开的实施例中，可以将在业务过程中已识别出的关键词存入业务词典，从而使得目标模型可以基于该业务词典从文本信息中提取关键词。这样，再进一步结合文本信息的语义特征，则可以更加准确地识别出文本信息中各个元素所属的关键词类别。
在一些实施例中，所述关键词类别包括时间关键词、地点关键词、人物关键词、事件关键词中的至少一种。可以理解的是，关键词类别还可以包括除此处列举的类别之外的其他类别。
另外，通过步骤102得到文本信息中各个元素所属的关键词类别之后，则可以根据各个元素所属的关键词类别，从文本信息中提取关键词，从而使得提取到的关键词更加准确。
此外，上述业务词典可以由业务方给出，也可以在目标模型实际部署上线后，由数据回流收集得到的置信度和支持度均比较高的关键词构成。
由上述步骤101至102可知，在本公开实施例中，能够获取待识别的文本信息，从而将文本信息输入目标模型，输出文本信息中的各个元素所属的关键词类别，其中，所述目标模型包括用于提取所述文本信息的语义特征的第一处理层、用于提取所述文本信息中存在于业务词典中的目标关键词的第二处理层、以及用于基于所述语义特征和所述目标关键词确定所述文本信息中的各个元素所属的关键词类别的第三处理层，所述第一处理层与所述第二处理层并行设置；所述业务词典包括在业务过程中已识别出的多个关键词。
由此可见，在本公开实施例中，上述目标模型在用于提取文本信息的语义特征的第一处理层的基础上，增加了提取业务词典中的关键词的第二处理层，这样，目标模型则可以结合语义特征和业务过程中已经识别出的关键词，来确定文本信息中各个元素所属的关键词类别，充分利用了已有的关键词业务知识，从而能够提升确定文本信息中各个元素所属关键词类别的准确度，进而提升从文本信息中提取关键词的准确度。
在一些实施例中，所述第一处理层包括词表征层、段表征层、位置表征层以及多注意力网络层，所述词表征层、所述段表征层和所述位置表征层并行设置，且所述词表征层、所述段表征层和所述位置表征层的输出分别输入至所述多注意力网络层。
由此可知，文本信息输入至第一处理层后，分别通过词表征层、段表征层和位置表征成进行处理，分别得到词向量、段向量和位置向量，进而词向量、段向量和位置向量输入至多注意力网络层进行处理，输出文本语义向量。
这里，词表征(Token Embeding)层，将输入的文本信息的元素转换成固定维度的词向量，例如对于英文，将一个单词会被拆成词根和词缀，然后分别将词根和词缀转换成固定维度的向量；对于中文，将每一个字转换成固定维度的向量；这里转换得到的向量即为词向量。
段表征(Segment Embedding)层用于区别句子在文中的顺序，并转换成对应的向量，这里转换得到的向量即为段向量。
位置表征(Position Embedding)层用于将文本信息中元素的位置信息编码成特征向量，这里转换得到的向量即为位置向量。
多注意力网络(Multi-attention)层能够将多方面的特征进行融合，则在本公开的实施例中，可以将通过词表征层、段表征层和位置表征层提取的这三方面的语义特征进行融合，进而得到能够更加准确地表征文本信息的语义特征的文本语义向量。
此外，上述第二处理层的维度与可以与Token Embedding层的维度相同，且初始化参数也与Token Embedding层相同；并且，还可以设置Keyword Embedding层的参数是可训练的。
在一些实施例中，所述方法还包括如下步骤A1至A3。
步骤A1：获取预训练文本信息。
步骤A2：将所述预训练文本信息中存在于所述业务词典中的关键词替换为掩码(即进行MASK操作)，得到预训练输入文本。
步骤A3：根据所述预训练输入文本，对所述第一处理层进行预训练。
例如在上述目标模型使用之前，可以采用上述预训练输入文本，对第一处理层进行预训练。
例如预训练文本信息为“你好，我是AB外卖，请下来取您的快递”，且命中业务词典的关键词为“AB外卖”，则可以将“AB外卖”替换为掩码<MASK>，即得到：“你好，我是<MASK><MASK><MASK><MASK>，请下来取您的快递”，则“你好，我是<MASK><MASK><MASK><MASK>，请下来取您的快递”即为预训练输入文本。
其中，在相关技术中的MLM任务(即指定周围词来预测中心词的任务)中，基于双向编码表示器(Bidirectional Encoder Representation Transformers，BERT)的Multi-attention设置的自注意力机制(self-attention)层的MASK矩阵是随机生成的，如图2所示。即在相关技术中的BERT模型在做MLM预训练时，通过随机掩盖一些词(替换为统一标记符[MASK])，然后预测这些被遮盖的词来训练双向语言模型，并且使每个词的 表征参考上下文信息，这样做会使得模型不能针对性的对关键词序列预测任务提升。
而本公开实施例中的预训练方法，对第一处理层的MASK矩阵做了特殊限制，仅仅对预训练输入文本中命中业务词典的关键词部分做MASK操作，例如图3所示，从而使得第一处理层可以针对关键词序列进行预测。
由此可知，相对于相关技术而言，在本公开实施例中的预训练过程中，进行MASK操作的内容具有针对性，则获得预训练输入文本所需的时间更短，这样，在同样的时间内，采用本公开实施例中的预训练方法，可以对更多的预训练文本信息进行处理，从而得到更多的预训练输入文本，进而进一步提升第一处理层的准确度。
可选地，步骤A3“根据所述预训练输入文本，对所述第一处理层进行预训练”，包括如下步骤B1至B3。
步骤B1：将所述预训练输入文本输入至所述第一处理层，输出对所述掩码的预测结果。
步骤B2：计算与同一预训练输入文本对应的预测结果与被替换的关键词之间的交叉熵。
步骤B3：根据所述交叉熵，调整所述第一处理层的参数。
其中，按照上述步骤A1至A2得到多个预训练输入文本之后，则可以将预训练文本输入至上述第一处理层，从而得到第一处理层的输出(即预训练输入文本中掩码被预测结果替换后内容)，这样，则可以根据第一处理层的输出以及进行MASK操作之前的预训练文本信息，调整第一处理层的参数。例如计算二者的交叉熵，从而根据交叉熵来调整第一处理层的参数。
但是，这里第一处理层的输出(即预训练输入文本中掩码被预测结果替换后内容)，与进行MASK操作之前的预训练文本信息存在重复的内容(例如前述示例：你好，我是，请下来取您的快递)，这些重复的内容则无需再计算交叉熵，因此，为了简化计算过程，可以仅计算第一处理层输出的对掩码的预测结果，与被掩码替换的关键词之间的交叉熵。
另外，计算第一处理层输出的对掩码的预测结果，与被掩码替换的关键词之间的交叉熵，即为计算二者的损失函数值并进行梯度回传，从而使得第一处理层在掩码的位置的输出能预测出“被掩码替换的关键词”，例如上述示例中：“你好，我是<MASK><MASK><MASK><MASK>，请下来取您的快递”，能够在“<MASK><MASK><MASK><MASK>”的位置处预测出“AB外卖”。
可选地，上述步骤102“将所述文本信息输入至目标模型，输出所述文本信息中的各 个元素所属的关键词类别”，包括如下步骤C1至C3。
步骤C1：将所述文本信息输入至所述第一处理层，输出用于表征所述文本信息的语义特征的文本语义向量。
步骤C2：将所述文本信息输入至所述第二处理层，通过所述第二处理层提取所述目标关键词，将所述目标关键词拼接为目标长度得到目标信息，并将所述目标信息转换为嵌入(Embeding)向量，其中，所述目标长度为所述文本信息的长度。
步骤C3：将所述文本语义向量和所述嵌入向量分别输入至所述第三处理层，输出所述文本信息中的各个元素所属的关键词类别。
其中，在步骤C2中，例如文本信息为“你好，我是AB外卖，请下来取您的快递。”，且其中命中业务词典的目标关键词为“AB外卖”，则可以将该目标关键词填充至与文本信息相同的长度，即：AB外卖[PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD]；然后将填充后得到的目标信息转换为Embeding向量。
由此可知，通过步骤C1至C3，将文本信息中命中业务词典的目标关键词与表征文本信息的语义特征的文本语义向量相结合，从而能够更加准确的得到文本信息中各个元素所属的关键词类别。
在一些实施例中，所述第三处理层包括至少一个标准全连接层和归一化处理层；
步骤C3“将所述文本语义向量和所述嵌入向量分别输入至所述第三处理层，输出所述文本信息中的各个元素所属的关键词类别”，包括如下步骤D1至D4。
步骤D1：将所述文本语义向量与所述嵌入向量相加，得到第一向量；
步骤D2：将所述第一向量输入至所述至少一个标准全连接层，输出第二向量；
步骤D3：将所述第二向量输入至所述归一化处理层，输出所述文本信息中各个元素属于预先确定的多个关键词类别的概率(probs)分布；
步骤D4：根据所述概率分布，输出所述文本信息中的各个元素所属的关键词类别。
其中，在步骤D4中，可以对文本信息中各个元素属于预先确定的多个关键词类别的probs分布取argmax操作(即在概率分布中，确定出各个元素对应的最大概率值)，从而可以得到文本信息中的各个元素所属的关键词类别。
这里，argmax是一种函数，是对函数求参数(集合)的函数。当存在一个函数y＝f(x)时，若有结果x0＝argmax(f(x))，则表示当函数f(x)取x＝x0的时候，得到f(x)取值范围的最大值；若有多个点使得f(x)取得相同的最大值，那么argmax(f(x))的结果就是一个点集。 即argmax(f(x))是使得f(x)取得最大值所对应的变量点x(或x的集合)。因此，在本公开实施例中，对文本信息中各个元素属于预先确定的多个关键词类别的probs分布取argmax操作，即为确定出各个元素对应的最大概率值。
另外，标准全连接层在这里起到分类器的作用，即文本语义向量和嵌入向量相加后的第一向量输入至标准全连接层后，通过标准全连接层的处理，将文本信息的各个元素划分为多个关键词类别，进而通过归一化层的处理，则可以得到文本信息中各个元素属于预先确定的多个关键词类别的概率分布，这样，在概率分布中，各个元素对应的最大概率值所属的关键词类别，即为该元素所属的关键词类别。
此外，所述至少一个标准全连接层例如可以包括两个第一标准全连接层和一个第二标准全连接层，所述第一标准全连接层的输出维度(outdim)设置可以与前文所述的词表征层中的隐藏层尺寸(hidden size)相同，其余参数可默认；所述第二标准全连接层的输入维度可以与前文所述的词表征层中的隐藏层尺寸(hidden size)相同，输出维度为k*2+1，k表示预先确定的关键词类别的数量。这里，“k*2+1”中“*2”是因为采用了BIO结构，每种类别有begin和I的区别；“+1”是因为非关键词的文本标注为O。
在一些实施例中，所述将所述文本信息输入至目标模型之前，所述方法还包括如下步骤E1至E2。
步骤E1：获取所述文本信息所属的领域信息。
步骤E2：将所述领域信息添加至所述文本信息中。
其中，可以采用相关技术中的任一方法获取文本信息所属的领域信息。
另外，可以将领域信息添加至文本信息的前端，二者可以采用预设字符(例如句号“。”)连接。
此外，相关技术中的序列标注技术在通讯助理这种领域细分程度高，关键词的槽位抽取涉及多种不同领域，类别数量大，长尾数据分布严重的场景上，容易因为不同场景内的槽位类型语义相似度较大，在领域间误判。而本公开的实施例中，可以在文本信息中融入领域信息，从而可以在识别文本信息中的元素所属的关键词类别的情况下，可以结合该领域信息，进而降低误判几率，进而提升关键词的提取准确度。
可以理解的是，在前文所述的预训练过程中，也可以融入领域信息，例如在步骤A1之后，且步骤A2之前，获取预训练文本信息的领域信息，并添加至该预训练文本信息中。
在一些实施例中，所述方法还包括如下步骤F1至F2。
步骤F1：采用序列标注方法，对训练文本信息中的元素进行标注，得到每一个所述 训练文本信息的标签。
步骤F2：根据所述训练文本信息以及所述训练文本信息的标签，对所述目标模型进行训练。
需要说明的是，这里对目标模型的训练过程，与前文所述的预训练的过程是不同的两个过程。即预训练是对第一处理层进行的，在对第一处理层预训练完成之后，在第一处理层的基础上添加上述第二处理层和第三处理层，进而构建得到目标模型之后，再对目标模型进行训练。
其中，序列标注方法即为：给定一个序列，对序列中的每个元素做一个标记，或者说给每一个元素打一个标签。打标使用标准的序列标注BIO结构(即B-begin，I-inside，O-outside)，例如训练文本信息为“你好，我是AB外卖，请下来取您的快递。”，标签则为：O O O O O B-外卖名称I-外卖名称I-外卖名称I-外卖名称O O O O O O O O O O。
此外，上述训练文本信息中也可以添加该训练文本信息的领域信息，例如上述训练文本信息为“你好，我是AB外卖，请下来取您的快递。”，添加领域信息“外卖”之后，对应的标签则为：O O O O O O O O B-外卖名称I-外卖名称I-外卖名称I-外卖名称O O O O O O O O O O。
可见，在目标模型的训练阶段，一个训练样本包括训练文本信息、训练文本信息的领域信息和标签。
其中，在上述步骤F2中，“根据所述训练文本信息以及所述训练文本信息的标签，对所述目标模型进行训练”，具体可以为：
将训练文本信息输入至目标模型，得到对应的输出，从而对比模型的输出与训练文本对应的标签，可以确定目标模型预测的训练文本信息中各个元素所属的关键词类别是否准确，进而可以基于此来调整目标模型的参数直到目标模型输出的结果的准确度可以达到一定要求为止，完成目标模型的训练。
综上所述，本公开实施例的识别关键词类别的方法的具体实施方式可如下所述：
(1)预训练阶段：
使用基础BERT中间层架构(即Token Embedding层、Segment Embedding层、Position Embedding层和Multi-attention层)，作为第一处理层进行预训练，具体过程如下步骤1.1至1.5所述。
1.1在其中一个预训练文本信息中融入领域信息，例如为：“外卖。你好，我是AB外卖，请下来取您的快递。”。
1.2识别融入领域信息后的预训练文本信息中，命中业务词典的关键词的位置，例如“外卖。你好，我是AB外卖，请下来取您的快递。”，命中业务词典的关键词为AB外卖，索引位置为：[8,12]。
1.3将命中业务词典的关键词的位置变为<MASK>，得到预训练输入文本，即得到：“外卖。你好，我是<MASK><MASK><MASK><MASK>，请下来取您的快递。”。
1.4将预训练输入文本输入至第一处理层(即分别输入至Token Embedding层、Segment Embedding层、Position Embedding层，然后这三层的输出再输入至Multi-attention层)，输出对<MASK><MASK><MASK><MASK>的预测结果Y(y1,y2,y3,y4)。
1.5计算Y(y1,y2,y3,y4)与原始文本即(A,B,外,卖)的损失函数值并进行梯度回传，使得第一处理层在<MASK><MASK><MASK><MASK>的输出能预测出“A,B,外,卖”四个字。
1.6所有预训练文本信息上重复上述1.1至1.5的过程，从而完成预训练过程。
(2)目标模型构建及训练阶段：
即对第一处理层进行预训练完成后，在第一处理层(即Token Embedding层、Segment Embedding层、Position Embedding层和Multi-attention层)的基础上，进一步添加Keyword Embeding层(即第二处理层)、两个第一标准全连接层、一个第二标准全连接层、归一化处理层和结果输出层，从而得到如图4所示的目标模型。
构建完成目标模型之后，则对构建的目标模型进行训练，训练过程如下步骤2.1至2.3所述。
2.1获取训练文本信息，并在训练文本信息中融入领域信息，例如为：“外卖。你好，我是AB外卖，请下来取您的快递。”。
2.2采用序列标注方法，对训练文本信息中的元素进行标注，得到每一个所述训练文本信息的标签，例如“外卖。你好，我是AB外卖，请下来取您的快递。”，得到的标签为：O O O O O B-外卖名称I-外卖名称I-外卖名称I-外卖名称O O O O O O O O O O。
2.3将训练文本信息输入至目标模型，得到对应的输出，从而对比模型的输出与训练文本对应的标签，可以确定目标模型预测的训练文本信息中各个元素所属的关键词类别是否准确，进而可以基于此来调整目标模型的参数直到目标模型输出的结果的准确度可以达到一定要求为止，完成目标模型的训练。
(3)目标模型的使用阶段：
获取到待识别的文本信息之后，通过如下步骤3.1至3.5的过程，得到该文本信息中 各个元素所属的关键词类别。
31.在文本信息中融入领域信息，例如文本信息为“你好，我是AB外卖，请下来取您的快递。”，融入领域信息后为：“外卖。你好，我是AB外卖，请下来取您的快递。”。
3.1：如图4所示，将融入领域信息后的文本信息分别输入至目标模型的Token Embedding层、Segment Embedding层、Position Embedding层、Keyword Embedding层，分别得到词向量、段向量、位置向量，然后这三个向量分别输入至Multi-attention层，输出文本语义向量。
其中，Keyword Embedding层从文本信息中提取命中业务词典的关键词，然后拼接至与文本信息相同的长度，再将其转换为嵌入向量。
3.2文本语义向量和嵌入向量相加后，得到第一向量。
3.3第一向量输入顺序输入至两个第一标准全连接层和一个第二标准全连接层，输出第二向量。
3.4第二向量输入至归一化处理层，输出文本信息中各个元素属于预先确定的多个关键词类别的概率分布。
3.5.结果输出层，根据所述概率分布，输出所述文本信息中的各个元素所属的关键词类别。
综上所述，本公开的实施例具有如下优点：
1.将待识别的文本信息的领域信息融入至该文本信息中，从而一同输入至目标模型，从而便于在目标模型中，对细分领域实现知识引导，尽可能地解决领域间相似实体误判的问题。
2.在目标模型中，输入的每一个Token(即对词的嵌入)，它的表征由其对应的词向量、段向量和位置向量以及命中业务词典的嵌入向量相加产生。并且，给出了Keyword Embedding与Multi-attention层输出的向量融合的计算方法，实现了Keyword Embedding层与上述第一处理层的结合。
其中，由于业务特殊性，关键词在通话中有很多重复性，但是在相关技术中，这些已经整理识别的关键词语义没有有效利用途径。
由上述可知，为了解决这个问题，在本公开的实施例中，在输入层设置了Keyword Embedding层，能够充分利用已有的关键词业务知识，这样，在训练和后续运营中都能使已有知识得到最大发挥。
3.本公开的实施例中，对上述第一处理层进行预训练的过程中，仅对命中业务词典的 关键词进行MAS操作，这样可对大量数据进行特殊关键词的语义知识嵌入训练，给这些关键词赋予特殊的embedding参数值，帮助提升模型准确度。即本公开的实施例中，通过独特的MASK方式，能将某些确定的关键词信息融入到对话语料中，从而可以提升模型泛化性，提升模型预测准确率。
以上介绍了本公开实施例提供的识别关键词类别的方法，下面将结合附图介绍本公开实施例提供的识别关键词类别的装置。
参见图5，本公开实施例还提供了一种识别关键词类别的装置，如图5所示，所述装置包括：第一获取模块501和识别模块502。
第一获取模块501，用于获取待识别的文本信息。
识别模块502，用于将所述文本信息输入至目标模型，输出所述文本信息中的各个元素所属的关键词类别；
其中，所述目标模型包括用于提取所述文本信息的语义特征的第一处理层、用于提取所述文本信息中存在于业务词典中的目标关键词的第二处理层、以及用于基于所述语义特征和所述目标关键词确定所述文本信息中的各个元素所属的关键词类别的第三处理层，所述第一处理层与所述第二处理层并行设置。
所述业务词典包括在业务过程中已识别出的多个关键词。
在一些实施例中，所述第一处理层包括词表征层、段表征层、位置表征层以及多注意力网络层，所述词表征层、所述段表征层和所述位置表征层并行设置，且所述词表征层、所述段表征层和所述位置表征层的输出分别输入至所述多注意力网络层。
在一些实施例中，所述装置还包括：第二获取模块，用于获取预训练文本信息；掩码处理模块，用于将所述预训练文本信息中存在于所述业务词典中的关键词替换为掩码，得到预训练输入文本；和预训练模块，用于根据所述预训练输入文本，对所述第一处理层进行预训练。
在一些实施例中，所述预训练模块具体用于：将所述预训练输入文本输入至所述第一处理层，输出对所述掩码的预测结果；计算与同一预训练输入文本对应的预测结果与被替换的关键词之间的交叉熵；和根据所述交叉熵，调整所述第一处理层的参数。
在一些实施例中，所述识别模块502包括：第一处理子模块，用于将所述文本信息输入至所述第一处理层，输出用于表征所述文本信息的语义特征的文本语义向量；第二处理子模块，用于将所述文本信息输入至所述第二处理层，通过所述第二处理层提取所述目标关键词，将所述目标关键词拼接为目标长度得到目标信息，并将所述目标信息转换 为嵌入向量，其中，所述目标长度为所述文本信息的长度；和第三处理子模块，用于将所述文本语义向量和所述嵌入向量分别输入至所述第三处理层，输出所述文本信息中的各个元素所属的关键词类别。
在一些实施例中，所述第三处理层包括至少一个标准全连接层和归一化处理层；所述第三处理子模块具体用于：将所述文本语义向量与所述嵌入向量相加，得到第一向量；将所述第一向量输入至所述至少一个标准全连接层，输出第二向量；将所述第二向量输入至所述归一化处理层，输出所述文本信息中各个元素属于预先确定的多个关键词类别的概率分布；和根据所述概率分布，输出所述文本信息中的各个元素所属的关键词类别。
在一些实施例中，所述装置还包括：第三获取模块，用于获取所述文本信息所属的领域信息；和领域融合模块，用于将所述领域信息添加至所述文本信息中。
在一些实施例中，所述装置还包括：标注模块，用于采用序列标注方法，对训练文本信息中的元素进行标注，得到每一个所述训练文本信息的标签；和
训练模块，用于根据所述训练文本信息以及所述训练文本信息的标签，对所述目标模型进行训练。
在一些实施例中，所述关键词类别包括时间关键词、地点关键词、人物关键词、事件关键词中的至少一种。
其中，方法和装置是基于同一申请构思的，由于方法和装置解决问题的原理相似，因此装置和方法的实施可以相互参见，重复之处不再赘述。
需要说明的是，本申请实施例中对单元的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个处理器可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器(processor)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory， RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
在此需要说明的是，本公开实施例提供的上述装置，能够实现上述方法实施例所实现的所有方法步骤，且能够达到相同的技术效果，在此不再对本实施例中与方法实施例相同的部分及有益效果进行具体赘述。
本公开的实施例还提供了一种电子设备，如图6所示，该电子设备包括存储器620、收发机610、处理器600。
存储器620，用于存储计算机程序。
收发机610，用于在处理器600的控制下接收和发送数据。
处理器600用于读取所述存储器620中的计算机程序并执行前述第一方面所述的识别关键词类别的方法。
其中，在图6中，总线架构可以包括任意数量的互联的总线和桥，具体由处理器600代表的一个或多个处理器和存储器620代表的存储器的各种电路链接在一起。总线架构还可以将诸如外围设备、稳压器和功率管理电路等之类的各种其他电路链接在一起，这些都是本领域所公知的，因此，本文不再对其进行进一步描述。总线接口提供接口。收发机610可以是多个元件，即包括发送机和接收机，提供用于在传输介质上与各种其他装置通信的单元，这些传输介质包括无线信道、有线信道、光缆等传输介质。处理器600负责管理总线架构和通常的处理，存储器620可以存储处理器600在执行操作时所使用的数据。
处理器600可以是中央处埋器(CPU)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field－Programmable Gate Array，FPGA)或复杂可编程逻辑器件(Complex Programmable Logic Device，CPLD)，处理器600也可以采用多核架构。
在此需要说明的是，本公开实施例提供的上述装置，能够实现上述方法实施例所实现的所有方法步骤，且能够达到相同的技术效果，在此不再对本实施例中与方法实施例相同的部分及有益效果进行具体赘述。
在本公开的一些实施例中，还提供了一种计算机程序，包括：指令，所述指令当由处理器执行时使所述处理器执行前面所述的识别关键词类别的方法。
本公开的实施例还提供了一种处理器可读存储介质，所述处理器可读存储介质存储有计算机程序，所述计算机程序用于使所述处理器执行上述所述的识别关键词类别的方法。例如，该处理器可读存储介质为非瞬时性处理器可读存储介质。
所述处理器可读存储介质可以是处理器能够存取的任何可用介质或数据存储设备，包括但不限于磁性存储器(例如软盘、硬盘、磁带、磁光盘(MO)等)、光学存储器(例如CD、DVD、BD、HVD等)、以及半导体存储器(例如ROM、EPROM、EEPROM、非易失性存储器(NAND FLASH)、固态硬盘(SSD))等。
本领域内的技术人员应明白，本申请的实施例可提供为方法、系统、或计算机程序产品。因此，本申请可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器和光学存储器等)上实施的计算机程序产品的形式。
本申请是参照根据本申请实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机可执行指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机可执行指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些处理器可执行指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的处理器可读存储器中，使得存储在该处理器可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些处理器可执行指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
显然，本领域的技术人员可以对本申请进行各种改动和变型而不脱离本申请的精神和范围。这样，倘若本申请的这些修改和变型属于本申请权利要求及其等同技术的范围之内，则本申请也意图包含这些改动和变型在内。

Claims (13)

  1. 一种识别关键词类别的方法，包括：
    获取待识别的文本信息；和
    将所述文本信息输入至目标模型，输出所述文本信息中的各个元素所属的关键词类别；
    其中，所述目标模型包括用于提取所述文本信息的语义特征的第一处理层、用于提取所述文本信息中存在于业务词典中的目标关键词的第二处理层、以及用于基于所述语义特征和所述目标关键词确定所述文本信息中的各个元素所属的关键词类别的第三处理层，所述第一处理层与所述第二处理层并行设置；
    所述业务词典包括在业务过程中已识别出的多个关键词。
  2. 根据权利要求1所述的方法，其中，所述第一处理层包括词表征层、段表征层、位置表征层以及多注意力网络层，所述词表征层、所述段表征层和所述位置表征层并行设置，且所述词表征层、所述段表征层和所述位置表征层的输出分别输入至所述多注意力网络层。
  3. 根据权利要求1或2所述的方法，还包括：
    获取预训练文本信息；
    将所述预训练文本信息中存在于所述业务词典中的关键词替换为掩码，得到预训练输入文本；和
    根据所述预训练输入文本，对所述第一处理层进行预训练。
  4. 根据权利要求3所述的方法，其中，所述根据所述预训练输入文本，对所述第一处理层进行预训练，包括：
    将所述预训练输入文本输入至所述第一处理层，输出对所述掩码的预测结果；
    计算与同一预训练输入文本对应的预测结果与被替换的关键词之间的交叉熵；和
    根据所述交叉熵，调整所述第一处理层的参数。
  5. 根据权利要求1至4任意一项所述的方法，其中，所述将所述文本信息输入至 目标模型，输出所述文本信息中的各个元素所属的关键词类别，包括：
    将所述文本信息输入至所述第一处理层，输出用于表征所述文本信息的语义特征的文本语义向量；
    将所述文本信息输入至所述第二处理层，通过所述第二处理层提取所述目标关键词，将所述目标关键词拼接为目标长度得到目标信息，并将所述目标信息转换为嵌入向量，其中，所述目标长度为所述文本信息的长度；和
    将所述文本语义向量和所述嵌入向量分别输入至所述第三处理层，输出所述文本信息中的各个元素所属的关键词类别。
  6. 根据权利要求5所述的方法，其中，所述第三处理层包括至少一个标准全连接层和归一化处理层；
    所述将所述文本语义向量和所述嵌入向量分别输入至所述第三处理层，输出所述文本信息中的各个元素所属的关键词类别，包括：
    将所述文本语义向量与所述嵌入向量相加，得到第一向量；
    将所述第一向量输入至所述至少一个标准全连接层，输出第二向量；
    将所述第二向量输入至所述归一化处理层，输出所述文本信息中各个元素属于预先确定的多个关键词类别的概率分布；和
    根据所述概率分布，输出所述文本信息中的各个元素所属的关键词类别。
  7. 根据权利要求1至6任意一项所述的方法，还包括：
    在将所述文本信息输入至目标模型之前，获取所述文本信息所属的领域信息；和
    将所述领域信息添加至所述文本信息中。
  8. 根据权利要求1至7任意一项所述的方法，还包括：
    采用序列标注方法，对训练文本信息中的元素进行标注，得到每一个所述训练文本信息的标签；和
    根据所述训练文本信息以及所述训练文本信息的标签，对所述目标模型进行训练。
  9. 根据权利要求1至8任意一项所述的方法，其中，所述关键词类别包括时间关键词、地点关键词、人物关键词、事件关键词中的至少一种。
  10. 一种识别关键词类别的装置，包括：
    第一获取模块，用于获取待识别的文本信息；和
    识别模块，用于将所述文本信息输入至目标模型，输出所述文本信息中的各个元素所属的关键词类别；
    其中，所述目标模型包括用于提取所述文本信息的语义特征的第一处理层、用于提取所述文本信息中存在于业务词典中的目标关键词的第二处理层、以及用于基于所述语义特征和所述目标关键词确定所述文本信息中的各个元素所属的关键词类别的第三处理层，所述第一处理层与所述第二处理层并行设置；
    所述业务词典包括在业务过程中已识别出的多个关键词。
  11. 一种电子设备，包括存储器、收发机和处理器：
    所述存储器用于存储计算机程序；所述收发机用于在所述处理器的控制下收发数据；所述处理器用于读取所述存储器中的计算机程序并执行如权利要求1至9中任一项所述的识别关键词类别的方法。
  12. 一种处理器可读存储介质，所述处理器可读存储介质存储有计算机程序，所述计算机程序用于使处理器执行如权利要求1至9中任一项所述的识别关键词类别的方法。
  13. 一种计算机程序，包括：
    指令，所述指令当由处理器执行时使所述处理器执行如权利要求1至9中任一项所述的识别关键词类别的方法。
PCT/CN2023/136792 2023-06-30 2023-12-06 识别关键词类别的方法、装置及电子设备 Ceased WO2025000910A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202310800741.7A CN117076666A (zh) 2023-06-30 2023-06-30 一种识别关键词类别的方法、装置及电子设备
CN202310800741.7 2023-06-30

Publications (1)

Publication Number Publication Date
WO2025000910A1 true WO2025000910A1 (zh) 2025-01-02

Family

ID=88718256

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/136792 Ceased WO2025000910A1 (zh) 2023-06-30 2023-12-06 识别关键词类别的方法、装置及电子设备

Country Status (2)

Country Link
CN (1) CN117076666A (zh)
WO (1) WO2025000910A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117076666A (zh) * 2023-06-30 2023-11-17 中国电信股份有限公司 一种识别关键词类别的方法、装置及电子设备
CN118228692A (zh) * 2024-04-07 2024-06-21 河南公路工程招标咨询有限公司 一种基于数据库的公开招标信息标准化处理方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112632292A (zh) * 2020-12-23 2021-04-09 深圳壹账通智能科技有限公司 业务关键词的提取方法、装置、设备及存储介质
CN113886577A (zh) * 2021-09-10 2022-01-04 润联软件系统(深圳)有限公司 一种文本分类方法、装置、设备及存储介质
CN114781366A (zh) * 2022-03-24 2022-07-22 Oppo广东移动通信有限公司 关键词提取方法、装置以及电子设备
US20230015606A1 (en) * 2020-10-14 2023-01-19 Tencent Technology (Shenzhen) Company Limited Named entity recognition method and apparatus, device, and storage medium
CN117076666A (zh) * 2023-06-30 2023-11-17 中国电信股份有限公司 一种识别关键词类别的方法、装置及电子设备

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110619889B (zh) * 2019-09-19 2022-03-15 Oppo广东移动通信有限公司 体征数据识别方法、装置、电子设备和存储介质
CN115408523A (zh) * 2022-08-31 2022-11-29 山东云海国创云计算装备产业创新中心有限公司 基于摘要提取和关键字提取的中长文本分类方法和系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230015606A1 (en) * 2020-10-14 2023-01-19 Tencent Technology (Shenzhen) Company Limited Named entity recognition method and apparatus, device, and storage medium
CN112632292A (zh) * 2020-12-23 2021-04-09 深圳壹账通智能科技有限公司 业务关键词的提取方法、装置、设备及存储介质
CN113886577A (zh) * 2021-09-10 2022-01-04 润联软件系统(深圳)有限公司 一种文本分类方法、装置、设备及存储介质
CN114781366A (zh) * 2022-03-24 2022-07-22 Oppo广东移动通信有限公司 关键词提取方法、装置以及电子设备
CN117076666A (zh) * 2023-06-30 2023-11-17 中国电信股份有限公司 一种识别关键词类别的方法、装置及电子设备

Also Published As

Publication number Publication date
CN117076666A (zh) 2023-11-17

Similar Documents

Publication Publication Date Title
TWI897104B (zh) 敏感性資料識別方法、裝置、設備及電腦存儲介質
CN112052329A (zh) 文本摘要生成方法、装置、计算机设备及可读存储介质
CN112084381A (zh) 一种事件抽取方法、系统、存储介质以及设备
CN113204956B (zh) 多模型训练方法、摘要分段方法、文本分段方法及装置
CN113887229A (zh) 地址信息的识别方法、装置、计算机设备及存储介质
WO2025000910A1 (zh) 识别关键词类别的方法、装置及电子设备
CN110738059B (zh) 一种文本相似度计算方法及系统
CN107169021A (zh) 用于预测应用功能标签的方法和设备
AU2019290658B2 (en) Systems and methods for identifying and linking events in structured proceedings
CN112036184A (zh) 基于BiLSTM网络模型及CRF模型的实体识别方法、装置、计算机装置及存储介质
CN109918647A (zh) 一种安全领域命名实体识别方法及神经网络模型
CN117931858B (zh) 数据查询方法、装置、计算机设备和存储介质
CN117910473B (zh) 融合实体类型信息的事件论元抽取方法及相关设备
CN118551068A (zh) 表情包检索方法、电子设备和计算机可读存储介质
CN118260589A (zh) 训练大语言模型的方法、装置、电子设备
CN113095083A (zh) 实体抽取方法及装置
CN114691716A (zh) Sql语句转换方法、装置、设备及计算机可读存储介质
CN114722832A (zh) 一种摘要提取方法、装置、设备以及存储介质
CN113515593A (zh) 基于聚类模型的话题检测方法、装置和计算机设备
CN112434136B (zh) 性别分类方法、装置、电子设备以及计算机存储介质
CN120125288A (zh) 数据标注方法、装置、电子设备及存储介质
CN115293165A (zh) 信息提取方法、装置、电子设备及存储介质
CN116186223A (zh) 一种金融文本处理方法、装置、设备和存储介质
WO2023173541A1 (zh) 基于文本的情绪识别方法、装置、设备及存储介质
CN114742053A (zh) 一种处理方法以及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23943386

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE