CN111382562A - Text similarity determination method and device, electronic equipment and storage medium - Google Patents

Text similarity determination method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN111382562A
CN111382562A CN202010147508.XA CN202010147508A CN111382562A CN 111382562 A CN111382562 A CN 111382562A CN 202010147508 A CN202010147508 A CN 202010147508A CN 111382562 A CN111382562 A CN 111382562A
Authority
CN
China
Prior art keywords
text
similarity
syllable
determining
matrix
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010147508.XA
Other languages
Chinese (zh)
Other versions
CN111382562B (en
Inventor
李艾宇
殷超
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Baidu Online Network Technology Beijing Co Ltd
Shanghai Xiaodu Technology Co Ltd
Original Assignee
Beijing Baidu Netcom Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baidu Netcom Science and Technology Co Ltd filed Critical Beijing Baidu Netcom Science and Technology Co Ltd
Priority to CN202010147508.XA priority Critical patent/CN111382562B/en
Publication of CN111382562A publication Critical patent/CN111382562A/en
Application granted granted Critical
Publication of CN111382562B publication Critical patent/CN111382562B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/02Feature extraction for speech recognition; Selection of recognition unit
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/08Speech classification or search
    • G10L15/10Speech classification or search using distance or distortion measures between unknown speech and reference templates
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/26Speech to text systems
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/02Feature extraction for speech recognition; Selection of recognition unit
    • G10L2015/027Syllables being the recognition units

Abstract

The method, the device, the electronic device and the storage medium for determining the text similarity relate to a natural language processing technology, and specifically, a first text and a second text to be processed and corresponding first syllable information and second syllable information are obtained; wherein the first syllable information includes a syllable of each character in the first text; the second syllable information includes a syllable of each character in the second text; determining each syllable in the first syllable information and a similarity value between each syllable in the second syllable information by using a preset syllable similarity dictionary; and determining the similarity between the first text and the second text according to the similarity values. The similarity obtained by the method is determined based on syllables corresponding to characters of the text, so that good recognition accuracy is achieved on the similar recognition of the voice of the user, and the accuracy of the question and answer output of the intelligent question and answer system is improved.

Description

Text similarity determination method and device, electronic equipment and storage medium
Technical Field
The present application relates to data processing technologies, and in particular, to a natural language processing technology.
Background
With the development of science and technology, the intelligent question-answering system is widely applied to various industries. In the intelligent question-answering system, the implementation step of similarity judgment for two texts is indispensable, generally, firstly, a voice is converted into a target text to be analyzed, and then a question-answering strategy of the intelligent question-answering system for the target text is determined by analyzing the similarity between the target text and a text in a history request in the intelligent question-answering system.
In the prior art, the judgment of the similarity between texts is determined based on the similarity of character glyphs in the texts, and specifically, the similarity between any two texts can be calculated by utilizing the length of the longest common subsequence or the edit distance of the two texts.
However, different users have different pronunciation habits, and when the voice is converted into the target text, the voice of the user cannot be converted into the correct target text, which may cause errors to occur easily when similarity calculation is performed based on the converted target text, so that the obtained determination result is not accurate enough, and the accuracy of the output question and answer of the intelligent question and answer system is further affected.
Disclosure of Invention
In view of the foregoing technical problems, the present application provides a method and an apparatus for determining text similarity, an electronic device, and a storage medium.
In a first aspect, an embodiment of the present application provides a method for determining text similarity, including:
acquiring a first text and a second text to be processed, and acquiring corresponding first syllable information and second syllable information; wherein the first syllable information comprises a syllable of each character in the first text; the second syllable information includes a syllable of each character in the second text;
determining each syllable in the first syllable information and a similarity value between each syllable in the second syllable information by using a preset syllable similarity dictionary;
and determining the similarity between the first text and the second text according to the similarity values.
In a second aspect, an embodiment of the present application provides a device for determining text similarity, including:
the syllable conversion module is used for acquiring a first text and a second text to be processed and acquiring corresponding first syllable information and second syllable information; wherein the first syllable information comprises a syllable of each character in the first text; the second syllable information includes a syllable of each character in the second text;
the similarity value obtaining module is used for determining similarity values between each syllable in the first syllable information and each syllable in the second syllable information by using a preset syllable similarity dictionary;
and the similarity determining module is used for determining the similarity between the first text and the second text according to each similarity value.
In a third aspect, an embodiment of the present application provides an electronic device, including: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the method of determining text similarity as described above in the first aspect and in various possible designs of the first aspect.
In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where computer-executable instructions are stored, and when a processor executes the computer-executable instructions, a method for determining text similarity according to the first aspect and various possible designs of the first aspect is implemented.
According to the method, the device, the electronic device and the storage medium for determining the text similarity, the first text and the second text to be processed and the corresponding first syllable information and second syllable information are obtained; wherein the first syllable information comprises a syllable of each character in the first text; the second syllable information includes a syllable of each character in the second text; determining each syllable in the first syllable information and a similarity value between each syllable in the second syllable information by using a preset syllable similarity dictionary; and determining the similarity between the first text and the second text according to the similarity values. Compared with the scheme of obtaining the similarity between texts based on the character font similarity of characters adopted by the prior art, the similarity obtained by the method is determined based on the syllables corresponding to the characters of the texts, so that the method has good recognition accuracy on the similar recognition of the voice of the user, and further the accuracy of the question and answer output of the intelligent question and answer system.
Other effects of the above-described alternative will be described below with reference to specific embodiments.
Drawings
The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
FIG. 1 is a schematic diagram of a network architecture on which the present application is based;
fig. 2 is a schematic flowchart of a text similarity determination method according to an embodiment of the present application;
fig. 3 is an interface schematic diagram of a text similarity determination method according to an embodiment of the present application;
fig. 4 is a schematic flowchart of another text similarity determination method according to an embodiment of the present application;
fig. 5 is a schematic diagram of a similarity matrix according to an embodiment of the present application;
fig. 6 is a schematic flowchart of another text similarity determination method according to an embodiment of the present application;
fig. 7 is a block diagram of a structure of a text similarity determination apparatus according to an embodiment of the present application;
fig. 8 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
In various applications and services, in order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
With the development of science and technology, the intelligent question-answering system is widely applied to various industries. In the intelligent question-answering system, the implementation step of similarity judgment for two texts is indispensable, generally, firstly, a voice is converted into a target text to be analyzed, and then a question-answering strategy of the intelligent question-answering system for the target text is determined by analyzing the similarity between the target text and a text in a history request in the intelligent question-answering system.
In the prior art, the judgment of the similarity between texts is determined based on the similarity of character glyphs in the texts, and specifically, the similarity between any two texts can be calculated by utilizing the length of the longest common subsequence or the edit distance of the two texts.
However, different users have different pronunciation habits, and when the voice is converted into the target text, the voice of the user cannot be converted into the correct target text, which may cause errors to occur easily when similarity calculation is performed based on the converted target text, so that the obtained determination result is not accurate enough, and the accuracy of the output question and answer of the intelligent question and answer system is further affected.
In order to solve the above problems, the technical solution provided by the present application utilizes the syllable characteristics of characters of the text, that is, utilizes the pronunciation similarity of syllables of the text to determine the similarity between the texts. Specifically, first syllable information and second syllable information are obtained; then, determining each syllable in the first syllable information by using a preset syllable similarity dictionary, and obtaining a similarity value between each syllable in the first syllable information and each syllable in the second syllable information; and finally, determining the similarity between the first text and the second text according to the similarity values. Compared with the scheme of obtaining the similarity between texts based on the character font similarity of characters adopted in the prior art, the method has good similarity calculation accuracy rate on the calculation of the text similarity in the intelligent question-answering system, particularly the calculation of the similarity of voice texts, and further improves the accuracy of output question-answering which can be obtained by the intelligent question-answering system based on the similarity.
Referring to fig. 1, fig. 1 is a schematic diagram of a network architecture based on the present application, and the network architecture shown in fig. 1 may specifically include a text similarity determination device 2 and a terminal 1.
The terminal 1 may specifically be a hardware device that can be used to collect voice and display images, such as a user mobile phone, a desktop computer, an intelligent home device, a tablet computer, and the like, and the text similarity determination device 2 is hardware or software that can interact with the terminal 1 through a network, and can be used to execute a text similarity determination method described in each example below, convert voice collected by the collection device of the terminal 1 to a text, perform corresponding similarity calculation, and output a processed result to the terminal 1.
Of course, the acquisition device and/or the display device on the terminal 1 may be integrated on the same hardware, or may be distributed on multiple hardware, and implement data interaction based on wired or wireless connection.
In the network architecture shown in fig. 1, when the text similarity determination apparatus 1 is hardware, it may include a cloud server with an arithmetic function; when the text similarity determination apparatus 1 is software, it may be installed in an electronic device with an arithmetic function, wherein the electronic device includes, but is not limited to, a laptop portable computer, a desktop computer, a terminal 1, and the like.
That is to say, the text similarity determination method based on the present application may be specifically based on the embodiment shown in fig. 1, and is applicable to various application scenarios, including but not limited to: the system is particularly suitable for scenes of an intelligent question-answering system.
In a text correction system scene based on voice input, the terminal 1 may collect input voice through a collection device, convert the voice into a text, and then recognize whether the text is ambiguous or not. When the situation occurs, the text similarity determining device provided by the application can be used for finding out the correct text which has higher similarity with the text, clear semantics and no ambiguity, and correcting the converted text by using the correct text.
Similar to the foregoing text correction system scenario of voice input, in a scenario such as a text translation platform based on voice, the terminal 1 may collect input voice through a collection device, convert the voice into a text, and then find a translation text with the highest similarity to the text from the texts provided by the translation platform library by using the text similarity determination device provided by the present application, and output a translation corresponding to the translation text.
Particularly, in the scenario of the intelligent question-answering system, the terminal 1 may collect input voices through the collection device, convert the voices into texts, and then find a question text with the highest text similarity among the question texts provided by the intelligent question-answering system by using the text similarity determination device provided by the present application, and output a response text corresponding to the question text.
It should be noted that, in each of the foregoing scenarios, the device for determining text similarity may also directly receive the voice acquired from the terminal 1, and convert the voice into a text by itself for processing.
In a first aspect, referring to fig. 2, fig. 2 is a schematic flowchart of a text similarity determination method provided in an embodiment of the present application. The method for determining the text similarity provided by the embodiment of the application comprises the following steps:
step 101, acquiring a first text and a second text to be processed, and acquiring corresponding first syllable information and second syllable information; wherein the first syllable information comprises a syllable of each character in the first text; the second syllable information includes a syllable of each character in the second text.
The main implementation of the determination method provided in this example is the foregoing determination device for text similarity, which may interact with a terminal to obtain a text. The texts are obtained by performing text conversion processing on the collected voice.
It should be noted that, in this embodiment, the first text is used to refer to a text corresponding to a voice acquired by the terminal, and the second text is used to refer to a text of similarity to be compared; in an optional embodiment, the first text is used to refer to a text with similarity to be compared, and the second text is used to refer to a text corresponding to the voice acquired by the terminal. For convenience of description, the first text is used to refer to a text corresponding to the voice acquired by the terminal, and the second text is used to refer to a text of the similarity to be compared, which is taken as an example for explanation. In addition, the text conversion process may be executed by a terminal, or may be executed by the text similarity determination apparatus provided in the present application, which is not limited in this embodiment.
Specifically, after acquiring the first text and the second text to be processed, the determining device further acquires corresponding first syllable information and second syllable information. Syllable information is information constituted by syllables of each character in text. The specific acquisition mode may be, for example: performing character segmentation processing on the acquired first text and the acquired second text to be processed to acquire characters forming the first text and characters forming the second text respectively; and performing syllable conversion processing on the characters of the first text and the characters of the second text to obtain a syllable corresponding to each character.
Generally, the number of syllables corresponding to each character is one or more, for example, in the syllable corresponding to the character "ang", the syllable thereof is represented as "ang", and for example, in the syllable corresponding to the character "first", the syllables thereof are represented as "sh" and "ou".
In other words, for Chinese, the syllables of the characters can be associated with pinyin, i.e., one initial syllable and one final syllable are included in the syllable corresponding to any one of the characters, or/and one final syllable is included in the syllable corresponding to any one of the characters. For example, the syllable corresponding to the aforementioned "ang" includes a final syllable "ang", and the syllable corresponding to the aforementioned "initial" includes a final syllable "ou" and an initial syllable "sh".
Furthermore, in order to further improve the accuracy in determining the similarity, the pitch of the syllable may also be taken into account when determining the syllable. That is, the syllable corresponding to each character includes the note and the corresponding tone that constitute the syllable. For example, in the syllable corresponding to the character "ang", it includes the musical note "ang" constituting the syllable and the corresponding tone "diphone", and it can be finally expressed as "ang 2"; in the syllable corresponding to the character "first", the note "sh" constituting the syllable is included, and since "sh" is the initial, the tone is absent, it can be expressed as "sh 0", and the tone "triphone" corresponding to the note "ou" constituting the syllable, and it can be finally expressed as "ou 3". In other words, based on the tone of the pinyin reading, the tone may be added on the basis of the note of the syllable of each character to achieve an accurate expression of the syllable. Typically, no tone is represented as a "0", one sound is represented as a "1", two sounds are represented as a "2", and so on.
And 102, determining each syllable in the first syllable information by using a preset syllable similarity dictionary, and obtaining a similarity value between each syllable in the first syllable information and each syllable in the second syllable information.
And 103, determining the similarity between the first text and the second text according to each similarity value.
Specifically, in the embodiment of the present application, the similarity value between each syllable in the first syllable information and each syllable in the second syllable information can be determined by using the existing syllable similarity dictionary. In general, the categories of syllables (including combinations of notes and tones) are exhaustive, and in order to indicate the degree of similarity between syllables, a syllable similarity dictionary may be established for looking up similarity values. The similarity value is generally based on the following factors, one is the similarity degree of the notes, for example, a certain similarity exists between 'ang' and 'an', and the similarity value of the two is higher under the condition of not considering other factors; the second is the similar degree of the tone, considering the problems of dialect or accent, etc., when the voice is converted into text, when the notes are completely the same, the influence degree of the difference of the tone on the similarity is low. It should be noted that the specific value of the similarity value may be determined based on experience in consideration of the above factors, and the application is not limited thereto.
And finally, determining the similarity between the first text and the second text based on the obtained similarity values. When the text similarity is determined by using the similarity values, the text similarity may be determined in various manners, such as based on a mean value, a total value, and the like of the similarity values, which is not limited in the present application.
For example, if the first text is "girl backward", the second text is "remainder backward". The first syllable information may be represented as "w 0", "ang 3", "h 0", "ou 4", "n 0", "v 3", "sh 0", "eng 1", and the second syllable information may be represented as "w 0", "ang 3", "h 0", "ou 4", "y 0", "v 2", "sh 0", "eng 1". By determining the similarity value between the syllables, it can be known that the similarity between each syllable of the first syllable information and the second syllable information is higher, that is, the similarity between the first text and the second text is higher. That is, if the first text is "girl later", the real semantic meaning may be "girl later", and the system or the platform may perform the subsequent processing based on the second text by using a corresponding reply method or processing method.
Fig. 3 is an interface schematic diagram of a text similarity determination method according to an embodiment of the present application, and as shown in fig. 3, in the interface, when a user inputs a voice, a text corresponding to the voice may be corrected by the text similarity determination device, and a correction result may be confirmed to the user.
In the method for determining the text similarity provided by this embodiment, a first text and a second text to be processed, and corresponding first syllable information and second syllable information are obtained; wherein the first syllable information includes a syllable of each character in the first text; the second syllable information includes a syllable of each character in the second text; determining each syllable in the first syllable information and a similarity value between each syllable in the second syllable information by using a preset syllable similarity dictionary; and determining the similarity between the first text and the second text according to the similarity values. Compared with the scheme of obtaining the similarity between texts based on the character font similarity of characters in the prior art, the similarity obtained by the method is determined based on the syllables corresponding to the characters of the texts, so that the method has good recognition accuracy on the similar recognition of the voice of the user, and further the accuracy of the output question and answer of the intelligent question and answer system.
On the basis of the foregoing embodiment, referring to fig. 4, fig. 4 is a schematic flowchart of a text similarity determination method provided in the embodiment of the present application.
Step 201, acquiring a first text and a second text to be processed, and acquiring corresponding first syllable information and second syllable information; wherein the first syllable information comprises a syllable of each character in the first text; the second syllable information includes a syllable of each character in the second text.
Step 202, determining each syllable in the first syllable information by using a preset syllable similarity dictionary, and obtaining a similarity value between each syllable in the second syllable information.
And 203, constructing a similarity matrix by taking the similarity values as matrix elements.
And step 204, determining one or more maximum similar paths in the similarity matrix by using a dynamic planning algorithm.
And step 205, determining the similarity between the first text and the second text according to the similarity values corresponding to the one or more maximum similar paths.
Similar to the foregoing embodiment, in the embodiment of the present application, the implementation subject is the foregoing text similarity determination apparatus, which may interact with a terminal to obtain a text. The texts are obtained by performing text conversion processing on the collected voice. In addition, for a specific implementation manner of the steps 201 and 202, reference may be made to corresponding portions of the steps 101 and 102 in the foregoing embodiment, which is not described in detail in this embodiment.
Different from the foregoing embodiment, in this embodiment, for determining the similarity between the first text and the second text according to each similarity value, the similarity may be determined based on a dynamic programming algorithm.
Specifically, when the similarity is determined, the characters in the first text and the second text may not be in one-to-one correspondence, for example, the first text is "i want to listen to girls backward", the second text is "afterward", the similarity value between girls and "afterward" may be higher, and the similarity value between "backward" and "backward" is higher. In this case, a similarity matrix may be established with each similarity value as a matrix element. Fig. 5 is a schematic diagram of a similarity matrix provided in an embodiment of the present application, as shown in fig. 5, in the similarity matrix, a row element is used to indicate each syllable of the second text, and a similarity value with each syllable in the first text is obtained, for example, a similarity value between "w 0" and "w 0" is 1, a similarity value between "w 0" and "x 0" is 0.5, and the like.
Correspondingly, the column element is used for representing each syllable of the first text, and the similarity value between each syllable of the first text and each syllable of the second text is 1, for example, the similarity value between "w 0" and "w 0" is 0.5, and the similarity value between "w 0" and "y 0" is 0.5.
And then, determining one or more maximum similar paths in the similarity matrix by using a dynamic programming algorithm, and determining the similarity between the first text and the second text according to the similarity value corresponding to the one or more maximum similar paths. The dynamic programming algorithm is a branch of operations research, and is a mathematical method for solving the optimization of a decision making process, and one or more maximum similar paths can be found from a similarity matrix by using the algorithm so as to represent the similarity between texts.
Specifically, one or more sub-matrices are determined in the similarity matrix, wherein the matrix elements of the sub-matrices are not overlapped, and the average value of the matrix elements along the diagonal line in each sub-matrix is greater than a preset similarity threshold (e.g., 0.8), and the sub-matrices are represented by rectangular boxes shown in fig. 5.
Then, summing up each matrix element along the diagonal line in each sub-matrix to obtain a similarity value of each sub-matrix, that is, the similarity value of the left sub-matrix is 0.5+0.83333+1+1 ═ 3.33333, and the similarity value of the right sub-matrix is 1+1+1+1 ═ 4.
And finally, determining the similarity between the first text and the second text according to the similarity value of each sub-matrix. Specifically, the sum of the similarity values of the maximum similar paths may be determined, and the average of the number of characters of the first text and the second text may be determined, where the sum of the similarity values of the maximum similar paths shown in fig. 5 is 7.33333; the average of the number of characters of the first text and the second text is (7+ 4)/2-5.5. And the ratio of the sum of the similarity values of the maximum similar paths to the average value constitutes the similarity of the first text and the second text, namely 1.33333.
On the basis of the foregoing embodiment, the embodiment adopts the dynamic programming algorithm, thereby achieving determination of the similarity, and has a good processing effect for a case where the characters of the first text and the second text cannot correspond to each other, so that the similarity accuracy in this case is high.
On the basis of the foregoing embodiment, referring to fig. 6, fig. 6 is a schematic flowchart of another text similarity determination method provided in the embodiment of the present application. The method for determining the text similarity provided by the embodiment of the application comprises the following steps:
step 301, collecting voice information input by a user;
step 302, performing text conversion processing on the voice information to obtain a first text, and taking any one of the historical texts as a second text; the history text is obtained by performing text conversion processing on voice information input by the user history.
Step 303, obtaining corresponding first syllable information and second syllable information; wherein the first syllable information comprises a syllable of each character in the first text; the second syllable information includes a syllable of each character in the second text;
304, determining each syllable in the first syllable information and a similarity value between each syllable in the second syllable information by using a preset syllable similarity dictionary;
and 305, determining the similarity between the first text and the second text according to the similarity values.
Step 306, according to the similarity between the first text and each historical text, determining the text with the highest similarity with the first text in each historical text, and taking the reply text corresponding to the text with the highest similarity as the output text corresponding to the first text.
Similar to the foregoing embodiment, in the embodiment of the present application, the implementation subject is the foregoing text similarity determination apparatus, which may interact with a terminal to obtain a text. The texts are obtained by performing text conversion processing on the collected voice. In addition, for a specific implementation manner of the above steps 303 to 305, reference may be made to corresponding parts of the foregoing embodiments, which are not described in detail in this embodiment.
Specifically, the user inputs the voice of the problem through the acquisition device of the terminal, the terminal can directly send the voice to the determination device, and the voice is converted into the first text based on the voice-text conversion technology of the determination device.
Subsequently, in order to be able to answer the question input by the user, the determination means will select any text from the history texts stored in the answering system as the second text, and calculate the similarity between the first text and the second text. It should be noted that the history text refers to an existing text in which a reply text has already been stored in the reply system, and the existing text may be preset or stored by acquiring and processing a question and answer initiated by the user history.
And then, processing the first text and the second text by adopting the mode, and selecting the next historical text as the second text after the similarity is obtained by processing until the similarity of all the historical texts is obtained.
And finally, determining the text with the highest similarity to the first text in each historical text according to the similarity between the first text and each historical text, and taking the reply text corresponding to the text with the highest similarity as the output text corresponding to the first text.
On the basis of the above embodiments, the present embodiment specifically describes the intelligent question-answering system, and the obtained similarity is determined based on syllables corresponding to characters of a text, so that there is a good recognition accuracy in the similar recognition of the user's voice, and the accuracy of outputting the question and answer by the intelligent question-answering system.
Fig. 7 is a block diagram of a text similarity determination apparatus according to an embodiment of the present application, which corresponds to the text similarity determination method according to the foregoing embodiment. For convenience of explanation, only portions related to the embodiments of the present application are shown. Referring to fig. 7, the apparatus for determining text similarity includes: the system comprises a syllable conversion module 10, a similarity value acquisition module 20 and a similarity determination module 30.
The syllable conversion module 10 is used for acquiring a first text and a second text to be processed and acquiring corresponding first syllable information and second syllable information; wherein the first syllable information comprises a syllable of each character in the first text; the second syllable information includes a syllable of each character in the second text;
the similarity value obtaining module is used for determining similarity values between each syllable in the first syllable information and each syllable in the second syllable information by using a preset syllable similarity dictionary;
and a similarity determining module 30, configured to determine a similarity between the first text and the second text according to each similarity value.
In an optional embodiment provided by the present application, the similarity determining module 30 is specifically configured to:
taking each similarity value as a matrix element to construct a similarity matrix; determining one or more maximum similar paths in a similarity matrix by using a dynamic programming algorithm; and determining the similarity between the first text and the second text according to the similarity values corresponding to the one or more maximum similar paths.
In an optional embodiment provided by the present application, the similarity determining module 30 is specifically configured to:
determining one or more sub-matrixes in the similarity matrix, wherein matrix elements of the sub-matrixes are not overlapped, and the mean value of the matrix elements along a diagonal line in each sub-matrix is greater than a preset similarity threshold value; the matrix element selection module is also used for summing matrix elements along the diagonal line in each sub-matrix to obtain a similarity value of each sub-matrix; and determining the similarity between the first text and the second text according to the similarity value of each sub-matrix.
In an optional embodiment provided by the present application, the similarity determining module 30 is specifically configured to: determining the sum of the similarity values of the maximum similar paths, and determining the average value of the number of the characters of the first text and the second text; and the ratio of the sum of the similarity values of the maximum similar paths to the mean value forms the similarity of the first text and the second text.
In an optional embodiment provided by the present application, the syllable conversion module 10 is configured to perform character segmentation processing on the obtained first text and the second text to be processed, so as to obtain characters constituting the first text and characters constituting the second text, respectively; and performing syllable conversion processing on the characters of the first text and the characters of the second text to obtain a syllable corresponding to each character.
In an alternative embodiment provided by the present application, the number of syllables corresponding to each character is one or more.
In an alternative embodiment provided by the present application, the syllable corresponding to any one of the characters includes an initial syllable and a final syllable, or/and the syllable corresponding to any one of the characters includes a final syllable.
In an alternative embodiment provided by the present application, the syllable corresponding to each character includes the musical notes constituting the syllable and the corresponding tone.
In an optional embodiment provided by the present application, the syllable conversion module 10 is specifically configured to collect voice information input by a user; performing text conversion processing on the voice information to obtain a first text, and taking any one of the historical texts as a second text; the history text is obtained by performing text conversion processing on voice information input by the user history.
In an optional embodiment provided by the present application, further comprising: an output module;
and the output module is used for determining the text with the highest similarity to the first text in each historical text according to the similarity between the first text and each historical text, and taking the reply text corresponding to the text with the highest similarity as the output text corresponding to the first text.
The device for determining text similarity provided by this embodiment obtains a first text and a second text to be processed, and corresponding first syllable information and second syllable information; wherein the first syllable information includes a syllable of each character in the first text; the second syllable information includes a syllable of each character in the second text; determining each syllable in the first syllable information and a similarity value between each syllable in the second syllable information by using a preset syllable similarity dictionary; and determining the similarity between the first text and the second text according to the similarity values. Compared with the scheme of obtaining the similarity between texts based on the character font similarity of characters in the prior art, the similarity obtained by the method is determined based on the syllables corresponding to the characters of the texts, so that the method has good recognition accuracy on the similar recognition of the voice of the user, and further the accuracy of the output question and answer of the intelligent question and answer system.
According to an embodiment of the present application, an electronic device and a readable storage medium are also provided.
Fig. 8 is a block diagram of an electronic device according to a determination method in an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 8, the electronic apparatus includes: one or more processors 501, memory 502, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to the interface). In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and multiple memories, as desired. Also, multiple electronic devices may be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). Fig. 8 illustrates an example of one processor 501.
Memory 502 is a non-transitory computer readable storage medium as provided herein. Wherein the memory stores instructions executable by at least one processor to cause the at least one processor to perform the determination methods provided herein. The non-transitory computer-readable storage medium of the present application stores computer instructions for causing a computer to perform the determination method provided by the present application.
The memory 502, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the determination methods in the embodiments of the present application (e.g., the acquisition module 10, the processing module 20, and the control module 30 shown in fig. 7). The processor 501 executes various functional applications of the server and data processing, i.e., a method of implementing the determination method in the above-described method embodiments, by executing non-transitory software programs, instructions, and modules stored in the memory 502.
The memory 502 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the electronic device of the determination method, and the like. Further, the memory 502 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 502 optionally includes memory located remotely from processor 501, which may be connected to an electronic device via a network. Examples of such networks include, but are not limited to, the internet, an intranet, a lan, a mobile 502, an input device 503, and an output device 504, which may be connected by a bus, as illustrated in fig. 8.
The input device 503 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic apparatus for the determination method, such as an input device such as a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointing stick, one or more mouse buttons, a track ball, a joystick, or the like. The output devices 504 may include a display device, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibrating motors), among others. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device can be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and the present invention is not limited thereto as long as the desired results of the technical solutions disclosed in the present application can be achieved.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims (22)

1. A method for determining text similarity includes:
acquiring a first text and a second text to be processed, and acquiring corresponding first syllable information and second syllable information; wherein the first syllable information comprises a syllable of each character in the first text; the second syllable information includes a syllable of each character in the second text;
determining each syllable in the first syllable information and a similarity value between each syllable in the second syllable information by using a preset syllable similarity dictionary;
and determining the similarity between the first text and the second text according to the similarity values.
2. The method for determining the text similarity according to claim 1, wherein the determining the similarity between the first text and the second text according to the similarity values includes:
taking each similarity value as a matrix element to construct a similarity matrix;
determining one or more maximum similar paths in a similarity matrix by using a dynamic programming algorithm;
and determining the similarity between the first text and the second text according to the similarity values corresponding to the one or more maximum similar paths.
3. The method for determining text similarity according to claim 2, wherein the determining one or more maximum similarity paths in the similarity matrix by using a dynamic programming algorithm includes:
determining one or more sub-matrixes in the similarity matrix, wherein matrix elements of the sub-matrixes are not overlapped, and the mean value of the matrix elements along a diagonal line in each sub-matrix is greater than a preset similarity threshold value;
correspondingly, determining the similarity between the first text and the second text according to the similarity value corresponding to the one or more maximum similar paths, including:
summing matrix elements along the diagonal line in each sub-matrix to obtain a similarity value of each sub-matrix;
and determining the similarity between the first text and the second text according to the similarity value of each sub-matrix.
4. The method for determining the text similarity according to claim 2, wherein the determining the similarity between the first text and the second text according to the similarity values corresponding to the one or more maximum similarity paths includes:
determining the sum of the similarity values of the maximum similar paths, and determining the average value of the number of the characters of the first text and the second text;
and the ratio of the sum of the similarity values of the maximum similar paths to the mean value forms the similarity of the first text and the second text.
5. The method for determining text similarity according to claim 1, wherein the obtaining corresponding first syllable information and second syllable information comprises:
performing character segmentation processing on the acquired first text and the acquired second text to be processed to acquire characters forming the first text and characters forming the second text respectively;
and performing syllable conversion processing on the characters of the first text and the characters of the second text to obtain a syllable corresponding to each character.
6. The method for determining text similarity according to claim 5, wherein the number of syllables corresponding to each character is one or more.
7. The method according to claim 5, wherein the syllable corresponding to any one of the characters comprises an initial syllable and a final syllable, or/and the syllable corresponding to any one of the characters comprises a final syllable.
8. The method of claim 5, wherein the syllable corresponding to each character comprises the musical notes and the corresponding tone of the syllable.
9. The method for determining text similarity according to any one of claims 1 to 8, wherein the obtaining of the first text and the second text to be processed includes:
collecting voice information input by a user;
performing text conversion processing on the voice information to obtain a first text, and taking any one of the historical texts as a second text; the history text is obtained by performing text conversion processing on voice information input by the user history.
10. The method for determining text similarity according to claim 9, further comprising:
and determining the text with the highest similarity to the first text in each history text according to the similarity between the first text and each history text, and taking the reply text corresponding to the text with the highest similarity as the output text corresponding to the first text.
11. A device for determining similarity between texts, comprising:
the syllable conversion module is used for acquiring a first text and a second text to be processed and acquiring corresponding first syllable information and second syllable information; wherein the first syllable information comprises a syllable of each character in the first text; the second syllable information includes a syllable of each character in the second text;
the similarity value obtaining module is used for determining similarity values between each syllable in the first syllable information and each syllable in the second syllable information by using a preset syllable similarity dictionary;
and the similarity determining module is used for determining the similarity between the first text and the second text according to each similarity value.
12. The apparatus for determining text similarity according to claim 11, wherein the similarity determining module is specifically configured to:
taking each similarity value as a matrix element to construct a similarity matrix; determining one or more maximum similar paths in a similarity matrix by using a dynamic programming algorithm; and determining the similarity between the first text and the second text according to the similarity values corresponding to the one or more maximum similar paths.
13. The apparatus for determining text similarity according to claim 12, wherein the similarity determining module is specifically configured to:
determining one or more sub-matrixes in the similarity matrix, wherein matrix elements of the sub-matrixes are not overlapped, and the mean value of the matrix elements along a diagonal line in each sub-matrix is greater than a preset similarity threshold value; the matrix element selection module is also used for summing matrix elements along the diagonal line in each sub-matrix to obtain a similarity value of each sub-matrix; and determining the similarity between the first text and the second text according to the similarity value of each sub-matrix.
14. The apparatus for determining text similarity according to claim 12, wherein the similarity determining module is specifically configured to: determining the sum of the similarity values of the maximum similar paths, and determining the average value of the number of the characters of the first text and the second text; and the ratio of the sum of the similarity values of the maximum similar paths to the mean value forms the similarity of the first text and the second text.
15. The apparatus for determining text similarity according to claim 11, wherein the syllable conversion module is configured to perform character segmentation on the obtained first text and the obtained second text to be processed, so as to obtain characters constituting the first text and characters constituting the second text, respectively; and performing syllable conversion processing on the characters of the first text and the characters of the second text to obtain a syllable corresponding to each character.
16. The apparatus for determining similarity between texts according to claim 15, wherein the number of syllables corresponding to each character is one or more.
17. The apparatus for determining similarity between texts as claimed in claim 15, wherein any one of said characters corresponds to a syllable comprising an initial syllable and a final syllable, or/and any one of said characters corresponds to a syllable comprising a final syllable.
18. The apparatus for determining text similarity according to claim 15, wherein the syllable corresponding to each character includes a note constituting the syllable and a corresponding tone.
19. The apparatus for determining text similarity according to any one of claims 11-18, wherein the syllable transformation module is specifically configured to collect voice information input by a user; performing text conversion processing on the voice information to obtain a first text, and taking any one of the historical texts as a second text; the history text is obtained by performing text conversion processing on voice information input by the user history.
20. The apparatus for determining similarity between texts according to claim 19, further comprising: an output module;
and the output module is used for determining the text with the highest similarity to the first text in each historical text according to the similarity between the first text and each historical text, and taking the reply text corresponding to the text with the highest similarity as the output text corresponding to the first text.
21. An electronic device, comprising: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the method of determining text similarity according to any of claims 1-10.
22. A computer-readable storage medium having stored thereon computer-executable instructions which, when executed by a processor, implement the text similarity determination method according to any one of claims 1 to 10.
CN202010147508.XA 2020-03-05 2020-03-05 Text similarity determination method and device, electronic equipment and storage medium Active CN111382562B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010147508.XA CN111382562B (en) 2020-03-05 2020-03-05 Text similarity determination method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010147508.XA CN111382562B (en) 2020-03-05 2020-03-05 Text similarity determination method and device, electronic equipment and storage medium

Publications (2)

Publication Number Publication Date
CN111382562A true CN111382562A (en) 2020-07-07
CN111382562B CN111382562B (en) 2024-03-01

Family

ID=71219809

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010147508.XA Active CN111382562B (en) 2020-03-05 2020-03-05 Text similarity determination method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN111382562B (en)

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103605694A (en) * 2013-11-04 2014-02-26 北京奇虎科技有限公司 Device and method for detecting similar texts
CN104464736A (en) * 2014-12-15 2015-03-25 北京百度网讯科技有限公司 Error correction method and device for voice recognition text
CN106548777A (en) * 2016-11-25 2017-03-29 北京光年无限科技有限公司 A kind of data processing method and device for intelligent robot
CN106815197A (en) * 2015-11-27 2017-06-09 北京国双科技有限公司 The determination method and apparatus of text similarity
CN108536657A (en) * 2018-04-10 2018-09-14 百融金融信息服务股份有限公司 The address text similarity processing method and system artificially filled in
CN108804624A (en) * 2013-12-18 2018-11-13 国网江苏省电力有限公司常州供电分公司 The method of text gear typing and comparison
CN109117464A (en) * 2018-03-02 2019-01-01 南京邮电大学 A kind of data similarity detection method based on editing distance
CN109299471A (en) * 2018-11-05 2019-02-01 广州百田信息科技有限公司 A kind of method, apparatus and terminal of text matches
CN109657213A (en) * 2018-12-21 2019-04-19 北京金山安全软件有限公司 Text similarity detection method and device and electronic equipment
CN110047515A (en) * 2019-04-04 2019-07-23 腾讯音乐娱乐科技(深圳)有限公司 A kind of audio identification methods, device, equipment and storage medium

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103605694A (en) * 2013-11-04 2014-02-26 北京奇虎科技有限公司 Device and method for detecting similar texts
CN108804624A (en) * 2013-12-18 2018-11-13 国网江苏省电力有限公司常州供电分公司 The method of text gear typing and comparison
CN104464736A (en) * 2014-12-15 2015-03-25 北京百度网讯科技有限公司 Error correction method and device for voice recognition text
CN106815197A (en) * 2015-11-27 2017-06-09 北京国双科技有限公司 The determination method and apparatus of text similarity
CN106548777A (en) * 2016-11-25 2017-03-29 北京光年无限科技有限公司 A kind of data processing method and device for intelligent robot
CN109117464A (en) * 2018-03-02 2019-01-01 南京邮电大学 A kind of data similarity detection method based on editing distance
CN108536657A (en) * 2018-04-10 2018-09-14 百融金融信息服务股份有限公司 The address text similarity processing method and system artificially filled in
CN109299471A (en) * 2018-11-05 2019-02-01 广州百田信息科技有限公司 A kind of method, apparatus and terminal of text matches
CN109657213A (en) * 2018-12-21 2019-04-19 北京金山安全软件有限公司 Text similarity detection method and device and electronic equipment
CN110047515A (en) * 2019-04-04 2019-07-23 腾讯音乐娱乐科技(深圳)有限公司 A kind of audio identification methods, device, equipment and storage medium

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
DHAVAL SALVI 等: "Handwritten text segmentation using average longest path algorithm", pages 505 - 512 *
TOMAS BALYO 等: "Optimal Longest Paths by Dynamic Programming", pages 1 - 11 *
何明 等: "一种文本相似性的度量方法和计算方法", pages 71 - 72 *
李静: "基于动态规划进行双序列全局对比研究", pages 006 - 36 *

Also Published As

Publication number Publication date
CN111382562B (en) 2024-03-01

Similar Documents

Publication Publication Date Title
KR102565659B1 (en) Method and apparatus for generating information
KR20210106397A (en) Voice conversion method, electronic device, and storage medium
CN112365880B (en) Speech synthesis method, device, electronic equipment and storage medium
JP2022028887A (en) Method, apparatus, electronic device and storage medium for correcting text errors
JP2019102063A (en) Method and apparatus for controlling page
CN111259671B (en) Semantic description processing method, device and equipment for text entity
JP7318159B2 (en) Text error correction method, apparatus, electronic device and readable storage medium
CN112926306B (en) Text error correction method, device, equipment and storage medium
JP2021119381A (en) Voice spectrum generation model learning method, device, electronic apparatus and computer program product
CN111241245B (en) Human-computer interaction processing method and device and electronic equipment
CN112509552B (en) Speech synthesis method, device, electronic equipment and storage medium
US20210210112A1 (en) Model Evaluation Method and Device, and Electronic Device
KR102475235B1 (en) Method for resource sorting, method for training sorting model and corresponding apparatuses
CN111241838B (en) Semantic relation processing method, device and equipment for text entity
JP2021111334A (en) Method of human-computer interactive interaction based on retrieval data, device, and electronic apparatus
JP7355776B2 (en) Speech recognition method, speech recognition device, electronic device, computer readable storage medium and computer program
KR20210041546A (en) Method and apparatus for determining causality, electronic device and storage medium
JP7264957B2 (en) Voice interaction method, device, electronic device, computer readable storage medium and computer program
CN111104516B (en) Text classification method and device and electronic equipment
US20220068265A1 (en) Method for displaying streaming speech recognition result, electronic device, and storage medium
CN114612749A (en) Neural network model training method and device, electronic device and medium
CN111563593A (en) Training method and device of neural network model
CN111709252A (en) Model improvement method and device based on pre-trained semantic model
JP2021114284A (en) Method and apparatus for predicting punctuation mark
JP2022028897A (en) Text translation method, device, electronic device and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20210518

Address after: 100085 Baidu Building, 10 Shangdi Tenth Street, Haidian District, Beijing

Applicant after: BAIDU ONLINE NETWORK TECHNOLOGY (BEIJING) Co.,Ltd.

Applicant after: Shanghai Xiaodu Technology Co.,Ltd.

Address before: 100085 Baidu Building, 10 Shangdi Tenth Street, Haidian District, Beijing

Applicant before: BAIDU ONLINE NETWORK TECHNOLOGY (BEIJING) Co.,Ltd.

GR01 Patent grant
GR01 Patent grant