DE102021006511A1 - Method and device for machine processing of texts - Google Patents

Abstract

The present invention relates to a method for machining a text. It comprises the following steps: First, a text is recorded and saved, with each recorded word being assigned a value. For example, this value can be represented by a k-dimensional vector. A first probability is then determined for a sequence of the text and for a word of the sequence, with which the word occurs in the sequence in combination with a specific state. This state is calculated from the values of words adjacent to the word in the sequence, for example by addition, possibly with subsequent scaling. The first probability that the word occurs in the sequence in combination with the state can be calculated based on the value of the word in relation to the state, for example as a frequency of the word for a discrete state.

Description

The invention relates to a method and a device for machine processing of texts. In particular, the invention relates to machine generation of sentences based on a given text base and with reference to a query sentence.

Technological background

According to current understanding, language is stored and retrieved in the human brain in a neural network. Something similar should also be achieved with artificial neural networks.

• A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, „Improving language understanding by generative pre-training“, Preprint, 2018.
• A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, „Language models are unsupervised multitask learners“, Preprint, 2019.
• T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, „Language models are few-shot learners,“ Preprint, 2020.
• Neuronale Netze werden anhand sehr großer Datenmengen trainiert. Die Information, wie Sprache erzeugt wird, liegt dabei im Wesentlichen in den Gewichten des Netzwerks vor. Das Training der Netzwerke ist sehr aufwendig und benötigt sehr viel Rechenleistung, da die Gewichte in der Regel durch Gradientenabstiegsverfahren trainiert werden.
• Für das Erzeugen natürlicher Sprache gibt es verschiedene Ansätze. Es kann eine Grammatik definiert werden, wie dies exemplarisch in den folgenden Schriften beschrieben ist:
• S. Bangalore and O. C. Rainbow, „System and method for natural language generation“, United States Patent US 7,562,005 , July 2009.
• A. Ratnaparkhi, „Trainable dynamic phrase reordering for natural language generation in conversational systems“, United States Patent Application US 2002/0116173 A1 , Aug. 2002.

Examples of this are described in the literature in the following articles:

• A Radford, K Narasimhan, T Salimans, and I Sutskever, Improving language understanding by generative pre-training, Preprint, 2018.
• Radford A, Wu J, Child R, Luan D, Amodei D, and Sutskever I, “Language models are unsupervised multitask learners”, Preprint, 2019.
• Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A, Agarwal S, Herbert-Voss A, G Krueger, T Henighan, R Child, A Ramesh, DM Ziegler, J Wu, C Winter, C Hesse, M Chen, E Sigler, M Litwin, S Gray, B Chess, Clark J, Berner C, McCandlish S, Radford A, Sutskever I, and Amodei D, "Language models are few-shot learners," Preprint, 2020.
• Neural networks are trained using very large amounts of data. The information on how speech is generated is essentially available in the weights of the network. Training the networks is very complex and requires a great deal of computing power, since the weights are usually trained using gradient descent methods.
• There are different approaches to generating natural language. A grammar can be defined, as is described in the following publications:
• S. Bangalore and OC Rainbow, "System and method for natural language generation", United States Patent U.S. 7,562,005 , July 2009.
• A. Ratnaparkhi, "Trainable dynamic phrase reordering for natural language generation in conversational systems", United States Patent Application U.S. 2002/0116173 A1 , Aug 2002.

It is also possible to define a large number of templates for speech generation, which are then filled with content. A disadvantage of such approaches is that the possible sentences are limited to those given by the templates.

• F. Schilder, „Systems and methods for natural language generation“, United States Patent Application US 2014/0149107 A1 , May 2014.
• B. Howald, R. Kondadadi, and F. Schilder, „Systems and methods for natural language generation,“ United States Patent US 9,424,254 , Aug. 2016.
• A. Kaeser, E. Vignon, and L. Stoeckl6, „Systems and methods for natural language generation“, United States Patent US 9,411,804, Aug. 2016.
• Letztendlich extrahieren künstliche neuronale Netze auch nur statistisch relevante Informationen aus den Datensätzen, die für das Training verwendet werden.

The vast majority of known approaches work with statistical methods, as described in the following documents, for example:

• F. Schilder, "Systems and methods for natural language generation", United States Patent Application U.S. 2014/0149107 A1 , May 2014.
• B. Howald, R. Kondadadi, and F. Schilder, "Systems and methods for natural language generation," United States Patent U.S. 9,424,254 , Aug 2016.
• A. Kaeser, E. Vignon, and L. Stoeckl6, "Systems and methods for natural language generation", United States Patent US 9,411,804, Aug. 2016.
• Ultimately, artificial neural networks also only extract statistically relevant information from the data sets used for training.

• M. Ebner, „Distributed storage and recall of sentences“, Bio-Algorithms and Med-Systems, vol. 13, no. 2, pp. 89-101, 2017.

Some time ago, the inventor had developed a method for breaking down sentences into n-tuples and then putting them back together again. This procedure is described in the article identified below, as well as in the DE 10 2016 125 162 B4 described:

• M. Ebner, "Distributed storage and recall of sentences", Bio-Algorithms and Med-Systems, vol. 13, no. 2, pp. 89-101, 2017.

Words with their associated frequencies are stored for each n-tuple. For processing, words are usually replaced by indices into an existing vocabulary.

The individual words are sorted in the vocabulary according to their frequency in the text, so that the word that occurs least frequently in the text corpus has the highest index. Punctuation marks are treated like words. For example, the end-of-sentence characters ".", "?" and "!" are given the indices 1,2,3. Index 0 is reserved for a word unknown in the vocabulary. This is followed in ascending order by the characters ",", ":" and ";". This is followed by the words "die", "der", "und", "das" in the German language. In the English language, the words "the", "to", "a", and "I" follow.

In general, let w ₁ w ₂ w ₃ ... w _l be a sentence of length I, where w _i are the indices of the words of the vocabulary. For training, the entire available text corpus is divided into sentences and the words are mapped to the indices. One calculates the probability p _n (w _i (|w _in ... w _i-1 ) for n > 0 by counting which words are listed how often for a given sequence w _i-1 ... w _i-1 follow this sequence.

Realistically, these probabilities are only calculated for n = 1 and n = 2, since the number of possible sequences increases exponentially. Considerably more storage space is required for n = 3 than for n = 2.

1. C. D. Manning and H. Schütze, Foundations of Statistical Natural Language Processing. Cambridge, Massachusetts: The MIT Press, 1999.
2. C. D. Manning, P. Raghavan, and H. Schütze, Introduction to Information Retrieval. New York, NY: Cambridge University Press, 2008.

A sequence of words in a sentence is referred to as an n-gram in computational linguistics, as is described in the following publications, for example:

1. CD Manning and H. Schütze, Foundations of Statistical Natural Language Processing. Cambridge, Massachusetts: The MIT Press, 1999.
2. CD Manning, P Raghavan, and H Schütze, Introduction to Information Retrieval. New York, NY: Cambridge University Press, 2008.

• A. Z. Broder, S. C. Glassman, M. S. Manasse, and G. Zweig, „Syntactic clustering of the web,“ SRC Technical Note, July 1997.
• T. Dunning, „Statistical identification of language,“ Technical Report MCCS 94-273, Mar. 1994.

Such n-grams can be used for a cluster analysis of websites or to differentiate between different languages, as is given by way of example in the following documents:

• AZ Broder, SC Glassman, MS Manasse, and G. Zweig, "Syntactic clustering of the web," SRC Technical Note, July 1997.
• T. Dunning, "Statistical identification of language," Technical Report MCCS 94-273, Mar. 1994.

The following is the teaching of DE 10 2016 125 162 B4 for a better understanding of the remaining problems, and for a better understanding of the present invention then described.

The probability p ₁ (w _i lw _i-1 ) indicates which word can follow word w _i-1 .

In general, such a probability can be determined for each n EN: $$p_n\left(w_i|w_{i-n}{w}_{i-\left(n-1\right)}...w_{i-1}\right).$$

If the term "probability" is used here and in the following, this term is to be understood in a somewhat broader sense than in the strictly mathematical sense. This means that a "probability" in the sense of the present application does not always necessarily have to assume a value between 0 and 1 (like a probability in the mathematical sense). It is often sufficient if a value is determined as the "probability" which makes it possible to compare the frequency of occurrence of a certain constellation (e.g. the occurrence of a word in combination with one or more other words in a given environment) with the frequency of occurrence to compare corresponding constellations. A correct understanding of the concept of "probability" then always results from the accompanying circumstances given.

In addition to the probability p _{n ,} a probability can also be specified for a word occurring as a function of a preceding word with a certain distance t: p _Δt=t (w _i |w _i-1-t ) for t ∈ {1 ,2, ..., t _max },

The further one goes back in the text, the less influence the word at position w _i-1-t usually has on the word w _i . A reasonable value for t _max is therefore usually at most 5.

The individual probabilities described above can be combined via one or more predetermined mathematical operations, such as a minimum, addition, multiplication or similar operation. In this way, individual word options or combinations can be eliminated.

Beispielsweise könnte dann mittels einer Kombination von p_Δt und p₂ die eingangs beschriebene, wegen des riesigen Speicherbedarfs praktisch kaum zu berechnende Wahrscheinlichkeit p(w_i |w₁w₂ ... w_i-1) wie folgt approximiert werden: $$p\left(w_i|w_1{w}_2\dots w_{i-1}\right)\approx op\left(p_2\left(w_i|w_{i-2}{w}_{i-1}\right);p_{\Delta t}\left(w_i\right)\right)$$

A link or combination of different probabilities using one of these operations is noted below as follows: If different distance-dependent probabilities p _Δt=t are to be combined for different t, this could briefly be: $$p_{\Delta t}=O{p}_{j=\mathrm{1,}...,t}\left\{p_{\Delta t=j}\left(w_i|w_{i-j}\right)\right\}$$

For example, using a combination of p _Δt and p ₂ , the probability p(w _i | w ₁ w ₂ ... w _i-1 ) described at the beginning, which is practically impossible to calculate due to the huge memory requirement, could be approximated as follows: $$p\left(w_i|w_1{\mathrm{<figure-callout\; id="2"\; label="w"\; filenames="DE102021006511A1\_0033.png,DE102021006511A1\_0034.png"\; state="\{\{state\}\}">w</figure-callout>}}_2...w_{i-1}\right)\approx Op\left(p_2\left(w_i|w_{i-2}{w}_{i-1}\right);p_{\Delta t}\left(w_i\right)\right)$$

However, as recognized earlier by the inventor, this approach can still be improved in order to generate sentences to be generated even more accurately with less resource consumption.

For this is in the DE 10 2016 125 162 B4 It has been proposed to record for each sentence a certain number of words with the lowest frequency in the text corpus when running through the sentence (this number is denoted by m _n , for example).

In 1 this is illustrated as an example.

Let _mi,j be the j-rarest word encountered while reading the given sentence from left to right up to position i. Let's read the example sentence
"I hope the sun will shine tomorrow.",
the word “sun” is in the 7th position in the sentence (the comma also occupies one position). If we read the sentence up to the 7th position, we have read “Sonne” as the rarest word, “Hope” as the second rarest word and “Tomorrow” as the third rarest word.

These words ("sun", "hope", "tomorrow") summarize what has been said so far.

Stop words, such as punctuation marks or articles, are eliminated.

We can then use this information to infer the following word "seems".

In 1 it is therefore shown in columns which m _n = 3 words have been found with the smallest frequency when running through the sentence in the reading direction. For each sentence position, this is in 1 specified.

In this context, the following probability p _m can be determined: $$p_m\left(w_i|w_{i-1},m_{i-2.1},m_{i-2.2}\right)$$

This probability p _m can also be combined or linked in the manner described with others, for example one or more of the above-described probabilities.

When using the probability p _m , since the two rarest words for all word positions in the sentence are combined with one another, a large number of tuples have to be stored in the memory.

für verschiedene j zu bestimmen und dann diese einzelnen Wahrscheinlichkeiten zu kombinieren, über eine der vorstehend beschriebenen Operationen: $$p_m\left(w_i|w_{i-1},m_{i-\mathrm{2,1}},m_{i-\mathrm{2,2}}\right)\approx p_{m\text{'}}\left(w_i|w_{i-1}\right)=o{p}_j\left\{p_{m=j}\left(w_i|w_{i-1},m_{i-\mathrm{2,}j}\right)\right\}$$

According to a preferred variant, which manages with a lower memory consumption, only the probability is sufficient $$p_{m=j}\left(w_i|w_{i-1},m_{i-\mathrm{2,}j}\right)$$

for various j and then combine these individual probabilities, via one of the operations described above: $$p_m\left(w_i|w_{i-1},m_{i-2.1},m_{i-2.2}\right)\approx p_{m\text{'}}\left(w_i|w_{i-1}\right)=O{p}_j\left\{p_{m=j}\left(w_i|w_{i-1},m_{i-\mathrm{2,}j}\right)\right\}$$

However, even according to the approaches described above, there are still unsolved aspects.

Because language is not used to reproduce what has been said before, but should also depict a current state of the speaker, for example. This cannot be achieved with the previously known approaches.

The object of the present invention is to propose a method and a device for machine processing of texts which take account of the disadvantages explained above.

This object is achieved by a method and a device having the features of the independent claims. Advantageous refinements and developments are specified in the dependent claims.

Summary of the Invention

A basic concept on which the present invention is built is to first assign a value to each word of the text corpus.

Such a value, which can be represented by a one- or multi-dimensional vector, for example, is used to classify or evaluate the respective word according to specified criteria.

In principle, any application-related classification or evaluation that is considered meaningful can be used here. In particular, empirically derived ratings can be used.

A classification known from the literature in connection with the machine processing of natural language is the so-called VAD classification (cf. S. M. Mohammad, "Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words", in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1), Melbourne, Australia Stroudsburg, PA: The Association for Computational Linguistics, July 2012, pp. 174-184).

VAD stands for Valence, Arousal, and Dominance. Here, words are classified based on primary emotional components.

Based on such an evaluation or classification, a probability can then be determined for a word in the text body that this word appears in a specific context, e.g. in a specific sentence, in combination with a specific state. This specific state in turn is determined in a predetermined manner from the values of words adjacent to the word in the text.

The term "neighboring" is to be understood here rather broadly in the sense of "within a certain distance", e.g. within a certain word spacing. In particular, adjacent does not necessarily mean "immediately adjacent" or "directly adjacent". How big the distance can be, e.g. "maximum five words" or "maximum ten words", or "within a sentence", or "within a paragraph", can vary depending on the application.

A state is first determined or calculated using a set of words that are in the text in the vicinity of the word in question, for example those words that are arranged in the relevant sentence in the reading direction in front of the word in question. The state is then determined based on the values associated with these words, for example by adding (and then scaling if necessary) these values. The state preferably takes on a scalar value. In principle, however, it is also possible to define the state as a vector.

The state can also be calculated in other ways, for example by multiplication. In this case, vector components of vector-valued represented values would first have to be transformed by, for example, an exponential function. Other forms of calculating the state on the basis of the values of the neighboring words are possible and the examples given below are not to be considered as limiting but as illustrative.

By means of the status, the evaluation, which is initially only available word by word, can be extended to a text passage, such as a sequence of words, one or more sentences, etc. The state thus reflects some kind of internal context or situation in which the word in question is found in the text, the context or situation depending on the way in which the individual words are evaluated.

The first probability determined for the word in question then indicates how likely it is that the word in question appears in the text in combination with the determined condition.

In doing so, this first probability is preferably calculated on the basis of the value of the word in question itself compared to the state.

The first probability is calculated for the state or as a function of the state. As explained above, the state results, for example, by mapping a sum of the vectors representing the values of the adjacent words onto a discrete value. There are again various possibilities for the type of mapping. One such possibility is to spatially subdivide the space in which the vectors are defined, e.g., into voxels. A vector sum, which can then also be regarded as a vector in the relevant space, can then be evaluated according to the voxel in which it is arranged. For this purpose, the voxels can be suitably sorted or numbered, as a result of which each voxel is unambiguously assigned a state corresponding to the numbering.

• Ein Text wird erfasst und gespeichert, wobei jedem erfassten Wort ein Wert zugeordnet wird. Dieser Wert kann dann zu dem Wort oder einem dem Wort zugeordneten Index gespeichert werden.

Accordingly, a preferred embodiment of a method for processing a text comprises the following steps:

• Text is captured and stored, with each captured word being assigned a value. This value can then be stored with the word or an index associated with the word.

A first probability is then determined for a sequence of words in the text, e.g., for a sentence, and for a word in the sequence.

This first probability gives a probability with which the word occurs in the sequence in combination with a certain state. This state is calculated from the values of words adjacent to the word in the sequence. The sequence does not necessarily have to contain only consecutive words.

For example, the value of a word can be represented by a one- or multi-dimensional vector.

The state calculated in relation to the neighboring words can then be calculated, for example, by adding the values assigned to the neighboring words. If necessary, a suitable scaling can also be carried out, e.g. by dividing the result of the addition by the number of summands. As mentioned, the state can be defined as vector-valued. However, the state is preferably defined as a scalar value, i.e. for example as a natural or real number.

In this way, for example, if a value assigned to a word is given by a 3-dimensional vector (as in the VAD classification) whose entries each have values in the interval [0, 1], the state can also be a 3 -dimensional vector with values in the same interval can be obtained, as shown in a simple example below.

The text sequence "I'm fine..." is given. The words in it are rated as follows: $$We\mathrm{right}t\left("mi\mathrm{right}"\right)=\left\{\begin{array}{c}0.1\\ 0.5\\ 0.7\end{array}\right\};We\mathrm{right}t\left("GeHt"\right)=\left\{\begin{array}{c}0.7\\ 0.7\\ 0.6\end{array}\right\},We\mathrm{right}t\left("es"\right)=\left\{\begin{array}{c}0.3\\ 0.6\\ 0.2\end{array}\right\}$$

How this evaluation comes about and how it is then evaluated semantically does not initially play a decisive role for the basic idea of the invention. The vector can then be converted to a direction vector. Other types of normalization or scaling are possible.

In the example above, the vector [0.5,0.5,0.5] would be subtracted, giving us positive and negative values in the components. Then the vector is normalized to a unit length. We obtain: $$v\left("mi\mathrm{right}"\right)=\left\{\begin{array}{c}-0.89\\ 0\\ 0.45\end{array}\right\};v\left("GeHt"\right)=\left\{\begin{array}{c}0.67\\ 0.67\\ 0.33\end{array}\right\},v\left("es"\right)=\left\{\begin{array}{c}-0.53\\ 0.27\\ -0.80\end{array}\right\}$$

A state with reference to this text sequence is then calculated, for example, by means of addition as follows: $$Z\mathrm{and}stan\mathrm{i.e}\left("Mi\mathrm{right}GeHtes"\right)=\left\{\begin{array}{c}-0.89\\ 0\\ 0.45\end{array}\right\}+\left\{\begin{array}{c}0.67\\ 0.67\\ 0.33\end{array}\right\}+\left\{\begin{array}{c}-0.53\\ 0.27\\ -0.80\end{array}\right\}=\left\{\begin{array}{c}-0.75\\ 0.94\\ 0.02\end{array}\right\}$$

A state defined in this way would be vector-valued. To calculate the first probability, this state could then suitably be used to calculate the probability of a subsequent word.

• Wenn wir nun den Raum in 3x3x3 Voxel mit Seitenlänge 1 unterteilen, können wir bestimmen, in welchem der verschiedenen Voxel der vektorwertige Zwischenzustand liegt. Im vorliegenden Beispiel entspricht der Vektor [-0.75,0.94,-0.02] dem Voxel [-1,1,0] aus der Menge der Voxel ([-1,-1,-1],... ,[0,0,0],... ,[1,1,1]).

According to a preferred embodiment, starting from the vector-valued intermediate state, a discrete state is to be calculated, as explained below:

• If we now subdivide the space into 3x3x3 voxels with side length 1, we can determine in which of the different voxels the vector-valued intermediate state lies. In the present example, the vector [-0.75,0.94,-0.02] corresponds to the voxel [-1,1,0] from the set of voxels ([-1,-1,-1],... ,[0,0 ,0],... ,[1,1,1]).

The voxels can, for example, be suitably sorted or numbered so that, for example, each voxel is assigned a unique number between 0 and 26. For example, the voxel in question [-1, 1, 0] has the number 5. We therefore identify the discrete state z=5.

If a first probability is to be determined that the word "good" occurs in the text in the preceding text passage ("I'm fine"), we count how often the word "good" in the text corpus matches the word "it" in the text State z=5 follows.

We thus get the probability p _z=5 ("good"|"it") and can use this probability as well as the other above-mentioned probabilities in combination with the calculated state to expand a sentence, ie add another word to this sentence to add.

As the example above shows, the vector representing the value of a word is a k-dimensional vector, where k is a natural number. The individual values associated with the words of the sentence, when added together, result in a path in k-dimensional space. The endpoint of this path is used to define the state.

To this end, the vector space around the starting point is discretized, for example by subdividing the space into individual voxels, as described. Each component of the vector, which was determined based on the ratings of the words of the previous sentence, can assume one of at least b discrete values, where b is also a natural number. A k-dimensional vector, each with b discrete entries, would then permit b ^k different states.

As already mentioned, a vector representing the value can contain a description of the word according to the so-called VAD classification.

On the other hand, it can make sense to use an alternative rating scheme for at least a subset of the text corpus, because VDA ratings, for example, are usually only available for a subset of all words occurring in the text corpus.

In this way it can be ensured that each word of the text corpus is assigned a value in an unambiguous manner.

An example of an alternative evaluation scheme results from assigning a random vector of a corresponding dimension to a word, i.e. a vector whose entries are, for example, (pseudo)random numbers, e.g. in the interval [0, 1].

In principle, different evaluation schemes can be suitably combined. This gives us a different status for each evaluation scheme. The more different states, the clearer the answer can be narrowed down.

Example: For evaluation scheme A we get the condition 5 for the sentence "I'm fine" and possible follow-up words: "good", "not". Possible following words are understood here to be words of the text corpus that follow the specified beginning of the sentence with a comparatively high first probability in the specified state.

For evaluation scheme B we get the state 11 for the sentence "I'm fine" and here the possible follow-up words: "good", "bad".

For the combination of evaluation scheme A with status 5 and evaluation scheme B with status 11, only the subsequent word "good" is possible.

The choice of random evaluation schemes has the advantage that any number of these evaluation schemes can be generated.

A class of words, for example those considered to carry most of the meaning of a text, can be evaluated in a first way, and semantically less important components of the text, such as filler words or punctuation marks, can be evaluated in a different way be rated.

In order to facilitate and standardize a determination of the state, it is advantageous if the respective evaluation schemes are formally compatible, such as VAD vectors and 3-dimensional random vectors with entries in [0, 1].

As already mentioned, the sequence considered together with the word in question can comprise one or more sentences of the text. The adjacent words may appear in the sequence in the reading direction before or after the word in question, or individual words may be omitted, such as those of high relative frequency (and low semantic importance).

The "first probability" supplements or expands the probabilities p _n □ and p _Δt' that were described at the beginning.

As described at the outset, the inventor recognized earlier that different probabilities can be combined when it comes to estimating how likely it is that a word in a text will occur at a specific point, e.g. in a specific sentence.

One of these probabilities is the “first probability” described above in relation to the present invention. The inventor has other probabilities that can be combined with this “first probability” in connection with the DE 10 2016 125 162 B4 already described.

For the sake of completeness, some of them are briefly explained again below with reference to a terminology that is then also found in the dependent claims.

After the text has been recorded and stored, a frequency value can be determined and stored for each recorded word, which represents how often this word occurs in the text compared to other words in the text.

For this purpose, for example, an absolute frequency with which the word occurs in the text can be determined or stored, or a relative frequency, or a number or an index of the word according to a sorting according to the frequency of occurrence. Based on the frequency value, it must only be possible to compare two words in the text with regard to the frequency with which these words occur in the text.

For at least one sentence of the text and for at least one word of the sentence, a second probability can then be determined and stored implicitly or explicitly, with which this word occurs in the sentence in combination with a first tuple of words, defined more precisely below. In the present context, a tuple is understood to be an ordered set, i.e. the order of the elements and their multiplicity must be observed.

The first tuple comprises at least two words of the sentence, which are arranged in front of the word when the sentence is traversed word-by-word, linearly. The sentence can be read in the usual reading direction, i.e. from left to right, or in the opposite reading direction, from right to left.

At least a first of the at least two words of the first tuple occurs with a lower frequency in the text compared to other words already traversed in the sentence.

Furthermore, it applies that at least a second of the at least two words of the first tuple in the sentence is arranged in a predetermined vicinity of the word.

• Der infrage stehende Satz bestehe aus der Wort-Sequenz w₁, w₂, ..., w₁₀ (wobei w_i jeweils einen Index des entsprechenden Wortes bezeichnet), das aktuell betrachtete Wort sei das Wort w₇, d.h. die Wörter w₁ bis w₆ seien bereits durchlaufen (in Leserichtung). Das Wort w₂ sei dasjenige Wort, welches in dem Text im Vergleich zu den weiteren in dem Satz bereits durchlaufenen Wörtern (w₁ bis w₆) mit der geringsten Häufigkeit vorkommen soll. Das Wort w₆ steht in Durchlaufrichtung unmittelbar vor dem Wort w₇.

For better understanding, the properties specified above should be illustrated using a simple, abstract example:

• The sentence in question consists of the word sequence w ₁ , w ₂ , ..., w ₁₀ (where w _i denotes an index of the corresponding word), the currently considered word is the word w ₇ , ie the words w ₁ up to w ₆ have already been run through (in the direction of reading). The word w ₂ is the word that is to occur in the text with the lowest frequency compared to the other words (w ₁ to w ₆ ) already run through in the sentence. Word w ₆ immediately precedes word w ₇ in the direction of flow.

The first tuple then consists of the two words w ₂ and w ₆ , for example: (w ₂ , w ₆ ). In other words, the "at least a first of the at least two words" of the first tuple corresponds to the word w ₂ . The “at least a second of the at least two words” of the first tuple is the word w ₆ , since it is arranged in a predetermined environment, in the example directly preceding the word w ₇ .

The "second probability", in the example a value p(w ₇ l(w ₂ , w ₆ )), does not necessarily have to be calculated and stored as a value between 0 and 1 (like a probability in the mathematical sense).

As already explained above, it is sufficient if a value is determined as the "second probability" which makes it possible to determine the frequency of occurrence of a relevant constellation, i.e. a given first tuple in combination with the word in question in a sentence to compare the frequency of occurrence of corresponding constellations ((tuple, word) in the sentence).

As a rule, this step is carried out for all sentences of the text and all words of a sentence. Any exceptions that may arise at the beginning or end of a sentence for technical reasons should be ignored for the time being.

Punctuation marks are also preferably regarded as words of the text. For technical reasons, a "sentence" usually includes the final punctuation mark, e.g. ".", "!" or "?", of the preceding sentence as the first "word". This makes it easier to recognize the beginning of a sentence.

The usefulness of using the "second probability" rests on two basic observations. These then also apply to the “fourth probability” described below, which is systematically very similar to the “second probability”.

According to a first observation, it has been shown that those words in a sentence are usually the most meaningful, i.e. they essentially convey the meaning of the sentence, which are clearly inferior to other words in the sentence in terms of frequency of occurrence in the text. In other words, rather “rare” words are more important than frequently occurring words in terms of the semantic content of a sentence.

The example already shows that 1 .

The words with the lowest frequency communicate “what the sentence is about”. In the example, the four words with the lowest frequency would result in the sentence "I hope the sun will shine tomorrow". The words with the lowest frequency in the sentence are therefore particularly meaningful. Interestingly, if it were the same sentence in English, the same words would be extracted. The words with the highest frequencies are just filler words that are necessary for a grammatically correct sentence but are not essential for understanding what is being said.

The second important observation is that the relative frequency of words within a sentence is essentially constant as soon as the processed text base becomes sufficiently large.

• Erstens können dadurch, dass das erste Tupel ein oder mehrere den Satz semantisch bestimmende Wörter umfasst, entsprechend dem „zumindest einen ersten Wort“ mit vergleichsweise geringer Häufigkeit, und dadurch, dass weniger wichtige Wörter (entsprechend Füllwörtern) mit hoher Häufigkeit ausgelassen werden, im Vergleich zum klassischen n-Gramm-Ansatz erhebliche Ressourcen, insbesondere Speicherplatz, eingespart werden. Das erste Tupel kann vergleichsweise „kurz“ gehalten werden und umfasst in der Regel z.B. zwei, drei oder vier Wörter. Trotzdem kann ähnlich viel Sinngehalt transportiert werden als in vergleichsweise langen n-Grammen.

These two observations have been processed with the definition of the "second probability":

• Firstly, because the first tuple comprises one or more words that semantically determine the sentence, corresponding to the "at least one first word" with a comparatively low frequency, and because less important words (corresponding to filler words) are omitted with a high frequency, in comparison Considerable resources, especially storage space, can be saved compared to the classic n-gram approach. The first tuple can be kept comparatively "short" and usually comprises two, three or four words, for example. Nevertheless, a similar amount of meaning can be transported as in comparatively long n-grams.

Secondly, the fact that the first tuple in the form of "at least one second word" additionally includes at least one word in a predetermined environment, preferably directly or almost directly adjacent to the word in question, for which the second probability is determined, ensures that at least a certain connection between the end of the currently processed sentence fragment, i.e. the part of the sentence to which the word follows, and the part of the sentence that includes the meaningful words, is retained. In this way, the basic disadvantage of the method described at the outset, which is known from the older prior art and in which only probabilities relating to individual words “in the past” in the sentence were determined separately, can be significantly alleviated. According to the invention, a connection between the end of the sentence and the beginning of the sentence is at least partially retained, even for longer sentences.

According to a preferred embodiment, the first tuple comprises at least three words. At least two of the at least three words occur in the text with a lower frequency than all the other words that have already been run through in the sentence.

In other words, according to this embodiment, the first tuple comprises at least two “meaningful” words. The number of "meaningful" words in the first tuple can vary, for example depending on the length of the sentence currently being considered, the basic structure of the text, the position of the word in the sentence, i.e. the part of the sentence that has already been run through , or similar.

According to a preferred variant of the above embodiment, the at least two “meaningful” words in the first tuple are sorted according to the frequency with which these words occur in the text.

According to a further embodiment, only a single "meaningful" word, preferably the rarest, is used and the at least a second of the at least two words of the first tuple when traversing the sentence is located immediately before the word in question. With that, a sufficient connection to the end of the currently existing sentence fragment, ie the sentence fragment that has already been passed through. However, a distance of eg one or two words is also acceptable here.

According to a further embodiment, a third probability can be determined in addition or as an alternative to the second probability.

This is done again, as described with reference to the second probability, on the basis of the determined values representing the frequencies. The third probability indicates a probability with which the word follows a sequence of consecutive words when running through the sentence, which is determined by a predetermined number of words preferably immediately preceding the word. This sequence of words may be part or all of the sentence, but may also overlap with adjacent sentences. I.e. the third probability can be understood across sentences, with reference to a given sequence of words, whereas the second probability is usually limited to the respective sentence. This means that the first tuple is always formed for a given sentence, not across sentence boundaries.

With reference to the example given above (the sentence in question consists of the word sequence w ₁ , w ₂ , ... , w ₁₀ , the currently considered word is the word w ₇ ), as a third probability, for a given Number three, e.g. the third probability p(w ₇ |w ₄ , w ₅ , w ₆ ) can be determined, ie the probability that the word w ₇ corresponds to a sequence of 3 words w ₄ , w ₅ , w ₆ follows.

According to a further embodiment, as an alternative or in addition to the third probability, a fourth probability can be determined and stored for the word in question. This fourth probability is determined analogously to the second probability, i.e. a probability is determined with which the word occurs in the sentence in combination with a second tuple of words. Like the first tuple, the second tuple comprises at least two words which are arranged before the word when the sentence is traversed. At least a first of the at least two words of the second tuple occurs with a lower frequency in the text compared to other words already traversed in the sentence. At least a second of the at least two words of the second tuple is located in the sentence in a predetermined neighborhood of the word.

The second tuple differs from the first tuple with regard to the definition of the at least one first word, with regard to the definition of the at least one second word, or with regard to the definition of both the at least one first word and the at least one second word.

For example, the first tuple could include the word with the lowest frequency as “the at least one first word”, wherein the second tuple can include the word with the second lowest frequency at this point. Analogously, a distinction can be made between “at least one second word” on the basis of a given environment, i.e. in particular with regard to a distance from the word in question.

The sentence can be traversed in the conventional reading order, left to right, to determine the first probability, the second probability, the third probability, or the fourth probability. Alternatively, the reverse flow direction can be selected, opposite to the reading direction, from right to left. Finally, it is possible and preferred that the respective probabilities for both directions of passage are determined and stored separately. This has the advantage that a sentence to be generated subsequently can be easily extended or supplemented in both directions, starting from any start word from the text.

The method steps described above, in which the explained probabilities (first to fourth) have been determined, describe a first phase of a method for the machine processing of texts. In this first phase, starting from a given text corpus, a database is built up from which sentences can be generated automatically in a second phase, which is described below.

The second phase starts with a step of providing a query set. This can be, for example, a question that is asked of the system that carries out the present method. The query sentence can also be a sentence which results from the fact that other machine devices, for example a speech recognition system or a system for the automatic translation of texts, output a sentence based on the text corpus, for example as a sentence from the speech recognition system tem recognized and output sentence or a sentence output as an automatic translation by the system for the automatic translation of texts.

The query sentence at least partially includes words from the recorded text, but can also include other words that have not previously been recorded. As a rule, query records received in this step are subsequently included in the database, i.e. processed as described above with reference to the first phase. This expands the database. Such an expansion of the database can take place at regular or irregular intervals. However, an embodiment is also conceivable in which one or more query records themselves form the database without any other text having been entered and processed beforehand. According to this embodiment, the at least one query sentence is first understood as a text basis and processed according to phase 1, and only then as an actual “query sentence” in the sense described below.

In a next step, a predetermined number of words is selected from the query sentence. The selection is preferably made in such a way that the selected words occur in the text with a lower frequency than non-selected words of the query sentence, which also occur in the text. In other words, "meaningful" words of the query sentence, recognizable by a comparatively low frequency of occurrence in the text, are selected. The number of words selected may vary depending, for example, on the length of the query sentence.

As described below, the selected words now serve as source or start words for automatically generating an answer sentence. This means that a response sentence to be generated initially consists of only one or a few of the selected words of the query sentence and is then successively supplemented or expanded in one or both possible directions, usually linearly. In the following steps, so-called supplementary words are selected from the recorded text for supplementing or expanding on the basis of the database built up in the first phase, according to the rules described below and added to the currently available answer sentence fragment.

The steps of determining a supplementary word and supplementing the answer sentence, i.e. the answer sentence fragment available at this point in time, can be carried out in relation to different flow directions, i.e. in the reading direction, against the reading direction or, which is preferred, in both directions.

In order to generate the answer sentence, the steps of determining a supplementary word and adding the supplementary word to the answer sentence are generally repeated until the supplementary word is recognized as a predetermined sentence end word. The punctuation marks usually arranged at the end of a natural language sentence, i.e. ".", "?", "!", and possibly also ":" or ";" are used as sentence final words. This applies to both reading directions because, as already described above, a “sentence” for the database is recorded in such a way that the first word of the sentence (in the reading direction) is the final punctuation mark of the previous sentence. The punctuation mark is thus processed as one word of two consecutive sentences. This makes it easy to recognize the beginning of a sentence if an answer sentence is added in the opposite direction to the usual reading direction.

In order to determine a supplementary word, a so-called supplementary probability is determined in a further step.

The supplement probability is determined based on the first probability described above, and specifically may match the first probability.

According to a variant, the complement probability can be determined based on the first probability in combination with the second probability or the third probability or the fourth probability.

As described above, the probabilities can be combined using various mathematical operations.

It is of course also possible to combine the first probability with more than one of the other probabilities, e.g. with the second and the third probability, or with the second and the fourth probability, or with the second, third and fourth probability.

For example, the complement probability can be determined as the minimum of the first and the second (or the first and the third) probability or as the sum of the first probability with the second probability and/or the third probability. Analogously, the probability of supplementation can take place through product formation on the basis of the first and the second (and/or third) probability.

From the above it is clear that the "supplement probability" does not necessarily have to be a probability in the mathematical sense, but merely a well-defined value by means of which different potential complement words can be compared in terms of how "probable" (measured with the first probability and, if necessary, the second/third/fourth probability) it is according to the detected and processed text corpus that the answer sentence fragment currently present continues in a predetermined direction of passage through the complement word.

• Es wird ein Ergänzungswort aus dem Text dadurch bestimmt, dass das Ergänzungswort aus einer Menge vorgegebener Größe von potentiellen Ergänzungswörtern ausgewählt wird, welche aus denjenigen potentiellen Ergänzungswörtern besteht, die mit Bezug zu einer vorgegebene Durchlaufrichtung und mit Bezug zu dem Antwortsatz die höchsten Ergänzungswahrscheinlichkeiten aufweisen.

In the subsequent step, as already generally described, an answer sentence is generated based on at least one of the selected words, based on the selected word as the start word and on the basis of the probability of completion determined. The following sub-steps are carried out for this purpose:

• A supplementary word is determined from the text by selecting the supplementary word from a set of a predetermined size of potential supplementary words, which consists of those potential supplementary words which have the highest supplementary probabilities in relation to a predetermined flow direction and in relation to the answer sentence.

In other words, based on the current answer sentence fragment and with reference to a given flow direction, the "most probable" additions are combined into a set based on the data processed in the first phase, and the additional word for the current step is then selected from this set.

According to a first simple variant, the quantity can have the size one, i.e. only include that supplementary word with the maximum probability of being completed. Then the step of selecting is unnecessary.

If the set includes a number of potential supplementary words, the selection can be made according to various criteria or also according to a heuristic. According to a variant that is preferred in this context, in order to select the supplementary word from the set of potential supplementary words, a similarity of one of the potential supplementary words to the query sentence can be determined according to a predetermined degree of similarity.

The similarity can be of a syntactic nature, for example, and simply measure the maximum character similarity of the supplementary word to a word of the query sentence. Alternatively or additionally, the similarity can include a semantic component and, for example, indicate a similarity on the basis of a predefined synonym database.

A potential supplementary word is then selected as supplementary word from a subset of the set of potential supplementary words, the subset consisting of those potential supplementary words in the set that have the greatest similarities to the query sentence within the set of potential supplementary words. A supplementary word can then be selected from the subset according to a predetermined rule or heuristic.

Finally, in the final sub-step, the answer sentence, i.e. the currently available answer sentence fragment, is supplemented or extended by the supplementary word in the direction of flow.

In the manner described, the answer sentence can be supplemented step by step in one or both directions by repeated application of the steps described above until the beginning and end of a sentence are recognized on the basis of recognized sentence terminating words. A response set recognized as complete can be output.

It goes without saying that according to the above method, starting from a start word, not only one answer sentence can be generated, but a plurality of answer sentence variants. Starting from a selected word, a plurality of answer sentence variants can be formed in that the answer sentence, ie the respective fragment of an answer sentence variant, is in each case supplemented by a supplementary word from the set of predetermined size of potential supplementary words. If the set of potential supplementary words always includes four supplementary words, four possible answer sentence variants can be generated in each stage, ie in each supplementary step; the number of generated variants is thus a power of 4. According to this embodiment, the similarity analysis with reference to the query sentence can also be used as an alternative or in addition to support the selection of a supplementary word, for example if the number of potential supplementary words is greater than the number of variants to be generated in each step.

The response sentence variants generated in each case can then be output. According to a preferred embodiment, the answer sentence variants can be evaluated using at least one of the following evaluation criteria in a third phase. At least one answer sentence can then be selected from the answer sentence variants based on the evaluation and output.

• • Summe der Wortfolgewahrscheinlichkeiten;
• • Summe der n-Gramm-Wahrscheinlichkeiten;
• • Informationsgehalt, auf Basis der Häufigkeiten, mit der die Wörter der jeweiligen Antwortsatzvariante im Text vorkommen;
• • Ähnlichkeit der Antwortsatzvariante zum Anfragesatz oder zu einem Teil des Anfragesatzes.

Examples of possible evaluation criteria are:

• • sum of word order probabilities;
• • sum of n-gram probabilities;
• • Information content, based on the frequency with which the words of the respective answer sentence variant appear in the text;
• • Similarity of the answer sentence variant to the query sentence or to a part of the query sentence.

• Ein Antwortsatz ist durch die Wortsequenz w₁, w₂, ..., w_n gegeben, der Anfragesatz wird mit $w_1^{\text{'}},w_2^{\text{'}},\dots ,w_{l_1}^{\text{'}}$
  
  bezeichnet.

The following designations are used to explain the evaluation criteria:

• A response sentence is given by the word sequence w ₁ , w ₂ , ..., w _n , the query sentence is given with $w_1^{\text{'}},w_2^{\text{'}},...,w_{l_1}^{\text{'}}$
  
  designated.

wobei mit p_r bzw. p_l jeweils die Wortfolgewahrscheinlichkeit in Leserichtung bzw. gegen die Leserichtung bezeichnet werden.A sum S ₁ of word order probabilities is determined as follows: $$s_1={\displaystyle {\sum }_{i=2}^n{p}_{\mathrm{right}}}\left(\left(w_i|w_{i-1}\right)+p_l\left(w_{i-1}|w_i\right)\right),$$

where p _r and p _l respectively denote the word sequence probability in the reading direction and against the reading direction.

A sum s ₂ of n-gram probabilities is determined as follows: $$s_2={\displaystyle {\sum }_{i=4}^n{p}_{\mathrm{right}}\left(\left(w_i|w_{i-3},w_{i-2},w_{i-1}\right)+p_l\left(w_{i-3}|w_{i-2},w_{i-1},w_i\right)\right)},$$

An information content S ₃ of a message is measured by sorting the words of the text based on frequency and indexing them accordingly, and then adding up the word indices w _i of the response sentence: $$s_3={\displaystyle \sum _{i=1}^n{w}_i}$$

wobei mit $w_1^{"},w_2^{"},\dots ,w_{l_2}^{"}$

die unterscheidbaren Wörter des Anfragesatzes $w_1^{\text{'}},w_2^{\text{'}},\dots ,w_{l_1}^{\text{'}}$

bezeichnet werden, und mit δ das Kroneckersymbol.For example, a similarity s ₄ of the response set to the query set can be calculated as follows: $$s_4={\displaystyle {\sum }_{i=1}^n{\displaystyle {\sum }_{j=1}^{l_2}\delta \left(w_i,w_j^{"}\right)}},$$

where with ${\mathrm{<figure-callout\; id="1"\; label="w"\; filenames="DE102021006511A1\_0032.png,DE102021006511A1\_0033.png"\; state="\{\{state\}\}">w</figure-callout>}}_1^{"},{\mathrm{<figure-callout\; id="2"\; label="w"\; filenames="DE102021006511A1\_0033.png,DE102021006511A1\_0034.png"\; state="\{\{state\}\}">w</figure-callout>}}_2^{"},...,{\mathrm{<figure-callout\; id="2"\; label="w\; l"\; filenames="DE102021006511A1\_0033.png,DE102021006511A1\_0034.png"\; state="\{\{state\}\}">w</figure-callout>}}_{l_{}}^{{}_2}$

the distinguishable words of the query sentence $w_1^{\text{'}},w_2^{\text{'}},...,w_{l_1}^{\text{'}}$

be denoted, and δ the Kronecker symbol.

wobei die Wörter $w_1^s,w_2^s,\dots ,w_k^s$

diejenigen Wörter des Anfragesatzes mit den geringsten Häufigkeiten sind.Alternatively, a similarity S ₅ of the response set to the query set can be determined as: $$s_5={\displaystyle {\sum }_{i=1}^n{\displaystyle {\sum }_{j=1}^k\left(k-j-1\right)\delta \left(w_i,w_j^s\right)}},$$

where the words $w_1^s,w_2^s,...,w_k^s$

are those words of the query sentence with the lowest frequencies.

Each of these evaluation criteria can be used individually or in a suitable combination with one or more other evaluation criteria.

Each of these scoring criteria can also be normalized by dividing the score S _i (a) for a set a=w ₁ , w ₁ ,..., w _n by the sum or maximum of all scores.

The ratings normalized in this way can be summed up, resulting in an overall rating S of a sentence: $$s\left(a\right)={\displaystyle {\sum }_{i=1}^5\frac{s_i\left(a\right)}{ma{x}_b{s}_i\left(b\right)}.}$$

Alternatively, the ratings could also be weighted differently.

Also, instead of normalizing the scores, a rank of the score could be determined (the larger the score, the larger the rank). The individual ranks could then also be summed or weighted to determine an overall score s'(a).

A preferred embodiment of a device according to the invention for machine processing of a text comprises at least one memory and at least one processor.

The device is set up to carry out a method as described above, at least the step of acquiring and storing the text and the step of determining the first probability, but preferably also the steps necessary for generating an answer sentence.

According to a first preferred embodiment, the device comprises a speech recognition system or a system for the automatic translation of texts.

The device is then set up, upon input of a query sentence, which is a sentence fragment output by the speech recognition system based on a partially processed speech input or a sentence fragment output by the system for automatic translation of texts as an automatic partial translation, to generate a response sentence which contains the respective Sentence fragment added.

The supplementation takes place in such a way that a recognition accuracy of the speech recognition system or a correctness of the system for the automatic translation of texts can be improved by restricting a set of supplementary words that are fundamentally used by the speech recognition system or the system for the automatic translation of texts on the basis of the supplementary probability.

This means that the further speech input or the rest of the text to be translated can be interpreted by means of the method according to the invention, depending on the previously understood speech input or the previously translated portion of a sentence.

According to a second preferred embodiment, the device can further comprise an input device for a user of the device to input at least part of the query sentence, for example a keyboard, a touch-sensitive display, a voice input or the like.

The device also includes an output device for outputting a plurality of supplementary words which have been determined as part of the generation of an answer sentence or an answer sentence variant, for example a display or a voice output.

Furthermore, the device includes a selection device, by means of which the user can select a supplementary word from the plurality of supplementary words output to supplement the answer sentence or the answer sentence variant.

This device can, for example, help speech-impaired people to articulate themselves in a simple way.

character list

• 1 basierend auf einer Datenbasis von ca. 38 Mio. Wörtern, welche m_n = 3 Worte mit der kleinsten Häufigkeit bei Durchlaufen eines angegebenen Satzes in Leserichtung an jeder Leseposition aufgefunden worden sind;
• 2a einen Pfad durch einen Zustandsraum aufgrund einer ersten Bewertung einzelner Worte eines Satzes;
• 2b zwei verschiedene Pfade durch den Zustandsraum, wenn zwei verschiedene Bewertungsschemata zur Bewertung verwendet werden, für die gleichen Worte des Satzes;
• 3 Schritte einer bevorzugten Ausführungsform eines Verfahrens zum maschinellen Verarbeiten von Texten;
• 4A eine erste bevorzugte Ausführungsform einer Vorrichtung zum maschinellen Verarbeiten von Texten;
• 4B eine zweite bevorzugte Ausführungsform einer Vorrichtung zum maschinellen Verarbeiten von Texten; und
• 4C eine dritte bevorzugte Ausführungsform einer Vorrichtung zum maschinellen Verarbeiten von Texten.

The invention is explained in more detail below using an exemplary embodiment and associated drawings. The figures show:

• 1 based on a database of approximately 38 million words, which m _n = 3 words have been found with the smallest frequency when running through a specified sentence in the reading direction at each reading position;
• 2a a path through a state space based on a first evaluation of individual words of a sentence;
• 2 B two different paths through the state space, if two different scoring schemes are used for scoring, for the same words of the sentence;
• 3 Steps of a preferred embodiment of a method for processing text;
• 4A a first preferred embodiment of a text machine processing apparatus;
• 4B a second preferred embodiment of a text machine processing apparatus; and
• 4C a third preferred embodiment of an apparatus for processing texts.

Detailed description of the invention

As already explained at the beginning, speech is not just used to reproduce what has been said before, but is sometimes also intended to reflect a current state that only results indirectly, e.g. from a text environment or based on measured sensor data.

Such a state can also reflect a state of a writer or speaker of the respective text. In this context, writer or speaker also means a robot or other machine that determines its status using sensors (temperature, distance sensors, position sensors).

Let's look at the example mentioned above. The beginning of the sentence "I'm fine..." should be continued. For example, we could use the words "good" or "bad" depending on a certain state. This state can also depict a situation of the speaker/writer.

welche angeben, wie wahrscheinlich das Wort w_i im Zustand Z1 und im Zustand Z2 erscheint.We calculate probabilities as explained above $$p_{Z\mathrm{and}stan\mathrm{i.e}=\mathrm{e.g}1}\left(w_i|w_{i-1}\right)\text{and}{p}_{Z\mathrm{and}stan\mathrm{i.e}=\mathrm{e.g}2}\left(w_i|w_{i-1}\right),$$

which indicate how probable the word w _i appears in state Z1 and in state Z2.

As explained above, assuming that a value associated with a word is a 3-dimensional vector with entries in the interval [0,1], and that a state to be computed via such values can be represented in the same way, it would be in a "positive" state Z1, which is characterized by rather high vector components, such as [1.0, 1.0, 1.0], the first probability of the word "good" (which also has a "positive" value, such as [0.8, 0.75 , 0.9]) higher than the first probability of the word "poor", which tends to have a low score (such as [0.2, 0.35, 0.1]).

In an analogous way, the first probability for the word "bad" in a state Z2, which describes more of a negative attitude (such as [0.0, 0.0, 0.0]), would be significantly higher than the first probability for the word "good".

Events from the outside as well as from the inside can bring about a change in the human condition. Basic emotions such as joy, surprise, anger, sadness, fear or disgust could be used to determine the condition.

Ein Wert von z_i = 0 würde z.B. die Abwesenheit eines Hormons signalisieren, während z_i = 1 die maximale Anwesenheit desselben Hormons signalisieren würde.Even more elementary, hormones or substances in the blood can be used to determine the condition. Let's take a k-dimensional vector to describe the state $$\mathrm{e.g}=\left({\mathrm{e.g}}_1,{\mathrm{e.g}}_2,...,{\mathrm{e.g}}_k\right)\text{with}{\mathrm{e.g}}_i\in \left[0.1\right].$$

For example, a value of _zi = 0 would signal the absence of a hormone, while _zi = 1 would signal the maximum presence of the same hormone.

If we discretize each component with b levels, we get b ^k different states. Each individual state z can be mapped to one of the b ^k different discrete states. Let d(z) be the mapping that assigns one of the discrete states to each vector z.

In addition to endogenous hormones, the words that have been spoken so far can also cause the speaker or the listener to enter a certain state. As already described, there are lexicons that specify how each word is to be classified along the three dimensions of valence/arousal/dominance, so-called VAD lexicons.

Let e(w _i )=[e _v , e _a , _ed ] be a 3-dimensional vector for such a classification, which defines a directional vector for a word w _i .

When processing a text sequence, such as a partial sentence, the values e(w _i ) can be summed up for all words in this sequence, it being possible to ensure by suitable limitation (as already mentioned) that the summation covers the defined range, eg [-1 , 1] or [0,1], does not leave componentwise.

We have already explained above with reference to the sub-sentence "It's me..." as an example of how corresponding probabilities can be determined with reference to such an evaluation.

$op\{(p_2\left(w_i|w_{i-2}{w}_{i-1}\right);$

Unmittelbarer Kontext $p_{\Delta t}\left(w_i\right));$

Kurzzeitlicher Kontext $p_{m^{\text{'}}}\left(w_i\right));$

Langfristiger Kontext $p_{Zustand=d}\left(w_i|w_{i-1}\right)\}$

Situationsbedingter Kontext „op {...}“ bezeichnet hier, wie bereits erwähnt, eine oder mehrere geeignete mathematische Operationen zum Kombinieren dieser Wahrscheinlichkeiten, wie z.B. Minimum, Addition oder Multiplikation.As also already mentioned, this probability can be combined with other probabilities that have been described above to produce a probability that cannot be calculated in practice for resource reasons
p(w _i |w ₁ w ₂ ... w _i-1 ) to approximate: $p\left(w_i|w_1{\mathrm{<figure-callout\; id="2"\; label="w"\; filenames="DE102021006511A1\_0033.png,DE102021006511A1\_0034.png"\; state="\{\{state\}\}">w</figure-callout>}}_2...w_{i-1}\right)\approx$

$Op\{(p_2\left(w_i|w_{i-2}{w}_{i-1}\right);$

immediate context $p_{\Delta t}\left(w_i\right));$

short-term context $p_{m^{\text{'}}}\left(w_i\right));$

long-term context $p_{Z\mathrm{and}stan\mathrm{i.e}=\mathrm{i.e}}\left(w_i|w_{i-1}\right)\}$

situational context As already mentioned, "op {...}" denotes one or more suitable mathematical operations for combining these probabilities, such as minimum, addition or multiplication.

wenn w_n die Wortalternative ist, ein Maß für die Worthäufigkeit oder ein Maß für die umgekehrte Worthäufigkeit.The probabilities can also be suitably transformed either before using the "op" operation or after. Possibilities include, for example, the calculation of the relative probabilities, the calculation of the probability of the alternatives taking into account the preceding words

if w _n is the word alternative, a measure of word frequency or a measure of inverse word frequency.

Instead of using a VAD lexicon, the vectors e(w) can, as mentioned, at least partially be replaced by random vectors. VAD lexicons usually only classify a subset of all words. Stop words usually do not have a VAD classification. When using random vectors however, all words can be assigned a non-zero classification vector. This means that neutral sentence beginnings, such as "I'm fine..." can also be used to classify possible follow-up words. In this way, a sentence can be broken down into fragments and put back together again.

The approach outlined above is in the 2A and 2 B illustrated.

Each word from the sentence changes the state according to the associated word-state vector. For each word, the subsequent probabilities p(w _t |w _i-1 ) can be divided into b ^k different probabilities P _state = _d (wi|w _i-1 ).

As long as ^bk is sufficiently large, sentences can be broken down into their individual parts and reassembled in this way.

bezeichnet.The probability p _state=d (w _i lw _i-1 ) is also called the situational probability

designated.

The method is also scalable through the use of random vectors.

verwendet werden, indem für $p_d\left(w_i|w_{i-1}\right)$

andere Bewertungsvektoren (VAD-Vektoren oder zufällig gewählte Vektoren) verwendet werden als für $p_d^{\text{'}}\left(w_i|w_{i-1}\right).$

There may well be multiple probabilities $p_{\mathrm{i.e}},p_{\mathrm{i.e}}^{\text{'}},p_{\mathrm{i.e}}^{"}$

be used by for $p_{\mathrm{i.e}}\left(w_i|w_{i-1}\right)$

other evaluation vectors (VAD vectors or randomly chosen vectors) are used than for $p_{\mathrm{i.e}}^{\text{'}}\left(w_i|w_{i-1}\right).$

This results in different end positions in the state space for one and the same beginning of a sentence. The possible subsequent words are thus distributed over the state space based on the beginning of the sentence and thus allow the sentence to be assembled.

(shown below) shows experimental results. Word classification b l p _d l q _n qe _{_} i none 3 0 0.390 0.342 12,247 VAD 3 1 0.390 0.342 12,247 Random 3 1 0.420 0.344 12.205 Random 5 1 0.440 0.355 11,981 Random 9 1 0.490 0.406 10,675 Random 3 2 0.430 0.350 12.172 Random 5 2 0.470 0.371 11,787 Random 9 2 0.520 0.429 9,598 Random 3 3 0.450 0.353 12,140 Random 5 3 0.470 0.382 11,698 Random 9 3 0.530 0.448 9,085
. Reproduction of sentences from the corpus. q _n is the proportion of sentences generated correctly (bigger is better). q _e Percentage of words that were correctly identified as the next word (bigger is better). i mean index of the correct following words in the sorted list of following words (smaller is better). VAD means that p _d was determined according to a VAD lexicon, Random expresses that random vectors were determined for the word description.

A text corpus with over 38 million words was used. From this text corpus, 100 sentences of length 10 were randomly selected.

Each sentence was shortened to 5 words in length and all sentences beginning with the same word were extracted from the text corpus.

Then the sentence fragment of length 5 was expanded to a complete sentence using the previously mentioned probabilities. The probability of selection was replaced by the word index after applying a minimum operator. This selects rare words first.

In the quota q _n of the correctly generated sentences, i.e. the sentences that are exactly as they are in the corpus, is given.

The number of probabilities lp _d | used was varied from {0, 1, 2, 3}. In addition, the results obtained when using a VAD lexicon are given for comparison.

In addition to the quota q _n, two other key figures were calculated. For this purpose, all possible subsequent words (due to the evaluation corpus) are determined for each sentence fragment. There are n _e following words possible. Then we determined all possible expansion words using the procedure and chose the n _e most likely words. We then calculate the proportion of words q _e that were correctly identified as follow-up words. In addition, we determine the normalized, mean index i of the correct subsequent words in the list of subsequent words determined by the method.

The above probabilities can be calculated for a given sentence according to the reading direction from left to right or alternatively in the opposite reading direction from right to left. In this way, a sentence fragment can be completed either from left to right or from right to left. This can be used for generating answer sets based on keywords like as already mentioned in the DE 10 2016 125 162 B4 had been described.

Devices that can respond to natural language input can be used in many areas: smartphones, tablets, PCs and other digital devices, cars, vending machines. The aforementioned technology can also be used to care for older people.

A preferred embodiment of a method for machining a text is referred to with the following 3 briefly described.

The process consists roughly of three phases.

In phase P1, the indices of the words are determined (step S1), then the “first probabilities” p _d are determined (step S2), which are also referred to as “situational probabilities”.

In addition, one or more of the described probabilities can be determined (step S3), such as the "second probability", the "third probability" or the "fourth probability".

Phase P2 responds to a query set.

To this end, the query record is first provided (step S4). Words that occur with low frequency in the query sentence are selected from the query sentence (step S5).

Each word from the query sentence is then iteratively completed by determining a completion probability (step S6.1) to form an answer sentence.

The complement probability is used to determine a complement word (step S6.2).

This supplementary word completes the answer sentence step by step (step S6.3), so that a complete answer sentence is finally created.

In phase P3, the generated answer sentences are evaluated (step S7) and then an answer sentence is selected (step S8).

As already explained in detail, the following can be used as evaluation criteria: the sum of the word order probabilities, the sum of the n-gram probabilities, the information content, based on the frequency with which the words of the respective answer sentence variant occur in the text, similarity of the answer sentence variant to the query sentence or to a part of the query sentence.

A corresponding procedure is in DE 10 2016 125 162 B4 , but already described in detail there without reference to the “first probability” according to the invention.

Finally, in the 4A . 4B and 4C show three preferred embodiments of a device 100, 101, 102 for machine processing of texts. These each include a memory 110 and at least one processor 120, with the devices 100, 100' each being set up to carry out a method of the type described above.

The embodiment 100 after 4A comprises an input device 500, for example a keyboard, a touch-sensitive display, a voice input or the like, and an output device 600, for example a display or a voice output. The device 100 is set up to output a response sentence via the output device 600 when a query sentence is input by means of the input device 500, said reply sentence having been generated by means of the method described above.

The embodiment 101 after 4B comprises a speech recognition system 200 or alternatively a system 300 for the automatic translation of texts.

The device 100 is set up, on the basis of a query sentence, which is a sentence fragment output by the speech recognition system 200 or a sentence fragment output by the system 300 for the automatic translation of texts as a partial automatic translation, to generate a response sentence in the manner described in detail above, which the respective sentence fragment is added.

This answer set can be used, for example, to improve the recognition accuracy of the speech recognition system 200 or the correctness of the system 300 for automatically translating texts, since a set of supplementary words (for the interpretation of speech data or for the translation of a term) that is fundamentally considered by the respective system is based on the according to the invention certain probability of supplementation can be significantly restricted.

In a speech recognition system, information is processed which, for example, was recorded by a microphone or stored in a file. During speech recognition (when part of the audio information has been processed), speech recognition can be improved using the method presented. For this purpose, what has been said so far or the audio information processed so far is regarded as an answer sentence fragment that needs to be supplemented. The method presented here provides a number of possible supplementary words for this answer sentence fragment. When recognizing spoken language, there may be alternative interpretation options. The alternative interpretation possibilities can be limited by the supplementary words provided by the procedure. This can improve the recognition rate. The probability of the possibility of interpretation can be increased if the corresponding supplementary word "is available" based on the procedure presented here, i.e. it has a positive supplementary probability. Thus, sentences that can be generated on the basis of the database are more likely to be recognized.

In addition, speaker-independent recognition can be promoted. It is no longer necessary to recognize the content of the spoken language. Rather, it is sufficient to determine which of the supplementary words the following audio information most closely corresponds to.

A distinction between word alternatives (the supplementary words) is therefore sufficient.

Improving translation systems works analogously. In a translation system, the text translated so far can be viewed as an answer sentence fragment. Alternative words may be available from the translation system to continue the sentence generated so far. The selection probabilities of the words in the target language that are suggested as supplementary words by the method presented here are then increased. This improves the quality of the translation, since the database serves as the basis for the translation.

The device 100 can additionally, by means of a comparison device (not shown), automatically compare the response set generated with the query set according to predefined rules, for example with regard to syntactic and/or semantic similarity.

The third embodiment 102 after 4C also includes an input device 500 as described above, which is set up for a user of the device to input at least part of the query record. The device 102 also includes an output device 600 as described above, which is set up to output a plurality of supplementary words which have been determined as part of the generation of an answer sentence or an answer sentence variant, for example a display or a voice output. Furthermore, the device 102 includes a selection device 700, by means of which the user can select a supplementary word from the plurality of supplementary words output to supplement the answer sentence or the answer sentence variant. The selection device 700 can be combined with the input device 500 . Likewise, as shown in the figure, a touch-sensitive display can serve as a combined input and output device 500, 600.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• US 7562005 [0003]
• US20020116173A1 [0003]
• US20140149107A1 [0005]
• US9424254 [0005]
• DE 102016125162 B4 [0006, 0013, 0022, 0081, 0190, 0203]

Claims (15)

A method for processing a text, comprising the steps of: - capturing and storing the text, assigning a value to each captured word; - For a sequence of text and for a word of the sequence: Determining (S1) a first probability with which the word occurs in the sequence in combination with a state that is calculated from the values of words in the sequence that are adjacent to the word. procedure after claim 1 , where the value of a word is represented by a one or more dimensional vector, and where the state calculated with respect to the neighboring words is calculated by adding the values assigned to the neighboring words. procedure after claim 2 , where the vector is a k-dimensional vector, where k is a natural number, and where each component of the vector can take on at least b discrete values, where b is a natural number. procedure after claim 2 or 3 , where the vector comprises a description of the word according to an empirically chosen classification, for example in the form of a VAD vector, or where the vector corresponds to a random vector. Procedure according to one of Claims 1 until 4 , wherein the sequence comprises one or more sentences of the text, and wherein the adjacent words are arranged in the sequence in the reading order before the word or wherein the adjacent words are arranged in the sequence after the word in the reading order. Procedure according to one of claims 2 until 5 , where the first probability that the word occurs in the sequence in combination with the state is calculated based on the value of the word compared to the state. Procedure according to one of Claims 1 until 6 , comprising the further steps: - for each detected word: determining a frequency value which represents how often this word occurs in the text compared to other words in the text; - For a sentence of the text and for a word of the sentence: determining and storing (S3) a second probability with which the word occurs in the sentence in combination with a first tuple of words, the first tuple comprising at least two words , which are placed before the word when traversing the sentence; - wherein at least a first of the at least two words occurs with a lower frequency in the text compared to other words already traversed in the sentence; and - wherein at least a second of the at least two words in the sentence is located in a predetermined neighborhood of the word, and/or determining and storing (S3) a fourth probability with which the word in the sentence in combination with a second tuple of words occurs, wherein the second tuple - comprises at least two words which are placed before the word when traversing the sentence; - wherein at least a first of the at least two words occurs with a lower frequency in the text compared to other words already traversed in the sentence; - wherein at least a second of the at least two words in the sentence is arranged in a predetermined environment of the word, and - wherein the second tuple differs from the first tuple with regard to the at least one first word and/or with regard to the at least one second word. Procedure according to one of Claims 1 until 7 , comprising the further step: determining and storing (S3) a third probability with which the word follows a sequence of consecutive words, which is determined by a predetermined number of words preceding the word, in the sentence or in a sentence adjacent to the sentence. Procedure according to one of Claims 1 until 8th , comprising the further steps: - providing (S4) a query set; - selecting (S5) a predetermined number of words from the query sentence, the selected words occurring in the text and compared to non-selected words of the query sentence, wel che also appear in the text appear in the text with a lower frequency; - determining (S6.1) a complement probability for determining (S6.2) a complement word; - based on the first probability, or - based on the first probability in combination with at least one of the second probability, the third probability and the fourth probability; - For at least one of the selected words: Generating (S6) an answer sentence, starting from the selected word, on the basis of the determined completion probability, by means of the following sub-steps: - Determining (S6.2) a complementary word from the text, the complementary word consisting of a Set of predetermined size of potential supplementary words is selected, which consists of those potential supplementary words that have the highest supplementary probabilities in relation to a predetermined flow direction with respect to the answer set. - Supplementing (S6.2) the answer sentence in the flow direction with the supplementary word. procedure after claim 9 , wherein to generate the answer sentence, the steps of determining a supplementary word and completing the answer sentence are repeated until the supplementary word is recognized as a sentence terminating word, wherein the steps of determining a supplementary word and completing the answer sentence can take place with respect to different traversal directions. Procedure according to one of claims 9 until 10 , starting from a selected word, a plurality of answer sentence variants are formed in that the answer sentence is each supplemented by a supplementary word from the set of predetermined size of potential supplementary words. procedure after claim 11 , comprising the further steps: evaluating (S7) the answer sentence variants using at least one of the following evaluation criteria: - sum of the word order probabilities; - sum of n-gram probabilities; - Information content, based on the frequency with which the words of the respective answer sentence variant occur in the text; - Similarity of the answer sentence variant to the query sentence or to a part of the query sentence, and selecting (S8) at least one answer sentence from the answer sentence variants on the basis of the evaluation. Device (100; 101; 102) for machine processing of a text, comprising at least one memory (110) and at least one processor (120), wherein the device is set up, a method according to one of Claims 1 until 12 to execute. Device (101) according to Claim 13 , comprising a speech recognition system (200) or a system (300) for the automatic translation of texts, wherein the device is set up, upon input of a query sentence, which is a sentence fragment output by the speech recognition system (200) on the basis of a partially processed speech input or one of the System (300) for the automatic translation of texts as an automatic partial translation output sentence fragment is to generate an answer sentence, which supplements the respective sentence fragment, so that a recognition accuracy of the speech recognition system (200) or a correctness of the system (300) for the automatic translation of texts can be improved by restricting a set of complement words used by the speech recognition system (200) or the system (300) for the automatic translation of texts on the basis of the complement probability. device (102). Claim 13 or 14 , further comprising an input device (500) for entering at least part of the query sentence by a user, and an output device (600) for outputting a plurality of supplementary words which have been determined during the generation of an answer sentence or an answer sentence variant, further comprising a selection device (700), by means of which the user can select a supplementary word from the plurality of supplementary words for supplementing the answer sentence or the answer sentence variant.

Images