RU2568272C2 - Device for semantic analysis of text data - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: claimed device comprises the set of input registers, input registers of the division unit set, set of the units of subtraction modulo, accumulation adder, division unit, comparator unit, indication unit, clock pulse oscillator and pulse distributor.

EFFECT: higher validity of test data authorship.

2 dwg, 1 tbl

Description

The invention relates to computer technology and can be used in calculations related to the identification of textual information in cases of determining its author.

Applicants are not aware that such a technical problem would be handled in this way. This is because there are no numerical methods for meaningful analysis of various parts of textual information.

The technical result of the claimed solution is to increase the level of reliability of determining the authorship of textual information and expand the arsenal of technical means. This makes it possible to make more objective decisions when protecting the copyrights of the creators of textual information.

A numerical content analysis of textual information is carried out as follows. Two passages of textual information can be compared by comparing information about the probabilities of a letter appearing in two different passages. The average value of the difference ΔP _cf between the probability of occurrence of the ith letter of passage "a" P _ai , and the probability of occurrence of the jth letter of passage "b" P _bj is estimated by the following formula:

$$\Delta P_{\mathrm{from}R}=\frac{\mathrm{one}}{n}{\displaystyle \sum _{k\equiv \mathrm{one}}^n{\left|P_{\mathrm{but}i}-P_{bj}\right|}_k},k=\mathrm{one,}...n,\left(\mathrm{one}\right)$$

i = 1, ..., m _аi , j = 1, ..., m _бj ,

where n is the number of letters in the alphabet

m _ai - the number of i-th letter in the passage "a",

m _bj is the number of the jth letter in the passage "b".

Comparing the value of ΔP _cf with a permissible value of ΔP _d , we can conclude that two passages “a” and “b” of textual information belong to one author. If ΔP _av ≤ΔP _d , then passages “a” and “b” belong to the same author. Otherwise (ΔP _cf > ΔP _d ) the authors of these passages are different.

The probabilities P _ai and P _{bj are} determined by the following formulas:

$$P_{\mathrm{but}i}=\frac{m_{\mathrm{but}i}}{N_{\mathrm{but}}},i=\mathrm{one}...,m_{\mathrm{but}},\left(2\right)$$

$$P_{bi}=\frac{m_{bi}}{N_b},j=\mathrm{one,}...,m_b,\left(3\right)$$

where N _a is the total number of letters in the passage "a",

N _b - the total number of letters in the passage "b".

To illustrate the developed approach, it is advisable to consider an example. As objects of meaningful analysis, one can use the poems of Joseph Brodsky “Odyssey Telemaku” (passage “a”) and “On the death of Zhukov” (passage “b”).

Odysseus Telemaco

My Telemac

Trojan war

over. I won’t remember who won.

The Greeks must be: so many dead

only Greeks can throw out ...

And still leading home

the road was too long

as if Poseidon while we are there

lost time, stretched out the space.

I don’t know where I am

what's in front of me. Some kind of dirty island

bushes, buildings, grunts of pigs,

overgrown garden, some queen,

grass and stones ... Dear Telemac,

all islands are alike

when you wander for so long and the brain

already getting lost counting the waves

a clogged eye crying

and watery meat will catch your ears.

I don’t remember how the war ended,

and how old are you now, I don’t remember.

Grow big, my Telemac, grow.

Only the gods know whether we will meet again.

You are now not the same baby

before which I held back the bulls.

When not Palamed, we lived together.

But maybe he is right: without me

you are delivered from the passions of the Oedipus

and your dreams, my Telemac, are sinless.

To the death of Zhukov

I see columns of frozen grandchildren

coffin on a gun carriage, horse croup.

The wind doesn't bring me sounds here

Russian military weeping pipes.

I see the corpse removed in regalia:

the fiery Zhukov leaves for death.

Warrior, before whom many fell

walls, even though the sword was enemy blunts,

the brilliance of the Hannibal maneuver

reminiscent of the Volga steppes.

Ending his days deaf, in disgrace,

like Belisarius or Pompey.

How much he shed the soldier’s blood

into a strange land! Well, grieving?

Did the dying civilian remember them

white bed? Complete failure.

What will he answer by meeting in hellish

areas with them? "I fought."

To the right case of the beetles Zhukov

will no longer attach to battle.

Go to sleep! The history of the Russian page

enough for those in the infantry formation

boldly entered foreign capitals,

but returned in fear to their own.

An analysis of these texts shows that the total number of letters in the passage "a" N _a = 745, and in the passage "b" N _b = 611. The number of letters in the Russian alphabet is n = 33. The number of the i-th letter m _ai in the passage “a” and the j-th letter m _bj in the passage “b” are shown in Table 1, the probability of occurrence of the i-th letter of the segment “a” P _ai , and the i-th letter of the segment “b” »P _bj defined by formulas, respectively (2) and (3), are also presented in table 1.

Table 1 The probabilities of the appearance of various letters in two poems. Letters of the Russian alphabet BUT B AT G D E E F 3 AND Th m _ai 60 fourteen 25 16 25 60 one 8 13 40 eighteen P _ai 0,081 0.019 0,034 0,021 0,034 0,081 0.001 0.011 0.017 0,054 0.024 m _bj 36 10 45 6 12 41 one 12 6 54 fifteen P _bj 0.059 0.016 0,074 0.010 0,020 0,067 0.002 0,020 0.010 0,088 0,025 | P _ai -P _bj | 0,022 0.003 0,040 0.011 0.014 0.014 0.001 0.009 0.007 0,034 0.001

Continuation of table 1 Letters of the Russian alphabet TO L M N ABOUT P R FROM T At F m _ai 33 thirty 34 46 83 fifteen 33 45 54 12 0 P _ai 0,044 0,040 0,046 0,062 0,111 0,020 0,044 0,060 0,072 0.016 0 m _bj 23 35 eighteen 34 61 twenty thirty 33 34 24 one P _bj 0,038 0,057 0,029 0.056 0,100 0,033 0,049 0,054 0.056 0,039 0.002 | P _ai -P _bj | 0.006 0.017 0.017 0.006 0.011 0.013 0.005 0.006 0.016 0,023 0.002

End of table 1 Letters of the Russian alphabet X Ts H W U B S B E YU I AM m _ai 5 four 8 6 one 0 16 13 one 5 17 P _ai 0.007 0.005 0.011 0.008 0.001 0 0,021 0.017 0.001 0.007 0,023 m _bj fourteen 3 7 6 3 0 10 8 0 7 2 P _bj 0,023 0.005 0.011 0.010 0.005 0 0.016 0.013 0 0.011 0.003 | P _ai -P _bj | 0.016 0 0 0.002 0.004 0 0.005 0.004 0.001 0.004 0,020

The bottom line of table 1 contains the magnitude of the difference modulus | P _ai -P _bj |.

The sum of these values is 0.334, the average value of the difference ΔP _cf between the probability of occurrence of the ith letter of passage "a" P _ai and the probability of occurrence of the i-letter of passage "b" P _bj is estimated by the formula (1):

$$\Delta P_{\mathrm{from}R}=\frac{\mathrm{one}}{33}\cdot 0.334=0.01$$

If we accept the acceptable value of this probability ΔP _d = 0.02, then we can conclude that the passages “a” and “b” belong to the same author.

The technical result is achieved by the fact that the device for the meaningful analysis of textual information contains the first and second groups of input registers consisting of n elements, the first to fourth input registers, the first and second groups of division blocks, consisting of n elements, a group of subtraction blocks modulo, consisting of n elements, an accumulative adder, a division unit, a comparison unit, an indication unit, a clock generator and a pulse distributor (RI), the clock input of which is connected to the output of the clock generator pulses, the first output of RI - with the recording inputs of the first and second groups of input registers, as well as with the recording inputs of the first, second, third and fourth input registers, the second output - with the reading inputs of the first and second groups of input registers, as well as the first and second input registers, the third and fourth outputs - with read inputs of the third and fourth input registers, respectively, information inputs from the first to the n-th elements of the first group of input registers are the input of the job of the initial information, to which the m _ai , characterizing the number of the i-th letter in the passage "a", the information inputs from the first to the n-th elements of the second group of input registers are the input of the job initial information, which receives the values m _bj , characterizing the number of the j-th letter in the passage "B", the information inputs from the first to the fourth input registers are inputs of the initial information, which receive respectively the value Ν _a , characterizing the total number of letters in the passage "a", the value Ν _b , characterizing the total number of letters in the passage e "b", the value of n, characterizing the number of letters in the alphabet, the value of ΔP _d , characterizing the value of the permissible value of the average difference between the probability of occurrence of the i-letter of the passage "a" and the probability of occurrence of the j-letter in the passage of "b", outputs, from the first to the nth, the elements of the first and second groups of input registers are connected to the inputs of the dividend of each corresponding element, respectively, of the first and second groups of division blocks, the inputs of the divider of which are connected to the outputs of the first and second input registers, respectively, and the outputs, respectively As regards the outputs of the reducible and the inputs of the subtracted group of subtraction blocks modulo, the outputs of which are connected to the inputs from the first to the nth accumulative adder, the output of which is connected to the input of the divisible division unit, the input of the divider of which is connected to the output of the third input register, and the output is with the information input of the comparison unit, the threshold input of which is connected to the output of the fourth input register, and the output to the input of the display unit.

In FIG. 1 is a functional diagram of a device for meaningful analysis of textual information (to eliminate the cumbersomeness of communication between RI and the control inputs of the respective blocks, they are not shown completely, but are indicated by numbering the inputs and outputs); in FIG. 2 shows a sequence diagram of the operation of the claimed device (the numbers of inputs of the RI are indicated on the ordinate axis, and the number of ticks on the abscissa axis), and the duration of various computational operations (addition and subtraction is one clock cycle, division is twelve cycles) in the upper part of FIG. 2.

A device for the meaningful analysis of textual information (Fig. 1) contains the first 1 and second 2 groups of input registers, each of which consists of n elements, the first 3, second 4, third 5 and fourth 6 input registers, the first 7 and second 8 groups of blocks divisions, each of which consists of n elements, a group of 9 subtraction blocks modulo consisting of n elements, an accumulative adder 10, a division unit 11, a comparison unit 12, an indication unit 13, a clock pulse generator 14 and a pulse distributor 15.

A device for a meaningful analysis of text information works as follows. In the data inputs of the first through n-th elements of the first group 1 of the input registers (FIG. 1) is saved into respectively magnitude m _a1, ... m _ai, ... m _AN, while the data inputs of the first through n-th elements of the second group of two input registers respectively, the values of m _b1 , ... m _bi , ... m _bn .

The information inputs of the first 3, second 4, third 5 and fourth 6 input registers are sent, respectively, the values of N _a , N _b , n and ΔP _d . Moreover, the control signal to the recording inputs of all elements of these groups of input registers and input registers is supplied from the first output of RI 15, the pace of which is set by the generator 14 clock pulses.

, подается на вход делимого блока 11 деления. На вход делителя этого блока по сигналу с третьего выхода РИ 15 направляется с выхода третьего входного регистра 5 величина n. С выхода блока 11 деления значение ΔP_ср, определяемое по формуле (1), поступает на информационный вход блока 12 сравнения, на пороговый вход которого по сигналу с четвертого выхода РИ 15 засылается с выхода четвертого входного регистра 6 величина ΔP_д.According to the signal from the second output of RI 15 to the read inputs of the first 1 and second 2 groups of input registers, the quantities m _ai and m _bj from their outputs are sent to the inputs of the dividend of the first 7 and second 8 groups of division blocks, respectively. The values ​​of N _a and N _{b are} sent to the inputs of the divider of these groups according to the signal from the second output of RI 15 from the outputs of the first 3 and second 4 input registers, respectively. From the outputs of the first 7 and second 8 groups of dividing blocks, the quantities P _ai and P _bj , determined by formulas (2) and (3), are respectively supplied to the inputs of the decremented and the inputs of the subtracted group of 9 subtraction blocks modulo. From the outputs of this group, the quantities | P _ai -P _bj | _{k are} sent to the inputs of the accumulative adder 10, from the output of which the value $$\frac{\mathrm{one}}{n}{\displaystyle \sum _{k\equiv \mathrm{one}}^n{\left|P_{\mathrm{but}i}\text{A.}-P_{bj}\right|}_k}$$

is fed to the input of the divisible division block 11. The value of n is sent to the input of the divider of this block by the signal from the third output of RI 15 from the output of the third input register 5. From the output of the division value ΔP _Wed defined by formula (1) of the block 11 is supplied to the data input of the comparator 12, a threshold which the signal output from the fourth input RI 15 6 to send a fourth input register output value ΔP _d.

If ΔP _av ≤ΔP _d (excerpts “a” and “b” belong to the same author), a signal will appear at the output of the comparison unit 12, which will cause the induction unit 13 to light up. Otherwise, when ΔP _sr > ΔP _{d there is no} signal at the output of the comparison unit 12 and the induction unit 13 does not light up, this will indicate that the passages “a” and “b” belong to different authors. The operating order of the device blocks is shown in the cyclogram of its operation (Fig. 2).

Thus, the technical result is achieved not at the expense of the mathematical apparatus, but at the expense of the technical means (blocks and elements) mentioned in the process of describing the operation of the device, which increases the level of reliability of determining the authorship of text information and expands the arsenal of technical means.

The industrial applicability of the invention is justified by the fact that it can be used in different areas (industries) in the calculations associated with the identification of textual information in cases of determining its author with the aim of making objective decisions in protecting copyright of the creators of textual information.

Claims (1)

A device for the meaningful analysis of text information containing the first and second groups of input registers consisting of n elements, the first to fourth input registers, the first and second groups of division blocks, consisting of n elements, a group of subtraction blocks modulo consisting of n elements, accumulative adder, division unit, comparison unit, display unit, clock generator and pulse distributor (RI), the clock input of which is connected to the output of the clock generator, the first output of the RI - with recording inputs the first and second groups of input registers, as well as with the recording inputs of the first, second, third and fourth input registers, the second output - with the reading inputs of the first and second groups of input registers, as well as the first and second input registers, the third and fourth outputs - with inputs reading the third and fourth input registers, the data inputs of the first through n-th elements of the first group of input registers are input reference source information, which are supplied with values _ai m characterizing quantity b i-th kvy in the passage "a" data inputs of the first through n-th elements of the second group of input registers are input reference source information, which are supplied with the values of m _bj that characterize the quantity of j-th letter in the passage "b", information inputs from first to fourth input registers are reference inputs initial information, which are supplied respectively Ν _a value representative of the total number of letters "a" passage Ν _b value representative of the total number of letters in the passage "b», n value representative Included quantity GUT letters in the alphabet, the value ΔP _∂, which characterizes the permissible value of the average difference between the probability of appearance of i-th letter "a" passage, and the probability of occurrence j-th letter in the passage "b", the outputs of the first through n-th, elements of the first and the second group of input registers are connected to the inputs of the dividend of each corresponding element, respectively, of the first and second groups of division blocks, the inputs of the divider of which are connected to the outputs of the first and second input registers, respectively, and the outputs, respectively, to the outputs of the decremented and input I will give a subtractable group of subtraction blocks modulo, the outputs of which are connected to the inputs, from the first to the nth, of the accumulative adder, the output of which is connected to the input of the divisible division unit, the input of which divider is connected to the output of the third input register, and the output to the information input of the block comparison, the threshold input of which is connected to the output of the fourth input register, and the output to the input of the display unit.

Images