Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
  • G06F16/33—Querying
  • G06F16/335—Filtering based on additional data, e.g. user or group profiles
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
  • G06F16/34—Browsing; Visualisation therefor
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/90—Details of database functions independent of the retrieved data types
  • G06F16/93—Document management systems
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F40/00—Handling natural language data
  • G06F40/20—Natural language analysis
  • G06F40/268—Morphological analysis
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
  • G06F2218/12—Classification; Matching
  • G06F2218/14—Classification; Matching by matching peak patterns

Definitions

• the field of the invention is telecommunications and, in particular, the analysis of digital document flows.
• the invention also applies to the analysis of large masses of digital documents.
• These digital documents may be e-mails, short GSM messages, messages, articles or comments posted on Internet sites, blogs, forums or social networks, instant messages and any other type of message or digital document posted or published. , in the form of text or which has a text or which can be analyzed by a device generating a text such a voice recognition device.
• These digital documents can be addressed specifically or implicitly to recipients or made public for a community or for everyone.
• These digital documents are associated with one or more dates5 of publication, sending or modification.
• the present invention relates to a method and a device for identifying and labeling abnormal or exceptional peaks, increases or variations in the throughput of a digital document stream from one or more social networks or a collection of blogs or websites, to alert an operator or produce a synthetic and explanatory view of the evolution of the flow.
• the general problem is to produce a synthetic and explanatory view of the evolution of the flow rate of a flow of digital documents, or to alert an operator by indicating the main subject or subjects of abnormalities or abnormal or exceptional variations thereof. debit.
• SUBSTITUTE SHEET (RULE 26) the flow at a given moment with the average flow over a longer period.
• More advanced methods rely on transformations, for example on discrete wavelet transforms, as described by A. Haar in the article "Zur Théorie der orthogonalen Funkti onensysteme” published in Mathe- matician Annalen 69 en 1910, no. 3, pages 331-371; there is abundant literature on the detection of peaks from such transformations, such as international patent application WO 2010/007486, or anomalies, such as CT communication Huang et al. entitled “Wavelet-based Real Time Detection of Network Traffic Anomalies" in Securecomm and Workshops published in 2006 by IEEE.
• This technical equipment generates huge amounts of digital data, ranging from tens of thousands to millions of messages or articles per day.
• the known means make it possible to construct an evolution curve of the information rate over time for the whole of a selected stream or for a selection corresponding to predetermined specifications (for example the flow of documents which contain such a word or combination of terms).
• predetermined specifications for example the flow of documents which contain such a word or combination of terms.
• the technical problem that arises is that of the real-time processing capacity of large volumes of data to carry out analyzes that make it possible to explain changes in volume or flow rate indicative of external events.
• the method according to the invention overcomes the disadvantages of traditional methods.
• the invention proposes a technical method, executed by a computer, comprising a succession of processing steps:
• this step identifies the changes in regime in the flow. This step involves determining one or more time intervals and ordering recording the digital documents corresponding to these intervals for subsequent processing;
• the second step concerns a technical processing consisting in extracting sequences of characters from the documents thus isolated, by cutting the texts into strings of characters and recording in another memory zone the strings of characters thus identified;
• the third step is to create an index of strings extracted from the second step, to associate the relevant documents and a quantitative indicator that measures the importance of a string of characters in these documents relative to the flow and then to determine the most important strings in relation to this quantitative indicator;
• the last step is to provide a label constructed from the documents associated with the strings identified in the third step.
• the invetion comprises, according to a first characteristic:
• the first method operates according to a high pass filter based on wavelets.
• the documents are counted per time unit (hour, day), and the sequence thus determined forms a signal on which a filtering is performed by eliminating the coefficients of the discrete wavelet decomposition which are below a certain threshold in absolute value. .
• the distinguished periods are defined as periods during which the signal recomposed after filtering has a strictly positive value. Compared to the naive and obvious approach for the skilled person to compare the number of documents per unit time compared to the average, this approach has the double advantage of identifying peaks or exceptional increases even when the average flow is high but the recent flow is lower than average, and limit peak periods more precisely than just exceeding the average;
• the first method works by comparing the signal with a periodic or quasi-periodic model.
• a periodic or quasi-periodic model is established a priori, for example as the linear combination of several periodic functions of period of 24 hours or 7 days.
• the model coefficients are obtained by the least squares method from the historical data.
• Distinguished periods are defined as the periods during which the difference between the signal and the model is greater than a certain threshold.
• the second method is a cutting of the digital documents according to the spaces and the punctuation.
• This approach has the advantage of being very simple and easy to implement.
• the cutting thus produced does not correspond to a very precise morphological analysis but is sufficient, in the context of the invention, to obtain labels for each of the peaks, increases or abnormal or exceptional variations in the flow rate;
• the second method is a cutting of digital documents according to a segmentation model based on statistical data, grammatical rules, dictionary or hidden Markov chains.
• a segmentation model based on statistical data, grammatical rules, dictionary or hidden Markov chains.
• This approach has the advantage of being able to extract strings of digital documents written in languages where the words are generally not separated by spaces or punctuations, such as Japanese, Chinese or Thai;
• the second method consists of a first step of identifying the language of the digital document and then a set of methods for separating the specialized words for each of the languages processed.
• the third method works by eliminating strings of characters determined by the second method those which appear in a list of empty words or tool words. This approach has the advantage of avoiding constructing labels from empty words or tool words;
• the third method works by calculating the product "TF-IDF" for the occurrences of the character strings extracted by the second method, then selecting the channel or chains for which this product is the highest;
• the fourth method works by searching the character string composed of a set of morphemes distinguished by the second method and present in the digital documents which maximizes a function defined as the sum of the frequencies of the set of substrings of characters of this chain in all digital documents;
• the process as a whole is implemented in a device which presents the operator with a graph of the flow rate and highlights the main peaks, increases or abnormal or exceptional variations in the flow rate and displays, statically or interactively, labels associated with these abnormal or exceptional peaks, increases or variations;
• the method as a whole is implemented in a device coupled with a parameterizable filtering system which presents to the operator a graph of the flow rate of a subset of the analyzed flow, highlights the main peaks, increases or variations; abnormal or exceptional flow and associates them with labels.
• This device advantageously allows the operator to adjust the filtering to analyze more particularly the flow rate with respect to these peaks, to obtain more information on these peaks or the rest of the curve, and possibly reveal other peaks;
• FIG. 2 represents a device which presents to the operator a graph of the flow rate by highlighting the main peaks, increases or abnormal or exceptional variations in the flow rate and which is coupled to a noti fication system;
• FIG. 1 represents the composition of the various processes and the flow (11) of digital documents through a device according to the invention.
• the digital documents are initially stored in alphanumeric form in a table of a relational database (10).
• Each digital document is stored on a line comprising a column with the text of the document, and a column with the date of publication of the document if it exists, or the date on which the document was retrieved, otherwise.
• the relational database is configured to index the column of the date with an ordered index, for example in the form of a tree of type B-Tree.
• the device (12) When the operator (27) interrogates the idle device (26), the device (12), at first, queries the relational database using the aforementioned index, to count, for each period of time (hour or day), the number of documents stored in the database, on a window chosen by the operator.
• This information makes it possible to draw the flow rate curve of the documents on the terminal (22) and an example of which is represented in FIG. 3.
• This curve can synthesize a very large mass of documents. This curve can be refreshed in real time when new documents are stored in the relational database (10).
• the device (12) implements the method (1) to identify periods of peaks, increases or abnormal or exceptional variations. These peak periods can be highlighted by a marker (31) at the local maximum on the interface of the terminal (22).
• the device (13) queries the relational database by using the aforementioned index to implement the method (2) in order to associate, with each document, a sequence of character strings representing a morpheme or a group of morphemes.
• the documents, associated with these string sequences, and the identified periods are then used by a device (14) implementing the method (3) to determine the most frequent character strings in each period identified with respect to the set of documents.
• This method (3) works by first eliminating words that are part of stop word lists, then for each of the character strings, the device calculates the product called "TF-IDF" and retains the n strings for which this product is the highest, n being a parameter of the process whose value can for example be 5.
• the documents, associated with string sequences representing a morpheme or group of morphemes, as well as the n most frequent character strings for each identified period, are used by a device (15) implementing the method (4) constructing, for each period, an associated tag (30).
• This tag (30) is constructed by looking for the character string that includes one or more of the n strings retained by the device (14), which is included in the documents of the period, which is composed of a set of morphemes distinguished by the device (13), and which maximizes the function defined as the sum of the frequencies of all the substrings of characters in all the documents processed by the device (12).
• FIG. 2 shows the integration of the various methods of the invention into a wider standby device (26).
• a number of streams are published on the Internet (25) and are captured and stored in a relational database (10). These streams are filtered by a device (21) that determines the messages on a given subject.
• the documents are then processed by a device (20) implementing a method according to the invention.
• This device presents to the operator (27) a graph like that shown in FIG. 3 on the terminal (22). This graph shows a number of labels (30) allowing the operator (27) to interpret the abnormal or exceptional peaks and variations of the flow rate.
• This operator (27) can then modify the parameters of the filtering device (21) via a feedback loop (24).
• the device (20) then produces a new curve (34) representing the flow rate of the stream defined by the filter parameters. This new curve has new peaks, increases or abnormal or exceptional variations that the device (20) identifies and for which it produces new labels (30).
• the device (20) is also coupled to a notification system that allows the operator (28) to receive an alert on the terminal (23) when the flow rate of the flow has a peak, an increase or an abnormal variation.
• This alert is associated with a tag (30) that allows the operator (28) to determine the cause of the peak and to decide whether it is necessary to analyze this variation via the terminal (22) or by searching in the digital documents which constitute the stream and which are stored in the database (10).
• FIG. 3 represents a graph as generated by a device according to the invention.
• the signal is represented in the form of a graph with abscissa (32), time, and ordinate, the flow per unit time (33).
• This signal forms a curve (34) with peaks identified by the method (1) and highlighted by a marker at the local maximum (31). These markers are associated with the labels (30).
• the morphemes or groups of morphemes are first extracted from the digital documents, which are stored in a relational database with the associated list of morphemes, before the process ( 1) identifies abnormal or exceptional peaks, increases or variations.
• the device (13) when the volume of documents is too important to obtain a response within a reasonable time for the operator, queries the relational database (10) for recovering only a uniform pseudo-random sample of digital documents.
• this random sample is skewed to favor periods of peaks and of recesses revealed by the device (12). It was found that sampling is justified when the number of digital documents recorded in the relational database (10) and corresponding to the selection of the operator exceeds 10,000. In this case, the sample is 10,000, independently the actual volume of documents saved in the database.
• relational database (10) is replaced by a buffer memory which may contain a certain number of digital documents and covering a sufficient period with respect to the interrogations of the operator.
• the digital documents are multimedia documents
• the method (2) of morphological analysis is composed of a text extraction method by speech recognition or by optical recognition.
• the morphological analysis method (2) is coupled to an automatic translation method.
• the method and the device according to the invention are particularly intended for community monitoring on social networks.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• Databases & Information Systems (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Data Mining & Analysis (AREA)
• Computational Linguistics (AREA)
• Health & Medical Sciences (AREA)
• Artificial Intelligence (AREA)
• Audiology, Speech & Language Pathology (AREA)
• General Health & Medical Sciences (AREA)
• Business, Economics & Management (AREA)
• General Business, Economics & Management (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=45875953

Family Applications (1)

Country Status (4)

Families Citing this family (10)

Family Cites Families (5)

• 2011
  • 2011-03-18 FR FR1100817A patent/FR2972822A1/fr not_active Withdrawn
• 2012
  • 2012-03-16 WO PCT/EP2012/054666 patent/WO2012146440A1/fr active Application Filing
  • 2012-03-16 US US14/005,803 patent/US20150205862A1/en not_active Abandoned
  • 2012-03-16 EP EP12710075.8A patent/EP2686782A1/de not_active Ceased

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events