JP2004500628A - Reverse computer system based on content access instead of address access and optimal implementation thereof - Google Patents

Abstract

A content accessible method and system for operating a computer. The main three parts of the invention are a first method for defining, classifying and indexing the content and a second method for specifying all real numbers so that real numbers can be easily arranged monotonically. And a third fast linear method that sorts by content according to the associated monotonic real numbers. The ergonomics problem is solved by simple dialog boxes and buttons that call for some advanced searching. This adds the possibility of searching by theme. A word, expression or text defines a theme, which is used to obtain the corresponding information. This system uses semantics, the meaning of words. When searching for one theme, of the words associated with the Boolean algebraic OR operation or proximity restriction on the words of the theme, ie the restriction that there must be at least N words in the same paragraph, Select the most corresponding word. Sort the results according to the degree of correspondence to reduce noise. FOCUS is not a full-text engine because it detects groups of words, roots of words, synonyms, and "concepts" (focusers) and stores them all in the repository. Any knowledge is extracted from the input data and stored in the FOCUS repository. Analysis means identifying the data format, decoding the data, detecting the language of the textual information, performing linguistic methods, and storing the results according to the FOCUS input format.

Description

[0001]
This application claims the benefit of provisional patent application Ser. No. 60 / 161,579, filed on Oct. 26, 2000.
[0002]
(Field of the Invention)
The present invention relates to content and context accessible computer systems.
[0003]
(background)
Most modern computer systems are not accessible by content, but rather are generally addressable by pathway, for example, disk drives, folders, and possibly subfolders and filenames. As an example of the address, "C: \ Program_Files \ Office WordProcessor \ Documento 3042_version_5" can be cited. An example of a subject that can be addressed by content is "my letter to Smith of last week" (my letter to Smith last week).
[0004]
To date, computer processors and computer systems have likewise operated on the principle of "giving an address to a computer and returning the content of the address". This is exactly the opposite of the way the human mind clearly does. Such a reverse process can take into account the fundamental problems that humans have when using a computer. That is, people and computers are processing information in reverse. People did not ask others, "Do you remember 7:34 pm on April 14, 1972?" Rather, "we were at this restaurant with a piano bar with spare ribs. Do you remember the time? " Restaurants, piano bars and spare ribs are basic memories, and people are basically not interested in the date of the event.
[0005]
So far, basic efforts have begun on addressing computers with content. However, the full content of computer processing must be considered. Relatively speaking, it is difficult to construct content-based addressing based on principles that evolved from the inverse principle of specific addressing of files and similar features of information arrays.
[0006]
To build a computer system that can be accessed with content in the optimal mode of the application, the content from any source needs to control data obtained, for example, from a network, or store the content programmatically. By optimizing any of the building blocks of the system, the system is fast enough to handle and process the content flow to the system. Making such a system portable and robust means designing the operating system independent as well as the size of the hardware, memory and items. The system accessible by the content itself is an operating system.
[0007]
This is illustrated by the following. No computer user is actually surprised to hear a file name when saving a text file. However, such a question is not asked in a good content addressable system. Rather than searching for a file by a name that no one knows, the file creator may have forgotten that a person even asks a simple question, such as "my letter to Smith of last week."
[0008]
Today, the realization of a system that can be addressed with good content is described, provided that the software package under the current operating system is realized. One can take advantage of current physical storage methods and certain software to access files and the system clock. These are the only system sources needed to implement a content accessible system. This guarantees overall portability.
[0009]
(Summary of the Invention)
The present invention relates to a content-accessible method and system for operating a computer. The three main parts of the invention are: first, a method for defining, classifying and indexing the content; second, specifying all real numbers, including integers, so that they can be easily arranged monotonically. Third, a fast linear method of sorting by content according to an associated monotonic real number, including an integer, to access the content.
[0010]
Computer systems using fully optimized content (FOCUS) are designed as language packages that are simple enough for the average user to handle and fast enough to handle the speed of the network. This system is an indexing and search system that combines simplicity, speed and efficiency. Unlike knowing the addresses where information may be stored, the system determines what information the user wants. Based on what users know.
[0011]
The above and other features and advantages of the present invention will become more apparent from the following detailed description.
[0012]
(Detailed description of preferred embodiments)
The following description is the presently best mode of carrying out the invention. This description is not meant to be limiting, but rather to illustrate the general principles of the invention. The scope of the invention should be determined with reference to the claims.
[0013]
The present invention relates to content-accessible methods and systems for operating a computer. Functionally, the main working part of the present invention is shown in FIG. 1, which is a first method for defining, classifying and indexing content (focuser 101) and a second, monotonic method. Includes a method (coding 102) for specifying all real numbers, including integers, for easy placement, and a third, fast linear method 103, which sorts by content according to the associated monotonic real numbers, including integers. The system provides twice the needs of the user, ie, (1) the ability to freely write requests as desired by the user, ease of use (question 104), and (2) appropriate and quick information 105. This system is designed to meet the gains. The information accessible by the content exists in the repository 106 functionally. These last two points define what can be considered an ideal information retrieval system.
[0014]
Point 1 is mainly about ergonomics. Simple dialog boxes and buttons to query some advanced searches solve this problem. On the other hand, the possibility of searching by theme is added. One theme is defined by words, notations or text and is used to get the corresponding information. Point 2 is achieved by semantics, a system that utilizes the meaning of words. Traditional systems that attempt to exploit semantics have ended with heavy computational overhead. Although the processes described below fall into the field of semantics, these processes are far from those currently used in this area, which have not been proven to be effective.
[0015]
The ergonomics of the FOCUS system are simple. That is, there are only (A) a text field, (B) a radio button theme, (C) a code search button, and (D) a search button based on meaning. Text fields are different from mere sequences of words (eg, using a question to define a group of words), and can enforce groups of words by using conventions. Code search deals with Boolean logic applied to search components. The search by meaning is performed through a file or a list of files corresponding to a given theme. These files may be lists of words. A theme is a list of simple words that the user has selected as the user's expression of a given concept, along with the frequency in a given text. This list of simple words can be sorted by correspondence using the frequency deviation from the reference.
[0016]
When searching for a theme, perform a Boolean algebraic OR operation on the words in the theme or select the maximum correspondence of these words in relation to proximity restrictions. There must be at least N words in the same paragraph. Here, N is at least 2, usually 3. These results are then sorted by correspondence to reduce noise. A paragraph is something that is separated by what is recognized as a paragraph separator (this code exists in Unicode, but needs to be analyzed against simple ASCII text). To be meaningful, a paragraph must have a certain minimum length.
[0017]
The theme radio button indicates that the word or expression to be searched should be considered as a theme. Since these themes are pre-calculated, they enable very fast semantic search. As a result, it can be displayed in three forms. The first form is to use this title when the title of the document can be identified. The second form is to use some kind of automatic abstract (the first sentence or first expression in the most corresponding paragraph in the file, or the first most deviant expression). The third form is a form that displays a text event. In this case, the view program highlights the corresponding paragraph and word, so that it is easier to navigate through the hits.
[0018]
It should be noted that the reference itself should be recorded as content. These contents are part of the computer system item that conveys meaning. For example, for each filename, one records the complete path, along with all of its components: drive ID, directory name, directory extension, file name, file extension and words inside them.
[0019]
Next, you must determine what the information is. Information can be described as having twice the nature. One property is the set of data that is being sought, and is also the manner in which that data set is sought. Within a text, a word can be some information so that the word can be used to find the word. However, one text is also a theme and a concept. Therefore, themes and concepts are extracted from the text. In languages such as German, where most words are compound words, even part of one word is information. The word "being" can be either a noun or a verb or information.
[0020]
Use the following steps to retrieve information. (A) Extract or calculate information from files stored in the system. This information is searchable and depends on the current technology level of the concept extraction software. (B) This information is stored so that it can be searched instantaneously. (C) Use a storage process having performance independent of the amount of data to be handled. (D) A search system is provided to access all stored elements. (E) Perform all of these as close to real time as possible.
[0021]
This search means building a repository that directly accesses all information content. Being linear in nature requires that all processes be linear for any set of data available at one time from the first step, ie, from a repository free sorting routine. Being independent of the amount of information for monitoring not only does not require the use of multiple repository files, but also avoids copying of the repository for updates. Real-time processing means handling operations as they occur, such as updating, renaming, moving and deleting, without interruption, ie without breaking on the timeline.
[0022]
Computer System Using Fully Optimized Content (FOCUS) Intelligent filtering as part of a FOCUS system does not merely provide a list of wanted or not wanted conditions. However, this method is quite different, although usually done with heavy computationally intensive techniques that require several minutes to determine the fate of a sentence. Semantic configurations only require that the user present a sample of text that is meant to indicate the desired theme to define.
[0023]
FOCUS feeding can be performed in various ways. For example, receiving data from a network can trigger analysis of that content. Updating a record in the database can trigger an analysis in the same way. Updating the word processor file can also trigger the analysis.
[0024]
Focuser concept
Focusers are defined as a set of the following elements:
(1) its name, (2) a list of words or expressions that indicate the focuser, (3) an optional but optional language description (French, English, ...), (4) a paragraph in the paragraph. A list of some parameters or words or expressions that govern existence. That is, (a) a small number of words or expressions in a paragraph indicates that the paragraph does not correspond to a focuser. A low number of them in a paragraph indicates that the paragraph does not correspond to a focuser. This number is used to detect the focuser. (B) The number of words or expressions in a paragraph that is less than if the paragraph corresponds, and the number of words or expressions that is greater than if the paragraph actually corresponds to a focuser. (C) At a threshold with a degree of correspondence lower than this number, the word does not belong to the focuser. This threshold is used to build the file.
[0025]
Building a focuser from text
Using known general frequencies of words in a given language, it is possible to automatically construct a focuser from text. Using this information, the degree of correspondence, which is a number indicating the uniqueness of the word in the paragraph, can be calculated. This number is the ratio of the frequency of the words in the text / the general frequency. You can specify all words from the source text by reducing the correspondence value. The word with the largest value corresponds most to this paragraph.
[0026]
It is also possible to increase the focuser manually by creating the following specific information: That is, the specific information is (A) a very corresponding specific expression, (B) a synonym (which can be automatically proposed by the system), (C) a word or expression to be excluded from the focuser (forced zero corresponding value), ( D) Words or expressions that distinguish the focuser, i.e., words that are automatically excluded from the focuser (negative corresponding values), and (E) words that are accepted excluding all expressions including the word. The forcer is then indicated by the part of the word that has the maximum value. The threshold can be selected by the program or by the user.
A very simple example of a text for Focuser is as follows:
[0027]
This is about horses. house, horseback
cavalry, @mounted_tropes. house.
[0028]
Here, “$ mounted_tropes” is a very appropriate expression. The word list excluding unsupported words,
That is, it looks like Cavarry, horse, houses, houseback, and pmounted_tropes.
[0029]
Focuser detection
Focusers do not mean anything by themselves, but take all the meanings when compared to text with paragraphs. Thus, the focuser is said to have a certain number of words or expressions relating to this focuser recognized in the same paragraph, this number being a parameter of the number of the focuser, whose value is usually about 3. The expression manually entered in the focuser will be three times the value of one word in the focuser.
[0030]
Automatic routing / filtering using text content
The routing defines the routing / filtering. Some profiles are a pair of positive and negative focusers. Whenever an email or HTML text or any text passes through the filter, all of the defined focusers are detected (see paragraph Focuser detection) and compared to the profile. If a negative focuser is recognized, the text is rejected, and if there is at least one positive focuser, the text is captured. Any combination of positive and negative focusers can be used.
[0031]
Semantic analysis methods evaluate every paragraph according to a given focuser. Instead of recording that rating along with the data, if a particular theme is associated with a particular destination, that rating can be used to route (or stop) entry information.
On the other hand, all of these can be maintained in the central repository, with each user performing an implicit Boolean algebra AND operation with their theme. This method allows some network managers to monitor what is being routed (or stopped) and where is being routed (stopped).
[0032]
If you want the filtering application to be efficient when it reaches the firewall, you can force it to monitor data that escapes the firewall. The firewall controller lets you pass information so you can check what has been stopped but what has been stopped, what has been stopped so far, and what has not been previously determined. You can tune your firewall as you try to defend it from the people you want. Of course, if a particular provider has sufficient evidence that it sends only undesired data, the controller can also declare that his or her ID (such as IP) will be rejected, Processed by FOCUS, but speeds up the process a lot.
[0033]
Automatically build focusers using text in the repository
Concepts associated with any word of text in the repository can be automatically constructed. The algorithm is as follows. Obtain all paragraphs containing this word, and then extract the concept from the sequence of this concept. Doing this for every word can be extremely time consuming, so that we can limit this extraction to, for example, expressions containing at least two significant words. Thus, the user can seek this predetermined expression, or this expression as a concept.
[0034]
Automaton Semantic Network, Thesaurus, Lexicon
Any kind of information linked to a string in memory can be displayed as follows: Ie
<Word><separator><type of information><separator><information>
If the information is small enough, it can be maintained directly in the automaton; otherwise, the information stored in the automaton will simply be the address to the information stored in the file, for example.
<Separator> and <type of information> must be small (usually one character) in order to be as compact as possible. Thus, semantic networks, thesauri, lexicons, etc. are displayed this way, provided they can fit in memory.
[0035]
Realization of semantic analysis / filtering in silicon
The basic routine of event manipulation can be implemented in silicon either on a general purpose processor based controller or on a dedicated RISC chip.
[0036]
Automatic viewpoint of text using deviation
The most relevant words or expressions associated with the focuser can be given without text. This focuser is called a viewpoint on the text. In order for the viewpoint to simply show the most deviated word or expression in both the text and the focuser, the expression need not be externally in the focuser. The reason for this is that the deviation of the expression is a compound deviation of the words in the expression. A compound deviation is the sum of the deviations of a single word contained in the expression divided by a number. If a particular viewpoint is not shown, the viewpoint is simply a focuser built from itself when appropriate.
[0037]
Automatic language and code page recognition
By comparing all terms in the sentence with all dictionaries of the planet coded in the possible character code pages, recognition of a particular language can be performed. It is easy to see that this method is overly heavy. A faster and more accurate solution is to characterize the language with a statistical distribution of n-applets. Experience has shown that obtaining a statistical distribution of four applets allows a fairly good discrimination of language and code page combinations. This is performed very straight forward using a sorting routine. Elements to be sorted start with an n-applet in the first byte, start with a 1 in the second byte, and so on. Next, the double is counted, and the result is compared with the database of the n-applet constructed in advance. This database is built by applying the same process to text that is large enough to be displayed. One megabyte of sample text is considered sufficient to build the database. This database consists of n-applets and their corresponding frequencies for each language / code page combination.
Comparing the text with the database, for each pair of language / code pages, adding the frequency of each occurrence of n-applet occurring in the text and selecting the highest number indicates the language / code page.
[0038]
This method is useful for recognizing the language and character code page for a given paragraph. Assuming a particular character code page, there is another more accurate way (in other words, it must be able to recognize words of text). This method can recognize languages in a single sentence based on using a small dictionary of frequent words for each language to be recognized. Each word in this sub-dictionary is associated with one or several languages. Next, when a word related to this language is recognized, the degree of correspondence of each language is incremented. Next, the highest language correspondence is selected.
[0039]
Computer accessible by content
A computer accessible with content considers any content as having three parameters: (1) a reference, (2) its own content, and (3) a location. A reference is a string of bytes that tells the host software how to reach the content. This reference can be a file name, an address in memory, a field in a database record, a file in an archive, and so on. The reference is preceded by a code indicating what software is required for the content, i.e., a filing system for file names, an archive program for archived references, a database management system for records, etc. The rest of this string is an argument sent to the host, for example an SQL request for the database manager. The coding used depends on the host.
[0040]
Next, consider the content. Every content in the computer is a string of bytes and the length of this string. Generally, these bytes are 8 bits to 1 byte. The third feature is location. This location is not a physical location in the computer's memory or in the computer's storage area. This position indicates something meaningful to the user. In a text file, for example, this position can be a chapter number, a section number, a paragraph number, and so on. In a database, this location can be a table number, a record number, a field number, a paragraph number, a sentence number, etc. In the image of the medical scanner, this position is in XYZ coordinates and is red-green-blue RGB pseudo color content. The position is mainly recorded to perform proximity management.
[0041]
Semantics
List
This technique provides a way to identify the content of a document before opening the document. This technique can be relatively comparable for large documents accessed over a network. The listing technique uses the idea of a document abstract, but automatically creates an abstract from the content of the entire document. When the FOCUS linguistics extracts a group of words, selecting the most distinctive group of documents by comparing the characteristics of the document with a statistical analysis of a given body of data. Relatively easy.
[0042]
For example, text used as a reference for plain text can be considered as normal English, French, etc., taken from a large plain text repository from FOCUS. When analyzing a database, field names are often very different from commonly spoken languages and are likely to be systematically selected for listing. It is easy to analyze all of the database and then select it with the corresponding focus and build a reference list of current words and expressions. A focuser is a text file showing a list of terms, synonyms and expressions of how a focuser owner is speaking when talking to another person about a given means. For a set of text that is deemed to correspond to a given region, this region can be described in the focuser, and the results can be used as described above to indicate a more accurate reference to this region.
[0043]
As long as FOCUS is fully implemented, these redefinitions and analysis can be performed locally on the FOCUS repository without accessing the document itself. Therefore, these techniques are also suitable for network environments. Once a document name is selected or displayed, it is easy to display the relevant listing recorded in FOCUS. By browsing these lists, users can quickly identify whether a document is of interest to their current question. The expression of the degree of the list can be used in the subsequent questions.
[0044]
Dynamic Focuser
A dynamic focuser is a focuser that is not compiled and created during filtering, but is created and analyzed in real time with one or several focus repositories. As noted above, a focuser is a text file that provides a list of terms, synonyms, and expressions of how a focuser owner is speaking when talking to another person for a given means. Focuser calculations are not limited to the time to analyze the document. In practice, this calculation can be done at any time, even when the user is asking a question. The format can be any. The user may type a series of words and / or expressions from any document, especially those documents discovered by another question in their FOCUS, may type actual sentences, You may capture paragraphs.
[0045]
The first task of the system is to ask all the words in this text. Next, sort these words according to the document and the location where the document resides. If the number of paragraphs is stored in the FOCUS memory, this feature is of great use; otherwise, the technique reverts to what is known as clustering. This clustering is linguistically insane. Making the whole paragraph available as a REAL question with an acceptable response time is a breakthrough in NATURAL LANGUAGE calculations. In addition, these questions can be mixed (Boolean) with regular questions in text, images, music, etc.
[0046]
Using a linear sorting routine, this operation is fast enough to be performed interactively.
Next, the choice of the winning paragraph is calculated according to the accuracy parameter (the number of words that should occur in approximately a given paragraph). This parameter can be monitored with a slider or "+" and "-" buttons, and the results can be quickly recalculated until the appropriate number of documents are selected. Next, the result of this inquiry is sorted by the degree of correspondence.
[0047]
If the user is satisfied with the "dynamic focuser" and he wants to use it again, FOCUS can record this definition "focuser" and its result (selected paragraph). This operation does not require access to the original document and may be appropriate for a network environment (eg, a web portal).
[0048]
The same technique can be used to update the focuser (if the focuser is dynamic or recorded in FOCUS). The only extra step is to eliminate the occurrence of the previous definition of the focuser, and this step does not require access to the original document.
The fact that the documents do not need to be referenced again merely stems from the fact that FOCUS has better knowledge of these documents than the documents themselves. In fact, FOCUS typically stores 15 times more information than the number of words in a given document. All of these updates occur in real time.
[0049]
Inverse focuser for e-commerce
In practice, this use of the focuser is not limited to E-commerce alone, which provides a good example. In an e-commerce portal, the vocabulary describing the item to be sold is usually in merchant terms. This makes navigation very difficult for customers who do not necessarily know the term. Classification is often not very useful because, for example, both sports or leisure can be considered the same item. The FOCUS solution to this dilemma is quite simple. First, start the portal with any descriptions available, and use these descriptions as focusers. The customer types a word, requests FOCUS for all focusers containing this word, and displays the answer as if the focuser was a category. If the user does not find the category he is thinking about, he rephrases (or stops) the query and the game starts over.
[0050]
If all of the user's queries are simply logged into any proprietary format, this log provides the E-commerce manager with two types of information of interest. Of course, the manager gets his customer's profile (and if the focuser is sufficiently consistent, he can create a focuser from the customer) but talks about what a person wants for each item being sold Also get the profile of These profiles can be used to update the focuser that describes the item, and FOCUS quickly learns how to answer customer inquiries with its own words (avoid quitting words). This process shows how standard FOCUS can be applied.
[0051]
Using lists to find similar documents
Requests for representations taken in a given list of documents can quickly provide similar documents (text, images, music, etc.).
There are two ways to do this.
One way is to use the token and make a new request for the list itself; another way is to make a request for each of the representations. The answers must be sorted using any parameters that the user considers appropriate, and the output is ranked by similarity.
[0052]
Automatic copy detection
It is very unlikely that two different documents will have the same list (this would typically result in up to 30 representations being the same in the two documents).
It is very easy to operate a separate item that queries a list and outputs a list belonging to more than one circuit. However, it is also possible to automatically issue a copy warning when storing the document parameters in the FOCUS repository, and this may be a copy if the list already exists. Of course, these techniques only provide copy candidates. On the other hand, even though the files have the same length and the date is not enough to declare these files as copies, this detection is format independent. Text processor documents that copy HTML documents produce the same list. Ultimately, one has to decide what to do, directly or through an application program (by copying.
[0053]
Open-ended, unlimited, variable-length coded numbers
Monotone floating point coding (MFPC)
In variable length coding, the length must be part of the code. Computer data is always a string of bits (or bytes). The first upper bit in the first byte of the string indicates the length of the string in bytes. Encode into the upper bit of the first byte. That is, the position of the first reset bit (0) indicates the length of the string.
[0054]
This special encoding uses all bits to indicate the value. Therefore, this method is optimal. The space required is an absolute minimum, and there is no limit on the number (setting all eight bits after the first byte causes the coding to proceed to the second byte). In addition, the strings are binary monotonic, which allows sorting and instruction testing without decoding. The coding and decoding routines are straight forward. This is independent of the processor representation of the number (large or small endian). If the computer agrees with a 16-bit binary representation, this encoding can be based on another unit, such as a 16-bit word. If the numeric fields in the database match the MFPC coding, all fields can be sorted as binary values without performing a dynamic match on the numeric fields.
[0055]
For numerical information, ie, real numbers including integers, FOCUS places no restrictions on the number of digits in the mantissa or the value of the exponent. As can be seen from FIG. 2A, the real number can be divided according to the codes of the mantissa 201 and the exponent 202. The overall value depends on this classification. FIG. 2B shows another classification. In this classification, for example, when + infinity 204 or −infinity 205 is separated, +1 206 and −1 207 and 0208 are also separated. These aren't really necessary, though they aren't really necessary. In FIG. 2A, a mantissa 201 in the first row with a + code 209 and an exponent 202 with a + code 210 can be seen, which are coded as 211 in D in this system. In the next case, the mantissa 201 is +212, the exponent -213 and the code are C214. C214 is smaller than D211 or D211 is inherently larger than C214. In the next case, the mantissa 201 is -216 and the exponent 202 is +217, which is coded as B218. B218 is smaller than C214 but larger than code A219 which has a mantissa 201 of -220 and an exponent of -221.
[0056]
As can be seen from FIG. 2B, not only the fourth line with mantissa + 212, exponent-213 and code F251, but also the eighth line with mantissa-216, exponent + 217 and code B218, the absolute value of the exponent is It should be understood that the higher the value, the lower the value of the number. If in the first case it is positive for the negative series and is a value, it is one more than a certain value and very large, and before the last line the mantissa is negative, ie a negative number, Since it is very positive, the number will be negative or some value for a very large series, so it will be a very large negative number. Therefore, in the regions F251 and B218, the larger the absolute value of the exponent, the smaller the value of the number. Thus, one may want this complement for a given base. For example, 12 in the base 10 is 99-12 = 87. Next, the absolute value of the mantissa is a predetermined coded value or a value for a predetermined base. The various bases that encode these components do not actually correspond, as long as these bases are coded to be the same for all coded elements of the number. (FIG. 2B) A265, B218, C266, D267, E264, F251, G264, H262, I261, unlimited monotonic coded exponent or unlimited real number to be a code combination that is a string of characters to the complement and mantissa Can be coded.
[0057]
Any real number can be described as a number of any length using the first unlimited integer of minimum length for the exponent and the mantissa. One way of multiplexing the numbers is to use the numbers described in base N, and since these numbers are described as base N + 1, there will not be N + 1 numbers in the numbers. When coding in base 2, it has multiple zeros and ones. When coding in base 3, the number is N + 1, where N = 2 has 1001, and has 2 and some other number 1001, etc., so larger numbers serve exactly as separators. Since the smaller the number N is, the less space is used, a number for N, ie 2 or 3, is preferred. Since 2 is preferable for the encoding speed and 3 is preferable for the decoding speed, the number actually used is determined by throwing coins.
[0058]
Linear sorting
In the case of linear sorting, basically one list must be sorted in a certain order. This classification can be performed at the binary level using the micro values of the selected components that you want to sort. For a byte, it has 8 bits or a power of 2-8, ie, a value of 256. When a 16-bit word is used, a value of 64K or the like is used. When sorting a list of bytes along with a byte, it carries an integer pointing to the next byte, with the available values carrying an array of start and end pointers. The byte pointer is initialized in a predetermined manner, typically in the order of a first item, a second item, a third item, and so on. When reading a byte of value X, update the X start and end pointers. If both pointers have not been used yet, the value will be a pointer to a byte; if all pointers already have a value, the end pointer will point to the byte, while the byte pointer for the previous end value will point to a new byte. Point. At the end of the scan, the storage of any value in the byte pointer is reported in the order of the match for the particular rank.
[0059]
FIG. 3 shows the status of the array of start pointers 301, the array of end pointers 301, and the array of pointers for the next pointer 303. In FIG. 3, the first three ranks, ie, the characters XYZ 304, have already been sorted and determined to be the same, and XYZ. . . . , The next character in the series, A2305, is being tested. The rank byte routine process is in the list. If there is more than one item in the rank list, only the next rank is processed. This is known when using any kind of flagging to terminate an input string. The next rank need not be R + 1, but can be in any desired rank.
[0060]
MFPC-based date / period A date or period can be expressed as a number of years, months, days, hours, minutes, seconds, and fractions of a second, or any combination thereof. For all these values, any length of time, whether large or small, can be represented by using MFPC. For dates, all but a few fractions of a year and seconds have restricted values and can be represented as simple binary integers. The search for the month of May does not end with token research, since the repository can come up with recording the complete date or period with the components here.
[0061]
Repository-logical structure
The best solution for the fastest access to any individual item uses an L-string. One L-string is an optional horizontal string and a vertical string of at least one byte, each byte of the vertical string being associated with a pointer to the next L-string. For example, to store “Albert”, “Bogart” and “Bowie”, the L-shaped string would be as follows (see FIG. 4): The “AB” 401 of the first string is purely vertical, and the “LBERT” 402 is clearly horizontal. In fact, "LBER" 403 is a horizontal string and "T" 404 is a vertical string with that pointer. The next string has “OGW” 405 as a full L-string. A "0" 406 is used to indicate that the string no longer points (i.e., the end of the string mark).
[0062]
Text information
FOCUS uses Unicode standard codification to record text information uniformly. This is an open-ended 2-byte encoding scheme that covers all glyphs used in the planet. An L-shaped string does not copy a given character. More than one representative can be considered to handle bad accent words or inconsistently written character words. For example, the number 12 can be written as a sequence of characters "twelve" or a sequence "12" or a number "12". There is nothing in FOCUS that prevents using these three displays at once. The same can be done for compound words like weakend, for example. This word can all be stored at the same address as "weekend", "week-end", "week" and "end".
[0063]
Automatic partitioning of repository when disk access becomes critical
When disk access becomes a bottleneck, it typically creates duplicates of the repository, guides heavy synchronization methods, and generally loses the potential to maintain real-time applications. Due to the algorithm of the sorting routine used in FOCUS, it is almost natural to split the repository instead of duplicating the repository.
It is easy to tell the sorting algorithm to send the start of the sort to one file and to switch to another file.
[0064]
The problem in this case is how to split the repository in the first place. One application used with FOCUS is to sequentially read out a repository. This program is used to build a new repository on the second disk by a point which is half the first. Then proceed to read, but write the second half again to another file. Delete the original file. This preferably keeps the original file as the second half and erases the data stored in the first half. The reason is that if another split is required, this would result in an unnecessary empty file. Store the repository number and name in the base repository (starting with the lowest string) as a special code (lowest code) that maintains overall control within FOCUS.
[0065]
An advantageous feature of this scheme is that the greater the number of repositories that must be possessed for the number of user accesses, the shorter the time it takes to execute the backup method. During a backup, the watchdog remains active but stops the feeding process so that the copy participates in a known state. Next, copy the repository. The more repositories, the shorter they will be, and the shorter the backup time.
[0066]
Repository (index) management
Increment / Decrement Index Update Copy w / o Index Normally, updates to the repository trigger a periodic update of the repository itself. This means that the available space is twice as large as the required space. The FOCUS routine performs its own garbage collection, thereby eliminating the need for a repository copy to remove slack.
[0067]
Catching index for real-time indexing
Not sorting the input data directly into the final repository allows for faster sorting and faster creation of a rapidly accessible cache repository, thus providing near real-time access to recorded or received information.
[0068]
Using monotonic access methods with index files containing numbers and dates / periods
The fact that different types of data have a monotonic encoding allows the internal engine to have a very limited set of functions. That is, it can have 1) a function of searching for a specific item, 2) a function of searching for a specific item, and 3) a function of searching for the next item. An important feature of FOCUS is that when no item is found, the next item available to the request program is sent, along with a flag stating that the request was unsuccessful. This allows the calling program to make the next call (see the replacement process).
[0069]
Generalized indexing of databases
This is where a reference to a table / record replaces a reference to a file name.
[0070]
Automatic reference monitoring for real-time indexing (according to the process used by a given OS) Builds a calendar log of events by intercepting operating system calls for file opening, file closing, writing to files, etc. Which can then be used to determine which files to delete, move, rename, or update. If a smarter method (OS dependent) cannot be used with the updated file, the update should be viewed as a deletion followed by a creation. Partial updates generally only result from new analyzes (eg, semantics) that generate new parameters and cancel other parameters. If the reference is for a field in the database, it is only necessary to have an automatic logging of edits for this database, ie a logging file containing all necessary parameters. An easy format for these logs is ODBC. More general changes and updates are well considered by their generation. Most processors allow some level of intervention at this level. In the worst case, one has to wait until a change is indicated by one of the processes having an access method (eg waiting for a change to be sent to one file).
[0071]
Non-text extension in indexing engine
"Full text" engines generally cannot record non-text information. There are two ways to do this. That is, there are a method of determining one or more prefixes and a method of determining one or more suffixes. For example, if the text information means that a prefix of “0” should be added, a prefix of “1” is added to the numerical information, a prefix of “2” is added to the date, and a prefix of “3” is added to MIDI. be able to. If the text information is file content, the suffix "0" can be determined. If this text information is a parameter for the filing system, the postfix will be “1”. If it is a file name, the same can be done by appending a "0" to the "1" postfix. The choice of prefixing or postfixing depends on the access used for the search. It is possible to record the copied data completely, one using a prefix and the other using a postfix, both indicating the same item. Here, when the Unicode standard considers two sets of 32 values (0-31 and 128-143) considered from the standard, no prefix or postfix is defined in the text information. From one of these 64 values, the code for the initial prefix and postfix is selected. The value 143 can be multiplexed so as to always allow for expansion in the unlikely type of code.
[0072]
FOCUS sequence
FOCUS feeding can be performed in various ways. For example, receiving data from a network can trigger analysis of that content. The analysis can also be triggered by updating records in the database. Updating the word processor file can also trigger the analysis. Whatever process is used, the desire of all FOCUS is to obtain a list of references to consider in a dedicated file. It can be immediately pointed out that the analysis process can also decide not to record or maintain data, such as for network filtering. However, the data can be flagged as unacceptable and the data stored in some way so that the network manager can trace the unacceptable information.
[0073]
For textual data, the analysis considers linguistics and semantics so that concepts can be identified on the fly. As can be seen from the above, these concepts can be used to route data flows and / or stored along with other information to be analyzed. Even when dealing with text, FOCUS detects groups of words, roots of words, synonyms, and "concepts" (focusers), and FOCUS is not a full text engine in storing all of them in its repository. This is the case when "knowledge management" is in a picture at the time of data reception and not at the time of a user request. Any knowledge that can be known by current technology is extracted at the time of data entry and stored in the FOCUS repository if it is immediately accessible. A reference is provided to the real-time input (network incoming data) along with its output. Perform the analysis when the input is just a notification of something that has occurred.
[0074]
Analysis means identifying the data format, decoding the data format, detecting the language on the text information, performing the linguistic procedure, and storing the results according to the FOCUS input format. Note: Identifying common business data formats is beyond the scope of this document. Many commercial products are available for this purpose. However, extracting embedded structure representations in formats such as HTML, SGML, and XML is part of FOCUS.
[0075]
At the end of these processes, the FOCUS input file is ready for processing. The problem of inserting this information into the repository is related to the sorting of these files. Despite the fact that the mathematicians have demonstrated that the sorting process is linear, no matter what routing is used for sorting, RAM-based techniques are more efficient than disk-based routing. Will also be faster. Thus, if multiple input files are available at the same time, FOCUS will concatenate these files and feed the sorting routing. If one file is available, the file is sorted at a time for immediate access to the data by the FOCUS query program.
[0076]
Since the output of the FOCUS sorting routing already uses the FOCUS repository format, the user can access the latest data very quickly. However, multiplexing these files on a particular physical disk drive means that the performance of the system is degraded at the time of inquiry and update (particularly at the time of deletion). Thus, the final step is to merge these temporary repositories on each physical drive into a single final repository. This process is relatively fast, but can be run in a low priority background (FOCUS manages its background priority). However, in some applications (if all of the repositories carry incoming data for a day and, for example, data should be deleted after a certain date), it is also possible to maintain a temporary repository instead of merging. .
[0077]
Record the reference in the repository and give it an ID (or handling) number (an unlimited integer). Also record the cross reference (from ID number to reference). When deleting a reference, the cross reference is updated as deleted, the entry is further deleted, and the cross reference is updated as free at the end of the deletion process. These free cross references are used by the reference allocation routine. Sometimes a user wants to record another cross-reference of interest, and the user is free to do so. FOCUS has nothing to do with the meaning associated with the data and only the user is free to do so.
[0078]
As used herein, the use of the word "unlimited" means that FOCUS does not impose restrictions on the user or some of the programmers. The application program nevertheless recognizes certain restrictions of itself. For example, FOCUS can store a number pi in the millionth place after the decimal point, but there is little point in doing this because the one seeking the pi is not ready to type a decimal in the millionth place. In addition, since FOCUS does not limit itself, selecting a limit can affect sizing a given buffer or cluster. The fact that the sorting routine can be executed with as little as 1K of RAM to handle terabytes of information means that doing so is unwise. In any case, using 1K for buffering does not allow sorting items larger than 1K. FOCUS is obviously limited by the amount of RAM and disk space available in a given system, however, if done well with the size and disk space of RAM, they will be large.
[0079]
If there is more input data than can fit in memory, the current method is to process the first rank and store all classes that ensure RAM fit for the final process. . This is less than the amount of RAM available proportionally between the RAM that has already been read and the remaining RAM that has not been read, assuming that all subsequent data has the same ratio of that class. Based on the fact that classes should be few. If the class is greater than the above amount, the next rank is used for classification and the process continues. Regardless of which processing is performed, the classes are read out in the order of collation, and the classification is performed completely. This happens only once. If the class is further divided, the first subclass is loaded together with the class before processing. Next, flush all classified data to disk, reconstruct the data in RAM, load the next subclass, and continue with the same work.
[0080]
This process is completely coherent and completely linear. Furthermore, the process is further absolutely optimized when considering bytes that are not necessary, and considers all bytes only once. If the special value (which is the case in FOCUS) does not give the end of the byte string, the length of the string must be used. By keeping track of the first and last values in the rank, it turns out that the speed is twice that of the method. The use of a bit-mac of the value used to do this allows the speed when written in assembly language to be further increased by verification.
[0081]
This technique of "unmerging" uses temporary storage equal to the final storage, and current techniques require extra storage approximately equal to the size of the input data. Rewriting this content to the final file requires the same space, as this temporary data can be deleted as soon as the temporary file is read into RAM. In fact, most operating systems store files in clusters, so temporary files use storage that is slightly larger than the end result, but use storage devices that are larger than the number of temporary files times the cluster size. do not do.
[0082]
The amount of RAM dedicated to the process is not important. Variations of only about 20% have been observed to date. The configuration of the RAM for the sorting routine here is one of the feasible methods. At any given time, there can be up to three lists in RAM. That is, there may be a list of items themselves, a list of files from previous loads, and a list of files stored by the current load. Therefore, the loading of the data into the buffer is performed between the end of the first list and the starting point (pointer) of the data. Loading ends when data size + pointer size overflows available RAM.
[0083]
This third list is created as follows. Consider whether the list is in the first rank and if the byte value is B. Also consider whether the size of the B class is OK to match an already existing file (which can be found by reading the second list during sorting). Next, all entries of the B class are attached to the B file, and the B list is skipped. Here, if the B file is created, the first entry of the B class is given a one-byte length, and this entry points (according to the collation) to the B + 1 class. When all items have been sent to their respective files, their sequence is replaced with a second list. These file names are moved to be attached to the first list, and the pointers are moved before the pointers in the first list. Next, these two lists are combined and sorted.
[0084]
Ascending order, for example, "0000", "0001",. . . . , And their surnames are the already classified characters that these filenames contain. These names can be appended to the surname to determine if the actual data is 4 bytes (in the example) longer than the string to be sorted. When the data is read, the corresponding file name is directly accessed. When skipping doublets, skipping occurs as soon as these doublets are detected (the first occurrence points directly to the next item and bypasses the doublet).
[0085]
Storing an excessive number of doublets is only a problem. These doublets require a dedicated file with the same signature as the doublet plus another file containing some attached strings. To distinguish them, a double name can be stored, and if the name is a normal file, 1 is added. In addition, the doublet can have a storage format (storing "her blue eyes" one million times in the primary file is ridiculous ...). Nevertheless, the doublet must be extended for final output.
[0086]
If RAM space is an issue, the storage of the filename can be done in a separate file. Also, sorting 1000 tera records with just one file name is obviously ridiculous and unlikely. As far as the personal computer history can be seen, the ratio of disk space to RAM was always about 1: 1000.
[0087]
Repository logic structure (as already mentioned)
Physical memory
Swapping data between RAM and disk implies some sort of clustering scheme. By design, FOCUS can accept cluster sizes of 4K bytes or more (the minimum value is based on analysis of 8-bit data and a small repository), which is convenient for the user and can be combined with the longest item (almost 2). Twice the size). For simplicity, the implementation limits the cluster size arbitrarily to a power of two, which is the size that the operating system swapping scheme uses anywhere. Such a clustering scheme performs fragmentation of the repository so that the L-string points either to another L-string in the same cluster or another L-string in another cluster, and this is flagged with a flag bit. Can be displayed.
[0088]
Similarly, when the string has finished seeking another L-string assignment that cannot be done in the same cluster, it initializes a new cluster that only carries the end of the string. Ends with a repository of. The new cluster does not complete with any other strings. By way of example, completing "n" with words like "mankind", "manhood", "manipulation" by storing the word "man" and having to export the last "n" Can be. If the last character of any of these other words has to be exported, it is clear that as far as English is concerned, there is an opportunity for these words not to be supplemented.
[0089]
In fact, if the repository is a few megabytes long, the slack condition can easily exceed 50%. Therefore, a finer approach, sub-clusters, is needed. For simplicity, the subcluster is a power of one half of the cluster. Thus, the general addresses of an L-shaped string are cluster, subcluster, and byte addresses. Whenever the length of the L-string changes, the L-string can move to sub-clusters of different sizes. In the FOCUS repository, all digits (length, address) are encoded using an unlimited single-adjustment number of the minimum length technique. Space allocation is performed using dynamic FAT-free allocation (described below).
[0090]
Structure of FOCUS content
The current FOCUS L-string builds the next string of bytes: content, reference ID, and numeric value. The last two values are coded as unlimited single adjustment numbers of minimum length. The first value is the ID, or handle, associated with the reference used to access the content, and its value will be a multiplexed number for proximity purposes. The L-shaped string of the cross reference FOCUS includes a cross reference code, an ID, and a reference. Here, "content" is the entire string.
[0091]
The composite structure of the FOCUS string means two subsequent fields. The length of the string is the length of the data, after which the numeric value includes its own length (see unlimited single adjustment number of minimum length). When the feeding is performed such that the ID is actually monotonous and the position is also within the ID, it is sufficient to sort the contents on condition that the copy is maintained in the original order.
[0092]
Dynamic FAT-free allocation for filing systems
FOCUS itself is an operating system and uses its own filing system. While this filing system can be used by any operating system with great advantages, it also makes it possible to consider mounting the system directly on a chip mounted on a disk controller. Although this filing system does not use a FAT (File Allocation Table), this filing system is extremely easy to create and can perform all normal optimizations relatively easily. Instead of concentrating cluster assignments, chain these clusters within the cluster itself.
[0093]
The basic idea is to make an empty file a chain of clusters. This chain means that each cluster has a pointer to the previous cluster and the next cluster. If there is no previous or next cluster, the pointer is set to any value (usually 0) that cannot be a valid logical / physical cluster address. This chain, called the free chain, can be initialized throughout the file, if the size is known at the time the file is created, or can be dynamically expanded by one or more clusters when needed.
[0094]
Cluster assignment is simply done from the free chain to another chain built on the same principle. Unassigning a cluster is returning it to the free chain. If the number of chains is limited, a first cluster (root) can be used to store the start and end points of these chains. If the number is not limited, the record itself uses the chaining technique. All of the usual functions of a file system can be easily implemented. For example, defragmentation is performed by copying the final cluster into the clusters of the free chain and truncating the file if the free chain is a continuous set of clusters at the end of the file.
[0095]
Multi-processing
Database
Database content reference
When indexing a database, a structure that can be used as data for indexing is implicitly provided. In general, knowing that a word is in table T, record R, field F, paragraph P allows more accurate access to this word. This information can be stored with the word itself using postfix coding techniques (also used for text structure information). A database of matches can be constructed using the information retrieved by the query. This ability is greatly enhanced by the concept extraction techniques used in the textual information below.
[0096]
There are several ways to evaluate items stored in the repository, such as titles, that belong to a database. These assessments can be recorded and further used to build a database from unstructured data. Merging structured and unstructured data can be easily achieved in FOCUS. The most interesting application of this is in the version that monitors the structure in the structure data. Here the type of coding is inappropriate. An important choice is related to the order in which these elements are stored. If multiple access paths are used, multiple stores in different orders can be used.
[0097]
Multi-processing technology does not call the operating system and is therefore independent of the OS. Using mailboxes between processors, these boxes are dedicated file names that can even carry parameters. This not only allows synchronization inside and outside some processes, but also allows data to be sent to these processes along with a file explorer type program. The same control can be obtained by executing the command anew with the new parameters, but the control technology of this Explorer type program allows the control panel of the process to be obtained by looking at the file names in a directory. To For example, the setting of the debug level can be controlled by a file name such as “debug.005”.
[0098]
When synchronizing processes across some kind of pipeline chain, it suffices that the first process outputs to "1 out.tmp" when the file is complete and renames it to "1 out.go". Next, a second process of searching for a particular file name is triggered. Making the two processes parallel, along with the processing of multiple files, simply means numbering the synchronized files, for example, "1 out001.go". The second process can easily retrieve these files in the order of increasing numbers. If the process is not resident, the coordination must be dedicated to the command program. This will start a different process depending on the existence of the process input file and other synchronization considerations. When the process is resident, a command program can be used to have a centralized scheduler.
[0099]
Application to text information
Text itself; optimized Unicode storage
In order to uniformly record text information, FOCUS uses a coding method according to the Unicode standard. This method is an open-ended two-byte encoding scheme that covers all glyphs used on the planet. Nevertheless, this scheme doubles the size of the text information. The way information is stored in the repository optimizes this space. That is, the L-shaped string does not copy the predetermined character. Although the Unicode standard allows for unique encoding for all hieroglyphs on this planet, available systems cannot handle the problem of badly accented words or inconsistently combined characters. Treat all of these with regular tokens to indicate the way character. The replacement processor (item 23) also handles misspelled words and broken words.
[0100]
One word may have more than one indication. The number 12 can be written as the sequence of characters "twelve" and as the sequence "12" or as the number 12. Nothing in FOCUS prevents using three displays at a time. A similar situation applies to compound words such as "weekend". This word can be stored at the same address as "weekend", "week-end", "week" and "end". For languages such as German or Swedish where most words are compound words, FOCUS stores the entire word with components such as the "weekend" example above.
[0101]
Addressing system based on grammar
Finding words in text can be done in several ways. Three of these will be described. a) the physical address, ie the number of the byte in the file where the first character of the word is present; b) the page number, the line number and the word number on the line where the word is present; c) the book number, chapter number, paragraph. Number, sentence number and word number. These initial information do not provide any information on the fact that there may be two consecutive words in the same sentence. The second information indicates the end of the line recorded in the file, information about how long a line exists, and what wrapping technique was used. The third piece of information relies on grammatical analysis. Using integer multiplexing coding techniques, all these techniques result in a single value representing the address. This means that other discovery schemes can be designed freely. In this regard, semantic information can be attached to a paragraph or sentence, all of which is performed when the text is sent to the repository. If you want to be independent of delimiters, mistypes, and want to be able to access text through the components themselves, compound words can be stored with those components. When using a grammar engine, numbers and dates can be stored under text and numeric forms without problems.
[0102]
Graphic rendering of algebraic / logical expressions
Those who work with mathematical formulas make good use of the usual display of mathematical formulas that use parenthesis pairs. It is difficult for these people to express complex conditions because those who have to express Boolean algebra usually do not practice this form of coding. By displaying the equations in a two-dimensional surface, the need for parentheses or pairing can be eliminated, giving a good understanding of complex equations. Such displays are also relatively intuitive, and people have used them for quite a short time.
[0103]
If you enter an item in a series of lines and the line is indented than the previous line, there is a parenthesis at the end of the expression formed by the line above the expression. Every new line can have its own Boolean operator shown on the buttons (AND, OR, NOT). In the FOCUS prototype, the intrinsic Boolean OR operator links all items on a particular line. Note: Boolean expressions can be applied in two ways, and the method used by computer scientists is confusing to ordinary people. When a database user looks for a record of "Brown" AND "Smith", the user must enter the expression "Brown" OR "Smith" (these two words are double with each other). The Boolean operator is an operator used by ordinary people.
[0104]
Co-occurrence
Co-occurrence is defined as a link between two words. This ink is just the fact that the two words are both in the same paragraph. Studies show that simple word co-occurrence does not yield interesting results due to semantic clarity. In contrast, groups or surfaces of words are semantically rich enough for use with co-occurrence networks. The basic function performed by co-occurrence is to provide a list of expressions from a given request. This representation is the one that occurs most frequently in this word, and all of the co-occurrences between all these representations are given to the graphic display.
All co-occurrence calculations are based on the results from a given request. The request can be a single word, a single expression, or a combination of Boolean algebra of words and expressions.
[0105]
Co-occurrence is an old idea in linguistics and means that interesting information can arise from words or expressions most frequently associated with a given term.
If n is the number of terms, the usual co-occurrence process is n raised to the third power. This means that even large base and RAM memories are impractical because of the enormous CPU time. Curiously, most sufficient co-occurrence systems use what is known as clustering. This technique consists in choosing an arbitrary value (usually about 2 Kbytes) in which the co-occurrence is calculated. This value has obviously no linguistic meaning. FOCUS can record the paragraph numbers of the terms it analyzes, and can perform concurrent calculations with more important linguistic elements.
Further, if FOCUS stores the cross-reference of paragraphs in the relevant words and tons, the whole process can be linear.
[0106]
Calculating the co-occurrence in FOCUS is simply sending a second query for the first given paragraph number. Then, sorting the results (linear as far as FOCUS is concerned) makes it easier to get the most frequent words and expressions related to the query. Under FOCUS, co-occurrence is not related to one term, but to the whole query.
[0107]
Right window
A key feature of the user's interface to the FOCUS interactive search function is what is referred to as the right window, which appears in the upper right corner of the prototype and first version interface sold. In the web interface (Java applet), this upper right window has been partially moved to the bottom window (listed with file references). The main use of this window is to interact with the user about each character typed.
When the user starts typing a word, this window shows the user's typing verbatim to indicate that FOCUS knows that he is going. Conversely, if the window is left empty, it indicates that some of the typed characters are not being reflected within FOCUS knowledge.
[0108]
As long as the user is within the word (ie, as long as the cursor is immediately after the character), the suggested display in the right window will be the word starting with the letter before the cursor, along with the next different letter for each word being displayed. Sample. For example, if the user's cursor is after the character "BE" and FOCUS has the words BEE, BEEF, BENATH, BENIGN, the screen space is simply limited and the user will not see the word starting with BEF, , BEEORE only, since it is enough for the user to know that it will not find BEFORE.
[0109]
When the user ends one word, ie the user enters one space to move on to the next word, we assume that FOCUS, along with the number of documents and the frequency of occurrence in these documents, (Set a mixed case, accent, and concatenated characters for these options). Finally, when the user enters the requested text and asks for an answer for each displayed document reference, the right window displays this list of documents.
[0110]
OCR type concatenated characters
The replacement engine used to handle alphabetically concatenated characters for optical character recognition errors can be used. Assuming an engine that does not make too many errors, that is, an engine that does not make errors so that the output can be read easily (our current criterion is Finereader V4), the data in the data for the replacement engine is " Common OCR errors can be declared, such as confusion between "rm" and "m". If you search for "modern", you will find misspellings like "modem". Such a function, which should be triggered by a menu option or other suitable means, allows to maintain a valid output from the OCR engine without manual correction and to search for suitable data between them.
[0111]
Java applet + Javascript structure
Current CGI (HTML) mode of operation on the web has a longer response time when compared to FOCUS. Therefore, it was necessary to design a new method to maintain the highest possible interactivity. Do this also if you use a Java applet to handle the question dialogue. Each time you press a key, only a few characters are sent to the network, assuming no pending queries. Advantageously, the response of the FOCUS can be kept very small (eg 2K), again minimizing the load on the network. In contrast, using a CGI script requires more than ten times the character sent to redraw the entire page. Similarly, when the query is completed and FOCUS sends the document list with their list, only the data is sent and the formatting is done locally by Javascript. A 10-fold improvement over the CGI method. The key point in all of these is to manage the dialog between the applet and the server so that queries are not sent, provided that one query is currently in the pipeline.
[0112]
Application to multimedia information
Multimedia
Data currently stored on the computer apart from text includes sounds and images. These codes can then be used as prefixes for data related to sound and images. Actually, apart from the text data, it is also possible to store numbers and dates or periods converted into a number of time units as numerical values. All these data types can be characterized by different prefixes. This keeps all data in one repository and guarantees both storage and search speed. If we decide to use the Unicode standard for text encoding, we know that some 2-byte codes cannot be valid Unicode values. Thus, the prefix for text in a prototype implementation of FOCUS is merely a potentially valid Unicode word.
[0113]
Automatic text and image correction
Some products can extract image signatures. After the linear conversion, these signatures can be inserted into the FOCUS. When taking an image database with text associated with the image, a number of corrections can be made between words and expressions, focusers and image signatures by statistical methods. Subsequent extraction of the signature from the type-like image makes it possible to propose words, expressions and focusers for this new image. That is, a method of automatic image description and text generation is obtained. For text-based documents, creating annotations with images (if they exist) is another way to do this. If nothing exists, the same output is obtained by getting the theme around the text. What has been said about images can be extended to sounds, music, drawings, videos, etc.
[0114]
Application to sound
As far as humans are concerned, we can plot sound in three-dimensional vector space: speech, music, noise. For example, indexing speech requires waiting for an efficient speech-to-text converter, but such converters are not currently available. The speech already has textual characteristics in the form of a MIDI file, which carries the name of the note to be played along with the text of the lyrics. That is, it carries discrete data different from continuous data recorded on a magnetic tape. Searching for MIDI frames can be performed in much the same way as searching for text by word. When the search reaches music in the form of a name file, there are wave-to-MIDI converters already available, but these converters are currently limited to monophonic music. There is no indication that a multi-melody converter will be available soon, but when it becomes available, it will revert to storing the MIDI information scheme. In the wave-to-MIDI conversion, it is also possible to identify the timbre of a musical instrument. Although noise can be an unwanted part of the signal, noise is typically handled by denoising devices. However, noise is also a special effect. Thus, noise is input for the analysis software. When considering the soundtrack of an opera, all three dimensions are obtained at the same time, far from being able to identify all these components. However, as soon as the appropriate analytical software becomes available, it will soon memorize the elements that can be provided and provided by the analytical program.
[0115]
Indexing MIDI files
Note that the values in the MIDI file are coded in 7 bits. Since the search proceeds without reference to the actual pitch of the notes, all that needs to be stored is the difference between successive notes. Therefore, the first recorded value is always zero, which means that it can be skipped. Musically, this means, for example, that in C key everything is chromatically transposed. Since the value is coded with only 7 bits, the difference between two notes could always be recorded in one single byte.
[0116]
It is known that such encoding is possible. The problem is what to choose for coding. There is a need for a software program that attempts to identify the melody channel (usually monaural) and cuts this melody into musical phrases. Such a slicing program can be basic at first because most people only remember the beginning of a berth or bridge. To search for a particular MIDI file, play a few notes on a keyboard or sing a few notes on a microphone equipped with a pitch for MIDI conversion. Because the information is transposed, perfect pitch is not required. Singers out of tune are assisted by basic fuzzy logic that uses a replacement engine.
[0117]
Music indexing by MIDI
Instruments and programs are commercially available for using wave-to-MIDI to convert monaural soundtracks to MIDI data.
[0118]
Creating standards in the music industry
This also applies to disc direct recording (D2D), CD, DVD, and the like. At the manufacturing level, mixing music eliminates the need for wave-to-MIDI conversion. Nevertheless, having someone play a melody on a MIDI instrument with music is quite cheap. This can be done by FOCUS and provide a repository for production. Any music phrase will be accessible.
[0119]
Application to images
There is one category of images that facilitates some kind of discrete access. That is, there is a category of an image created by a CAD system such as Corel Draw, AutoCad, DesignCad or a vector drawing program. In these images, all elements are discrete and their structure is defined even if they are not publicly documented. The text in these images is often maintained in a computerized text form, and all information about the image components is available. The location of some meaningful points (eg, the center for the picture) can be used as a location parameter.
Some attempts have been made to analyze the ability to find special patterns in certain graphic fields or to characterize some of the features of an image, such as a still image, which is a type of color composition, texture in an image. These programs provide a binary bit pattern that can be recorded in the repository. The fastest search results are obtained if these bit patterns have some sort of order in the pattern, so that only a portion of the repository is scanned to answer the query.
[0120]
For a video sequence, this process is exactly the same as the sequence being cut into uniform parts. Each uniform portion features the first image, and all subsequent images are determined as displacements from the first image. Similarly, one film is simply represented by a small number of images, returning to the problem of still images. Some representations related to motion can also be extracted from the video sequence. Access to music or video sequences can be provided by the time code of the industry standard SMPTE (Association of Motion Picture Technology Engineers). The value of this code at the start of the sequence can be used as a positioning parameter.
[0121]
Second, all these types of data usually do not occur alone, but they are accompanied by textual information. Textual information that is readily available in movies is scripts and dialogues (subtitles). The music information can be stored by placing the music information on the MIDI track of the melody along with the video sequence (the cost is not a major issue since a keyboard player with ordinary skills can do this at the same time). When the FOCUS is operated on these text data and the SMPTE code is used as an access method, it is possible to directly access a predetermined sequence using the text parameter and the music parameter.
[0122]
Search by image signature
This application assumes that external technologies can extract meaningful signatures, such as colors, compositions, textures, etc., from a given image. At the same time, these techniques tend to use parameters and seek concordance by convolution or other CPU-based techniques. FOCUS uses these techniques in very different ways. In fact, the general idea is to build a "graphic focuser". The same operation to search for parameters that are restrictions on the blue image from a group of images selected because it is quite blue, and to find the mean (or bracket) of the corresponding parameters and any kind of texture, composition, etc. It is easy to build a focuser that characterizes blue images. When analyzing a new image, a set of parameters is compared to the value of each graphic focuser, as was done for text. Record these values along with the image reference and the original parameters (if these parameters are available). Access is then granted through a user graphic, such as a graphic editor, for a set of images. This method is still appropriate and takes place immediately.
[0123]
Using a repository structure to dynamically manage structured data domains
A domain is just a subset that is the response to a query. As the database is involved and changes, the same query will obviously generate different domains. Thus, the domain is simply defined by the query itself, and from now on applies to everything analyzed by FOCUS. The definition of the domain can be done dynamically by the focuser and is stored for use as a reference. A domain is needed to describe the action fields of such things as focusers and synonyms (thesaurus). Their primary use is to serve as a category or taxonomy. Given the terms, FOCUS knows the definition of the domain in which this domain is used and can assign these domains to the user so that the user can limit their questions to it. For example, typing cell allows you to make a choice between spreadsheet and biology. Since a domain can be defined not only by one or more focusers but also by a larger number of query items (eg, date brackets, directories, URLs, etc.), the notation of the domain is wider than that of the focuser.
[0124]
Cross reference
(Like any other non-text element, it is given a special code in FOCUS. This cross-reference is used extensively every time an indirect access must be made. For example, in the document "mydoc" There is going to be a given word, this document name can be changed (by a rename or move operation), so coding it as "mydoc" with this word is extremely ridiculous. "Mydoc" is given an ID (usually one number) that must point to the actual name (mydoc for the time being) .When moving or renaming a document, this ID does not need to be changed; Update only the cross-reference to the document name You can use a cross-reference for each item that is comprehensive for the size, language, etc., as well as field names in the database, a set of data, and if you can change this item, use this cross-reference. Especially enforced. For fixed data, not using a cross-reference only wastes space, but there may be applications that need this space. To remember if there is one, our implementation of co-occurrence also uses a cross-reference.
[0125]
Application to information displayed by a list of networks
Chemistry and other networks
FOCUS has been studied for storing a special type of data, for example, images of chemical molecules. What is actually to be handled is a network with different nodes and different links.
The quick answer (which deals with the 24M molecule, which is patented today, despite the quickness) is a one-way display-only hierarchical method for each node, for all of the networks (ie starting at any node). Describe the possible indications. Where nodes and links are represented by numerical codes, uniqueness can be imparted by simply commanding different ways of navigation through the network in numerical sorting. The loop is just a special link (we suggest giving the lowest value after the value of the end of the string), this link is the number of backward links to the common node and the forward link (in the description) to the loop node. Jump to the node described by the number of links to. Special groups of nodes and links can be easily displayed as special node values. The recorded network is entered at the time of description. The query network selects a non-ambiguous node to start (preferably the furthest node of the ambiguous nodes), describes the network in all possible forms (according to the group of nodes that the network may contain), and describes the FOCUS Executing the corresponding request for. Due to the fuzzy definition of a given group of nodes, not all replies to inquiries to FOCUS can be replies to the user. Therefore, it is necessary to select the correct answer.
Already registered 24M molecules require a 90 Gb repository and access is usually adequate, no longer than one hundredth of a second.
[0126]
Embedding FOCUS in current applications
As already mentioned, FOCUS allows re-visits to all of the computer applications we currently have. This is not just a new feature that can be called from an application as an external routine, but it can actually be embedded in part of all current programs. For example, in a file explorer, FOCUS can be used to replace the current limited access to content. Also, if the user can type a word associated with a file name, FOCUS will immediately access all file names containing that word. No further navigation is provided by an endless directory tree structure. Assuming a large number of people are connected to the network, the agreement on directory names and / or certain limited conventions in the architecture allows people to focus on other directories at once.
[0127]
In word processing, spreadsheet, and database applications, the focusing process on the current paragraph or text field allows the user to get at once all relevant paragraphs for all data that has been accessed. It is sufficient to click on any word, expression or cell to recall all relevant data. In addition to displaying the information found, calls to the FOCUS search routine can also be used to provide the results to the current application.
[0128]
Target system
Instead of using the clipboard copy or drag and drop techniques that are always needed to move from one window to another, the target system can be used with the viewing routing associated with the query interface. The target is identified by a single letter, and the target consists of either a file or an application that sends the selected item by typing the relevant letter. By using the Ctrl or Alt key while holding down the contextual menu and / or character keys, the user can monitor the target, thereby allowing the various names associated with the file name, target application, and data transmission. Options can be changed. Basically, if the target is a file name, all information sent to that file is simply appended. This not only saves a lot of time compared to current copy / drag and drop technology, but also allows users to send the results of their queries to a given batch (journalists can Consider creating citations for different papers at once). A batch application host can also be used as the output of FOCUS. That is, it sends all found paragraphs to a particular processing unit. Such a system is what FOCUS already uses for automatic focusing. This capability can be associated with the target technology, that is, sending all hits to the target file or application of the current file, or all hits or queries to the target.
[0129]
RDBM link
Basic access to data on a typical computer is through a "database". Since the most complex database is "Relational", the abbreviation RDBMS for Relational DataBase Management Systems has been created. This method, which was considered to be an easier way to access the data, has proven to be extremely flawed in handling data and is heavy and slow. For FOCUS, the structure is simply another type of content, such as a tree structure for directories. The following shows how DBMS data is sent to FOCUS. Optional, but recommended, allows one administrator to select which fields to record in the FOCUS. Next, an initial analysis, which can be performed simply by dumping the database in ASCII, Unicode or SGML format, is performed, and then this format is analyzed by FOCUS (eg, as with regular text). For small databases (hundreds of megabytes), if real-time updates are not a problem, the method can be repeated periodically. However, if the base is several terabytes in size or a real-time response is desired, an update method is needed. In each case, the alias file redirects the reference stored by FOCUS back to the original record / document, as would be done for portal or access to the network document. Assume that the intermediate format is XML (HTML, SGML and all structured text formats are equivalent for this purpose).
[0130]
Data selection: The administrator selects the field to be analyzed from the base structure and indicates whether to change or keep the field name in the XML tag that FOCUS associates with the content.
Initial Analysis: As described above, this analysis is a simple dump of the base selected field.
Update: This solution is to use a virtual table with the same structure as the original structure, and dump this virtual table periodically, as for an initial analysis with the same parameters. FOCUS handles the history of the record by removing previous occurrences, as it does for all other documents. The second solution is more complex but more efficient. This method consists in reading the internal logging format of the DBMS to achieve the same result. The first solution implies slight changes to all applications using DBMS, but does not imply knowledge of internal coding. On the other hand, the second solution does not mean interference with the application, but does mean knowledge of the internal structure.
[0131]
The invention disclosed by the specific embodiment and its application has been described above. However, those skilled in the art can make various modifications and changes without departing from the scope of the invention described in the claims.
[Brief description of the drawings]
FIG.
Figure 6 illustrates functional access of a computer system accessed by content.
FIG. 2
(A) shows the naming of a monotonic numberless sequence, and (B) shows more details for the naming of a monotonic numbering sequence.
FIG. 3
This shows a part of the linear sorting technology.
FIG. 4
3 shows an L-shaped repository file.

Claims (36)

(A) defining content;
(B) classifying the content;
(C) indexing the content, comprising: a content accessible system for operating a computer comprising: The method of claim 1, further comprising the step of specifying all real numbers in a scheme, wherein the real numbers can be easily and monotonically located. The method of claim 1, further comprising the step of sorting by content according to a linear and fast sorting method. A method for a content accessible system for operating a computer, comprising the step of specifying all real numbers according to a scheme, wherein said real numbers can be easily and monotonically arranged. 5. The method of claim 4, further comprising the step of sorting by content according to an associated monotonic real number, said sorting step being linear and fast. A method for a content accessible system for operating a computer, comprising the step of sorting by content according to a linear and fast sorting method. (A) including a step of designating all real numbers by a scheme, wherein the real numbers can be easily and monotonously arranged;
(B) A method for a computer-operated content-accessible system that further comprises sorting by content according to an associated monotonic real number, wherein the sorting process is linear and fast. (A) defining, classifying and indexing the content;
(B) A method for a content accessible system for operating a computer, further comprising sorting by content according to a linear and fast sorting method. (A) defining, classifying and indexing the content;
(B) A method for a content accessible system for operating a computer, comprising the step of specifying all real numbers, wherein said real numbers can be easily and monotonically arranged. (A) defining, classifying and indexing the content;
(B) A method for a content accessible system for operating a computer, comprising the step of specifying all real numbers, wherein said real numbers can be easily and monotonically arranged.
(C) A method for a content accessible system for operating a computer, comprising the step of sorting by content according to an associated monotonic real number, said sorting step being linear and fast. (A) utilizing a Focuser, the step further comprising: (b) utilizing an L-shaped string further comprising an optional horizontal string and at least one byte of a vertical string;
The method of claim 10, further comprising: (c) pointing to a next L-shaped string by at least one byte of an associated vertical string. The method of claim 11, further comprising utilizing defragmentation to efficiently perform software garbage collection. (A) using a sub-cluster;
12. The method of claim 11, further comprising: (b) defining a subcluster as a power of a half of the cluster; and (c) addressing the L-shaped string generally by consecutive clusters, subclusters, and byte addresses. Method. (A) encoding all numbers, including length and address, as a single adjustment number in the repository of Focus (a computer system using fully optimized content);
The method of claim 11, further comprising: (b) encoding the integer according to the monotonic real number. (A) constructing a string of bytes for the L-shaped string of the focuser;
15. The method of claim 14, further comprising: (b) configuring the string with content, reference ID, and numerical value. (A) utilizing two numeric fields, data length + data field length;
(B) encoding the numeric field with a single adjustment number;
(C) utilizing a monotonic ID and a monotonic location within the ID to sort by content. (A) selecting an empty file as a chain of clusters;
(B) using two pointers in each cluster;
(C) pointing one pointer to the previous cluster;
(D) pointing the second pointer to the next cluster;
(E) setting a pointer to any invalid address value if the next or previous cluster does not exist. (A) initializing an empty chain of the cluster as a free chain; and (b) initializing the free chain across all files if the size of the file is known when the file is created.
(C) dynamically expanding the free chain with one or more required clusters;
(D) allocating from a free chain to another chain;
(E) deallocating the cluster by returning the cluster to the free chain;
(F) storing a start point and an end point of each chain in a first cluster, wherein the first cluster is a root;
18. The method of claim 17, further comprising the step of: (g) performing defragmentation by copying the final cluster into a cluster of a free chain, and truncating files when the free chain is a contiguous set of clusters at the end of the file. The described method. (A) using an operating system-independent multiprocessing technique that does not call the operating system;
(B) using a mailbox between processors;
(C) dedicating a file name for the mailbox carrying parameters. (A) performing synchronization inside and outside the plurality of processors;
(B) sending data to a plurality of processors by a file explorer type program;
20. The method of claim 19, further comprising the step of: (c) obtaining a control panel of the processor by searching for a file name in the directory with a file explorer type program. (A) synchronizing processes across the pipeline chain;
(B) outputting the output of the first process to “1 out.tmp”;
(C) renaming the file to “1 out.go” when the file is completed;
(D) searching for a specific file name;
(E) triggering a second process when a file name is encountered;
(F) synchronizing files to be numbered, such as “1 out001.go”;
21. The method of claim 20, further comprising: (g) parallelizing the two processes with the processing of multiple files. (A) dedicating a command program to coordination when the process is not resident;
(B) starting a different process according to the existence of the output file of the process;
The method of claim 21, further comprising the step of: (c) utilizing the command program as a centralized scheduler when the process is resident. (A) indexing the database;
(B) using an implicitly provided structure for indexing;
(C) storing data structure information for one word;
(D) using a word in table T, record R, field F, paragraph P for the information;
(E) utilizing a post-fix coding method to store the information along with the words themselves. (A) identifying a name for the focuser;
(B) associating a list of words and expressions indicating a focuser;
(C) identifying the language used by the focuser;
(D) identifying some parameters of the focuser, wherein identifying the parameters comprises:
(1) including indicating the number of words or expressions in the paragraph, the lesser number found in the paragraph indicating that the paragraph for the focuser does not correspond;
(2) further comprising the step of examining the number of words or expressions in the paragraph, if the number is small, the paragraph may correspond; if it is larger, the paragraph actually corresponds to the focuser And
(3) The method according to claim 10, further comprising determining whether the degree of correspondence is above or below a threshold, and if it is lower than the threshold, determining that the word does not belong to the focuser. (A) constructing a focuser from text utilizing a known general frequency of words in a given language;
(B) determining, as a whole, the number of correspondences as a ratio of the frequency of words in the text divided by the general frequency of words in the language;
(C) calculating the number of degrees of correspondence;
(D) displaying the uniqueness of the word found in the paragraph;
(E) ordering all words from the source text by decreasing the degree of correspondence value;
26. The method of claim 24, further comprising the step of: (f) displaying the word having the highest degree of correspondence value as the most corresponding word for the paragraph. 25. The method of claim 24, further comprising automatically constructing a focuser by utilizing a corresponding number of words in the text. (A) manually raising the focuser;
(B) further comprising the step of creating specific information, the information being (1) a specific expression that corresponds very closely; (2) synonyms (automatically suggested by the system); and (3) a focuser. (4) to distinguish the focuser, ie words or expressions automatically excluded from the focuser (negative correspondence values) 25. The method of claim 24, comprising: (5) a word that is acceptable except for all expressions that include the word. (A) recognizing the number of paragraphs of the word or expression relating to the focuser in the same paragraph;
28. The method of claim 27, further comprising the step of: (b) detecting the focuser. 29. The method of claim 28, further comprising setting the number of paragraphs of a word or expression related to a focuser as three. (A) routing by a profile that is a set of positive and negative focusers;
(B) defining filtering by a profile that is a set of positive and negative focusers;
29. (c) rejecting the text if a negative focuser is recognized and accepting the text if at least one positive focuser is detected. (A) automatically constructing concepts associated with any word in the text repository;
26. The method of claim 25, further comprising: (b) following an algorithm. (A) collecting all paragraphs in the text including the word;
(B) extracting the concepts from the concatenation of paragraphs. 33. The method of claim 32, further comprising the step of restricting said extraction to expressions containing at least two meaningful words. (A) further comprising the step of identifying the document prior to opening the document, wherein the identifying comprises:
(B) using one or more focusers;
(C) applying one or more focusers to documents in the one or more focuser repositories;
34. The method of claim 33, comprising: (d) generating the list of documents, wherein the list is an abstract of the document created by the one or more focusers. (A) analyzing one or more FOCUS repositories and creating them in real time;
35. The method of claim 34, further comprising: (b) compiling and creating a dynamic focuser, wherein the dynamic focuser is generated in real time. 36. The method of claim 35, further comprising the step of splitting the repository, wherein the overflowed portion can then be split without leaving unnecessary empty disk space.

Images