WO2001050327A2

WO2001050327A2 - Method and apparatus for flexibly assigning tokenization procedures

Info

Publication number: WO2001050327A2
Application number: PCT/US2001/000177
Authority: WO
Inventors: Jacek Ambroziak
Original assignee: Sun Microsystems, Inc.
Priority date: 2000-01-06
Filing date: 2001-01-02
Publication date: 2001-07-12
Also published as: JP2004501412A; EP1386248A2; AU2757901A; WO2001050327A3

Abstract

One embodiment of the present invention provides a system that tokenizes text within a document by converting the text into tokens that correspond to individual meaning-carrying units of text in order to facilitate searching through the text. The system operates by receiving the document to be tokenized, and retrieving a set of tokenizing instructions associated with the document. Next, the system tokenizes the document by translating the document into tokens that correspond to individual meaning-carrying units of text in a manner that is specified by the set of tokenizing instructions.

Description

METHOD AND APPARATUS FOR FLEXIBLY ASSIGNING TOKENIZATION PROCEDURES

BACKGROUND

The present invention relates to indexing structures to facilitate computerized searches through data. More specifically, the present invention relates to a method and an apparatus for flexibly assigning tokenization procedures to convert a document into tokens associated with individual meaning-carrying units of text, such as wordforms or numbers.

The explosive growth of the Internet has been strongly tied to the development of search engines that allow users to rapidly search through large volumes of textual data from thousands and even millions of different web sites. A user who is interested in a particular topic merely has to enter a number of keywords into a search engine in order to receive linkages to different web pages containing the key words.

Search engines typically create an "index" of documents (such as web pages) that are available on the world wide web. An index typically stores individual words (or other meaning carrying textual strings) in a more compact and easily searchable form known as "tokens."

The process of building an effective index can be greatly complicated by the fact that documents can exist a wide variety of different forms which need to be indexed differently. For example, an efficient index for a technical paper might contain the abstract and title of the technical paper, but not the body of the technical paper, whereas an efficient index for a television schedule might contain ratings for individual television programs. The process of creating an index is also complicated by the fact that for common document formats, such the Hypertext Markup Language (HTML) or the Extensible

Markup Language (XML), much of the important information for search purposes is stored within attribute fields, and is not within the normal text of the document. Furthermore, the structure of a document may change over time, which can require the structure of the index to change. For example, suppose the structure of a product catalog is updated to include consumer reviews for individual products. This change may require the index to change to include these consumer reviews.

Existing systems create indexes for documents using ad hoc rules. For example, one ad hoc rule is to create an index for all textual information that is not within attribute fields. Unfortunately, such ad hoc rules often include much unimportant information in the index, and often exclude important information.

A similar problem exists in converting the document into tokens (tokenizing the document) during the index creation process. During the index creation process, relevant portions of a document are converted into tokens associated with individual meaning- carrying units of text, such as wordforms or numbers. In the English language, wordforms are typically delineated by white spaces and punctuation marks. Hence, the tokenizing process is relatively easy. In contrast, languages such as Japanese have no such delineation. Consequently, the tokenization process depends on contextual information and can be very complicated.

The tokenization process can also be domain dependent. For example, periods within an email address, such as "person.dept@companyx.com" are linking elements, whereas periods within other textual information typically delineate word and sentence boundaries. Hence, the tokenization process varies between languages and between domains.

SUMMARY

One embodiment of the present invention provides a system that tokenizes text within a document by converting the text into tokens that correspond to individual meaning-carrying units of text in order to facilitate searching through the text. The system operates by receiving the document to be tokenized, and retrieving a set of tokenizing instructions associated with the document. Next, the system tokenizes the document by translating the document into tokens that correspond to individual meaning- carrying units of text in a manner that is specified by the set of tokenizing instructions.

In one embodiment of the present invention, tokenizing the document involves using a first set of tokenizing instructions to tokenize a first section of the document, and using a second set of tokenizing instructions to tokenize a second section of the document. In one embodiment of the present invention, the set of tokenizing instructions is contained in a plug-in module.

In one embodiment of the present invention, the set of tokenizing instructions is activated through an object defined within an object-oriented programming system.

In one embodiment of the present invention, the system additionally uses the tokenized document in generating an index for the document. In a variation on this embodiment, the system makes this index available to a search engine so that the search engine is able to scan through the index.

In one embodiment of the present invention, the system retrieves the set of tokenizing instructions across a network from a remote address. In one embodiment of the present invention, the set of tokenizing instructions is appended to the document.

In one embodiment of the present invention, the set of tokenizing instructions is contained in a tokenizing procedure associated with the document.

In one embodiment of the present invention, the document is received from a client at a tokenizing server. In this embodiment, the tokenizing server returns the tokenized document to the client.

In one embodiment of the present invention, the set of tokenizing instructions is provided by a remote service across a network. BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a distributed computing system in accordance with an embodiment of the present invention.

FIG. 2 illustrates how an indexing server creates an index for different document types in accordance with an embodiment of the present invention.

FIG. 3 illustrates how indexing stylesheets and tokenizing procedures are used to create and index for a document in accordance with an embodiment of the present invention.

FIG. 4 illustrates an example of the tokenization process. FIG. 5 is a flow chart illustrating the process of creating an index in accordance with an embodiment of the present invention.

FIG. 6 is a flow chart illustrating the process of dynamically creating an updated index for an old document in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION

The following description is presented to enable any person skilled in the art to make and use the invention, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present invention. Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

The data structures and code described in this detailed description are typically stored on a computer readable storage medium, which may be any device or medium that can store code and/or data for use by a computer system. This includes, but is not limited to, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact discs) and DVDs (digital video discs), and computer instruction signals embodied in a transmission medium (with or without a carrier wave upon which the signals are modulated). For example, the transmission medium may include a communications network, such as the Internet.

Distributed Computing System FIG. 1 illustrates a distributed computing system 100 in accordance with an embodiment of the present invention. Distributed computing system 100 includes clients 102 and 118, which are coupled to indexing server 112 and search engine 122 through network 110.

Network 110 can include any type of wire or wireless communication channel capable of coupling together computing nodes. This includes, but is not limited to, a local area network, a wide area network, or a combination of networks. In one embodiment of the present invention, network 110 includes the Internet.

Clients 102 and 118 can include any node on network 110 including computational capability and including a mechanism for communicating across network 110.

Client 102 contains a number of documents 104-106, which are to be integrated into index 116 within indexing server 112. Indexing server 112 can include any node on a computer network including a mechanism for servicing requests from a client for computational and/or data storage resources. More specifically indexing server 112 includes resources for creating index 116 within database 114 for indexing documents 104-106. Database 114 can include any type of mechanism for storing data in nonvolatile form. In one embodiment of the present invention, database 114 includes the ORACLE 8™ database distributed by the Oracle Corporation of Redwood Shores, California. Client 118 includes a browser 120 that communicates with search engine 122 in order to scan through index 116. Browser 120 can include any type of web browser capable of viewing a web site, such the INTERNET EXPLORER™ browser distributed by the Microsoft Corporation of Redmond, Washington. Search engine 122 can include any type of computing system or application that is able to search through data.

During operation, indexing server 112 retrieves documents 104-106 from client 102, and uses documents 104-106 to create index 116. Note that client 102 may send documents 104-106 to indexing server 112. Or, alternatively, indexing server 112 can gather documents 104- 106 from client 102.

Indexing server 112 creates index 116 by tokenizing selected portions of documents 104-106 and by creating index 116 from the tokens. Note that client 102 may itself be a server that makes documents 104-106 available over network 110.

After index 116 is created, client 118 submits a query 124 to search engine 122 through browser 120. Query 124 may specify keywords of interest to a user of client 118. In response to query 124, search engine 122 searches through index 116 to find documents containing matching keywords within index 116. If such documents are located, search engine 122 returns the documents in a list of query hits 126 to browser 120.

Indexing Server

FIG. 2 illustrates how indexing server 112 creates an index for different document types in accordance with an embodiment of the present invention. In FIG. 2, indexing server 112 receives many different document types from different sources. Indexing server 112 integrates these different document types into index 116. Note that index 116 may include a single index containing many different document types. Alternatively, index 116 may contain a different index for each document type.

An unlimited number of document types can be indexed. For example, FIG. 2 illustrates a news document 202, a product catalog 204, a television programming schedule 206, a document containing user documentation 208 and a document containing financial information 210. Each of these different document types can have a different document structure, which can be defined in a language such as XML. Each of these different document structures can be associated with a different indexing scheme. In some documents, specific attributes should be included in the index. For example, for some types of user documentation it may be advantageous to index an attribute that indicates whether the user documentation is suitable for a novice user or an expert user.

Indexing Stylesheets and Tokenization Procedures

FIG. 3 illustrates how stylesheets for indexing and procedures for tokenizing are used to create and index for a document in accordance with an embodiment of the present invention. In FIG. 3, an index building mechanism 310 within indexing server 112 takes as input a document 302 and produces an index 312 for the document 302. Index 312 feeds into a larger index 116 for a collection of documents, which is contained within database 114 (from FIG. 1).

During the index building process, index builder 310 refers to index stylesheet 304 and tokenizing procedures 306-307. Index stylesheet 304 contains a set of instructions for creating index 312 for document 302. For example, index stylesheet 304 can specify which sections of document 302 should be skipped in producing an index 312. Index stylesheet 304 can also specify attributes of document 302 to be included in index stylesheet 304. For example, an attribute can specify a minimum security-level for a person to have access to document 302. Another attribute can specify a content rating (G, PG-13, PG, R, X) for document 302. Tokenizing procedures 306-307 specify how certain portions of document 302 are to be tokenized. For example, tokenizing procedure 306 may specify how a first portion of document 302 is to be tokenized, while tokenizing procedure 307 specifies how a second portion of document 302 is to be tokenized. Although most documents will likely use of a single tokenizing procedure, other documents may includes sections in different languages or from different domains, which require different tokenizing procedures.

Note that index stylesheet 304 is analogous to format stylesheets that are specified by the XML standard. Format stylesheets are used to specify display attributes, such as font and color, in order to display a XML document. Similarly, index stylesheet 304 specifies how an index for document 302 is to be created. Note that although FIG. 3 illustrates indexing instructions and tokenizing instructions in the form of stylesheets and procedures, other representations are possible.

For example, the indexing and tokenizing instructions can be contained within plug-in modules that can be plugged in to index builder 310. These indexing and tokenizing instructions can also be referenced through an object defined within an object-oriented programming system. For example, an index parameter object can include a method that retrieves the instructions for building an index for document 302.

The tokenizing procedures 306-307 can additionally assume the form of code modules containing tokenization instructions, or may be provided by a remote service across a network.

Also note that index stylesheet 304 may contain references specifying where tokenizing procedures 306-307 can be retrieved from.

Note that index builder 310 includes standardized interfaces that are able to accept input from many different index stylesheets and tokenizing procedures. This allows index builder 310 to create indexes for many different document types using many different tokenization rules.

Tokenization Process FIG. 4 illustrates an example of the tokenization process. In this example, a piece of text "MAY 7, 2000" is divided into three tokens 402-404. Token 402 contains the word "MAY." Token 403 contains the day number "7," and token 404 contains the year number "2000." Each of these tokens is associated with a unique token number, and the unique token numbers are used to create the index. Using token numbers leads to a more compact representation because token numbers take up less space than large character strings. Furthermore, during the searching process it is easier to lookup a number than a character string. Process of Creating an Index

FIG. 5 is a flow chart illustrating the process of creating an index in accordance with an embodiment of the present invention. The system starts by downloading a configuration file (step 502). This process may involve downloading a configuration file across a network. Next, the system parses the configuration file (step 504) and identifies an address of an index stylesheet 304 within the configuration file (step 506).

Next, the system downloads the index stylesheet 304 from the identified address

(step 508). This may involve retrieving the stylesheet across a network from a location specified by a universal resource locator (URL). Alternatively, the stylesheet can be appended to the document, in which case, the index stylesheet can be easily retrieved.

Next, the system parses the index stylesheet 304 so that the instructions within the index stylesheet can be used during the index creation process (step 510).

The system similarly identifies an address of a tokenizing procedure 306 (step 512). (Note that the address of tokenizing procedure 306 may be contained within the configuration file, or alternatively, within index stylesheet 304). Next, the system downloads tokenizing procedure 306 from the identified address (step 514). In one embodiment of the present invention, tokenizing procedure 306 is retrieved from the same location as the index stylesheet 304. In another embodiment, tokenizing procedure 306 is retrieved from another location.

Next, the system inputs a number of documents into index 116. This is accomplished by downloading a document into index builder 310 (step 518) and then parsing the document using the instruction specified in index stylesheet 304 (step 520).

The system converts the parsed document into tokens using tokenizing procedure 306 (step 522) and then creates an index using the tokens (step 524). This process is repeated for each document that is inputted into index 116.

After index 116 is complete, the system makes index 116 available to search engine 122 (from FIG. 1), so that search engine 122 can scan through index 116 to process queries (step 526). FIG. 6 is a flow chart illustrating the process of dynamically creating an updated index for a document in accordance with an embodiment of the present invention. For certain types of searches data has a limited life. For example, in searching for current weather data, old weather data is of no interest. In these circumstances, one embodiment of the present invention operates as follows. The system receives a search request (step 602). In response the search request, the system checks document creation dates for any documents that are involved in the search (step 604). If the system determines that a document is old (by perhaps comparing the age of the document against an age limit), the system causes a new version of the document to be generated (for example, by gathering new weather data) (step 606). Next, the system creates an index for the new version of the document (step 608). This process implicitly removes the index for the old version of the document (step 610). Finally, the system performs a search involving the newly updated index.

The foregoing descriptions of embodiments of the invention have been presented for purposes of illustration and description only. They are not intended to be exhaustive or to limit the present invention to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art.

For example, although the present invention is described in the context of a distributed computing system including clients and servers, the present invention is not necessarily limited to a distributed client-server computing system. In general, the present invention can apply to any system that generates an index for textual information or any system that tokenizes textual information.

Additionally, the above disclosure is not intended to limit the present invention. The scope of the present invention is defined by the appended claims.

Claims

What Is Claimed Is:

1. A method for tokenizing text within a document by converting the text into tokens that correspond to individual meaning-carrying units of text in order to facilitate searching through the text, the method comprising: receiving the document to be tokenized; retrieving a set of tokenizing instructions associated with the document; and tokenizing the document by translating the document into tokens that correspond to individual meaning-carrying units of text in a manner that is specified by the set of tokenizing instructions.

2. The method of claim 1 , wherein tokenizing the document involves using a first set of tokenizing instructions to tokenize a first section of the document, and using a second set of tokenizing instructions to tokenize a second section of the document.

3. The method of claim 1, wherein the set of tokenizing instructions is contained in a plug-in module.

4. The method of claim 1, wherein the set of tokenizing instructions is activated through an object defined within an object-oriented programming system.

5. The method of claim 1 , further comprising using the tokenized document in generating an index for the document.

6. The method of claim 5, further comprising making the index available to a search engine so that the search engine is able to scan through the index.

7. The method of claim 1 , wherein retrieving the set of tokenizing instructions involves retrieving the set of tokenizing instructions across a network from a remote address.

8. The method of claim 1 , wherein the set of tokenizing instructions is appended to the document.

9. The method of claim 1, wherein the set of tokenizing instructions is contained in a tokenizing procedure associated with the document.

10. The method of claim 1 , wherein the document is received from a client at a tokenizing server; and further comprising returning the tokenized document from the tokenizing server to the client.

11. The method of claim 1, wherein the set of tokenizing instructions is provided by a remote service across a network.

12. A computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for tokenizing text within a document by converting the text into tokens that correspond to individual meaning-carrying units of text in order to facilitate searching through the text, the method comprising: receiving the document to be tokenized; retrieving a set of tokenizing instructions associated with the document; and tokenizing the document by translating the document into tokens that correspond to individual meaning-carrying units of text in a manner that is specified by the set of tokenizing instructions.

13. The computer-readable storage medium of claim 12, wherein tokenizing the document involves using a first set of tokenizing instructions to tokenize a first section of the document, and using a second set of tokenizing instructions to tokenize a second section of the document.

14. The computer-readable storage medium of claim 12, wherein the set of tokenizing instructions is contained in a plug-in module.

15. The computer-readable storage medium of claim 12, wherein the set of tokenizing instructions is activated through an object defined within an object- oriented programming system.

16. The computer-readable storage medium of claim 12, wherein the method further comprises using the tokenized document in generating an index for the document.

17. The computer-readable storage medium of claim 16, wherein the method further comprises making the index available to a search engine so that the search engine is able to scan through the index.

18. The computer-readable storage medium of claim 12, wherein retrieving the set of tokenizing instructions involves retrieving the set of tokenizing instructions across a network from a remote address.

19. The computer-readable storage medium of claim 12, wherein the set of tokenizing instructions is appended to the document.

20. The computer-readable storage medium of claim 12, wherein the set of tokenizing instructions is contained in a tokenizing procedure associated with the document.

21. The computer-readable storage medium of claim 12, wherein the document is received from a client at a tokenizing server; and wherein the method further comprises returning the tokenized document from the tokenizing server to the client.

22. The computer-readable storage medium of claim 12, wherein the set of tokenizing instructions is provided by a remote service across a network.

23. An apparatus that tokenizes text within a document by converting the text into tokens that correspond to individual meaning-carrying units of text in order to facilitate searching through the text, the apparatus comprising: a receiving mechanism that is configured to receive the document to be tokenized; an instruction retrieving mechanism that is configured to retrieve a set of tokenizing instructions associated with the document; and a tokenizing mechanism that is configured to tokenize the document by translating the document into tokens that correspond to individual meaning-carrying units of text in a manner that is specified by the set of tokenizing instructions.

24. The apparatus of claim 23, wherein if the document is of a mixed token type, the tokenizing mechanism is configured to use a first set of tokenizing instructions to tokenize a first section of the document, and to use a second set of tokenizing instructions to tokenize a second section of the document.

25. The apparatus of claim 23, wherein the set of tokenizing instructions is contained in a plug-in module.

26. The apparatus of claim 23, wherein the instruction retrieving mechanism is configured to retrieve the set of tokenizing instructions through an object defined within an object-oriented programming system.

27. The apparatus of claim 23, further comprising an index generation mechanism that is configured to use the tokenized document in generating an index for the document.

28. The apparatus of claim 27, further comprising an access mechanism that is configured to make the index available to a search engine so that the search engine is able to scan through the index.

29. The apparatus of claim 23, wherein the instruction retrieving mechanism is configured to retrieve the set of tokenizing instructions across a network from a remote address.

30. The apparatus of claim 23, wherein the set of tokenizing instructions is appended to the document.

31. The apparatus of claim 23 , wherein the set of tokenizing instructions is contained in a tokenizing procedure associated with the document.

32. The apparatus of claim 23, wherein the apparatus is contained with a tokenizing server that receives the document from a client; and wherein the tokenizing server is configured to return the tokenized document to the client.

33. The apparatus of claim 23, wherein the set of tokenizing instructions is provided by a remote service across a network.