USRE46967E1 - System, apparatus, method, and computer program product for indexing a file - Google Patents
System, apparatus, method, and computer program product for indexing a file Download PDFInfo
- Publication number
- USRE46967E1 USRE46967E1 US11/513,423 US51342306A USRE46967E US RE46967 E1 USRE46967 E1 US RE46967E1 US 51342306 A US51342306 A US 51342306A US RE46967 E USRE46967 E US RE46967E
- Authority
- US
- United States
- Prior art keywords
- file
- files
- web
- search
- graphic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Lifetime
Links
Images
Classifications
-
- G06F17/30864—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24542—Plan optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9532—Query formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9535—Search customisation based on user profiles and personalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9538—Presentation of query results
-
- G06F17/30463—
-
- G06F17/30867—
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/912—Applications of a database
- Y10S707/913—Multimedia
- Y10S707/915—Image
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/953—Organization of data
- Y10S707/959—Network
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99931—Database or file accessing
- Y10S707/99933—Query processing, i.e. searching
- Y10S707/99934—Query formulation, input preparation, or translation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99941—Database schema or data structure
- Y10S707/99943—Generating database or data structure, e.g. via user interface
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99941—Database schema or data structure
- Y10S707/99944—Object-oriented database structure
- Y10S707/99945—Object-oriented database structure processing
Definitions
- the invention concerns methods and apparatus for representing data file contents for searching the data files and reporting selected data file addresses, especially hypertext markup language files accessed using an Internet search engine (i.e., Web pages).
- An Internet search engine i.e., Web pages.
- One process develops a database representing the text content of data files on a network.
- Another process renders graphic representations of the files according to a default configuration and stores a compressed graphic file for each.
- a further process selects file hits according to user criteria and reports their addresses with associated presentation of the stored graphic file.
- a search engine is a useful facility for browsing the Internet or World Wide Web.
- Popular browsers such as Microsoft Internet Explorer and Netscape Navigator display visual outputs using hypertext markup language or “html.”
- html hypertext markup language
- Html files typically contain, for example, text and numeric information, typographical symbols, information defining formatting particulars by which the text is to appear on a display of the file, and uniform resource location references (URLs), which are hypertext links that address other files.
- the user can highlight and select a URL by pointing and clicking using his/her mouse, whereupon the browser loads and displays the identified page.
- the link may be such that this point-and-click method causes the browser to jump to a display of a different position in the file, or to perform an identified action such as downloading and playing an audio or video file, or may cause the browser to alter its display of the present data, such as inserting or enlarging a display of a graphic file.
- the link may also cause the browser to invoke an applications program or a process, etc.
- the html files which are addressed typically contain certain formatting information. All users who download the html file obtain the identical file and formatting. However, the display and processing of the files is not necessarily the same from one user's browser to another.
- the html page does not contain a fixed graphic data display.
- the html page contains text, addresses and encoding information which are processed by the browser and the system operating the browser, to prepare and present a graphic data display.
- Browsers from different software suppliers are not identical and operate somewhat differently.
- the same browser program can be set up by user options for display of data in selected ways, including for example choices of font size and font type.
- applications programs that may be run within the browser (often called plug-ins) or which are invoked when a file of a particular type is selected.
- the operating system e.g., Microsoft Windows
- the display may be configured to employ a certain X-Y pixel size and color display resolution.
- the user may have selected one of several available font sizes, which in combination with the X-Y pixel size of the display field determines the vertical and horizontal size of each character. These choices affect pagination and the layout of text within text subdivisions such as paragraphs or tables.
- the browser may allow the user to select a default character alphabet.
- the browser may also allow the user to select how and whether background and foreground colors are displayed, or whether colors are even used in certain situations, such as to distinguish links from other text or to highlight a link when selected by the cursor or mouse.
- the typical html source file contains text and may include or contain addresses identifying static or dynamic files and information, but the source files are usually not limited to text.
- the source files contain header, footer, paragraph and section markers, font and color changes which may distinguish sections, markers indicating text strings to be interpreted as html links (URL addresses that are delineated as such), and other formatting and instructions.
- markers which include hidden text tags and textual start/stop markers, are not themselves displayed but instead are used to carry undisplayed information or as specifications for display of the remaining text according to preset rules and configuration choices in the browser and the operating system.
- “going to” the web page is a misnomer.
- the process actually involves sending a message to a remote server or user station on the web that requests transmission of the html source code stored there.
- the source code is processed locally by the browser so as to produce data representing a graphic display.
- the graphic display data is stored in a memory buffer in the system RAM or in an associated display driver card from which the luminance, saturation and hue of each pixel in the display are determined.
- the browser may store a copy of the source code locally so that using the “Back” function reloads the page without the need to wait for another exchange of messages over the Web.
- Search engines now operating do not search web pages on demand. Instead the search engine operators use various means to build a limited database reflecting the contents of a number of web pages.
- the users' search criteria are applied to the database to identify the addresses of web pages that meet the search criteria, at least from a subset of all existing web pages. Web page content can be changed.
- the search is current up to the most recent time at which the search engine database was updated to reflect the latest content of the web pages subject to search.
- the search engine operator can use various methods to find or select web page addresses that will be loaded and analyzed or indexed in building the database.
- the methods may be chosen to expand or to limit the number of web pages that the search engine will access. As a result, the results of searches vary among the different search engines.
- a web crawler or similar routine might attempt to load and analyze pages corresponding to all the top level domain names that are found to be registered with public domain name services or listed in a directory service [e.g., http://www.[domain].com].
- Search engine services also can queue for indexing all pages that they are specifically requested to index (which request might be submitted by the page owner or another).
- the list can be expanded by parsing the received pages for hypertext links and URL addresses that identify additional pages, and then loading and analyzing all the pages that are connected to the initial pages in that way. This process can be extended indefinitely. A smaller set of pages might be obtained by only indexing the top level pages or only links to top level pages out to a certain number of links from the originally targeted page.
- the search engine compares the criteria to available information for web pages and sends to the user a report identifying the web pages that meet the criteria.
- the report to the user is transmitted in html source code.
- the search engine finds URLs for the selected web pages and inserts a list of these URLs into a shell form (i.e., an “empty” html source code file).
- the shell form has text and formatting to display title headers, possibly also ad banners and similar information.
- the user can select among the results and point and click or similarly highlight and invoke the html link addressing the page that the search engine considered to meet the user's criteria. This then loads the html source code directly from the remote page that was selected and the browser displays the current contents of the referenced web page according the html source code found there at that time.
- the typical search engine reports more to the search than the URLs of the indexed pages that meet the searcher's selection criteria.
- the URLs themselves which are formatted as hypertext links in the search report, sometimes provide information as to whether or not a search hit is pertinent to the user's desires. For example the domain name associated with the page may identify an owner known to be in a pertinent business, or on the contrary may show that the search result is plainly not relevant to the search.
- the search engine typically also stores and includes in the search report listing one or two of the first lines of the web page that is referenced, which frequently includes a title that may be helpful to show quickly whether the selected page is of interest. The search listing also may show the date at which the web page was last updated or the date that it was indexed.
- the usual success rate in finding a pertinent page or website in one try or only a few tries is actually rather low.
- the success rate varies with the subject matter, but in a typical search the user's search criteria may turn out to be unduly broad and may select so many pages that they cannot all be reviewed, or may be so narrow that much desired content is excluded, either of which can be an unsatisfactory and perhaps frustrating experience. Balancing the needs to include relevant material and to exclude irrelevant material can result in a substantial expenditure of time, much of which is effectively wasted.
- search results could be supplemented to more effectively assist a user running a search to quickly and meaningfully separate the pertinent and irrelevant results.
- a capability will only be useful if it can be accomplished without unduly adding processing time and storage requirements to the steps involved in preparing database information for search and in presenting the results to the user.
- Such parameters can specify font type and sizes, colors, backgrounds, screen pixel resolution and the like.
- a web crawler or automated information review process to find and load or otherwise accept and process html pages.
- previously processed pages are again accessed and the database is periodically updated.
- the abbreviated snapshot representation can be provided in combination with or in lieu of a tabular listing of the associated hypertext link and perhaps also an introductory portion of the text of the html pages.
- a hypertext link can be associated with the graphic snapshot such that the user (searcher) can point and click on the graphic to load and view the associated web page.
- the improved search engine of the invention for managing user search and selection of data files stored at distributed systems coupled at network addresses.
- the search engine is effective to improve searching of hypertext web pages on the Internet.
- the search engine has an associated web crawler operable to address and load successive web pages, and to index text data associated with the successive web pages. In this manner the search engine obtains parameter information such as words appearing in documents, word proximity and other information that can be used to distinguish at least groups of the web pages from one another when conducting a search.
- the web crawler stores the parameter information in a manner that cross references the paramater information with the associated web addresses or URLs of the web pages.
- the search engine accepts user-submitted search criteria and conducts a search or the parameter information to select the associated addresses of web pages that met all or part of the search criteria.
- the results can potentially be ranked, subdivided into categories and similarly handled according to known search engine operation.
- the crawler in conjunction with obtaining the parameter information for at least a subset of the web pages subject to search, the crawler renders a display image of the web page that is being indexed, and processes the image to provide a reduced size graphic image file corresponding to a static visual presentation of each of the indexed web pages.
- This graphic image file preferably is stored in a compressed graphic file format such as GIF, JPG, or a similar file, the file address or URL of which is stored and cross referenced to the criteria in the database that identifies the corresponding web page.
- a search is conducted and results in a hit on a web page, its graphic snapshot is linked to the search results reported to the user.
- acceptance of the user search criteria and reporting of the results are handled by html page exchange communications between the search engine and the user.
- the search engine is accessed by the user and provides a form page having CGI boxes or the like for accepting text and/or other selections from the user.
- the search engine conducts a search which identifies one or more hits that are reported to the user by sending an html search results page.
- the search results page is composed by the search engine as a function of the search results and may contain no hits or a number of hits.
- Each of the hits is identified in the search results by the graphic snapshot, and preferably also by text information that reflects the content of the web page hit.
- the image file is loaded by the user's browser when processing the search results page, which generally occurs after the display of text has been accomplished.
- the search results appearing on the user's browser include links to the web pages that were found to meet the criteria (hits), and also a snapshot graphic image of the way that the web page appeared when rendered at the time of indexing.
- the invention is applicable to a wide range of search systems.
- the invention in addition to use with a web crawler and a text indexed word association database (or instead of automated text indexing), the invention is applicable to produce and associate representative graphic snapshots with websites that reside in a human reviewed directory such as Yahoo, wherein subjective characteristics of the data (a text form of which is sometimes termed “descriptors”) are stored in the database for comparison with user criteria in finding hits.
- characteristics such as an arbitrary business or art classification may categorize the web pages for selection in a manner similar to text string aspects used such as the presence of selected strings, word associations, proximity and the like.
- the invention is also applicable to automated categorizing processes such as used by Northern Light.
- the graphic image file that is produced is not necessarily identical to the appearance of the page when ultimately loaded by the user after a search.
- the rendering is accomplished according to a predetermined display configuration of the crawler when rendered. Nevertheless, the graphic is a useful and very quick means for a user to sift through search results and determine immediately whether or not at least some of the hits bear further investigation.
- FIG. 1 is a schematic block diagram illustrating a first embodiment of the invention.
- FIG. 2 is a block diagram illustrating the elements associated with collecting, processing and organizing a database of information according to the invention, to be used to conduct searches.
- FIG. 3 is a block diagram illustrating operation of the invention in connection with executing and reporting the results of searches.
- the invention is described in this disclosure with primary reference to the preferred application to an Internet coupled search engine in which the data files searched are html pages on the Internet or worldwide web 50 .
- files are accessible for loading and review by other users via browsers and search engines, they generally contain hypertext markup language (html) text, comments or tags, formatting commands, and links addressing other files.
- the data may contain text, media, scripts, programs, etc., and may be addressable at the same network address or a different address.
- the files may contain information that is not displayed when rendering the file, but nevertheless can be used to categorize the content of the files.
- the basic files are addressable using standard uniform resource locator (URL) addresses, containing a high, mid and low level domain name that is resolvable by a domain name server into a numeric Transmission Control Protocol/Internet Protocol (TCP/IP) address by which packets of data are directed from one computer system on the Internet to another.
- URL uniform resource locator
- TCP/IP Transmission Control Protocol/Internet Protocol
- packets as transmitted to the system 52 containing the web page to be subject to search request transmission of an addressed web page (see FIG. 1 ). That system 52 responds by transmitting the contents addressed.
- the packets are reassembled for or by the receiving system.
- the browser or a similar process of the receiving system processes the data, normally but not necessarily for visual display on a local monitor.
- the invention is likewise applicable to other environments such as searching within a company intranet or other group of accessible data stores which have a visual aspect.
- the invention is also applicable to platforms and user interfaces other than PCs and browsers, such as the various Unix processes which are run on PCs or mainframes, etc.
- the invention is applicable to various wireless communication architectures. These environments and platforms are not limited to consumer and business use, and have applications in technical, military and other situations as well.
- FIG. 1 A block diagram showing an improved Internet search engine 20 according to the invention, for managing user search and selection web pages stored at distributed systems 52 coupled at network addresses to the Internet 50 or the like, is shown generally in FIG. 1 .
- FIG. 2 illustrates a succession of method steps and/or programmed operations of the system for building and adding to or updating a database 62 of searchable information.
- FIG. 3 illustrates a method and apparatus for conducting searches by accepting user queries 54 , conducting searches of the database 62 and reporting search results in the form of a composed search report 80 containing visual representations or snapshots 35 that depict a presentation of how the selected pages would have appeared according to a default display configuration at the time they were accessed by the crawler 60 .
- the search engine 20 in the embodiment shown in FIG. 1 has an associated web crawler 60 operable to address and load successive web pages from remote servers 52 on network 50 , and to index or to otherwise accept or generate descriptors that characterize text data associated with the successive web pages that are loaded.
- crawler 60 develops parameter information on the successive web pages that can distinguish at least groups of the web pages from one another, and at times can be used selectively to identify a single web page, provided some encoded aspect of that page is unique among the pages loaded and processed.
- the crawler 60 stores the parameter information and associated addresses of the web pages as a database 62 in a storage medium 64 that is accessible to a search processor 78 that accepts the user criteria 54 and prepares and sends search reports 80 to the query submitting user 30 .
- the search engine portal or processor 78 responds to user submitted search criteria by searching the parameter information in the database 62 and reporting to user 30 at least the associated addresses of data files that met the search criteria when indexed. In particular, search portal/processor 78 reports the URL addresses 82 of web pages meeting the user criteria.
- the web pages are generally maintained on web servers 52 ( FIG. 1 ) that are “remote” from the querying user 30 and from the search engine 78 , but actually could be anywhere that is addressable on the particular network, including on the user's own system.
- the web servers 52 in known manner, store text and graphic data or addresses of graphic data found elsewhere. That information is available upon request and in the case of the Internet and other TCP/IP protocol type networks is transmitted in packet form to any user that requests the web page by directing a request to the web server identifying the TCP/IP address of the web server 52 , the sender's address or identity, and the address of the desired page.
- the same sort of URL addressing is used internally in html pages to address image and other files that may be located at the same web server or elsewhere on the worldwide web, namely by providing a hyperlink that states the network address of the text or other content, as opposed to containing the content itself.
- Such hyperlinks can also be invoked to move around in a given file, for example from one subheading to another.
- the source of a displayed page can be displayed by selecting “View” and “Source” from the toolbar.
- a crawler 60 collects web page data and is generally shown in FIG. 2 .
- Crawler 60 can be operated preliminarily but preferably operates continuously during operation of the other components to collect additional data and/or to update data already collected.
- Crawler 60 has one or more fetching processes 66 , several being shown in FIG. 1 and identified as Agent A (fetch) processes.
- the crawler 60 via its fetching processes 66 determines web pages to load and attempts to load them.
- the crawler 60 may test TCP/IP addresses (known as scanning) or attempt to load pages from particular domain name addresses where servers might be up and running, obtained for example, from a domain name server (not shown).
- the text portion of any data obtained by the fetching processes 66 from a particular URL address is parsed or divided into discrete terms and statements. These terms and statements are compared to predetermined reserved terms and formats that represent URLs, file addresses and the like. When the comparison indicates that a hyperlink to another file or web server has been found (or that a given string so resembles a hyperlink as to be interpreted as such), the found address is added to a list of addresses and an attempt is made in due course to load a file at that address, thus increasing the field of files that have been consulted.
- Agent A fetching processes 66 The general function of the Agent A fetching processes 66 is to obtain the files available from remote web servers 52 and to note the addresses of the files (URLs for the Internet) that when invoked will address and load the file. As a result of communication delays, it is preferred to employ a plurality of concurrently active requests for files so that one file can be processed while waiting to receive another.
- This aspect is represented in the drawing by plural Agent A processes 66 , which obtain the fetched files and store at least part of the fetched files in a buffer memory or queue 92 .
- the data includes html source code, addressed files containing images, audio or other media, which are stored in buffer 92 together with the addresses from which they were obtained.
- the collected information from downloaded files, particularly text files, is processed according to a generally conventional text processing or categorizing technique 68 to build a text or descriptor index in database 62 as shown.
- the database 62 contains an index developed from automatic analysis (generally “indexing”) or human review (categorization) of the text and other data, indexed to the URLs of the pages from which they were obtained.
- index might represent any attributes of the content of the respective web sites, not limited to words in their displayed text. For example terms in hidden meta tags, comments in the source code, strings found in addresses and the like are also potential data points that may be collected.
- index could contain information as to the type of links found in the source, the date of the last update, the country of origin or language, whether the site appears to be academic or commercial, an entry for rating content as adult or “general admission” for keying child protection interests, and so forth.
- the crawler 60 that is operable to receive the web pages and to extract the parameter information from them, generates a file 72 of graphic image data corresponding to an appearance of each of the web pages, which is stored, preferably as a reduced-size and compressed image data file 75 , in association with the database data respecting the page.
- the search engine reports the associated URL addresses 82 of web pages that met the search criteria in a conventional manner, preferably inserting a hypertext link to each identified page into an html page reported to the user, optionally a short description or excerpt, and also inserts into the report page the graphic image snapshot file by inserting into the source of the report page a link to the stored compressed graphic image file 75 .
- the user's browser displays the search results in conventional form, namely by showing a selectable hyperlink to the addresses and optionally a description or excerpt, and displays a snapshot of how the identified page is likely to appear if or when it is loaded by the user's browser, should the user point and click to the link to invoke the URL of the page hit.
- the search portal 78 that performs the search by reference to the database 62 in storage media 64 , reports the search by composing a web page containing the search results, assembling the search report using hypertext markup language.
- the search report contains headers and information identifying the portal and perhaps contains advertising.
- Such a report is conventional in an html source search report. It typically also has a description or excerpt and may be arranged in a pyramid or hierarchy of categories.
- the search engine also inserts the URL address of the graphic file that has been processed by a further process identified in FIG. 2 as Web Agent B 95 , to contain a snapshot reduced/compressed graphic 35 representing the page hit.
- the user's browser when displaying the search results also displays the graphic snapshot image, as shown in FIG. 3 .
- the invention has three main components, shown generally in FIG. 1 .
- these include the crawler processes that fetch files from web pages in the universe of web pages to be subject to search, and the processes that index or catalog the pages and render the fetched files into graphic image files.
- the processes in FIG. 2 can generally be considered the processes that obtain raw data and process it to provide a searchable database and information that may be included in search reports when a web page becomes a hit.
- the crawler processes 66 that are associated with collecting the raw data files, which experience communication delays, are separate from the processes 68 , 95 that process the raw data into a form apt for storage in database 62 in preparation for searching.
- FIG. 3 illustrates the processes 78 that interface with a user who seeks to search the web 50 , including presenting the web page hit information to the user in html form for browser display.
- the search engine includes or is associated with web crawler 60 , which is an engine that conducts web page addressing, loading and analyzing, and stores representative data in a storage device 64 containing a database 62 .
- the stored representative data characterizes the web pages that the crawler loads and that are analyzed for content by process 68 .
- preparation of database 62 allows a search to be conducted more quickly by reference to the processed database information gleaned from the field of possibly-selected files, than would be possible if the search engine attempted to load and analyze the entire universe of files after the user had submitted query 54 ( FIG. 3 ), namely while the user was awaiting search results.
- the process of preparing database 62 includes determining URLs (or perhaps TCP/IP addresses or other addressing strings) for the files to be searched, and then loading and analyzing the files to note the occurrence and juxtaposition of text strings.
- the files are categorized for other aspects, for example by human review and assignment of arbitrary descriptor categories that tend to distinguish files by their content or owner or type, etc.
- the files or webpages consist essentially of ASCII characters stored in a text file that is known to be or is identified as hypertext markup language, often having an “htm” or “html” extension on the file name).
- the ASCII character strings in the web page are searched for combinations of characters that conform to specific code name and character rules whereby they can be interpreted as commands or links or other specific forms of information in html.
- Html is a form of standardized markup language in which various tags are associated with ASCII character strings.
- Many of the character strings and tags used in html webpages concern the appearance of the associated text and the visual aspects that are to be displayed concurrently with the text.
- Such commands can specify a header, a background pattern, color or complete image, set or reset a font type, font size, capitalization or color, change justification, centering and margins, specify lines, a table or frames, call for insertion of a graphic figure in any of several formats, which may be static or animated, and otherwise generally vary the appearance of the page and the text on the page.
- the strings also can address additional files.
- indexing results in a database of information in which each text string found during the analysis of all the files or pages searched is referenced to the URL address where the files or pages can be found.
- indexing can be construed to include other methods for categorizing data files in a manner that allows distinctions to be drawn that are useful for searching, including human reviewer categorization and discrimination for non-text factors such as the revision date, country of origin or the like.
- the database 62 is generated by preparing or obtaining a set of characterizing parameters concerning the fetched files, or their addresses or content or the like.
- Database 62 contains a cross reference between criteria and the identity (normally the URL address) of the file that matches the criteria. Assuming that the criteria concerns a concatenation of terms (e.g, “quick brown fox”), all the URLs of files that contain that string are available by searching for the string. Likewise the URLs of all the files containing the component terms are available (“quick” or “brown” or “fox”), and these terms or phrases can be combined with other terms or arbitrary categorizations to find a page (such as the Quick Brown Fox Hardware Store).
- the indexing and/or categorization particulars can be objective or arbitrary, and wholly or partly driven by human review or by automated means, and can concern any aspect that tends to be unique to individual files or common to subsets of files only.
- the search database includes any collection of information prepared in a manner that enables search criteria to be compared to stored criteria to distinguish files from one another.
- the search criteria involves combinations of categorizations and/or text strings and other factors, chosen by the user in an effort to target the files or pages that have a desired subject or include reference to a particular datum.
- each criterion is not applicable to every page reviewed, and as a result it is possible both to collect files that meet a user's criteria and to eliminate files that do not meet the criteria and thus are irrelevant to the particular search.
- the universe of files and pages can comprise, for example, all the high level pages of registered domain names on the Internet, plus a series of additional lower level pages.
- the lower level pages can include all the pages to which the high level pages are linked by hyperlinks in the content of the high level pages and/or frequently encountered subpage names such as “index” and “home”.
- Various such processes are conventionally practiced using so-called web crawlers that are operated constantly, often during low traffic hours, to find, load and analyze (index) a very large universe of web pages.
- Conventional web crawlers prepare a database that records and can be used by searchers to select (or de-select) web pages primarily on text strings and Boolean combinations of text strings found in the content of the web pages and indexed in the search engine database.
- the web crawler/search engine database also can be arranged to record and permit searchers to select and de-select on the type of media linked to a page, on a window of dates, the language of the web site or page, the location of the registered domain, the depth of a particular web page in the directory structure of the target site, and other aspects.
- the configuration of the user's system also affects the appearance of a web site content when displayed.
- the user can opt to display particular font types, and also can specify font sizes. These configuration choices affect the appearance of a retrieved page even if the page defines specific fonts that are available to the browser.
- the browser may also permit the user to select whether or not to use the background colors of retrieved sites and other features affecting the display.
- the user can opt for different display options such as the number of pixels and the color resolution employed. These aspects also affect the display.
- retrieved web pages appear differently on different user's displays when retrieved. For the most part, differences due to such configuration choices do not grossly affect the appearance of the web site, but they do cause an identically encoded page to appear differently on differently configured systems and/or browsers.
- the search/reporting steps of the browser include accepting search criteria 54 from user 30 , for example using a CGI script technique in which the user enters selections including text strings, literal strings of plural terms, additional encoded aspects such as media types, date windows or limits, countries of origin, etc.
- the user may also select Boolean relationships (AND, OR, NOT, XOR).
- the search portal may require commands or may permit selection using point-and-click steps.
- the search engine compares the search criteria to the pre-prepared database of information gleaned from the web pages it has loaded and analyzed from the field. The results are reported to the user by preparing and formatting an html source reporting page into which hyperlinks are entered that name and point to the addresses of the files that were found to meet the criteria.
- the report includes other information such as the date the page was last updated before it was indexed, and a few lines of introductory text from the page, which provide a hint to assist the user in determining without loading the page whether the page is likely to be relevant to the search.
- the user finds a link that appears to be pertinent, the user selects and engages the hyperlink. This causes the browser to load the html source found at the URL address shown in the search report, and any referenced files and links therein.
- the page may have changed between the time that the indexing was accomplished and may have totally different content than it had when indexed. The page may no longer exist. In those cases, the search fails except to advise the user that the page formerly held information that might have been of interest.
- Text can also be placed in “ALT” descriptions of images and graphics, thereby indexed by the crawler but not seen by the user.
- a particular term can be included one or many times to improve rankings, by one of the foregoing techniques, or by overloading keywords in “META” tags included in web pages and not displayed.
- Another technique is to temporarily post a page to be textually indexed by the crawler/search engine and then to replace its content after it has been indexed, or similarly, meta-refreshing the web page so as to redirect the user to another page address.
- the user can visually distinguish pages having undesired content and not waste time on them. Search engine corruption using the aforementioned techniques to provide misleading text is averted due to the visual nature of the present invention.
- a system of the type that indexes or categorizes information on web pages for searching is improved by encoding and providing in the search report 80 a standardized graphic representation 35 of the appearance and rendering of each page at the time that the page is indexed.
- the graphic representation 35 preferably is in the form of a compressed image of the page, described herein as a snapshot, stored in a standard compressed file graphics format at a location accessible to the search portal process 78 .
- the snapshot is acquired when the page is initially loaded by the crawler 60 for indexing ( FIG. 2 ).
- the snapshot is rendered, converted to the compressed format and stored.
- the individual snapshots transmitted to the user are the individual snapshots, which have been stored locally to the search portal processor 78 , in association with the index/categorization database.
- the snapshots 35 of the hit page (which may be one of a number of hits that are reported to user 30 ) is shown when providing the search report.
- the snapshots 35 can be contained in formatted image files (e.g., GIF, JPG, etc.).
- the snapshot image files, or URL addresses pointing to the image files preferably are stored in the database 62 that also contains the URL addresses of the indexed pages.
- the search engine 78 inserts a link 82 aiming to the snapshot image file 35 into the html search results page 80 .
- the search results appear on the users browser 84 as a link to selected pages with an associated snapshot of the page when indexed, as shown in FIG. 3 .
- One problem with acquisition of the snapshots is due the very large number of websites that must be physically rendered, namely every website that is indexed and is available in the universe of websites subject to search.
- the website content including any referenced image files, must be downloaded by the crawler Agent A process(es) 66 and rendered by the rendering Agent B process(es) at acceptable speeds, and preferably also reduced to obtain reasonably sized image files 35 .
- the image files must be accessibly stored and downloaded from the search engine 78 to the searcher (specifically the user's browser 84 ) at acceptable speeds as well.
- the invention applies particular technology to solve these and other problems.
- Major search engine portals each have a usually-proprietary “robot” or automated process that crawls the web as described above.
- a robot or crawler that accepts or finds website URL addresses, accesses websites by TCP/IP addressing and loads their source code.
- the crawler robot automatically parses the text of a website, namely dividing the strings found in the source code into units separated by delimiters such as spaces or punctuation.
- the strings and the succession of strings are compared to stored parameters whereby certain strings are construed as links or formatting commands, which is noted accordingly.
- the occurrence and proximity of these strings and the free content strings that are to appear as text in the web page when displayed on a browser are all noted and stored in a database where this information is cross referenced to the URL address of the website from which the page was loaded.
- the website text can be analyzed and indexed at an extremely high rate of speed because the page is treated only as a succession of text strings. No processing time is spent to load and process or otherwise handle any embedded or referenced graphics, media, scripting, Java, or animations. Such files are not helpful for traditional indexing and thus are not requested.
- the html tags that might be used to find and load files for non-text content may be textually parsed, but their associated data files are never requested and not retrieved by the traditional text crawlers employed by the major search engines.
- no time is devoted by the crawler for data transfer that might be needed to request and receive packets containing the graphic or other media files.
- the load on the crawler is minimized because the portion of the website that is loaded and processed, namely the text portion, represents little “weight” in communications bandwidth requirements, processing time and the like for most web pages. Without the need to download and process large graphic and media files, simple text indexing in the traditional sense by conventional crawlers is very efficient, simple, and fast.
- a web page will contain one single text file, but in contrast may contain dozens of graphic and media files.
- Traditional text crawling by the major search engines require that only the one single text file be transmitted and parsed.
- full graphic rendering employed by the current invention requires that each and every graphic, image, and media file be transmitted and subsequently rendered into a full visual depiction of a web page.
- the text data portion of a web page is most commonly five to ten Kbytes in length and is received in less than a second on a typical network connection.
- the text file is normally the first file sent from the originating web server. Image files and script or other code, if requested, follow afterwards.
- the robotic processes of requesting a text file, retrieving packets and reassembling the text file, parsing the text file by finding terms within delimiters, and indexing its contents, can be accomplished under normal circumstances in 0.5 to 1.5 seconds. Assuming a one second average processing time, one computer processor operating, for example, 25 text processing web crawler robots (which may be conservative), can obtain and index the text of 25 web pages per second every second. Operating continuously, such a crawler could process over 15 million web pages per week.
- a website server is usually prompt in sending short files, such as the requested text of a particular page, and short file transmissions are more frequently successful than longer ones due to the additional packet handling for reassembling the file, and the increased possibility of transmission errors requiring retransmission.
- the browser receiving and processing the graphic file seems to pause or to stick on presenting a particular graphic section during the resulting delay.
- the transmission may pause at any point, even on the last packet of a number of successively transmitted files.
- the receiving browser or other processor cannot complete the total and full rendering of the web page, for display or otherwise, until the delay elapses.
- the receiving computer simply waits before completing the display of the page.
- the browser or page rendering robot normally requires on the average 30-45 seconds per page to receive and process a web page into a graphically visual layout (an approximation that incorporates a variety of factors including changes in bandwidth, server lag, and lost packets which can result in web pages being delayed).
- the graphic layout of the page usually comprises a series of image files.
- Each file consists of or is unpacked into an array of digital data words representing the saturation, luminance and hue or the respective RGB levels of each pixel in an X-Y field corresponding to the display screen area.
- the image file is loaded into a series of memory locations accessed by the display driver to drive the monitor display, either in the processor random access memory or in the memory of a video display driver card (or both).
- the process of rendering a page into a visually graphic layout usually requires devoting a full display memory field to this function, and particular aspects of processors are often devoted to handling a limited number of display images.
- At least two independent types of intelligent web agents are cooperatively operated to handle different aspects of the job of retrieving, rendering, and processing websites, in a manner that makes it possible to produce graphic data in the form of a compressed or reduced graphic file representing the appearance of a rendered website, and to do so at an acceptable rate.
- the first type of intelligent web agent (now to be referred to as “Web Agent A”) requests, retrieves, and downloads each and every file associated with a particular website, including but not limited to the source code text file, graphic files (e.g., GIFs, JPGs and others), script files, Java executeable files, Flash technology files, Shockwave files, animations, and so forth.
- the rendering process by Web Agent B comprises processing the text and html tagging data to prepare a visual representation. All the files necessary to render the image have preferably been obtained by Web Agent A before then, and such files are stored in the buffer.
- Web Agent B produces a full visual representation, such as a bitmap file containing a pixel data array, which if coupled to a display driver could be used to display the web page layout on the video monitor at full screen dimensions. In short, Web Agent B prepares a visual image as might be provided by a browser.
- the visual display of the web page is then compressed by Web Agent B or a process associated with it, to a predetermined and preferably small image size, for example a 2 in. ⁇ 2 in. image on a 17 inch diagonally measured display screen.
- This process may involve sampling or local area averaging techniques as known in the art.
- the reduced size bitmap image then is digitally compressed and/or encoded to minimize storage requirements and to permit quick transmission over an ASCII-only data channel.
- the reduced size bitmap image can be converted into a JPG, GIF or similar format for an image file suitable for web transmission. That image file, which represents the rendered appearance of the associated web page at a particular point in time, is stored in a mass memory accessible to the search engine.
- the mass memory can be in one or more hard drives, ram caches, writable CD ROMs or other media that is useful as a high capacity RAM.
- the mass memory can be a peripheral on the search system or can be accessible to the search engine, for example using communications over a local area network, provided that the image files are very quickly recallable using a minimum of data communications and/or communications that are direct rather than over the web.
- the mass memory can have a subdirectory naming system and file naming system based on the network addresses or URLs of the web pages from which the graphic files were generated, or alternatively the files can be arbitrarily named or stored and can be found using a cross reference table in the search engine whereby the address or URL of the web page and its associated image file are cross referenced.
- the search engine memory also comprises text indexing data or human categorization directory data (or both), that is obtained in a conventional web crawler manner and includes an association between the text data found at each web page and the web address or URL of the originating web page.
- the search engine can call up the graphic file representing its appearance when rendered at some time in the past.
- the search engine can determine the matching web pages, report their URLs and provide a graphic file showing a miniature window version of how they would have appeared if loaded by a browser at substantially the time when their data was loaded and indexed.
- Web Agent B preferably has additional functions, including keeping status information such as storing log files containing addresses and/or linked file names that have been attempted and obtained, optionally including a queue of files that presented problems when first tried and should be re-tried or after a time will be rendered with missing-graphic gaps, web addresses that have been completely rendered, etc.
- the logs and status indicators are sufficient to permit an operator to monitor operation by reference to by readouts or by displaying stored data.
- Web Agent B also preferably generates error messages and/or alarms in the event of any crucial errors. Status readouts available can include rudimentary data such as the current URL being processed, the rendering state of the current URL, the number of URLs processed since inception or last clear, any error messages and so on.
- the search engine can comprise one or a number of processors and the processors can be in direct communication or linked on a local network or other arrangments, the key being quick access to the stored database of data representing the universe of web pages that have been processed and therefore are searchable.
- the search engine accepts user search criteria in a conventional way, such as using CGI form boxes to enter text strings into an associated search engine entry html page that is addressable by a browser.
- the search engine permits selections to be made according to at least one search criterion and preferably accepts a variety of different criteria types and combinations. These aspects of the search engine can be of the type conventionally used by current search engines such as Hotbot, Yahoo, Alta Vista, Northern Light, etc.
- the search engine is operable to select web page hits as a function of user supplied search criteria and to determine the URL addresses of web pages (hits) that wholly or partly meet the criteria.
- the search engine may store and retrieve a brief exemplary text string such as the initial few lines of text in the web page hit.
- the search engine reports search results to the user that entered the search criteria, by composing an html source page and transmitting it to the user.
- This html report page may identify no hits or a long list of hits, depending on the search results.
- the search engine typically shows the search criteria used, and displays indicia summarizing or similarly identifying each web page hit.
- the search report can identify hits by the URL of the originating web page.
- Preferably a short text selection such as the first few lines of text is shown.
- the html coded report page prepared by the search engine includes an associated hyperlink to the URL of each hit.
- the user reviews the URLs, sample text or other information and activates the hyperlink of a selected web page identified in the results, thereby loading the web page presently found at the address of the originating page when processed by the crawler robots.
- the composed search report page prepared by the search engine includes but is not limited to the URL of each web page, the title of each web page, a description of each web page, and a graphic depiction of each web page.
- the user's browser immediately loads the source code, which contains the text portion of the search report.
- the user's browser encounters the links to the image files that were included by the search engine when composing the search report page and obtains the image file.
- the report page composed by the search engine places the graphic for the web page hits immediately adjacent to the associated text and hyperlink.
- the graphic image was rendered under certain assumptions as to the display configuration and represents a snapshot of the web page frozen in time.
- the snapshot is at least an approximation of how the web page will appear if the link is activated and the page is loaded by the user (i.e., if the page is unchanged and the user's display configuration is equal to the default configuration assumed by the Agent B of the invention). Unless the page has been substantially changed by its owner, the graphic depiction will substantially assist the user in sifting the pages that are definitely interesting versus possibly interesting, neutral, unlikely to contain pertinent material or definitely irrelevant.
- Agent A for fetching
- Agent B for handling image content
- Agent A is subject to communication delays involved in requesting, receiving and storing the needed files from the internet, which can delay a single robot, but in fact is ameliorated by running multiple copies of Agent A in the background.
- Agent B has more data to process, but due to the preloading by numerous Agents A in the background can process the data quickly from local copies.
- Agent B is free to monopolize the display in the foreground while multiple Agents A in the background acquire necessary files from the internet and feed them into a temporary data buffer.
- web agents of type A operate in conduction with each web agent of type B.
- a plurality of web agents of type A continuously fetch and feed into a buffer or queue all web page files of targeted web pages, including their source code and their graphic images, such as JPG, GIF, Java, Flash, etc., all being stored locally.
- One or more web agents of type B preferably one for a number of Agents A (e.g., 32) continuously processes and removes files from this buffer to produce and render one web page snapshot image after another. Concurrently with this process, the text portion of the web page data is indexed or categorized.
- the ratio of Agents A to Agents B can be determined from experience such that the contents of the buffer or queue remain substantially stable for the particular search engine. Alternatively, the ratio can be changed on the fly so as to keep Agent B constantly working and to keep the size of the buffer or queue stable. If the queue continues to grow, the ratio of Agents A to Agents B can be reduced, thereby committing more of the available CPU time to Agent B, which should cause the buffer to shrink.
- the buffer should not be allowed to shrink indefinitely, or Agent B will become idle or will lose efficiency or even stall, waiting for complete sets of web files to become available.
- an optimal buffer size is assigned, such as some hundreds of MBytes. Additionally, this buffer is maintained relatively static by the deletion of data after it is used by Agent B.
- Agent A processes can be added until a substantial portion of the available communications time is filled with active Agent A messages. If the buffer grows continuously, Agent A processes are reduced in number relative to the number of Agents B, and vice versa. Inasmuch as the optimum ratio is in part due to communication delays due to web congestion, the ratio of Agents A to B can be varied throughout a processing day.
- Web Agent B continuously renders and processes web pages one after another according to a specified queue.
- Web Agent B does not suffer from the limitations and overhead of requesting and transferring files over the internet because these problems are solved by the team of web agents of type A, for example thirty-two of which may be busy addressing and loading files from different sources.
- a single web Agent B was employed in a computer engaged as described above.
- Conventional browser and display driver routines were used to render bitmap display files from html pages that had been revised such that all included image reference links pointed to graphics files that had been previously downloaded by one of the plurality of operating Agent A processes and stored in the queue or buffer, namely on the system hard drive.
- An image conversion utility then converted the display bitmaps into a GIF image files under file names referenced to the corresponding URL of the originating web page. This arrangement proved to be an efficient and fast method to obtain snapshot renderings of web pages.
- Web Agent B in such an arrangement controls and manipulates all processing and system resources for graphical display but is not held back by the delay of retrieving and storing of the necessary files, which is collectively performed by all the Web Agents of type A, running as concurrent processes in the background and thus not requiring many of the system resources, including the display buffers and drivers.
- the system proved efficiently cabable of rendering at least one web page per second, and if run continuously would render 86,400 pages per day, 604,800 per week. This may seem like an adequate rate, but assuming a desired universe of 100 million pages, a single computer system crawling at that rate would still need approximately 3 years to complete a crawling cycle. During that time, the content of most of the web pages would have been changed.
- the multiple Agent A processes operating concurrently are such that the usual reason for waste of communications time, namely waiting for a response from a remote web page server, is minimized because delay experienced by one of the Agent A processes is used by the other Agent A processes that are operating at the same time.
- the invention can perform on any bandwidth connection, including 28 Kbps. Of course a high bandwidth connection is preferred, such as one or more T 1 or T 3 connections (if not even higher).
- the Unix platform is alternatively useful according to the invention due to its capability of handling multiple simultaneous processes.
- the respective software robots can run on the Unix platform as applications programmed, for example, in C, C++, Perl or one of the other languages.
- numerous computers are employed simultaneously, each having its own connection to the internet and each employing its own embodiment of the current invention.
- the computers can reside on a network and feed off of and simultaneously contribute to a common database maintained by one of the computers on the network.
- a cycle of processing preferably is completed and the index and snapshot files that result are loaded into a database or are used to update a database, maintained on the server that accepts user search criteria and composes and sends to the user the search results.
- the search system of the invention preferably permits anyone to suggest a web page to be added to the universe of searchable pages.
- the suggested web page is added to the compilation, and the search engine's robots crawl the web by loading the suggested page, noting and loading the pages linked to the suggested page and continuing on to the pages that are linked to the linked pages, etc. Duplicates are removed. URLs that have been recently visited can be flagged for deferred reload, or removed.
- a “human surfer” or web page reviewer may be more dependable than a robotic agent in categorizing the content of web sites (e.g., “The Electric Factory” is the identifier of a concert promoter and supplier of tickets to entertainment events). Both methods can be utilized to compile a database of websites.
- a team of human surfers can be employed for the task, each visiting successive websites and making determinations, for example, as to an appropriate title, description, category or the like.
- the processing is accomplished in a network of programmed processors that are in a data communication with one another and each of which has a TCP/IP communication link to the web.
- the database containing the universe of crawled or to-be-crawled target web sites, which may number in the millions, can be stored in a controlling processor or can be part of a shared data store used to allocate individual URLs to client computers on the search system network, such as by permitting Web Agent A to obtain the next URL from the list and to flag the URL as in use. It is not strictly necessary to use the network paradigm.
- each Web Agent A or each client computer running multiple Web Agents of type A can contain its own database with a subset of the URLs of the universe, and the databases of a number of robots or clients can be synchronized periodically to eliminate duplicates, flag URLs after they have been crawled, and similarly updated.
- the database serves out a URL to the next Web Agent A in the queue and moves an index or “pointer” to refer to the next URL to be served out.
- a framed web page can accept and display any of a number of other web pages as an inset frame. This complicates matters in that the end-of-file that actually concerns only part of the framed page might erroneously trigger the Web Agent to move on to the next website and to process the frame but not the framed content.
- Frames also present a problem for the crawler robot regarding embedded html links to other web pages.
- the owner of a frames web page can include html links to web pages of others. If a surfing browser attempts to load the linked page by selecting (clicking on) the link on the frames web page, the browser will load the linked page but it will be within the frame of the first web page owner. The browser is not linked independently in that case and instead is linked through the frames page.
- the html target address that appears in the browser toolbar and is recorded in the browser's history list is not a link to the selected site. Instead it is a link to the frames page, with a modifier that identifies the selected site. When that target address is invoked, the frame is loaded and the linked web page is inserted into the frame.
- Web Agent A In queuing embedded links found on pages for processing, Web Agent A distinguishes framed links from direct links. When processing a framed page, preferably, the crawler invokes the framed page's internal links to find and queue additional links, but does not treat every framed link as a new web page. Insofar as Web Agent A encounters websites with frames, it processes the data local to that web site and checks for the presence of a website with frames. When a frame page is detected, the Web Agent A checks for a download complete message (end-of-file) for every framed element and processes the text and graphics of the frame and the contents both.
- end-of-file end-of-file
- Either Web Agent A or Web Agent B can process the target site to link to or to present the static display marker for such files.
- an icon or character can indicate when a link to an audio file is detected.
- the icon also can be chosen as a function of the file extension to indicate the type of audio file found, such as WAV, MID, MP3, etc.
- pop up windows are ignored or suppressed.
- Dialog boxes unlike pop up windows, are somewhat more complex and may obstruct the display of background page features when displayed.
- a dialog or data-entry CGI box may suspend the processing of a page until the dialog box is handled.
- a dialog box is detected and triggers running of a “cancel” routine in response to a dialog box. Assuming that the site is operating password-control or a similar process, that process is discontinued for failure to enter the password or the like, but Web Agent A can continue on and may obtain additional graphic file data or text after the dialog box or similar prompt has been passed.
- Animated GIFs and other changing features can also be identified by an icon indicating the presence of that feature. Preferably these animated features are selectively processed to provide a static image.
- Animated GIFs and some other technologies such as Macromedia Flash, provide an action sequence in the form of a plurality of images that are displayed in quick succession, normally in a loop. It is a problem with animations, especially those pertaining to Macromedia Flash Technology to select which frame will be captured or selected as representative of the animation.
- Animated GIFs begin with a graphic and the subsequent “frames” may be limited only to those pixels that have changed color from one frame to the next. Flash Technology usually begins with a blank screen or blank square.
- the Web Agent B can employ a timer to wait a predetermined time before capturing the rendered image in a file of the type that starts as a blank or fades in. It may be a matter of luck what in particular will be present at the moment captured in the changing portion of the display.
- An alternative is to generate a static image as a sum or average of two or more changing frames, which may produce a smeared static image.
- Another alternative is to disable the Flash plug in by a suitable message to the target site when loading the page.
- Disabling the Flash plug may eliminate any graphic data, namely if the website operators did not provide a static HTML page as an alternative to be presented for users who are not outfitted for Flash. Often, a user without Flash is presented with a blank screen with a tiny caption at the bottom reading “If you do not have Flash, click here.” A rendering and subsequent snapshot of a screen similar to this could be misleading to the user if viewed within the search results of a search engine, so a timed capture is preferred.
- Flash it is an aspect of the current invention to provide an icon or similar indication within the search results as to whether or not a particular website contains Flash Technology. This alleviates possible inconsistencies in processing and rendering a Flash movie, and subsequent interpretation by the user of a search engine who may be viewing the snapshots. Moreover, for Flash and similar technologies that are optional for users, adding an indication of their presence benefits users of the search results. Specifically in the case of Flash, a user who has loaded the Flash plugin or otherwise has the capability to process the content will prefer to access pages that contain Flash content if other factors are equal. Users with browsers incapable of processing Flash technology might be forewarned that their browser may have difficultly rendering that particular website, or at the least would be neutral about that aspect of the web site. The use of Flash, RealAudio and other “value added” technologies is often an indication that a particular website has superior content.
- the presence of Flash content is detected.
- a static page is captured according to one or more of the foregoing alternatives, preferably by disabling the Flash Plug-in.
- a conventional static graphic is displayed in the snapshot image, and adjacent to the static graphic an icon is inserted to show that the site is a Flash site.
- the same technique can be used to identify other dynamic displays, such as Shockwave Movies and the like, preferably using distinct icons for each type.
- each computer employed by the search engine system has one database, a plurality of Web Agent A's, and a single Web Agent of type B. While the Web Agent A's are occupied with downloading necessary text data and graphic files in the background, the single Web Agent B is busy in the foreground rendering pages and performing coordinate based screen captures. Most commonly, screen captures are performed at a bit depth or resolution of 24-bits, and thus comprising 16.7 million possible colors in the captured image. To minimize data overhead and to maximize efficiency, a coordinate based system is utilized to execute the desired image capture.
- Such facilities may include, as available, display processing hardware, software, firmware, coprocessors, memory caches and possibly peripherals such as display driver cards, which might normally be used to facilitate fast updates to a display during the foreground operation of a program.
- the system as described can be configured to operate using a plurality of independent computers that are in data communication (e.g., on a common network, or having access to a particular memory store either concurrently or by virtue of preparing a mass memory media such as CD ROMs containing the storage media database, using one or more computers, and then processing the database for searches using one or more additional computers.
- one computer or a subset of a group of computers
- This large number of Web Agents of type A or processes incorporating similar capabilities deposits downloaded files into a datastore, such as a hard drive, removable drive, or the like. The deposited data can then be transferred via network or on movable media to a different computer running Web Agent of type B.
- Web Agent B processes the data to provide reduced/compressed web page images or snapshots on graphic data files.
- This second computer running Web Agent B accesses the datastore to render and process websites according to a specified queue.
- Separation of the index preparation function in which the storage database is processed to ready it for searching, and the searching and reporting functions in which user queries are accepted, the storage database is searched and a report is composed and reported, can also be separated onto additional computers that each serve particular functions. In this way, operating together and preferably including allocation of additional resources at any processing and communication bottlenecks, the system can obtain data, prepare the data for searching by preprocessing the data, including producing graphic image files, and conduct and report searches via interaction with remote users.
- both types of Web Agents run on the same machine.
- one of the Web Agents of type A downloads a web page, it stores all elements of the page, both text and graphics and including files that may be linked to each subject page but stored at a a different server address, and saves the URL address and the associated file names.
- the URL is added to Web Agent B's input queue. All of the Web Agents of type A perform this same process, namely attempting downloads and when a download is complete, placing that URL in the Web Agent B's queue. In this way the Web Agent B normally cannot outpace the Web Agent As, even though the latter are occupied to some extent with waiting for transmitted data to be sent by a remote website server.
- the Web Agent B undertakes processing after all the files necessary to complete the processing have been downloaded and accessibly stored.
- Web Agent A or Web Agent B (or another process such as a process that parses the received source code for text indexing) scans through the source code and thereby determines the files that are needed for processing, namely the files or addresses to which hyperlinks are found in the source code.
- the web page can wait to be queued for Web Agent B until Web Agent A has successfully loaded and stored all the files.
- a list of the associated files can be prepared by Web Agent B or by another process with access to the source code, and Web Agent B can check the list before attempting to process the data for the web page.
- the processing capacity of Web Agent B is substantially devoted to processing pages that are complete when their processing commences.
- Web Agent A or another process, can be arranged to continue to attempt to load any of the necessary files that has not been loaded.
- Web Agent B can undertake a new communication on the web in an effort to retrieve the missing file, or can queue one of the Web Agents of type A, or another process, to obtain the file or to reload the missing file(s) or perhaps the entire web page and associated files. Reloading the entire web page deals with the possibility that a file that is found to be missing or unavailable may no longer be linked in the web page source code, and thus is unnecessary. Discontinuance of the link might also be the reason why the file has not been found (i.e., it was removed and deleted).
- Web Agent B can render and capture graphic image files or snapshots at an improved rate.
- Agent B and several Agent A processes were operative on one computer to accumulate stored files locally and to queue Agent B, Agent B was found able to produce graphic snapshot files at a rate of about one web page per second. This is much faster than downloading and rendering one page at a time as such would be the case with a normal browser, wherein transmission waits make the typical rate one web page per 45 seconds.
- the rendered image file is captured from the display buffer memory of the operating system, and then is resized, processed to increase image quality, and compressed. It is then stored on disk in a standard format under a file name associated with the URL of the originating web page.
- the search engine database Upon completion of a full crawl, rendering of each and every desired web site, and full data storage of the resulting graphic snapshots, the search engine database is ready to accept user queries.
- the user presents combinations of text string expressions in a known manner.
- search criteria known in other search engine applications (e.g., HotBot, Alta Vista, Yahoo, etc.)
- the criteria are compared to the indexed text information.
- the search engine selects and prepares a list of the web page hits discovered by comparing the search criteria to the contents of the indexed database.
- a report listing is prepared by generating a reporting web page in html source code, which is then sent to the user.
- the reporting web page includes a list of hits wherein each entry on the list comprises an html link to the URL from which the associated web page was downloaded.
- the entries Preferably, and as already done with most search engines, the entries also include at least a line or two of text from the web page, such as the first three lines.
- the browser When the user reviews the search report using a browser, the browser inserts the graphic snapshot image adjacent to the listing of the URL link to the subject web page.
- the user can determine whether a page entry in the search results is of interest, not only from the text information included with the URL link such as a description and title, but also from a small size presentation of what the web page looked like when it was indexed.
- the user can click on the hypertext link to the URL of the web page, whereupon the user's browser loads the web page directly from its original web page server.
- the snapshot image preferably is associated with the hypertext link redundantly, so that the user can click either on the hypertext link or on the snapshot image and in either case will be linked by the URL to the originating web page.
- Web Agent B upon complete rendering of a web page, programmatically manipulates the aforementioned algorithms and subsequently ensures the proper storage of the resulting graphic snapshot onto disk. Additionally, Web Agent B performs a test to determine whether the graphic snapshot is of a higher quality in GIF format or JPG format. It should be noted that new algorithms or other existing algorithms may be operable and may be preferable in other operating situations.
- An element of the snapshot software system is to programmatically control software to perform these actions, for example by manipulation of subroutines from commercial software. This can be accomplished using C++ programming to access certain files and processes normally regarded as internal to the computer operating system.
- the memory locations containing the bitmap image intended for the browser display, generated by the operating system e.g., browser, display drivers, etc.
- the operating system e.g., browser, display drivers, etc.
- a bitmap-to-GIF or bitmap-to-JPG conversion is effected on the contents of the display buffer stored in RAM. Exactly which conversion is determined by Web Agent B.
- the original bitmap screen capture into a snapshot Upon the completion of processing the original bitmap screen capture into a snapshot, all the raw data files used to render the image now captured are deleted to prevent the data store from overfilling. That is, the original html source code can be deleted together with the graphic files addressed in the source code and downloaded for inclusion in the rendering, leaving in storage only the representation of the web site in the database by its URL address, its text indexing and/or categorization and the ultimate graphic snapshot in an image file cross referenced to the stored address.
- the bit size of the graphic snapshot file is approximately 1/200 the size of the original data.
- a popular search engine or website portal may receive numerous server requests per day.
- the search engine opening page may have numerous included text, graphic and interactive elements, each element requiring a communication request for transmission of a data file.
- Top search engines are visited by millions of users every day, and each search can generate numerous “hits”. Some of the search portals personalize the presentation to users. If a search engine is visited by millions of users per day, it has to serve multiple millions of operations and data transfers. According to the present invention, the search engine can report improved information without the corresponding overhead.
- Search engine visitors are very impatient and tests show that they are not willing to wait very long for results to be reported.
- the invention expedites the search and search reporting process while improving the content of the results, and encourages users to remain loyal to their preferred search engine.
- the snapshot rendering feature is preferably enabled and disabled by user option, which can be a point upon which the user personalizes his/her access to the web portal containing the search engine of the invention.
- user option can be a point upon which the user personalizes his/her access to the web portal containing the search engine of the invention.
- the user can be assigned a code that is stored in a cookie that is sent to or made available to the portal, or a cookie containing bit flags in which the user can set and unset options such as snapshot reporting.
- the snapshot servers access and deliver graphic snapshots from storage, to the network address of a user to whom a search is being reported.
- the snapshot servers can conduct packet data transmission, serve requests for re-send, etc.
- the snapshot servers remove the overhead of reporting graphic files from the searching processes, and thus ensure that search reports are as quick as possible.
- the search process passes the graphic file names and user network address to the snapshot servers.
- the snapshot servers transmit the graphic snapshot files corresponding to each search report following shortly after the search report text.
- the snapshot servers employ a RAM cache for storage of some or all of the snapshot images to be reported to users. This contributes further to the reporting speed because it is not necessary to await the addressing and loading of image snapshot files from the system hard disk and the hard disk does not become an undue bottleneck.
- the library of quick access snapshot graphics can be copied from a hard drive into the RAM cache.
- the cached files can be all of the snapshot graphic files or only those found after experience to be most frequently addressed.
- the snapshot servers preferably share or employ a large cache, for example at least 1 Gigabyte and preferably 10 Gigs or greater.
- the snapshot servers preferably contain a program or process that counts or calculates the two million most requested snapshots. This can be updated on a weekly basis.
- a predetermined number of those found to be most requested such as two million, are kept directly in the memory cache (hence a cache size of 10 Gigs, or approximately 5 KBytes per image).
- the status of a given page as being among the predetermined number (e.g., two million) that are most often requested or at least most often reported in searches, can be indicated in the graphic results, for example by adding a frame to the snapshot that is reported by transmitting an additional frame graphic.
- the textual portion of search results always is sent and caused to appear first, prior to the snapshots corresponding to those results.
- the text portion appears first. If a user so desires, he can abort the transmission of the results based on review of the initially received portion. This is accomplished through programming within the snapshot server system that queues the text portion of the search results to be “released” or transmitted first, preferably even before addressing (or perhaps even checking for the presence on the corresponding snapshots.
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Operations Research (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
A search engine manages the indexing of web page contents and accepts user selection criteria to find and report hits that meet the search criteria. The inventive search engine has an associated crawler function wherein display images of the web pages are rendered and stored as snapshots, preferably when the pages are indexed. The search engine reports search results by composing an html page with links to the corresponding page hits and containing snapshot reduced size graphic images showing the web pages as they appeared when fetched and stored as snapshots.
Description
More than one reissue application has been filed for the reissue of U.S. Pat. No. 6,643,641. The reissue applications are U.S. patent application Ser. No. 11/266,750, filed Nov. 4, 2005 (the parent reissue application) and U.S. patent application Ser. No. 11/513,423 filed Aug. 31, 2006 (the present continuation reissue application of the parent reissue application), all of which are reissues of U.S. Pat. No. 6,643,641.
1. Field of the Invention
The invention concerns methods and apparatus for representing data file contents for searching the data files and reporting selected data file addresses, especially hypertext markup language files accessed using an Internet search engine (i.e., Web pages). One process develops a database representing the text content of data files on a network. Another process renders graphic representations of the files according to a default configuration and stores a compressed graphic file for each. A further process selects file hits according to user criteria and reports their addresses with associated presentation of the stored graphic file.
2. Prior Art
A search engine is a useful facility for browsing the Internet or World Wide Web. Popular browsers such as Microsoft Internet Explorer and Netscape Navigator display visual outputs using hypertext markup language or “html.” An enormous variety of information is stored in html format in subscriber homepages and the like on the Web, and much of the information is accessible on the Web by simply pointing one's browser to the associated page or file. Html files typically contain, for example, text and numeric information, typographical symbols, information defining formatting particulars by which the text is to appear on a display of the file, and uniform resource location references (URLs), which are hypertext links that address other files. Some of the URLs address or point to other hypertext pages that are linked to a displayed page. The user can highlight and select a URL by pointing and clicking using his/her mouse, whereupon the browser loads and displays the identified page. Alternatively, the link may be such that this point-and-click method causes the browser to jump to a display of a different position in the file, or to perform an identified action such as downloading and playing an audio or video file, or may cause the browser to alter its display of the present data, such as inserting or enlarging a display of a graphic file. The link may also cause the browser to invoke an applications program or a process, etc.
The html files which are addressed typically contain certain formatting information. All users who download the html file obtain the identical file and formatting. However, the display and processing of the files is not necessarily the same from one user's browser to another. The html page does not contain a fixed graphic data display. The html page contains text, addresses and encoding information which are processed by the browser and the system operating the browser, to prepare and present a graphic data display.
Browsers from different software suppliers are not identical and operate somewhat differently. The same browser program can be set up by user options for display of data in selected ways, including for example choices of font size and font type. There are also alternative choices for applications programs that may be run within the browser (often called plug-ins) or which are invoked when a file of a particular type is selected.
Using font size as an example, the operating system (e.g., Microsoft Windows) and the display may be configured to employ a certain X-Y pixel size and color display resolution. In the browser, the user may have selected one of several available font sizes, which in combination with the X-Y pixel size of the display field determines the vertical and horizontal size of each character. These choices affect pagination and the layout of text within text subdivisions such as paragraphs or tables. The browser may allow the user to select a default character alphabet. The browser may also allow the user to select how and whether background and foreground colors are displayed, or whether colors are even used in certain situations, such as to distinguish links from other text or to highlight a link when selected by the cursor or mouse.
The typical html source file contains text and may include or contain addresses identifying static or dynamic files and information, but the source files are usually not limited to text. The source files contain header, footer, paragraph and section markers, font and color changes which may distinguish sections, markers indicating text strings to be interpreted as html links (URL addresses that are delineated as such), and other formatting and instructions. These and other markers, which include hidden text tags and textual start/stop markers, are not themselves displayed but instead are used to carry undisplayed information or as specifications for display of the remaining text according to preset rules and configuration choices in the browser and the operating system.
Users often refer to the display of a particular web page as “going to” the web page. In fact, “going to” the web page is a misnomer. The process actually involves sending a message to a remote server or user station on the web that requests transmission of the html source code stored there. Upon receipt the source code is processed locally by the browser so as to produce data representing a graphic display. The graphic display data is stored in a memory buffer in the system RAM or in an associated display driver card from which the luminance, saturation and hue of each pixel in the display are determined. After “going to” a web page, the browser may store a copy of the source code locally so that using the “Back” function reloads the page without the need to wait for another exchange of messages over the Web.
Users may know the URL for a web site they wish to load, but also may need to find files with selected content without knowing the corresponding URL. For this purpose the user can “search the Web” using a search engine. Early search engines did live web page searches and came to be known as “web crawlers.” The number of searchable pages has multiplied, however, and it would be an immensely large job to attempt to address, load and search all the possible URLs that might identify a web page today. This web crawling method is now impractical for on-demand searching.
Search engines now operating do not search web pages on demand. Instead the search engine operators use various means to build a limited database reflecting the contents of a number of web pages. The users' search criteria are applied to the database to identify the addresses of web pages that meet the search criteria, at least from a subset of all existing web pages. Web page content can be changed. The search is current up to the most recent time at which the search engine database was updated to reflect the latest content of the web pages subject to search.
The web pages to be reflected in the database are indexed to build a record of the terms that appear in each web page. Search engines vary but typically the index database reflects at least the presence of single words to enable selection by Boolean combinations. At least some proximity relationships and/or the presence of exact phrases can be made searchable. The indexing can include a selection of field information, such as revision dates, country of domain and other fields, which in some cases are automatically generated and in others require human review (e.g., to define a business category).
The search engine operator can use various methods to find or select web page addresses that will be loaded and analyzed or indexed in building the database. The methods may be chosen to expand or to limit the number of web pages that the search engine will access. As a result, the results of searches vary among the different search engines.
For example a web crawler or similar routine might attempt to load and analyze pages corresponding to all the top level domain names that are found to be registered with public domain name services or listed in a directory service [e.g., http://www.[domain].com]. Search engine services also can queue for indexing all pages that they are specifically requested to index (which request might be submitted by the page owner or another).
When indexing an initial collection of web pages, the list can be expanded by parsing the received pages for hypertext links and URL addresses that identify additional pages, and then loading and analyzing all the pages that are connected to the initial pages in that way. This process can be extended indefinitely. A smaller set of pages might be obtained by only indexing the top level pages or only links to top level pages out to a certain number of links from the originally targeted page.
Examples of search engines include Hotbot, Alta Vista, Yahoo, NorthernLight, Excite, etc. In addition, there are some search engine portals that run the same user query through a plurality of other search engines. The search engine comprises a processor that maintains a web page which the user loads by aiming his browser at the search engine URL (e.g., Excite's URL is http://www.excite.com/). The received page (namely the processed version of the html source code that is displayed) typically includes one or more Common Gateway Interface (CGI) boxes or similar form processing means by which a user who wishes to make a search enters one or more letter strings as search criteria. Boolean combinations of two or more strings often can be included or will be implied if not stated. The criteria typically are construed met if the specified words or phrases are found anywhere in the html source code of the target pages when last indexed. This includes portions that are not displayed (e.g., meta-tags and comments). The criteria can specify attributes other than the presence anywhere of a certain text string. This may be helpful, for example, to limit search results to finding files of a certain type (e.g., with URLs linking to a certain file extension type to find a certain kind of media). The criteria can also bracket out files in a selected date window.
The search engine compares the criteria to available information for web pages and sends to the user a report identifying the web pages that meet the criteria. The report to the user is transmitted in html source code. To generate the report, the search engine finds URLs for the selected web pages and inserts a list of these URLs into a shell form (i.e., an “empty” html source code file). The shell form has text and formatting to display title headers, possibly also ad banners and similar information. The URL list that is produced is inserted into the html shell. Each URL is flagged in the html source as identifying an html link (href=[etc.]). Thus when the list is displayed by the users browser, the user can select among the results and point and click or similarly highlight and invoke the html link addressing the page that the search engine considered to meet the user's criteria. This then loads the html source code directly from the remote page that was selected and the browser displays the current contents of the referenced web page according the html source code found there at that time.
After running a search and loading the web page referenced in a URL that is mentioned by the search engine as meeting the search criteria, it is not unusual that the user may not find the loaded web page to contain the terms used as the search criteria. This occurs because the content of the page was changed to eliminate the search term between the time that it was indexed by the search engine and loaded by the user who ran the search. For the same reasons, linked pages that are reported by a search engine sometimes no longer exist.
It would be possible to employ a web crawler process not only to find and index web pages but also to update the pages already indexed. The job of indexing web pages is growing constantly, and the job of also revising indexing work that already has been completed is that much larger of a job. The operator of the search engine must make some decisions on allocating available resources of memory, processing power and communication bandwidth to the jobs of seeking out web pages, indexing and storing usefully complete database information on the pages, and updating their database, as well as to handle user search requests and reports.
The typical search engine reports more to the search than the URLs of the indexed pages that meet the searcher's selection criteria. The URLs themselves, which are formatted as hypertext links in the search report, sometimes provide information as to whether or not a search hit is pertinent to the user's desires. For example the domain name associated with the page may identify an owner known to be in a pertinent business, or on the contrary may show that the search result is plainly not relevant to the search. The search engine typically also stores and includes in the search report listing one or two of the first lines of the web page that is referenced, which frequently includes a title that may be helpful to show quickly whether the selected page is of interest. The search listing also may show the date at which the web page was last updated or the date that it was indexed.
The usual success rate in finding a pertinent page or website in one try or only a few tries is actually rather low. The success rate varies with the subject matter, but in a typical search the user's search criteria may turn out to be unduly broad and may select so many pages that they cannot all be reviewed, or may be so narrow that much desired content is excluded, either of which can be an unsatisfactory and perhaps frustrating experience. Balancing the needs to include relevant material and to exclude irrelevant material can result in a substantial expenditure of time, much of which is effectively wasted.
It would be advantageous if the presentation of search results could be supplemented to more effectively assist a user running a search to quickly and meaningfully separate the pertinent and irrelevant results. However, such a capability will only be useful if it can be accomplished without unduly adding processing time and storage requirements to the steps involved in preparing database information for search and in presenting the results to the user.
It is an object of the invention to provide an abbreviated representation of searchable data files, in particular Internet/Intranet/Extranet html data pages, which represents their text and linked graphics in a visual snapshot form to supplement representations such as introductory text passages and URL addresses. It is a further object to collect and process the necessary information before conducting searches and to store a relatively small graphic file in association with the search database for representing each potential hit. The respective graphics file is reported to the user when a search results in a hit on the file, namely by inserting a hyperlink to the stored file in the search report sent to the user as the search results.
It is another object of the invention to overcome problems associated with the fact that different user configurations result in differences in the manner of displaying files, by preparing a graphic snapshot presentation as described, according to a default set of configuration parameters. Such parameters can specify font type and sizes, colors, backgrounds, screen pixel resolution and the like.
It is a further object to generate and store such an abbreviated visual presentation or shapshot as part of the process of building one or more databases using a web crawler or automated information review process to find and load or otherwise accept and process html pages. Preferably previously processed pages are again accessed and the database is periodically updated. Optionally, the abbreviated snapshot representation can be provided in combination with or in lieu of a tabular listing of the associated hypertext link and perhaps also an introductory portion of the text of the html pages. A hypertext link can be associated with the graphic snapshot such that the user (searcher) can point and click on the graphic to load and view the associated web page.
It is another object to permit such snapshot representation to be initially processed, or reloaded, processed and updated at times or at a frequency that is different from that at which the web crawler database is updated with respect to the text content of the web pages.
These and other objects are accomplished by the improved search engine of the invention, for managing user search and selection of data files stored at distributed systems coupled at network addresses. In particular the search engine is effective to improve searching of hypertext web pages on the Internet. The search engine has an associated web crawler operable to address and load successive web pages, and to index text data associated with the successive web pages. In this manner the search engine obtains parameter information such as words appearing in documents, word proximity and other information that can be used to distinguish at least groups of the web pages from one another when conducting a search. The web crawler stores the parameter information in a manner that cross references the paramater information with the associated web addresses or URLs of the web pages. The search engine accepts user-submitted search criteria and conducts a search or the parameter information to select the associated addresses of web pages that met all or part of the search criteria. The results can potentially be ranked, subdivided into categories and similarly handled according to known search engine operation. According to an inventive aspect, in conjunction with obtaining the parameter information for at least a subset of the web pages subject to search, the crawler renders a display image of the web page that is being indexed, and processes the image to provide a reduced size graphic image file corresponding to a static visual presentation of each of the indexed web pages. This graphic image file preferably is stored in a compressed graphic file format such as GIF, JPG, or a similar file, the file address or URL of which is stored and cross referenced to the criteria in the database that identifies the corresponding web page. When a search is conducted and results in a hit on a web page, its graphic snapshot is linked to the search results reported to the user. In a preferred embodiment, acceptance of the user search criteria and reporting of the results are handled by html page exchange communications between the search engine and the user. The search engine is accessed by the user and provides a form page having CGI boxes or the like for accepting text and/or other selections from the user. The search engine conducts a search which identifies one or more hits that are reported to the user by sending an html search results page. The search results page is composed by the search engine as a function of the search results and may contain no hits or a number of hits. Each of the hits is identified in the search results by the graphic snapshot, and preferably also by text information that reflects the content of the web page hit. Preferably, the search results page is composed to include a hypertext link to the URL address where the graphic snapshot file has been stored by the web-crawler/database/search-engine processes, for example by an IMG SRC=[path\filename] command inserted in html source code. As a result, the image file is loaded by the user's browser when processing the search results page, which generally occurs after the display of text has been accomplished.
As a result, the search results appearing on the user's browser include links to the web pages that were found to meet the criteria (hits), and also a snapshot graphic image of the way that the web page appeared when rendered at the time of indexing.
The invention is applicable to a wide range of search systems. For example, in addition to use with a web crawler and a text indexed word association database (or instead of automated text indexing), the invention is applicable to produce and associate representative graphic snapshots with websites that reside in a human reviewed directory such as Yahoo, wherein subjective characteristics of the data (a text form of which is sometimes termed “descriptors”) are stored in the database for comparison with user criteria in finding hits. In that situation characteristics such as an arbitrary business or art classification may categorize the web pages for selection in a manner similar to text string aspects used such as the presence of selected strings, word associations, proximity and the like. The invention is also applicable to automated categorizing processes such as used by Northern Light.
According to an inventive aspect, the graphic image file that is produced is not necessarily identical to the appearance of the page when ultimately loaded by the user after a search. In addition to the fact that the web page may have changed since it was rendered into the graphic file, the rendering is accomplished according to a predetermined display configuration of the crawler when rendered. Nevertheless, the graphic is a useful and very quick means for a user to sift through search results and determine immediately whether or not at least some of the hits bear further investigation.
There are shown in the drawings certain non-limiting examples illustrating embodiments of the invention as presently preferred. The same reference numbers are used throughout the drawings to identify corresponding elements in the respective figures.
According to the invention as generally shown in FIGS. 1-3 , the reporting of search results by a search engine 20, is improved and facilitated by offering each searcher or user 30 a visual representation 35 of the web pages found to meet the user's search criteria submitted to the search engine. The invention is particularly applicable to an Internet search engine but can also be applied to other networks 50 where the search engine 20 is available for managing user search and selection of web pages or similar files, stored at distributed systems 52 coupled to the network. The web pages, which may be considered data files, are found at addresses to which the search engine can link to load the data files, for example being accessible using URL addressing of the pages as hypertext markup language (html), file transfer protocol (ftp), telnet or other such file types. The data files may have embedded links to other data file or to graphics or other media files. The search engine 20 of the invention accepts user queries that characterize files of interest, searches for the files and reports to each such user the results of the search including network addresses of the files found to at least partly meet the query, enabling the user to link directly to the files, and also a snapshot of how the file will appear according to the most recent rendering performed by the crawler of the search engine.
The invention is described in this disclosure with primary reference to the preferred application to an Internet coupled search engine in which the data files searched are html pages on the Internet or worldwide web 50. Insofar as such files are accessible for loading and review by other users via browsers and search engines, they generally contain hypertext markup language (html) text, comments or tags, formatting commands, and links addressing other files. The data may contain text, media, scripts, programs, etc., and may be addressable at the same network address or a different address. The files may contain information that is not displayed when rendering the file, but nevertheless can be used to categorize the content of the files.
In the preferred example, the basic files (e.g., web pages), as well as the other files and systems to which they refer, are addressable using standard uniform resource locator (URL) addresses, containing a high, mid and low level domain name that is resolvable by a domain name server into a numeric Transmission Control Protocol/Internet Protocol (TCP/IP) address by which packets of data are directed from one computer system on the Internet to another. In this case such packets as transmitted to the system 52 containing the web page to be subject to search request transmission of an addressed web page (see FIG. 1 ). That system 52 responds by transmitting the contents addressed. The packets are reassembled for or by the receiving system. The browser or a similar process of the receiving system processes the data, normally but not necessarily for visual display on a local monitor.
Although described with respect to browser searching on the Internet, the invention is likewise applicable to other environments such as searching within a company intranet or other group of accessible data stores which have a visual aspect. The invention is also applicable to platforms and user interfaces other than PCs and browsers, such as the various Unix processes which are run on PCs or mainframes, etc. Furthermore, the invention is applicable to various wireless communication architectures. These environments and platforms are not limited to consumer and business use, and have applications in technical, military and other situations as well.
A block diagram showing an improved Internet search engine 20 according to the invention, for managing user search and selection web pages stored at distributed systems 52 coupled at network addresses to the Internet 50 or the like, is shown generally in FIG. 1 . FIG. 2 illustrates a succession of method steps and/or programmed operations of the system for building and adding to or updating a database 62 of searchable information. FIG. 3 illustrates a method and apparatus for conducting searches by accepting user queries 54, conducting searches of the database 62 and reporting search results in the form of a composed search report 80 containing visual representations or snapshots 35 that depict a presentation of how the selected pages would have appeared according to a default display configuration at the time they were accessed by the crawler 60.
It should be appreciated that the invention is discussed in connection with processes organized in functional blocks in the drawings. This illustration is helpful to illustrate the input and output sources and destinations, the operational steps undertaken, the various memory stores and data types involved and other aspects. However the illustration is not intended to exclude arrangements, for example, wherein separately illustrated units are sequential operations of the same processing element or wherein illustrated functions or storage capacity are distributed over separated units, especially separate processors coupled to a common network. The separately illustrated or commonly illustrated elements can be combined or separated as convenient, without departing from the invention and while serving the same functions.
The search engine 20 in the embodiment shown in FIG. 1 has an associated web crawler 60 operable to address and load successive web pages from remote servers 52 on network 50, and to index or to otherwise accept or generate descriptors that characterize text data associated with the successive web pages that are loaded. In this way crawler 60 develops parameter information on the successive web pages that can distinguish at least groups of the web pages from one another, and at times can be used selectively to identify a single web page, provided some encoded aspect of that page is unique among the pages loaded and processed. The crawler 60 stores the parameter information and associated addresses of the web pages as a database 62 in a storage medium 64 that is accessible to a search processor 78 that accepts the user criteria 54 and prepares and sends search reports 80 to the query submitting user 30. The search engine portal or processor 78 responds to user submitted search criteria by searching the parameter information in the database 62 and reporting to user 30 at least the associated addresses of data files that met the search criteria when indexed. In particular, search portal/processor 78 reports the URL addresses 82 of web pages meeting the user criteria.
The web pages are generally maintained on web servers 52 (FIG. 1 ) that are “remote” from the querying user 30 and from the search engine 78, but actually could be anywhere that is addressable on the particular network, including on the user's own system. The web servers 52, in known manner, store text and graphic data or addresses of graphic data found elsewhere. That information is available upon request and in the case of the Internet and other TCP/IP protocol type networks is transmitted in packet form to any user that requests the web page by directing a request to the web server identifying the TCP/IP address of the web server 52, the sender's address or identity, and the address of the desired page. This normally involves addressing using URLs that identify the type of communication desired, such as transmission of an html page (versus a linked graphic or media file, or perhaps a different type of interface such as ftp or telnet), and an address that represents the domain name and a subdirectory path leading to the actual html file or other file.
The same sort of URL addressing is used internally in html pages to address image and other files that may be located at the same web server or elsewhere on the worldwide web, namely by providing a hyperlink that states the network address of the text or other content, as opposed to containing the content itself. Such hyperlinks can also be invoked to move around in a given file, for example from one subheading to another. The hyperlinks are embodied by automatically recognizable codes (e.g., “href=” or “img src=”) that appear in the source code together with the various start and stop tags that specify text formatting, colors and other aspects of the page as it should be displayed, for example using a browser. In a browser such as MS Internet Explorer, the source of a displayed page can be displayed by selecting “View” and “Source” from the toolbar.
According to the invention, a crawler 60 collects web page data and is generally shown in FIG. 2 . Crawler 60 can be operated preliminarily but preferably operates continuously during operation of the other components to collect additional data and/or to update data already collected. Crawler 60 has one or more fetching processes 66, several being shown in FIG. 1 and identified as Agent A (fetch) processes. The crawler 60 via its fetching processes 66 determines web pages to load and attempts to load them. For example, the crawler 60 may test TCP/IP addresses (known as scanning) or attempt to load pages from particular domain name addresses where servers might be up and running, obtained for example, from a domain name server (not shown). The text portion of any data obtained by the fetching processes 66 from a particular URL address is parsed or divided into discrete terms and statements. These terms and statements are compared to predetermined reserved terms and formats that represent URLs, file addresses and the like. When the comparison indicates that a hyperlink to another file or web server has been found (or that a given string so resembles a hyperlink as to be interpreted as such), the found address is added to a list of addresses and an attempt is made in due course to load a file at that address, thus increasing the field of files that have been consulted.
The general function of the Agent A fetching processes 66 is to obtain the files available from remote web servers 52 and to note the addresses of the files (URLs for the Internet) that when invoked will address and load the file. As a result of communication delays, it is preferred to employ a plurality of concurrently active requests for files so that one file can be processed while waiting to receive another. This aspect is represented in the drawing by plural Agent A processes 66, which obtain the fetched files and store at least part of the fetched files in a buffer memory or queue 92. In connection with html web pages, the data includes html source code, addressed files containing images, audio or other media, which are stored in buffer 92 together with the addresses from which they were obtained.
The collected information from downloaded files, particularly text files, is processed according to a generally conventional text processing or categorizing technique 68 to build a text or descriptor index in database 62 as shown. The database 62 contains an index developed from automatic analysis (generally “indexing”) or human review (categorization) of the text and other data, indexed to the URLs of the pages from which they were obtained. Insofar as the automatic or human generated descriptors and addresses are described herein as a “text” index, it should be appreciated that index might represent any attributes of the content of the respective web sites, not limited to words in their displayed text. For example terms in hidden meta tags, comments in the source code, strings found in addresses and the like are also potential data points that may be collected. Any arbitrary characterization that my be automatically assigned or assigned by a human reviewer can be deemed an indexed point. For example, the index could contain information as to the type of links found in the source, the date of the last update, the country of origin or language, whether the site appears to be academic or commercial, an entry for rating content as adult or “general admission” for keying child protection interests, and so forth.
According to an inventive aspect, the crawler 60 that is operable to receive the web pages and to extract the parameter information from them, generates a file 72 of graphic image data corresponding to an appearance of each of the web pages, which is stored, preferably as a reduced-size and compressed image data file 75, in association with the database data respecting the page. When search results are reported to the user (FIG. 3 ), the search engine reports the associated URL addresses 82 of web pages that met the search criteria in a conventional manner, preferably inserting a hypertext link to each identified page into an html page reported to the user, optionally a short description or excerpt, and also inserts into the report page the graphic image snapshot file by inserting into the source of the report page a link to the stored compressed graphic image file 75. The user's browser displays the search results in conventional form, namely by showing a selectable hyperlink to the addresses and optionally a description or excerpt, and displays a snapshot of how the identified page is likely to appear if or when it is loaded by the user's browser, should the user point and click to the link to invoke the URL of the page hit.
The search portal 78 that performs the search by reference to the database 62 in storage media 64, reports the search by composing a web page containing the search results, assembling the search report using hypertext markup language. The search report contains headers and information identifying the portal and perhaps contains advertising. The search report also lists the hits that resulted from the search. More particularly, the search engine inserts (in list or table form) a text string showing the URL address of each web page hit (i.e., the pages found to meet the user criteria) together with a hypertext linkage to that URL (e.g., an “href=” statement), causing the user's browser to show a link that can be invoked (pointed and clicked) to load the page at the stated address. Such a report is conventional in an html source search report. It typically also has a description or excerpt and may be arranged in a pyramid or hierarchy of categories. According to the foregoing inventive aspect, the search engine also inserts the URL address of the graphic file that has been processed by a further process identified in FIG. 2 as Web Agent B 95, to contain a snapshot reduced/compressed graphic 35 representing the page hit.
The link to the compressed rendered graphic file can be made, for example, by use of a IMG SRC=<domain>/<path><filename> command in the html source. The graphic can be associated with a hypertext link to the hit page URL as well as linking using an HREF=<URL of hit page> command as mentioned above. As a result, the user's browser when displaying the search results also displays the graphic snapshot image, as shown in FIG. 3 .
The invention has three main components, shown generally in FIG. 1 . As shown in FIG. 2 , these include the crawler processes that fetch files from web pages in the universe of web pages to be subject to search, and the processes that index or catalog the pages and render the fetched files into graphic image files. The processes in FIG. 2 can generally be considered the processes that obtain raw data and process it to provide a searchable database and information that may be included in search reports when a web page becomes a hit. Preferably, according to the invention the crawler processes 66 that are associated with collecting the raw data files, which experience communication delays, are separate from the processes 68, 95 that process the raw data into a form apt for storage in database 62 in preparation for searching. FIG. 3 illustrates the processes 78 that interface with a user who seeks to search the web 50, including presenting the web page hit information to the user in html form for browser display.
Referring to FIG. 2 ., the search engine includes or is associated with web crawler 60, which is an engine that conducts web page addressing, loading and analyzing, and stores representative data in a storage device 64 containing a database 62. The stored representative data characterizes the web pages that the crawler loads and that are analyzed for content by process 68. Of the main activities to be effected by the search engine system (i.e., by the crawler and the search processor), preparation of database 62 allows a search to be conducted more quickly by reference to the processed database information gleaned from the field of possibly-selected files, than would be possible if the search engine attempted to load and analyze the entire universe of files after the user had submitted query 54 (FIG. 3 ), namely while the user was awaiting search results.
The process of preparing database 62 includes determining URLs (or perhaps TCP/IP addresses or other addressing strings) for the files to be searched, and then loading and analyzing the files to note the occurrence and juxtaposition of text strings. Alternatively or in addition, the files are categorized for other aspects, for example by human review and assignment of arbitrary descriptor categories that tend to distinguish files by their content or owner or type, etc. The files or webpages consist essentially of ASCII characters stored in a text file that is known to be or is identified as hypertext markup language, often having an “htm” or “html” extension on the file name). As a result the ASCII character strings in the web page are searched for combinations of characters that conform to specific code name and character rules whereby they can be interpreted as commands or links or other specific forms of information in html.
Html is a form of standardized markup language in which various tags are associated with ASCII character strings. Many of the character strings and tags used in html webpages concern the appearance of the associated text and the visual aspects that are to be displayed concurrently with the text. Such commands can specify a header, a background pattern, color or complete image, set or reset a font type, font size, capitalization or color, change justification, centering and margins, specify lines, a table or frames, call for insertion of a graphic figure in any of several formats, which may be static or animated, and otherwise generally vary the appearance of the page and the text on the page. The strings also can address additional files.
The encoding of a representation of the occurrence and juxtaposition of text strings is generally known as indexing, and results in a database of information in which each text string found during the analysis of all the files or pages searched is referenced to the URL address where the files or pages can be found. According to the present invention, such indexing can be construed to include other methods for categorizing data files in a manner that allows distinctions to be drawn that are useful for searching, including human reviewer categorization and discrimination for non-text factors such as the revision date, country of origin or the like.
The database 62 is generated by preparing or obtaining a set of characterizing parameters concerning the fetched files, or their addresses or content or the like. Database 62 contains a cross reference between criteria and the identity (normally the URL address) of the file that matches the criteria. Assuming that the criteria concerns a concatenation of terms (e.g, “quick brown fox”), all the URLs of files that contain that string are available by searching for the string. Likewise the URLs of all the files containing the component terms are available (“quick” or “brown” or “fox”), and these terms or phrases can be combined with other terms or arbitrary categorizations to find a page (such as the Quick Brown Fox Hardware Store). The indexing and/or categorization particulars can be objective or arbitrary, and wholly or partly driven by human review or by automated means, and can concern any aspect that tends to be unique to individual files or common to subsets of files only.
Automated indexing and similar characterization systems may seem objective but the results are determined in part by usage chosen by the author of the content, which is to some extent arbitrary. Human review is subject to potentially arbitrary choices by the reviewer. The search database as discussed herein includes any collection of information prepared in a manner that enables search criteria to be compared to stored criteria to distinguish files from one another. The search criteria involves combinations of categorizations and/or text strings and other factors, chosen by the user in an effort to target the files or pages that have a desired subject or include reference to a particular datum. At the same time, each criterion is not applicable to every page reviewed, and as a result it is possible both to collect files that meet a user's criteria and to eliminate files that do not meet the criteria and thus are irrelevant to the particular search.
Referring again to FIG. 2 , the universe of files and pages can comprise, for example, all the high level pages of registered domain names on the Internet, plus a series of additional lower level pages. The lower level pages can include all the pages to which the high level pages are linked by hyperlinks in the content of the high level pages and/or frequently encountered subpage names such as “index” and “home”. Various such processes are conventionally practiced using so-called web crawlers that are operated constantly, often during low traffic hours, to find, load and analyze (index) a very large universe of web pages.
Conventional web crawlers prepare a database that records and can be used by searchers to select (or de-select) web pages primarily on text strings and Boolean combinations of text strings found in the content of the web pages and indexed in the search engine database. The web crawler/search engine database also can be arranged to record and permit searchers to select and de-select on the type of media linked to a page, on a window of dates, the language of the web site or page, the location of the registered domain, the depth of a particular web page in the directory structure of the target site, and other aspects.
Although it is possible and useful to encode and to select web pages based on attributes that are determined from letter strings found in their text or perhaps in the particulars of their URL address, it is not readily possible for an automated web crawler and associated processor to encode much of the appearance of a web page. In the event that the web page contains a link to a graphic image file, for example, the URL address of the graphic image file, including its file name, will be found in that web page, but the graphic image could have any content and may or may not be consistent with the file name. Therefore, known search engines cannot discriminate among web sites by virtue of most of the attributes that affect the graphic appearance of a site's contents when displayed on a browser or the like. However users can readily discriminate among web sites, particularly some forms of web sites, by appearance only.
The configuration of the user's system also affects the appearance of a web site content when displayed. On the level of the browser, the user can opt to display particular font types, and also can specify font sizes. These configuration choices affect the appearance of a retrieved page even if the page defines specific fonts that are available to the browser. The browser may also permit the user to select whether or not to use the background colors of retrieved sites and other features affecting the display. On the level of the operating system, the user can opt for different display options such as the number of pixels and the color resolution employed. These aspects also affect the display. As a result of such user choices, retrieved web pages appear differently on different user's displays when retrieved. For the most part, differences due to such configuration choices do not grossly affect the appearance of the web site, but they do cause an identically encoded page to appear differently on differently configured systems and/or browsers.
The search/reporting steps of the browser, generally shown in FIG. 3 , include accepting search criteria 54 from user 30, for example using a CGI script technique in which the user enters selections including text strings, literal strings of plural terms, additional encoded aspects such as media types, date windows or limits, countries of origin, etc. The user may also select Boolean relationships (AND, OR, NOT, XOR). The search portal may require commands or may permit selection using point-and-click steps. The search engine compares the search criteria to the pre-prepared database of information gleaned from the web pages it has loaded and analyzed from the field. The results are reported to the user by preparing and formatting an html source reporting page into which hyperlinks are entered that name and point to the addresses of the files that were found to meet the criteria. Often the report includes other information such as the date the page was last updated before it was indexed, and a few lines of introductory text from the page, which provide a hint to assist the user in determining without loading the page whether the page is likely to be relevant to the search. If the user finds a link that appears to be pertinent, the user selects and engages the hyperlink. This causes the browser to load the html source found at the URL address shown in the search report, and any referenced files and links therein. However, the page may have changed between the time that the indexing was accomplished and may have totally different content than it had when indexed. The page may no longer exist. In those cases, the search fails except to advise the user that the page formerly held information that might have been of interest.
Deliberate as well as inadvertent “search engine corruption” sometimes occurs. It may be crucial for marketing or other purposes for a web site to be found in user searches on search engines, and it can be lucrative or otherwise beneficial for a web site operator if his/her site is ranked high in the search results for particular terms. Thus, a great number of website operators have ways to misrepresent the content of their pages. Keywords intended to cause the page to be selected and to rate highly in particular categories can be included and may or may not be displayed. Misleading text can be placed in miniscule font at the bottom of a page or misleading text can be hidden by making it the same color as the background on which it appears. Text can also be placed in “ALT” descriptions of images and graphics, thereby indexed by the crawler but not seen by the user. A particular term can be included one or many times to improve rankings, by one of the foregoing techniques, or by overloading keywords in “META” tags included in web pages and not displayed. Another technique is to temporarily post a page to be textually indexed by the crawler/search engine and then to replace its content after it has been indexed, or similarly, meta-refreshing the web page so as to redirect the user to another page address. According to an aspect of the present invention, the user can visually distinguish pages having undesired content and not waste time on them. Search engine corruption using the aforementioned techniques to provide misleading text is averted due to the visual nature of the present invention.
According to an inventive aspect, a system of the type that indexes or categorizes information on web pages for searching is improved by encoding and providing in the search report 80 a standardized graphic representation 35 of the appearance and rendering of each page at the time that the page is indexed. The graphic representation 35 preferably is in the form of a compressed image of the page, described herein as a snapshot, stored in a standard compressed file graphics format at a location accessible to the search portal process 78. The snapshot is acquired when the page is initially loaded by the crawler 60 for indexing (FIG. 2 ). The snapshot is rendered, converted to the compressed format and stored. When the subject page is selected in a search (FIG. 3 ), transmitted to the user are the individual snapshots, which have been stored locally to the search portal processor 78, in association with the index/categorization database. In this way the snapshots 35 of the hit page (which may be one of a number of hits that are reported to user 30) is shown when providing the search report.
The snapshots 35 can be contained in formatted image files (e.g., GIF, JPG, etc.). The snapshot image files, or URL addresses pointing to the image files, preferably are stored in the database 62 that also contains the URL addresses of the indexed pages. In reporting search results, the search engine 78 inserts a link 82 aiming to the snapshot image file 35 into the html search results page 80. The search results appear on the users browser 84 as a link to selected pages with an associated snapshot of the page when indexed, as shown in FIG. 3 .
These operations impose challenges that are addressed according to the invention. One problem with acquisition of the snapshots is due the very large number of websites that must be physically rendered, namely every website that is indexed and is available in the universe of websites subject to search. The website content, including any referenced image files, must be downloaded by the crawler Agent A process(es) 66 and rendered by the rendering Agent B process(es) at acceptable speeds, and preferably also reduced to obtain reasonably sized image files 35. The image files must be accessibly stored and downloaded from the search engine 78 to the searcher (specifically the user's browser 84) at acceptable speeds as well. The invention applies particular technology to solve these and other problems.
Major search engine portals each have a usually-proprietary “robot” or automated process that crawls the web as described above. In each search portal or system a robot or crawler that accepts or finds website URL addresses, accesses websites by TCP/IP addressing and loads their source code. The crawler robot automatically parses the text of a website, namely dividing the strings found in the source code into units separated by delimiters such as spaces or punctuation. The strings and the succession of strings are compared to stored parameters whereby certain strings are construed as links or formatting commands, which is noted accordingly. The occurrence and proximity of these strings and the free content strings that are to appear as text in the web page when displayed on a browser, are all noted and stored in a database where this information is cross referenced to the URL address of the website from which the page was loaded.
In operating a conventional crawler and indexing routine as discussed, the website text can be analyzed and indexed at an extremely high rate of speed because the page is treated only as a succession of text strings. No processing time is spent to load and process or otherwise handle any embedded or referenced graphics, media, scripting, Java, or animations. Such files are not helpful for traditional indexing and thus are not requested. The html tags that might be used to find and load files for non-text content may be textually parsed, but their associated data files are never requested and not retrieved by the traditional text crawlers employed by the major search engines. In addition to avoiding processing overhead, no time is devoted by the crawler for data transfer that might be needed to request and receive packets containing the graphic or other media files. The load on the crawler is minimized because the portion of the website that is loaded and processed, namely the text portion, represents little “weight” in communications bandwidth requirements, processing time and the like for most web pages. Without the need to download and process large graphic and media files, simple text indexing in the traditional sense by conventional crawlers is very efficient, simple, and fast.
Although simple text indexing is quick and simple, the exact opposite is the case for full graphic rendering of a web page. Before the display of a web page can be completed, it is necessary for the browser to wait so that each and every required file is downloaded. The browser must wait for all necessary files to be received before a full rendering of the display. Additionally, any script or otherwise dynamic content normally awaits receipt of the entire file before processing begins. Furthermore, image, graphic and media files are very data intensive and thus require substantially increased transmission times in comparison to text.
A web page will contain one single text file, but in contrast may contain dozens of graphic and media files. Traditional text crawling by the major search engines require that only the one single text file be transmitted and parsed. By contrast, full graphic rendering employed by the current invention requires that each and every graphic, image, and media file be transmitted and subsequently rendered into a full visual depiction of a web page.
In a conventional web crawler installation, dozens of robots can run on the same processor simultaneously, all executing their individual tasks without regard of the other robots present. Employing a large number of robots on the same computer processor facilitates conventional text indexing. Also, the conventional crawler is only concerned with processing text data. The crawler processes need not include many steps required of a browser to handle the graphical content. Specifically, conventional crawler processes do not include generating and presenting a visual display, which would require additional network communication (to obtain graphics, etc), consume time and processing power, and require devotion of system resources such as the visual display itself (e.g. monitor).
The text data portion of a web page is most commonly five to ten Kbytes in length and is received in less than a second on a typical network connection. The text file is normally the first file sent from the originating web server. Image files and script or other code, if requested, follow afterwards. The robotic processes of requesting a text file, retrieving packets and reassembling the text file, parsing the text file by finding terms within delimiters, and indexing its contents, can be accomplished under normal circumstances in 0.5 to 1.5 seconds. Assuming a one second average processing time, one computer processor operating, for example, 25 text processing web crawler robots (which may be conservative), can obtain and index the text of 25 web pages per second every second. Operating continuously, such a crawler could process over 15 million web pages per week. Certain factors limit the rate at which pages can be processed. Web congestion, long files, long transmission sequences, low bandwidth server connections, and other factors that vary from one website to another and one time of day to another may limit processing speed. Nevertheless, a search engine portal that has several computers with multiple robots devoted to crawling the web, might complete an entire crawling sequence through a reasonable universe of selected web pages, in three or four weeks.
By comparison, complete and total processing of web pages, including rendering all graphics requires a substantial increase in resources. If a typical website has text content of about 5 Kbytes, that same text file may have any number of associated graphic files, each of which is several times the size of the entire text file. All the web page data must be downloaded totally and processed before accurately rendering the web page, because the data may affect the rendering even if the data itself will not appear on the screen.
A website server is usually prompt in sending short files, such as the requested text of a particular page, and short file transmissions are more frequently successful than longer ones due to the additional packet handling for reassembling the file, and the increased possibility of transmission errors requiring retransmission. The browser receiving and processing the graphic file seems to pause or to stick on presenting a particular graphic section during the resulting delay. The transmission may pause at any point, even on the last packet of a number of successively transmitted files. The receiving browser or other processor cannot complete the total and full rendering of the web page, for display or otherwise, until the delay elapses. The receiving computer simply waits before completing the display of the page.
For rendering a page layout including graphics, the browser or page rendering robot normally requires on the average 30-45 seconds per page to receive and process a web page into a graphically visual layout (an approximation that incorporates a variety of factors including changes in bandwidth, server lag, and lost packets which can result in web pages being delayed).
The graphic layout of the page usually comprises a series of image files. Each file consists of or is unpacked into an array of digital data words representing the saturation, luminance and hue or the respective RGB levels of each pixel in an X-Y field corresponding to the display screen area. On a computer running a browser, the image file is loaded into a series of memory locations accessed by the display driver to drive the monitor display, either in the processor random access memory or in the memory of a video display driver card (or both). The process of rendering a page into a visually graphic layout usually requires devoting a full display memory field to this function, and particular aspects of processors are often devoted to handling a limited number of display images. As a result, only a single image processing application or graphical robot can visually produce the intended web page layout on the screen at any one time. In other words, rendering the page layout of the website at its intended dimensions (displaying a full screen) can only be accomplished using a single graphical application or web browser at a time.
Web pages are intended by their creators to be seen at a size rendered at or near full screen dimensions. Obviously, only one full screen web page can be displayed at any given time on one screen, and as a result, only one graphic robot and its associated hardware can be active to render that display at any single instance in time. This situation is thus unlike the way text is processed by traditional web crawlers, wherein a single computer processor is capable of running dozens of textual web crawlers simultaneously “in the background.” This is because requesting, retrieving, and indexing text from a web page does not commit visual or display generating resources. Without the requirement to share this type of resource, any number of the text indexing type of web crawlers can run at one time.
Because of the limitations, constraints and resources used for rendering and display, crawling the entire web for the purpose of successively rendering web pages to produce a display can be impractically slow. If the conventional text retrieving robot is capable of indexing 1 page per second, a graphic rendering robot is capable of processing 1 page display every 45 seconds. As a result, a computer running 25 simultaneous text retrieving robots can index an estimated 15 million (15,000,000) web pages per week, but the same computer running a 1 graphic rendering robot would process and estimated 15 thousand (15,000) web pages per week. If there are 100 million web pages in the desired universe, graphically rendering the entire universe of searchable web sites on one computer processor would require approximately 6,600 weeks or nearly 127 years to complete. Even employing 25 different computer systems would require over 5 years to complete a graphical rendering of the desired 100 million web pages.
According to an aspect of the present invention, at least two independent types of intelligent web agents are cooperatively operated to handle different aspects of the job of retrieving, rendering, and processing websites, in a manner that makes it possible to produce graphic data in the form of a compressed or reduced graphic file representing the appearance of a rendered website, and to do so at an acceptable rate. The first type of intelligent web agent (now to be referred to as “Web Agent A”) requests, retrieves, and downloads each and every file associated with a particular website, including but not limited to the source code text file, graphic files (e.g., GIFs, JPGs and others), script files, Java executeable files, Flash technology files, Shockwave files, animations, and so forth. Web Agent A is arranged to communicate or pass data into one or more memory buffers or queues accessible to a second type of intelligent web agent (now to be referred to as “Web Agent B”), which siphons out the website files as needed to produce and render complete graphical displays of the web page.
The rendering process by Web Agent B comprises processing the text and html tagging data to prepare a visual representation. All the files necessary to render the image have preferably been obtained by Web Agent A before then, and such files are stored in the buffer. Web Agent B produces a full visual representation, such as a bitmap file containing a pixel data array, which if coupled to a display driver could be used to display the web page layout on the video monitor at full screen dimensions. In short, Web Agent B prepares a visual image as might be provided by a browser.
The visual display of the web page is then compressed by Web Agent B or a process associated with it, to a predetermined and preferably small image size, for example a 2 in.×2 in. image on a 17 inch diagonally measured display screen. This process may involve sampling or local area averaging techniques as known in the art. The reduced size bitmap image then is digitally compressed and/or encoded to minimize storage requirements and to permit quick transmission over an ASCII-only data channel. The reduced size bitmap image can be converted into a JPG, GIF or similar format for an image file suitable for web transmission. That image file, which represents the rendered appearance of the associated web page at a particular point in time, is stored in a mass memory accessible to the search engine. The mass memory can be in one or more hard drives, ram caches, writable CD ROMs or other media that is useful as a high capacity RAM. The mass memory can be a peripheral on the search system or can be accessible to the search engine, for example using communications over a local area network, provided that the image files are very quickly recallable using a minimum of data communications and/or communications that are direct rather than over the web.
The mass memory can have a subdirectory naming system and file naming system based on the network addresses or URLs of the web pages from which the graphic files were generated, or alternatively the files can be arbitrarily named or stored and can be found using a cross reference table in the search engine whereby the address or URL of the web page and its associated image file are cross referenced.
The search engine memory also comprises text indexing data or human categorization directory data (or both), that is obtained in a conventional web crawler manner and includes an association between the text data found at each web page and the web address or URL of the originating web page. In this way, the text indexed or categorized data, and the graphic file location, are both indexed to the URL. By selecting a URL, the search engine can call up the graphic file representing its appearance when rendered at some time in the past. After receiving a selection containing one or more text strings, Boolean combinations, file extension types or other criteria, the search engine can determine the matching web pages, report their URLs and provide a graphic file showing a miniature window version of how they would have appeared if loaded by a browser at substantially the time when their data was loaded and indexed.
Web Agent B preferably has additional functions, including keeping status information such as storing log files containing addresses and/or linked file names that have been attempted and obtained, optionally including a queue of files that presented problems when first tried and should be re-tried or after a time will be rendered with missing-graphic gaps, web addresses that have been completely rendered, etc. Preferably the logs and status indicators are sufficient to permit an operator to monitor operation by reference to by readouts or by displaying stored data. Web Agent B also preferably generates error messages and/or alarms in the event of any crucial errors. Status readouts available can include rudimentary data such as the current URL being processed, the rendering state of the current URL, the number of URLs processed since inception or last clear, any error messages and so on.
The search engine can comprise one or a number of processors and the processors can be in direct communication or linked on a local network or other arrangments, the key being quick access to the stored database of data representing the universe of web pages that have been processed and therefore are searchable. The search engine accepts user search criteria in a conventional way, such as using CGI form boxes to enter text strings into an associated search engine entry html page that is addressable by a browser. The search engine permits selections to be made according to at least one search criterion and preferably accepts a variety of different criteria types and combinations. These aspects of the search engine can be of the type conventionally used by current search engines such as Hotbot, Yahoo, Alta Vista, Northern Light, etc. The search engine is operable to select web page hits as a function of user supplied search criteria and to determine the URL addresses of web pages (hits) that wholly or partly meet the criteria. In addition to determining the URLs of hits, the search engine may store and retrieve a brief exemplary text string such as the initial few lines of text in the web page hit.
The search engine reports search results to the user that entered the search criteria, by composing an html source page and transmitting it to the user. This html report page may identify no hits or a long list of hits, depending on the search results. In composing the report page, the search engine typically shows the search criteria used, and displays indicia summarizing or similarly identifying each web page hit. For example, the search report can identify hits by the URL of the originating web page. Preferably a short text selection such as the first few lines of text is shown. The html coded report page prepared by the search engine includes an associated hyperlink to the URL of each hit. The URL can be shown in plain text and provided with an associated hypertext link (href=[URL]). The user reviews the URLs, sample text or other information and activates the hyperlink of a selected web page identified in the results, thereby loading the web page presently found at the address of the originating page when processed by the crawler robots.
According to the invention, the composed search report page prepared by the search engine includes but is not limited to the URL of each web page, the title of each web page, a description of each web page, and a graphic depiction of each web page. The user's browser immediately loads the source code, which contains the text portion of the search report. In processing the source, the user's browser encounters the links to the image files that were included by the search engine when composing the search report page and obtains the image file. Preferably the report page composed by the search engine places the graphic for the web page hits immediately adjacent to the associated text and hyperlink. The graphic image was rendered under certain assumptions as to the display configuration and represents a snapshot of the web page frozen in time. The snapshot is at least an approximation of how the web page will appear if the link is activated and the page is loaded by the user (i.e., if the page is unchanged and the user's display configuration is equal to the default configuration assumed by the Agent B of the invention). Unless the page has been substantially changed by its owner, the graphic depiction will substantially assist the user in sifting the pages that are definitely interesting versus possibly interesting, neutral, unlikely to contain pertinent material or definitely irrelevant.
It is an aspect of the invention that the assets and processing power of the search engine system are proportioned to coordinate the operation of Agent A (for fetching) and Agent B (for handling image content), whereby neither one substantially lags the other. Agent A is subject to communication delays involved in requesting, receiving and storing the needed files from the internet, which can delay a single robot, but in fact is ameliorated by running multiple copies of Agent A in the background. Agent B has more data to process, but due to the preloading by numerous Agents A in the background can process the data quickly from local copies. Agent B is free to monopolize the display in the foreground while multiple Agents A in the background acquire necessary files from the internet and feed them into a temporary data buffer.
In view of the communication delays and to maintain the pace, it is presently preferred that 32 web agents of type A operate in conduction with each web agent of type B. Thus a plurality of web agents of type A continuously fetch and feed into a buffer or queue all web page files of targeted web pages, including their source code and their graphic images, such as JPG, GIF, Java, Flash, etc., all being stored locally. One or more web agents of type B, preferably one for a number of Agents A (e.g., 32) continuously processes and removes files from this buffer to produce and render one web page snapshot image after another. Concurrently with this process, the text portion of the web page data is indexed or categorized.
The ratio of Agents A to Agents B can be determined from experience such that the contents of the buffer or queue remain substantially stable for the particular search engine. Alternatively, the ratio can be changed on the fly so as to keep Agent B constantly working and to keep the size of the buffer or queue stable. If the queue continues to grow, the ratio of Agents A to Agents B can be reduced, thereby committing more of the available CPU time to Agent B, which should cause the buffer to shrink. The buffer should not be allowed to shrink indefinitely, or Agent B will become idle or will lose efficiency or even stall, waiting for complete sets of web files to become available. Preferably, an optimal buffer size is assigned, such as some hundreds of MBytes. Additionally, this buffer is maintained relatively static by the deletion of data after it is used by Agent B. After startup of an estimated optimum ratio of Agents A to Agents B, additional Agent A processes can be added until a substantial portion of the available communications time is filled with active Agent A messages. If the buffer grows continuously, Agent A processes are reduced in number relative to the number of Agents B, and vice versa. Inasmuch as the optimum ratio is in part due to communication delays due to web congestion, the ratio of Agents A to B can be varied throughout a processing day.
Web Agent B continuously renders and processes web pages one after another according to a specified queue. Web Agent B does not suffer from the limitations and overhead of requesting and transferring files over the internet because these problems are solved by the team of web agents of type A, for example thirty-two of which may be busy addressing and loading files from different sources.
In one embodiment tested, a single web Agent B was employed in a computer engaged as described above. Conventional browser and display driver routines were used to render bitmap display files from html pages that had been revised such that all included image reference links pointed to graphics files that had been previously downloaded by one of the plurality of operating Agent A processes and stored in the queue or buffer, namely on the system hard drive. An image conversion utility then converted the display bitmaps into a GIF image files under file names referenced to the corresponding URL of the originating web page. This arrangement proved to be an efficient and fast method to obtain snapshot renderings of web pages. Web Agent B in such an arrangement controls and manipulates all processing and system resources for graphical display but is not held back by the delay of retrieving and storing of the necessary files, which is collectively performed by all the Web Agents of type A, running as concurrent processes in the background and thus not requiring many of the system resources, including the display buffers and drivers. The system proved efficiently cabable of rendering at least one web page per second, and if run continuously would render 86,400 pages per day, 604,800 per week. This may seem like an adequate rate, but assuming a desired universe of 100 million pages, a single computer system crawling at that rate would still need approximately 3 years to complete a crawling cycle. During that time, the content of most of the web pages would have been changed. Therefore, the invention is preferably applied running a number of computers operating concurrently. Networking to a common database and running 18 computers concurrently would allow a complete rendering of a desired 100 million web sites every 2 months. It is preferred that such a 2 month cycle be utilized to maintain a fresh and updated database of graphic snapshots.
It is not unusual during an initial attempt to retrieve a web page using a browser, including retrieval of its included or referenced graphic files, that at least one of the files is not successfully transferred. This may be due, for example, to congestion or other factors causing the website server to time out and issue an error message. Sometimes a file is garbled in transmission and this is detected by the receiving browser, which visually marks the displayed page to show that there is a missing file (e.g., a rectangle is placed at the image position with a red “X” indicating that the transfer was unsuccessful or the received file was defective and could not be decoded and/or displayed as an image). In that situation, the browser “refresh” function often can be invoked to make one or more additional tries to retrieve the rest of the webpage, at the user's point/click command.
According to the invention, in such a situation a built-in redundancy deals with damaged or missing files. Web Agent A is responsible for retrieving and storing the graphics files, and all associated files of a particular web page. In so doing the originating server or an intermediate router may time out or transmit a damaged version of the file. If a file is not received or a received file is defective, which is not infrequent when browsing, Web Agent B of the invention detects that the file is missing or defective (in other words, Web Agent B notices that the necessary file is in fact, not in the buffer as it should be). Web Agent B can be arranged to attempt one or more times to retrieve missing files from the address specified in the html source code, (i.e. to obtain the graphic file again “live”, directly off the web). Preferably, however, if Web Agent B is ready to render a file and one or more graphic files is not found, then Web Agent B can signal one of the Web Agent A processes to attend to fetching the file, and during the delay Web Agent B proceeds to render another file whose component files are all available. With a redundancy or retry capability, the system is likely to successfully render the whole webpage, with all its graphics and all its associated files, more dependably than a browser responsive to live file downloads. In fact, this redundancy brings the success rate to nearly 100%.
The respective crawling, communication, indexing, and rendering program functions can be written in any of a variety of available programming languages and can run on any of a number of different platforms. The program has been found to be readily embodied in C++ running on a Windows NT operating system.
It is an aspect of the invention that available communications bandwidth is used efficiently. The multiple Agent A processes operating concurrently are such that the usual reason for waste of communications time, namely waiting for a response from a remote web page server, is minimized because delay experienced by one of the Agent A processes is used by the other Agent A processes that are operating at the same time. The invention can perform on any bandwidth connection, including 28 Kbps. Of course a high bandwidth connection is preferred, such as one or more T1 or T3 connections (if not even higher).
Apart from the example of Windows NT, the Unix platform is alternatively useful according to the invention due to its capability of handling multiple simultaneous processes. The respective software robots can run on the Unix platform as applications programmed, for example, in C, C++, Perl or one of the other languages. To finish crawling cycles reasonably promptly, in a preferred arrangement numerous computers are employed simultaneously, each having its own connection to the internet and each employing its own embodiment of the current invention. The computers can reside on a network and feed off of and simultaneously contribute to a common database maintained by one of the computers on the network.
The two general functions associated with preparing the database of information which is then subject to search and reporting, are the functions of retrieving all webpage data (performed by Web Agent A), and generating a “snapshot” file from the data (performed by Web Agent B). It is found that these functions can operate concurrently with or apart from the search engine processor or processors that search the database of information and return results to the requesting user. The preferred embodiment, however, is to perform all processing in regards to rendering, resizing, and compressing the snapshot prior to being accessible to surfers on the web. A cycle of processing (crawling, indexing, rendering) preferably is completed and the index and snapshot files that result are loaded into a database or are used to update a database, maintained on the server that accepts user search criteria and composes and sends to the user the search results.
Web Agent A attempts sequentially (or randomly or otherwise) to load all the web pages listed in a large database of URL addresses that were compiled previously from various sources. A compilation of URL addresses might be built up by trying to download composed URLs based on dictionary words (e.g., http://www.aardvark.com . . . ) or company names from a name directory (e.g., http://www.acme.com . . . ) or known URLs from a domain name service, or even all sequential string combinations one after another. The tried and true way to compile a list of addresses for a web crawl is to start with URL addresses from an existing compilation of web page addresses, such as a domain name listing; to load each one sequentially; and to scan through the source of the loaded pages for all the hypertext links to other URL domain names and/or URL web page addresses. These latter linked web pages are then added to the compilation of URL addresses, and crawled (loaded and also scanned for links) at some later time.
The search system of the invention preferably permits anyone to suggest a web page to be added to the universe of searchable pages. The suggested web page is added to the compilation, and the search engine's robots crawl the web by loading the suggested page, noting and loading the pages linked to the suggested page and continuing on to the pages that are linked to the linked pages, etc. Duplicates are removed. URLs that have been recently visited can be flagged for deferred reload, or removed.
Another preferred method incorporates the use of a human reviewed and compiled database. A “human surfer” or web page reviewer may be more dependable than a robotic agent in categorizing the content of web sites (e.g., “The Electric Factory” is the identifier of a concert promoter and supplier of tickets to entertainment events). Both methods can be utilized to compile a database of websites. A team of human surfers can be employed for the task, each visiting successive websites and making determinations, for example, as to an appropriate title, description, category or the like. The current invention provides additional enhancement to a human compiled database in that the content of a website is even more quickly apparent if any descriptive terms or titles are considered together with a snapshot of the content, even if miniaturized to the extent that most or all of the text shown in the snapshot may be too small to be readily discerned.
In a preferred arrangement of the invention, the processing is accomplished in a network of programmed processors that are in a data communication with one another and each of which has a TCP/IP communication link to the web. The database containing the universe of crawled or to-be-crawled target web sites, which may number in the millions, can be stored in a controlling processor or can be part of a shared data store used to allocate individual URLs to client computers on the search system network, such as by permitting Web Agent A to obtain the next URL from the list and to flag the URL as in use. It is not strictly necessary to use the network paradigm. Instead, each Web Agent A or each client computer running multiple Web Agents of type A can contain its own database with a subset of the URLs of the universe, and the databases of a number of robots or clients can be synchronized periodically to eliminate duplicates, flag URLs after they have been crawled, and similarly updated. In a typical application, the database serves out a URL to the next Web Agent A in the queue and moves an index or “pointer” to refer to the next URL to be served out.
Web Agent A receives the URL, makes a TCP/IP request for the web page over the web, and attempts to download the source code and all the necessary graphic files and data needed to render that website. Web Agents of type A are preferably programmed to “patiently” request and await download of files, but also intelligent as to which of the files to ignore (for example audio files are ignored) and whether to continue to attempt downloading if successive attempts have been unsuccessful. Integrity, byte count, parity and similar checks can be performed to ensure that the download is complete and correct.
In dealing with websites containing “frames,” which are actually multiple documents that are loaded and displayed in tandem at a defined and potentially variable portion of a browser display screen, each document typically has an end-of-file code and issues a download complete message to the Operating System.
Often a framed web page can accept and display any of a number of other web pages as an inset frame. This complicates matters in that the end-of-file that actually concerns only part of the framed page might erroneously trigger the Web Agent to move on to the next website and to process the frame but not the framed content.
Frames also present a problem for the crawler robot regarding embedded html links to other web pages. The owner of a frames web page can include html links to web pages of others. If a surfing browser attempts to load the linked page by selecting (clicking on) the link on the frames web page, the browser will load the linked page but it will be within the frame of the first web page owner. The browser is not linked independently in that case and instead is linked through the frames page. Thus the html target address that appears in the browser toolbar and is recorded in the browser's history list is not a link to the selected site. Instead it is a link to the frames page, with a modifier that identifies the selected site. When that target address is invoked, the frame is loaded and the linked web page is inserted into the frame.
In queuing embedded links found on pages for processing, Web Agent A distinguishes framed links from direct links. When processing a framed page, preferably, the crawler invokes the framed page's internal links to find and queue additional links, but does not treat every framed link as a new web page. Insofar as Web Agent A encounters websites with frames, it processes the data local to that web site and checks for the presence of a website with frames. When a frame page is detected, the Web Agent A checks for a download complete message (end-of-file) for every framed element and processes the text and graphics of the frame and the contents both.
Web Agent A preferably detects dynamic occurrences that are programmed into web sites, from the html source code that is received. Agent A can keep only a portion of the content of a particular file, such as the first frame of an animated GIF, or can wholly ignore the file, such as an audio file, a data entry form script or video clip, etc. There are a variety of situations in which a web site may be arranged to display text or graphics sequentially or conditionally, or to link the user to different files. These include automatic re-routing to a further link after a delay or after a user input such as a mouse click, pop up windows for temporary display of a graphic on top of a background, CGI prompt boxes for entering data, data that varies inherently such as video windows, sound files, animated GIF images and other similar occurrences.
According to an inventive aspect, Web Agent A of the invention deals with changing data by loading as much of the text and graphic data as the target web page will supply, and storing a sufficient collection of the graphics and linked files to prepare a static version of the target page upon initial access. This requires Web Agent A to search the source code received from a site for indications of dynamic content and to suppress the dynamic aspect of the content. However, the dynamic aspect is preferably not omitted entirely, and instead is limited to a static display of the initial content encountered.
Accordingly, sound files (WAV, MID, MP3, etc.) are suppressed and ignored. For example in downloading the html source, Agent A deletes links, as a function of their file extensions, before storing the file, and of course does not attempt to download the files themselves. Animated graphics preferably are partially loaded (e.g., only the first frame of an animated GIF) or the graphic files are fully loaded by the Web Agent A but are only partly processed by the Web Agent B. Video content can be processed to obtain an initial frame, but preferably video is ignored and is replaced by a link to a static graphic that marks the video and the file type. For example, MOV video files can be marked by a static Apple Quicktime icon, or ASF files marked by a static Windows MediaPlayer static icon, etc. The static markers preferably are chosen by file extension (e.g., for video, RAM=RealPlayer, ASF=Windows MediaPlayer, MOV=Quicktime), or a generic marker is used for all these formats, or perhaps only for the generic formats that all the players can process (e.g., MPG). Either Web Agent A or Web Agent B can process the target site to link to or to present the static display marker for such files.
Similar markers can be used to indicate the presence of media that is not displayed. For example, an icon or character (e.g., “_”) can indicate when a link to an audio file is detected. As in the foregoing discussion of video, the icon also can be chosen as a function of the file extension to indicate the type of audio file found, such as WAV, MID, MP3, etc.
According to further aspects of the invention, pop up windows are ignored or suppressed. Dialog boxes, unlike pop up windows, are somewhat more complex and may obstruct the display of background page features when displayed. A dialog or data-entry CGI box, may suspend the processing of a page until the dialog box is handled. Rather than permitting a dialog box such as a name or password box to suspend operation of Web Agent A, a dialog boxes is detected and triggers running of a “cancel” routine in response to a dialog box. Assuming that the site is operating password-control or a similar process, that process is discontinued for failure to enter the password or the like, but Web Agent A can continue on and may obtain additional graphic file data or text after the dialog box or similar prompt has been passed.
Animated GIFs and other changing features can also be identified by an icon indicating the presence of that feature. Preferably these animated features are selectively processed to provide a static image. Animated GIFs and some other technologies such as Macromedia Flash, provide an action sequence in the form of a plurality of images that are displayed in quick succession, normally in a loop. It is a problem with animations, especially those pertaining to Macromedia Flash Technology to select which frame will be captured or selected as representative of the animation. Animated GIFs begin with a graphic and the subsequent “frames” may be limited only to those pixels that have changed color from one frame to the next. Flash Technology usually begins with a blank screen or blank square. Choosing the first frame of a Flash movie as the designated frame to process and render would certainly be unaccepteable. According to alternative solutions, the Web Agent B can employ a timer to wait a predetermined time before capturing the rendered image in a file of the type that starts as a blank or fades in. It may be a matter of luck what in particular will be present at the moment captured in the changing portion of the display. An alternative is to generate a static image as a sum or average of two or more changing frames, which may produce a smeared static image. Another alternative is to disable the Flash plug in by a suitable message to the target site when loading the page. Disabling the Flash plug may eliminate any graphic data, namely if the website operators did not provide a static HTML page as an alternative to be presented for users who are not outfitted for Flash. Often, a user without Flash is presented with a blank screen with a tiny caption at the bottom reading “If you do not have Flash, click here.” A rendering and subsequent snapshot of a screen similar to this could be misleading to the user if viewed within the search results of a search engine, so a timed capture is preferred.
It is an aspect of the current invention to provide an icon or similar indication within the search results as to whether or not a particular website contains Flash Technology. This alleviates possible inconsistencies in processing and rendering a Flash movie, and subsequent interpretation by the user of a search engine who may be viewing the snapshots. Moreover, for Flash and similar technologies that are optional for users, adding an indication of their presence benefits users of the search results. Specifically in the case of Flash, a user who has loaded the Flash plugin or otherwise has the capability to process the content will prefer to access pages that contain Flash content if other factors are equal. Users with browsers incapable of processing Flash technology might be forewarned that their browser may have difficultly rendering that particular website, or at the least would be neutral about that aspect of the web site. The use of Flash, RealAudio and other “value added” technologies is often an indication that a particular website has superior content.
Therefore, in a preferred embodiment, the presence of Flash content is detected. A static page is captured according to one or more of the foregoing alternatives, preferably by disabling the Flash Plug-in. A conventional static graphic is displayed in the snapshot image, and adjacent to the static graphic an icon is inserted to show that the site is a Flash site. The same technique can be used to identify other dynamic displays, such as Shockwave Movies and the like, preferably using distinct icons for each type.
In the preferred arrangement shown in FIGS. 1-3 , each computer employed by the search engine system has one database, a plurality of Web Agent A's, and a single Web Agent of type B. While the Web Agent A's are occupied with downloading necessary text data and graphic files in the background, the single Web Agent B is busy in the foreground rendering pages and performing coordinate based screen captures. Most commonly, screen captures are performed at a bit depth or resolution of 24-bits, and thus comprising 16.7 million possible colors in the captured image. To minimize data overhead and to maximize efficiency, a coordinate based system is utilized to execute the desired image capture.
By operating Web Agent B in the foreground, the invention can take advantage of certain display facilities without corresponding processing overhead. Such facilities may include, as available, display processing hardware, software, firmware, coprocessors, memory caches and possibly peripherals such as display driver cards, which might normally be used to facilitate fast updates to a display during the foreground operation of a program.
According to an inventive aspect, the system as described can be configured to operate using a plurality of independent computers that are in data communication (e.g., on a common network, or having access to a particular memory store either concurrently or by virtue of preparing a mass memory media such as CD ROMs containing the storage media database, using one or more computers, and then processing the database for searches using one or more additional computers. In one particular arrangement, one computer (or a subset of a group of computers) exclusively runs Web Agent A processes, for downloading data, files, media, graphics, etc. This large number of Web Agents of type A or processes incorporating similar capabilities, deposits downloaded files into a datastore, such as a hard drive, removable drive, or the like. The deposited data can then be transferred via network or on movable media to a different computer running Web Agent of type B.
Web Agent B processes the data to provide reduced/compressed web page images or snapshots on graphic data files. This second computer running Web Agent B accesses the datastore to render and process websites according to a specified queue. The point is that it is not necessary to have both types of Web Agents on the same computer to enable proper execution of the system, and it may be efficient to separate these functions as described. Separation of the index preparation function in which the storage database is processed to ready it for searching, and the searching and reporting functions in which user queries are accepted, the storage database is searched and a report is composed and reported, can also be separated onto additional computers that each serve particular functions. In this way, operating together and preferably including allocation of additional resources at any processing and communication bottlenecks, the system can obtain data, prepare the data for searching by preprocessing the data, including producing graphic image files, and conduct and report searches via interaction with remote users.
In a preferred form of the invention, both types of Web Agents run on the same machine. When one of the Web Agents of type A downloads a web page, it stores all elements of the page, both text and graphics and including files that may be linked to each subject page but stored at a a different server address, and saves the URL address and the associated file names. The URL is added to Web Agent B's input queue. All of the Web Agents of type A perform this same process, namely attempting downloads and when a download is complete, placing that URL in the Web Agent B's queue. In this way the Web Agent B normally cannot outpace the Web Agent As, even though the latter are occupied to some extent with waiting for transmitted data to be sent by a remote website server.
The Web Agent B undertakes processing after all the files necessary to complete the processing have been downloaded and accessibly stored. For example, Web Agent A or Web Agent B (or another process such as a process that parses the received source code for text indexing) scans through the source code and thereby determines the files that are needed for processing, namely the files or addresses to which hyperlinks are found in the source code. In an embodiment in which Web Agent A handles this process, the web page can wait to be queued for Web Agent B until Web Agent A has successfully loaded and stored all the files. Alternatively, a list of the associated files can be prepared by Web Agent B or by another process with access to the source code, and Web Agent B can check the list before attempting to process the data for the web page. In any event, preferably the processing capacity of Web Agent B is substantially devoted to processing pages that are complete when their processing commences.
Web Agent A, or another process, can be arranged to continue to attempt to load any of the necessary files that has not been loaded. Alternatively, Web Agent B can undertake a new communication on the web in an effort to retrieve the missing file, or can queue one of the Web Agents of type A, or another process, to obtain the file or to reload the missing file(s) or perhaps the entire web page and associated files. Reloading the entire web page deals with the possibility that a file that is found to be missing or unavailable may no longer be linked in the web page source code, and thus is unnecessary. Discontinuance of the link might also be the reason why the file has not been found (i.e., it was removed and deleted).
Performing from locally stored text and data files, Web Agent B can render and capture graphic image files or snapshots at an improved rate. In an embodiment wherein one Agent B and several Agent A processes were operative on one computer to accumulate stored files locally and to queue Agent B, Agent B was found able to produce graphic snapshot files at a rate of about one web page per second. This is much faster than downloading and rendering one page at a time as such would be the case with a normal browser, wherein transmission waits make the typical rate one web page per 45 seconds.
The rendered image file is captured from the display buffer memory of the operating system, and then is resized, processed to increase image quality, and compressed. It is then stored on disk in a standard format under a file name associated with the URL of the originating web page.
Upon completion of a full crawl, rendering of each and every desired web site, and full data storage of the resulting graphic snapshots, the search engine database is ready to accept user queries. The user presents combinations of text string expressions in a known manner. According to the same sort of search criteria known in other search engine applications (e.g., HotBot, Alta Vista, Yahoo, etc.), the criteria are compared to the indexed text information. By whatever means used (e.g., all words, any word, exact phrase, Boolean combinations, with or without results ranking or categorization, etc.) the search engine selects and prepares a list of the web page hits discovered by comparing the search criteria to the contents of the indexed database.
A report listing is prepared by generating a reporting web page in html source code, which is then sent to the user. The reporting web page includes a list of hits wherein each entry on the list comprises an html link to the URL from which the associated web page was downloaded. Preferably, and as already done with most search engines, the entries also include at least a line or two of text from the web page, such as the first three lines. Additionally, according to the invention the entry also has an html link to the graphic file on the search engine computer where the snapshot of the rendered web page is stored. This link can be an IMG SRC=[path] [filename] command.
When the user reviews the search report using a browser, the browser inserts the graphic snapshot image adjacent to the listing of the URL link to the subject web page. Thus the user can determine whether a page entry in the search results is of interest, not only from the text information included with the URL link such as a description and title, but also from a small size presentation of what the web page looked like when it was indexed.
If the user is interested in reviewing the web page to which the search report entry is directed, the user can click on the hypertext link to the URL of the web page, whereupon the user's browser loads the web page directly from its original web page server. The snapshot image preferably is associated with the hypertext link redundantly, so that the user can click either on the hypertext link or on the snapshot image and in either case will be linked by the URL to the originating web page.
There are some timing issues. Between the time that the web page was downloaded and the time that the user clicks on a search result entry to review the page, the contents of the page may have changed. If a website operator updated or changed the layout of that website since it was rendered and processed by the snapshot software (Web Agent A and Web Agent B), it is possible that the visual aspect as seen through the user's browser no longer coincides with the snapshot image in the search results. Nevertheless, the snapshot normally shows a mostly consistent visual representation of the current content of the web page.
Numerous algorithms were tested to generate the ideal snapshot from the raw image data, (effectively to convert a bitmap image in the display memory of the computer to a GIF or JPG file to be stored on the disk of the search engine computer). It is utterly essential to the performance of the system to obtain a high image quality and small file size. However, it is a fact of computer science that these two objectives contradict each other. Under normal circumstances, you can have one or the other, but not both. The higher the image quality, the larger the file size, and subsequently, the longer a user of the search engine has to wait for the snapshot to download. On the other hand, creating a small file with less data, will result in a faster download for the user but will also result in poor, unacceptable image quality as it pertains to the snapshot. Not all algorithms are programmed the same, and in fact, some are found to be superior to others.
The algorithms necessary to control resizing, image quality, and compression are programmatically controlled to create the resultant graphic snapshots. To provide a perpetual, never ending crawl and graphical rendering of websites on the internet, it is necessary to automate all functions, including those found in commercial software so that they may be performed without human intervention. Web Agent B, upon complete rendering of a web page, programmatically manipulates the aforementioned algorithms and subsequently ensures the proper storage of the resulting graphic snapshot onto disk. Additionally, Web Agent B performs a test to determine whether the graphic snapshot is of a higher quality in GIF format or JPG format. It should be noted that new algorithms or other existing algorithms may be operable and may be preferable in other operating situations.
Obviously individually resizing, sharpening, compressing, and converting each and every bitmap screen capture to produce the desired representative snapshot would prove prohibitive. A manner by which automation and speed to perform this process is warranted. An element of the snapshot software system is to programmatically control software to perform these actions, for example by manipulation of subroutines from commercial software. This can be accomplished using C++ programming to access certain files and processes normally regarded as internal to the computer operating system. In particular, the memory locations containing the bitmap image intended for the browser display, generated by the operating system (e.g., browser, display drivers, etc.) is co-opted and used as the source file for generation of a compressed graphic image file in an efficient format for storage and data transmission. In particular, a bitmap-to-GIF or bitmap-to-JPG conversion is effected on the contents of the display buffer stored in RAM. Exactly which conversion is determined by Web Agent B.
Upon the completion of processing the original bitmap screen capture into a snapshot, all the raw data files used to render the image now captured are deleted to prevent the data store from overfilling. That is, the original html source code can be deleted together with the graphic files addressed in the source code and downloaded for inclusion in the rendering, leaving in storage only the representation of the web site in the database by its URL address, its text indexing and/or categorization and the ultimate graphic snapshot in an image file cross referenced to the stored address. The bit size of the graphic snapshot file is approximately 1/200 the size of the original data. By automatically and continuously deleting raw files from the data store after processing, the necessary data for searching and reporting with a rendered depiction remains available (albeit a small depiction), and storage capacity requirements remain manageable.
A popular search engine or website portal may receive numerous server requests per day. The search engine opening page may have numerous included text, graphic and interactive elements, each element requiring a communication request for transmission of a data file. Top search engines are visited by millions of users every day, and each search can generate numerous “hits”. Some of the search portals personalize the presentation to users. If a search engine is visited by millions of users per day, it has to serve multiple millions of operations and data transfers. According to the present invention, the search engine can report improved information without the corresponding overhead.
Search engine visitors are very impatient and tests show that they are not willing to wait very long for results to be reported. The invention expedites the search and search reporting process while improving the content of the results, and encourages users to remain loyal to their preferred search engine.
Conventional search engines report results in a format that apart from advertising and preset information is limited to text and text formatted as links to the URL addresses of pages in the hit list. This textual form can be reported in a very small file size as compared to the number of hits reported, thereby limiting server overhead and decreasing internet download time. If a browser was arranged to attempt to load and render snapshots during the receipt and display of a search report, a serious technical challenge and communication load would result.
According to a further aspect of the present invention, the snapshot rendering feature is preferably enabled and disabled by user option, which can be a point upon which the user personalizes his/her access to the web portal containing the search engine of the invention. For this purpose, the user can be assigned a code that is stored in a cookie that is sent to or made available to the portal, or a cookie containing bit flags in which the user can set and unset options such as snapshot reporting.
It is a further aspect of the invention to employ a system comprising a plurality of optimized “snapshot servers”. The snapshot servers access and deliver graphic snapshots from storage, to the network address of a user to whom a search is being reported. The snapshot servers can conduct packet data transmission, serve requests for re-send, etc. The snapshot servers remove the overhead of reporting graphic files from the searching processes, and thus ensure that search reports are as quick as possible. The search process passes the graphic file names and user network address to the snapshot servers. The snapshot servers transmit the graphic snapshot files corresponding to each search report following shortly after the search report text.
In a preferred embodiment, the snapshot servers employ a RAM cache for storage of some or all of the snapshot images to be reported to users. This contributes further to the reporting speed because it is not necessary to await the addressing and loading of image snapshot files from the system hard disk and the hard disk does not become an undue bottleneck. Upon system startup the library of quick access snapshot graphics can be copied from a hard drive into the RAM cache. The cached files can be all of the snapshot graphic files or only those found after experience to be most frequently addressed. The snapshot servers preferably share or employ a large cache, for example at least 1 Gigabyte and preferably 10 Gigs or greater.
For determining the frequency of addressing, the snapshot servers preferably contain a program or process that counts or calculates the two million most requested snapshots. This can be updated on a weekly basis. Although any number of snapshots could be maintained within rapid access of the search engine's database, a predetermined number of those found to be most requested, such as two million, are kept directly in the memory cache (hence a cache size of 10 Gigs, or approximately 5 KBytes per image). The status of a given page as being among the predetermined number (e.g., two million) that are most often requested or at least most often reported in searches, can be indicated in the graphic results, for example by adding a frame to the snapshot that is reported by transmitting an additional frame graphic.
In a preferred embodiment, the textual portion of search results always is sent and caused to appear first, prior to the snapshots corresponding to those results. As a result, regardless of whether the user has turned the snapshots capability “ON” or “OFF”, the text portion appears first. If a user so desires, he can abort the transmission of the results based on review of the initially received portion. This is accomplished through programming within the snapshot server system that queues the text portion of the search results to be “released” or transmitted first, preferably even before addressing (or perhaps even checking for the presence on the corresponding snapshots.
A number of additional variations and further embodiments are possible and will become apparent to persons skilled in the art in view of this disclosure. The invention is not intended as limited to the precise arrangements disclosed as examples. Accordlingly, reference should be made to the appended claims for assessing the scope of exclusive rights claimed.
Claims (44)
1. A method for processing data files stored at distributed addresses on a data processing network, at least some of the data files having text and graphic content, the method comprising:
analyzing at least a subset of the data files to produce a database of information characterizing aspects of the data files that tend to distinguish the data files from one another, and cross referencing said information to addresses of the data files;
generating an image of at least a portion of the subset of data files, and storing a graphic file of said image in a manner cross referenced to the addresses of the data files, whereby the graphic file represents an image of the data files at a time of generation;
receiving search queries and applying the search queries to the database for selecting a hit list from among the data files;
reporting the hit list in a search report including the addresses of each of the data files selected and the image corresponding to the data files in the hit list at the respective time of generation.
2. The method of claim 1 , wherein the data files comprise hypertext markup language text and linked graphic format files, and wherein said analyzing comprises at least one of indexing the text and reviewing at least a portion of the data files for assignment of an arbitrary categorization.
3. The method of claim 2 , wherein the data files comprise hypertext markup language text and linked graphic format files on one of an intranet and the World Wide Web, and wherein said generating comprises rendering an image corresponding to the data files according to a predetermined display configuration defining a default choice of at least one of a pixel display size, font type, color pallet, color resolution and use of colors.
4. The method of claim 2 , wherein said analyzing and said generating are accomplished using at least two processes, one of said processes collecting the hypertext markup language text and the linked graphic format files and another of said processes rendering the graphic files as a presentation of respective said data files.
5. The method of claim 4 , comprising a greater number of said processes collecting the files than a number of said processes rendering the presentation.
6. The method of claim 4 , comprising storing in a buffer each of the files collected by said processes collecting the files, queuing the process for generating the image, and deleting the files in the buffer after generating the image.
7. The method of claim 4 , comprising operating said process rendering the presentation file using at least part of a computer's display facility to produce a bitmap, and converting the bitmap into a graphic format file.
8. The method of claim 4 , comprising operating said process rendering the presentation file by reducing a display size of the bitmap and converting the bitmap into a graphic format file.
9. The method of claim 2 , wherein said reporting of the hit list comprises composing a hypertext report page containing selectable links for addressing corresponding said data files, and transmitting the hypertext report page to a user submitting a query, and wherein the report page additionally includes an image link addressing the graphic file for at least a portion of the hits.
10. The method of claim 1 , further comprising stripping at least one variable aspect of the data files, said aspect comprising at least one of a time changing display feature, a user interactive feature and a nonvisual media feature.
11. A network search engine for managing user selection of information contained on data files stored at distributed network addresses on a global information processing network wherein distributed users have control over associated data files accessible by other users, each of said data files having at least some associated text and each of the data files having at least one mode of graphic presentation, comprising:
a crawler having at least one processor operable to address and load successive data files comprising at least a subset of said data files stored at said distributed network addresses, the crawler being operable to produce and store a database of information characterizing aspects of the data files that tend to distinguish the data files from one another, cross referenced to addresses of the data files; and,
wherein the crawler is further operable to produce graphic image files representing at least some of the data files, the graphic image files each corresponding to content of corresponding said data files at a point in time, and wherein the crawler is operable to store the graphic image file so as to cross reference the graphic image file to the data files in the database.
12. The network search engine of claim 11 , further comprising programmed processes operable to receive search queries from network users, to apply the search queries to the database for selecting a hit list from among the data files and to report the hit list in a search report including the addresses of each of the data files selected and the image corresponding to the data files in the hit list at the respective time of generation.
13. The network search engine of claim 11 , wherein the data files comprise hypertext markup language text and linked graphic format files, and wherein said analyzing comprises at least one of: indexing the text for storing a text index cross referenced to network addresses of the data files; and reviewing at least a portion of the data files for assignment of an arbitrary categorization and for storing a categorization cross referenced to the network addresses of the data files.
14. The network search engine of claim 11 , wherein the crawler operates at least two discrete processes for collecting the data files and files linked thereto, and for producing the graphic image files.
15. The network search engine of claim 14 , wherein the discrete processes operate together on at least one processor, and wherein the processes for collecting are more numerous than at least one said process for producing the graphic image files.
16. The network search engine of claim 15 , wherein the process for producing the graphic image file renders an image of the data files from downloaded copies of the hypertext markup language text and linked graphic format files, and converts a resulting display image file into the graphic format file.
17. The network search engine of claim 16 , wherein the process for producing the graphic image file renders the image of the data files according to a configuration selected as a default configuration with respect to at least one of use of changing visual features, presentation of user interactive features, presentation of non-visual media, display pixel resolution, color pallette, color resolution, and use of colors.
18. The network search engine of claim 17 , further comprising a programmed process for producing the graphic image file, which co-opts a display bitmap from a processor programmed to present the data files, and converts the bitmap to a compressed graphic format file stored on the search engine.
19. An improved Internet search engine for managing user search and selection of web pages stored at distributed systems coupled at network addresses to the Internet, the search engine having an associated web crawler operable to address and load successive web pages, and to index text data associated with said successive web pages so as to obtain parameter information that distinguishes at least groups of the web pages from one another, the crawler storing the parameter information and associated addresses of the web pages, and the search engine being operable responsive to user submitted search criteria to search the parameter information and to report at least the associated addresses of web pages that met the search criteria when indexed, wherein the improvement comprises:
said crawler being operable in conjunction with obtaining the parameter information for at least a subset of said successive web pages to generate a graphic image file containing a visual image that is substantially identical to an appearance of said web pages, for display in a size proportionally smaller than said web pages; and
wherein the search engine is operable when reporting the associated addresses of web pages that met the search criteria to include a representation of the graphic image file in said proportionally smaller size.
20. The improved Internet search engine of claim 19 , wherein the crawler generates the graphic image file with an appearance of the web pages according to a predetermined default display configuration of a browser.
21. The improved Internet search engine of claim 20 wherein the predetermined default display configuration defines a selection of at least one of relative font size and type, colors and pixel aspect ratio.
22. The improved Internet search engine of claim 21 , wherein the search engine reports to the user the associated addresses of the web pages that met the search criteria, in a form of hypertext source data containing URL links to said web pages, and wherein the graphic image file is displayed in association with a URL link to the web page represented by the graphic image file.
23. The improved Internet search engine of claim 21 , wherein the graphic image file comprises a compressed pixel image of a bitmap corresponding to said web pages.
24. The improved Internet search engine of claim 22 , wherein the graphic image file is transmitted as an image link in the hypertext source data to a file compressed by at least one of MIME, Binhex and Base64.
25. A system comprising:
a fetching agent configured to receive a website file via at least one network interface, wherein the website file is associated with a web page;
a rendering agent configured to generate, based on the website file, a visual representation file that represents a rendered appearance of the web page that is substantially identical to an appearance of the web page and to compress the visual representation file of the web page into a reduced image file,
wherein the reduced image file represents a reduced-size rendered appearance of the web page for display in a size proportionally smaller than the web page, and
wherein the rendering agent is further configured to limit a dynamic aspect of dynamic content in the website file to a static display, wherein the static display comprises an image from the dynamic content in the web page at a fixed time;
a memory, configured to store the reduced image file and at least one network address associated with a network location of the website file, wherein the memory is further configured to cross reference the reduced image file with the at least one network address; and
a first plurality of fetching agents and a second plurality of rendering agents, and wherein a ratio of the first plurality of fetching agents to the second plurality of rendering agents is modified during processing of website files to maintain a consumption of the memory within a range of fractions of a capacity of the memory.
26. The system of claim 25, wherein the fetching agent is further configured to utilize a plurality of concurrently active requests for web pages.
27. The system of claim 25, wherein the fetching agent is further configured to remove dynamic content from the website file.
28. The system of claim 25, wherein the web page includes nontext data.
29. The system of claim 25, further comprising:
a search portal configured to:
receive search criteria; and
generate a report based on the search criteria, the report comprising the at least one network address and one of:
a link to the reduced image file, and
a copy of the reduced image file.
30. The system of claim 25, wherein the at least one network address comprises a uniform resource locator (URL) address of the web page.
31. The system of claim 25, wherein the static display comprises a frame of the dynamic content at the fixed time.
32. The system of claim 25, wherein the rendering agent is further configured to include an icon in the reduced image file indicating that the web page includes dynamic content.
33. The system of claim 32, wherein the icon indicates a type of the dynamic content.
34. A method comprising:
receiving, at a computer, a website file via at least one network interface, wherein the website file is associated with a web page;
generating, at the computer, a visual representation file that represents a rendered appearance of the web page that is based on the website file, wherein the generating limits a dynamic aspect of dynamic content in the website file to a static display, wherein the static display comprises an image from the dynamic content in the web page at a fixed time;
compressing, at the computer, the visual representation file of the web page into a reduced image file, wherein the reduced image file represents a reduced-size rendered appearance of the web page;
storing, at the computer, the reduced image file and at least one network address associated with a network location of the website file, wherein storing the reduced image file and the at least one network address comprises cross referencing the reduced image file with the at least one network address; and
modifying a ratio of first plurality of fetching agents to second plurality of rendering agents during processing of website files to maintain a consumption of memory within a range of fractions of a capacity of the memory.
35. The method of claim 34, wherein the web page includes nontext data.
36. The method of claim 34, further comprising:
receiving search criteria; and
generating a report based on the search criteria, the report comprising the at least one network address and one of:
a link to the reduced image file, and
a copy of the reduced image file.
37. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions comprising:
instructions for receiving a website file via at least one network interface, wherein the website file is associated with a web page,
instructions for generating, based on the website file, a visual representation file that represents a rendered appearance of the web page, wherein the generating limits a dynamic aspect of dynamic content in the website file to a static display, and wherein the static display comprises an image from the dynamic content in the web page at a fixed time;
instructions for compressing the visual representation file of the web page into a reduced image file, wherein the reduced image file represents a reduced-size rendered appearance of the web page;
instructions for storing the reduced image file and at least one network address associated with a network location of the website file, wherein the instructions for storing the reduced image file comprise instructions for cross referencing the reduced image file with the at least one network address; and
instructions for modifying a ratio of first plurality of fetching agents to second plurality of rendering agents during processing of website files to maintain a consumption of memory within a range of fractions of a capacity of the memory.
38. The non-transitory computer-readable storage medium of claim 37, wherein the instructions for receiving, the instructions for generating, the instructions for compressing, and the instructions for storing are configured to operate on a single processor.
39. The non-transitory computer-readable storage medium of claim 37, wherein the instructions for receiving, the instructions for generating, the instructions for compressing, and the instructions for storing are configured to operate on multiple processors.
40. The non-transitory computer-readable storage medium of claim 39, wherein at least one of the multiple processors exclusively executes the instructions for receiving.
41. The non-transitory computer-readable storage medium of claim 37, wherein the web page includes nontext data.
42. The non-transitory computer-readable storage medium of claim 37, further comprising:
instructions for receiving search criteria; and
instructions for generating a report based on the search criteria, the report comprising the at least one network address and one of:
a link to the reduced image file, and
a copy of the reduced image file.
43. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions comprising:
instructions for receiving a file via at least one network interface, wherein the file includes formatting information;
instructions for generating, based on the formatting information, a visual representation file that represents a rendered appearance of the file, wherein the generating limits a dynamic aspect of dynamic content in the file to a static display, wherein the static display comprises an image from the dynamic content in a web page at a fixed time;
instructions for compressing the visual representation file into a reduced image file, wherein the reduced image file represents a reduced-size rendered appearance of the file;
instructions for storing the reduced image file and at least one network address associated with a network location of the file, wherein the instructions for storing the reduced image file comprise instructions for cross referencing the reduced image file with the at least one network address; and
instructions for modifying a ratio of first plurality of fetching agents to second plurality of rendering agents during processing of website files to maintain a consumption of memory within a range of fractions of a capacity of the memory.
44. The non-transitory computer-readable storage medium of claim 43, further comprising:
instructions for receiving search criteria; and
instructions for generating a report based on the search criteria, the report comprising the at least one network address and one of:
a link to the reduced image file, and
a copy of the reduced image file.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/513,423 USRE46967E1 (en) | 2000-04-27 | 2006-08-31 | System, apparatus, method, and computer program product for indexing a file |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/559,501 US6643641B1 (en) | 2000-04-27 | 2000-04-27 | Web search engine with graphic snapshots |
US11/266,750 USRE42413E1 (en) | 2000-04-27 | 2005-11-04 | Web search engine with graphic snapshots |
US11/513,423 USRE46967E1 (en) | 2000-04-27 | 2006-08-31 | System, apparatus, method, and computer program product for indexing a file |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/559,501 Reissue US6643641B1 (en) | 2000-04-27 | 2000-04-27 | Web search engine with graphic snapshots |
Publications (1)
Publication Number | Publication Date |
---|---|
USRE46967E1 true USRE46967E1 (en) | 2018-07-24 |
Family
ID=29270918
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/559,501 Ceased US6643641B1 (en) | 2000-04-27 | 2000-04-27 | Web search engine with graphic snapshots |
US11/266,750 Expired - Lifetime USRE42413E1 (en) | 2000-04-27 | 2005-11-04 | Web search engine with graphic snapshots |
US11/513,423 Expired - Lifetime USRE46967E1 (en) | 2000-04-27 | 2006-08-31 | System, apparatus, method, and computer program product for indexing a file |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/559,501 Ceased US6643641B1 (en) | 2000-04-27 | 2000-04-27 | Web search engine with graphic snapshots |
US11/266,750 Expired - Lifetime USRE42413E1 (en) | 2000-04-27 | 2005-11-04 | Web search engine with graphic snapshots |
Country Status (1)
Country | Link |
---|---|
US (3) | US6643641B1 (en) |
Families Citing this family (495)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3484325B2 (en) * | 1997-09-02 | 2004-01-06 | 富士通株式会社 | Agent system device with conditional answer and program storage medium |
US6594662B1 (en) * | 1998-07-01 | 2003-07-15 | Netshadow, Inc. | Method and system for gathering information resident on global computer networks |
US7346605B1 (en) * | 1999-07-22 | 2008-03-18 | Markmonitor, Inc. | Method and system for searching and monitoring internet trademark usage |
US6983311B1 (en) * | 1999-10-19 | 2006-01-03 | Netzero, Inc. | Access to internet search capabilities |
US6832245B1 (en) | 1999-12-01 | 2004-12-14 | At&T Corp. | System and method for analyzing communications of user messages to rank users and contacts based on message content |
US20020038348A1 (en) * | 2000-01-14 | 2002-03-28 | Malone Michael K. | Distributed globally accessible information network |
US8019757B2 (en) * | 2000-01-14 | 2011-09-13 | Thinkstream, Inc. | Distributed globally accessible information network implemented to maintain universal accessibility |
US7114124B2 (en) * | 2000-02-28 | 2006-09-26 | Xerox Corporation | Method and system for information retrieval from query evaluations of very large full-text databases |
US9736209B2 (en) | 2000-03-17 | 2017-08-15 | Facebook, Inc. | State change alerts mechanism |
US7624172B1 (en) | 2000-03-17 | 2009-11-24 | Aol Llc | State change alerts mechanism |
US20020087577A1 (en) * | 2000-05-31 | 2002-07-04 | Manjunath Bangalore S. | Database building method for multimedia contents |
US6670968B1 (en) * | 2000-07-10 | 2003-12-30 | Fuji Xerox Co., Ltd. | System and method for displaying and navigating links |
US6666377B1 (en) | 2000-07-18 | 2003-12-23 | Scott C. Harris | Bar code data entry device |
GB0019151D0 (en) * | 2000-08-07 | 2000-09-27 | Pace Micro Tech Plc | Deferred internet page reformatting |
US7007008B2 (en) * | 2000-08-08 | 2006-02-28 | America Online, Inc. | Category searching |
US7359951B2 (en) * | 2000-08-08 | 2008-04-15 | Aol Llc, A Delaware Limited Liability Company | Displaying search results |
US7225180B2 (en) * | 2000-08-08 | 2007-05-29 | Aol Llc | Filtering search results |
US7047229B2 (en) * | 2000-08-08 | 2006-05-16 | America Online, Inc. | Searching content on web pages |
US6957224B1 (en) * | 2000-09-11 | 2005-10-18 | International Business Machines Corporation | Efficient retrieval of uniform resource locators |
JP2004515846A (en) * | 2000-10-03 | 2004-05-27 | ラングフォード,ロナルド,エヌ | How to find web pages by using visual images |
US7143143B1 (en) * | 2000-10-27 | 2006-11-28 | Microsoft Corporation | System and method for distributed caching using multicast replication |
US7043473B1 (en) * | 2000-11-22 | 2006-05-09 | Widevine Technologies, Inc. | Media tracking system and method |
US7653551B2 (en) * | 2000-12-05 | 2010-01-26 | Ipwealth.Com, Inc. | Method and system for searching and submitting online via an aggregation portal |
US7409382B2 (en) * | 2000-12-08 | 2008-08-05 | Fujitsu Limited | Information processing system, terminal device, method and medium |
KR20020046647A (en) * | 2000-12-15 | 2002-06-21 | 윤종용 | Method for transmitting display data of mobile telecommunication terminal |
US20020078099A1 (en) * | 2000-12-15 | 2002-06-20 | International Business Machines Corporation | Method and system for off-loading and retrieving document content in a document processing system |
JP2002183203A (en) * | 2000-12-18 | 2002-06-28 | Yamaha Corp | Information retrieving method and information storage medium |
ATE518194T1 (en) * | 2001-01-19 | 2011-08-15 | Panasonic Corp | DEVICE AND PROGRAM FOR DISPLAYING HYPERTEXT INCLUDING A FUNCTION FOR PREVIOUS HISTORY |
US20020138621A1 (en) * | 2001-02-08 | 2002-09-26 | Rutherford Jan R. | System and method for displaying remotely stored content on a web page |
US6813620B2 (en) * | 2001-03-07 | 2004-11-02 | Broadcom Corporation | Binary search engine and method |
US7620622B1 (en) * | 2001-03-08 | 2009-11-17 | Yahoo! Inc. | Method and system for indexing information and providing results for a search including objects having predetermined attributes |
US6928429B2 (en) * | 2001-03-29 | 2005-08-09 | International Business Machines Corporation | Simplifying browser search requests |
US7461369B2 (en) * | 2001-03-30 | 2008-12-02 | Bmc Software, Inc. | Java application response time analyzer |
US7792948B2 (en) * | 2001-03-30 | 2010-09-07 | Bmc Software, Inc. | Method and system for collecting, aggregating and viewing performance data on a site-wide basis |
US7506047B2 (en) * | 2001-03-30 | 2009-03-17 | Bmc Software, Inc. | Synthetic transaction monitor with replay capability |
US20040054968A1 (en) * | 2001-07-03 | 2004-03-18 | Daniel Savage | Web page with system for displaying miniature visual representations of search engine results |
US6981029B1 (en) * | 2001-07-17 | 2005-12-27 | Cisco Technology, Inc. | System and method for processing a request for information in a network |
US6990494B2 (en) * | 2001-07-27 | 2006-01-24 | International Business Machines Corporation | Identifying links of interest in a web page |
US7130841B1 (en) * | 2001-07-31 | 2006-10-31 | America Online, Inc. | Enabling a search for both local and remote electronic content |
US7483872B2 (en) * | 2001-08-23 | 2009-01-27 | Michael Meiresonne | Supplier identification and locator system and method |
US8214342B2 (en) * | 2001-08-23 | 2012-07-03 | Michael Meiresonne | Supplier identification and locator system and method |
US7089233B2 (en) * | 2001-09-06 | 2006-08-08 | International Business Machines Corporation | Method and system for searching for web content |
US7774711B2 (en) | 2001-09-28 | 2010-08-10 | Aol Inc. | Automatic categorization of entries in a contact list |
US20030105794A1 (en) * | 2001-11-09 | 2003-06-05 | Jasinschi Radu S. | Systems for sensing similarity in monitored broadcast content streams and methods of operating the same |
US6910077B2 (en) * | 2002-01-04 | 2005-06-21 | Hewlett-Packard Development Company, L.P. | System and method for identifying cloaked web servers |
US6829617B2 (en) * | 2002-02-15 | 2004-12-07 | International Business Machines Corporation | Providing a snapshot of a subset of a file system |
US7716161B2 (en) * | 2002-09-24 | 2010-05-11 | Google, Inc, | Methods and apparatus for serving relevant advertisements |
US7171625B1 (en) * | 2002-06-18 | 2007-01-30 | Actify, Inc. | Double-clicking a point-and-click user interface apparatus to enable a new interaction with content represented by an active visual display element |
US7568002B1 (en) | 2002-07-03 | 2009-07-28 | Sprint Spectrum L.P. | Method and system for embellishing web content during transmission between a content server and a client station |
US7801945B1 (en) | 2002-07-03 | 2010-09-21 | Sprint Spectrum L.P. | Method and system for inserting web content through intermediation between a content server and a client station |
US7360210B1 (en) | 2002-07-03 | 2008-04-15 | Sprint Spectrum L.P. | Method and system for dynamically varying intermediation functions in a communication path between a content server and a client station |
US7949689B2 (en) * | 2002-07-18 | 2011-05-24 | Accenture Global Services Limited | Media indexing beacon and capture device |
US20040030697A1 (en) * | 2002-07-31 | 2004-02-12 | American Management Systems, Inc. | System and method for online feedback |
US7418661B2 (en) * | 2002-09-17 | 2008-08-26 | Hewlett-Packard Development Company, L.P. | Published web page version tracking |
US7640306B2 (en) | 2002-11-18 | 2009-12-29 | Aol Llc | Reconfiguring an electronic message to effect an enhanced notification |
US8122137B2 (en) | 2002-11-18 | 2012-02-21 | Aol Inc. | Dynamic location of a subordinate user |
US8005919B2 (en) | 2002-11-18 | 2011-08-23 | Aol Inc. | Host-based intelligent results related to a character stream |
US8701014B1 (en) | 2002-11-18 | 2014-04-15 | Facebook, Inc. | Account linking |
US7590696B1 (en) | 2002-11-18 | 2009-09-15 | Aol Llc | Enhanced buddy list using mobile device identifiers |
US7428580B2 (en) | 2003-11-26 | 2008-09-23 | Aol Llc | Electronic message forwarding |
US8965964B1 (en) | 2002-11-18 | 2015-02-24 | Facebook, Inc. | Managing forwarded electronic messages |
WO2004046867A2 (en) | 2002-11-18 | 2004-06-03 | America Online, Inc. | People lists |
US7899862B2 (en) | 2002-11-18 | 2011-03-01 | Aol Inc. | Dynamic identification of other users to an online user |
JP4024137B2 (en) * | 2002-11-28 | 2007-12-19 | 沖電気工業株式会社 | Quantity expression search device |
JP2004178490A (en) * | 2002-11-29 | 2004-06-24 | Oki Electric Ind Co Ltd | Numerical value information search device |
US7925246B2 (en) | 2002-12-11 | 2011-04-12 | Leader Technologies, Inc. | Radio/telephony interoperability system |
US8195714B2 (en) | 2002-12-11 | 2012-06-05 | Leaper Technologies, Inc. | Context instantiated application protocol |
US7640336B1 (en) | 2002-12-30 | 2009-12-29 | Aol Llc | Supervising user interaction with online services |
US7945674B2 (en) | 2003-04-02 | 2011-05-17 | Aol Inc. | Degrees of separation for handling communications |
US7263614B2 (en) | 2002-12-31 | 2007-08-28 | Aol Llc | Implicit access for communications pathway |
US7593915B2 (en) * | 2003-01-07 | 2009-09-22 | Accenture Global Services Gmbh | Customized multi-media services |
US20040135902A1 (en) * | 2003-01-09 | 2004-07-15 | Eventshots.Com Incorporated | Image association process |
US20040172451A1 (en) * | 2003-02-27 | 2004-09-02 | Nigel Biggs | System and method for sharing digital images |
US7337030B2 (en) * | 2003-03-12 | 2008-02-26 | Right Hemisphere Limited | Automated derivative view rendering system |
US7092974B2 (en) * | 2003-03-12 | 2006-08-15 | Right Hemisphere Limited | Digital asset server and asset management system |
US20040181515A1 (en) * | 2003-03-13 | 2004-09-16 | International Business Machines Corporation | Group administration of universal resource identifiers with members identified in search result |
US20040237037A1 (en) * | 2003-03-21 | 2004-11-25 | Xerox Corporation | Determination of member pages for a hyperlinked document with recursive page-level link analysis |
US20050188300A1 (en) * | 2003-03-21 | 2005-08-25 | Xerox Corporation | Determination of member pages for a hyperlinked document with link and document analysis |
US7613776B1 (en) | 2003-03-26 | 2009-11-03 | Aol Llc | Identifying and using identities deemed to be known to a user |
EP1462952B1 (en) * | 2003-03-27 | 2007-08-29 | Exalead | Method for indexing and searching a collection of internet documents |
DE10320711A1 (en) * | 2003-05-08 | 2004-12-16 | Siemens Ag | Method and arrangement for setting up and updating a user interface for accessing information pages in a data network |
JP2006526837A (en) * | 2003-06-03 | 2006-11-24 | 株式会社Access | How to browse content using page save file |
JP2005038414A (en) * | 2003-06-30 | 2005-02-10 | Fuji Photo Film Co Ltd | File management program, file management method, file management device, imaging device, and recording medium |
US7653693B2 (en) | 2003-09-05 | 2010-01-26 | Aol Llc | Method and system for capturing instant messages |
US7567979B2 (en) * | 2003-08-15 | 2009-07-28 | Microsoft Corporation | Expression-based web logger for usage and navigational behavior tracking |
US20050055265A1 (en) * | 2003-09-05 | 2005-03-10 | Mcfadden Terrence Paul | Method and system for analyzing the usage of an expression |
WO2005031526A2 (en) * | 2003-09-23 | 2005-04-07 | Amazon.Com, Inc. | Personalized searchable library based on user ownership |
US7496560B2 (en) * | 2003-09-23 | 2009-02-24 | Amazon Technologies, Inc. | Personalized searchable library with highlighting capabilities |
US20070136348A1 (en) * | 2003-10-27 | 2007-06-14 | Koninklijke Philips Electronics N.V. | Screen-wise presentation of search results |
US8234373B1 (en) | 2003-10-27 | 2012-07-31 | Sprint Spectrum L.P. | Method and system for managing payment for web content based on size of the web content |
US20050097080A1 (en) * | 2003-10-30 | 2005-05-05 | Kethireddy Amarender R. | System and method for automatically locating searched text in an image file |
WO2005048052A2 (en) * | 2003-11-05 | 2005-05-26 | Dipsie, Inc. | Identifying cataloging and retrieving web pages using client-side scripting and web forms |
US20050111756A1 (en) * | 2003-11-25 | 2005-05-26 | Turner Robert W. | System and method for generating coherent data sets of images from various sources |
US7548968B1 (en) | 2003-12-10 | 2009-06-16 | Markmonitor Inc. | Policing internet domains |
US7984175B2 (en) * | 2003-12-10 | 2011-07-19 | Mcafee, Inc. | Method and apparatus for data capture and analysis system |
US7299222B1 (en) * | 2003-12-30 | 2007-11-20 | Aol Llc | Enhanced search results |
US20050182726A1 (en) * | 2004-02-17 | 2005-08-18 | Avocent Corporation | Network virtual computing devices and framework |
US7769874B2 (en) * | 2004-02-20 | 2010-08-03 | Akamai Technologies, Inc. | Highly scalable, fault-tolerant file transport using vector-exchange |
US7657124B2 (en) * | 2004-02-27 | 2010-02-02 | The Boeing Company | Multiple image data source information processing systems and methods |
US7287042B1 (en) * | 2004-03-03 | 2007-10-23 | A9.Com, Inc. | Search engine system supporting inclusion of unformatted search string after domain name portion of URL |
US8898239B2 (en) | 2004-03-05 | 2014-11-25 | Aol Inc. | Passively populating a participant list with known contacts |
US8595146B1 (en) | 2004-03-15 | 2013-11-26 | Aol Inc. | Social networking permissions |
US20050256868A1 (en) * | 2004-03-17 | 2005-11-17 | Shelton Michael J | Document search system |
US7853782B1 (en) | 2004-04-14 | 2010-12-14 | Sprint Spectrum L.P. | Secure intermediation system and method |
US8522131B1 (en) * | 2004-04-14 | 2013-08-27 | Sprint Spectrum L.P. | Intermediation system and method for enhanced rendering of data pages |
US9172679B1 (en) | 2004-04-14 | 2015-10-27 | Sprint Spectrum L.P. | Secure intermediation system and method |
US7421322B1 (en) | 2004-04-30 | 2008-09-02 | Carfax, Inc. | System and method for automatic identification of vehicle identification number |
US20100217686A1 (en) * | 2004-05-03 | 2010-08-26 | Superlative, Inc. | System for managing communication between a real estate agent and clients |
US7693856B2 (en) * | 2004-06-25 | 2010-04-06 | Apple Inc. | Methods and systems for managing data |
US8131674B2 (en) | 2004-06-25 | 2012-03-06 | Apple Inc. | Methods and systems for managing data |
US7730012B2 (en) * | 2004-06-25 | 2010-06-01 | Apple Inc. | Methods and systems for managing data |
US9026534B2 (en) * | 2004-07-21 | 2015-05-05 | Cisco Technology, Inc. | Method and system to collect and search user-selected content |
GB2417342A (en) * | 2004-08-19 | 2006-02-22 | Fujitsu Serv Ltd | Indexing system for a computer file store |
US7540051B2 (en) * | 2004-08-20 | 2009-06-02 | Spatial Systems, Inc. | Mapping web sites based on significance of contact and category |
US8341135B2 (en) * | 2004-09-07 | 2012-12-25 | Interman Corporation | Information search provision apparatus and information search provision system |
US7512973B1 (en) | 2004-09-08 | 2009-03-31 | Sprint Spectrum L.P. | Wireless-access-provider intermediation to facilliate digital rights management for third party hosted content |
JP4557650B2 (en) * | 2004-09-13 | 2010-10-06 | キヤノン株式会社 | COMMUNICATION SYSTEM, COMMUNICATION DEVICE, AND CONTROL METHOD THEREOF |
US7383510B2 (en) * | 2004-09-13 | 2008-06-03 | Third Generation Search, Ltd | System for organization, display, and navigation of digital information |
US8522148B1 (en) | 2004-09-13 | 2013-08-27 | Pran, Cho & Shici Pte., Llc | System for organization, display, and navigation of site information |
JP2006092099A (en) * | 2004-09-22 | 2006-04-06 | Fuji Xerox Co Ltd | Image forming method and image forming device |
US8364668B2 (en) * | 2004-09-29 | 2013-01-29 | Google Inc. | User interfaces for a document search engine |
US20060085741A1 (en) * | 2004-10-20 | 2006-04-20 | Viewfour, Inc. A Delaware Corporation | Method and apparatus to view multiple web pages simultaneously from network based search |
US7600011B1 (en) | 2004-11-04 | 2009-10-06 | Sprint Spectrum L.P. | Use of a domain name server to direct web communications to an intermediation platform |
US20060100982A1 (en) * | 2004-11-08 | 2006-05-11 | International Business Machines Corporation | Storage configuration loader with automatic error recovery |
US8001476B2 (en) | 2004-11-16 | 2011-08-16 | Open Text Inc. | Cellular user interface |
US8418075B2 (en) * | 2004-11-16 | 2013-04-09 | Open Text Inc. | Spatially driven content presentation in a cellular environment |
US7769579B2 (en) | 2005-05-31 | 2010-08-03 | Google Inc. | Learning facts from semi-structured text |
US20060184655A1 (en) * | 2004-12-30 | 2006-08-17 | Brandon Shalton | Traffic analysis |
US7953720B1 (en) | 2005-03-31 | 2011-05-31 | Google Inc. | Selecting the best answer to a fact query from among a set of potential answers |
US8682913B1 (en) | 2005-03-31 | 2014-03-25 | Google Inc. | Corroborating facts extracted from multiple sources |
US7587387B2 (en) * | 2005-03-31 | 2009-09-08 | Google Inc. | User interface for facts query engine with snippets from information sources that include query terms and answer terms |
US9208229B2 (en) | 2005-03-31 | 2015-12-08 | Google Inc. | Anchor text summarization for corroboration |
US8239394B1 (en) | 2005-03-31 | 2012-08-07 | Google Inc. | Bloom filters for query simulation |
US11232768B2 (en) | 2005-04-12 | 2022-01-25 | Douglas G. Richardson | Embedding animation in electronic mail, text messages and websites |
US12051391B2 (en) | 2005-04-12 | 2024-07-30 | Douglas G. Richardson | Embedding animation in electronic mail, text messages and websites |
US7519580B2 (en) * | 2005-04-19 | 2009-04-14 | International Business Machines Corporation | Search criteria control system and method |
US20060265388A1 (en) * | 2005-05-20 | 2006-11-23 | Woelfel Joseph K | Information retrieval system and method for distinguishing misrecognized queries and unavailable documents |
US8996470B1 (en) | 2005-05-31 | 2015-03-31 | Google Inc. | System for ensuring the internal consistency of a fact repository |
US7831545B1 (en) | 2005-05-31 | 2010-11-09 | Google Inc. | Identifying the unifying subject of a set of facts |
US7725476B2 (en) | 2005-06-14 | 2010-05-25 | International Business Machines Corporation | System and method for automated data retrieval based on data placed in clipboard memory |
US7433895B2 (en) | 2005-06-24 | 2008-10-07 | Microsoft Corporation | Adding dominant media elements to search results |
US7797299B2 (en) * | 2005-07-02 | 2010-09-14 | Steven Thrasher | Searching data storage systems and devices |
TWI284818B (en) * | 2005-07-21 | 2007-08-01 | Bridgewell Inc | Database searching engine system |
US20070033224A1 (en) * | 2005-08-04 | 2007-02-08 | Allen Richard B | Method and system for automatic construction and management of sets of hierarchical interest points |
US7725407B2 (en) * | 2005-08-11 | 2010-05-25 | International Business Machines Corporation | Method of measuring a large population of web pages for compliance to content standards that require human judgement to evaluate |
US7849093B2 (en) * | 2005-10-14 | 2010-12-07 | Microsoft Corporation | Searches over a collection of items through classification and display of media galleries |
US20070112758A1 (en) * | 2005-11-14 | 2007-05-17 | Aol Llc | Displaying User Feedback for Search Results From People Related to a User |
US20070112738A1 (en) * | 2005-11-14 | 2007-05-17 | Aol Llc | Displaying User Relevance Feedback for Search Results |
US7899829B1 (en) | 2005-12-14 | 2011-03-01 | Unifi Scientific Advances, Inc. | Intelligent bookmarks and information management system based on same |
US8533199B2 (en) | 2005-12-14 | 2013-09-10 | Unifi Scientific Advances, Inc | Intelligent bookmarks and information management system based on the same |
US8117196B2 (en) | 2006-01-23 | 2012-02-14 | Chacha Search, Inc. | Search tool providing optional use of human search guides |
US7962466B2 (en) | 2006-01-23 | 2011-06-14 | Chacha Search, Inc | Automated tool for human assisted mining and capturing of precise results |
US8266130B2 (en) | 2006-01-23 | 2012-09-11 | Chacha Search, Inc. | Search tool providing optional use of human search guides |
US8196205B2 (en) * | 2006-01-23 | 2012-06-05 | University Of Washington Through Its Center For Commercialization | Detection of spyware threats within virtual machine |
US7644373B2 (en) | 2006-01-23 | 2010-01-05 | Microsoft Corporation | User interface for viewing clusters of images |
US7725451B2 (en) * | 2006-01-23 | 2010-05-25 | Microsoft Corporation | Generating clusters of images for search results |
US7836050B2 (en) * | 2006-01-25 | 2010-11-16 | Microsoft Corporation | Ranking content based on relevance and quality |
US8954426B2 (en) | 2006-02-17 | 2015-02-10 | Google Inc. | Query language |
US8260785B2 (en) | 2006-02-17 | 2012-09-04 | Google Inc. | Automatic object reference identification and linking in a browseable fact repository |
US8055674B2 (en) | 2006-02-17 | 2011-11-08 | Google Inc. | Annotation framework |
US7925676B2 (en) | 2006-01-27 | 2011-04-12 | Google Inc. | Data object visualization using maps |
US20070185870A1 (en) | 2006-01-27 | 2007-08-09 | Hogue Andrew W | Data object visualization using graphs |
US7814040B1 (en) | 2006-01-31 | 2010-10-12 | The Research Foundation Of State University Of New York | System and method for image annotation and multi-modal image retrieval using probabilistic semantic models |
US7752060B2 (en) * | 2006-02-08 | 2010-07-06 | Health Grades, Inc. | Internet system for connecting healthcare providers and patients |
US20070203888A1 (en) * | 2006-02-24 | 2007-08-30 | Cun Wang | Simple hierarchical Web search engine |
US7493403B2 (en) | 2006-03-13 | 2009-02-17 | Markmonitor Inc. | Domain name ownership validation |
US8538822B1 (en) | 2006-03-31 | 2013-09-17 | Michael Meiresonne | Method and system to establish a communicative connection between a supplier and a consumer over the internet |
US20070299830A1 (en) * | 2006-06-26 | 2007-12-27 | Christopher Muenchhoff | Display of search results |
US20070299986A1 (en) * | 2006-06-27 | 2007-12-27 | Craig Jeremy S | Systems and methods for automatic website construction |
US20080052668A1 (en) * | 2006-06-27 | 2008-02-28 | Craig Jeremy S | Systems and methods for automatic website construction |
US20070299985A1 (en) * | 2006-06-27 | 2007-12-27 | Craig Jeremy S | Systems and methods for template based website construction |
US20100299201A1 (en) * | 2006-06-30 | 2010-11-25 | Steven Thrasher | Searching data storage systems and devices |
US20080082599A1 (en) * | 2006-09-28 | 2008-04-03 | Vladislav Sinaniyev | System and method for integrating ads into presentation environments |
US8954412B1 (en) | 2006-09-28 | 2015-02-10 | Google Inc. | Corroborating facts in electronic documents |
US7707208B2 (en) * | 2006-10-10 | 2010-04-27 | Microsoft Corporation | Identifying sight for a location |
US7657504B2 (en) * | 2006-10-10 | 2010-02-02 | Microsoft Corporation | User interface for displaying images of sights |
US8122026B1 (en) | 2006-10-20 | 2012-02-21 | Google Inc. | Finding and disambiguating references to entities on web pages |
WO2008054948A1 (en) * | 2006-10-31 | 2008-05-08 | Nielsen Media Research, Inc. | Methods and systems to retrieve information from data sources |
US20080288588A1 (en) * | 2006-11-01 | 2008-11-20 | Worldvuer, Inc. | Method and system for searching using image based tagging |
US20080109250A1 (en) * | 2006-11-03 | 2008-05-08 | Craig Allan Walker | System and method for creating and rendering DICOM structured clinical reporting via the internet |
US10192031B1 (en) | 2006-11-03 | 2019-01-29 | Vidistar, Llc | System for extracting information from DICOM structured reports |
US10503867B1 (en) | 2006-11-03 | 2019-12-10 | Vidistar, Llc | System for interacting with medical images |
WO2008064934A1 (en) * | 2006-11-30 | 2008-06-05 | International Business Machines Corporation | Method, system and computer program for downloading information based on a snapshot approach |
US20090157631A1 (en) * | 2006-12-14 | 2009-06-18 | Jason Coleman | Database search enhancements |
US8601387B2 (en) | 2006-12-15 | 2013-12-03 | Iac Search & Media, Inc. | Persistent interface |
US20080147709A1 (en) * | 2006-12-15 | 2008-06-19 | Iac Search & Media, Inc. | Search results from selected sources |
US20080270932A1 (en) * | 2006-12-15 | 2008-10-30 | Iac Search & Media, Inc. | Toolbox editing |
US20080148164A1 (en) * | 2006-12-15 | 2008-06-19 | Iac Search & Media, Inc. | Toolbox minimizer/maximizer |
US8621092B2 (en) * | 2006-12-19 | 2013-12-31 | International Business Machines Corporation | Remote portlet consumer with enhanced resource URL processing |
US20080162477A1 (en) * | 2006-12-20 | 2008-07-03 | Benjamin Rodefer | Method, apparatus, system, and computer readable medium for media distribution |
US20080195590A1 (en) * | 2007-02-08 | 2008-08-14 | Mitsuo Nakamura | Network device, image forming device, and data searching method |
US20080263436A1 (en) * | 2007-02-13 | 2008-10-23 | Ahrens Mark H | Methods and apparatus to reach through to business logic services |
US20080208831A1 (en) * | 2007-02-26 | 2008-08-28 | Microsoft Corporation | Controlling search indexing |
US8347202B1 (en) | 2007-03-14 | 2013-01-01 | Google Inc. | Determining geographic locations for place names in a fact repository |
US8001138B2 (en) * | 2007-04-11 | 2011-08-16 | Microsoft Corporation | Word relationship driven search |
US8239751B1 (en) | 2007-05-16 | 2012-08-07 | Google Inc. | Data from web documents in a spreadsheet |
US7970766B1 (en) | 2007-07-23 | 2011-06-28 | Google Inc. | Entity type assignment |
CN101364970B (en) * | 2007-08-09 | 2012-06-20 | 鸿富锦精密工业(深圳)有限公司 | Webpage material download control system and method |
US7925641B2 (en) * | 2007-09-21 | 2011-04-12 | Sap Ag | Indexing web content of a runtime version of a web page |
JP2009087071A (en) * | 2007-09-28 | 2009-04-23 | Fujifilm Corp | Information processor and information processing method |
US8191002B2 (en) | 2007-10-15 | 2012-05-29 | International Business Machines Corporation | Summarizing portlet usage in a portal page |
US7904818B2 (en) * | 2007-10-15 | 2011-03-08 | International Business Machines Corporation | Summarizing portlet usage captured responsive to trigger events in a portal page |
US8812435B1 (en) | 2007-11-16 | 2014-08-19 | Google Inc. | Learning objects and facts from documents |
US8515028B2 (en) | 2007-11-23 | 2013-08-20 | Foncloud, Inc. | System and method for externally mapping an Interactive Voice Response menu |
US9270817B2 (en) * | 2007-11-23 | 2016-02-23 | Foncloud, Inc. | Method for determining the on-hold status in a call |
US9288316B2 (en) * | 2007-11-23 | 2016-03-15 | Foncloud, Inc. | System and method for eliminating hold-time in phone calls |
US8908847B2 (en) * | 2007-11-23 | 2014-12-09 | Foncloud, Inc. | System and method for deep dialing phone systems |
US9386151B2 (en) | 2007-11-23 | 2016-07-05 | Foncloud, Inc. | System and method for replacing hold-time with a call-back in a contact center environment |
FI124000B (en) * | 2007-12-11 | 2014-01-31 | Documill Oy | Method and arrangement for processing data retrieval results |
JP2009157482A (en) * | 2007-12-25 | 2009-07-16 | Kyocera Corp | Document editing device and document editing method |
WO2009088893A1 (en) * | 2008-01-11 | 2009-07-16 | Recommended Reading, Inc. | Data picker application |
US20090254529A1 (en) * | 2008-04-04 | 2009-10-08 | Lev Goldentouch | Systems, methods and computer program products for content management |
US9639531B2 (en) * | 2008-04-09 | 2017-05-02 | The Nielsen Company (Us), Llc | Methods and apparatus to play and control playing of media in a web page |
US8849672B2 (en) * | 2008-05-22 | 2014-09-30 | Core Wireless Licensing S.A.R.L. | System and method for excerpt creation by designating a text segment using speech |
US8375089B2 (en) * | 2008-05-28 | 2013-02-12 | Red Hat, Inc. | Methods and systems for protecting E-mail addresses in publicly available network content |
US9646308B1 (en) | 2008-06-06 | 2017-05-09 | Carfax, Inc. | Tool for selling and purchasing vehicle history reports |
US20100017385A1 (en) | 2008-07-16 | 2010-01-21 | International Business Machines | Creating and managing reference elements of deployable web archive files |
US8306255B1 (en) * | 2008-08-28 | 2012-11-06 | Intuit Inc. | Snapshot-based screen scraping |
US20100080411A1 (en) * | 2008-09-29 | 2010-04-01 | Alexandros Deliyannis | Methods and apparatus to automatically crawl the internet using image analysis |
US20100131488A1 (en) * | 2008-11-26 | 2010-05-27 | Yahoo! Inc. | Digital Images of Web Pages |
US20100161344A1 (en) * | 2008-12-12 | 2010-06-24 | Dyson David S | Methods and apparatus to prepare report requests |
JP5322660B2 (en) * | 2009-01-07 | 2013-10-23 | キヤノン株式会社 | Data display device, data display method, and computer program |
CN101499098B (en) * | 2009-03-04 | 2012-07-11 | 阿里巴巴集团控股有限公司 | Web page assessed value confirming and employing method and system |
US20100287148A1 (en) * | 2009-05-08 | 2010-11-11 | Cpa Global Patent Research Limited | Method, System, and Apparatus for Targeted Searching of Multi-Sectional Documents within an Electronic Document Collection |
US9135277B2 (en) | 2009-08-07 | 2015-09-15 | Google Inc. | Architecture for responding to a visual query |
US9087059B2 (en) | 2009-08-07 | 2015-07-21 | Google Inc. | User interface for presenting search results for multiple regions of a visual query |
US20110047006A1 (en) * | 2009-08-21 | 2011-02-24 | Attenberg Joshua M | Systems, methods, and media for rating websites for safe advertising |
US8495048B2 (en) | 2009-08-26 | 2013-07-23 | International Business Machines | Applying user-generated deployment events to a grouping of deployable portlets |
US20110106835A1 (en) * | 2009-10-29 | 2011-05-05 | International Business Machines Corporation | User-Defined Profile Tags, Rules, and Recommendations for Portal |
US20100077349A1 (en) * | 2009-11-06 | 2010-03-25 | Health Grades, Inc. | Patient direct connect |
US20110113352A1 (en) * | 2009-11-06 | 2011-05-12 | Research In Motion Limited | Portable electronic device and method of web page rendering |
US20110109634A1 (en) * | 2009-11-06 | 2011-05-12 | Research In Motion Limited | Portable electronic device and method of information rendering on portable electronic device |
US20110119330A1 (en) * | 2009-11-13 | 2011-05-19 | Microsoft Corporation | Selective content loading based on complexity |
US9135261B2 (en) * | 2009-12-15 | 2015-09-15 | Emc Corporation | Systems and methods for facilitating data discovery |
US20110225152A1 (en) * | 2010-03-15 | 2011-09-15 | Microsoft Corporation | Constructing a search-result caption |
JP2011204158A (en) * | 2010-03-26 | 2011-10-13 | Sony Corp | Terminal apparatus, processing system, processing method and program |
US8346755B1 (en) | 2010-05-04 | 2013-01-01 | Google Inc. | Iterative off-line rendering process |
US9015139B2 (en) * | 2010-05-14 | 2015-04-21 | Rovi Guides, Inc. | Systems and methods for performing a search based on a media content snapshot image |
KR101665512B1 (en) * | 2010-05-25 | 2016-10-12 | 삼성전자 주식회사 | Web page managing method, apparatus and storage medium thereof |
US8732017B2 (en) | 2010-06-01 | 2014-05-20 | Integral Ad Science, Inc. | Methods, systems, and media for applying scores and ratings to web pages, web sites, and content for safe and effective online advertising |
CN103052950A (en) * | 2010-08-20 | 2013-04-17 | 惠普发展公司,有限责任合伙企业 | Systems and methods for filtering web page contents |
US8769429B2 (en) | 2010-08-31 | 2014-07-01 | Net-Express, Ltd. | Method and system for providing enhanced user interfaces for web browsing |
US8775868B2 (en) | 2010-09-28 | 2014-07-08 | Pure Storage, Inc. | Adaptive RAID for an SSD environment |
US9244769B2 (en) | 2010-09-28 | 2016-01-26 | Pure Storage, Inc. | Offset protection data in a RAID array |
US20120221546A1 (en) * | 2011-02-24 | 2012-08-30 | Rafsky Lawrence C | Method and system for facilitating web content aggregation initiated by a client or server |
US8869097B2 (en) | 2011-03-23 | 2014-10-21 | Infosys Limited | Online integrated development environment with code assist |
US8732571B2 (en) * | 2011-03-31 | 2014-05-20 | Google Inc. | Methods and systems for generating and displaying a preview image of a content area |
US9348894B2 (en) | 2011-03-31 | 2016-05-24 | Infosys Limited | Facet support, clustering for code query results |
US9009664B2 (en) * | 2011-03-31 | 2015-04-14 | Infosys Limited | Structural search of source code |
US11841912B2 (en) | 2011-05-01 | 2023-12-12 | Twittle Search Limited Liability Company | System for applying natural language processing and inputs of a group of users to infer commonly desired search results |
US8326862B2 (en) | 2011-05-01 | 2012-12-04 | Alan Mark Reznik | Systems and methods for facilitating enhancements to search engine results |
US8645353B2 (en) | 2011-06-23 | 2014-02-04 | Microsoft Corporation | Anchor image identification for vertical video search |
US8645354B2 (en) * | 2011-06-23 | 2014-02-04 | Microsoft Corporation | Scalable metadata extraction for video search |
US8468145B2 (en) | 2011-09-16 | 2013-06-18 | Google Inc. | Indexing of URLs with fragments |
US11068532B2 (en) | 2011-09-21 | 2021-07-20 | Horsetooth Ventures, LLC | Interactive image display and selection system |
US9734167B2 (en) | 2011-09-21 | 2017-08-15 | Horsetooth Ventures, LLC | Interactive image display and selection system |
US20130104032A1 (en) * | 2011-10-19 | 2013-04-25 | Jiyoun Lee | Mobile terminal and method of controlling the same |
US20130117252A1 (en) * | 2011-11-09 | 2013-05-09 | Google Inc. | Large-scale real-time fetch service |
CN102622405B (en) * | 2012-01-16 | 2013-08-21 | 北京工业大学 | Method for computing text distance between short texts based on language content unit number evaluation |
US9529926B2 (en) * | 2012-04-23 | 2016-12-27 | Excalibur Ip, Llc | Snapshot refreshment for search results page preview |
US9218419B2 (en) * | 2012-04-23 | 2015-12-22 | Yahoo! Inc. | Snapshot generation for search results page preview |
US9378283B2 (en) * | 2012-04-23 | 2016-06-28 | Excalibur Ip, Llc | Instant search results with page previews |
US9672258B2 (en) * | 2012-05-10 | 2017-06-06 | Pitney Bowes Inc. | Systems and methods for dynamically selecting graphical query result display modes |
US10387911B1 (en) | 2012-06-01 | 2019-08-20 | Integral Ad Science, Inc. | Systems, methods, and media for detecting suspicious activity |
US9646039B2 (en) | 2013-01-10 | 2017-05-09 | Pure Storage, Inc. | Snapshots in a storage system |
US9563678B2 (en) * | 2013-03-04 | 2017-02-07 | Avaya Inc. | Systems and methods for indexing and searching reporting data |
US11163736B2 (en) * | 2013-03-04 | 2021-11-02 | Avaya Inc. | System and method for in-memory indexing of data |
US20140281980A1 (en) | 2013-03-15 | 2014-09-18 | Chad A. Hage | Methods and Apparatus to Identify a Type of Media Presented by a Media Player |
KR20140126800A (en) * | 2013-04-22 | 2014-11-03 | 한국전자통신연구원 | Apparatus for displaying hyperlinks using keywords and method thereof |
US11630585B1 (en) | 2016-08-25 | 2023-04-18 | Pure Storage, Inc. | Processing evacuation events in a storage array that includes a plurality of storage devices |
US9760369B2 (en) | 2013-12-13 | 2017-09-12 | Infosys Limited | Assessing modularity of a program written in object oriented language |
CN106462582B (en) | 2014-06-26 | 2020-05-15 | 谷歌有限责任公司 | Batch optimized rendering and fetching architecture |
EP3161610B1 (en) * | 2014-06-26 | 2020-08-05 | Google LLC | Optimized browser rendering process |
EP3161662B1 (en) | 2014-06-26 | 2024-01-31 | Google LLC | Optimized browser render process |
US9489132B2 (en) | 2014-10-07 | 2016-11-08 | Pure Storage, Inc. | Utilizing unmapped and unknown states in a replicated storage system |
US10594820B2 (en) * | 2014-11-24 | 2020-03-17 | Google Llc | Conditionally joining data from cookies |
TW201624253A (en) * | 2014-12-31 | 2016-07-01 | 萬國商業機器公司 | Method, computer program product and computer system for displaying information of a parent webpage associated with a child tab on a graphical user interface |
US9521200B1 (en) | 2015-05-26 | 2016-12-13 | Pure Storage, Inc. | Locally providing cloud storage array services |
US9716755B2 (en) | 2015-05-26 | 2017-07-25 | Pure Storage, Inc. | Providing cloud storage array services by a local storage array in a data center |
US11102298B1 (en) | 2015-05-26 | 2021-08-24 | Pure Storage, Inc. | Locally providing cloud storage services for fleet management |
US9594678B1 (en) | 2015-05-27 | 2017-03-14 | Pure Storage, Inc. | Preventing duplicate entries of identical data in a storage device |
US10021170B2 (en) | 2015-05-29 | 2018-07-10 | Pure Storage, Inc. | Managing a storage array using client-side services |
US11503031B1 (en) | 2015-05-29 | 2022-11-15 | Pure Storage, Inc. | Storage array access control from cloud-based user authorization and authentication |
US9444822B1 (en) | 2015-05-29 | 2016-09-13 | Pure Storage, Inc. | Storage array access control from cloud-based user authorization and authentication |
US9300660B1 (en) | 2015-05-29 | 2016-03-29 | Pure Storage, Inc. | Providing authorization and authentication in a cloud for a user of a storage array |
US9588691B2 (en) | 2015-06-10 | 2017-03-07 | Pure Storage, Inc. | Dynamically managing control information in a storage device |
US9594512B1 (en) | 2015-06-19 | 2017-03-14 | Pure Storage, Inc. | Attributing consumed storage capacity among entities storing data in a storage array |
US10310740B2 (en) | 2015-06-23 | 2019-06-04 | Pure Storage, Inc. | Aligning memory access operations to a geometry of a storage device |
US9547441B1 (en) | 2015-06-23 | 2017-01-17 | Pure Storage, Inc. | Exposing a geometry of a storage device |
US10296236B2 (en) | 2015-07-01 | 2019-05-21 | Pure Storage, Inc. | Offloading device management responsibilities from a storage device in an array of storage devices |
US9892071B2 (en) | 2015-08-03 | 2018-02-13 | Pure Storage, Inc. | Emulating a remote direct memory access (‘RDMA’) link between controllers in a storage array |
US9851762B1 (en) | 2015-08-06 | 2017-12-26 | Pure Storage, Inc. | Compliant printed circuit board (‘PCB’) within an enclosure |
US11625181B1 (en) | 2015-08-24 | 2023-04-11 | Pure Storage, Inc. | Data tiering using snapshots |
US10198194B2 (en) | 2015-08-24 | 2019-02-05 | Pure Storage, Inc. | Placing data within a storage device of a flash array |
US11294588B1 (en) | 2015-08-24 | 2022-04-05 | Pure Storage, Inc. | Placing data within a storage device |
US9384082B1 (en) | 2015-10-23 | 2016-07-05 | Pure Storage, Inc. | Proactively providing corrective measures for storage arrays |
US11360844B1 (en) | 2015-10-23 | 2022-06-14 | Pure Storage, Inc. | Recovery of a container storage provider |
US10514978B1 (en) | 2015-10-23 | 2019-12-24 | Pure Storage, Inc. | Automatic deployment of corrective measures for storage arrays |
US10284232B2 (en) | 2015-10-28 | 2019-05-07 | Pure Storage, Inc. | Dynamic error processing in a storage device |
US10374868B2 (en) | 2015-10-29 | 2019-08-06 | Pure Storage, Inc. | Distributed command processing in a flash storage system |
US9740414B2 (en) | 2015-10-29 | 2017-08-22 | Pure Storage, Inc. | Optimizing copy operations |
US10353777B2 (en) | 2015-10-30 | 2019-07-16 | Pure Storage, Inc. | Ensuring crash-safe forward progress of a system configuration update |
US9760479B2 (en) | 2015-12-02 | 2017-09-12 | Pure Storage, Inc. | Writing data in a storage system that includes a first type of storage device and a second type of storage device |
US11762764B1 (en) | 2015-12-02 | 2023-09-19 | Pure Storage, Inc. | Writing data in a storage system that includes a first type of storage device and a second type of storage device |
US10326836B2 (en) | 2015-12-08 | 2019-06-18 | Pure Storage, Inc. | Partially replicating a snapshot between storage systems |
US11616834B2 (en) | 2015-12-08 | 2023-03-28 | Pure Storage, Inc. | Efficient replication of a dataset to the cloud |
US10162835B2 (en) | 2015-12-15 | 2018-12-25 | Pure Storage, Inc. | Proactive management of a plurality of storage arrays in a multi-array system |
US11347697B1 (en) | 2015-12-15 | 2022-05-31 | Pure Storage, Inc. | Proactively optimizing a storage system |
US10346043B2 (en) | 2015-12-28 | 2019-07-09 | Pure Storage, Inc. | Adaptive computing for data compression |
US10606618B2 (en) * | 2016-01-19 | 2020-03-31 | Adp, Llc | Contextual assistance system |
US9886314B2 (en) | 2016-01-28 | 2018-02-06 | Pure Storage, Inc. | Placing workloads in a multi-array system |
US10572460B2 (en) | 2016-02-11 | 2020-02-25 | Pure Storage, Inc. | Compressing data in dependence upon characteristics of a storage system |
US9760297B2 (en) | 2016-02-12 | 2017-09-12 | Pure Storage, Inc. | Managing input/output (‘I/O’) queues in a data storage system |
US9959043B2 (en) | 2016-03-16 | 2018-05-01 | Pure Storage, Inc. | Performing a non-disruptive upgrade of data in a storage system |
US11995315B2 (en) | 2016-03-16 | 2024-05-28 | Pure Storage, Inc. | Converting data formats in a storage system |
US9841921B2 (en) | 2016-04-27 | 2017-12-12 | Pure Storage, Inc. | Migrating data in a storage array that includes a plurality of storage devices |
US11112990B1 (en) | 2016-04-27 | 2021-09-07 | Pure Storage, Inc. | Managing storage device evacuation |
US11809727B1 (en) | 2016-04-27 | 2023-11-07 | Pure Storage, Inc. | Predicting failures in a storage system that includes a plurality of storage devices |
US9811264B1 (en) | 2016-04-28 | 2017-11-07 | Pure Storage, Inc. | Deploying client-specific applications in a storage system utilizing redundant system resources |
US10303390B1 (en) | 2016-05-02 | 2019-05-28 | Pure Storage, Inc. | Resolving fingerprint collisions in flash storage system |
US11231858B2 (en) | 2016-05-19 | 2022-01-25 | Pure Storage, Inc. | Dynamically configuring a storage system to facilitate independent scaling of resources |
US9507532B1 (en) | 2016-05-20 | 2016-11-29 | Pure Storage, Inc. | Migrating data in a storage array that includes a plurality of storage devices and a plurality of write buffer devices |
US10250656B2 (en) | 2016-05-26 | 2019-04-02 | Facebook, Inc. | Systems and methods for generating, providing, and rendering quick load articles |
US10691567B2 (en) | 2016-06-03 | 2020-06-23 | Pure Storage, Inc. | Dynamically forming a failure domain in a storage system that includes a plurality of blades |
US11706895B2 (en) | 2016-07-19 | 2023-07-18 | Pure Storage, Inc. | Independent scaling of compute resources and storage resources in a storage system |
US10459652B2 (en) | 2016-07-27 | 2019-10-29 | Pure Storage, Inc. | Evacuating blades in a storage array that includes a plurality of blades |
US10474363B1 (en) | 2016-07-29 | 2019-11-12 | Pure Storage, Inc. | Space reporting in a storage system |
CN106294848A (en) * | 2016-08-22 | 2017-01-04 | 深圳市茁壮网络股份有限公司 | A kind of web analysis, acquisition methods and device |
CN106909603A (en) | 2016-08-31 | 2017-06-30 | 阿里巴巴集团控股有限公司 | Search information processing method and device |
US10908966B1 (en) | 2016-09-07 | 2021-02-02 | Pure Storage, Inc. | Adapting target service times in a storage system |
US10146585B2 (en) | 2016-09-07 | 2018-12-04 | Pure Storage, Inc. | Ensuring the fair utilization of system resources using workload based, time-independent scheduling |
US11481261B1 (en) | 2016-09-07 | 2022-10-25 | Pure Storage, Inc. | Preventing extended latency in a storage system |
US10331588B2 (en) | 2016-09-07 | 2019-06-25 | Pure Storage, Inc. | Ensuring the appropriate utilization of system resources using weighted workload based, time-independent scheduling |
US11886922B2 (en) | 2016-09-07 | 2024-01-30 | Pure Storage, Inc. | Scheduling input/output operations for a storage system |
US11960348B2 (en) | 2016-09-07 | 2024-04-16 | Pure Storage, Inc. | Cloud-based monitoring of hardware components in a fleet of storage systems |
US10671439B1 (en) | 2016-09-07 | 2020-06-02 | Pure Storage, Inc. | Workload planning with quality-of-service (‘QOS’) integration |
US10235229B1 (en) | 2016-09-07 | 2019-03-19 | Pure Storage, Inc. | Rehabilitating storage devices in a storage array that includes a plurality of storage devices |
US11531577B1 (en) | 2016-09-07 | 2022-12-20 | Pure Storage, Inc. | Temporarily limiting access to a storage device |
US10007459B2 (en) | 2016-10-20 | 2018-06-26 | Pure Storage, Inc. | Performance tuning in a storage system that includes one or more storage devices |
US11379132B1 (en) | 2016-10-20 | 2022-07-05 | Pure Storage, Inc. | Correlating medical sensor data |
US10162566B2 (en) | 2016-11-22 | 2018-12-25 | Pure Storage, Inc. | Accumulating application-level statistics in a storage system |
US11620075B2 (en) | 2016-11-22 | 2023-04-04 | Pure Storage, Inc. | Providing application aware storage |
US10503803B2 (en) | 2016-11-23 | 2019-12-10 | Google Llc | Animated snippets for search results |
US10198205B1 (en) | 2016-12-19 | 2019-02-05 | Pure Storage, Inc. | Dynamically adjusting a number of storage devices utilized to simultaneously service write operations |
US11461273B1 (en) | 2016-12-20 | 2022-10-04 | Pure Storage, Inc. | Modifying storage distribution in a storage system that includes one or more storage devices |
US10489307B2 (en) | 2017-01-05 | 2019-11-26 | Pure Storage, Inc. | Periodically re-encrypting user data stored on a storage device |
US11307998B2 (en) | 2017-01-09 | 2022-04-19 | Pure Storage, Inc. | Storage efficiency of encrypted host system data |
US10503700B1 (en) | 2017-01-19 | 2019-12-10 | Pure Storage, Inc. | On-demand content filtering of snapshots within a storage system |
US11340800B1 (en) | 2017-01-19 | 2022-05-24 | Pure Storage, Inc. | Content masking in a storage system |
US11163624B2 (en) | 2017-01-27 | 2021-11-02 | Pure Storage, Inc. | Dynamically adjusting an amount of log data generated for a storage system |
US11803453B1 (en) | 2017-03-10 | 2023-10-31 | Pure Storage, Inc. | Using host connectivity states to avoid queuing I/O requests |
US11442825B2 (en) | 2017-03-10 | 2022-09-13 | Pure Storage, Inc. | Establishing a synchronous replication relationship between two or more storage systems |
US11941279B2 (en) | 2017-03-10 | 2024-03-26 | Pure Storage, Inc. | Data path virtualization |
US10521344B1 (en) | 2017-03-10 | 2019-12-31 | Pure Storage, Inc. | Servicing input/output (‘I/O’) operations directed to a dataset that is synchronized across a plurality of storage systems |
US12056383B2 (en) | 2017-03-10 | 2024-08-06 | Pure Storage, Inc. | Edge management service |
US11169727B1 (en) | 2017-03-10 | 2021-11-09 | Pure Storage, Inc. | Synchronous replication between storage systems with virtualized storage |
US11675520B2 (en) | 2017-03-10 | 2023-06-13 | Pure Storage, Inc. | Application replication among storage systems synchronously replicating a dataset |
US11089105B1 (en) | 2017-12-14 | 2021-08-10 | Pure Storage, Inc. | Synchronously replicating datasets in cloud-based storage systems |
US10503427B2 (en) | 2017-03-10 | 2019-12-10 | Pure Storage, Inc. | Synchronously replicating datasets and other managed objects to cloud-based storage systems |
US10454810B1 (en) | 2017-03-10 | 2019-10-22 | Pure Storage, Inc. | Managing host definitions across a plurality of storage systems |
US10459664B1 (en) | 2017-04-10 | 2019-10-29 | Pure Storage, Inc. | Virtualized copy-by-reference |
US9910618B1 (en) | 2017-04-10 | 2018-03-06 | Pure Storage, Inc. | Migrating applications executing on a storage system |
US11868629B1 (en) | 2017-05-05 | 2024-01-09 | Pure Storage, Inc. | Storage system sizing service |
US10552090B2 (en) | 2017-09-07 | 2020-02-04 | Pure Storage, Inc. | Solid state drives with multiple types of addressable memory |
US12086650B2 (en) | 2017-06-12 | 2024-09-10 | Pure Storage, Inc. | Workload placement based on carbon emissions |
US11340939B1 (en) | 2017-06-12 | 2022-05-24 | Pure Storage, Inc. | Application-aware analytics for storage systems |
US12086651B2 (en) | 2017-06-12 | 2024-09-10 | Pure Storage, Inc. | Migrating workloads using active disaster recovery |
US11989429B1 (en) | 2017-06-12 | 2024-05-21 | Pure Storage, Inc. | Recommending changes to a storage system |
US10613791B2 (en) | 2017-06-12 | 2020-04-07 | Pure Storage, Inc. | Portable snapshot replication between storage systems |
US11442669B1 (en) | 2018-03-15 | 2022-09-13 | Pure Storage, Inc. | Orchestrating a virtual storage system |
US11593036B2 (en) | 2017-06-12 | 2023-02-28 | Pure Storage, Inc. | Staging data within a unified storage element |
US11210133B1 (en) | 2017-06-12 | 2021-12-28 | Pure Storage, Inc. | Workload mobility between disparate execution environments |
US11016824B1 (en) | 2017-06-12 | 2021-05-25 | Pure Storage, Inc. | Event identification with out-of-order reporting in a cloud-based environment |
US10853148B1 (en) | 2017-06-12 | 2020-12-01 | Pure Storage, Inc. | Migrating workloads between a plurality of execution environments |
US12061822B1 (en) | 2017-06-12 | 2024-08-13 | Pure Storage, Inc. | Utilizing volume-level policies in a storage system |
US11422731B1 (en) | 2017-06-12 | 2022-08-23 | Pure Storage, Inc. | Metadata-based replication of a dataset |
US10884636B1 (en) | 2017-06-12 | 2021-01-05 | Pure Storage, Inc. | Presenting workload performance in a storage system |
US10417092B2 (en) | 2017-09-07 | 2019-09-17 | Pure Storage, Inc. | Incremental RAID stripe update parity calculation |
US11592991B2 (en) | 2017-09-07 | 2023-02-28 | Pure Storage, Inc. | Converting raid data between persistent storage types |
US10976962B2 (en) | 2018-03-15 | 2021-04-13 | Pure Storage, Inc. | Servicing I/O operations in a cloud-based storage system |
EP3612922A1 (en) | 2017-06-12 | 2020-02-26 | Pure Storage, Inc. | Accessible fast durable storage integrated into a bulk storage device |
US11609718B1 (en) | 2017-06-12 | 2023-03-21 | Pure Storage, Inc. | Identifying valid data after a storage system recovery |
US10845953B1 (en) * | 2017-06-28 | 2020-11-24 | Amazon Technologies, Inc. | Identifying actionable content for navigation |
US11561714B1 (en) | 2017-07-05 | 2023-01-24 | Pure Storage, Inc. | Storage efficiency driven migration |
US11477280B1 (en) | 2017-07-26 | 2022-10-18 | Pure Storage, Inc. | Integrating cloud storage services |
US10831935B2 (en) | 2017-08-31 | 2020-11-10 | Pure Storage, Inc. | Encryption management with host-side data reduction |
US10452444B1 (en) | 2017-10-19 | 2019-10-22 | Pure Storage, Inc. | Storage system with compute resources and shared storage resources |
US12067466B2 (en) | 2017-10-19 | 2024-08-20 | Pure Storage, Inc. | Artificial intelligence and machine learning hyperscale infrastructure |
US11455168B1 (en) | 2017-10-19 | 2022-09-27 | Pure Storage, Inc. | Batch building for deep learning training workloads |
US10671435B1 (en) | 2017-10-19 | 2020-06-02 | Pure Storage, Inc. | Data transformation caching in an artificial intelligence infrastructure |
US10360214B2 (en) | 2017-10-19 | 2019-07-23 | Pure Storage, Inc. | Ensuring reproducibility in an artificial intelligence infrastructure |
US11861423B1 (en) | 2017-10-19 | 2024-01-02 | Pure Storage, Inc. | Accelerating artificial intelligence (‘AI’) workflows |
US11494692B1 (en) | 2018-03-26 | 2022-11-08 | Pure Storage, Inc. | Hyperscale artificial intelligence and machine learning infrastructure |
US10671494B1 (en) | 2017-11-01 | 2020-06-02 | Pure Storage, Inc. | Consistent selection of replicated datasets during storage system recovery |
US10467107B1 (en) | 2017-11-01 | 2019-11-05 | Pure Storage, Inc. | Maintaining metadata resiliency among storage device failures |
US10817392B1 (en) | 2017-11-01 | 2020-10-27 | Pure Storage, Inc. | Ensuring resiliency to storage device failures in a storage system that includes a plurality of storage devices |
US10484174B1 (en) | 2017-11-01 | 2019-11-19 | Pure Storage, Inc. | Protecting an encryption key for data stored in a storage system that includes a plurality of storage devices |
US10509581B1 (en) | 2017-11-01 | 2019-12-17 | Pure Storage, Inc. | Maintaining write consistency in a multi-threaded storage system |
US10929226B1 (en) | 2017-11-21 | 2021-02-23 | Pure Storage, Inc. | Providing for increased flexibility for large scale parity |
US10936238B2 (en) | 2017-11-28 | 2021-03-02 | Pure Storage, Inc. | Hybrid data tiering |
US10990282B1 (en) | 2017-11-28 | 2021-04-27 | Pure Storage, Inc. | Hybrid data tiering with cloud storage |
US10795598B1 (en) | 2017-12-07 | 2020-10-06 | Pure Storage, Inc. | Volume migration for storage systems synchronously replicating a dataset |
US11036677B1 (en) | 2017-12-14 | 2021-06-15 | Pure Storage, Inc. | Replicated data integrity |
US10929031B2 (en) | 2017-12-21 | 2021-02-23 | Pure Storage, Inc. | Maximizing data reduction in a partially encrypted volume |
US10992533B1 (en) | 2018-01-30 | 2021-04-27 | Pure Storage, Inc. | Policy based path management |
US10521151B1 (en) | 2018-03-05 | 2019-12-31 | Pure Storage, Inc. | Determining effective space utilization in a storage system |
US10942650B1 (en) | 2018-03-05 | 2021-03-09 | Pure Storage, Inc. | Reporting capacity utilization in a storage system |
US11972134B2 (en) | 2018-03-05 | 2024-04-30 | Pure Storage, Inc. | Resource utilization using normalized input/output (‘I/O’) operations |
US11861170B2 (en) | 2018-03-05 | 2024-01-02 | Pure Storage, Inc. | Sizing resources for a replication target |
US11150834B1 (en) | 2018-03-05 | 2021-10-19 | Pure Storage, Inc. | Determining storage consumption in a storage system |
US10296258B1 (en) | 2018-03-09 | 2019-05-21 | Pure Storage, Inc. | Offloading data storage to a decentralized storage network |
US10917471B1 (en) | 2018-03-15 | 2021-02-09 | Pure Storage, Inc. | Active membership in a cloud-based storage system |
US11048590B1 (en) | 2018-03-15 | 2021-06-29 | Pure Storage, Inc. | Data consistency during recovery in a cloud-based storage system |
US12066900B2 (en) | 2018-03-15 | 2024-08-20 | Pure Storage, Inc. | Managing disaster recovery to cloud computing environment |
US11288138B1 (en) | 2018-03-15 | 2022-03-29 | Pure Storage, Inc. | Recovery from a system fault in a cloud-based storage system |
US11210009B1 (en) | 2018-03-15 | 2021-12-28 | Pure Storage, Inc. | Staging data in a cloud-based storage system |
US10924548B1 (en) | 2018-03-15 | 2021-02-16 | Pure Storage, Inc. | Symmetric storage using a cloud-based storage system |
US11171950B1 (en) | 2018-03-21 | 2021-11-09 | Pure Storage, Inc. | Secure cloud-based storage system management |
US11095706B1 (en) | 2018-03-21 | 2021-08-17 | Pure Storage, Inc. | Secure cloud-based storage system management |
US10838833B1 (en) | 2018-03-26 | 2020-11-17 | Pure Storage, Inc. | Providing for high availability in a data analytics pipeline without replicas |
US11436344B1 (en) | 2018-04-24 | 2022-09-06 | Pure Storage, Inc. | Secure encryption in deduplication cluster |
US11392553B1 (en) | 2018-04-24 | 2022-07-19 | Pure Storage, Inc. | Remote data management |
US11455409B2 (en) | 2018-05-21 | 2022-09-27 | Pure Storage, Inc. | Storage layer data obfuscation |
US10992598B2 (en) | 2018-05-21 | 2021-04-27 | Pure Storage, Inc. | Synchronously replicating when a mediation service becomes unavailable |
US12086431B1 (en) | 2018-05-21 | 2024-09-10 | Pure Storage, Inc. | Selective communication protocol layering for synchronous replication |
US11675503B1 (en) | 2018-05-21 | 2023-06-13 | Pure Storage, Inc. | Role-based data access |
US11954220B2 (en) | 2018-05-21 | 2024-04-09 | Pure Storage, Inc. | Data protection for container storage |
US10871922B2 (en) | 2018-05-22 | 2020-12-22 | Pure Storage, Inc. | Integrated storage management between storage systems and container orchestrators |
US11403000B1 (en) | 2018-07-20 | 2022-08-02 | Pure Storage, Inc. | Resiliency in a cloud-based storage system |
US11416298B1 (en) | 2018-07-20 | 2022-08-16 | Pure Storage, Inc. | Providing application-specific storage by a storage system |
US11632360B1 (en) | 2018-07-24 | 2023-04-18 | Pure Storage, Inc. | Remote access to a storage device |
US11146564B1 (en) | 2018-07-24 | 2021-10-12 | Pure Storage, Inc. | Login authentication in a cloud storage platform |
US11954238B1 (en) | 2018-07-24 | 2024-04-09 | Pure Storage, Inc. | Role-based access control for a storage system |
US11860820B1 (en) | 2018-09-11 | 2024-01-02 | Pure Storage, Inc. | Processing data through a storage system in a data pipeline |
US12026381B2 (en) | 2018-10-26 | 2024-07-02 | Pure Storage, Inc. | Preserving identities and policies across replication |
US10671302B1 (en) | 2018-10-26 | 2020-06-02 | Pure Storage, Inc. | Applying a rate limit across a plurality of storage systems |
US11526405B1 (en) | 2018-11-18 | 2022-12-13 | Pure Storage, Inc. | Cloud-based disaster recovery |
US11340837B1 (en) | 2018-11-18 | 2022-05-24 | Pure Storage, Inc. | Storage system management via a remote console |
US12026061B1 (en) | 2018-11-18 | 2024-07-02 | Pure Storage, Inc. | Restoring a cloud-based storage system to a selected state |
US11023179B2 (en) | 2018-11-18 | 2021-06-01 | Pure Storage, Inc. | Cloud-based storage system storage management |
US12026060B1 (en) | 2018-11-18 | 2024-07-02 | Pure Storage, Inc. | Reverting between codified states in a cloud-based storage system |
US10963189B1 (en) | 2018-11-18 | 2021-03-30 | Pure Storage, Inc. | Coalescing write operations in a cloud-based storage system |
US11650749B1 (en) | 2018-12-17 | 2023-05-16 | Pure Storage, Inc. | Controlling access to sensitive data in a shared dataset |
US11003369B1 (en) | 2019-01-14 | 2021-05-11 | Pure Storage, Inc. | Performing a tune-up procedure on a storage device during a boot process |
US11042452B1 (en) | 2019-03-20 | 2021-06-22 | Pure Storage, Inc. | Storage system data recovery using data recovery as a service |
US11221778B1 (en) | 2019-04-02 | 2022-01-11 | Pure Storage, Inc. | Preparing data for deduplication |
US11068162B1 (en) | 2019-04-09 | 2021-07-20 | Pure Storage, Inc. | Storage management in a cloud data store |
US11392555B2 (en) | 2019-05-15 | 2022-07-19 | Pure Storage, Inc. | Cloud-based file services |
US11853266B2 (en) | 2019-05-15 | 2023-12-26 | Pure Storage, Inc. | Providing a file system in a cloud environment |
US11327676B1 (en) | 2019-07-18 | 2022-05-10 | Pure Storage, Inc. | Predictive data streaming in a virtual storage system |
US11126364B2 (en) | 2019-07-18 | 2021-09-21 | Pure Storage, Inc. | Virtual storage system architecture |
US12001355B1 (en) | 2019-05-24 | 2024-06-04 | Pure Storage, Inc. | Chunked memory efficient storage data transfers |
US11487715B1 (en) | 2019-07-18 | 2022-11-01 | Pure Storage, Inc. | Resiliency in a cloud-based storage system |
US11797197B1 (en) | 2019-07-18 | 2023-10-24 | Pure Storage, Inc. | Dynamic scaling of a virtual storage system |
US11093139B1 (en) | 2019-07-18 | 2021-08-17 | Pure Storage, Inc. | Durably storing data within a virtual storage system |
US11422751B2 (en) | 2019-07-18 | 2022-08-23 | Pure Storage, Inc. | Creating a virtual storage system |
US11861221B1 (en) | 2019-07-18 | 2024-01-02 | Pure Storage, Inc. | Providing scalable and reliable container-based storage services |
US11526408B2 (en) | 2019-07-18 | 2022-12-13 | Pure Storage, Inc. | Data recovery in a virtual storage system |
US11086553B1 (en) | 2019-08-28 | 2021-08-10 | Pure Storage, Inc. | Tiering duplicated objects in a cloud-based object store |
US11693713B1 (en) | 2019-09-04 | 2023-07-04 | Pure Storage, Inc. | Self-tuning clusters for resilient microservices |
US11360689B1 (en) | 2019-09-13 | 2022-06-14 | Pure Storage, Inc. | Cloning a tracking copy of replica data |
US11797569B2 (en) | 2019-09-13 | 2023-10-24 | Pure Storage, Inc. | Configurable data replication |
US12045252B2 (en) | 2019-09-13 | 2024-07-23 | Pure Storage, Inc. | Providing quality of service (QoS) for replicating datasets |
US11573864B1 (en) | 2019-09-16 | 2023-02-07 | Pure Storage, Inc. | Automating database management in a storage system |
US11669386B1 (en) | 2019-10-08 | 2023-06-06 | Pure Storage, Inc. | Managing an application's resource stack |
US11943293B1 (en) | 2019-12-06 | 2024-03-26 | Pure Storage, Inc. | Restoring a storage system from a replication target |
US11709636B1 (en) | 2020-01-13 | 2023-07-25 | Pure Storage, Inc. | Non-sequential readahead for deep learning training |
US11733901B1 (en) | 2020-01-13 | 2023-08-22 | Pure Storage, Inc. | Providing persistent storage to transient cloud computing services |
US11720497B1 (en) | 2020-01-13 | 2023-08-08 | Pure Storage, Inc. | Inferred nonsequential prefetch based on data access patterns |
US12014065B2 (en) | 2020-02-11 | 2024-06-18 | Pure Storage, Inc. | Multi-cloud orchestration as-a-service |
US11868622B2 (en) | 2020-02-25 | 2024-01-09 | Pure Storage, Inc. | Application recovery across storage systems |
US11637896B1 (en) | 2020-02-25 | 2023-04-25 | Pure Storage, Inc. | Migrating applications to a cloud-computing environment |
US11321006B1 (en) | 2020-03-25 | 2022-05-03 | Pure Storage, Inc. | Data loss prevention during transitions from a replication source |
US12038881B2 (en) | 2020-03-25 | 2024-07-16 | Pure Storage, Inc. | Replica transitions for file storage |
US11630598B1 (en) | 2020-04-06 | 2023-04-18 | Pure Storage, Inc. | Scheduling data replication operations |
US11301152B1 (en) | 2020-04-06 | 2022-04-12 | Pure Storage, Inc. | Intelligently moving data between storage systems |
US11494267B2 (en) | 2020-04-14 | 2022-11-08 | Pure Storage, Inc. | Continuous value data redundancy |
US11921670B1 (en) | 2020-04-20 | 2024-03-05 | Pure Storage, Inc. | Multivariate data backup retention policies |
US11431488B1 (en) | 2020-06-08 | 2022-08-30 | Pure Storage, Inc. | Protecting local key generation using a remote key management service |
US11442652B1 (en) | 2020-07-23 | 2022-09-13 | Pure Storage, Inc. | Replication handling during storage system transportation |
US11349917B2 (en) | 2020-07-23 | 2022-05-31 | Pure Storage, Inc. | Replication handling among distinct networks |
US12079222B1 (en) | 2020-09-04 | 2024-09-03 | Pure Storage, Inc. | Enabling data portability between systems |
US11397545B1 (en) | 2021-01-20 | 2022-07-26 | Pure Storage, Inc. | Emulating persistent reservations in a cloud-based storage system |
US11853285B1 (en) | 2021-01-22 | 2023-12-26 | Pure Storage, Inc. | Blockchain logging of volume-level events in a storage system |
US12086649B2 (en) | 2021-05-12 | 2024-09-10 | Pure Storage, Inc. | Rebalancing in a fleet of storage systems using data science |
US11816129B2 (en) | 2021-06-22 | 2023-11-14 | Pure Storage, Inc. | Generating datasets using approximate baselines |
US11893263B2 (en) | 2021-10-29 | 2024-02-06 | Pure Storage, Inc. | Coordinated checkpoints among storage systems implementing checkpoint-based replication |
US11714723B2 (en) | 2021-10-29 | 2023-08-01 | Pure Storage, Inc. | Coordinated snapshots for data stored across distinct storage environments |
US11914867B2 (en) | 2021-10-29 | 2024-02-27 | Pure Storage, Inc. | Coordinated snapshots among storage systems implementing a promotion/demotion model |
US11922052B2 (en) | 2021-12-15 | 2024-03-05 | Pure Storage, Inc. | Managing links between storage objects |
US11847071B2 (en) | 2021-12-30 | 2023-12-19 | Pure Storage, Inc. | Enabling communication between a single-port device and multiple storage system controllers |
US12001300B2 (en) | 2022-01-04 | 2024-06-04 | Pure Storage, Inc. | Assessing protection for storage resources |
US11860780B2 (en) | 2022-01-28 | 2024-01-02 | Pure Storage, Inc. | Storage cache management |
US11886295B2 (en) | 2022-01-31 | 2024-01-30 | Pure Storage, Inc. | Intra-block error correction |
US11958500B1 (en) | 2022-05-10 | 2024-04-16 | Ghost Autonomy Inc. | Autonomous vehicle model training and validation using low-discrepancy sequences |
Citations (63)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5297249A (en) * | 1990-10-31 | 1994-03-22 | International Business Machines Corporation | Hypermedia link marker abstract and search services |
US5367623A (en) * | 1990-09-25 | 1994-11-22 | Sharp Kabushiki Kaisha | Information processing apparatus capable of opening two or more windows on screen, one window containing a page and other windows containing supplemental information |
US5572643A (en) * | 1995-10-19 | 1996-11-05 | Judson; David H. | Web browser with dynamic display of information objects during linking |
WO1997004409A1 (en) | 1995-07-18 | 1997-02-06 | Nomura Research Institute, Ltd. | File searching device |
JPH09138803A (en) | 1995-11-14 | 1997-05-27 | Toshiba Corp | Picture file retrieval device |
US5737599A (en) * | 1995-09-25 | 1998-04-07 | Rowe; Edward R. | Method and apparatus for downloading multi-page electronic documents with hint information |
US5737619A (en) * | 1995-10-19 | 1998-04-07 | Judson; David Hugh | World wide web browsing with content delivery over an idle connection and interstitial content display |
US5761655A (en) * | 1990-06-06 | 1998-06-02 | Alphatronix, Inc. | Image file storage and retrieval system |
US5790114A (en) * | 1996-10-04 | 1998-08-04 | Microtouch Systems, Inc. | Electronic whiteboard with multi-functional user interface |
JPH10207898A (en) | 1997-01-20 | 1998-08-07 | Terumi Ishizawa | Picture displaying browser and its storage device and picture displaying method |
EP0863638A2 (en) | 1997-03-04 | 1998-09-09 | Nortel Networks Corporation | Data access method and apparatus for communications network |
US5848409A (en) * | 1993-11-19 | 1998-12-08 | Smartpatents, Inc. | System, method and computer program product for maintaining group hits tables and document index tables for the purpose of searching through individual documents and groups of documents |
US5864863A (en) | 1996-08-09 | 1999-01-26 | Digital Equipment Corporation | Method for parsing, indexing and searching world-wide-web pages |
JPH11110323A (en) | 1997-10-07 | 1999-04-23 | Oki Electric Ind Co Ltd | Document information providing system |
JPH11149469A (en) | 1997-11-14 | 1999-06-02 | Canon Inc | Device and method for processing document and storage medium |
US5920856A (en) * | 1997-06-09 | 1999-07-06 | Xerox Corporation | System for selecting multimedia databases over networks |
JPH11250104A (en) | 1997-09-17 | 1999-09-17 | Sun Microsyst Inc | Picture information retrieving device, its method, method for selecting representative picture from video display, system to display picture from video data base and medium for recording picture information retrieving program |
US5983218A (en) * | 1997-06-30 | 1999-11-09 | Xerox Corporation | Multimedia database for use over networks |
US5982369A (en) | 1997-04-21 | 1999-11-09 | Sony Corporation | Method for displaying on a screen of a computer system images representing search results |
US6011537A (en) | 1997-01-27 | 2000-01-04 | Slotznick; Benjamin | System for delivering and simultaneously displaying primary and secondary information, and for displaying only the secondary information during interstitial space |
US6014668A (en) * | 1996-10-07 | 2000-01-11 | Ricoh Company, Ltd. | Document information management system and method of generating a human-computer interactive document sheet allowing relevant information retrieval from a remote site |
US6064755A (en) * | 1996-07-05 | 2000-05-16 | Fuji Photo Film Co., Ltd. | Image analyzing apparatus for producing density profile data of an image |
US6070176A (en) | 1997-01-30 | 2000-05-30 | Intel Corporation | Method and apparatus for graphically representing portions of the world wide web |
US6154738A (en) * | 1998-03-27 | 2000-11-28 | Call; Charles Gainor | Methods and apparatus for disseminating product information via the internet using universal product codes |
US6157935A (en) * | 1996-12-17 | 2000-12-05 | Tran; Bao Q. | Remote data access and management system |
US6182072B1 (en) | 1997-03-26 | 2001-01-30 | Webtv Networks, Inc. | Method and apparatus for generating a tour of world wide web sites |
US6182090B1 (en) | 1995-04-28 | 2001-01-30 | Ricoh Company, Ltd. | Method and apparatus for pointing to documents electronically using features extracted from a scanned icon representing a destination |
US6237011B1 (en) * | 1997-10-08 | 2001-05-22 | Caere Corporation | Computer-based document management system |
US20010011365A1 (en) | 1996-02-09 | 2001-08-02 | Jonathan Isaac Helfman | Method and apparatus for passively browsing the internet |
US6271840B1 (en) | 1998-09-24 | 2001-08-07 | James Lee Finseth | Graphical search engine visual index |
US6313855B1 (en) | 2000-02-04 | 2001-11-06 | Browse3D Corporation | System and method for web browsing |
US6335742B1 (en) * | 1997-07-24 | 2002-01-01 | Ricoh Company, Ltd. | Apparatus for file management and manipulation using graphical displays and textual descriptions |
US6351467B1 (en) * | 1997-10-27 | 2002-02-26 | Hughes Electronics Corporation | System and method for multicasting multimedia content |
US6356908B1 (en) | 1999-07-30 | 2002-03-12 | International Business Machines Corporation | Automatic web page thumbnail generation |
US20020038299A1 (en) | 2000-03-20 | 2002-03-28 | Uri Zernik | Interface for presenting information |
US6374260B1 (en) | 1996-05-24 | 2002-04-16 | Magnifi, Inc. | Method and apparatus for uploading, indexing, analyzing, and searching media content |
US6389417B1 (en) | 1999-06-29 | 2002-05-14 | Samsung Electronics Co., Ltd. | Method and apparatus for searching a digital image |
US6415282B1 (en) | 1998-04-22 | 2002-07-02 | Nec Usa, Inc. | Method and apparatus for query refinement |
US6456334B1 (en) * | 1999-06-29 | 2002-09-24 | Ati International Srl | Method and apparatus for displaying video in a data processing system |
US20020166122A1 (en) * | 1999-06-25 | 2002-11-07 | Dan Kikinis | Image-oriented electronic programming guide |
US6523063B1 (en) * | 1999-08-30 | 2003-02-18 | Zaplet, Inc. | Method system and program product for accessing a file using values from a redirect message string for each change of the link identifier |
US6665838B1 (en) | 1999-07-30 | 2003-12-16 | International Business Machines Corporation | Web page thumbnails and user configured complementary information provided from a server |
US6678673B1 (en) | 1998-02-24 | 2004-01-13 | Koninklijke Philips Electronics N.V. | System and method for providing appropriate hyperlink based on identified keywords from text messages sent between users |
US6708172B1 (en) | 1999-12-22 | 2004-03-16 | Urbanpixel, Inc. | Community-based shared multiple browser environment |
US6711567B2 (en) | 2000-02-24 | 2004-03-23 | Inktomi Corporation | Delivering non-default items in association with search results |
US6725221B2 (en) | 1994-10-03 | 2004-04-20 | Hitachi Ltd | Image data transfer method and system therefor |
US6725214B2 (en) | 2000-01-14 | 2004-04-20 | Dotnsf | Apparatus and method to support management of uniform resource locators and/or contents of database servers |
US6829619B1 (en) * | 1999-02-02 | 2004-12-07 | Fujitsu Limited | Information providing server |
US20050010860A1 (en) | 2000-02-23 | 2005-01-13 | Yuval Weiss | Systems and methods for generating and providing previews of electronic files such as Web files |
US6845214B1 (en) * | 1999-07-13 | 2005-01-18 | Nec Corporation | Video apparatus and re-encoder therefor |
US20050047681A1 (en) * | 1999-01-28 | 2005-03-03 | Osamu Hori | Image information describing method, video retrieval method, video reproducing method, and video reproducing apparatus |
US6904227B1 (en) * | 1999-02-15 | 2005-06-07 | Nec Corporation | Device and method for editing video and/or audio data recorded in a disc storage medium |
US6934964B1 (en) * | 2000-02-08 | 2005-08-23 | Koninklijke Philips Electronics N.V. | Electronic program guide viewing history generator method and system |
US6964018B1 (en) | 1999-10-29 | 2005-11-08 | Sony Corporation | Document editing processing method and apparatus and program furnishing medium |
US20050257400A1 (en) * | 1998-11-06 | 2005-11-24 | Microsoft Corporation | Navigating a resource browser session |
US7000242B1 (en) * | 2000-07-31 | 2006-02-14 | Jeff Haber | Directing internet shopping traffic and tracking revenues generated as a result thereof |
US7039860B1 (en) * | 1999-10-01 | 2006-05-02 | Netspinner Solutions As | Creating web pages category list prior to the list being served to a browser |
US7047180B1 (en) | 1999-04-30 | 2006-05-16 | Autodesk, Inc. | Method and apparatus for providing access to drawing information |
US7051019B1 (en) | 1999-08-17 | 2006-05-23 | Corbis Corporation | Method and system for obtaining images from a database having images that are relevant to indicated text |
US20060184886A1 (en) | 1999-12-22 | 2006-08-17 | Urbanpixel Inc. | Spatial chat in a multiple browser environment |
US20080148330A1 (en) * | 2000-02-03 | 2008-06-19 | Gad Liwerant | Method and system for sharing video over a network |
WO2010124721A1 (en) * | 2009-04-27 | 2010-11-04 | Nokia Siemens Networks Oy | Optimized resource allocation signaling on a physical downlink control channel |
US8464302B1 (en) * | 1999-08-03 | 2013-06-11 | Videoshare, Llc | Method and system for sharing video with advertisements over a network |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6562077B2 (en) * | 1997-11-14 | 2003-05-13 | Xerox Corporation | Sorting image segments into clusters based on a distance measurement |
US6665841B1 (en) * | 1997-11-14 | 2003-12-16 | Xerox Corporation | Transmission of subsets of layout objects at different resolutions |
US7080098B2 (en) * | 2002-05-02 | 2006-07-18 | Smirniotopoulos James G | Medical multimedia database system |
-
2000
- 2000-04-27 US US09/559,501 patent/US6643641B1/en not_active Ceased
-
2005
- 2005-11-04 US US11/266,750 patent/USRE42413E1/en not_active Expired - Lifetime
-
2006
- 2006-08-31 US US11/513,423 patent/USRE46967E1/en not_active Expired - Lifetime
Patent Citations (63)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5761655A (en) * | 1990-06-06 | 1998-06-02 | Alphatronix, Inc. | Image file storage and retrieval system |
US5367623A (en) * | 1990-09-25 | 1994-11-22 | Sharp Kabushiki Kaisha | Information processing apparatus capable of opening two or more windows on screen, one window containing a page and other windows containing supplemental information |
US5297249A (en) * | 1990-10-31 | 1994-03-22 | International Business Machines Corporation | Hypermedia link marker abstract and search services |
US5848409A (en) * | 1993-11-19 | 1998-12-08 | Smartpatents, Inc. | System, method and computer program product for maintaining group hits tables and document index tables for the purpose of searching through individual documents and groups of documents |
US6725221B2 (en) | 1994-10-03 | 2004-04-20 | Hitachi Ltd | Image data transfer method and system therefor |
US6182090B1 (en) | 1995-04-28 | 2001-01-30 | Ricoh Company, Ltd. | Method and apparatus for pointing to documents electronically using features extracted from a scanned icon representing a destination |
WO1997004409A1 (en) | 1995-07-18 | 1997-02-06 | Nomura Research Institute, Ltd. | File searching device |
US5737599A (en) * | 1995-09-25 | 1998-04-07 | Rowe; Edward R. | Method and apparatus for downloading multi-page electronic documents with hint information |
US5737619A (en) * | 1995-10-19 | 1998-04-07 | Judson; David Hugh | World wide web browsing with content delivery over an idle connection and interstitial content display |
US5572643A (en) * | 1995-10-19 | 1996-11-05 | Judson; David H. | Web browser with dynamic display of information objects during linking |
JPH09138803A (en) | 1995-11-14 | 1997-05-27 | Toshiba Corp | Picture file retrieval device |
US20010011365A1 (en) | 1996-02-09 | 2001-08-02 | Jonathan Isaac Helfman | Method and apparatus for passively browsing the internet |
US6374260B1 (en) | 1996-05-24 | 2002-04-16 | Magnifi, Inc. | Method and apparatus for uploading, indexing, analyzing, and searching media content |
US6064755A (en) * | 1996-07-05 | 2000-05-16 | Fuji Photo Film Co., Ltd. | Image analyzing apparatus for producing density profile data of an image |
US5864863A (en) | 1996-08-09 | 1999-01-26 | Digital Equipment Corporation | Method for parsing, indexing and searching world-wide-web pages |
US5790114A (en) * | 1996-10-04 | 1998-08-04 | Microtouch Systems, Inc. | Electronic whiteboard with multi-functional user interface |
US6014668A (en) * | 1996-10-07 | 2000-01-11 | Ricoh Company, Ltd. | Document information management system and method of generating a human-computer interactive document sheet allowing relevant information retrieval from a remote site |
US6157935A (en) * | 1996-12-17 | 2000-12-05 | Tran; Bao Q. | Remote data access and management system |
JPH10207898A (en) | 1997-01-20 | 1998-08-07 | Terumi Ishizawa | Picture displaying browser and its storage device and picture displaying method |
US6011537A (en) | 1997-01-27 | 2000-01-04 | Slotznick; Benjamin | System for delivering and simultaneously displaying primary and secondary information, and for displaying only the secondary information during interstitial space |
US6070176A (en) | 1997-01-30 | 2000-05-30 | Intel Corporation | Method and apparatus for graphically representing portions of the world wide web |
EP0863638A2 (en) | 1997-03-04 | 1998-09-09 | Nortel Networks Corporation | Data access method and apparatus for communications network |
US6182072B1 (en) | 1997-03-26 | 2001-01-30 | Webtv Networks, Inc. | Method and apparatus for generating a tour of world wide web sites |
US5982369A (en) | 1997-04-21 | 1999-11-09 | Sony Corporation | Method for displaying on a screen of a computer system images representing search results |
US5920856A (en) * | 1997-06-09 | 1999-07-06 | Xerox Corporation | System for selecting multimedia databases over networks |
US5983218A (en) * | 1997-06-30 | 1999-11-09 | Xerox Corporation | Multimedia database for use over networks |
US6335742B1 (en) * | 1997-07-24 | 2002-01-01 | Ricoh Company, Ltd. | Apparatus for file management and manipulation using graphical displays and textual descriptions |
JPH11250104A (en) | 1997-09-17 | 1999-09-17 | Sun Microsyst Inc | Picture information retrieving device, its method, method for selecting representative picture from video display, system to display picture from video data base and medium for recording picture information retrieving program |
JPH11110323A (en) | 1997-10-07 | 1999-04-23 | Oki Electric Ind Co Ltd | Document information providing system |
US6237011B1 (en) * | 1997-10-08 | 2001-05-22 | Caere Corporation | Computer-based document management system |
US6351467B1 (en) * | 1997-10-27 | 2002-02-26 | Hughes Electronics Corporation | System and method for multicasting multimedia content |
JPH11149469A (en) | 1997-11-14 | 1999-06-02 | Canon Inc | Device and method for processing document and storage medium |
US6678673B1 (en) | 1998-02-24 | 2004-01-13 | Koninklijke Philips Electronics N.V. | System and method for providing appropriate hyperlink based on identified keywords from text messages sent between users |
US6154738A (en) * | 1998-03-27 | 2000-11-28 | Call; Charles Gainor | Methods and apparatus for disseminating product information via the internet using universal product codes |
US6415282B1 (en) | 1998-04-22 | 2002-07-02 | Nec Usa, Inc. | Method and apparatus for query refinement |
US6271840B1 (en) | 1998-09-24 | 2001-08-07 | James Lee Finseth | Graphical search engine visual index |
US20050257400A1 (en) * | 1998-11-06 | 2005-11-24 | Microsoft Corporation | Navigating a resource browser session |
US20050047681A1 (en) * | 1999-01-28 | 2005-03-03 | Osamu Hori | Image information describing method, video retrieval method, video reproducing method, and video reproducing apparatus |
US6829619B1 (en) * | 1999-02-02 | 2004-12-07 | Fujitsu Limited | Information providing server |
US6904227B1 (en) * | 1999-02-15 | 2005-06-07 | Nec Corporation | Device and method for editing video and/or audio data recorded in a disc storage medium |
US7047180B1 (en) | 1999-04-30 | 2006-05-16 | Autodesk, Inc. | Method and apparatus for providing access to drawing information |
US20020166122A1 (en) * | 1999-06-25 | 2002-11-07 | Dan Kikinis | Image-oriented electronic programming guide |
US6456334B1 (en) * | 1999-06-29 | 2002-09-24 | Ati International Srl | Method and apparatus for displaying video in a data processing system |
US6389417B1 (en) | 1999-06-29 | 2002-05-14 | Samsung Electronics Co., Ltd. | Method and apparatus for searching a digital image |
US6845214B1 (en) * | 1999-07-13 | 2005-01-18 | Nec Corporation | Video apparatus and re-encoder therefor |
US6356908B1 (en) | 1999-07-30 | 2002-03-12 | International Business Machines Corporation | Automatic web page thumbnail generation |
US6665838B1 (en) | 1999-07-30 | 2003-12-16 | International Business Machines Corporation | Web page thumbnails and user configured complementary information provided from a server |
US8464302B1 (en) * | 1999-08-03 | 2013-06-11 | Videoshare, Llc | Method and system for sharing video with advertisements over a network |
US7051019B1 (en) | 1999-08-17 | 2006-05-23 | Corbis Corporation | Method and system for obtaining images from a database having images that are relevant to indicated text |
US6523063B1 (en) * | 1999-08-30 | 2003-02-18 | Zaplet, Inc. | Method system and program product for accessing a file using values from a redirect message string for each change of the link identifier |
US7039860B1 (en) * | 1999-10-01 | 2006-05-02 | Netspinner Solutions As | Creating web pages category list prior to the list being served to a browser |
US6964018B1 (en) | 1999-10-29 | 2005-11-08 | Sony Corporation | Document editing processing method and apparatus and program furnishing medium |
US6708172B1 (en) | 1999-12-22 | 2004-03-16 | Urbanpixel, Inc. | Community-based shared multiple browser environment |
US20060184886A1 (en) | 1999-12-22 | 2006-08-17 | Urbanpixel Inc. | Spatial chat in a multiple browser environment |
US6725214B2 (en) | 2000-01-14 | 2004-04-20 | Dotnsf | Apparatus and method to support management of uniform resource locators and/or contents of database servers |
US20080148330A1 (en) * | 2000-02-03 | 2008-06-19 | Gad Liwerant | Method and system for sharing video over a network |
US6313855B1 (en) | 2000-02-04 | 2001-11-06 | Browse3D Corporation | System and method for web browsing |
US6934964B1 (en) * | 2000-02-08 | 2005-08-23 | Koninklijke Philips Electronics N.V. | Electronic program guide viewing history generator method and system |
US20050010860A1 (en) | 2000-02-23 | 2005-01-13 | Yuval Weiss | Systems and methods for generating and providing previews of electronic files such as Web files |
US6711567B2 (en) | 2000-02-24 | 2004-03-23 | Inktomi Corporation | Delivering non-default items in association with search results |
US20020038299A1 (en) | 2000-03-20 | 2002-03-28 | Uri Zernik | Interface for presenting information |
US7000242B1 (en) * | 2000-07-31 | 2006-02-14 | Jeff Haber | Directing internet shopping traffic and tracking revenues generated as a result thereof |
WO2010124721A1 (en) * | 2009-04-27 | 2010-11-04 | Nokia Siemens Networks Oy | Optimized resource allocation signaling on a physical downlink control channel |
Non-Patent Citations (11)
Title |
---|
Bellamy et al., "Dynamic Visual Metaphors for News Story Abstractions," System Science, 1999. |
Borek, "A Hypertext Markup Language Interface for Multi-Product Database Libraries," Information Technology Conference, IEEE, 1998. |
Cox et al., "Visual Dialog Showing Speech Interaction with an Intelligent Agent," IBM TDB, Jan. 1996. |
Frankel et al., "WebSeer: An Image Search Engine for the World Wide Web," Aug. 1, 1996. |
Image and Video Searching (www.ditto.com, www.broadcast.com_ (Company Business and Marketing) Home Office Computing, 17, 11, 20 (Nov. 1999) Gale Group Magazine DB (TM). |
Li et al., "Multimedia Content Description in the Infopyramid," IEEE, 19998. |
Ma et al., "A Generalized Content-Based Image Retrieval System," ACM 1998. |
Mukheriea et al., "Amore: A World-Wide Web Image Retrieval Engine," ACM, May 1999. |
So, Siu-Kai et al., "Allocating Data Objects to Multiple Sites for Fast Browsing of Hypermedia Documents", Proceedings of the 22nd Annual International Computer Software and Applications Conference, Aug. 19-21, 1998, pp. 406-411. |
Taycher et al., Image Digestion and Relevance Feedback in the ImageRover WWW Search Engine, Visual 1997. |
Terveen et al., "Constructing, Organizing, and Visualizing Collections of Topically Related Web Resources," ACM Transactions on Computer-Human Interaction, vol. 6, No. 1, Mar. 1999. |
Also Published As
Publication number | Publication date |
---|---|
USRE42413E1 (en) | 2011-05-31 |
US6643641B1 (en) | 2003-11-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
USRE46967E1 (en) | System, apparatus, method, and computer program product for indexing a file | |
US8495049B2 (en) | System and method for extracting content for submission to a search engine | |
US7031968B2 (en) | Method and apparatus for providing web site preview information | |
JP3987133B2 (en) | Search hypertext information using profiles and topics | |
US7058944B1 (en) | Event driven system and method for retrieving and displaying information | |
US7716569B2 (en) | Framework for providing visual context to WWW hyperlinks | |
US5999929A (en) | World wide web link referral system and method for generating and providing related links for links identified in web pages | |
US6651065B2 (en) | Search and index hosting system | |
US7941416B2 (en) | Harvesting of media objects from searched sites without a user having to enter the sites | |
US5826031A (en) | Method and system for prioritized downloading of embedded web objects | |
US8832058B1 (en) | Systems and methods for syndicating and hosting customized news content | |
US6338059B1 (en) | Hyperlinked search interface for distributed database | |
US6970863B2 (en) | Front-end weight factor search criteria | |
US20030014415A1 (en) | Systems and methods for generating and providing previews of electronic files such as web files | |
US6347320B1 (en) | Search parameters | |
US20090094327A1 (en) | Method and apparatus for mapping a site on a wide area network | |
US20110238662A1 (en) | Method and system for searching a wide area network | |
JP2009059353A (en) | Device and method for retrieving information selectively and enabling display of information after that | |
CN1494680A (en) | Gethering enriched server activity data of cached web content | |
WO2008141295A1 (en) | Keyword generation system and method for online activity | |
KR20040017008A (en) | System and method for offering information using a search engine | |
US6463455B1 (en) | Method and apparatus for retrieving and analyzing data stored at network sites | |
US7085801B1 (en) | Method and apparatus for printing web pages | |
CA2452269A1 (en) | Method and system for standard bookmark classification of web sites | |
US8131752B2 (en) | Breaking documents |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: BAYARD CHIMNEY ROCK LLC, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SNYDER, RUSSELL;REEL/FRAME:021424/0368 Effective date: 20040915 |
|
AS | Assignment |
Owner name: MINERAL LASSEN LLC, NEVADA Free format text: MERGER;ASSIGNOR:BAYARD CHIMNEY ROCK LLC;REEL/FRAME:036928/0582 Effective date: 20150811 |