JP6568985B2 - Batch optimized rendering and fetch architecture - Google Patents

Description

  The present invention relates to a batch optimized rendering and fetch architecture.

  The World Wide Web is rich in information. Today, it is estimated that there are over 1 trillion unique web pages. Many of these pages are dynamically created on the New York Times home page, for example, and include links to embedded content such as images and videos that can affect the content and appearance of the rendered web page. Have. For example, when a browser executes a script such as JavaScript code, this will affect how the web page will look to the user after the browser has finished rendering the web page. And can change the content and / or appearance of the page. As another example, some web pages use style sheets that tell the browser how to change the appearance of the text. A typical web page can have handlers for such additional embedded items, some of which are specifically designed for or directed to the browser rendering engine. Additional information generated by the rendering process can be useful for downstream systems, such as Internet search engines. Rendering a single web page in real time for a single user's web browser is relatively easy, but many more pages, like all the pages on the World Wide Web (1 trillion pages) Or rendering only the top 1% on the World Wide Web (10 billion pages) in real time.

  Embodiments include a rendering server and a fetch server optimized for batch rendering of web pages for downstream users, such as a web page indexing system. When a downstream user identifies a web page (eg, using its URL) that has one or more embedded items, the downstream user renders the URL for the rendering server to generate a rendering result. May be requested. A rendering server can include multiple (eg, tens of thousands) rendering engines. Each rendering engine simulates a browser kernel optimized for batch rendering that eliminates many rendering errors. During rendering, when the rendering engine finds an embedded item, the rendering engine requests the embedded item from the fetch server. The fetch server includes an embedded item data store matched by an identifier for each embedded item (eg, its URL) and content for items retrieved by a web-crawler. Prior to looking for a data store for an embedded item, the fetch server may rewrite the URL using rewrite rules. When the content for the UR overlaps with another embedded item (eg, represented by a redirect URL), the rewrite rule may replace the URL with the redirect URL. If the requested embedded item is a duplicate, the fetch server may rewrite the URL to use the redirect UR, which instead of fetching the content for the requested URL, Make the already searched content available for. Such a deduplication method can dramatically reduce the actual number of crawl requests made by the fetch server and can improve the response time of the rendering engine. The rewrite rule may indicate that the URL is blacklisted. In some embodiments, the fetch server may store the size rather than the actual content of the embedded image. When the rendering engine requests an image, the fetch server may generate a mock image having the size of the image and return the mock image to the rendering engine. When the rendering engine has finished rendering the web page, it can also provide the rendering result to downstream users, such as an index engine, that can use the rendering result information to improve the processing of the web page. Good.

  In one aspect, a computer system includes at least one processor and a memory when the memory is executed by a content data store for an embedded item and at least one processor. Stores instructions that cause the system to perform operations. The operations include receiving a request to render a web page from a batch process and identifying an embedded item on the web page. The operation determines, based on rewrite rules, that the embedded item has content that is a duplicate of content for a previously fetched embedded item, and in response to the determination, from the data store Providing the content for a fetched embedded item, generating rendering results for the web page using the content for a previously fetched embedded item, and a batch process Providing rendering results.

  One or more embodiments of the subject matter described in this specification can include one or more of the following features. For example, determining that an embedded item has content that is a duplicate of content for a previously fetched embedded item includes matching the embedded item to a rewrite rule template, wherein the rewrite rule includes a redirect identifier. Can be included. In such embodiments, providing the content for previously fetched embedded items may include a URL without the content and / or query string for previously fetched embedded items. Using the redirect identifier to indicate a template.

  As another example, the embedded item may be a first embedded item and the action may include identifying a second embedded item on the web page, wherein the second embedded item is blacklisted. If the second embedded item is blacklisted with no fetch content for the second embedded item, returns an error, and no content for the second embedded item It may include generating a rendering result. As another example, when generating a rendering result, the action may include the use of a virtual clock, which proceeds independently of real time. In another example, when generating a rendering result, the action may include the use of a virtual clock and does not proceed while the virtual clock waits for the provided content of the previously fetched embedded item.

  As another example, the embedded item may be a first embedded item, the action identifying a second embedded item on the web page, and determining that the second embedded item includes an image. , Using a size table to generate a mock image that specifies a size for the second embedded item and using the mock image when generating a rendering result.

  In another aspect, a computer-implemented method includes receiving a request to render a web page from a batch process, and initializing a virtual clock and a task list for rendering the web page, The virtual clock includes an initialization step that is still stopped when a request for an embedded item is outstanding and the task is ready. The method also includes generating a rendering result for the web page and generating a rendering result for the batch process when the virtual clock matches the runtime for the stop task in the task list. .

  One or more embodiments of the subject matter described in this specification can include one or more of the following features. For example, initializing the task list may include adding a stopped task with a runtime set at a predetermined time added to the virtual clock. The predetermined time may be at least 5 seconds. As another example, the method uses a virtual clock to the runtime of a task in the task list when a request for an embedded item is not outstanding and there is only one task in the task list that has a runtime greater than the virtual clock. A further step may be included. As another example, the method includes identifying an embedded image of a web page, requesting content for the embedded image, and specifying a size for the embedded image in response to the request, but empty. Receiving a mock image having the content of and using the mock image when generating a rendering result. As another example, the batch process may be an index engine and the method further includes demoting a rank for the web page based on rendering result information and / or dynamically The method further includes using the rendering result to index the generated content.

  In another aspect, a method includes receiving a request from a batch rendering process for a uniform resource locator (URL) of an embedded item in a web page and applying a rewrite rule that determines the rewritten URL. . The method may include determining whether content for the rewritten URL exists in the data store and providing the content to the batch rendering process when the content is present. When content is not present, the method initializes content fetching, wherein the batch rendering process is configured to wait without timeout between fetches, and content from the web patrol engine. Receiving, providing content for a batch rendering process, and storing data store content. The content may be used by a batch rendering process to claim a web page rendering result.

  One or more embodiments of the subject matter described in this specification can include one or more of the following features. For example, the step of applying the rewrite rule may include the step of matching the URL to a template associated with the redirect URL, where the redirect URL is the rewritten URL when the URL matches the template. When the URL does not match the template, it is determined that the URL is a rewritten URL. As another example, the method may determine that the content for a rewritten URL is stale based on the rate of change or the type of embedded item stored in the data store; In response to determining that the content is stale, receiving updated content from the web patrol engine, updating the data store with the updated content, and content for the rewritten URL Providing updated content may also be included.

  In another aspect, a computer system includes at least one processor and a memory that stores a table of sizes stored by image identification and instructions that, when executed by the at least one processor, cause the system to perform an action. The operations may include identifying an embedded item on the web page, determining a size for the embedded image from a size table, and using the size to generate a mock image. The operation may include generating a rendering result for the web page using the mock image.

  In another aspect, a computer-implemented method includes receiving a request for rendering a web page from a batch process and identifying at least one embedded image of the web page. The method also includes receiving a mock image from the fetch server and using the mock image to render a rendering result for the web page, the mock image having an embedded image size and empty content. In some embodiments, the method may provide rendering results to a batch process that requested a web page.

  In another aspect, a non-transitory computer readable medium may include instructions executable by at least one processor formed on a circuit board to cause a computer system to perform one or more of the above-described methods.

  One or more embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages. As an example, since the batch rendering engine is not coupled to an input device (e.g., keyboard, mouse) or output device (e.g., display, touch screen, etc.), the rendering engine may be a machine friendly API rather than a user friendly API, for example. Can be simpler and more sophisticated than an actual browser renderer. Also, since the rendering engine does not need to display the final rendered page or interact with the user, the rendering engine can use a virtual clock that progresses based on the finished task rather than the actual time, This speeds up the rendering process and avoids common errors. For example, fetching in a batch environment can be much slower than in a personal web environment and can cause many timeout errors. The virtual clock hides fetch latency and avoids timeout errors. Virtual clocks also allow for more definitive results. For example, instead of replacing a date / time component with a fixed time in a URL that includes a date / time component, the system may use the value of a virtual clock. This means that not all time parameters of a web page have the same value, but every time a web page is rendered, a particular time parameter has the same value. This flexibility allows the system to advance the time, which is important in some web pages for the accuracy of the rendered results, but the requested set of URLs remains the same throughout the rendering. Is always guaranteed (reduces crawl requests). The system may avoid fetching unnecessary items such as blacklisted items. Saving the size for the image rather than the actual content of the image reduces storage requirements for the image at the fetch server, requires less data to be transferred to the rendering engine, and render time in the rendering engine Is further improved. Rewriting the URL avoids fetching duplicate content and makes the batch rendering process even faster.

1 illustrates an example system in accordance with the disclosed subject matter. It is a block diagram of the web page which has an embedded item. FIG. 2 is a block diagram of a batch rendering engine according to an embodiment. 4 is a flow diagram illustrating an example process by which a batch rendering engine may render a web page with embedded objects, according to an embodiment. 4 is a flow diagram illustrating an example process by which a batch rendering engine advances a virtual clock, according to an embodiment. 3 is a flow diagram illustrating an example process for a fetch server to provide content for an embedded item to a batch rendering engine, according to an embodiment. 6 is a flow diagram illustrating an example process in which a fetch server provides mock images to a batch rendering engine, according to an embodiment. 6 illustrates an example of a computing device that can be used to implement the disclosed technology. 6 illustrates an example of a distributed computing device that can be used to implement the disclosed technology.

  In order to fully render a web page, the content of all embedded external resources of the web page must first be obtained. Such resources include, but are not limited to, external images, JavaScript code, and style sheets. Often, the same external resource is embedded in many different web pages. While it is efficient for a single user's web browser to request external web page resources such as Google Analytics JavaScript code in real time (ie when a web page with embedded resources is rendered), batch rendering It is neither feasible nor efficient that the engine does. For example, a batch rendering engine for the web page indexing process is designed to quickly render a large number of web pages at once, efficiently and quickly. However, fetching embedded external resources can be slow, and such resources are for batch processing purposes (eg, without a human user seeing the final rendered product) It may not be important. To improve the processing time of rendering web pages in a batch environment, the rendering engine works with a virtual clock, works with a fetch server to avoid duplication and unnecessary fetching, and visual processing or web Other user-oriented elements on the page may be minimized.

  FIG. 1 is a block diagram of a system according to an example embodiment. System 100 may be used to efficiently and quickly render web pages in batch mode for a request process. The request process illustrated in system 100 is an index engine for an Internet search engine, but implementations are not limited to the index engine as a downstream user of the rendered web page. For example, the request process may be an analysis engine that analyzes the page and troubleshoots delay, or an analysis engine that determines whether a tool like Google Analytics is set up correctly, or It may be an advertising system or another system that relies on automated interaction with complex web pages, such as filling out forms and clicking on elements. Accordingly, system 100 may be described as using batch generated rendering results for indexing, and system 100 may be used for other batch systems where the information provided in the rendering results is useful. Can be done.

  System 100 may be a computing device or a device having many different device forms. For example, the system 100 may be a nest-shot server, a group of such servers, a client server system, or a rack server system. Further, the system 100 may be implemented with a personal computer. The system 100 may be a computer device example 800 as illustrated in FIG. 8, or a computer device 900 as illustrated in FIG.

  The system 100 includes a web patrol engine 130, a request process such as an index engine 110, a rendering server 140, and a fetch server 150. Web patrol engine 130, rendering server 140, and fetch server 150 may work together to efficiently render a large number of web pages, such as web pages found on the World Wide Web.

  Index engine 100 includes one or more processors configured to execute one or more machine-executable instructions or pieces of software, firmware, or combinations thereof to create index 115. Can be included. For example, the index engine 110 may receive information from the server via the web patrol engine 130. The index engine 110 may process the content of the received web page to generate the index 115. Server 190 may be any type of computing device accessible via the Internet that hosts one or more web pages or resources embedded in one or more web pages. Web pages accessed by the patrol engine 130 may include embedded items such as style sheets, Java Scripts, images, etc., some of which may change the content and layout of the rendered web page. The index engine 110 can index what is provided via the patrol engine 130. The index engine can request the rendering server 140 to provide rendering results for a browser-rendered web page that includes layout information and dynamic content otherwise not available to the index engine 110. The index engine 110 can use the rendered result content to improve the information available about the documents in the index 115. For example, the index engine 110 may change the rank of the text element of the web page based on the text position or size of the image of the web page. As an example, text that represents a major news (eg, can be viewed without scrolling) may be considered more important than the text of an advertisement. As another example, the text in an advertisement may be considered less important for a web page. In addition, when some content is dynamically generated, for example when the web page is not available until it is rendered, the index engine 110 uses the rendering results to index the dynamically generated content. May be. Although not shown in FIG. 1 for the sake of brevity, in some embodiments the index engine 110 may be distributed across one or more separate computing devices.

  Similar to index engine 110, query engine 120 may include one or more servers that use index 115 to identify the search results of query 182 using, for example, conventional or other information retrieval techniques. . Query engine 120 may include one or more servers that receive query 182 from a requester, such as client 180. Query engine 120 may use index 115 to identify the document in response to the query, and may provide information from the response document to the requester as search results 184. In some embodiments, the query engine 120 may use the rendering results of the rendering results data store 148 to provide thumbnails as part of the search results 184. Query engine 120 may include, for example, a ranking engine that uses one or more ranking signals to calculate a score for the document in response to the query. The one or more ranking signals can be based on content obtained from rendering results associated with the document. The ranking engine may use the score to rank documents found in response to the query.

  The system may include a web patrol engine 130. Web patrol engine 130 may include one or more processors configured to execute one or more machine-executable instructions or portions of software, firmware, or combinations thereof. The web patrol engine 130 may be a computing device such as a standard server, a group of such servers, a client server system, or a rack server system. In some embodiments, the web patrol engine 130 may share components such as a memory or hardware processor with other components of the system 100 such as the fetch server 150 or the index engine 110. The web patrol engine 130 may crawl web pages that can be found on the World Wide Web. When the web patrol engine 130 receives a crawled web page, i.e., receives content for a crawled web page, the web patrol engine 130 may be the index engine 110 or the fetch server 150. You may provide content. Web patrol engine 130 may store the content in a data store (not shown) and provide the location to the requestor. The web page content used herein refers to the HTML code provided to the web page rendering engine and used to render the web page for display in a web browser, such as a style sheet, Java Script , Any web page, or any link to an external object embedded in the web page, such as an image file. Web patrol engine 130 may be used by fetch server 150 to fetch the contents of these embedded items. The web patrol engine 130 may provide embedded item content to the fetch server 150, or store the fetched content in a data store such as the embedded item table 152. The web patrol engine 130 may notify the requester when an embedded item is crawled.

  As described above, the system 100 includes the fetch server 150. The fetch server 150 may include one or more processors configured to execute one or more machine-executable instructions or portions of software, firmware, or combinations thereof. The fetch server 150 may be a computing device such as a standard server, a group of such servers, a client server system, or a rack server system. In some embodiments, the fetch server 150 may share components such as a memory or hardware processor with other components of the system 100 such as the rendering server 140, the web patrol engine 130, or the index engine 110. Good. Fetch server 150 is configured to request that web crawling engine 130 fetch the content of a particular embedded item, for example by its URL, and receive the fetched content and crawl time of the requested embedded item. The The fetch server 150 may receive content and fetch times directly from the web patrol engine 130 or from an embedded item table 152 that is updated by the web patrol engine. The fetch server 150 may receive a request for an embedded item from the rendering engine 142. The fetch server 150 may provide a response to the requesting rendering engine 142. The response may include content fetched or stored in the embedded item table 152, a mock image based on the image size table 156, or an error response. In some embodiments, the fetch server 150 may provide embedded item content by sending the embedded item content and crawl time to the rendering engine 142 of the rendering server 140 that requested the embedded item. Alternatively, the fetch server 150 can provide content by notifying the rendering engine 142 that the embedded item content and crawl times are available via a specified location in the embedded item table 152, and rendering The engine 142 can obtain web page content and crawl times from a data store.

  The fetch server 150 may apply the URL rewrite rule 154 to the requested embedded item (eg, the requested URL). The URL rewriting rule 154 includes a rule for rewriting a URL when the URL is associated with the same content as another URL. This is when the website owner wants the browser to download content each time a resource is requested and provides a dynamically generated or cache-busting URL for that purpose. Often occurs. Such URLs often have a timestamp or embedded random string as part of the URL, for example, each time a web page is rendered by JavaScript that generates a cache busting URL, the URL is unique. To be. However, the content for dynamically generated URLs provided from the hosting server is not changed or changed in a meaningful way for batch rendering purposes. Fetch server 150 may use URL rewrite rules 154 to respond more efficiently to requests for embedded items. For example, the URL rewrite rule 154 may include a pattern or template, and URLs that match the rule template return the same content, eg, duplicate content. In some embodiments, the template may be determined offline or by a batch process that uses fetch logs to compare the content of various URLs and identify common URL patterns for URLs with duplicate content. . The fetch log may be maintained by the web patrol engine 130 or the fetch server 150, for example. The template may be user input. If the requested embedded item has a URL that matches one of the templates, the URL rewrite rule 154 may tell the fetch server 150 that the requested item is a duplicate, and the requested URL You may instruct it to rewrite with a redirect URL. This is associated with URLs that have previously fetched content, eg, embedded item table 152 content. The previously fetched embedded item URL may be considered a redirect URL. This allows the fetch server 150 to avoid unnecessary fetches, speeds up the response to the requesting batch rendering engine 142, and removes stress on the hosting server caused by excessive fetch requests. Of course, if the requested URL does not match the template of the URL rewriting rule 154, rewriting the URL results in the requested URL not being changed.

  The URL rewrite rules 154 may include blacklisted URL patterns or templates. If the requested embedded item matches the URL pattern on the flack list, the system may return a predetermined error rather than trying to fetch the content for the URL. The blacklisted URL may be entered by the user into the URL rewrite rules 154 after determining that the content is not required for batch rendering purposes. An example of this is the Google Analytics JavaScript code included in many web pages. This JavaScript code may not be considered critical to the layout of the rendered page and need not be executed for the purpose of the batch rendering engine. Thus, for rendering efficiency, some embedded items may be blacklisted using URL rewrite rules 154. In some embodiments, rather than returning an error for a blacklisted URL, the system will enter an entry in the embedded item table 152 that does not expire and has predetermined content suitable for the embedded item: The URL may be rewritten using the redirect URL described above. In some embodiments, URL rewrite rules may flag URLs as blacklisted when they match a template. URL rewrite rules 154 can dramatically reduce the number of embedded items fetched through the web patrol engine 130, improve response time to any request for resources of the fetch server 150, and identify The fetch amount of the server 190 is minimized. Minimizing the fetch amount ensures that the system does not burden the server 190 with fetch requests. In some embodiments, fetch server 150 and / or web crawling engine 130 may be configured to limit the number of fetch requests destined for server 190, and if the request exceeds the limit, the system may request May begin to queue. If the queue is too large, the system may fail fetch requests. Therefore, the URL rewrite rule 154 can also minimize the fetch amount.

  In some embodiments, the fetch server 150 may include an image size table 156. The image size table 156 may be a key value store that associates an image URL with a known size for the image. The known size may be determined when the image is fetched. Using the requested image size, fetch server 150 may generate a mock image that has the same size as the requested image, but is empty content or has simple tiles as content. A mock image is a valid image having the same size as the requested image but not the same image data. Since fetch server 150 fetches content for the batch rendering engine, the actual image may not be important to the rendering result, but the size of the image can affect the layout of the rendered page. Using a mock image instead of a real image results in a very small file size (eg, tens of bytes per image), which is the network bandwidth for transferring mock images, and for batch rendering engines Save processor and memory resources. In some embodiments, the image dimension table 156 may be a key value storage device such as SSTable, but the dimension table 156 may be any data structure that stores sizes by image identifier.

  System 100 may include an embedded item table 152. Embedded item table 152h may be keyed by URL and may store fetched content for embedded items returned from web patrol engine 130. In some embodiments, the embedded item table 152 may also store a crawl history. For example, in some embodiments, the embedded item table 152 may include content fetched over a period of time, eg, 7 days, 2 weeks, etc. The embedded item table 152 may include a change rate based on the crawl history. In some embodiments, the embedded item table 152 may be implemented as a BigTable, a relational database, a Hadoop Distributed File, etc. Fetch server 150 may use embedded item table 152 to quickly return content for previously fetched embedded items. Since fetch server 150 can handle requests for thousands of batch rendering systems, it is likely that the requested embedded item has been previously fetched in response to a previous fetch request. When the fetched content is located in the embedded item table 152, rather than asking the web patrol engine 130 to provide the content, the fetch server 150 may respond to the request using the content of the embedded item table 152. Good. This reduces the burden on server 190 that stores the fetched content and allows fetch server 150 to respond more quickly to requests for embedded items. The fetch server 150 can further reduce crawl requests with URLs that eliminate duplication using the URL rewrite rules 154.

  At any stage of the rendering process, if the content of one or more requested embedded items is not stored in the embedded item table 152 or is out of date, the fetch server 150 schedules the crawling of the requested embedded item. Therefore, the web patrol engine 130 may be instructed. Once the web patrol engine 130 crawls the rendered embedded item, it notifies the fetch server 150. The fetch server 150 may then store the fetched content item in the embedded item table 152 along with the crawl time. If the embedded item is an image, the fetch server 150 may alternatively or additionally store the size of the fetched image in the image dimension table 156 along with the crawl time. The fetch server 150 may then send the requested content back to the requesting rendering engine 142 instead of an image file to which a mock image having an image size may be sent.

  The system 100 includes a rendering server 140. Rendering server 140 may include one or more processors configured to execute one or more machine-executable instructions or portions of software, firmware, or combinations thereof. The rendering server 140 may be a computing device such as a standard server, a group of such servers, a client server system, or a rack server system. In some embodiments, the rendering server 140 may share a component such as a memory or hardware processor with another component of the system 100 such as the rendering server 140, the fetch server 150, or the index engine 110. . The rendering server 140 receives a request from the index engine 110 or another request process to render a particular web page. In other words, the rendering server 140 may receive the URL of the requested web page. The rendering server 140 may include one or more rendering engines 142. In some embodiments, the rendering server 140 includes tens of thousands of rendering engines 142 and may perform load balancing to select a rendering engine 142 that renders the web page. Once the rendering engine 142 is selected, the rendering engine 142 attempts to render the web page for the rendering result. A web page usually includes additional embedded items and may be referred to as an embedded web page.

  Each rendering engine 142 is configured to emulate a renderer for a personal web browser, but with optimization for batch rendering. Therefore, after the rendering engine 142 receives the web page to be embedded, the task representing the work that the rendering engine 142 performs to generate the rendering result may start to be entered in the task list. Many tasks may be scheduled to be executed immediately, but some tasks may be scheduled in the future. One batch optimization of the rendering engine 142 is to use a virtual clock and add a task to the task list indicating that the rendering is complete at a particular time. For example, in some embodiments, the task may indicate that the rendering is complete in the current time + 20 seconds. The predetermined time may be based on the time it takes to design a web page so that a large number of web page designers can see it completely, eg, any animation or layout change is designed to end within a predetermined time. Most users are not willing to wait very long for the page to load, so the predetermined time may be between 5 and 20 seconds, but can be longer in some situations. Rendering engine 142 does not take all 20 seconds to use the virtual clock, and often when embedded items have not been crawled (eg, fetch server 150 can show the contents of embedded item table 152). Full rendering can occur in seconds. Accordingly, the task that generates the final rendering result may be added to the task list at a start time of 20 seconds from the current time. The current time is based on the initialization time of the virtual clock, which can be zero or the current time of the actual clock.

  As part of the rendering process, eg, one of the rendering tasks, the rendering engine 142 may determine whether the embedded web page includes any embedded items such as style sheets, image files, Java Scripts, etc. . These embedded items are called primary embedded objects. If the web page does not contain any embedded objects, the rendering engine can process the web page immediately into the rendering result and store the rendering result in the rendering result data store 148. However, if the web page includes embedded items, the rendering engine 142 may extract all the embedded items and send a request to the fetch server 150 for the embedded item content. Each requested embedded item is represented by a respective URL. The rendering engine 142, however, does not stop rendering or time out while waiting for fetched resources. Rather, since the rendering engine 142 uses a virtual clock, waiting for resources fetched via the web patrol engine 130 will not advance the clock and the rendering engine will not time out, as described in more detail below.

  When content for a requested embedded item is received, the rendering engine 142 may add a task to the task list to process the content. Part of the content processing may include finding out whether the requested embedded object (ie, the primary embedded object) itself has an embedded object (ie, a secondary embedded object). If the primary embedded object does not include a secondary embedded object, the rendering engine 142 can continue to work on a rendering task (eg, executing JavaScript code) that changes the properties of the image. However, if the primary embedded object includes one or more secondary embedded objects, the rendering engine 142 requests a secondary embedded object from the fetch server 150. This process of discovering and requesting embedded objects causes the rendering engine to discover, request, and receive all objects (eg, primary, secondary, third, etc.) embedded in the rendered web page. Repeat until.

  Each embedded item request may be a task in the task list that is deleted once fetch server 150 returns the content for that time. When content is returned, the rendering engine 142 may add tasks to process the content, which in turn may add additional tasks, such as changing the opacity of the image, running a script, etc. . Each task may be associated with a runtime. Some tasks may have a future runtime. For example, to fade in (or out) an image, the browser may add several tasks to the task list, each changing the opacity of the image over a time interval. As described in more detail below, the rendering engine 142 may use a virtual clock rather than real time for the task list to determine when a task is ready to run.

  The rendering engine 142 works on the process of rendering tasks in the task list until rendering is complete, eg, until a rendering result is generated. The rendering engine 142 may then store the rendering results in the rendering results data store 148 and / or provide the rendering results to a rendering process (eg, the index engine 110). A rendering process such as the index engine 110 may then use the information extracted from the rendering results when processing the web page. For example, the request process may use JavaScript errors, layout information, style information, ad space information, fetched resource list, performance statistics, etc., all of which are included in the rendering result, but for the request process It cannot be used otherwise.

  System 100 may communicate with client 180 and server 190 via network 170. The network 170 may be, for example, the Internet 170, a network 170 that may be a wired or wireless local area network (LAN), a wide area network (WAN), or combinations thereof, such as gateway devices, bridges, switches, or Similar implementations may be used. Via the network 170, the query engine, the web patrol engine 130 and / or the fetch server 150 may communicate with the client 180 and / or the server 190 and send and receive data.

  System 100 may include other components not shown for the sake of brevity. For example, one or more of the index engine 110, query engine 120, web traversal engine 130, rendering server 140, and fetch server 150 may be distributed across one or more computing devices. Similarly, the index 115, rendering result data store 148, embedded item table 152, and image size table 156 may be stored across multiple computing devices. In some embodiments, the various components of the system 100 may share the hardware components of the computing device or may be logical partitions of the same computing device.

  FIG. 2 is a block diagram of a web page having embedded objects. As shown, the web page 200 can include a plurality of embedded items. These embedded objects can include, but are not limited to, other web pages 210, style sheets 220, image files 230, so-called cache busting URLs 240, and JavaScript code 250. Different types of embedded objects added are of course possible. Furthermore, each object embedded in the web page 200 may embed another object. For example, the web page 210 embedded in the web page 200 may embed other web pages, image files, style sheets, and the like. Similarly, the style sheet 220 embedded in the web page 200 may embed other objects such as a background image file. Further, each of the objects embedded in the web page 210 or the style sheet 220 may further embed an object. In order to fully render such a web page into an image file, the batch rendering engine requires each of the embedded objects 210-250 (primary embedded objects) and all objects embedded in the embedded objects 210-250 ( Secondary embedded objects), and all objects embedded in objects embedded in embedded objects 210-250 (third embedded objects) and others must be requested.

  As mentioned above, each user's web browser can efficiently request all of these embedded objects and use them to fully render and display the web page 200 in real time. , The batch rendering engine is optimized to avoid fetching duplicate or unnecessary content, to wait for embedded object content and not to time out, and to finish rendering as soon as possible regardless of the internal timing of the task Can be. Thus, a system such as that disclosed in FIG. 1 can be used to efficiently render a large number of crawled web pages.

  FIG. 3 is a block diagram of several components of the batch rendering engine 142, according to one embodiment. The batch rendering engine may include additional components not shown in FIG. The batch rendering system 142 includes a page task list 305, a virtual clock 310, and a rendering result 315. The virtual clock 310 may be used to distort the timeline to load a web page and avoid numerous errors that can occur by waiting for fetched resources. The virtual clock 310 may be initialized to zero or the current clock time at the start of the rendering process, and the page task list 305 is currently ready to execute when the rendering engine is not waiting for the embedded item fetch. You may proceed only when there is a completed task. When the virtual clock is advanced, the rendering engine 142 advances the virtual clock 310 based on the page task list 305. In other words, the rendering engine 142 advances the virtual clock 310 to the time represented by the next task to occur. In this sense, fetching embedded items and executing Java Scripts does not take virtual time and avoids the entire class of errors generated by live (or personal) browsers. Further, the rendering process may end in real time much earlier than the time specified in the task list. For example, the “Generate Final Rendering” task is set to occur in 20 seconds, but the virtual clock usually depends on how long it takes to actually complete the tasks in the page task list 305, Advances 20 seconds in actual seconds. The “generate final rendering” task in the page task list 305 is an example of a stop task that is notified to the batch rendering engine 142 when rendering is completed.

  The rendering engine 142 may render the rendering result 315 of the embedded web page. The rendering result 315 may include various components. For example, the rendering result 315 may include an image 316 of the rendered page. The image 316 may be an image that is displayed to a user of a live (or personal) web browser and can be used, for example, to display a thumbnail of the rendered page to the user. The rendering result 315 can also include a document object model (DOM) tree 317. The DOM tree 317 represents the HTML structure of a web page. For example, the system may determine the text of the document visible to the token or user by processing the DOM tree. The rendering result 315 may include a layout 318. Layout 318 includes a box for each element of the web page, and the box determines the coordinates of the elements of image 316. For example, a layout can include a box representation of DOM nodes in a DOM tree (not all DOM elements have a corresponding rendering box). Boxes can be organized in a tree structure called a rendering tree. Thus, for example, a table may be represented by a box in the layout and a paragraph may be represented by another box in the layout. Thus, the layout 318 provides instructions such as where elements occur on the web page and how much space is required on the web page. Thus, the layout 318 determines how much of the advertisement is on the web page, how prominent the paragraph is (eg, above-the-fold or below-the-fold). )), Provide information such as whether the element is visible. Thus, layout 318 provides geometric information about the elements of the web page. The rendering result 315 may include an error 320. The error 320 includes an error caused as a result of executing a script such as Java Script. The rendering result 315 may include a list of embedded resources 319 fetched during rendering and may include other elements generated as part of the rendering process. Thus, the rendering result 315 provides information that is not available to the requesting process only through fetching content from the hosting server. For example, the indexing engine can rank the elements of the index, provide invisible elements as part of the fragment, and use rendering result information to index dynamically generated content . The dynamically generated content is content that exists after rendering the web page, and is not present in the crawled content.

  FIG. 4 is a flow diagram illustrating an example process 400 by which a batch rendering engine can render a web page with embedded objects, according to an embodiment. Process 400 may be performed by a system such as system 100 of FIG. A system such as an advertising system, or Internet index system, may use process 400 to generate a batch mode web page rendering result at the request of a downstream process. In some embodiments, process 400 may be performed by a rendering server's batch rendering engine and may be initiated in response to a request from a requesting process.

  Process 400 may begin by receiving a request to render a web page (405). In some embodiments, the request may include the URL of the requested web page and / or the fetched content and associated metadata (eg, crawl time). In some embodiments, rather than receiving web page content, the batch rendering engine receives notification that web page content is available in the database, and the content and associated metadata (eg, from the database) , Crawl time). The fetched content may be provided, for example, because the requested process has already fetched the content. The batch rendering engine may begin by initializing the virtual clock and adding a stop task to the task list (410). For example, the batch rendering engine may add a stop task to the task list that sets the virtual clock to zero and causes the rendering engine to determine that the rendering engine has completed in a predetermined time. The runtime associated with this stop task may be the time it takes for most web pages to complete loading on an individual user's machine. For example, the time may be 15 or 20 seconds. As part of the rendering start, the batch rendering engine adds other tasks to the task list, such as fetching web page content (if no content was provided) and processing web page content. Also good. These tasks may be added at virtual time zero and can thus be started immediately, for example.

  The batch rendering engine may then begin to work on the tasks in the task list (415). For example, as part of processing web page content, the batch rendering engine may identify one or more embedded items (420). The batch rendering engine may then request the content of the embedded item from the fetch server (425). The fetch server may be the fetch server 150 of FIG. In some embodiments, the batch rendering engine may track whether the identified embedded items and the fetch server have returned content for each embedded item. In some embodiments, this list of embedded items may be included in the rendering results of the web page. After the batch rendering engine requests an embedded item, the batch rendering engine may continue working on another task (415) that is ready to execute while waiting for the fetch server to return content. If there are no tasks ready to run at the current virtual time, the batch rendering engine may wait for a response from the fetch server. Although the fetch is outstanding, the batch rendering engine does not advance the virtual clock, so the batch rendering engine does not time out waiting for the fetch.

  Upon receiving a response from the fetch server, the batch rendering engine processes the content of the embedded item (430). For example, in response to receiving the content, the batch rendering engine may add a task to the task list, such as analyzing the received content of the embedded item. These tasks may be given the start time of the current virtual clock, which indicates that the task is ready to run (eg, the current time on the virtual clock). Parse the received content and the batch rendering process may add additional tasks to the task list, whether for the originally requested web page or for the embedded item. For example, analyzing the contents of an embedded item may detect additional embedded items (eg, secondary embedded items), which causes the batch rendering engine to request the embedded item and return the content when it is returned. It may be analyzed. If the content includes a script such as JavaScript, running the script may cause the additional task to perform additional tasks such as generating a layout or changing the appearance of one or more elements on a web page . Some of these tasks may be scheduled to start in the future. For example, if the opacity of the image is changed at regular intervals, it is displayed to the user as if the image is fading in. Each change in opacity is a task, and the script causes the task list to add several such tasks, each adding the amount specified in the current virtual clock runtime.

  As part of the rendering process, the batch rendering engine may determine whether rendering is complete (435). This determination may be performed, for example, every time the batch rendering engine completes a task, or at a predetermined time interval. Rendering may end when the virtual clock reaches the time specified in the stop task. Since the virtual clock does not advance while the embedded item fetch is outstanding, the batch rendering engine is guaranteed to receive a request for each fetch request when the virtual clock reaches the time specified by the stop task. . Thus, the batch rendering engine never times out waiting for resources.

  If the rendering is not complete (435, No), the batch rendering engine may continue to work on the tasks in the task list, wait for a response to a request for one or more embedded items, and so on. If the render is complete (435, yes), the batch rendering engine may finalize the rendering results for the requested web page (440) and return the rendering results to the requesting process. The rendering result element may have been previously generated as a result of a task completed by the batch rendering engine. For example, a list of embedded items fetched during script execution and errors encountered may be generated before rendering is complete. Another element, such as determining the layout, may occur after rendering is complete. In some embodiments, the batch rendering engine does not determine the layout until rendering is complete unless a script that is executed as part of the rendering process requires the position of the element. Even if the layout is generated before rendering is complete, the batch rendering engine may eventually generate the layout as part of finalizing the rendering results. Thus, finalizing the rendering result may include generating new elements and collecting already generated elements. In some embodiments, the batch rendering engine may store the rendering results in memory and provide the location of the rendering results to the requesting process. In some embodiments, the system may store the rendering result with a time stamp indicating when it was generated, and may store more than one version of the rendering result. Producing rendering results in batch mode with batch processing optimization, then process 400 ends.

  FIG. 5 is a flow diagram illustrating an example process 500 for a batch rendering engine to advance a virtual clock, according to one embodiment. Process 500 may be performed as part of determining whether rendering is complete (eg, step 435 of FIG. 4), but may be performed at other times (eg, periodically). . Process 500 may begin by determining whether the batch rendering engine is waiting for a request for an embedded item (505). For example, if the batch rendering engine requests an embedded item from the fetch server and has not yet received a response from the fetch server, the batch rendering engine is waiting. If the batch rendering engine is waiting (505, yes), the virtual clock is not advanced and the batch rendering engine may work on a task that is ready to run at the current virtual time, if it exists, or It may wait (510). This step may be performed as part of step 415 of FIG. If the batch rendering engine is not waiting for a fetch request (505, No), the batch rendering engine may determine whether there is a task ready for execution in the task list (515). For example, if a task in the task list has a runtime equal to the virtual clock, the task is ready for execution. If the task is ready for execution (515, yes), the batch rendering engine may work on the task (520). Working on a task may add another task to the task list, some of which may be ready to run, others of which may have a future runtime ( For example, current virtual time plus predetermined time). This step may be performed as part of step 415 of FIG. If there are no pending tasks ready for execution (515, no), the batch rendering engine may advance the virtual clock to the next runtime specified in the task list. In other words, the batch rendering engine may distort the virtual clock first so that the next task on the line in the task list can be performed immediately.

  If the next task on the line in the task list is a stop task (530, yes), rendering is complete. Otherwise, the batch rendering engine may continue to work on pending tasks (520). Process 500 shows how the virtual clock does not advance while there is a pending task ready to execute or while waiting for an embedded item fetch. Thus, the class of errors encountered when the virtual clock is “stopped” for these events and the rendering engine uses the actual clock can be avoided. In addition, process 500 shows how the virtual clock can be distorted before, so that in some examples, the rendering process can take the timing indicated by the task (eg, image fades). In real time or wait for a time interval to play an animation). This is especially true when embedded items are returned without crawling, as described in more detail herein. Of course, it is understood that the order of pending task checks (515) and fetch requests (505) can be reversed, and embodiments are not limited to the order shown in FIG.

  FIG. 6 is a flow diagram illustrating an example process in which a fetch server provides embedded item content to a batch rendering engine, according to an embodiment. Process 600 may be performed by a system such as system 100 of FIG. The system may use a process 600 corresponding to fetch resources for embedded items from multiple batch rendering systems. In some embodiments, process 600 may be performed by a fetch server and may be initiated in response to a request from one of the batch rendering engines.

  Process 600 may begin with a fetch server receiving a URL for an embedded item (605). The ULR may be provided by the batch rendering engine and may be one of multiple URLs requested by the batch rendering engine. The fetch server may apply the rewrite rule to the URL of the requested embedded item (610). The rewrite rule may be the URL rewrite rule 154 of FIG. The rewrite rule may include a template and a redirect URL. Applying the rewrite rule may include determining whether the URL matches a pattern or template for one of the rewrite rules. For example, the template may be a URL with any query string removed, and the system may remove the query string from the URL of the embedded item requested to see if it matches the template. . As another example, the template indicates where any character or string can match a wildcard character, * and? Wildcard characters such as

  If the URL matches the pattern, the rewrite rule may provide a redirect URL, and the fetch server may substitute the URL of the requested embedded item with the redirect URL. One reason for applying the rewrite rules is that the fetch server uses redirect URLs to identify URLs and identify URLs that return the same content, so that unnecessary fetches need not be scheduled This is to make it possible. A specific type of generally embedded item has a dynamically generated URL. For example, the URL of some embedded items depends on a random number generated by a random number generator or a current date and time returned by a date and time function. These embedded objects, called cache-busting tracking URLs, are typically used to measure the unique hits or views of a web page for the purpose of measuring advertising costs or revenue. . The content of such an embedded object is usually the same, but a unique URL is generated for that object each time it is detected by the rendering engine. Thus, for a web page containing such an embedded item, the rendering engine sees a new and different URL for the object each time it tries to render the web page, and the fetch server is the same without applying rewrite rules. Fetch content many times. To avoid this, the rewrite rule may apply a template to allow the fetch server to identify these URLs and redirect the fetch request to previously acquired content stored under the redirect URL.

  Another reason for applying the rewrite rule is to identify URLs on the black list. The rewrite rule may include a rule for identifying a URL on a black list, or a pattern or template for a URL on a black list. For example, a rewrite rule may include a template and an associated redirect URL, error, or flag. If the URL for the requested embedded item matches a blacklisted URL or a template for a blacklisted URL, the fetch server identifies the URL as being blacklisted. May be. In some embodiments, applying a rewrite rule may cause a URL to be replaced with a redirect URL. In some embodiments, applying the rewrite rule may flag the URL as blacklisted, or return in response to a request for an embedded item identified by the URL. An error may be provided.

  If the URL is blacklisted (615, yes), the fetch server may return an error to the requesting batch rendering engine (620). The error may be a standard error indicating that the resource was not found, or a specific error that informs the rendering engine that the resource is not needed or can be skipped. The error may be provided from the embedded item table by a matching rewrite rule selected from the embedded item table based on a flag such as a rewrite rule or a hard code when the rewrite rule provides a redirect URL. The fetch request for this URL is then complete and process 600 ends.

  If the URL is not on the black list (615, no), the fetch server may look for the rewritten URL in the embedded item data store (625). The embedded item data store may be the embedded item table 152 of FIG. If the original URL matches the identified pattern of the rewrite rule, the rewritten URL may be a redirect URL provided by the rewrite rule. If the URL does not match any template in the rewrite rule, the rewritten URL may be the original URL. If the URL is in the embedded item data store (625, yes), the fetch server may optionally determine (630) whether the requested URL is for an image. This is optional and in an embodiment that does not test for images, step 630 may be omitted. Whether the requested embedded item is an image may be determined based on request information, the URL itself, or based on a field in the embedded item data store for the rewritten URL. If the embedded item is an image (630, yes), the system looks in the size table for the size of the image and returns a mock image with the size, as described in more detail with respect to the process 700 of FIG. Also good. It is understood that in some embodiments, the fetch server may perform step 630 before applying the rewrite rules after the embedded item data store or after determining whether the entry is stale. .

  If the requested embedded item is not an image (630, No), the fetch server may determine whether the entry in the embedded item table is stale (645). Whether an entry is stale depends on several factors, such as the rate of change of the item, the type of embedded item (eg script, stylesheet, image, etc.), the importance of the web page that the browser rendering engine is rendering, etc. It may depend. In some embodiments, the embedded item table may have a field or value that indicates that the entry will not age, such as for a redirect URL for a blacklisted embedded item URL. If the entry is not stale (645, no), the fetch server may return the contents of the embedded item table for the rewritten URL to the requesting batch rendering engine (650) and process for this embedded item. 600 ends. In some embodiments, returning the content includes the fetch server providing the location of the entry in the embedded item table as a response, and the batch rendering process using the location to access the content. But you can.

  If the entry in the embedded item table is old (645, yes) or if the rewritten URL is not in the embedded item data store (625, no), the fetch server may be a web such as the web patrol engine 130 of FIG. A URL fetch may be requested from the patrol (635). When the fetch server receives crawl content, the received content may be stored as an entry in the embedded item data store (640) without communication or further processing. In some embodiments, the fetch server may save the embedded item content and crawl time without overwriting the previous crawl content and crawl time of the embedded item. In some embodiments, the fetch server may keep one entry in the embedded item table and may not maintain a previous crawl of the embedded item. In any case, once stored in the embedded item table, the content is cached and does not need to be fetched again until it is out of date. The fetch server may return 650 the fetched content to the requesting batch rendering engine, and process 600 ends.

  FIG. 7 is a flow diagram illustrating an example process 700 for a fetch server to provide mock images to a batch rendering engine, according to one embodiment. Process 700 may be performed by a system such as system 100 of FIG. The system may use process 700 to respond to fetch requests for images embedded in web pages from multiple batch rendering engines. In some embodiments, process 700 may be performed by a fetch server and may be initiated in response to a request from one of the batch rendering engines. In some embodiments, the fetch server may perform process 700 independently of other embedded items (eg, process 600 of FIG. 6). In another embodiment, the fetch server may incorporate the elements of process 700 into a process that includes other embedded items, such as process 600 of FIG.

  Process 700 may begin with the fetch server determining whether the requested image has an entry in the size table (705). The image size table may be the image size table 156 of FIG. The image size table contains the size for the image, which is stored by an identifier for the image, such as a URL. If the image is not in the size table (705, no), or if the image is in the size table (705, yes) and old (710, yes), the fetch server may be a web such as the web patrol engine 130 of FIG. An image fetch (715) may be scheduled via the patrol engine. In some embodiments, the fetch server may use the size table information to determine whether an entry is stale. In some embodiments, the fetch server may use information from a separate embedded item table, as described above with respect to step 645 of FIG. 6, to determine whether the size is old. Thus, in some embodiments, the fetch server may perform step 710 in conjunction with or as part of step 645 of FIG. When content for an image is received, the fetch server may add an entry for the image to the size table, where the entry includes the size of the fetched image (720). In some embodiments, the fetch server may store the fetched content in an embedded item table, as described above as part of step 640 of FIG.

  If the image is in the size table (705, yes) and not old (710, no), or after the image has been fetched and stored (720), the system mockes using the size from the size table (725) An image may be generated. The mock image may have image file format data that specifies the empty content that is the same size as the requested image. The system returns the mock image (730) to the request batch rendering engine and process 700 ends.

  It will be appreciated that in some embodiments, some of the steps of process 700 may be optional or performed as part of other processing. For example, the step of determining whether the size of the image is old may be executed as part of step 645 of FIG. 6 or may be based on information in the embedded item table. Further, step 715 may be performed as part of or in conjunction with step 635 of FIG. In other words, the fetch server may combine aspects of the process 600 and process 700, such as fetching content for an image, determining whether cached and fetched content is stale, etc. Of course, the fetch server may execute a process 700 that is completely independent of the process 600. Accordingly, implementations may include variations of process 700.

  FIG. 6 illustrates an example of a general purpose computing device 800 that may operate as the system 100 of FIG. 1 and / or the client 170 and may be used with the techniques described herein. The computing device 800 includes various examples of computing devices such as laptops, desktops, workstations, personal digital assistants, mobile phones, smartphones, tablets, servers, and other computing devices including wearable devices. It is intended to represent. The components shown herein, their connections and relationships, and their functions are meant to be examples only and are intended to illustrate the embodiments of the invention described and / or claimed in this document. It is not meant to be limiting.

  The computing device 800 includes a processor 802, such as a silicone-based hardware processor, a memory 804, a storage device 806, and an expansion port 810 connected via an interface 808. In some embodiments, the computing device 800 includes a transceiver 846, a communication interface 844, and a GPS (Global Positioning System) receiver module 848, among other components connected via an interface 808. obtain. Device 800 may communicate wirelessly through communication interface 844, which may include digital signal processing circuitry, if desired. Each of the components 802, 804, 806, 808, 810, 840, 844, 846, and 848 can be mounted on a general motherboard or in other manners as needed.

  The processor 802 can process instructions for execution in the computing device 800, including instructions stored in the memory 804 or on the storage device 806, on an external input / output device, such as the display 816. Displays graphical information for the GUI. Display 816 may be a monitor or a flat touch screen display. In some embodiments, multiple processors and / or multiple buses may be used with multiple memories and multiple types of memory as required. A plurality of computing devices 800 may also be connected with each device that provides a portion of the required operations (eg, as a server bank, a group of blade servers, or a multiprocessor system).

  Memory 804 stores information within computing device 800. In one embodiment, the memory 804 is a unit or units of volatile memory. In another embodiment, the memory 804 is a unit or units of non-volatile memory. The memory 804 may also be another form of computer readable medium, such as a magnetic or optical disk. In some embodiments, the memory 804 may include expansion memory provided through an expansion interface.

  Storage device 806 can provide mass storage for computing device 800. In one embodiment, the storage device 806 is a computer readable medium such as a floppy disk device, hard disk device, optical disk device, or tape device, flash memory or other similar solid state memory device, or storage area network or It can be, or can include, an array of devices, including devices in other configurations. A computer program product may be explicitly embodied in such computer readable media. A computer program product may also include instructions that, when executed, perform one or more methods, such as those described above. The computer or machine readable medium is a storage device, such as memory 804, storage device 806, or memory on processor 802.

  Interface 808 may be a high speed controller that handles bandwidth consuming operations for computing device 800, or a low speed controller that handles lower bandwidth consuming operations, or a combination of such controllers. An external interface 840 may be provided to allow proximity area communication of the device 800 with other devices. In some embodiments, controller 808 can be coupled to storage device 806 and expansion port 814. Expansion ports that can include various communication ports (eg, USB, Bluetooth®, Ethernet®, wireless Ethernet®) can be connected to one or more input / output devices, eg, through a network adapter, for example. , Keyboards, pointing devices, scanners, or networking devices such as switches or routers.

  The computing device 800 may be implemented in many different forms, as shown in the drawings. For example, it may be implemented multiple times as a standard server 830 or in a group of such servers. It can also be implemented as part of a rack server system. In addition, it may be implemented on a personal computer, such as a laptop computer 832, a desktop computer 834, or a smartphone 836. The entire system may be comprised of multiple computing devices 800 that communicate with each other. Other configurations are possible.

  FIG. 9 illustrates an example of a general purpose computing device 900, which can be the system 100 of FIG. 1 and can be used with the techniques described herein. Computing device 900 is intended to represent various example forms of large data processing devices such as servers, blade servers, data centers, mainframes, and other large computing devices. The computing device 900 may be a distributed system having multiple processors, possibly including network attached storage nodes that are interconnected by one or more communication networks. The components shown herein, their connections and relationships, and their functions are meant to be examples only and limit the embodiments of the invention described or claimed in this document. Does not mean to do.

  The distributed computing system 900 may include any number of computing devices 980. The computing device 980 may include a server or rack server, mainframe, etc. that communicates over a local or wide area network, dedicated optical links, modems, bridges, routers, switches, wired or wireless networks, and the like.

  In some embodiments, each computing device may include multiple racks. For example, the computing device 980a includes a plurality of racks 958a-958n. Each rack may include one or more processors, such as processors 952a-952n and 962a-962n. The processor may include data processors, network attached storage devices, and other computer control devices. In some embodiments, one processor may operate as a master processor and manage scheduling and data distribution tasks. Processors can be interconnected through one or more rack switches 958, and one or more racks can be connected through switches 978. The switch 978 can handle communication between multiple connected computing devices 900.

  Each rack may include memory, such as memory 954 and memory 964, and storage, such as 956 and 966. Storage 956 and 966 may provide mass storage and may be volatile or non-volatile storage, such as network attached disks, floppy disks, hard disks, optical disks, tapes, flash memory or other similar solid state memory devices, or storage area networks. Or an array of devices, etc., including devices in other configurations. Storage 956 or 966 may include a computer-readable medium that stores instructions executable by one or more processors that may be shared among multiple processors, multiple racks, or multiple computing devices. Memories 954 and 964 may be, for example, a volatile memory unit or units, a non-volatile memory unit or units, and / or other forms of computer readable media, such as magnetic or optical disks, flash memory, It may include cache, random access memory (RAM), read only memory (ROM), combinations thereof, and the like. Memory, such as memory 954, may also be shared between processors 952a-952n. A data structure such as an index may be stored across the storage 956 and memory 954, for example. The computing device 700 may include other components not shown, such as a controller, a bus, an input / output device, a communication module, and the like.

  An entire system, such as system 100, may be comprised of multiple computing devices 900 that communicate with each other. For example, device 980a may communicate with devices 980b, 980c, and 980d, which may be collectively known as system 100. As another example, the system 100 of FIG. 1 can include one or more computing devices 900. Some of the computing devices may be located geographically close to each other and others may be located geographically apart. The layout of the computing device 900 is merely an example, and the system can take other layouts or configurations.

  Various embodiments can be combined for receiving data and instructions from a storage system, at least one input device, and at least one output device, and for sending data and instructions to them. Embodiments can be included in one or more computer programs that can be executed or interpreted on a programmable system that includes at least one programmable processor formed on a circuit board, which can be a general purpose.

  These computer programs (also known as programs, software, software applications, or code) include machine language instructions for programmable processors, in high-level procedural and / or object-oriented programming languages, and / or assembly / Can be implemented in machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any non-transitory computer program used to provide machine language instructions and / or data to a programmable processor. Product, apparatus, and / or device (eg, magnetic disk, optical disk, memory (including read-only memory), programmable logic device (PLD)).

  The systems and techniques described herein include a computing system that includes a back-end component (eg, as a data server), or a computing system that includes a middleware component (eg, an application server), or a front-end component (eg, a user). Computing system comprising a graphical user interface or client computer with a web browser that can interact with the implementation of the systems and technologies described herein, or such back-end, middleware, or front-end components Can be implemented in any combination. The components of the system can be connected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), and the Internet.

  The computing system can include clients and servers. A client and server are usually remote from each other and typically interact through a communication network. The client-server relationship arises from the effectiveness of computer programs that run on each computer and have a client-server relationship with each other.

  A number of embodiments have been described. Nevertheless, various modifications can be made without departing from the spirit and scope of the invention. Further, the logical flows shown in the figures need not be in the specific order shown or in sequential order to achieve the desired result. In addition, other steps may be provided, or multiple steps may be omitted from the described flow, and other elements may be added to or removed from the described system. . Accordingly, other embodiments are within the scope of the appended claims.

110 Indexing Engine 115 Index 120 Query Engine 130 Web Tour Engine 140 Rendering Server 142 Rendering Engine 148 Rendering Result 150 Fetch Server 152 Embedded Item Table 154 URL Rewrite Rule 156 Image Size Table 170 Network 180 Client 182 Query 184 Response 190 Server 220 Style Sheet 230 Image file 240 Cache busting URL
305 page task list 310 virtual clock 315 rendering result 316 image 317 DOM tree 318 layout 319 embedded resource 320 error

Claims (18)

Receiving a request to render a web page from a batch process;
Initializing a virtual clock and a task list for rendering the web page using at least one processor, wherein the virtual clock is open when a request for an embedded item is outstanding Initializing, which is still stationary when is ready
Executing the tasks in the task list according to the virtual clock;
Generating a rendering result for the web page using the at least one processor when the virtual clock matches a runtime for a stopped task in the task list;
Providing the rendering results for the batch process.   The method of claim 1, wherein initializing the task list comprises adding the stopped task with a runtime set to a predetermined time added to the virtual clock.   The method of claim 1, wherein the batch process includes an index engine, and the method further comprises using the rendering results to rank index tokens.   Further comprising advancing the virtual clock to the runtime of a task in the task list when a request for an embedded item is not outstanding and there is only one task in the task list having a runtime greater than the virtual clock; The method of claim 1.   The method of claim 1, wherein the batch process is an index engine and the method further comprises demoting a rank for the web page based on the rendering result information.   The method of claim 1, wherein the batch process is an indexing engine and the method further comprises using the rendering results to index dynamically generated content.   In response to the step of executing a task in the task list according to the virtual clock determining that a request for an embedded item is not outstanding and no task has a runtime that matches the virtual clock. The method of claim 1, comprising advancing to a time represented by a next occurring task in the task list. Identifying an embedded item on the web page;
Requesting content for the embedded item;
Receiving the content in response to the request;
The method of claim 1, further comprising: adding a task to the task list to process the content, wherein the added task has a runtime equal to the virtual clock. the method of. Requesting content for the embedded item comprises:
In response to receiving the requested content, adding a task to the task list to process the content, the added task having a start time equal to the virtual clock 9. The method of claim 8 , comprising the step of adding. 9. The method of claim 8 , wherein the actual time spent processing the task list is greater than a predetermined time due to the amount of time waiting for the content.   The method of claim 1, wherein upon determining that all of the tasks with a start time equal to the virtual clock have been completed, the virtual clock advances to a time represented by a next occurring task in the task list. . At least one processor;
A batch rendering engine, the batch rendering engine comprising:
Receiving a request to render a web page from a request process;
Initializing a virtual clock for the web page;
Generating a task list to render the web page, each task in the task list having an associated start time;
Adding a stopped task to the task list;
Performing the task in the task list according to the virtual clock for the web page, wherein the virtual clock for the web page occurs next in the task list for the web page Proceed by setting to the time represented by the task, and the virtual clock will remain unchanged as long as the pending tasks in the task list have a runtime that matches the virtual clock And
Generating a rendering result for the web page when the virtual clock matches a start time for a stopped task in the task list;
Providing the rendering result to the requesting process;
A computer system configured to do the following. The computer system of claim 12 , wherein the stop task has a start time set to a predetermined time. The computer system of claim 13 , wherein an actual time spent processing the task list is less than the predetermined time if an embedded item has not been crawled . The task list includes a first task with a first start time and a second task with a first start time, wherein the first task is an embedded resource fetch, The rendering engine
Requesting the embedded resource from a server;
Tackling the second task while waiting for the request;
The computer system of claim 12 , wherein the computer system is configured to: The rendering engine is
Receiving content for the embedded resource from the server;
Responsive to receiving content for the embedded item, adding a task to the task list, the task having a respective start time set equal to a current value of the virtual clock; Adding
The computer system of claim 15 , further configured to: The computer system of claim 16 , wherein the task added to the task list executes a script identified in content for the embedded item. The computer system of claim 16 , wherein the task added to the task list fetches a second embedded item identified in the content.

Images