WO2013177770A1

WO2013177770A1 - Browser-driven content provisioning for webpages

Info

Publication number: WO2013177770A1
Application number: PCT/CN2012/076305
Authority: WO
Inventors: Yuzhu Shen
Original assignee: Google Inc.
Priority date: 2012-05-30
Filing date: 2012-05-30
Publication date: 2013-12-05

Abstract

Method, computer-readable media, and systems for browser-driven retrieval and presentation of respective set of similar or related images for images within a rendered webpage are disclosed. The browser-driven image retrieval and presentation are independent of invocation of any hyperlinks or instructions embedded within the webpage.

Description

BROWSER-DRIVEN CONTENT PROVISIONING FOR WEBPAGES

BACKGROUND

This specification relates generally to retrieval and presentation of Internet content. The Internet provides access to a vast amount of electronic content (e.g., text, images, blog posts, videos, music files, articles, news items, ads, and so on). A content source is a host of content items that provides the content items to user devices over one or more networks. Examples of a content source include servers of a website, a blog, an online discussion board, an online gaming community, an online marketplace, and other forums where content are provided to users over the Internet or another network.

The content provided by the content sources can be accessible in the form of webpages. The author or publisher of a webpage provides specifications and instructions that describe what content items are to be included in the webpage and the layout and styles by which the content items are to be presented in the webpage, for example. Typically, instructions and specifications are stored or dynamically generated at the servers in one or more documents written in a structured language, such as Hypertext Markup Language (HTML) or Extended Hypertext Markup Language (XHTML). The styles of the webpage can be specified in a Cascade Style Sheet (CSS) file downloaded with the substantive content of the webpage, for example. A client device can request the data for rendering a webpage by a unique Uniform Resource Locator (URL) associated with the webpage, and the server of the webpage provides the data to the client device in response to the request.

Specifically, a web browser application installed on a user device can submit a request (e.g., an HTTP request) to a content source and receive data including content items as well as instructions and specifications on how the content items are to be presented on the user device. The web browser application utilizes the received instructions and

specifications to generate one or more user interfaces (e.g., browser windows) on the user device and present the content items in the user interfaces.

Examples of the perceivable content rendered on the client device include static content such as text, images, audio and video content, as well as interactive elements such as hyperlinks, embedded media players, and interactive user interface elements. Functionalities of the interactive elements can be provided by scripts (e.g., program code written in JavaScript) downloaded from the content source, or by the built-in components of the web browser application. One of the basic functionalities provided by the web browser application is the navigation from a currently displayed webpage to another webpage when a hyperlink in the currently displayed webpage is selected (or "clicked") by the user.

Typically, when a user requests a webpage, for example, by entering the URL of the webpage in the destination address input field of a web browser application or by clicking on a hyperlink in a currently displayed webpage, the web browser requests and receives the data needed to render the requested webpage. Then, the browser application renders the webpage in a browser window according to the data received. The rendered webpage may include text, images, media files, as well as interactive elements such as hyperlinks, buttons, or forms.

Sometimes, the user may see an image in the rendered webpage for which he or she wishes to obtain similar or related images. Similar images are images that are of similar in one or more aspects such as subject matter and composition. Related images are different images but are related in terms of subject matter. There are a few ways for the user to fulfill the desire to see similar or related images of the image(s) currently shown in the webpage.

For example, if the author of the webpage already anticipates the user's needs and has available such similar or related images, the author could provide a hyperlink in the webpage that links to the similar or related images. In such a case, the user can simply click on the hyperlink, and the web browser would navigate to another page showing the similar or related images according to the URL specified in the hyperlink. Unfortunately, most webpages do not provide such a hyperlink for each image in the webpage.

Alternatively, the author of the webpage can embed a script in the webpage, where the script includes instructions that monitor the user's interaction with the webpage, dynamically determine whether the user is interested in a particular image based on the interactions, and dynamically retrieve related or similar images of that particular image from a remote server when certain kinds of interactions are detected. However, in most cases, the authors of webpages do not have the time, resources, intention, and/or skills to predict and fulfill each user's needs to see similar or related images of a particular image in their webpages.

If the publisher or author of the webpage do not already provide a convenient way to access similar or related images of the images in the webpage, the user has to either manually perform an image search using an image search engine, or settle for the lack of such similar or related images. Many users, however, often forego such a search, and thus many users thus leave their interest to see the similar or relate images, for example, higher quality images, unfulfilled. Furthermore, to initiate such as search, a user must leave the web page of interest and navigate to a search engine interface. Accordingly, the content of primary interest to the user is then relegated to another tab or sometimes even lost.

SUMMARY

This specification describes technologies relating to browser-driven content retrieval and presentation for webpages. In an example implementation, when a webpage containing images is rendered in a browser window, the browser application detects user interaction with an image presented in the rendered webpage. If the user interaction is indicative of the user being interested in the image, then, independent of invocation of any hyperlink or instructions embedded in the webpage, the browser application automatically initiates an image search query to an image search engine to find images related to or similar to that image. The browser application presents the retrieved images or a subset thereof

contemporaneously with the webpage in the browser user interface.

In general, one aspect of the subject matter described in this specification can be embodied in a method that includes the actions of rendering a webpage in a browser user interface. The rendered webpage presents a first image among other content. User input registered in proximity to the first image in the browser user interface is detected. In response to detecting the user input and while the rendered webpage is presented in the browser user interface, an image search is initiated at one or more other search engines to retrieve other images related to the first image. The initiation of the image search is independent of invocation of any hyperlink or instruction embedded within the webpage. One or more related images of the first image are obtained as a result of the image search. The one or more related images of the first image are rendered contemporaneously with the webpage in the browser user interface. Other embodiments of this aspect include

corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. These and other embodiments can optionally include one or more of the following features. In some implementations, detecting the user input registered in proximity to the image in the browser user interface can further include detecting hovering of a pointer in proximity to the first image in the browser user interface for at least a threshold amount of time. The initiation of the image search can be independent of any user input registered within the rendered webpage in the browser user interface. It can be verified that at least one of a plurality of result images retrieved from the image search satisfy one or more similarity criteria. Upon the verifying, a user interface element can be displayed in proximity to the first image in the rendered webpage. The rendering of the one or more related images in the browser user interface can be in response to a user input invoking the user interface element. Verifying that at least one of multiple result images retrieved from the image search satisfy one or more similarity criteria can include deriving first topics from text surrounding the first image in the webpage, deriving second topics from text of at least one image search result, and determining that the first and second topics meet a similarity threshold. Before initiating the image search for the first image, it can be determined that the first image is located within a predetermined section of the rendered webpage. Initiating the image search can further include submitting one or more keywords extracted from the webpage along with a copy of the first image to the one or more image search engines. The one or more related images of the first image can include one or more images that are similar to the first image in appearance. The one or more related images of the first image can include one or more images that are related to the first image in terms of subject matter of the images. Rendering the one or more related images contemporaneously with the webpage in the browser user interface can further include rendering the one or more related images in an overlay placed in proximity to the first image within the rendered webpage in the browser user interface.

Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

For example, according to some implementations, the browser application is able to automatically detect a user's interest in an image in a webpage currently presented in the browser window, and automatically initiate an Internet image search at one or more image search engines without requiring the user to take any explicit search actions or navigating away from the current webpage. Therefore, the browser can satisfy the user's information needs and provide a better user experience, rather than letting the user go through the trouble of doing an explicit image search or settle for an unfilled interest in seeing other images similar to or related to the image currently available in the webpage. The manner by which the browser presents the retrieved similar or related images also ensures that the images autonomously retrieved by the browser do not interfere with the user's normal viewing of the webpage.

In addition, the browser detects a user's interest in particular images in a webpage and initiates the image searches on the Internet independent of the invocation of any hyperlinks or programming instructions embedded in the webpage. Therefore, the browser alleviates from the author or publisher of the webpage the burden to fulfill the user's interest in seeing related or similar images to particular images presented in the webpage. Since the publishers or authors of webpages do not always have the resources, time, and/or skills to obtain the related or similar images and to embed hyperlinks or scripts in the webpage for the automatic retrieval of the related or similar images, the techniques described in this specification fulfills the needs not met by the publishers and authors of the webpages.

In addition, many webpages are dynamically generated and it is difficult to predict which images may be included in the webpage before the webpage is rendered in the user's browser. At the same time, it is difficult to predict which image each particular user may find interesting before the webpage is rendered in the user's browser. It is more efficient to fulfill a user's interest in seeing related or similar images of an image present in a webpage by using the browser to determine user interest on images in a currently displayed webpage, and only initiating image search query or presenting retrieved related or similar images when such interest is detected.

The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1A illustrates an example graphical user interface of a web browser.

FIG. IB illustrates an example implementation of the browser-drive content retrieval and presentation techniques described in this specification. FIGS. 1C-1D illustrate another example implementation of the browser-drive content retrieval and presentation techniques described in this specification.

FIGS. IE- IF illustrate an example implementation of the browser-drive content retrieval and presentation techniques described in this specification.

FIG. 2 is a block diagram of an example web browser application implementing the browser-drive content retrieval and presentation techniques described in this specification.

FIG. 3 is a flow diagram of an example process for browser-drive content retrieval and presentation.

Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

In this specification, implementations of browser-driven content retrieval and presentation techniques are described. According to an example implementation of the browser-driven content retrieval and presentation techniques, a web browser optionally detects a user's interest in a particular image in the rendered webpage, and automatically initiates an image search on the Internet (e.g., at a remote image search engine) for similar or related images of the particular image. The web browser then automatically makes the retrieved similar or related images available for the user to see alongside the rendered webpage. The browser's actions are independent of the invocation of any hyperlink or programming instructions embedded in a webpage currently rendered in the browser window.

The browser-driven content retrieval and presentation techniques described in this specification alleviates the burden to perform the manual search from the user, and alleviates the burden to anticipate the user's interests and provide mechanisms to fulfill such an interest in the webpage itself from the authors of webpages. The browser-driven content retrieval and presentation techniques function generically for all images in webpages rendered in the browser regardless of whether hyperlinks or scripts are associated with the images.

In some implementations, the browser distinguishes various zones or regions in a rendered webpage as have different roles or importance. For example, the browser can be configured such that only images within the primary pane of the rendered webpage will trigger the image search and presentation described in this specification. Alternatively, the browser-driven image search and presentation can be performed for all images currently within view on the user's device. The images currently within view can include images on the webpage under input focus and images within pop-up windows that are not obfuscated by other windows, for example.

FIG. 1A illustrates an example graphical user interface 100 of a web browser application installed on a user device. In this example, the graphical user interface 100 is an application window in which one or more webpages can be rendered, e.g., in separate tabs 102a and 102b. In some implementations, the web browser application can open multiple browser windows for rendering webpages. Unless otherwise stated, in this specification, a webpage rendered in a browser window and/or tab of the browser application refers to the webpage that is rendered in the browser window and/or tab that currently has input focus of the user device.

As shown in FIG. 1A, the web browser application has retrieved and rendered a webpage 104 within the tab 102a of the browser window 100. The rendered webpage 104 includes content such as text 106 and images 108a, 108b, and 108c. The webpage 104 also optionally includes a script (e.g., either literally included in the source code of the webpage 104 or referenced and subsequently downloaded with the source code of webpage 104). When the script is executed by the browser application, the executed script optionally detects user interaction with the text or image on the webpage 104, generates a pop-up window 110 alongside the browser window 100, and cause the browser to dynamically download and present an advertisement image 112 in the popup window 110. The webpage 104 also optionally includes hyperlinks 114a, 114b, and 114c, where the hyperlinks 114a and 114b have their respective anchor text in the textual portions 106 of the webpage 104, and the hyperlink 114c has the image 108b as its anchor image. If the user clicks on one of the hyperlinks 114a-c in the webpage 104, the browser will navigate to the webpage associated with the URL of the clicked hyperlink. The browser window 100 also presents other user interface elements for loading a new webpage and navigating from page to page.

In this example, the browser application that provides the user interface 100 in FIG. 1A implements the browser-driven content retrieval and presentation techniques described in this specification. Independent of any instructions embedded in the downloaded and rendered webpages, the browser application is capable of monitoring the user's interaction with a webpage currently rendered in the browser window and determining whether to initiate and carry out additional content retrieval and presentation actions.

For example, the browser application optionally determines which browser window or webpage has the current input focus, and detects the location of the input focus within that window or webpage. The browser also has information on the layout and positions of the content items on the webpage having the current input focus. Therefore, browser can determine whether the user's interactions with the webpage focus on any particular portion or item on the webpage. When the browser application determines according to predetermined criteria that the user's interactions with the webpage are indicative of the user's interest in a particular portion or item on the webpage, the detection can be used as a trigger for subsequent actions taken by the browser application. These actions of the browser application can be initiated and carried out regardless of the hyperlinks or instructions embedded in or downloaded with the webpage, and therefore, the actions are browser-driven.

In this example implementation, the browser application monitors whether the user's input focus (e.g., as represented by the pointer 116) hovers over any of the images presented on the webpage for at least a predetermined amount of time. If so, the browser determines that the user may be interested in seeing similar or related images of that image, and proceeds to initiate a search for the image. The search may include the image and text surrounding the image. As a result of the search, the browser retrieves one or more images that are similar or related to the searched image. The browser application then presents the retrieved images to the user alongside the webpage, either automatically without further input from the user or upon being prompted by the user. In some implementations, the browser evaluates the retrieved images, and makes only a subset of the retrieved images for the user to view. For example, duplicates of the image already present in the webpage are filtered out from the retrieved images before the remaining images are made available to the user.

FIG. IB illustrates an example implementation of the browser-drive content retrieval and presentation techniques described in this specification. As shown in FIG. IB, the browser has detected that the user's pointer 116 has hovered over the image 108a for at least a predetermined threshold amount of time, and initiated an Internet search for the image 108a. The image 108a is not associated with any hyperlink or script, and the Internet search for the image 108a is browser-driven. In some implementations, the browser saves the image or the URL of the image, and initiates a search in the background at one or more Internet image search engines using the saved image or URL. In some implementations, the browser can verify that the images retrieved as a result of the search are indeed similar or related images of the searched image, and only make those images that pass the verification available to the user.

For example, if the browser has been configured to retrieve related images of the image, the browser optionally analyzes the text surrounding the image to extract one or more keywords describing the subject matter of the image. Then, the browser optionally sends the one or more keywords with the image or URL thereof when performing the search and requires that the search results be related to those keywords. In some implementations, the browser performs its own analysis of the retrieved images and verifies that the retrieved images are indeed related to the image shown in the webpage in terms of subject matter, and are not duplicates of that image at different resolution or qualities. Various techniques exist in the art for verifying the subject matter of an image and determining whether two images are different versions of the same image.

For another example, if the browser has been configured to retrieve images that are similar to an image in the webpage, before making the retrieved images to the user, the browser optionally verifies that the images retrieved from the Internet search are similar in terms of composition, or subject matter, or both, to the searched image. In some

implementations, the browser analyzes the composition of the image and each of the retrieved images, and filters out images that do not have a similar composition as the searched image. To determine if two images are similar in terms of subject matter, the browser optionally analyzes the text surrounding the image to extract one or more keywords describing the subject matter of the image. Then, the browser optionally sends the one or more keywords with the image when performing the search and requires that the search results be related to those keywords. In some implementations, the browser can perform its own analysis of the retrieved images and the text included with the search results to verify that the retrieved images are indeed similar to the image shown in the webpage in terms of subject matter and/or composition, and are not duplicates of that image at different resolution or qualities. After the browser application has successfully identified images that are similar or related to the searched image, the browser application renders the identified images alongside the image that was the subject of the image search. In the example shown in FIG. IB, related images of image 108a are retrieved and the rendering of the retrieved images is automatic and without further prompt from the user. The techniques described with respect to FIG. IB for presenting the related images are also applicable to presenting similar images of the searched image.

As shown in FIG. IB, a user interface element 118 (e.g., a pop-up window) is placed in proximity to the rendered webpage 104, for example, directly below the rendered webpage 104. The user interface element 118 presents multiple images (e.g., images 120a-c) that have been retrieved as a result of an Internet image search for the image 108a. In some implementations, the images presented in the user interface element 118 have been verified (e.g., by the image search engine or the browser) to be related to the image 108a in terms of subject matter. In some implementations, the related images can also be presented in a pop- up window or overlay positioned within the browser window 104, such as in proximity to the image 108a. The user interface element 118 presenting the related images optionally includes one or more user interface elements for manipulating each of the related images, such as controls for saving the related image on the user device, zoom into the related image, or closing the pop-up window 118. In some implementations, no all related images that have been retrieved are immediately visible in the pop-up window 118, and the user may scroll to see additional related images by selecting the browsing control 122a or 122b.

In some implementations, if the pop-up window or overlay 118 is placed in proximity to the image 108a, the pop-up window or overlay 118 can be automatically closed when the user's input focus (e.g., as indicated by the pointer 116) moves away from the image 108a and/or the window 118. If the retrieved image is presented in a separate pop-window, the user can close the pop-up window by selecting the appropriate user interface element available in the pop-up window. After the pop-up window or overlay 118 showing the related images is closed, the user interface 100 returns to its original appearance as shown in FIG. 1A.

If the user subsequently moves the pointer 116 to another image (e.g., image 108b) shown in the webpage, and hovers the pointer 116 over the image for an extended period of time, the browser application will perform the image search using that image (e.g., image 108b). If the search returns related images of the image 108b, the browser application presents the retrieved images in the same manner as described with respect to image 108a. If no related images of the image 108b are found through the automatic search and verification process, the user interface is not altered.

In some implementations, the browser application may present, in addition to an option for the user to save the related or similar images that have been retrieved, an option to navigate to each of the source webpages of the images. For example, the pop-up window 118 showing the related or similar images can provide an options menu or menu bar showing these various options. Since the related or similar images may come from one or more content sources different from that providing the webpage 104 currently rendered in the browser window 100, by selecting the option to go to the source of a related or similar image, the browser may cause the webpage 104 to be replaced with a different webpage originally containing the related or similar image.

In some implementations, the related or similar images presented in the user interface element 118 may be scaled such that the images are shown to have substantially the same size on the display. Each image presented in the user interface element 118 can be expanded to its original scale once the user has clicked on it.

FIG. IB shows one example manner in which image search for related and/or similar images can be triggered and retrieved images can be presented on the user device. FIGS. 1C- 1D illustrate another implementation in which the retrieved images are not presented immediately after they are received by the browser application. Instead, a user interface element indicating the availability of such related or similar images is presented alongside the webpage currently rendered in the browser window, and the retrieved images are only rendered in the browser user interface when the user interface element is invoked by user input.

In this example, suppose that the browser application has been configured to retrieve and show images that are similar to an image currently rendering in the browser window. As shown in FIG. 1C, as soon as the browser application determines that the user may be interested in seeing related images of the image 108b, for example, based on the pointer 116 having hovered over the image 108b for more than a threshold amount of time, the browser application proceeds to perform an image search on the Internet for the image 108b. As soon as the browser obtains the result images from the search, and identifies from the result images one or more images that are similar to the searched image 108b in terms of subject matter and composition, the browser presents a user interface element 124 next to the image 108b on the webpage 104. The user interface element 124 indicates that the web browser has retrieved images that are similar to the image 108b from the Internet. If the user wishes to see those images similar to the image 108b, the user can simply invoke the user interface element 124 by clicking on it using the pointer 116, for example.

FIG. ID shows that once the user has invoked the user interface element 124, the retrieved similar images (e.g., images 128a-c) are presented in a pop-up window or overlay 126 displayed in the browser user interface (e.g., in proximity to the image 108b in the rendered webpage 104 in the browser window 100). In some implementations, the pop-up window 126 can also be presented outside of the browser window 100, such as below the browser window 100.

In some implementations, only a subset of all eligible images are immediately visible in the pop-up window 126, and the user can scroll to see more of the images by clicking on the controls 130a and 130b, for example. In some implementations, the user can close the pop-up window or overlay 126 by either moving the pointer 116 away from the image 108b and pop-up window 126 or manually closing the pop-up window or overlay 126, for example. By providing the user interface element 124 rather than providing the retrieved images immediately on the user interface, the browser allows the user more control over the browser interface, and avoid unnecessary cluttering of the browser window. In addition, the user can always go back to the user interface element 124 to cause the retrieved images to be displayed again without causing the browser to perform the search again.

In some implementations, the browser does not have to wait for the user to hover the pointer over a particular image in order to initiate the image search on the Internet. Instead, the search can begin as long as there is bandwidth and computing resources available for the background process to initiate the image searches. For example, in some implementations, the browser application initiates the image search for each image or a subset of the images (e.g., images within the primary pane of the webpage, or images that have been determined not to be a graphical icon) on the rendered webpage as soon as the webpage is loaded and rendered on the user device. Then, for each image for which the browser has received one or more similar or related images, the browser presents a corresponding user interface element in proximity to the image. If no similar or related image is received for a particular image, no such user interface element is presented in proximity to the particular image.

As shown in FIG. IE, without regard to the user' prior interactions with the webpage 102, the browser has retrieved at least one similar or related image for each of several images presented on the webpage 104. For those images (e.g., images 108a and 108b) that have similar or related counterparts, the browser application has rendered corresponding user interface elements (e.g., element 132a, 132b, and 132c) in proximity to the images. For the images (e.g., image 112) that no similar or related images have been found, the images are left as they were originally rendering the webpage 104. The user can select any one of the user interface elements 132a-c to see the corresponding group of similar or related images presented in the browser user interface.

As shown in FIG. IF, the user has selected the user interface element 132b displayed in proximity to the image 108b rendered in the webpage 104, and the similar or related images of the image 108b is presented in a pop-up window or overlay 134 in proximity to the image 108b as a result.

FIGS. 1 A- IF are merely illustrations of example user interfaces of a browser application that implements the browser-driven content retrieval and presentation techniques described in this specification. Other variations of the manner by which browser-driven content retrieval and presentation functions can be implemented are possible.

FIG. 2 is a block diagram of an environment 200 in which an example browser application 202 that implements the browser-driven content retrieval and presentation techniques described in this specification operates. In the example environment 200, the browser application 202 is installed on a user device 204 that communicates with servers of content sources 206 and image search engines 208 through one or more networks 210.

Examples of the networks include combinations of one or more local area networks (LAN), wide area networks (WAN), peer-to-peer networks, wireless networks, and/or other equivalent communication networks. The user device 204 can be data processing apparatus such as personal computers, smart phones, tablet computers, and so on. The browser application 202 is a software application that download data 212 (e.g., source code and content items of webpages) for generating content interfaces (e.g., the user interface 102a shown in FIG. 1A) from content sources 206, and render the content interfaces on the user device 204 according to the data 212. The browser application 202 is further capable of executing scripts downloaded from the content sources 206, for example, as part of the data 212, and performing operations according to the instructions specified in the scripts.

As shown in FIG. 2, the browser application 202 installed on a client device 204 is capable of communicating with one or more content sources 206 (e.g., servers of one or more websites) through a network component 214. For example, the network component 214 is capable of sending a request for a webpage (e.g., an HTTP request specifying the URL of the webpage) to a server of the webpage (or one of the content sources 206), and receiving the data 212 needed for rendering the webpage on the client device 204. The data 212 received from the content source 206 include the source code of the webpage (e.g., in one or more HTML or XHTML documents) as well as content items that are to be presented on the webpage and/or scripts to be embedded in the webpage.

Once the network component 214 receives the data needed to render the webpage on the client device 204, a rendering component 216 of the browser application 202 interprets the data and determines the layout and styles by which the content items are to be rendered in the webpage. The rendering component 216 renders the webpage in a browser window of the browser application 202 according to the layout and styles specified in the received data or the default styles set in the browser's configuration settings.

Once the webpage is rendered in the browser user interface (e.g., a browser window), a user interface component 218 of the browser application 202 can monitor the browser user interface, and detect any user interactions with the browser user interface and the webpage rendered inside the browser user interface. For example, the UI component is able to determine the location of the pointer inside the browser window, the particular content item or text located at that location, and any user input (e.g., clicks, hovering, etc.) registered at that location. For example, if the user selects a hyperlink inside the webpage, the user interface component 218 of the browser application 202 will detect the user input at a location on the rendered webpage, determine that there is a hyperlink located at that location, and navigate to the webpage identified by the URL associated with the selected hyperlink. The user interface component 218 can communicate with the network component 214 to carry out the actions initiated by the user input registered within the browser user interface by the user interface component 218.

In some implementations, if the webpage contains embedded scripts, the browser application 202 can execute the script, which may utilize the various components of the browser application 202 through the various APIs provided by the browser application 202. In some implementations, the browser application 202 also provides APIs for browser extensions and plug-in software components to extend the default functions of the browser application 202 by utilizing the components 214, 216, and 218 of the browser application 202, for example.

In this example, as indicated in FIG. 2, the browser application 202 also includes a content-retrieval component 220 that initiates Internet image searches for images on a webpage currently rendered in the browser window. The content-retrieval component 220 is capable of initiating Internet image searches for images presented in a webpage currently rendered in the browser independent of any instructions or hyperlinks embedded in the webpage. The content-retrieval component 220 communicates with the user interface component 218 and instructs the user interface component 218 to monitor the user's interactions with the webpage currently presented in the browser window. The content- retrieval component 220 then determines whether the user's interactions (e.g., as reported by the user interface component 218) indicate the user's interest in any particular image in the webpage currently rendered in the browser window. The content-retrieval component 220 can include one or more criteria for evaluating the user's interest based on parameters (e.g., proximity to image, input type, input duration, and so on) received from the user interface component 218, for example.

In some implementations, if the content-retrieval component 220 determines that the user's interaction with the webpage indicates the user is interested in a particular image in the rendered webpage, the content-retrieval component 220 initiates an image search for the particular image via the network component 214 of the browser application 202. For example, the content-retrieval component 220 can contact one or more Internet image search engines and perform the image search using a copy of the image or the URL of the image. The search results received by the browser application 202 may include images that are identical to the search image and images that are similar or related to the searched image in one or more aspects.

In some implementations, the content retrieval component 220 specifies one or more criteria for evaluating the search results. For example, if the browser has been configured to retrieve and present only images that are related to a searched image, then the browser will include component for determining the respective subject matter of the searched image and each image retrieved from the search, and determines whether the retrieved image relates to the searched image in terms of subject matter. For example, topics derived from text surrounding the image in the webpage may be compared to topics derived from text included with the image search result. If the topics match, or are determined to meet a similarity threshold, then the referenced image may be determined to be related to the image in the web page. Likewise, image feature analysis can be performed on the image of the webpage and the image of the search result, and if the image of the search result meets a similarity threshold that measures the similarity of the image to the image of the web page, then the referenced image may be determined to be related to the image in the web page.

In some implementations, if the browser has been configured to retrieve and present only images that are similar to a searched image, then the content retrieval component 220 will include sub-component(s) for determining the respective subject matter as well as compositions of the searched image and each image retrieved from the search, and determines whether the retrieved image is similar to the searched image in terms of both subject matter and composition. In addition, the content-retrieval component 220 also includes sub-components that filter out result images that are identical to the search image or are different from the searched image only in terms of quality or resolution. In some implementations, a result images that is a higher quality /resolution version of the searched image is considered by the content-retrieval component 220 as a similar image of the searched image.

In some implementations, image search engines provide search results in a search result page with a thumbnail and a hyperlink for each result image. In such cases, the content-retrieval component 220 follows each of the hyperlinks to retrieve the result image and determine whether the result image satisfies the above-mentioned criteria. After the retrieved result images are evaluated by the content-retrieval component 220, for each image that one or more similar or related images have been retrieved, the content-retrieval component 220 utilizes the rendering component 216 to render the similar or related images on the user device 204 alongside the webpage. Examples of how the retrieved images are presented on the user device are described with respect to FIGS. IB- IF.

FIG. 2 is merely an illustration of the functionalities of the browser application 202. The browser application 202 may have more or fewer components than those shown, and the functionalities of the components may be combined into fewer components or divided into more components. In some implementations, the content-retrieval component 220 of the browser application 202 may be a native component of the browser application program. Alternatively, the content-retrieval component may be implemented as a browser extension or a plug-in component that interfaces with the main browser application through the extension or plug-in API of the browser application. The content-retrieval component 220 performs its functions independent of the instructions downloaded with the webpages.

Optionally, the content-retrieval component 220 performs the same functions for all webpages currently displayed in the browser, even if the webpage is a locally stored webpage (e.g., a saved webpage).

FIG. 3 is a flow diagram of an example process 300 implemented in a browser application (e.g., browser application 202 in FIG. 2) executing on a client device.

In the example process 300, the browser application renders a webpage in a browser user interface, the rendered webpage presenting a first image among other content (302). The browser application detects user input registered in proximity to the first copy of the image in the browser user interface (304). In response to detecting the user input and while the rendered webpage is presented in the browser user interface, the browser application initiates an image search at one or more image search engines to retrieve other images related to the first image, where the initiation of the image search is independent of invocation of any hyperlink or instruction embedded within the webpage (306). Then, the browser application obtains one or more related images of the first image as a result of the image search (308). Then, the browser renders the one or more related images of the first image contemporaneously with the webpage in the browser user interface (310). In some implementations, detecting the user input registered in proximity to the image in the browser user interface further includes detecting hovering of a pointer in proximity to the first image in the browser user interface for at least a threshold amount of time.

In some implementations, the initiation of the image search is independent any user input registered within the rendered webpage in the browser user interface. For example, the browser application can autonomously initiate image searches for images within the rendered webpage without regard to whether the user has interacted with the webpage prior to the initiation of the image searches. In such implementations, the presentation of the retrieved images is optionally triggered based on whether the user has expressed interest in seeing the retrieved images for a particular image rendered in the webpage. The user's interest can be indicated by the user selecting a user interface element indicating availability of the retrieved images, for example.

In some implementations, the browser application verifies that at least one of a plurality of result images retrieved from the image search satisfy one or more similarity criteria, and provides a user interface element indicating availability of the similar images in proximity to the first image only upon such verification. In some implementations, upon the verification, the browser application displays a user interface element in proximity to the first image in the rendered webpage, wherein the rendering of the one or more related images in the browser user interface is in response to a user input invoking the user interface element.

In some implementations, before initiating the image search for the first image, the browser application determines that the first image is located within a predetermined section of the rendered webpage. For example, in some implementations, the browser application will only initiate image searches for images located within the primary pane of the rendered webpage, and not in less important sections of the page, for example, footer, sidebar, or the like. In addition, the browser application can also determine whether a particular image rendered in the webpage is an icon image, and if so, no image search is initiated for that particular image.

In some implementations, initiating the image search at the one or more image search engines further includes submitting a copy of the first image to the image search engines. In some implementations, initiating the image search at the one or more image search engines further includes submitting a Uniform Resource Locator (URL) of the first image to the one or more image search engines. In some implementations, initiating the image search further includes submitting one or more keywords extracted from the webpage along with a copy of the first image to the one or more image search engines.

In some implementations, the one or more related images of the first image include one or more images that are similar to the first image in appearance. In some

implementations, the one or more related images of the first image include one or more images that are related to the first image in terms of subject matter of the images.

In some implementations, rendering the one or more related images

contemporaneously with the webpage in the browser user interface further rendering the one or more related images in an overlay placed in proximity to the first image within the rendered webpage in the browser interface. In some implementations, the related images are rendered in a designated area within or outside of the browser window, such as immediately below the browser window.

The example process shown in FIG. 3 is provided for illustrative purposes. Other processes are described with respect to FIGS. 1A-2, for example. In addition, most of the examples are provided for retrieval and presentation of images, however similar techniques can be implemented to retrieve and present other types of content, such as video and audio clips. For example, the browser can initiate Internet searches and retrieval of similar or related video clips for a short video clip embedded in a webpage currently rendered in the browser window, independent of invocation of any hyperlink or instructions embedded in the webpage. The retrieved content can be presented to the user in the manner similar to that described in this specification. The initiation of the content retrieval can be automatic or upon determination that the user may be interested in seeing such similar or related content of the content shown in the webpage. In some implementations, the browser does not impose a limitation that the similar or related content retrieved and presented for a content item on the webpage has to be of the same type of the content item. For example, for an image rendered in a webpage, the browser optionally presents a video clip that includes a frame showing the image.

Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus.

Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures. A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices;

magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network ("LAN") and a wide area network ("WAN"), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer- to-peer networks).

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate

embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

What is claimed is:

Claims

1. A computer-implemented method, the method being performed by one or more processors and comprising:

rendering a webpage in a browser user interface, the rendered webpage presenting a first image among other content;

detecting user input registered in proximity to the first image in the browser user interface;

in response to detecting the user input and while the rendered webpage is presented in the browser user interface, initiating an image search at one or more image search engines to retrieve other images related to the first image, the initiation of the image search being independent of invocation of any hyperlink or instruction embedded within the webpage; obtaining one or more related images of the first image as a result of the image search; and

rendering the one or more related images of the first image contemporaneously with the webpage in the browser user interface.

2. The method of claim 1 , wherein detecting the user input registered in proximity to the image in the browser user interface further comprises:

detecting hovering of a pointer in proximity to the first image in the browser user interface for at least a threshold amount of time.

3. The method of claim 1, wherein the initiation of the image search is independent any user input registered within the rendered webpage in the browser user interface.

4. The method of claim 1, further comprising:

verifying that at least one of a plurality of result images retrieved from the image search satisfy one or more similarity criteria; and

upon the verifying, displaying a user interface element in proximity to the first image in the rendered webpage, wherein the rendering of the one or more related images in the browser user interface is in response to a user input invoking the user interface element.

5. The method of claim 4, wherein verifying that at least one of a plurality of result images retrieved from the image search satisfy one or more similarity criteria comprises:

deriving first topics from text surrounding the first image in the webpage;

deriving second topics from text of at least one image search result; and

determining that the first and second topics meet a similarity threshold.

6. The method of claim 1, further comprising:

before initiating the image search for the first image, determining that the first image is located within a predetermined section of the rendered webpage.

7. The method of claim 1, wherein initiating the image search further comprises:

submitting one or more keywords extracted from the webpage along with a copy of the first image to the one or more image search engines.

8. The method of claim 1, wherein the one or more related images of the first image include one or more images that are similar to the first image in appearance.

9. The method of claim 1, wherein the one or more related images of the first image include one or more images that are related to the first image in terms of subject matter of the images.

10. The method of claim 1, wherein rendering the one or more related images

contemporaneously with the webpage in the browser user interface further comprises:

rendering the one or more related images in an overlay placed in proximity to the first image within the rendered webpage in the browser interface.

11. A computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, cause the processors to perform operations comprising:

rendering a webpage in a browser user interface, the rendered webpage presenting a first image among other content; detecting user input registered in proximity to the first image in the browser user interface;

12. The medium of claim 11 , wherein detecting the user input registered in proximity to the image in the browser user interface further comprises:

13. The medium of claim 11 , wherein the initiation of the image search is independent any user input registered within the rendered webpage in the browser user interface.

14. The medium of claim 11, the operations further comprising:

15. The medium of claim 14, wherein verifying that at least one of a plurality of result images retrieved from the image search satisfy one or more similarity criteria comprises: deriving first topics from text surrounding the first image in the webpage;

deriving second topics from text of at least one image search result; and

determining that the first and second topics meet a similarity threshold.

16. The medium of claim 11, the operations further comprising:

17. The medium of claim 11 , wherein initiating the image search further comprises: submitting one or more keywords extracted from the webpage along with a copy of the first image to the one or more image search engines.

18. The medium of claim 11, wherein the one or more related images of the first image include one or more images that are similar to the first image in appearance.

19. The medium of claim 11 , wherein the one or more related images of the first image include one or more images that are related to the first image in terms of subject matter of the images.

20. The medium of claim 11, wherein rendering the one or more related images contemporaneously with the webpage in the browser user interface further comprises:

21. A system, comprising:

one or more processors; and

memory having instructions stored thereon, the instructions, when executed by one or more processors, cause the processors to perform operations comprising:

in response to detecting the user input and while the rendered webpage is presented in the browser user interface, initiating an image search at one or more image search engines to retrieve other images related to the first image, the initiation of the image search being independent of invocation of any hyperlink or instruction embedded within the webpage;

obtaining one or more related images of the first image as a result of the image search; and

rendering the one or more related images of the first image

contemporaneously with the webpage in the browser user interface.

22. The system of claim 21 , wherein detecting the user input registered in proximity to the image in the browser user interface further comprises:

23. The system of claim 21, wherein the initiation of the image search is independent any user input registered within the rendered webpage in the browser user interface.

24. The system of claim 21, the operations further comprising:

25. The system of claim 24, wherein verifying that at least one of a plurality of result images retrieved from the image search satisfy one or more similarity criteria comprises: deriving first topics from text surrounding the first image in the webpage;

deriving second topics from text of at least one image search result; and

determining that the first and second topics meet a similarity threshold.

26. The system of claim 21, the operations further comprising: before initiating the image search for the first image, determining that the first image is located within a predetermined section of the rendered webpage.

27. The system of claim 21, wherein initiating the image search further comprises:

28. The system of claim 21, wherein the one or more related images of the first image include one or more images that are similar to the first image in appearance.

29. The system of claim 21, wherein the one or more related images of the first image include one or more images that are related to the first image in terms of subject matter of the images.

30. The system of claim 21, wherein rendering the one or more related images contemporaneously with the webpage in the browser user interface further comprises: