JP2008535098A - System and method for transferring web page data - Google Patents

Abstract

  A system and method that can include accessing an Internet site by a proxy server, converting image data from the Internet site into a multi-resolution representation, and sending the image data of the multi-resolution representation to a client device. Disclose.

Description

  It has long been recognized that it is useful to browse the web with small devices such as personal digital assistants (PDAs), mobile phones, or wireless terminals. The problems associated with implementing web browsing on such devices generally do not arise due to limitations in computing power. Rather, such problems arise from the following two causes. First, the display is usually very small, making it difficult to display web pages designed specifically for larger displays or windows (eg, usually at least 800 × 200 pixels or 1024 × 768 pixels). . Second, small wireless devices are usually connected to wide area networks with low bandwidth, and traditional web browsing is due to the need to download the entire HTML (Hypertext Markup Language) page before viewing. Become uncomfortable.

  There are several partial solutions to this problem. For example, a computer hardware manufacturer has designed a new platform for wireless distribution of content to small devices based on the BREW (Binary Runtime Environment for Wireless). However, this type of solution is that content providers agree to supply their content using BREW technology, or that third-party content aggregators use specialized formatting for the content or application Existing content needs to be continuously repackaged and again using BREW technology to deliver the results.

  Another partial solution is Opera For Mobile in combination with Opera Mobile Accelerator and Small-Screen Rendering ™. These technologies are based on Opera, a popular web browser that has been ported to mobile wireless platforms. Small-Screen Rendering ™ attempts to dynamically reformat web pages to fit within the horizontal dimensions of the mobile terminal display. The Opera Mobile Accelerator uses a proxy server to access the web page in the original HTML format on demand, and then compresses the HTML and transfers the compressed version to the mobile terminal. According to Opera, the Opera Mobile Accelerator compresses the size of the web page by about 50% to about 70%, but has the ability to reduce the size of the web page to 90%. Considering this, data is retrieved faster and web pages are displayed more quickly. In general, the speed increase depends on the type of site the user is trying to visit. These two technologies present a partial solution to the mobile terminal display size and bandwidth issues related to web browsing.

  However, there is a need in the prior art for improved systems and methods for supplying web page data to digital devices with limited bandwidth or other communication and / or processing power limitations.

  According to one aspect, the present invention accesses an Internet site by a proxy server, converts image data from the Internet site into a multi-resolution representation, and transmits the image data of the multi-resolution representation to a client device. A method that can include: Preferably, the method further includes navigating the web page by the client device. Preferably, the step of navigating comprises: (a) navigating the image data of the internet site by a proxy server; and (b) navigating the multi-resolution image data by the client device. Navigating) and navigating in step (b) include occurring substantially simultaneously.

  Preferably, navigating includes storing at least substantially the entire multi-resolution image data of the web page at the proxy server and navigating the stored multi-resolution image data by the client device. Preferably, the converting step includes continuously converting internet site image data that needs to be navigated by the client device. Preferably, the step of converting is pre-converted with the proxy server pre-converting the selected image data by the proxy server before the selected image data from the internet site is needed for client device navigation. Storing stored image data on a proxy server.

  Preferably, the method further includes transmitting the stored image data to the client device as needed by client device navigation. Preferably, the method further comprises ensuring that the stored pre-converted image data is not stale before sending the stored pre-converted image data to the client device. Preferably, the guaranteeing step compares one of the time stamp and checksum of the pre-converted image data with a respective one of the corresponding image data time stamp and checksum originating from the internet site. Including doing. Preferably, the pre-converting step includes converting the image data of at least one web page by a proxy server and storing the converted image data of at least one web page at the proxy server.

  Preferably, the method further includes enabling two-way interactive communication between the client device and the Internet site. Preferably, the interactive communication includes sending a navigation command from the client device to the proxy server. Preferably, the method further comprises emulating dynamic internet graphics for the client device by a proxy server. Preferably, the step of emulating includes emulating at least one of dynamic HTML (Hypertext Markup Language) and an applet. Preferably, navigating includes selecting at the client device a resolution level for viewing at least a portion of the web page.

  Preferably, navigating includes selecting an area of the web page for display on the client device. Preferably, navigating further includes selecting a resolution for viewing an area of the web page. Preferably, the step of navigating includes selecting a plurality of portions of the web page viewed on the client device. Preferably, the step of navigating further comprises selecting a plurality of respective resolutions for viewing the plurality of web page portions. Preferably, navigating the multi-resolution image data includes panning the multi-resolution image data. Preferably, navigating the multi-resolution image data includes zooming the multi-resolution image data.

  According to another aspect, the present invention is a proxy server that communicates with a client device, the client device and an Internet site, and acts as an intermediate between the client device and the Internet site, the image data from the Internet site. Can be provided that can include a proxy server that is operable to convert the video to a multi-resolution visual data format. Preferably, the multi-resolution visual data format is JPEG2000. Preferably, the client device is a mobile phone, and the mobile phone includes at least one mechanism for navigating image data in a multi-resolution visual data format. Preferably, the mechanism is a touchpad that allows panning of image data in a multi-resolution visual data format. Preferably, the mechanism allows zooming in on the image data within the multi-resolution visual data format. Preferably, the zooming mechanism includes a roller that is rotatable by a user of the client device. Preferably, the client device is one of the group consisting of a mobile phone, a PDA (Personal Digital Assistant), a notebook computer, a desktop computer, a tablet computer, and a television set.

  Other aspects, features, benefits, etc. will become apparent to those skilled in the art when the description of the preferred embodiments of the invention herein is taken in conjunction with the accompanying drawings.

  While the presently preferred form is shown in the drawings for purposes of illustrating various aspects of the invention, it is to be understood that the invention is not limited to the precise arrangements and instrumentalities shown.

  FIG. 1 is a block diagram illustrating a communication system 140 that includes a proxy server 150 and a client device 230 in accordance with one or more embodiments of the present invention. The communication system 140 can include the Internet 180, the client device 230, and the proxy server 150, which can include a cache 160. Internet 180 can be interpreted to include the communication links, computing hardware, and communication hardware necessary to enable proxy server 150 to browse the Internet in a conventional manner. Although a high bandwidth connection is preferred for use between the proxy server 150 and the Internet 180, the proxy server 150 can be connected to the Internet 180 using any communication link known in the art. In general, proxy server 150 may be connected to an Internet service provider (ISP) (not shown) that acts as a gateway to the Internet as a whole. In one or more embodiments herein, the Internet 180 is intended to include such Internet service providers.

  Proxy server 150 may be a personal computer or other computing device suitably configured to communicate with the Internet and process data including image data downloaded from the Internet. The proxy server 150 is preferably capable of sending commands to the Internet 180 and receiving image data from the Internet 180.

  Proxy server 150 may include a cache 160 that stores such data to allow quick and easy access to data including image data. The cache 160 can be physically incorporated within the server 150, such as by being in the same physical enclosure as the proxy server 150. Alternatively, the cache 160 may be stored separately from the proxy server 150, but the cache 160 may be accessible.

  Client device 230 is preferably a portable computing device capable of communicating with proxy server 150. However, the client device can be a larger non-portable computing device, such as a desktop computer. Client device 230 may be one of a mobile phone, a personal digital assistant (PDA), a notebook computer, a desktop computer, a tablet computer, and a television set. However, the client device 230 is not limited to being one of the aforementioned items. Preferably, when the client device 230 is a television set, the television set can receive multi-resolution image data and send navigation commands to an external device such as but not limited to the proxy server 150. it can. The client device 230 may also include a data storage function that includes multi-resolution image data received from the proxy server 150, including volatile storage and / or non-volatile storage. Such data storage can include, but is not limited to, RAM (random access memory, ROM (read only memory), hard drive, CD-ROM drive, CD read-write, and bubble memory. The client device 230 can include a cache memory in the data storage.

  Proxy server 150, as described in more detail with respect to FIG. 2, after converting image data from the Internet 180 to a multi-resolution format, allows the client device 230 to browse such data. It can act as an intermediate between 180 and client device 230.

  FIG. 2A is a diagram illustrating the front of a client device 230, which can be a mobile phone, customized according to one or more embodiments of the present invention. FIG. 2B is a front view of the back side of the client device 230 of FIG. 2A. 2C is a front view of the back side of the client device of FIG. 2A.

  In one or more embodiments included in the diagram of FIG. 2, client device 230 is a mobile phone or PDA specifically configured to browse multi-resolution image data. Client device 230 may include a main body 232, a display 234, an antenna 236, a touch pad 238, a joystick 240, and / or a roller 242. In FIG. 2, the roller 242 is shown protruding from the body 232 of the client device 230 to a certain extent. This protrusion is shown for illustration. The roller or wheel 242 need only be accessible from the user of the client device 230. Thus, the roller or wheel 242 can protrude slightly from the body 232. Alternatively, a recess (not shown) can be provided on the side of the body 232, and the roller 242 can be contained in the recess so that the wheel 242 is a straight edge on the side of the body 232 of the client device 230. It becomes possible not to protrude beyond the part which has.

  In one or more embodiments, the display 234 can be positioned in front of the client device 230 (FIG. 2A). However, in other embodiments, the display 234 can be located anywhere on the client device 230. The touchpad 238 can be located on the back of the client device 230 (FIG. 2B) and can be used to pan image data received from the proxy server 150. However, alternatively, the touchpad 238 can be placed on any surface of the client device 230. Alternatively, the touch pad 238 can be used for zooming the image data. In one or more other embodiments, the touchpad 238 can be used for panning and zooming image data.

  A user of the client device 230 can pan or zoom the image data by moving a finger along the surface of the touchpad 238. Additionally or alternatively, touchpad 238 can be used to move the cursor over one or more image areas or web page areas displayed on display 234. When multiple images (possibly but not necessarily from two different web pages) are displayed on display 234, the use of a cursor is useful for identifying one of the images for performing a navigation step. It may be.

  In an alternative embodiment, joystick 240 (FIG. 2C) can be used to pan image data either in addition to or instead of touchpad 238. The joystick can be used to pan the image data by pressing the joystick 240 in a direction corresponding to the direction that the user of the client device 230 wants to pan across the image. Similar to touch pad 238, joystick 240 may be placed on any surface of client device 230.

  In one or more embodiments, zooming is performed by using a separate device, such as a wheel 242 or other device, such as a slider (not shown), or up and down at one edge of the touchpad 238 surface. This can be accomplished by using routine gestures on the touchpad 238, such as sliding the screen. In the latter case, the index finger of both hands can be used simultaneously on the touchpad (one hand pans, the other hand zooms).

  In one or more embodiments that include both the touchpad 240 and the wheel 242, the user of the client device 230 can switch back and forth between zoom and pan, or pan and zoom at the same time. You can even do it. In this specification, the wheel 242 is also referred to as a “roller”.

  By using one or more of the features described above, a tactile experience as if the user of the client device 230 is looking through a larger surface display (screen) 234 is panning and / or zooming. Can be. The index finger of one hand can stroke the touch pad 238 to pan the image and provide the feeling that a larger image is moved from behind the display 234 using the touch pad 238. In a similar manner, using the wheel 242 to zoom in on the image, the movement of the wheel 242 moves the “larger image” closer or farther to the display 234 depending on the direction of rotation of the wheel 242. It can bring a sense of doing.

  The haptic interface described above can realize a natural and familiar “feel” with a shallow learning curve. One or more embodiments compare to a more conventional solution of using a touch screen that allows an unobstructed view of the display while a finger is stroking the touch pad 238 to pan the image Can also bring benefits. Further, in one or more embodiments, whether using touchpad 238 and wheel 242 or other navigation mechanisms, can be easily alternated between pan and zoom and / or the two activities Can be performed simultaneously or substantially simultaneously to achieve a synergistic benefit.

  In one or more embodiments, the touchpad 238 may have a dimension of about 2 inches (50.8 mm) × about 1.5 inches (38.1 mm), In either dimension, it can be less than or greater than the indicated value.

  FIG. 3 is a flow diagram of a method 320 for navigating image data by a client device 230 according to one or more embodiments of the invention. The steps shown in FIG. 3 are not limited to execution in the order shown. In certain cases, more than one of the illustrated steps can be performed simultaneously. The following lists and summarizes the steps and then describes the application of the steps to basic interactive browsing and variations thereof.

  The method 320 may include accessing 322 an Internet site on the Internet 180 by the proxy server 150. The proxy server can convert 324 image data from the accessed Internet site into a multi-resolution visual data format (multi-resolution image data) that supports spatial random access. JPEG 2000 is one possible example of such a visual data format. However, the present invention is not limited to using this standard. As used herein, the term “convert” generally corresponds to the term “repackage” as applied to converting conventional web page data to multi-resolution image data.

  The method 320 may include transmitting 326 the multi-resolution image data to the client device 230 by the proxy server 150 for final display 328 at the client device 230. Client device 230 may navigate 330 the transmitted multi-resolution image data either after or at the same time as the transmission at step 326. The transmitted multi-resolution image data navigated 330 by the client device 230 can include emulated Internet graphical features such as, but not limited to, dynamic HTML and applet functionality.

  In one embodiment, the proxy server 150 converts 324 image data from a web page on the Internet 180 into multi-resolution image data. This is because the navigation 330 of the client device 230 requires such data.

  The multi-resolution image data is then preferably sent 326 to the client device 230 where it is displayed 328. In this embodiment, access 322 to an Internet site, which can be a web page, by proxy server 150 and navigation of multi-resolution image data by client device 230 can occur substantially simultaneously. Thus, in this embodiment, the transformation 324 of web page (or other type) image data is performed substantially “on the fly” in response to navigation 330 performed by client device 230. Navigation 330 can include, but is not limited to, panning and zooming by client device 230.

  In another embodiment, the proxy server converts or “pre-converts” one or more entire web pages or other forms of Internet image data into multi-resolution image data, and converts the converted data to the client It can be stored in a cache 160 or other memory device accessible to the proxy server 150 for rapid accessibility by the device 230 and transmission to the client device 230. This pre-conversion can be performed before the relevant web page data is needed in response to the navigation 330 command of the client device 230.

  In one or more embodiments, when proxy server 150 services a request for image data from client device 230, it verifies that the original HTML content has not been modified compared to the cached version. This ensures that the cached pre-converted multi-resolution image data is not “old”. This verification can be accomplished by time stamp comparison, checksum comparison, or the use of other well-known methods. A certain tolerance for “age” can be built in to place an upper limit on the number of HTML requests that the proxy server must make. In this disclosure, data that is not “old” is “fresh”.

  In one or more embodiments, proxy server 150 serves as an intermediate between Internet 180 or, more specifically, a site or page of Internet 180 and client device 230. As a result, communication of the proxy server 150 with the Internet 180 is preferably responsive to browsing of multi-resolution data by the client device 230. Also, several of the steps shown in FIG. 3 can be performed interactively and / or simultaneously, rather than strictly following the sequence shown one at a time.

  In general, web page image data flows from the Internet 180 to the proxy server 150, and the proxy server 150 can convert this data into multi-resolution image data. At that point, the multi-resolution image data can be stored in the cache 160 and / or sent to the client device 230 for display on the client device 230. The client device can store the received multi-resolution image data if necessary.

  Navigation commands or browsing commands generally go in the opposite direction. Specifically, navigation requests and / or other commands related to browsing a web page or Internet site originate from the client device 230 and go to the proxy server 150, and then possibly in a modified form. To the Internet 180. Appropriate translation or reconfiguration of commands received from the client device 230 at the proxy server 150 may be performed at the proxy server 150 before retransmitting such commands to the Internet 180. For example, if the client device 230 clicks on a link to another web page, the proxy server 150 can resend the linking command to the Internet 180 to access the target web page. Navigation commands such as pan and zoom can be included in commands that the client device 230 can send. Additional commands include clicking, typing, speaking (if the voice command can be understood by the client device 230) using a mouse or other device, gestures, finger or hand movement along the surface of the touchpad 238, wheel 242 Movement, or movement of the joystick 240 can be included.

  Consider a first example in which a single web page is accessed 322, converted to multi-resolution image data 324, and stored in cache 160. After being stored in the cache 160, the multi-resolution image data can be browsed by the client device 230 as needed.

  Continuing with this example, assume that client device 230 receives the first rendition of the converted web page at a low resolution level. From this starting point (“resolution level 1”), the user of the client device 230 zooms in on the selected area “A” of the web page. Assuming the client device does not have higher resolution image data in its local storage, this zoom command is preferably sent to the proxy server 150, which retrieves the necessary data from the cache 160. The image data can be transmitted to the client device 230.

  Preferably, the range of zoom requested by the client device 230 can be associated by the proxy server 150 with a selected resolution level (“resolution level 2” for this example). The resulting resolution level can be selected from one of the predefined resolution level sets of the selected region (region “A”). Alternatively, if the selected resolution level does not exactly correspond to one of the predefined resolution levels, the required resolution level is 1 as disclosed in the document incorporated herein. It can be calculated by performing one or more interpolation calculations.

  Next, the display 234 of the client device 230 preferably displays the selected area “A” at the selected resolution. Presumably, substantially only region A of the converted web page is displayed on display 234.

  Continuing this example, the user of client device 230 then preferably maintains region “A” in region “A” while maintaining region “A” in region “A”. Consider a situation where you want to pan sideways from A ”to region“ B ”(region not shown). Assuming that multi-resolution image data in region “B” is not available in client device 230, the step of panning toward region “B” preferably causes proxy server 150 to send the image in region “B”. The data is searched from the cache 160 or another data storage device accessible from the proxy server 150, and this data is transmitted to the client device 230. Upon receiving this data, client device 230 must be able to display region “B” at “resolution level 2”. If resolution level 2 image data is not available in cache 160, proxy server 150 preferably accesses the relevant web page again from the Internet 180, and the web page data is stored in cache 160. Note that it has been converted to multi-resolution image data having a higher resolution than that stored.

  Continuing with this example, consider the situation in which the user of client device 230 wishes to activate a hyperlink in the first web page, preferably in region B of the first web page.

  When the user of the client device 230 clicks on a hyperlink, the proxy server 150 can determine that the web page image data associated with the hyperlink has not been placed in the cache 160. Accordingly, proxy server 150 preferably replicates the request for the target web page of the hyperlink in communication with the Internet 180.

  Preferably, proxy server 150 accesses the target web page of the hyperlink. Proxy server 150 then converts the image data from the target web page to multi-resolution image data and dynamically streams the multi-resolution image data to client device 230 for display on client device 230. Can do. Alternatively or additionally, proxy server 150 may store all or part of the multi-resolution image data of the target web page in cache 160 for later access by client device 230 as needed. it can. Such storage in cache 160 or other suitable storage device may prevent client device 230 from receiving and displaying all of the multi-resolution image data of the target (hyperlinked) web page at once. Because there is, it may be beneficial. This inability to receive and display is more likely to exist when the client device 230 selects a high resolution level that displays one or more regions of the target web page. Note that in general, the higher the resolution (displayed area) at which the client device displays multi-resolution image data, the smaller the displayed area.

  In one or more embodiments, the client device 230 may be able to display multiple regions of a web page or multiple regions derived from multiple respective web pages. In this case, the client device 230 is preferably capable of selecting a display resolution independently for each such displayed area. In this case, the touchpad 238 or other mechanism within the client device 230 can preferably move the cursor over each such image to select the desired resolution of each image.

  Furthermore, the resolution need not be constant in any one image that can form only a portion of a web page. Resolution refers to avoiding the burden of displaying a region of interest at high resolution and receiving data storage and the amount of multi-resolution image data required to display the entire high-resolution image. It can be changed as necessary to make competing priorities compatible.

  One or more embodiments of the techniques described herein for proxy web browsing can provide the features described below.

  As mentioned above, zoom functionality can be provided along with proxy web browsing. Thus, a web page formatted to display a wider display than the client device 230 can still be viewed in its entire column width on a small display, such as the display 234. If the text is small relative to the column size, it may be beneficial to pan horizontally to read the entire single line. However, the use of continuous zoom makes it possible to read the entire number of text strings even in a 176 pixel wide display (this width is characteristic of some commercial mobile phones).

  In one or more embodiments, this is possible without reformatting that would change the layout of the web page. FIG. 4 shows a column of web pages displayed in this manner (originally designed for display on an 800 × 200 pixel display).

  If the conversion to the multi-resolution image data format is reasonably fast or the cached multi-resolution image content is already available in the cache 160, the first view of the entire web page Can be rendered visibly on the client device 230 immediately after the request for the relevant web page. This capability can be realized for web pages of any length. This is true whether the initial view of the web page is a zoomed out view of the entire page or a zoomed in view of the top (or any other part) of the web page.

  FIGS. 5A and 6A show these situations, respectively, showing a general overview and top view of a large web page after about 2 seconds of loading over a 20 kbps connection. FIGS. 5B and 6B show clearer versions of FIGS. 5A and 6A, respectively, after a data load time of about 10 seconds.

  For complex web pages with various content, such as newspapers, clients can immediately focus on areas of interest without waiting for the entire document to be downloaded. Please refer to FIG.

  Web pages of virtually unlimited length can be viewed without overburdening the resources of the client device 230, without delaying the time to first view, and without causing performance degradation.

  In one or more embodiments, a web page may have a very large image, and in order to see the fine details of this very large image, the client device 230 places a heavy burden on the client device 230 resources. Without zooming in, without delaying the time to first display of the web page and / or without causing performance degradation.

  In some cases, the display of the small device is higher than its width (eg, 176 pixels wide by 220 pixels high for the Samsung® SGH-D410 mobile phone). Still, for web browsing, the most restrictive variable is column width, which may make it desirable for cellular displays to use a “landscape” page setup rather than a “portrait” page setup. Multi-resolution image data allows rotation of visual content by a selectable angular distance so that larger sized displays can be most effectively used during content browsing. See Figures 8A and 8B.

  In one or more embodiments, the client device 230 accessing multi-resolution image data originates from any server that can access the multi-resolution image data, not just the multi-resolution image data generated by the proxy server 150. Can be enabled to watch. Thus, such a client device 150 provides a large number of large images, large multi-resolution vector maps, and other types of image content that may be difficult or impossible to serve using conventional HTML. It may be enabled to navigate continuously.

  The benefits listed above allow multi-resolution image data browsing using proxy server 150 not only through mobile phone users, but also through narrowband connections such as PDA or palmtop wireless devices, notebook computers with wireless Internet, modems, etc. Can be sufficiently compelling to be useful to users of larger devices, including but not limited to home computer users connected to the Internet and / or home broadband users There is sex.

  For home broadband users, one or more embodiments of a system that uses proxy server 150 to transmit multi-resolution image data to client device 230 provides features including but not limited to the following: be able to.

a) Web pages can be continuously zoomed to improve the readability of small fonts or to examine image details,
b) Web pages designed for small displays can be browsed with clearer details (for text or vector components) on high resolution displays;
c) Relocate, manipulate, and resize multiple web pages in a rich 2D (2D) or 3D (3D) multiscale environment, providing a user experience that replaces traditional multi-window or tabbed browsing And / or
d) Through the judicious use of proxy server 150 caching, a potentially unlimited number of web pages can be created without the need for excessive memory or other resources on the client device 230, and live large-scale “views” of the web. Can be aggregated.

  Finally, displaying images in multi-resolution image data format can provide complementary benefits for large, high-bandwidth and small-display, low-bandwidth platforms, so this form of display is for different platforms. Rather than having to provide different versions of visual content and / or browsers, the user experience can be unified and cross-platform engineering and cross-platform development can be simplified.

  FIG. 9 is a block diagram of a computing system 800 that is adaptable for use with one or more embodiments of the invention. In one or more embodiments, a central processing unit (CPU) 802 can be coupled to the bus 804. Further, the bus 804 may be coupled to a random access memory (RAM) 806, a read only memory (ROM) 808, an input / output (I / O) adapter 810, a communication adapter 822, a user interface adapter 806, and a display adapter 818. it can.

  In one or more embodiments, RAM 806 and / or ROM 808 may hold user data, system data, and / or programs. The I / O adapter 810 can connect a storage device, such as a hard drive 812, a CD-ROM (not shown), or other mass storage device, to the computing system 800. Communication adapter 822 may couple computing system 800 to a local area network, a wide area network, or an Internet network 824. User interface adapter 816 can couple user input devices such as keyboard 826 and / or pointing device 814 to computing system 800. Further, the display adapter 818 can be driven by the CPU 802 to control display on the display device 820. The CPU 802 can be any general purpose CPU.

  Any known processor, programmable digital device or method or apparatus described earlier and / or later in this document that operates to execute standard digital circuits, analog circuits, software and / or firmware programs Note that any known technology can be used, such as a programmable digital system, a programmable array logic device, or any combination of the above. One or more embodiments of the present invention may also be implemented in a software program for storage on a suitable storage medium and execution by a processing unit.

  Although the invention herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present invention. Accordingly, numerous modifications can be made to these exemplary embodiments and other arrangements can be made without departing from the spirit and scope of the invention as defined by the claims appended hereto. Please understand that it can be devised.

  Note that additional disclosure text has been added prior to the summary of this application.

System and method for accurate rendering with a zooming user interface

[Technical field]
The present invention relates generally to a computer graphical zooming user interface (ZUI). More specifically, the present invention is computationally efficient (resulting in good user responsiveness and interactive frame rate), vector drawing, text, and other non-photo content ultimately does not resample Both in the sense that resampling will usually lead to degradation of image quality and in the sense that it will be drawn without interpolation of other images (which should also lead to degradation) A system and method for progressively rendering zoomable visual content in such a way.

[Background]
Most current graphical computer user interfaces (GUIs) are designed using fixed spatial scale visual components. However, since the advent of the field of computer graphics, it has been recognized that visual components can be expressed and manipulated in such a way that they can be zoomed in or out, rather than having a fixed spatial scale on the display. It was. The desirability of zoomable components is evident in many application areas, such as displaying maps, browsing large heterogeneous text layouts such as newspapers, viewing digital photo albums, and working with large dataset visualizations. It is. Even when viewing regular documents such as spreadsheets and reports, it is often useful to be able to see an overview of the document and then zoom in on the area of interest. Microsoft® Word® and other Office® products (zoom under display menu), Adobe® Photoshop®, Adobe® Acrobat®, Many modern computer applications, such as QuarkXPress®, include zoomable components. In most cases, these applications allow zooming in and out of a document, but not necessarily zooming in and out of the visual components of the application itself. Furthermore, zoom is usually a peripheral aspect of user interaction with the software, and zoom settings are only occasionally changed. Continuous panning over a document is standard (ie, using a scroll bar or cursor to translate the document being viewed left, right, up, or down), but is user friendly. The ability to continuously zoom and pan in any way does not exist in prior art systems.

  First, several definitions are shown. A display is one or more devices used to output a rendered image to a user. The frame buffer is used to dynamically represent the contents of at least a portion of the display. The display refresh rate is the rate at which a physical display or part thereof is refreshed using the contents of the frame buffer. The frame rate of the frame buffer is a rate at which the frame buffer is updated.

  For example, in a normal personal computer, the display refresh rate is 60 to 90 Hz. For example, most digital video has a frame rate of 24-30 Hz. Thus, each frame of digital video is actually displayed at least twice when the display is refreshed. Multiple frame buffers can be utilized at different frame rates and thus can be displayed substantially simultaneously on the same display. This should occur, for example, when two digital videos with different frame rates are being played in different windows on the same display.

  One problem with the zooming user interface (ZUI) is that visual content must be displayed at different resolutions when the user zooms. The ideal solution to this problem should be to display an accurate newly calculated image based on the underlying visual content at every successive frame. The problem with such an approach is that the exact recalculation of each resolution of the visual content in real time as the user zooms is computationally impractical when the underlying visual content is complex It is.

  As a result of the foregoing, prior art ZUI systems use multiple pre-computed images, each representing a different resolution of the same visual content. We call each of these different pre-computed images a level of detail (LOD). A complete set of LODs, organized conceptually as a stack of decreasing resolution images, is referred to as a LOD pyramid. Please refer to FIG. In such prior art systems, when zoom occurs, the system interpolates between LODs and displays the resulting image at the desired resolution. Although this approach solves the computational problem, it is often blurred and unrealistic, and ultimately the corrupted image, often due to the loss of information due to the fact that it represents a different LOD interpolation. Is displayed. These interpolation errors are particularly noticeable when the user stops zooming and gets the opportunity to view a still image at a selected resolution that does not exactly match the resolution of any LOD.

  Another problem with interpolation between precomputed LODs is that this approach typically treats vector data in the same way as photographic or image data. Vector data such as blueprints or line drawings are displayed by processing a set of abstract instructions using a rendering algorithm, which renders straight lines, curves, and other primitive shapes at all desired resolutions can do. Text rendered using scalable fonts is an important special case of vector data. Image data or photo data (including text rendered using bitmap fonts) is not generated as such, but by interpolation between precomputed LODs or by resampling the original image Must be displayed in either We refer to this latter as non-vector data.

  Prior art systems that use rendering algorithms to redisplay vector data at a new resolution for each frame during the zoom sequence constrain itself to only simple vector drawings to achieve interactive frame rates. There must be. On the other hand, prior art systems that pre-calculate LOD for vector data and interpolate between them, for non-vector data, the sharp edges inherent in most vector data renditions are particularly sensitive to interpolation errors, so It has a bad visual quality as a drawback. This degradation is usually unacceptable for text content that is a special case of vector data.

[Non-Patent Document 1] Lance Williams. "Pyramidal Parametrics," Computer Graphics (Proc. SIGGRAPH '83) 17 (3): 1-11 (1983)
[Disclosure of the Invention]
[Problems to be solved by the invention]
An object of the present invention is to create a ZUI that duplicates the zoom effect that a user would see if the user actually viewed the physical object and moved the physical object closer to the user himself.

  An object of the present invention is to create a ZUI that displays an image at an appropriate resolution, but avoids or reduces interpolation errors in the final displayed image. Another object of the present invention is to allow a user to arbitrarily zoom in on vector content while maintaining a clear, unblurred view of the content and maintaining an interactive frame rate.

  Another object of the present invention is to allow the user to arbitrarily zoom out to obtain an overview of complex vector content while preserving the overall appearance of the content and maintaining an interactive frame rate. is there.

  Another object of the present invention is to reduce user perception of transitions between interactive LOD or rendition quality.

  Another object of the present invention is to enable graceful degradation of image quality due to blur when the information normally required to render portions of an image is still incomplete.

  Another object of the present invention is to progressively increase image quality by focusing more sharply on the image as more complete information necessary to render portions of the image becomes available. It is.

  An object of the present invention is to optimally and independently render both vector data and non-vector data.

  These and other objects of the present invention will become apparent to those skilled in the art from a review of the following specification.

[Means for solving problems]
The above and other problems of the prior art are overcome in accordance with the present invention, which allows the user to zoom in, zoom out, rotate, pan, or otherwise change that user's view of the image. In particular, it relates to a hybrid strategy for implementing a ZUI that allows images to be displayed at dynamically changing resolutions. All such changes in the view are referred to as navigation. Zooming the image to a resolution that is not equal to any resolution of the predefined LOD is accomplished by displaying the image at a new resolution interpolated from the predefined LOD that “surrounds” the desired resolution. “Enclosing LOD” means the lowest resolution LOD higher than the desired resolution and the highest resolution LOD lower than the desired resolution. If the desired resolution is either higher than the resolution of the LOD with the highest available resolution or lower than the resolution of the LOD with the lowest resolution, there is only a single “enclosing LOD”. Dynamic interpolation of an image at a desired resolution based on a precomputed set of LODs is referred to in the literature as mip mapping or trilinear interpolation. The latter term further indicates that bilinear sampling is used to resample the surrounding LODs, followed by linear interpolation between these resampled LODs (thus trilinear) ( For example, refer nonpatent literature 1). The foregoing document is incorporated herein by reference in its entirety. An obvious change or extension to the mip mapping technique introduced by Williams uses nonlinear resampling and / or interpolation of surrounding LODs. In the present invention, it is immaterial whether the resampling and interpolation operations are zero order (nearest neighbor), linear, higher order, or more generally non-linear.

  In accordance with the invention described herein, when the user defines the exact desired resolution (this resolution is almost never one of the predefined LODs), the final image Is preferably displayed by first displaying the intermediate final image. The intermediate final image is the first image displayed at the desired resolution before the image is refined as described later herein. The intermediate final image may correspond to the image that should be displayed at the desired resolution using conventional techniques.

  In a preferred embodiment, the transition from the intermediate final image to the final image can be gradual as described in detail below.

  In an enhanced embodiment, the present invention provides an irrational increment (ie, a scaling factor between successive LODs that cannot be expressed as a ratio of two integers, as described in more detail below. ) Allows the LOD to be spaced at any resolution increment.

  In another enhanced embodiment, portions of each different LOD image are represented as tiles, and such tiles are rendered in an order that minimizes all perceived imperfections to the viewer. Is done. In other embodiments, the displayed visual content consists of a plurality of LODs (potentially a superset of the surrounding LODs described above), each of these LODs being finalized on the display in a manner that conceals imperfections. It is displayed in the correct position at the correct ratio to fade in the image gradually.

  The renditions of various tiles in multiple LODs are accepted to allow the system to operate on standard computers with normal clock speeds that can be used on most laptop computers and desktop personal computers. Achieved in an order that optimizes the visual content appearance while remaining within the computational complexity of the levels.

  The present invention uses a hybrid strategy where the image is displayed using a predefined LOD during quick zoom and pan, but when the view is sufficiently stable, the exact LOD is rendered and displayed. . The exact LOD is rendered and displayed at the exact resolution selected by the user, and this resolution is usually different from the predefined LOD. Because the human visual system is insensitive to the fine details of visual content while the visual content is still moving, this hybrid strategy creates the illusion of continuous "perfect rendering" with much less computation be able to.

[Best Mode for Carrying Out the Invention]
FIG. 11 shows a flow diagram of the basic technique of implementation of the present invention. The flowchart of FIG. 11 represents an exemplary embodiment of the present invention and begins execution when the image is displayed at the initial resolution. Note that although the invention can be used in a client / server model, the client and server may be on the same machine or on different machines. Thus, for example, a discrete set of LODs stored remotely at a host computer can be provided, and a user can be connected to the host via a local PC. The actual hardware platform and system utilized is not critical to the present invention.

  The flow diagram is entered at start block 241 with an initial view of the image of a particular resolution. In this example, the image is interpreted as static. The image is displayed at block 242. The user can navigate the image, for example, by moving a computer mouse. The initial view displayed at block 242 changes as the user navigates the image. Note that the underlying image itself can be dynamic, such as in the case of motion video, but in this example the image itself is treated as static. As explained above, all displayed images can also have text data or other vector data and / or non-vector data such as photographs and other images. The present invention and the entire discussion below are applicable regardless of whether the image contains vector data or non-vector data or both.

  Regardless of the type of visual content displayed at block 242, the method transfers control to decision point 243, which can detect navigation input. If no such input is detected, the method loops back to block 242 and continues to display still visual content. If navigation input is detected, control is transferred to block 244 as shown.

  Decision point 243 may be implemented by a continuous loop of software looking for a specific signal to detect movement, a hardware interrupt system, or any other desired methodology. The particular technique utilized to detect and analyze navigation requests is not critical to the present invention. Regardless of the methodology used, the system can detect the request and thus indicate the desire to navigate the image. Although much of the discussion herein relates to zooming, it should be noted that these techniques are applicable to zooming, panning, or other forms of navigation. Indeed, the techniques described herein are applicable to all types of dynamic transformations or perspective changes on the image. Such transformations reveal, for example, three-dimensional translation and rotation, application of image filters, local stretching, dynamic spatial distortion applied to selected areas of the image, or more information All other types of distortion that could be included. Another example is a virtual magnifying glass that can move over the image and magnify the lower part of the virtual magnifying glass within the image. When decision point 243 detects that the user is about to begin navigation, block 244 renders and displays a new view of the image, which is different from, for example, the previously displayed view. Resolution may be used.

  One simple prior art technique for displaying a new view is based on interpolating the LOD as the user zooms in or out. The selected LODs can be two LODs that “enclose” the desired resolution, ie the resolution of the new view. Interpolation occurs constantly in conventional systems when the user zooms, and is therefore often implemented directly in hardware to achieve speed. The combination of motion detection at decision point 245 and a substantially immediate display of the appropriate interpolated image at block 244 ensures that the image appears to continuously zoom as the user navigates. Bring. Since the image is moving during zooming in or out, the interpolated image is sufficient to look realistic and clear. Interpolation errors are hidden by a constantly changing view of the image and are therefore only minimally detectable by the human visual system.

  At decision point 245, the system tests whether the movement has substantially stopped. This can be achieved using various techniques including, for example, measuring the rate of change of one or more parameters of the view. That is, the methodology ascertains whether the user has reached a point where zooming has been completed. Upon such stabilization at decision point 245, control is transferred to block 246, where the correct image is rendered, and then control returns to block 243. Thus, at all desired resolutions, the system eventually displays an accurate LOD.

  Notably, the display is simply rendered by interpolation of two predefined LODs, not rendered, but instead renders text data or other vector data when the initial view is displayed at block 242. Can be rendered and displayed by re-rendering the vector data using the original algorithm used to do this. Non-vector data can also be resampled for rendering and display at the exact required LOD. The requested re-rendering or re-sampling is not only the exact resolution required for display at the desired resolution, but also the correct position of the display pixel relative to the underlying content, calculated based on the desired view. It can also be performed on a precisely corresponding sampling grid. As an example, a ½ pixel translation in the display plane of the image on the display does not change the required resolution, but changes the sampling grid and therefore requires accurate LOD re-rendering or resampling. To do.

  The aforementioned system of FIG. 11 uses pre-defined LOD-based interpolation while the view is changing (eg, navigation is taking place), but when the view is substantially stationary. Represents a hybrid approach where an accurate view is rendered and displayed.

  In the description herein, “rendering” refers to the computer generation of tiles in a particular LOD based on vector data or non-vector data. For non-vector data, these non-vector data can be re-rendered at any resolution by re-sampling the original image at a higher or lower resolution.

  We now move on to a methodology for rendering and displaying the different parts of the visual content necessary to achieve the correct final image represented by block 246 in FIG. Referring to FIG. 12, when it is determined that navigation has stopped, control is transferred to block 333 and the interpolated image is immediately displayed as if it were zoomed. We refer to this interpolated image that can be temporarily displayed after navigation stops as the intermediate final image or simply the intermediate image. This image is generated from interpolation of the surrounding LOD. In some cases, as described in more detail below, the intermediate image may be interpolated from more than two discrete LODs or from two other discrete LODs that surround the desired resolution.

  Once the intermediate image has been displayed, block 334 is entered, which causes the image to begin to fade progressively toward the exact rendition of the image we call the final image. The final image differs from the intermediate image in that the final image may not involve any predefined LOD interpolation. Instead, the final image or portions thereof can include newly rendered tiles. In the case of photographic data, the newly rendered tiles can be derived from resampling of the original data, and in the case of vector data, the newly rendered tiles are taken at the desired resolution. It can be derived from rasterization.

  Note also that it is possible to skip directly from blocks 333 to 335 and immediately replace the interpolated image with the final accurate image. However, in the preferred embodiment, step 334 is performed, so the transition from the intermediate final image to the final image is made progressively smooth. This gradual fade, sometimes referred to as blending, causes the image to gradually focus when navigation stops, creating an effect similar to the autofocus of a camera or other optical device. The physical illusion created by this effect is an important aspect of the present invention.

  The following is a discussion of how this fade or blending can be done to minimize perceived irregularities, sudden changes, seams, and other imperfections in the image. However, it should be understood that the particular technique of fading is not critical to the present invention and numerous variations will be apparent to those skilled in the art.

  Different LODs differ in the number of samples per physical area of the underlying visual content. Thus, the first LOD may occupy a 1 inch (25.4 mm) x 1 inch (25.4 mm) area of visible object and produce a single 32 x 32 sample tile. However, rendering this information by taking the same 1 inch (25.4 mm) x 1 inch (25.4 mm) area and representing it as a tile that is 64 x 64 samples, and therefore at a higher resolution. You can also

  We define a concept called irrational tiling. The tiling granularity, described as the variable g, is defined as the ratio of the higher resolution LOD linear tile grid size to the next lower resolution LOD linear tiling grid size. In the Williams paper introducing trilinear interpolation, g = 2. This same value of g has been used in other prior art. The LOD may be subdivided into tiles in any way, but in the exemplary embodiment, each LOD is subdivided into a grid of square or rectangular tiles containing a certain number of samples (at the edge of the visual content). Unless necessary). Conceptually, when g = 2, each tile in a LOD is “split” into 2 × 2 = 4 tiles in the next higher resolution LOD as shown in FIG. , Potentially excluding edges).

  There are fundamental drawbacks to tiling with a particle size of 2. Typically, when the user zooms in on a random point in the tile, every g-fold increase in zoom is simply a corresponding to the next higher resolution LOD near the point that the user is zooming towards it. Requires one additional tile rendition. However, if the user is zoomed in on a grid line in the tiling grid, two new tiles need to be rendered, one on each side of the line. Finally, if the user is zoomed in on the intersection of two grid lines, four new tiles need to be rendered. If these events (requiring one, two, or four new tiles with each g-zoom zoom) are randomly scattered throughout the long zoom sequence, the overall performance is consistent. Become a thing. However, the grid lines for all integer granularity tilings (ie when g is an integer) will remain grid lines for all higher resolution LODs.

  For example, consider zooming in on the center of a very large image tiled with granularity 2. We have adopted the convention that visual content is contained in a square with corners (0,0), (0,1), (1,0), and (1,1), and (x, y) Write the coordinates as (1/2, 1/2). Since the center is at the intersection of the two grid lines, when the user reaches each higher resolution LOD, four new tiles need to be rendered each time, which is the zoom at this particular point. Results in slow performance and inefficiencies. On the other hand, assume that the user zooms in on an irrational point (meaning a grid point (x, y) that cannot be expressed as a ratio of two integers). Examples of such numbers are pie (= 3.141459 ...) and the square root of 2 (= 1.414213 ...). Next, it is easy to see that the sequence of 1, 2, and 4 given by the number of tiles that need to be rendered for all zooms of g times is pseudo-random, ie does not follow a periodic pattern. Can be demonstrated. This type of pseudo-random sequence is clearly more desirable from a performance standpoint, and there is no distinction regarding zoom from a performance standpoint.

  Irrational tiling solves this problem. g itself is made to be an irrational number, usually the square root of 3, 5, or 12. This means that an average of 3, 5, or 12 tiles (corresponding to g) at a given LOD is contained within a single tile of the next lower resolution LOD. Note that tiling grids in successive LODs no longer “match” any grid lines in this manner (potentially the leading edge of visual content, ie x = 0 and y = 0, Except for some other pre-selected single grid line along each axis). If g is chosen to be no nth root of any integer (pie is such a number), no LOD shares any grid lines (again, potentially x = 0 and except y = 0). Thus, it can be shown that each tile can randomly overlap one, two, or four tiles with the next lowest LOD, but this number is always 1 for g = 2. .

  Thus, with irrational tiling granularity, zooming in on any point creates a pseudo-random stream of one, two, or four tile requests and the performance is no matter where you zoom in It becomes uniform on average. Perhaps the greatest benefit of irrational tiling appears in relation to panning after deep zooming. When the user pans the image after deep zooming in, at some point the grid lines are moved on the display. Normally, the area on the other side of this grid line will correspond to a lower resolution LOD than the rest of the display, but the difference between these resolutions should be as small as possible. However, using an integer g, this difference is often extremely large since grid lines can overlap across many consecutive LODs. This creates a resolution “deep crack” that spans the node region, as shown in FIG.

  On the other hand, irrational tiling grid lines never overlap with adjacent LOD grid lines (again, one grid line in each direction, which can be at one corner of the image). Since there are possible exceptions), discontinuities in resolution of multiple LODs do not occur. This increased smoothness in relative resolution allows the illusion of spatial continuity to be much more compelling.

  FIG. 15 (b) shows the benefits obtained by irrational tiling granularity. FIG. 15 shows a cross section through multiple LODs of visual content, where each bar represents a cross section of a rectangular tile. Therefore, the second level from the top has 2 bars, but can be 2 × 2 = 4 tile LOD. Curve 631, drawn from top to bottom, represents the boundaries of the visible content of the visual content at the associated LOD during the zoom operation, increasing the resolution (zoom in to reveal more details). Sometimes the area under inspection decreases. Darker colored bars (eg, 632) represent tiles that have already been rendered during the zoom process. The lighter colored bars have not yet been rendered and therefore cannot be displayed. When tiling is integer as shown in FIG. 15 (a), an unexpected change in resolution across space is common, and if the user pans immediately after zooming, the arrow Note that the four LODs unexpectedly “finish” at the spatial boundary indicated by. The resulting image looks sharp on the left side of this boundary, but appears extremely blurred on the right side. Identical visual content represented using irrational tiling granularity does not have such a “crack” of resolution, and adjacent LODs share tile boundaries except those shown on the left edge do not do. Mathematically, this shared boundary can occur at most at one location on the x-axis and one location on the y-axis. In the illustrated embodiment, this shared boundary is positioned at y = 0 and x = 0, but if present, they can be placed in any other location.

  Another benefit of irrational tiling granularity is that there are far more irrational numbers than integers, especially over a useful range where g is not too large, so irrational tiling granularity allows finer control of g. It is to be. This additional freedom may be useful for tuning the zoom performance of certain applications. When g is set to an integer irrational square root (such as sqrt (2), sqrt (5), or sqrt (8)), in the embodiment described above, alternating LOD grid lines are accurate If g is an irrational cube root, every third LOD should be correctly aligned, and so on. This provides an additional benefit related to limiting the complexity of the composite tile defined below.

  An important aspect of the present invention is the order in which tiles are rendered. More specifically, different tiles of different LODs are optimally rendered so that all visible tiles are rendered first. Invisible tiles may not be rendered at all. Within the set of visible tiles, renditions proceed in increasing resolution order so that tiles in the low resolution LOD are rendered first. Within any particular LOD, tiles are rendered in order of increasing distance from the center of the display, we refer to this as a focused rendering. A number of sorting algorithms such as heap sorting, quick sorting, or other sorting can be used to sort such tiles in the order described. To implement this ordering, lexicographic keys can be used to sort "requests" and tiles can be rendered with the outer subkey being visible and the middle subkey being in samples per physical unit. Resolution, the inner subkey is the distance to the center of the display. Other methods of ordering tile rendering requests can also be used. The actual rendering of the tiles is best performed as a parallel process with the navigation and display described herein. When rendering and navigation / display proceed as a parallel process, user responsiveness may remain high even when tile rendering is slow.

  The process of rendering tiles in an exemplary embodiment will now be described. If a tile represents vector data, such as alphabetic typography for stroke-based fonts, rendering this tile executes an algorithm that rasterizes the alphabetic data and possibly sends that data from the client to the server. Accompanied by. Alternatively, data supplied to the rasterization algorithm can be sent to the client, which can execute an algorithm to rasterize the tiles. In another example, rendering a tile with digitally sampled photographic data can be used to resample that data to generate a tile with an appropriate LOD. For pre-stored discrete LODs, rendering can only use simple transmission of tiles to the client computer for later display. For tiles that are included during discrete LODs, such as tiles in the final image, some further calculations as described above may be required.

  At any given time, when a tile is rendered and the image begins to fade towards the correct image, the actual display may contain a different mixture of different tiles from different LODs. Thus, any portion of the display may include, for example, 20% from LOD 1, 40% from LOD 2, and 40% from LOD 3. Regardless of the tiles that are displayed, the algorithm will render tiles from various LODs in a priority order that is best suited to supply tiles rendered for display when they are most needed. Try. The actual display of the rendered tile will be described in more detail later with respect to FIG.

  The following describes how to draw multiple LODs using an algorithm that can guarantee the spatial and temporal continuity of image details. This algorithm uses high resolution tiles over lower resolution tiles that cover the same display area, and uses spatial blending to avoid sharp boundaries between LODs, with higher details. When available, all at that time (ie when higher resolution tiles have been rendered), using temporally progressive blending weights to blend in that higher detail Designed to best utilize rendered tiles. Unlike the prior art, this algorithm and its variants can result in three or more LODs being blended together at a given point on the display, with a blending factor that varies smoothly across the display area. And a blending factor that gradually changes over time even after the user has stopped navigating. In this exemplary embodiment, the algorithm is nevertheless computationally efficient, as will become apparent below, and the overall variation of the image as partially transparent or across image regions. Can be used to render with transparency.

  We define herein a composite tile region or simply a composite tile. To define a composite tile, consider all of the LODs stacked on top of each other. Each LOD has its own tile grid. A composite grid is formed by projecting all grids from all LODs onto a single plane. The composite grid is composed of various composite tiles of different sizes, defined by tile boundaries from all of the different LODs. This is conceptually illustrated in FIG. FIG. 16 shows tiles from three different LODs 701-703, all representing the same image. It can be imagined that LODs 701 to 703 are stacked on top of each other. In that case, when the corners 750 of each of these LODs are aligned and stacked on top of each other, the region represented by 740 will be inside the region represented by 730 and represented by 730 and 740. The area to be done is inside the area represented by 720. Region 710 in FIG. 16 indicates that there is a single “composite tile” 710. Each of the composite tiles is examined during each frame, and the frame rate can typically exceed 10 frames per second. Note that as explained above, this frame rate is not necessarily the display refresh rate.

  FIG. 14 shows a flowchart of an algorithm for updating the frame buffer when a tile is rendered. The arrangement of FIG. 14 is intended to manipulate all composite tiles in the displayed image each time the frame buffer is updated. Thus, for example, if the frame duration is 1/20 of a second, each of the composite tiles of the entire screen is preferably examined and updated during each 1/20 of a second. When a composite tile is manipulated by the process of FIG. 14, the composite tile may lack an associated tile in one or more LODs. The process of FIG. 14 attempts to display each composite tile as a weighted average of all available superimposed tiles within which the composite tile is located. Note that composite tiles are defined as being contained in exactly one tile for all given LODs, and thus the weighted average can be expressed as a relative proportion of each LOD. This process determines the appropriate weights for each LOD in the composite tile to progressively fade the image towards the final image described above, and progressively these weights over space and time. And try to change.

  The composite grid includes a plurality of vertices defined to be any intersection or corner of the grid lines in the composite grid. These are referred to as composite grid vertices. We define opacity at each composite grid vertex for each LOD. This opacity can be expressed as a weight between 0.0 and 1.0, so the sum of all LOD weights at each vertex is the desired result that the image is completely opaque. In some cases it must be 1.0. The current weight at any particular time at each vertex per LOD is maintained in memory.

  The algorithm for updating the vertex weights proceeds as described below.

  The following variables are selected to be a number between 0.0 and 1.0, but are kept in memory for each tile: centerOpacity, cornerOpacity at each corner (tiling is a rectangular grid 4) and edge Opacity per edge (4 if tiling is a rectangular grid). When a tile is first rendered, all of its opacity listed directly above is usually set to 1.0.

  During the drawing pass, the algorithm walks through composite tiling, starting with the highest resolution LOD, once for each associated LOD. In addition to the per-tile variables, this algorithm maintains the following variables: levelOpacityGrid and opacityGrid. Both of these variables are also numbers between 0.0 and 1.0 and are maintained for each vertex in the composite tiling.

  The algorithm walks through each LOD in turn, performing the following operations in order from highest resolution to lowest resolution. First, 0.0 is assigned to the levelOpacityGrid of all vertices. Then, for each rendered tile in that LOD (which may be a subset of the set of tiles in that LOD if some tiles have not yet been rendered), the algorithm uses the centerOpacity of that tile. Based on the value, the cornerOpacity value, and the edgeOpacity value, the part of the levelOpacityGrid that touches the tile is updated as follows.

  If the vertex is completely inside the tile, the vertex is updated using centerOpacity.

  If the vertex is at the left edge of the tile, for example, the vertex is updated using the left edgeOpacity.

  If the vertex is at the upper right corner, for example, the vertex is updated using the upper right cornerOpacity.

  “Update” means the following. If the pre-existing levelOpacityGrid value is greater than 0.0, the new value is set to the minimum of the current value or the value that this variable is about to be updated to. If the pre-existing value is 0 (ie, this vertex has not yet been manipulated), set the levelOpacityGrid value to the value that this variable is about to be updated to. The net result is that the levelOpacityGrid at each vertex position is set to the minimum non-zero value to which this variable is updated.

  The algorithm then walks through the levelOpacityGrid and sets 0.0 to all vertices that touch the unrendered tile, called the hole. This means that blending spatial continuity, that is, whenever a composite tile is included in a hole, the drawing opacity must fade to zero at all vertices that touch the hole at the current LOD. Guarantee.

  In an enhanced embodiment, the algorithm then relaxes the levelOpacityGrid value to further improve the spatial continuity of LOD blending. The situation described so far can be drawn as follows. All vertices are similar to a tent post where the levelOpacityGrid is the height of the tent post. The algorithm so far has that the tent post has a height of zero at every point that touches the hole, and inside the rendered tile, the (possibly) non-zero value at the tent post is set. Guarantee. In extreme cases, perhaps all values inside the rendered tile are set to 1.0. For purposes of illustration, assume that the rendered tile does not yet have a rendered adjacent tile, and therefore the boundary value is 0.0. In the present specification, it has not been specified how narrow the “margin” is between the 0.0 boundary tent column and the 1.0 internal tent column. If this margin is too small, the transition may be too sharp when measured as an opacity derivative across space, even if the blending is technically continuous. The relaxation action smoothes the tent and always preserves the value of 0.0, but possibly lowers the other tent struts to make the function defined by the tent surface smoother, ie its maximum spatial derivative Limit. Which of the various methods is used to implement this behavior is not important to the present invention, and one approach uses, for example, selective low-pass filtering, leaving zero untouched. However, locally replacing all non-zero values with a weighted average of its neighboring values. Other methods will be apparent to those skilled in the art.

  The algorithm then walks through all the composite grid vertices and examines the corresponding values of levelOpacityGrid and opacityGrid for each vertex, and if the levelOpacityGrid is greater than 1.0-opacityGrid, the levelOpacityGrid is set to 1.0. Set -opacityGrid Next, again, for each vertex, the corresponding value of levelOpacityGrid is added to opacityGrid. Due to the previous step, this can never cause the capacityGrid to exceed 1.0. These steps of this algorithm are such that when higher resolution LOD becomes available, as much opacity as possible is contributed by those higher resolution LODs, and lower resolution only if there are holes. Guarantees that the LOD of the “LOD” is allowed to “see through”.

  The final step in the current LOD traversal is to actually draw a composite tile of the current LOD using levelOpacityGrid as the opacity value per vertex. In an enhanced embodiment, levelOpacityGrid can be multiplied by a scalar overallOpacity variable in the range of 0.0 to 1.0 just before drawing, thereby using partial transparency given by overallOpacity to image It becomes possible to draw the whole. Note that drawing a polygon containing an image, such as a rectangle, with different opacity at each vertex is a standard procedure. This can be accomplished, for example, using industry standard texture mapping functions using the OpenGL graphics library or the Direct3D graphics library. In practice, the opacity drawn within the interior of each such polygon is spatially interpolated, resulting in a smooth change in opacity across the polygon.

  In another enhanced embodiment of the algorithm described above, the tiles are not only the current values of centerOpacity, cornerOpacity, and edgeOpacity (called the current value), but also the targetCenterOpacity, targetCornerOpacity, and the targetEdget value. It also maintains a parallel set (called the target value). In this enhanced embodiment, the current values are all set to 0.0 when the tile is first rendered, but the target values are all set to 1.0. Then, after each frame, the current value is adjusted to a new value that is closer to the target value. This can be done using several mathematical formulas, but as an example, it can be done in the following way: New value = old value * (1-b) + target value * b, where b is a rate greater than 0.0 and less than 1.0. A value of b close to 0.0 results in a very slow transition toward the target value, and a value of b close to 1.0 results in a very fast transition toward the target value. This method of updating opacity results in exponential convergence towards the target, giving a visually satisfying impression of temporal continuity. Other formulas can achieve the same result.

  FIG. 17 is a diagram illustrating successive stages of rendering in a forbidden order of the second layer of the LOD pyramid of FIG. 17 in accordance with one or more embodiments of the present invention.

  The foregoing describes preferred embodiments of the present invention. The present invention is not limited to such preferred embodiments, and various modifications consistent with the appended claims are also included in the present invention.

System and method for multi-node display

[Technical field]
The present invention relates to a zooming user interface (ZUI) of a computer.

[Background]
Most today's graphical computer user interfaces are designed with fixed spatial scale visual components. The visual component can be manipulated by zooming in or out or otherwise navigating it. However, the accuracy with which the coordinates of various objects can be represented is extremely limited by the number of bits, usually between 16 and 64, specified to represent such coordinates. Because of this limited representation size, there is limited accuracy.

  In the context of a zooming user interface, the user can easily zoom in and fill the entire display with an area that previously covered only a single pixel. Conversely, the user can zoom out and reduce the content of the entire display to the size of a single pixel. Since each zoom-in or zoom-out may multiply or divide the xy coordinates by a number of digits, only a few such zooms will fully use the precision available using, for example, 64-bit floating point numbers. Use up. Thereafter, rounding significantly degrades the image quality.

[Patent Document 1] US Patent Application No. 10/790253
[Disclosure of the Invention]
[Problems to be solved by the invention]
It is an object of the present invention to provide a ZUI that allows a larger range of zoom.

  Another object of the present invention is to provide a ZUI in which the accuracy with which coordinates are represented is related to the required accuracy required at a particular zoom detail level. Another object of the present invention is to define a pannable and zoomable two-dimensional space of finite physical size, but with arbitrarily high complexity or resolution, to define a larger pannable and zoomable two-dimensional space. It is possible to embed in a defined area.

  Another object of the present invention is that zooming out after deep zooming in can behave like a “back” button in a web browser, allowing the user to turn back his step through visual navigation. It is to be.

  Another object of the present invention is to allow zoom-in immediately after zoom-out to behave like a “forward” button in a web browser, allowing the user to accurately reverse the effect of arbitrarily long zoom-out. Is to make it possible.

  Another object of the present invention is to allow a node, ie a visual object defined more precisely below, to have a very large number of child nodes (eg up to 10 ^ 28).

  Another object of the present invention is to allow nodes to programmatically create their own children on the fly, allowing content to be dynamically defined, created or modified during navigation. It is to be.

  Another object of the invention is that even if the content is ultimately represented using a very large amount of data, that data is stored at a remote location and shared via a low bandwidth network. Is to allow almost immediate viewing of arbitrarily complex visual content.

  Another object of the present invention is to allow a user to arbitrarily zoom in on visual content while maintaining an interactive frame rate.

  Another object of the present invention is to allow the user to arbitrarily zoom out to get an overview of complex visual content, both in the process of preserving the overall appearance of the content and maintaining the interactive frame rate. It is to be.

  These and other broader objects of the present invention will become apparent to those skilled in the art from a review of the following specification.

[Means for solving problems]
The above and other objects of the present invention are achieved by displaying visual content as a plurality of “nodes”. Each node preferably has its own coordinate system and rendering method, but can be contained within one parent node and can be represented in the parent node's coordinate system and rendering method. When the user navigates this visual content, for example by zooming in or out, the node is only “activated” when the zoom provides the appropriate level of detail. Invoking a node causes that node to be represented in its own coordinate system and / or rendering method, rather than the coordinate system and / or rendering method of a different node.

  Before a node is activated, it is either represented in the parent node's coordinate system or not represented at all. By activating a node only when it is needed, the accuracy of the coordinate system becomes a function of the zoom detail level of what is being displayed. This allows a variable level of accuracy up to and including the maximum value allowed by the memory of the computer on which the system operates.

  For the purpose of illustration, the presently preferred form is shown in the drawings, but it should be understood that the invention is not limited to the precise arrangements and instrumentalities shown.

[Best Mode for Carrying Out the Invention]
We assume a user interface metaphor where the display is a camera, through which the user can see a part of a two-dimensional surface or 2D universe. For convenience, it is not necessary to do so, but the physical dimension is attributed to this universe, so that it can be a square of, for example, 1 meter. The present invention is equally applicable to N-dimensional representations.

  An exemplary universe includes 2D objects or nodes, has a visual representation, and can also be dynamic or interactive (ie, video clips, applications, editable text documents, CAD drawings, or still images). . In order for a node to be visible, it must be associated with a rendering method that can draw the node in whole or in part in an area of the display. Each node is also given a finite precision local coordinate system. For illustration purposes, we assume that the node is rectangular and is represented by a local coordinate system.

  These two parameters, the rendering method and the coordinate system, specify how the node is displayed and the position of the item within the node. Each node can have zero or more child nodes, and these child nodes are addressed by reference. A node need not contain all the information of each child node, but generally does not contain it, but instead contains only an address that provides the information necessary to obtain the child node. As the user navigates, eg zooms in and zooms out, the nodes are displayed on the screen as shown for example in FIG.

  In general, a “node” is the basic unit of functionality in the present invention. Most nodes appear visually on the user's display during navigation, and some nodes may be animated and / or respond to user input. Nodes are hierarchical in that a node can contain child nodes. The containing node is called the parent node. When a parent node includes a child node, the child node's visual manifest station is also included within the parent visual manifest station. Each node has a logical coordinate system, and the entire extent of the node is contained in an exemplary rectangle defined within the logical coordinate system, for example, a node has a rectangular (0,0)-( 100, 100) can be defined as a logical coordinate system.

  Each node can have the following data defining its properties.

The logical coordinate system of the node, including its logical size (100x100 in the example above);
○ The identity, position, and size of any child node specified in the (parent) node's logical coordinate system,
○ Optionally, any required user data and
− Executable code that defines the following behavior or “method”: ○ Initialization of node data based on “construction argument”;
○ Rendering all or part of the visual appearance of the node (the output of this method is the rendered tile);
○ Optionally, respond to user input such as keyboard or mouse events.

  Executable code defines a “node class” and can be shared among multiple “node instances”. Node instances differ in their data content. Thus, the node class can define the logic necessary to render a JPEG image. The “construction argument” given to the initialization code should include the URL of the displayed JPEG image. A node that displays a particular image should be an instance of a JPEG node class. Similar to the way a software application can be instantiated multiple times simultaneously, multiple instances of a node can be displayed in the same visual content.

  Note that in complex visual documents or applications, it is usually possible to divide the required functionality into nodes in many different ways. For example, a web page-like document written by a script that includes multiple images, pull-down menus, and buttons can be implemented as a single node with complex rendering and user input methods. Alternatively, the document can be implemented as a parent node that only defines the overall layout of the page, and all constituent images and buttons can be implemented as child nodes. This has the obvious benefit of more effectively reusing or “factoring” the functionality, and the buttons all have the same behavior and are therefore all instances of the same node class Images can all be in the same format, and thus can be instances of a common node class, and so on. This also simplifies layout rearrangements, where a parent node can easily move child nodes or resize child nodes.

  According to the present invention, visual content can be displayed in a form that depends on the state of navigation input by a user. For example, FIG. 18 shows a node 105 that can be an image of a part of a city. The node 105 can include child nodes 101-103. Node 101 can be an image of a building in the city, node 107 can be an image of a playground, and node 103 can be a sports arena. At the level of zoom shown, nodes 101-103 are relatively small and therefore represent them as small darkened areas without details in node 105 that are placed in the correct position in the coordinate system of node 105. be able to. Only the coordinate system of the node 105 and the rendering method are required.

  Consider the case where the user zooms in, resulting in a different level of detail (LOD) such as that shown in FIG. In the LOD of FIG. 19, nodes 101 and 107 are no longer visible on the screen due to the fact that visual content is displayed much larger. Furthermore, it should be noted that since the size at which the sports arena node 103 is displayed is now much larger, details of the sports arena such as individual seats, fields, etc. should now be displayed.

  To facilitate the foregoing, the sports arena node 103 is now displayed using its own coordinate system and rendering method, rather than being displayed as a darkened area without details in the coordinate system of the node 105. "Activated" to be displayed. Details such as seating, competition fields, etc. should be shown individually when displayed using its own coordinate system and rendering method. The other functions described above and the functions related to the node 103 should also start executing when the node 103 is activated. The particular navigation conditions that cause the activation of node 103 or all nodes related thereto are a function of design choice and are not critical to the present invention.

  The accuracy with which the node 103 is displayed is the combined accuracy of the coordinate system used by the node 105 and the coordinate system used by the node 103. Thus, for example, if each coordinate system of the node uses 8 bits, the combined accuracy is only used by the coordinate system of the node 103 specifying the position of the item in the node 103, Since the overall position of the node 103 in the node 105 is specified in the coordinate system of the node 105, it becomes 16 bits. Note that this nesting may continue repeatedly if the sports arena 103 itself includes additional nodes within it. For example, one such node 201 may actually be a specific in-store counter within this sports arena. This is represented without much detail in the coordinate system of node 103 and the rendering method. When the user continues zooming in on the sports arena 103, the node 201 is activated at some point. If node 201 is displayed using 8 bits of precision, these 8 bits specify where in the node 201 coordinate system a particular item must be displayed. Nevertheless, the position of the node 201 in the node 103 is maintained with an 8-bit precision in the coordinate system of the node 103, and the position of the node 103 is maintained using 8 bits in the coordinate system of the node 105. . Thus, the items in node 201 are ultimately displayed using 24-bit precision.

  By nesting nodes within nodes, the accuracy with which visual content can ultimately be displayed is limited only by the memory capacity of the computer. The final accuracy with which visual content within a node is displayed after the node is activated effectively becomes the combined accuracy of all parent nodes and the accuracy of the activated node. Thus, depending on the level of nesting, the accuracy can be increased as needed, limited only by the storage capacity of the computer, which is almost always much more than sufficient. Furthermore, the increased accuracy is only used when necessary. This is because, if the image is in a LOD that does not require booting, according to the above description, the image will only be displayed with the accuracy of the node included when the node is booted. It is. Thus, for nodes nested within other nodes, activated nodes can be traversed until they eventually reach a node that has not yet been activated when moving inward from the outermost node. All such unstarted nodes and also the nodes within them are displayed only with the accuracy of the last traversed node that was started.

  This results in an “accordion” type of accuracy, where the accuracy with which the visual content is displayed expands and contracts as needed according to instructions from the user's navigation input, when needed for higher accuracy. Maximize system resource efficiency by using limited system resources.

  It should also be noted that when a node is activated, its display changes from that based on the parent node's coordinates and rendering method to the child node's coordinates and rendering method. The change is optimally gradual through the use of blending, as described, for example, in US Pat. However, other methodologies that change progressively from displaying information in the parent node's coordinate system and rendering method to child nodes are possible. For example, a system where parent-to-child blending occurs over a specific range can be programmed. Next, when the user traverses through the range during zooming, a switchover occurs unless navigation is stopped in the range, in which case the blending is in the appropriate coordinate system. You can continue until it is fully displayed.

  An additional problem solved by the present invention relates to a system that maintains the spatial interrelationship between all nodes being displayed. More specifically, many different coordinate systems are used to display potentially different nodes during dynamic navigation such as zooming and panning. Some nodes are simply displayed as images in the coordinate system of other nodes, as described above, and some nodes are displayed in their own coordinate system. In fact, the entire visual display can capture nodes that appear at different positions in different coordinate systems, and the coordinate system and accuracy used for the various nodes can be determined when the node is activated during navigation. May change. Therefore, it is important to ensure that the nodes are placed correctly with respect to each other, since each node only knows its own coordinate system. The present invention propagates relative position information between all nodes to ensure that each node “knows” the correct position within the overall view that it must render, and Provides techniques for updating information.

  The foregoing can be accomplished with the addition of fields to the node structure and additional address stack data structures. The extended node definition includes a field used by the node to position itself relative to the entire display, which we call the “view” field. The view field represents the image of the display rectangle at the node's visible area, ie, the node's coordinates, at the node's coordinates. This rectangle may only partially overlap the area of the node, such as when the node is partially off-screen. Obviously, the view field cannot always be kept updated for all nodes. This is because it is not always possible to traverse the entire directed graph of nodes in real time when navigation occurs.

  The stack structure is defined as follows.

Stack <Address>viewStack;
Here, this stack is a global variable of the client (computer connected to the display). For purposes of illustration, assume that navigation begins with an overview of the universe of content defined by the root node. This root node can then be pushed to viewStack and the view field of the root node can be initialized to be the entire area of the root node, ie
rootNode.view = rootNode.coodSystem;
Push (viewStack, rootNode);
It is.

  In general, viewStack specifies the address of a sequence of “pierced” nodes by one point relative to the display, which in this example implementation is interpreted as the center of the display. Is done. This sequence must start at the root node, but may be infinite and therefore must be truncated. In the exemplary embodiment, this sequence is truncated when the “pierced” node is smaller than some minimum size, defined as minimumArea. The current view is then represented by the view field of all nodes in the viewStack, each of which specifies the current view with respect to the node's local coordinate system. If the user zooms very deep into the universe, the detailed position of the display is most accurately given by the view field of the last node in the stack. However, the view field of the last element only specifies the user's viewpoint relative to its local coordinates, not relative to the entire universe. On the other hand, the view field of the root node specifies where in the universe the user is looking. Thus, nodes closer to the “fine edge” of viewStack specify the view position with increasing accuracy, albeit relative to the gradually narrowing area in the universe. This is conceptually shown in FIG. 20. In FIG. 20, among the three activated nodes 301, 302, and 303, the node 303 has the coordinate system “the finest”, so that the user sees it. It can be seen that node 301 provides information on a much larger area of visual content, although not as fine as it provides.

  The problem is then translated as follows: the view of all visible nodes (ie the view field) is synchronized when the user navigates through the universe, ie pan and zoom Must be kept on. Failure to keep them synchronized should lead to the appearance that the nodes move on the display independently of each other, rather than behaving as a coherent and physically consistent 2D surface. .

  View changes during all navigation operations proceed as follows. Since the last node in viewStack has the most accurate representation of the view, the first step is to change the view field of this last node. This modified view is taken as the correct new view and all other visible nodes must go after. The second step is to propagate this new view “upward” towards the root node, which may involve making progressively smaller changes to the previous node's view field in the stack. included. If the user zooms deeply, at some point in the upward propagation, changes to the view can be so small that it cannot be accurately represented, and upward propagation stops at this node. At each stage of upward propagation, changes are propagated downward to other visible nodes. Thus, first the parent view of the last node is changed, and then in the downward propagation, the “sister” view of the last node is changed. The next upward propagation changes the grandfather's view, and the second downward propagation changes the first uncle and then the first cousin. Downward propagation, as before, is stopped when the area of the “cousin node” is smaller than minimumArea or when a node is completely offscreen.

  The technique described above involves converting the layout of the various nodes into a tree, which is conceptually illustrated in FIGS. As can be seen from FIGS. 21 and 22, there is a corresponding tree of a particular displayed set of nodes, and this tree structure can be used to propagate view information as previously described. Each of nodes 401 to 408 corresponds to a point having the same symbol in the tree shown in FIG. Conceptually shown in FIG. 20 is information that should be stored for a node, such as, for example, a pointer, information indicating the size and position of another node. FIG. 23 is a block diagram corresponding to a portion of the tree diagram of FIG. 22 where exemplary values are used to indicate data that can be used to define the properties of the node 403. In this exemplary case, node 403 has a pointer to child node 406, information regarding the logical size (100 × 100) of child node 406, and data specifying the position of child node 406 in the coordinates of node 403, ie 10 , 20.

  A pan operation may move the last node far enough that the last node no longer belongs to viewStack. Instead, zooming in may expand the child to the extent that it is necessary to extend viewStack, or zooming out will make the area of the last node less than the minimum area and require viewStack truncation. There is a case. In all of these cases, the identity of the last node changes. These situations are detected during downward propagation, which changes the viewstack accordingly and potentially makes the viewstack longer or shorter.

  One simple case described above is that during zooming, a node is activated, so that it needs to be placed on the view stack. Another example is that zooming out makes previously visible nodes so small that they must be removed from the view stack.

An extension of this idea is to avoid truncating the immediate viewStack in response to long outward zooms. The viewstack truncation is necessary only when the user subsequently pans. A long outward zoom makes the view field of a deeply zoomed node very large (and therefore numerically inaccurate), but is a field that represents the center point of the view rectangle
Point2D viewCenter;
Can be added to the Node structure, so zoom without panning does not change the viewCenter field of any node. This configuration makes it possible to immediately continue the zoom-in back to the long outward zoom. Since the viewStack remains unchanged, the user can accurately return to the starting view. This behavior is similar to the “back” and “forward” buttons of a web browser, where “back” is similar to zooming out and “forward” is similar to zooming back. In a web browser, the user uses “back” to return to the previous web page, but it is this point that “forward” stops working when following alternative links. Thus, following an alternate link is similar to panning after zooming out.

  The foregoing provides that visual content can be displayed and navigated in a variety of ways, with virtually unlimited accuracy limited only by the capacity of the computer system on which the application is running. The visual content displayed at any given time is displayed as an assembly of nodes, where only the nodes needed for a particular view have been activated and all other nodes are in another node's Displayed as part of it, or not displayed at all. It should be understood that various other embodiments will be apparent to those skilled in the art and that the invention is not limited to the embodiments described herein.

Method and apparatus for navigating an image

[Technical field]
The present invention relates to a method and apparatus for navigating, such as zooming and panning over an image of an object, in a manner that provides a smooth and continuous navigation motion appearance.

[Background]
Most conventional graphical computer user interfaces (GUIs) are designed with a fixed spatial scale visual component, but make the visual component no longer have a fixed spatial scale on the display. It has long been recognized that it can be expressed and manipulated, and in fact, visual components can be panned and / or zoomed in or out. The ability to zoom in and out of an image, for example, to view a map, browse a text layout such as a newspaper, view a digital photo, view a blueprint or drawing, and view other large data sets Desirable in relation.

  Many existing computer applications, such as Microsoft® Word, Adobe® Photo Shop, Adobe® Acrobat, include zoomable components. In general, the zoom function provided by these computer applications is a peripheral aspect of user interaction with the software, and the zoom function is only occasionally used. These computer applications allow the user to pan smoothly and continuously over the image (eg, using a scroll bar or cursor to translate the viewed image left, right, up, or down). enable. However, an important problem with such computer applications is that these computer applications do not allow the user to zoom smoothly and continuously. In fact, these computer applications provide zoom in discrete steps such as 10%, 25%, 50%, 75%, 100%, 150%, 200%, 500%, etc. The user uses the cursor to select the desired zoom, and in response, the image unexpectedly changes to the selected zoom level.

  The undesirable quality of discontinuous zoom also exists in Internet-based computer applications. A computer application based on the website of Non-Patent Document 2 shows this point. The MapQuest website allows a user to enter one or more addresses and receive an image of a road map in response. 1-4 are examples of images that can be obtained from the MapQuest website in response to a regional map query for Long Island, New York. The MapQuest website allows users to zoom in and out to discrete levels, such as 10 levels. FIG. 24 is a rendition at zoom level 5 which is approximately 100 meters / pixel. FIG. 25 is a zoom level 6 image that is approximately 35 meters / pixel. FIG. 26 is an image at zoom level 7 which is approximately 20 meters / pixel. FIG. 27 is an image at zoom level 9 which is approximately 10 meters / pixel.

  As can be seen by comparing FIGS. 1-4, the unexpected transition between zoom levels results in the loss of sudden unexpected details when zooming out and the addition of sudden unexpected details when zooming in. For example, local roads, secondary roads, or connecting roads cannot be seen in FIG. 24 (zoom level 5), but secondary roads and connecting roads appear suddenly in FIG. 25, which is exactly the next zoom level. Such unexpected discontinuities are very unpleasant when using the MapQuest website. However, it should be noted that the results are still unsatisfactory even if the MapQuest software application is modified to allow, for example, a general road view at zoom level 5 (FIG. 24). The visual density of the map should change with the zoom level, and at some zoom level the results may be satisfactory (eg, at level 7 or FIG. 26), but when zoomed in The road should not be thick and the map should look very sparse. When zoomed out, the roads will eventually blend together and quickly form a filled population within which the individual roads will be indistinguishable.

  The ability to provide smooth and continuous zoom on road map images is problematic because of the various levels of roughness associated with road categories. In the United States, there are about five categories of roads (according to classification under TIGER / Line Data distributed by the US Census Bureau), A1 or primary road, A2 or primary road, A3 or state road, secondary road, And connecting roads, A4 or ordinary roads, city streets and country roads, and A5 or unpaved roads. These roads can be thought of as elements of the overall object (ie road map). The roughness of the road element appears because there are significantly more A4 roads than A3 roads, considerably more A3 roads than A2 roads, and much more A2 roads than A1 roads. Furthermore, the physical dimensions of the road (eg, its width) vary greatly. The A1 road may be about 16 meters wide, the A2 road may be about 12 meters wide, the A3 road may be about 8 meters wide, and the A4 road is about 5 meters wide And the A5 road may be about 2.5 meters wide.

  The MapQuest computer application handles this varying level of roughness by displaying only those road categories that may be appropriate at a particular zoom level. For example, a country-wide view can show only A1 roads, a state-wide view can show A1 and A2 roads, and a county-wide view can show A1, A2, and A3 roads. Can show. Even when MapQuest is modified to allow continuous zooming of the road map, this approach leads to the sudden appearance and disappearance of the road category being zoomed, which is confusing, visual Is unpleasant.

[Patent Document 2] US Provisional Patent Application No. 60/475897
[Patent Document 3] US Provisional Patent Application No. 60/453897
[Patent Document 4] US Provisional Patent Application No. 60/553803
[Non-Patent Document 2] www.mapquest.com
[Disclosure of the Invention]
[Problems to be solved by the invention]
In view of the foregoing, the art has complexities that preserve the visual distinction between elements based on the size or importance of the elements of the object, while at the same time allowing smooth and continuous zooming of the image. There is a need for new methods and apparatus for navigating images of complex objects.

[Means for solving problems]
In accordance with one or more aspects of the present invention, a method and apparatus for performing various actions including zooming in or out of an image having at least one object is contemplated, wherein at least some of the at least one object. These elements are scaled up and / or scaled down in a manner that is non-physically proportional to one or more zoom levels associated with zoom.

Non physically proportional scaling can be represented by the following formula ^{p = c · d · z a} , p is an linear size in pixels of one or more elements of the object at that zoom level , C are constants, d is the linear size in physical units of one or more elements of the object, z is the zoom level in physical linear size / pixel units, and a is the scale Should be a ≠ -1.

  Under non-physical scaling, a to be scaled is not equal to -1 within the zoom levels z0 and z1 (usually -1 <a <0), and z0 is a physical linear size / less than z1. Has pixels. Preferably, at least one of z0 and z1 can vary for one or more elements of the object. Note that a, c, and d can also vary from element to element.

  At least some elements of the at least one object may also be scaled up and / or scaled down in a manner that is physically proportional to one or more zoom levels associated with zoom. The physically proportional scaling can be represented by the following formula p = c · d / z, where p is the linear size in pixels of one or more elements of the object at that zoom level, and c Is a constant, d is the linear size in physical units of one or more elements of the object, and z is the physical linear size / zoom level in pixels.

  Any known processor, programmable digital device or method or apparatus described above and / or described later in this document that operates to execute standard digital circuits, analog circuits, software and / or firmware programs Note that any known technology can be used, such as a programmable digital system, a programmable array logic device, or any combination of the above. The invention can also be implemented in software programs for storage on suitable storage media and execution by processing units.

  The elements of the object can have varying degrees of roughness. For example, as described above, the roughness of the elements of the road map object is such that there are considerably more A4 roads than A3 roads, much more A3 roads than A2 roads, and much more A2 roads than A1 roads. It appears because there is. The degree of road category roughness also appears in properties such as average road length, intersection frequency, and maximum curvature. The roughness of the elements of other image objects may appear in other ways that are too many to list in their entirety. Thus, the scaling of an element in a given given image is physically based on at least one of (i) the degree of roughness of the element and (ii) the zoom level of the given given image. Or it may be non-physically proportional. For example, the object can be a road map, the object element can be a road, and the degree of change can be a road hierarchy. Thus, the scaling of a given road within a given given image can be physically or non-physically based on (i) the road hierarchy of the given road and (ii) the zoom level of the given given image. Can be proportional to each other.

  According to one or more further aspects of the present invention, receiving a plurality of pre-rendered images of varying zoom levels of a road map at a client terminal, one or more user navigation commands including zoom information. Pre-rendering to obtain the intermediate image so that the client terminal receives and displays the intermediate image at the intermediate zoom level corresponding to the zoom information of the navigation command on the client terminal provides a smooth navigation appearance Methods and apparatus are contemplated that perform various actions including blending two or more of the rendered images.

  According to one or more further aspects of the present invention, receiving at a client terminal a plurality of pre-rendered images of varying zoom levels of at least one object, wherein at least some of the at least one object Are scaled up and / or scaled down to produce a plurality of predetermined images, the scaling being (i) physically proportional to the zoom level and (ii) non-physically proportional to the zoom level. Receiving at least one of the following: receiving at the client terminal one or more user navigation commands including zoom information; and an intermediate image at an intermediate zoom level corresponding to the zoom information of the navigation command Pre-render to get Performing the method comprising blending two or more of the grayed by image, the various actions and displaying the intermediate image at the client terminal, method and apparatus are contemplated.

  According to one or more further aspects of the present invention, transmitting a plurality of pre-rendered images of varying zoom levels of a road map to a client terminal over a communication channel, and a plurality of pre-rendered images At the client terminal, using the client terminal to issue one or more user navigation commands including zoom information, and an intermediate image client terminal at an intermediate zoom level corresponding to the zoom information of the navigation command A method and apparatus for performing various actions including blending two or more of the pre-rendered images to obtain an intermediate image so that the display on the screen provides a smooth navigation appearance Is contemplated.

  According to one or more further aspects of the present invention, transmitting a plurality of pre-rendered images of varying zoom levels of at least one object to a client terminal via a communication channel, the method comprising: At least some elements of the object are scaled up and / or scaled down to create a plurality of predetermined images, the scaling being (i) physically proportional to the zoom level and (ii) to the zoom level Transmitting at least one of non-physically proportional; receiving a plurality of pre-rendered images at a client terminal; and using the client terminal to include zoom information Issuing multiple user navigation commands and navigating Various methods including blending two of the pre-rendered images and displaying the intermediate image at the client terminal to obtain an intermediate image at an intermediate zoom level corresponding to the zoom information of the navigation command Methods and apparatus for performing various actions are contemplated.

  Other aspects, features, and benefits of the present invention will become apparent to those skilled in the art when interpreting the description herein in conjunction with the accompanying drawings.

  For the purpose of illustrating the invention, there are shown in the drawings forms that are presently preferred, but it is to be understood that the invention is not limited to the precise arrangements and instrumentalities shown.

[Best Mode for Carrying Out the Invention]
Referring now to the drawings, where like symbols indicate like elements, FIGS. 5-11 show a series of images representing the road network of Long Island, New York, each image being Different zoom levels (or resolutions). Before exploring the technical details of how the present invention is implemented, these images should be smoothed out while maintaining the desired characteristics of the results of using the present invention, ie, at least while maintaining information integrity. We will now discuss this in relation to continuous navigation, especially the appearance of zooming.

  Note that the various aspects of the invention described below can be applied in contexts other than navigation of road map images. Indeed, the range of images and implementations for which the present invention can be used is too much to list in its entirety. For example, features of the present invention can be used to navigate human anatomy, complex topography, engineering drawings such as wiring diagrams or blueprints, images such as gene ontology. However, it has been found that the present invention has particular applicability to navigating images with varying levels of detail or roughness of the elements. Accordingly, for the sake of clarity and clarity, various aspects of the present invention will be described in the context of a particular example, namely a road map image.

  While it is impossible to show a smooth and continuous zoom appearance in a patent document, this feature has been achieved through experiments and prototype development by running an appropriate software program on a Pentium®-based computer. Has been demonstrated. The road map image 100A shown in FIG. 28 is a zoom level at which features can be expressed in units of physical length / pixel (or physical line size / pixel). In other words, the zoom level z represents the actual physical line size represented by a single pixel of the image 100A. In FIG. 28, the zoom level is about 334 meters / pixel. Those skilled in the art will appreciate that zoom levels can be expressed in other units without departing from the spirit and scope of the claimed invention. FIG. 29 is an image 100B of a road map that is the same as FIG. 28 but has a zoom level z of about 191 meters / pixel.

  In accordance with one or more aspects of the present invention, a user of a software program implementing one or more aspects of the present invention zooms in or out between the levels shown in FIGS. be able to. Note that such a zoom has the appearance of a smooth continuous transition from / to the 334 meter / pixel level (FIG. 28) to the 191 meter / pixel level (FIG. 29) and to any level in between. is important. Similarly, the user has z = 109.2 meters / pixel (FIG. 30), z = 62.4 meters / pixel (FIG. 31), z = 35.7 meters / pixel (FIG. 32), z = 20.4. Other levels can be zoomed, such as meters / pixel (FIG. 33) and z = 11.7 meters / pixel (FIG. 34). Again, transitions through these zoom levels and all levels in between advantageously have a smooth and continuous motion appearance.

  Another important feature of the present invention shown in FIGS. 5-11 is that when zooming from one level to another, little or no details appear and disappear unexpectedly. The details shown in FIG. 31 (z = 62.4 meters / pixel zoom level) can also be found in FIG. 28 (334 meters / pixel zoom level). This is the case even if the image object, which in this example is a road map, includes elements with varying degrees of roughness (ie roads). In fact, the road map 100D of FIG. 31 includes at least A1 main roads such as 102, A3 secondary roads such as 104, and A4 local roads such as 106. Nevertheless, these details can still be seen in the image 100A of FIG. 28, even on the A4 local road 106, which is considerably zoomed out compared to the image 100D of FIG.

  In addition, the A4 local road 106 can be seen at a zoom level of z = 334 meters / pixel (FIG. 28), but the A1, A2, A3, and A4 roads can be distinguished from each other. Even the difference between the A1 primary road 102 and the A2 primary road 108 can be distinguished from each other with respect to the relative weights given to such roads in the rendered image 100A.

  The ability to distinguish between road hierarchies is also advantageously maintained when the user continues to zoom in to a zoom level of, for example, z = 20.4 meters / pixel shown in image 100F of FIG. The weight of the A1 primary road 102 is greatly increased compared to the zoom level of z = 62.4 meters / pixel in FIG. 31, but is so large as to obscure other details such as the A4 local road 106 or the A5 dirt road. Will not increase. Nevertheless, the weight of lower level roads such as the A4 local road 106 is greatly increased in weight compared to the counterpart at zoom level z = 62.4 meters / pixel in FIG.

  Thus, the dynamic range of the zoom level between that shown in FIG. 28 and that shown in FIG. 34 is substantial and details remain substantially consistent (ie, the road is The information that the user seeks to obtain at a given zoom level is not obscured by unwanted artifacts. For example, at a zoom level of z = 334 meters / pixel (FIG. 28), the user may want to get a general feeling of which primary roads exist and in which direction they extend. This information can be easily obtained even when the A4 local road 106 is also displayed. At a zoom level of z = 62.4 meters / pixel (FIG. 31), the user decides whether a particular A1 primary primary road 102 or A2 primary road 108 is beneficial to or in a particular city. May want. Again, the user can obtain this information without interference from other much more detailed information, such as the presence and spread of A4 local road 106 or A5 dirt road. Finally, at a zoom level of 11.7 meters / pixel, the user may be interested in finding a particular A4 local road, such as 112, without interference from a significantly larger road, such as the A1 primary primary road 102. Can do it.

  One or more software that causes one or more computing devices to perform appropriate actions to accomplish one or more of the various aspects of the invention described above. It is intended to execute the program. In this regard, reference is now made to FIGS. 12-13, which are flowcharts illustrating the process steps preferably performed by one or more computing devices and / or associated equipment.

  This process flow is preferably performed by a commercially available computing device (such as a Pentium®-based computer), but the spirit and scope of the claimed invention can be any of a number of other techniques. Can be used to perform these process steps without departing from the above. In practice, the hardware used is one or more of standard digital circuits, analog circuits, all known processors that operate to execute software and / or firmware programs, programmable read only memory (PROM), etc. Can be implemented utilizing any other known or below-developed technology, such as a programmable digital device or programmable digital system, a programmable array logic device (PAL), or any combination of the above. Furthermore, the method of the present invention can be implemented in a software program that can be stored on either known or underdeveloped media.

  FIG. 35 illustrates an embodiment of the present invention in which multiple images are prepared in action 200 (each at a different zoom level or resolution), and two or more of these images are combined together. Blended to achieve a smooth navigational appearance such as zoom (action 206). Although not necessary to practice the invention, it is contemplated that the approach shown in FIG. 35 is used in connection with a service provider-client relationship. For example, the service provider spends resources to prepare multiple pre-rendered images (action 200) and make these images available to the user's client terminal via a communication channel such as the Internet (action 202). Alternatively, the pre-rendered image can be an integral or related part of the application program that the user loads onto the user's computer and executes there.

  Through experiments, it has been found that when a blending approach is used, the following set of zoom level images works well when the image object is a road map: 30 meters / pixel, 50 meters / pixel, 75 Meter / pixel, 100 meter / pixel, 200 meter / pixel, 300 meter / pixel, 500 meter / pixel, 1000 meter / pixel, and 3000 meter / pixel. However, it should be noted that any number of images can be used at any number of resolutions without departing from the scope of the present invention. In fact, other image objects in other contexts may be best served by more or fewer images when the particular zoom level is different from the above example.

  Regardless of how the image is obtained by the client terminal, in response to a user initiated navigation command, such as a zoom command (action 204), the client terminal displays an intermediate resolution image that matches the navigation command. It is preferred to operate to blend two or more images to create (action 206). This blending can be accomplished by a number of methods, such as the well-known trilinear interpolation technique described in Non-Patent Document 1, the entire disclosure of which is incorporated herein by reference. Other techniques for image interpolation, such as bicubic linear interpolation, are also useful in connection with the present invention, and other techniques may be developed in the future. It should be noted that the present invention does not require or depend on any particular one of these blending methods. For example, as shown in FIG. 31, the user may wish to navigate to a zoom level of 62.4 meters / pixel. This zoom level can be between two of the pre-rendered images (in this example, between a zoom level of 50 meters / pixel and a zoom level of 75 meters / pixel), so 62.4 meters The desired zoom level of / pixel can be achieved using trilinear interpolation techniques. Further, all zoom levels between 50 meters / pixel and 75 meters / pixel can be obtained using the blending technique described above, which, when performed, is smooth and smooth. Provide a continuous navigation look fast enough. This blending technique can be accomplished to other zoom levels, such as 35.7 meters / pixel shown in FIG. In that case, this blending technique can be performed between the 30 meter / pixel and 50 meter / pixel pre-rendered images of the examples described so far.

  The above blending approach is that the computing power of the processing unit in which the present invention is implemented is: (i) first to perform the rendering operation and / or (ii) high image for smooth navigation. It can be used when image rendering is not high enough to perform “just in time” or “on the fly” (eg, in real time) to achieve the frame rate. However, as described below, other embodiments of the present invention provide for high performance processing units known or developed below that are capable of rendering at client terminals for blending and / or high frame rate applications. Intended for use.

  The process flow of FIG. 36 illustrates the detailed steps and / or actions that are preferably performed to prepare one or more images in accordance with the present invention. At action 220, information about one or more image objects is obtained using any of the known or developed techniques below. Typically, such image objects are modeled using appropriate primitives such as polygons, lines, points, etc. For example, when the image object is a road map, it is possible to easily obtain models of roads in all Universal Mercator (UTM) zones. This model is usually in the form of a list of line segments (in an arbitrary coordinate system) that includes roads in the zone. This list is either a known or under-developed rendering process determines the weight (eg, apparent thickness or actual thickness) of a given primitive in a pixel (spatial) region As long as it incorporates some sort of technique, it can be converted to an image (pixel image) in the spatial domain using its rendering process. In accordance with the road map example above, this rendering process must incorporate some sort of technique that determines the line weights that model the roads of the road map within the spatial domain. These techniques are described below.

  In action 222 (FIG. 36), the elements of the object are classified. In the case of road map objects, this classification can take the form of recognizing categories that already exist, namely A1, A2, A3, A4, and A5. In fact, these road elements have varying degrees of roughness and can be rendered differently based on this classification, as described below. Action 224 applies mathematical scaling to different road elements based on the zoom level. As described in more detail below, this mathematical scaling can also vary based on element classification.

  As background, there are two conventional techniques for rendering image elements such as map roads: real physical scaling and preset pixel width. Real physical scaling techniques specify that a road map is rendered as if it were looking at the actual physical image of the road at different scales. For example, the A1 main road may be 16 meters wide, the A2 road may be 12 meters wide, the A3 road may be 8 meters wide, and the A4 road may be 5 meters wide. It may be metric wide and the A5 road may be 2.5 meters wide. This may be acceptable to the viewer when zoomed in on a small area of the map, but when zoomed out, all roads, both major and non-important, are detailed Too thin. At some zoom level, for example the state level (eg about 200 meters / pixel), the road should not be seen at all.

  The preset pixel width approach specifies that all roads are a certain pixel width, such as one pixel wide on the display. Major roads such as main roads can be highlighted by making them 2 pixels wide or the like. Unfortunately, this approach changes the visual density of the map as a person zooms in and out. The result may be satisfactory at some zoom level, for example, at a small size county level. However, when zooming in, the road does not get thicker and the map becomes overly sparse. In addition, when zooming out, the roads will mix with each other and quickly form a filled population within which the individual roads will be indistinguishable.

  In accordance with one or more aspects of the present invention, at action 224, the image is (i) physically proportional to the zoom level, at least some image elements depending on the parameters described in more detail below. Or (ii) scaled up and / or scaled down either non-physically proportional to the zoom level.

A scaling level “physically proportional to the zoom level” means that the size of the element should appear to change with distance from the human eye, so the number of pixels representing the road width increases with the zoom level, or Note that it means to decrease. The perspective formula that gives the apparent length y of an object of physical size d is
y = c · d / x
Where c is a constant that determines the angled perspective and x is the distance of the object from the viewer.

In the present invention, the line size p in units of display pixels of the object of the physical line size d ′ is
p = d ′ · z ^a
Where z is the zoom level in physical line size / pixel (eg, meters / pixel) and a is the power law. When a = −1 and d ′ = d (the actual physical line size of the object), this equation is dimensionally correct and is equivalent to the perspective formula, p = y and z = x / c is there. This represents the equivalent between physical zoom and perspective transformation, zooming in is equivalent to moving the object closer to the viewer, and zooming out is equivalent to moving the object further away. is there.

  In order to implement non-physical scaling, a power law other than −1 can be set to a, and a physical line size other than the actual physical line size d can be set to d ′. In the context of a road map, p can represent the displayed width of the road in pixels, and d ′ can represent the resulting width in physical units, but “non-physically proportional to the zoom level. “Do” means that the road width in display pixels increases or decreases with zoom level in a manner other than being physically proportional to the zoom level, ie, a ≠ −1. Scaling is distorted in a way that achieves some desired result.

Note that "straight line size" means a one-dimensional size. For example, when a two-dimensional object is considered and its “straight line size” is doubled, the area is multiplied by 4 = 2 ² . In the two-dimensional case, the linear size of an object's elements can include length, width, radius, diameter, and / or any other measurement that can be read using a ruler on the Euclidean plane. Line thickness, line length, circle or disk diameter, polygon side length, and distance between two points are all examples of a straight line size. In this sense, “linear size” in two dimensions is the distance between two identified points of an object on a 2D Euclidean plane. For example, the line size can be calculated by taking the square root of (dx ² + dy ² ), where dx = x1-x0, dy = y1-y0 and the two identified points are Cartesian Given by the coordinates (x0, y0) and (x1, y1).

The concept of “straight line size” naturally extends to more than two dimensions, for example, when considering volumetric objects, doubling the straight line size increases the volume by 8 = 2 ³ times. With that. Similar measurements of line size can also be defined for non-Euclidean spaces, such as the surface of a sphere.

  Every power law a <0 causes the rendered size of an element to decrease when zooming out and increase when zooming in. When a <−1, the rendered size of the element decreases more quickly than proportional physical scaling when zooming out. Conversely, when -1 <a <0, the size of the rendered element decreases more slowly than proportional physical scaling when zooming out.

  In accordance with at least one aspect of the present invention, p (z) is allowed to be substantially continuous for a given length of a given object, so that during navigation, the user can No longer experience sudden jumps or discontinuities in the size of the elements of the image (unlike conventional approaches that allow the most extreme discontinuities, ie, the sudden appearance or disappearance of elements during navigation). Furthermore, p (z) decreases monotonically with zoom-out so that zoom-out makes the object's elements smaller (eg, the road becomes thinner) and zoom-in makes the object's elements larger. Is preferred. This gives the user a sense of physicality about the object of the image.

The scaling features described above can be more fully understood with respect to FIG. 37, which is a log-log graph of pixel-rendered line width of A1 main road vs. zoom level in meters / pixel. (Plotting log (z) on the x-axis and log (p) on the y-axis is convenient because the plot is a straight line due to the relationship log (x ^a ) = a · log (x)). The basic characteristics of a line (road) width versus zoom level plot are as follows.

(I) Road width scaling can be physically proportional to the zoom level when zooming in (eg, up to about 0.5 meters / pixel);
(Ii) Road width scaling can be non-physically proportional to the zoom level when zooming out (eg, up to about 0.5 meters / pixel);
(Iii) Road width scaling shall be physically proportional to the zoom level when further zooming out (eg, greater than about 50 meters / pixel or more, depending on the parameters described in more detail below). Can do.

Respect zone road width scaling is physically proportional to the zoom level, p = d '· z ^a scaling formulas are used that, where it is a = -1. In this example, a reasonable value for the physical width of the actual A1 main road is about d ′ = 16 meters. Thus, the rendered width of the line representing the A1 main road decreases monotonically with physical scaling when zooming out to at least some zoom level z0, eg, z0 = 0.5 meters / pixel.

  The zoom level for z0 = 0.5 is selected to be the inner scale below which physical scaling is applied. This prevents the non-physical appearance when the road map is combined with other finer scale GIS content with actual physical dimensions. In this example, z0 = 0.5 meters / pixel, or 2 pixels / meter, when expressed as a map scale on a 15 inch (38.1 centimeter) display (with 1600 × 1200 pixel resolution) 1: corresponds to a scale of 2600. At d = 16 meters, which is a reasonable actual physical width of the A1 road, the rendered road appears to be its actual size when zoomed in (less than 0.5 meters / pixel). At a zoom level of 0.1 meters / pixel, the rendered line is approximately 160 pixels wide. At a zoom level of 0.5 meters / pixel, the rendered line is approximately 32 pixels wide.

Road width scaling is with respect to the zone of non-physically proportional to the zoom level, p = d '· z ^a scaling formula is used that is a -1 <a <0 (in the range of zoom level z0 and z1). In this example, non-physical scaling is performed between approximately z0 = 0.5 meters / pixel and z1 = 3300 meters / pixel. Again, when -1 <a <0, the width of the rendered road decreases more slowly when zooming out than with proportional physical scaling. Advantageously, this allows the A1 road to remain visible (and distinguishable from other smaller roads) when zoomed out. For example, as shown in FIG. 28, A1 road 102 is visible at a zoom level of z = 334 meters / pixel and remains distinguishable from other roads. Assuming that the physical width of the A1 road is d ′ = d = 16 meters, the width of the line rendered using physical scaling is about 0.005 at a zoom level of about 3300 meters / pixel. Pixels and should be rendered virtually invisible. However, using non-physical scaling, when -1 <a <0 (in this example, a is approximately -0.473), the width of the rendered line is a zoom level of 3300 meters / pixel. Is approximately 0.8 pixels and is rendered clearly visible.

  Note that the value of z1 is chosen so that a given road is the most zoomed out scale that still has a “higher than physical importance” importance. For example, if the entire United States is rendered on a 1600 × 1200 pixel display, this resolution should be about 3300 meters / pixel or 3.3 kilometers / pixel. When looking at the world as a whole, there may be no reason why major US roads have an increased importance relative to the country-only view.

Thus, at zoom levels above z1, which is about 3300 meters / pixel in the above example, the road width scaling is still physically proportional to the zoom level, but preferably due to the continuity of p (z) Has a large d ′ (much larger than the actual width d). In this zone, the scaling formula p = d ′ · z ^a is used, where a = −1. In order for the rendered road width to be continuous at z1 = 3300 meters / pixel, a new attributed physical width of the A1 main road, for example d ′ = 1.65 km, is selected. The new values of z1 and d ′ are preferably chosen so that the rendered width of the line is a reasonable number of pixels at the outer scale z1. In this case, at a zoom level (about 3300 meters / pixel) where the whole country can be seen on the display, the A1 road can be about 1/2 pixel wide, which is thin but still clear. Visible, which corresponds to a resulting physical road width of 1650 meters or 1.65 kilometers.

  The above implies a specific set of equations for the rendered line width as a function of zoom level:

When p (z) = d0 · z ⁻¹ and z ≦ z0 When p (z) = d1 · z ^a and z0 <z <z1 When p (z) = d2 · z ⁻¹ and z ≧ z1 p The upper form of (z) has six parameters: z0, z1, d0, d1, d2, and a. z0 and z1 mark the scale on which the behavior of p (z) changes. In the zoomed-in zone (z ≦ z0), the zoom is physical (ie, the index of z is −1) and has a physical width of d0, which is equal to the actual physical width d. It is preferable to correspond. In the zoomed out zone (z ≧ z1), the zoom is still physical but has a physical width of d1, which generally does not correspond to d. Between z0 and z1, the rendered line width is scaled using the power law of a, which can be a value other than -1. Given the preference that p (z) is continuous, specifying z0, z1, d0, and d2 uniquely determines d1 and a, as clearly shown in FIG. Enough to do.

  The technique described above for the A1 road can be applied to other road elements of the road map object. An example of the application of this scaling technique to A1, A2, A3, A4, and A5 roads is shown in the log-log graph of FIG. In this example, z0 = 0.5 meters / pixel for all roads, but this value can vary from element to element depending on context. Since the A2 road is generally somewhat narrower than the A1 road, d0 = 12 meters. In addition, A2 roads are “important”, eg, at the US state level, and thus z1 = 312 meters / pixel, which is roughly the rendering resolution for a single state (approximately 1 of the country on a linear scale). / 10). At this scale, a line width of 1 pixel has been found desirable, so d2 = 312 meters is a reasonable setting.

  Using the general approach outlined above for the A1 and A2 roads, the parameters of the remaining elements of the road map object can be established. For the A3 road, d0 = 8 meters, z0 = 0.5 meters / pixel, z1 = 50 meters / pixel, and d2 = 100 meters. For the A4 street, d0 = 5 meters, z0 = 0.5 meters / pixel, z1 = 20 meters / pixel, and d2 = 20 meters. For A5 unpaved roads, d0 = 2.5 meters, z0 = 0.5 meters / pixel, z1 = 20 meters / pixel, and d2 = 20 m. Note that using these parameter settings, the A5 dirt road looks more like a street at the zoomed out zoom level, but the physical scale when zoomed in is half as wide. .

  The logarithmic plot of FIG. 38 summarizes the scaling behavior for road types. Note the apparent width of A1> A2> A3> A4> = A5 at all scales. Also note that all of the power laws, with the exception of dirt roads, appear in the vicinity of a = −0.41. All of the dashed lines have a slope of −1 and represent physical scaling with different physical widths. From top to bottom, the corresponding physical widths of these dashed lines are 1.65 kilometers, 312 meters, 100 meters, 20 meters, 16 meters, 12 meters, 8 meters, 5 meters, and 2.5 meters.

  When interpolation between multiple pre-rendered images is used, in many cases the resulting interpolation will be all lines of the correct pixel width determined by physical and non-physical scaling equations Or it is possible to ensure that humans are indistinguishable or nearly indistinguishable from ideal renditions of other primitive geometric elements. In order to appreciate this alternative embodiment of the present invention, some background regarding anti-aliased line drawings is presented below.

  The discussion of anti-aliased line art is presented according to the example of a road map described at length above, where all primitive elements are lines and line widths are described earlier. As governed by the scaling equation. Referring to FIG. 39A, a one pixel wide vertical line drawn in black on a white background and the horizontal position of the line precisely aligned to the pixel grid is simply black on a white background. It consists of a 1 pixel wide column of pixels. In accordance with various aspects of the present invention, it is desirable to address and address the case where the line width is a non-integer number of pixels. Referring to FIG. 39B, the left and right pixel columns in the center column on an anti-aliased graphics display when the line endpoint remains fixed but the line weight is increased to 1.5 pixel width. Is drawn in 25% gray. Referring to FIG. 39C, at 2 pixels wide, the columns located on these sides are drawn in 50% gray. Referring to FIG. 39D, at 3 pixels wide, the columns located on these sides are drawn in 100% black, and the result is three filled black columns as expected.

  This technique of drawing non-integer width lines on a pixelated display provides a visual continuity sensation (or illusion) when the line width changes, with different width lines being a fraction of one pixel in width Make sure that the lines are clearly distinguishable, even if they are different. In general, this technique is known as anti-aliased line drawing, but it is a line of luminance function perpendicular to the line drawn (or “1-luminance” function for black lines on a white background). Designed to ensure that the integral is equal to the line width. This method is easily generalized to lines whose endpoints are not exactly in the center of the pixel, lines with orientations other than vertical, and curves.

  Drawing the anti-aliased vertical lines of FIGS. 16A-D is accomplished by alpha blending of two images where one (image A) is one pixel wide and the other (image B) is three pixels wide. Note that you can also. Alpha blending assigns (1-alpha) * (corresponding pixel in image A) + alpha * (corresponding pixel in image B) to each pixel on the display. When alpha changes between 0 and 1, the effective width of the rendered line changes smoothly between 1 and 3 pixels. This alpha blending technique produces a visual result that may be limited to the most common case where the difference between the two rendered line widths in images A and B is less than or equal to one pixel, otherwise, Lines may appear behind the light with an intermediate width. This same approach can be applied to the rendering of points, polygons, and many other primitive geometric elements of different line sizes.

  Turning again to FIGS. 16A-D, the 1.5 pixel wide line (FIG. 39B) and the 2 pixel wide line (FIG. 39C) are the 1 pixel wide line (FIG. 39A) and the 3 pixel wide line (FIG. 39D). ) Between and alpha blending. Referring to FIGS. 17A-C, a 1 pixel wide line (FIG. 40A), a 2 pixel wide line (FIG. 40B), and a 3 pixel wide line (FIG. 40C) are shown in any orientation. The same principle as when the lines are precisely aligned to the pixel grid applies to any orientation of FIGS. 17A-C, but the spacing of the line widths between which alpha blending takes place is a good result. Therefore, it may be necessary to make the pixel finer than 2 pixels.

  In the context of the current map example, a set of images with different resolutions can be selected for pre-rendition with respect to the log-log plot of FIGS. For example, referring now to FIG. 41, FIG. 41 is substantially similar to FIG. 37 except that it includes a set of horizontal and vertical lines. A horizontal line indicates a line width between 1 and 10 pixels in 1 pixel increments. The vertical lines are spaced so that the line width across the interval between two adjacent vertical lines does not change more than two pixels. Thus, a vertical line represents a set of zoom values appropriate for pre-rendition, and alpha blending between two adjacent such pre-rendered images is a continuously variable line of road representing a road. Produces almost the same characteristics as rendering.

  Interpolation between the six resolutions represented by the vertical lines shown in FIG. 41 is sufficient to accurately render the A1 main road using the illustrated scaling curve above about 9 meters / pixel. A rendition of less than about 9 meters / pixel would render such a vector into a vector without the need for a prior rendition, as such views are very zoomed in and thus show very few roads. Make it computationally more efficient (and more efficient with respect to data storage requirements) than interpolation between pre-rendered images. At resolutions above about 1000 meters / pixel (such views include large portions of the Earth's surface), only the final pre-rendered image can be used. This is because this is a rendition using a 1 pixel wide line. Lines thinner than a single pixel render the same pixel faintly. Thus, to create an image where the A1 line is 0.5 pixels wide, an image of a 1 pixel wide line can be multiplied by 0.5 alpha.

  In practice, over each interval between resolutions, a slightly larger set of resolutions is pre-rendered so that all of the scaling curves in FIG. 38 do not change more than one pixel. Reducing the allowed variation to 1 pixel can result in improved rendering quality. Notably, the co-pending patent document 1 filed on March 1, 2004 entitled “SYSTEM AND METHOD FOR EXACT RENDERING IN A ZOOMING USER INTERFACE,” the entire disclosure of which is incorporated herein by reference. The tiling technique contemplated and described in (Attorney Docket No. 489/2) can be considered in connection with the present invention. This tiling technique can be used to resolve an image at that level, even if a particular zoom level does not match the pre-rendered image. When each image in a slightly larger set of resolutions is pre-rendered and tiled at the appropriate resolution, the result is that all lines are in width according to the scaling equation disclosed herein. It is a complete system for navigation that zooms and pans through arbitrarily complex road maps that appear to change continuously.

  Additional details regarding other techniques for blending images that can be used in connection with implementations of the present invention are described in “SYSTEM AND METHOD FOR THE EFFICIENT, DYNAMIC”, the entire disclosure of which is incorporated herein by reference. It can be found in Patent Document 2 filed June 5, 2003 entitled “AND CONTINUOUS DISPLAY OF MULTI RESOLUTIONAL VISUAL DATA”. For further details regarding blending techniques that can be used in connection with the implementation of the present invention, see “SYSTEM AND METHODFOR FOVEATED, SEAMLESS,” filed Mar. 12, 2003, the entire disclosure of which is incorporated herein by reference. It can be found in Patent Document 3 entitled “PROGRESSIVE RENDERING IN A ZOOMING USER INTERFACE”.

  Advantageously, using the above-described aspects of the present invention, the user enjoys a smooth and continuous navigation look through various zoom levels. Furthermore, little or no detail appears or disappears unexpectedly when zooming from one level to another. This represents a significant advancement over the technical state of the art.

  It is contemplated that various aspects of the present invention can be applied in numerous products, such as interactive software applications on the Internet, automotive-based software applications, and the like. For example, the present invention can be used by an Internet website that supplies maps and driving directions to client terminals in response to user requests. Alternatively, various aspects of the invention can be used in a GPS navigation system in an automobile. The present invention can also be incorporated into medical imaging equipment, whereby detailed information relating to, for example, the patient's circulatory system, nervous system, etc. can be rendered and navigated as described above. While many applications of the present invention are too numerous to list in their entirety, those skilled in the art will appreciate that they are contemplated herein and are within the scope of the claimed invention.

  The present invention can also be used in connection with other applications in which rendered images provide a means to promote and otherwise facilitate commercial transactions. Additional details regarding these aspects and uses of the present invention can be found in US Pat. (Attorney reference number 489/7).

  Although the invention herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present invention. Accordingly, many modifications can be made to these exemplary embodiments and other arrangements can be devised without departing from the spirit and scope of the invention as defined by the appended claims. I want you to understand.

[Title of Invention] Efficient data cache

[Background]
“MRU caching” (MRU stands for “most recently used”) is a well-known concept for the implementation of client-side memory in a client-server system. Assume that the server can access the client and supply a large number of data objects to the client, and these data objects can occupy a large amount of memory in a cluster. However, the available bandwidth between the client and server is limited, and therefore client requests that require data objects to be sent from the server are time consuming. MRU caching is client-server when access to data objects is reasonably “coherent” (meaning that an object that a client recently needed is likely to be needed again in the near future). The efficiency of the system can be increased. Using this approach, the client typically keeps a limited amount of memory (generally much less than what is needed to store all of the objects on the server), and in this memory (cache), It stores as many of the most recently requested objects as possible. When a new object is sent from the server to the client and the client's cache space is exhausted, the least recently used (LRU) object is purged from the cache, creating a data storage space where the new object can be stored. create.

  In general, when a client needs a data object, the cache is first checked to see if the object is cached. When cached, the cached representation is used, eliminating the need for slow or computationally expensive server requests. Normally, using a cached representation also “promotes” the object to the MRU end of the cache. This approach generally provides substantial performance benefits over having to request data from the server for every data object accessed.

  Erasing the least recently used object from the cache when a new object is accessed by the computing system and stored in the cache can cause inefficiencies in cache usage. Even the longest unused object to be erased in the cache may be requested again by the server. When this occurs, the server may engage in a relatively slow or computationally expensive task of retrieving this object from a more remote source of data storage such as main memory or mass storage device. Given the finite size of the cache memory, object erasure can occur at a certain frequency, which allows data that was once conveniently stored in the cache memory to a server or other computing system. Access to more remote memory to get more resources.

[Disclosure of the Invention]
[Problems to be solved by the invention]
Thus, there is a need in the prior art for a more efficient and flexible approach to cache memory management.

[Means for solving problems]
According to one aspect, the present invention provides a cache in a computing system having an initial group of cache objects, each cache object having an initial compression ratio and including stored data; A method comprising reducing the amount of data storage space occupied by at least one cache object other than a given cache object and increasing the amount of data storage space occupied by a given cache object Can be provided. Preferably, reducing includes reducing the amount of data storage space by a given amount. Preferably, increasing includes increasing the amount of data storage space occupied by a given cache object by a given amount. Preferably, the reducing includes increasing the initial compression ratio of the at least one cache object. Preferably, increasing includes decreasing the initial compression ratio of a given cache object. Preferably, a given cache object is the most recently used cache object among the cache objects. Preferably, the at least one cache object that undergoes the reducing step includes a cache object that has been least recently used among the cache objects.

  Preferably, the reducing includes removing a portion of the stored data for the at least one cache object. Preferably, increasing includes augmenting stored data for a given cache object. Preferably, the amount of data storage space available for each of the cache objects may be equal to one of a finite number of discrete values. Preferably, reducing includes reducing the amount of data storage space for at least one randomly selected cache object other than a given cache object among the cache objects. Preferably, reducing includes reducing the amount of data storage space for the at least one randomly selected cache object to a randomly determined range. Preferably, the randomly selected cache object is selected using one of a random method and a pseudo-random method. Preferably, the selection of a randomly selected cache object is guided by a heuristic.

  Preferably, the method further includes storing the given cache object in a lossless compressed form after incrementing. Preferably, the method further includes storing the given cache object in uncompressed form after incrementing. Preferably, the reducing comprises removing at least one cache object other than the given cache object among the cache objects.

  According to another aspect, the present invention provides a computing system having at least one processor functionally communicable with main memory and a cache in a computing system having an initial group of cache objects, wherein each cache The object has an initial compression ratio and contains stored data, and the computing system reduces the amount of data storage space occupied by at least one cache object other than the given cache object, and the given cache object An apparatus can be provided that operates to increase the amount of data storage space occupied by the. Preferably, reducing includes reducing the amount of data storage space by a given amount. Preferably, increasing includes increasing the amount of data storage space occupied by a given cache object by a given amount. Preferably, the reducing includes increasing the initial compression ratio of the at least one cache object. Preferably, increasing includes decreasing the initial compression ratio of a given cache object.

  Preferably, a given cache object is the most recently used cache object among the cache objects. Preferably, the reducing includes removing a portion of the stored data for the at least one cache object. Preferably, increasing includes augmenting stored data for a given cache object. Preferably, the amount of data storage space available for each of the cache objects may be equal to one of a finite number of discrete values. Preferably, reducing includes reducing the amount of data storage space for at least one randomly selected cache object other than a given cache object among the cache objects. Preferably, reducing includes reducing the amount of data storage space for the at least one randomly selected cache object to a randomly determined range.

  According to another aspect, the present invention provides a cache in a computing system, the cache having an initial condition, and insufficient data storage space is in the cache under the initial condition. If present and storing at least one new object in the cache, compressing at least one object in the cache to clear the data storage space for the at least one new object; and at least one Storing a new object in a cache. Preferably, the initial condition corresponds to the cache being empty. Preferably, this method stores new objects in the cache without compressing the objects stored in the cache until insufficient data storage space remains in the cache to store any additional new objects. Further comprising continuing to do. Preferably, the method stores at least one new object in the cache without compression if sufficient space exists in the cache under initial conditions for storing at least one new object. Further included.

  Other aspects, features, benefits, etc. will become apparent to those skilled in the art when the description of the preferred embodiments of the invention herein is taken in conjunction with the accompanying drawings.

  For the purpose of illustrating various aspects of the invention, there are shown in the drawings embodiments that are presently preferred; it should be understood, however, that the invention is not limited to the precise arrangements and instrumentalities shown.

[Best Mode for Carrying Out the Invention]
In this disclosure, reference is made to the LRU end and the MRU end of the cache. Objects are generally added to the MRU end and are generally erased from the LRU end. However, the present invention is not limited to such a method. Note that the physical layout of the cache objects in the cache need not correspond to the LRU-MRU layout. The layout of the cache simply allows the computing system to find, insert and / or delete objects in the manner described herein. A linear LRU-MRU arrangement is a convenient mechanism for describing the behavior of a cache, but represents only a number of possible implementations of cache memory. In this specification, the terms “cache” and “cache memory” are used interchangeably.

  Although MRU caching and its extensions disclosed herein are discussed in the context of a client / server architecture, similar principles apply to efficient hard disk access on a single computer (access to hard disk to RAM). Note that it also applies to a number of other scenarios, such as RAM, which is slower to access and thus RAM is used to cache the most recently used content on the hard disk. In one or more other embodiments, data is collected from the environment or generated computationally, rather than being loaded from disk or transmitted over a network. In any case, the client can access a small but fast temporary cache memory and a larger but slower data source from which information is repeatedly requested. This slower data source is generally referred to herein as a “server”.

  The following discussion of convergence series is provided as an introduction to the cache memory apparatus and method disclosed herein.

  The infinite sum of the series y (n) = n ^ -p where n varies from 1 to infinity and p> 1 is finite. Similarly, the sum of y = 1 / b ^ n is finite for b> 1. For example, in the latter case, if b = 2, the sum is exactly 2. Using the concepts underlying such convergence series, one or more embodiments of the efficient data caching method and apparatus described herein can be implemented.

  One or more embodiments of the methods and apparatus described herein may use the concepts relating to the “Zenon paradox” described below. While this discussion provides a conceptual basis applicable to one or more embodiments described herein, the present invention is not limited by the conceptual aspects described below.

<Concept of Zenon Caching>
Zenon is a runner fast enough to go half a distance from his current position to the end of all the racetracks in one step (for the sake of discussion, he will assume one step per second). The paradox is that he never moves into the final line of the competition field, even though he moves forward one step at a time. This paradox is simply related to the above 1 / b ^ n series, with b = 2 and taking the sum from n = 2 to infinity. This concept can be extended to store cache objects by allowing them to be progressively compressed to a larger extent as usage or access decreases (the cache itself is a “stadium”). To be compared). Thus, in the progression from the MRU end of the cache to its LRU end, a theoretically infinite number of additional cache objects of increasingly decreasing size can be introduced without running out of space. This principle is referred to herein as the Zenon cache concept.

  The cache objects involved in this specification are preferably compressible, which in this disclosure corresponds to following a lossy data compression technique. Lossy data compression is characterized by the ability to represent a data object with fewer bytes than the full representation of the data object. A higher compression ratio generally incurs a higher distortion of the data object and a lower quality of the image rendered using the compressed data (this object includes one or more image files). Without limitation, lossy compression techniques may be applicable to sound, video, and many other data types.

  In one or more embodiments, a compressed version of the data may be suitable as a substitute for uncompressed data. Below a given level of distortion, the compressed representation of the data may be adequate enough, and when above a given level of distortion, the compressed representation is used by the client to create a higher quality version of the data. It may be appropriate as a temporary measure while waiting. The higher quality version can simply be compressed lower than the temporarily used version, or it can be lossless or uncompressed.

  In one or more embodiments, lower quality representations can be a subset of higher quality representations, since improved representation quality on the client side augments lower quality representations. This means that data transmission can simply be used to provide a higher quality representation. Preferably, with this approach, the burden of sending a completely new set of data associated with a high quality representation need not be incurred. This approach preferably avoids redundancy and thus preferably substantially increases efficiency.

  Consistent with the technique described above, the reverse process of reducing the representation quality of an object does not require compression or recompression of the data used for the high quality representation of the image, but its high quality representation. It can be used to simply remove some of the data used in the process. This property also preferably increases the efficiency of the caching apparatus and methods disclosed herein.

  In one or more embodiments, compression techniques can provide objects at a compression level that is scaled from lossy to lossless. This feature creates a reversible representation of a data object in multiple steps, from large lossy to lossless, with little or no extra total cost relative to initially sending the lossless version. Can be made possible. An example of a data type and compression technology that enables the above features is image wavelet compression, embodied by the JPEG 2000 standard. However, the present invention is not limited to the use of the JPEG2000 standard.

  Given the above properties, if the memory is “continuous” (ie, not discretized into bytes), theoretically, simply the compressed size of the object will converge as previously described herein. By enforcing the constraint of following series rules, it should be possible to cache an infinite number of data objects in a finite amount of memory. The operation of a cache that can function according to the theory described above is described below in connection with FIGS.

  In the graph of FIG. 42, the variable “N” preferably corresponds to the number of each cache object, and the value of the number of each cache object represents the freshness of use of each such cache object, increasing N. The value corresponds to a decreasing newness of use of the cache object associated with that number. The variable “Y” preferably corresponds to the size of each cache object. For the “Y” variable, a value of “1” may correspond to the highest quality condition, ie the size of the cache object when not compressed at all. The most recently used objects can be represented with less distortion, and objects that have not been used for a longer time may be subject to a range of compression consistent with their last used freshness. it can. In FIG. 42, it can be seen that the value of the cache object size Y after compression decreases as the usage of the relevant cache object decreases. Note that the “Y” variable may correspond to the absolute size (expressed in arbitrary units) of each object (whether compressed or not) in the cache. Instead, in one or more other embodiments, “Y” can correspond to a compression ratio, eg, the value “1” corresponds to a full-size object, and the value “0.5”. Corresponds to an object occupying half of the uncompressed size.

  Referring to FIG. 43, the cumulative sum of size Y of objects numbered from 1 to N can still be finite as shown in FIG. 43 for each value of N. . The unit of the variable “Y” may be a unit of data storage space (or data storage space that it requires) corresponding to the size of one representative fully expanded (uncompressed) cache object. it can. 1 and 2 assist in understanding the theory of one or more embodiments of the present invention, so information describing the size of a cache object in bits and / or bytes of data is not provided herein.

  In one or more embodiments, the theoretical implementation described above is preferably modified for two reasons. First, in practice, the memory storage preferably consists of discrete storage units. Thus, for example, usually it does not make sense to compress a cache object to occupy an amount of storage space less than one bit. Second, the total number of operations performed on the cache is preferably finite. In contrast, forcing a continuous curve of compression ratio described by one of the convergence formulas above reduces the size of all cache objects in the cache each time additional cache storage space is needed May be accompanied. This should require a large number of operations impractically.

  In one or more embodiments, the number of objects in the cache is actually finite. However, if the concept of Zenon cache is used, this number can be much larger than would be possible with conventional MRU caching. In addition, cached objects may have the property that they can be stored at a high quality level (somewhere from low level distortion or compression lossyness to lossless compression, uncompressed data) when used recently. it can. This quality level of the cache object can be progressively worse with each successive cache memory access where the cache object is not accessed (ie, subject to progressively higher levels of distortion or compression lossyness).

  Since computer memory is discrete and there may be a minimum compressed size of cache objects below which it may not have value to the user, the cached representation is It may be under the control of the maximum compression ratio that causes the size. Thus, in one or more embodiments, the maximum number of cache objects that can be stored in the cache is the total data storage space in the cache, after the minimum compression described above, when all of the objects are of equal size. Divided by the amount of data storage space occupied by one cache object having a size of However, the cache objects need not all be the same size.

  There are a number of ways to design a series that is limited by one of the above mentioned equations (or any other convergent sum) and thus has a finite sum. An additional constraint, specifically the likelihood that any given value repeats in a series of consecutive values is greater than N so that the number of different values of Y used can be limited to a reasonable number You can also introduce a constraint of increasing by value.

  Examples of such series are 1, 1/4, 1/4, 1/16, 1/16, 1/16, 1/16, 1/64, 1/64, 1/64, 1/64, 1/64, 1/64, 1/64, 1/64, 1/256, and the like.

Obviously, the sum of the series 1, 2 1/4, 4 1/16, 8 1/64, etc. is exactly the same as y = 1/2 ^ n, as described earlier in this specification. Is 2. However, if this series is taken up to n = 16000, only about log ₂ (16000) values of y (object data storage size) or about 14 values can be used.

  In one or more embodiments, the logarithmic function described above provides one form of increasing the number of available values of Y (the possible size of the cache object) much more slowly than the value of N. To do. However, the present invention is not limited to the use of this logarithmic function, and other mathematical operations that increase the number of Y values more slowly than the value of N can be used in connection with the present invention.

  In one or more embodiments, as few as 20 values of Y can be used when N = million (determined using the log-based formula provided above). This is because when a space has to be freed up in the cache, the majority of cache objects preferably occupy a certain amount of data storage space that does not need to be changed, so only a few operations Imply that it may be necessary to establish an appropriate allocation of data storage space between.

  Other mathematical series can also meet the desired criteria for use in cache memory management systems and methods. Further, it is possible to use a series that theoretically does not converge (ie, the sum is infinite). This is because in practice, a finite number of terms are summed up in any case.

  In one or more embodiments, a random algorithm can be used to improve the basic algorithm in several ways. In one or more embodiments, the series described above, such as 2 * 1/4, 4 * 1/16, includes only a small number of available cache object sizes, possibly between different objects in the cache. May lead to a clear difference in compression ratio. Use random selection to “squeeze” a randomly selected subset of cache objects in a weighted manner until a target amount of space is made available for a new cache object (used by it) Reduced data storage space). This approach can yield beneficial results. This is because the exact location of the cache object in the cache can decrease in importance with the increasing number of objects in the cache. The amount by which each object is “squeezed” can also be at least partially randomized. Obvious discontinuities or thresholds in cache object quality that can be perceived in images rendered using cache objects stored in the cache by using a randomization algorithm similar to that described herein Can be reduced.

  The following presents an illustrative example of management of cache objects in a data cache according to one or more aspects of the present invention.

  FIG. 44 is a block diagram of a data cache 300 that includes a plurality of cache objects 302-310, according to one or more embodiments of the invention. FIG. 45 is a block diagram of the data cache 300 of FIG. 44 in which the cache objects 302, 304, 308, and 310 have been resized according to one or more embodiments of the present invention. FIG. 46 is a block diagram of the data cache of FIG. 45 in which the accessed cache object 306 has been resized and restored to the cache 300 in accordance with one or more embodiments of the present invention.

  In one or more embodiments, including those shown in FIG. 44, cache 300 includes five cache objects (abbreviated as “CO” in FIGS. 3-6 for simplicity of illustration) CO 1 302, CO 2 304, CO 3 306, CO 4 308, and CO 5 310 can be included. The number (5) of cache objects shown in FIG. 44 is selected for convenience in showing various concepts described in this specification. However, fewer or more than five cache objects can be included in the cache 300. In principle, there is no lower limit on the number of cache objects that can be included in the cache 300. In principle, the upper limit on the number of cache objects that can be included in the cache 300 corresponds to the total size of the cache divided by the smallest acceptable cache object size (discussed elsewhere herein). It can be.

  3-5, for purposes of illustrating the various concepts disclosed herein, the width of each illustrated cache object is intended to be proportional to the data storage space used by that cache object. . Also, in FIGS. 3 to 5, advancing from the leftmost cache object to the rightmost cache object corresponds to the newness of increasing access to the displayed cache object, and the cache object that has not been used for the longest time is displayed. The most recently used cache object is shown at the right end.

  FIG. 45 illustrates the cache 300 after the CO 3 306 has been accessed by a computing system, such as the computing system 700 that uses the cache 300. In this example, CO 3 306 is not shown in its original location in cache 300. This is because this position has been overwritten in the state of the cache 300 shown in FIG. In addition, free space 402 is created to make room for the expanded version of CO 3 306, which is more data storage in cache 300 than the original version of CO 3 in FIG. May take up space. In FIG. 46, the expanded version of the CO 306 is written in the portion of the cache 300 that was occupied by the free space 402 under the cache 300 conditions shown in FIG.

  FIG. 47 is a flow diagram of a method 600 for accessing a cache object in the data cache 300 and updating the data cache 300 according to one or more embodiments of the invention. Reference is now made to FIGS.

  In step 602, cache objects 302, 304, 306, 308, and 310 are provided to programs that can access cache 300. The groups of cache objects that initially exist in the cache 300 are shown in FIG. This initial state of the cache 300 may arise from default settings in the programs or from program steps previously executed in the one or more programs. In any case, what is important in the following discussion is the changes made to the cache 300 after the establishment of the initial state shown in FIG. Any of a number of programs running on any of the various computing systems can communicate with the cache 300, but for convenience, software that can access the cache 300 is described below. It is simply called “program”.

  In step 604, an indication of which cache object will be used next by the program may be provided. At step 606, the indicated cache object, which in this example is CO 3 306, can be accessed programmatically.

  In step 608, a determination can be made as to whether the accessed cache object is expandable. As used herein, a cache object is expandable when it can occupy more data storage space by receiving a lower compression ratio. Such expansion is achieved by augmenting data that already exists in the cache object, rather than by supplying a completely new set of data corresponding to the new compression rate (or corresponding to the lack of compression). be able to.

  If the accessed cache object is not expandable, the cache object is preferably restored to the cache 300 at step 610. Preferably, at step 610, the restored cache object occupies the same amount of data storage space after the access as it occupied before the access. Consistent with the principle of LRU-MRU caching, accessed cache objects can be written to the right end or MRU end of the cache 300 when restored to the cache 300. However, instead, the accessed cache object can be written to any part of the cache 300. Continuing with this branch of method 600, method 600 preferably ends at step 612.

  Referring to step 608, if an accessed cache object, such as cache object 306, is expandable, the cache object is preferably expanded in accordance with one or more embodiments of the present invention ( Step 614). As previously described herein, the deployment of the cache objects described above provides an arrangement in which the most recently and / or most frequently accessed cache objects are stored in the cache 300 at the highest quality level. It helps to help.

In one or more embodiments, if there are “N” cache objects in the cache, the number of possible sizes of such cache objects (measured in units of data storage space) is expressed as log ₂ ( N) can be limited to an amount equal to N). Establishing a limited finite number of possible cache object sizes as described above favors the computational expense of determining a new expanded size for a cache object, such as CO 306, that is expanded in step 614. Restrict.

  In one or more embodiments, the amount of data storage space required for the expanded (or in other words, less compressed) version of CO 306 is a computing system (not shown) that can access cache 300. Can be calculated by: If the cache 300 is not yet ready to receive the expanded version of the CO 306, the expanded version of the CO 306 is stored in another memory storage location (not shown) for temporary storage therein. Can be written on.

  In step 616, the data storage space 402 required to store the expanded version of CO 306 is preferably made available in cache 300. If there is enough space in the cache 300 to store the expanded version of the CO 306 without changing any cache objects in the cache 300, then one or more cache objects in the cache 300 Size reduction can be omitted. However, if all or substantially all of the storage space in the cache 300 was occupied before the CO 306 was accessed, one or more cache objects other than the CO 306 may be expanded in the cache 306. In order to free up space in the cache 300 for storage of the modified version, a reduction in size may be experienced.

  In one or more embodiments, the number of cache object size reduction operations can be reduced if there is a limited number of possible cache object sizes. Limiting the number of cache object size reduction operations preferably operates to reduce the computational burden on the computing system accessing the cache 300 and preferably results in overall computing system efficiency.

  In one or more embodiments, there may be various ways of achieving the desired amount of data storage space clearing. As used herein, the term “clear” corresponds to making data storage space available in the cache 300 by reducing the data storage space allocated to one or more cache objects in the cache 300. Can be.

  In one or more embodiments, the amount of data storage space that is cleared is that the deployed cache object exceeds the space that the deployed cache object occupied prior to the most recent access by the computing system. It can correspond to the amount of additional storage required. However, in other embodiments, the amount of space that is cleared can be less than or more than the amount space that the most recently accessed cache object increases in size.

  In one or more embodiments, the space to be cleared for the most recently used deployed cache object shall be at one end of the cache 300, as shown in FIG. be able to. However, in other embodiments, the space to be cleared can be placed in other locations within the cache 300.

  In one or more embodiments, the available data storage space is provided at the expense of one or more of the cache objects of FIG. 44 other than CO 3 306 (the most recently used cache object). Can do. Specifically, it is possible to provide the additional space required by reducing the size of just one remaining cache object or by reducing the size of all cache objects except the most recently used cache object. It may be. Furthermore, any number of cache objects between the two extremes can be made to discard storage space in preference to the most recently used cache object being deployed. In the following, all cache objects other than the most recently accessed cache object are considered “suitable for size reduction”.

  In one or more embodiments, the size reduction range of one or more cache objects suitable for size reduction can be selected according to one or more considerations. In one embodiment, a cache object suitable for size reduction can discard an equal or substantially equal amount of storage space. In another embodiment, the preferred cache object discards an equal or substantially equal ratio of its pre-shrink size to clear the space for the most recently used cache object being expanded. be able to.

  In one or more other embodiments, the size reduction range for each cache object may be based on how recently the cache object was last accessed. Specifically, a cache object suitable for size reduction can gradually throw away more storage space with the decreasing newness of its last access. Therefore, under this approach, the most recently used cache objects suitable for size reduction can throw away a relatively small amount of storage space, while the least recently used cache objects A large amount of data storage space can be thrown away, and a cache object between the two extremes throws away an intermediate amount of storage space.

  The discussion of storage space reduction herein is primarily directed to merely reducing the size of cache objects that are not the most recently accessed, but in one or more embodiments, one or more Cache objects can be removed from the cache 300 to clear the data storage space. Further, such cache object removal can be practiced either alone or in combination with cache object data storage space reduction of cache objects remaining in the cache 300.

  In the embodiment of FIG. 46, all four cache objects 302, 304, 308, and 310 that remain in cache 300 clear the data storage space for writing CO 306 to the location shown at the right end of cache 300. In order to be reduced in size. However, in alternative embodiments, three or fewer of the four cache objects 302, 304, 308, and 310 suitable for size reduction may experience size reduction. Preferably, the method ends at step 620.

  In one or more embodiments, rather than managing objects in the cache using only the freshness of use of each cache object as a variable in determining the cache object size, which objects are in cache object management. It can also include intelligent guesses about what might be needed next. Thus, objects that are less likely to be needed can be “squeezed” before objects that have a higher likelihood of being needed in the future. In one or more embodiments, this inference approach can be used to randomly select objects in the cache for squeezing, and to generate a randomly varying amount of squeezing of the selected objects; Can be combined.

  FIG. 48 is a block diagram of a computing system 900 that is adaptable for use with one or more embodiments of the invention. In one or more embodiments, a central processing unit (CPU) 902 can be coupled to the bus 904. In addition, the bus 904 is connected to an inventive cache 300, random access memory (RAM) 906, read only memory (ROM) 908, input / output (I / O) adapter 910, communication adapter 922, user interface adapter 906, and display adapter 918. Can be combined.

  In one or more embodiments, RAM 906 and / or ROM 908 may hold user data, system data, and / or programs. The I / O adapter 910 can connect a storage device, such as a hard drive 912, a CD-ROM (not shown), or other mass storage device, to the computing system 900. Communication adapter 922 can couple computing system 900 to a local area network, a wide area network, or an Internet network 924. User interface adapter 916 can couple user input devices such as keyboard 926 and / or pointing device 914 to computing system 900. Further, the display adapter 918 can be driven by the CPU 902 to control the display on the display device 920. The CPU 902 can be any general purpose CPU.

  Any known processor, programmable digital device or programmable digital system capable of executing standard digital circuits, analog circuits, software and / or firmware programs can be used to implement the methods and apparatus described above and / or described hereinbelow. Note that any known technology, such as a programmable array logic device, or any combination of the above may be utilized. One or more embodiments of the present invention may also be implemented in a software program for storage on a suitable storage medium and execution by a processing unit.

  Although the invention herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present invention. Accordingly, it will be appreciated that numerous modifications can be made to these exemplary embodiments and that other arrangements can be devised without departing from the spirit and scope of the invention as defined by the appended claims. I want to be.

System and method for efficiently encoding data

[Background]
Recently, image compression standards such as JPEG 2000 / JPIP (JPEG 2000 Interactive Protocol) have been developed to meet demanding engineering goals, i.e. very small images (i.e. gigapixel in size) from server to client. It has been introduced to allow for incremental or selective delivery over bandwidth communication channels. (For example, see David Kabdu's implementation of Kadadu (see, for example, Non-Patent Document 3)). When such images are viewed at full resolution, only a limited area of each image can fit on the client's graphical display at any given time. These new standards are directed to selectively accessing such areas and transmitting only the data associated with those areas over the communication channel. If this “region of interest” or ROI changes continuously, the continuous interaction between the client and the server via the low bandwidth channel ensures that the representation of the client in the region inside the ROI is accurate. May continue.

[Non-Patent Document 3] www.kakadusoftware.com
[Disclosure of the Invention]
[Problems to be solved by the invention]
However, existing approaches generally require the transmission of a significant amount of data to substantially shift the position of the region of interest, thus limiting the speed with which such a shift can be implemented. Furthermore, existing approaches generally rely on sequential access to a linear string of text, thereby placing a heavy burden on text navigation when the client seeks to significantly change the location of the region of interest. Therefore, there is a need for an improved method for transmitting data from a server to a client.

[Means for solving problems]
According to one or more embodiments, the present invention converts data identifying a plurality of visual features into a plurality of pixel characteristic data values, and an image file having pixels having respective pixel characteristic data values. Forming a method. Preferably, the plurality of visual features includes at least one graphical symbol. Preferably, the visual feature identification data includes an ASCII code. Preferably, the visual feature identification data includes frequency-order positions of visual features. Preferably, the visual feature identification data is determined based on a combination of a) frequency of use of visual features and b) transition probabilities between successive visual features of the visual features. Preferably, the pixel characteristic data value includes at least one data type selected from the group consisting of a pixel luminance data value, a pixel color data value, and an image contrast data value.

  Preferably, the pixel characteristic data value includes a pixel color data value. Preferably, the pixel color data values are RBG (red, blue, green) pixel color data values, HSV (hue, saturation, lightness) pixel color data values, and CMYK (cyan, magenta, yellow, black) pixel color data. It includes at least one color data value type selected from the group of values.

  Preferably, the method further includes reversibly compressing the image file. Preferably, the visual feature identification data includes a difference in values of identification data of successive visual features among the visual features. Preferably, the visual feature includes alphanumeric characters. Preferably, the method further includes placing pixels in the image file in the same order that the alphanumeric characters are ordered in the original document. Preferably, the method further includes transmitting the image file to the client. Preferably, the method further includes decoding at least a region of the image file into alphanumeric characters for viewing by the client. Preferably, the method further includes allowing the client to browse the decoded image file region in the first dimension.

  Preferably, the method further includes allowing the client to browse the decoded image file region in the first dimension and the second dimension. Preferably, browsing in the first dimension includes traveling along a line of alphanumeric characters, and browsing in the second dimension preferably scrolls across multiple lines of alphanumeric characters. including. Preferably, browsing includes browsing the decoded file in a manner that emulates browsing within the original document. Preferably, the method further includes correlating the position of the pixel in the image file with the position of the alphanumeric character corresponding to the pixel in the original document containing the alphanumeric character.

  Preferably, the position of the pixel in the image file corresponds at least substantially to the position of the corresponding alphanumeric character in the original document. Preferably, the position of the pixel in the image file is substantially the same as the position of the corresponding alphanumeric character in the original document. Preferably, the method further includes transmitting at least a region of the image file from the server to the client. Preferably, the method further includes requesting an area of the image file by the client from the server.

  Preferably, the method further includes transmitting the requested area of the image file by the server. Preferably, the transmitting step includes transmitting compressed data suitable for decoding by the client.

  According to another aspect, one or more embodiments of the present invention is an apparatus that can include a processor operating under the direction of a software program, the software program identifying a plurality of visual features. An apparatus is provided for causing an apparatus to perform actions including converting data to be converted into a plurality of pixel characteristic data values and forming an image file having pixels having respective pixel characteristic data values.

  According to another aspect, one or more embodiments of the present invention provide, under the direction of a software program, data that identifies a plurality of visual features to a plurality of pixel characteristic data values to a device that includes a processor. A storage medium can be provided that includes a software program operable to perform actions including converting and forming an image file having pixels having respective pixel characteristic data values.

  Other aspects, features, benefits, etc. will become apparent to those skilled in the art when the description of the preferred embodiments of the invention herein is taken in conjunction with the accompanying drawings.

  While the presently preferred form is shown in the drawings for purposes of illustrating various aspects of the invention, it is to be understood that the invention is not limited to the precise arrangements and instrumentalities shown.

[Best Mode for Carrying Out the Invention]
One or more embodiments of the present invention include extending selectively decompressable image compression and transmission technologies to text data or other data that can be identified using a binary representation. Can do.

  As used herein, binary data that identifies and / or describes visual features can be converted from an initial format such as ASCII (American Standard Code for Information Interchange) or other formats, such as pixel luminance data. It can be converted into a format suitable for incorporation into image data, but not limited thereto.

  The visual features referred to above can include, but are not limited to, graphical symbols and / or alphanumeric characters. However, as used herein, visual features can include all visual image features that can be described and / or identified using binary data. Further, such data can be encoded into pixel luminance data, but the present invention is not limited to encoding in this format.

  In one or more embodiments, the initial data (visual feature identification data) includes a plurality of possibilities including, but not limited to, pixel luminance data, pixel color data, image contrast data, and / or other forms of image data. Can be converted into various types of pixel characteristic data. The pixel color data described above includes red, blue, green (RBG) pixel color data, hue, saturation, lightness (HSV) pixel color data, cyan, magenta, yellow, black (CMYK) pixel color data, and Other forms of pixel color data may be included, but are not limited to these.

  In the next discussion, we will consider a large text, specifically the book “Ulysses” by James Joyce. In one or more embodiments, this text can be formatted by placing each chapter in its own column and arranging successive chapter columns from left to right. However, it should be appreciated that other arrangements of chapters can be implemented. Assume that the column has a maximum width in characters, such as 100 characters. 49. Ulysses text encoded as an image of this shape, where each pixel in FIG. 49 corresponds to a single text character (or character position that does not contain text, such as an empty space) in the original text document. Show the whole. FIG. 50 shows a text image of the first five chapters of Ulysses using the same encoding method as FIG.

  In one or more embodiments, the luminance value of each pixel can be set equal to the ASCII code of the character encoded in that pixel. Since both grayscale pixels and ASCII characters can be represented using an 8-bit sequence (both can have values in the range of 0-255), the correspondence between pixel values and characters Can be easily implemented. In this disclosure, text characters and other characters can be represented using pixels using ASCII codes as pixel luminance values, although other codes for text characters and other characters may be used for this purpose. I want to understand what I can do.

  In general, Ulysses' full text in ordinary ASCII representation (ie, as a standard text file) occupies 1.5 MB of storage space, which is large enough to be transmitted entirely over a narrowband communication channel. It may be too much. The pixel characteristic data representation of character data encoded as a reversible JPEG2000 image in FIG. 49 (referred to herein as “character pixel image”, also referred to as “text image”) is approximately 2.2 MB (megabytes) of data storage. Requires space. This storage space requirement can be significantly reduced if the chapter of the book is more even in length, resulting in less empty space (encoded as 0) in the character pixel image (text image). .

  However, more important than the total file size is the ability of an ordinary JPIP server to selectively and incrementally supply this file to the client. Specifically, what is important here is the server's ability to serve selected portions of the file in controllable increments of resolution.

  In one or more embodiments, a client viewing text at a zoom level sufficient to read characters (which may need to be well above 1 pixel / character on the client-side display) Using JPIP (or other suitable protocol), only the relevant part of the text can be requested. This operation is efficient and has a very low bandwidth connection to the server, ie, downloading the entire text becomes prohibitive due to the size of the data associated with such a download Appropriate performance can be achieved for the reader of the text even under conditions.

  In one or more embodiments, a similar effect can be achieved using client / server technology specifically designed for selective access to large text, but the character pixel image approach described above. Has several benefits listed below over conventional implementations.

  1) The character pixel image approach can use existing technologies and protocols designed for very large amounts of data.

  2) The character pixel image approach can be easily scaled up to text that is many gigabytes in size or larger without any performance degradation.

  3) Access to text or other visual features in the region of interest is preferably two-dimensional in one or more embodiments of the invention. Thus, in many situations (eg, when text is viewed in a row as in the case of Ulysses), the character pixel image approach disclosed herein is preferably an existing one that treats text as a one-dimensional string. Allows browsing more efficiently than is available using techniques.

  4) Since wavelets are used in JPEG2000, character pixel images are preferably the subject of multi-resolution representations. Other visual features identified using text or visual feature identification data are generally not readable except at the final (most detailed) resolution, but the wider spatial support for lower resolution wavelets is Preferably providing intelligent client-side caching of text and / or other visual features near the region of interest. Thus, ROI movement across various regions of the original text, which may occur when scrolling text in a traditional manner, can be efficiently performed using one or more embodiments of the present invention. Can be executed.

  In one or more embodiments, the above concepts can be easily extended to handle formatted text, Unicode, or other metadata. This is because all such data can be represented using ASCII text strings, possibly with embedded escape sequences.

  In the selected application, JPEG2000 can be used as a lossy compression format, which means that the decoded image byte is not necessarily identical to the original byte. As used herein, the term “decode” refers to converting pixel data in a text image back to original text data or other visual feature data. Lossless compression is generally unacceptable when image bytes represent text. However, one of the design goals of JPEG 2000 was to efficiently support lossless compression. This is because it is important for certain imaging functions (such as medical and scientific applications). The lossless compression ratio of a photographic image is usually only about 2: 1 compared to a visually acceptable lossy image that can be easily compressed, usually 24: 1.

  Both lossy and lossless image compression generally works best for images with good spatial continuity (meaning that the difference between adjacent pixel luminance values is small). Raw ASCII encoding is not optimal from this perspective. This is because ASCII text-encoded consecutive text characters may have values that vary greatly. Therefore, some alternatives are considered below.

  FIG. 51 is a text image of the first five chapters of Ulysses encoded using frequency-order encoding according to one or more embodiments of the present invention.

<Frequency encoding>
In one or more embodiments of the present invention, encoding by reordering characters according to their frequency of occurrence from highest frequency to lowest frequency in the relevant text, in English or in another language. Efficiency can be improved. Thus, in one or more embodiments, an empty space has a code 0, and a character pixel image has a pixel value of 0. A “space” character can receive the code “1” (the corresponding value in the character pixel image is also “1”). e, t, a, o, i, n, s, r, h, l. . . Can be converted to successive pixel values starting with “2” (corresponding to “e”) and going up to a value of 255.

  In converting all characters of a large text into pixel values using this approach, all 255 pixel values may eventually be used. However, due to the very nature of the conversion from text characters (or other visual features) to pixel values as contemplated herein, pixel value appearances preferably become increasingly rare as the number increases.

  The image in FIG. 51 shows this point. Pixel values tend to be close to zero. At least equally important, the difference in pixel values between successive character pixel values in the embodiment of FIG. 51 is preferably smaller than in the embodiment of FIG.

  In general, 8 bits are needed to represent each pixel value, where all pixel values in the range 0-255 can appear equally. However, in embodiments where some pixel values appear much more frequently than others, those pixel values can preferably be represented using fewer bits without losing any information.

  An example is considered to illustrate this point. In this extreme example, the pixel value is equal to 0 for 99% and has another value between 0 and 255 otherwise. In this case, the encoding algorithm encodes the value of 0 using a single “0” bit and the non-zero value with the leading “1” bit (to signal the presence of a non-zero value) followed by a non-zero. Can be encoded with an 8-bit representation of the value. Thus, this approach saves 7 bits per pixel for 99 out of 100 pixels, but uses one extra pixel to represent a non-zero pixel that occurs only in the 1% case. Accordingly, a decoding algorithm corresponding to the encoding algorithm described above preferably interprets a “0” bit as representing a 0-bit value, and converts a bit sequence starting from a “1” bit value to a leading “1”. Interpret as having the value represented by the bit following the bit.

  However, even in less extreme situations, the presence of pixel values that occur much more frequently than other pixel values does not incur any loss of pixel data, and is represented by pixel data through logical expansion. A significant savings in storage space can be made without incurring any loss of visual feature data. In general, the frequency of occurrence of values in two or more categories can be established using a gradually increasing number of bits to represent values that generally occur with decreasing frequency. In this way, smaller bit sequences can be used in most cases, thereby saving data storage space requirements and data communication bandwidth requirements.

  In the middle example, 5 bits can be used to represent the most frequently occurring pixel values, and 9 bits can be used for less frequently occurring values. For the most frequently occurring visual features, a leading bit can be provided that can be "0", followed by 4 bits that represent the actual value of the pixel. For pixel values that occur less frequently, a leading bit can be provided that can be "1", followed by 8 bits that represent the actual value of the pixel.

  In one or more other embodiments, frequency encoding can benefit from spatial coherence to represent a text image using a reduced number of bits. Specifically, the image can be divided into 8 × 8 pixel blocks, thus providing 64 pixels within each block, each pixel being a frequency encoded visual feature (which can be a text character). Represents. This encoding method can then review each block and determine the number of bits needed to represent the pixel value having the maximum value in that block. The determined number of bits can then be used to represent all pixel values in the block.

  For many of the blocks in any given image, the pixel with the largest value may be represented using 4 bits or less. Thus, considerable savings in data storage requirements can be obtained when using this block-by-block approach.

  In one or more embodiments, when a Ulysses frequency encoded text image is reversibly compressed as a JPEG2000 image, the file size is slightly larger than the raw ASCII text image (1.5 MB) and the ASCII encoding 1.6 MB which is 37% smaller than the text image. With further optimization of character encoding, the compressed file size can be reduced below the ASCII text file size. For further optimization, use the character transition probabilities to expand the encoding (Markov-1), not just the frequency (Markov-0), and / or the character and the next character, not the character itself. Encoding deltas or differences between them as pixels can be included, but is not limited to these.

  With these additional optimizations, text ready to be delivered in this form may actually occupy less data storage space than raw ASCII.

  Although one or more embodiments of the invention described herein include using JPEG2000 / JPIP as a selective image decompression technique, the invention is not limited to the use of this image compression technology. Other image compression formats and image compression protocols can be used with the present invention including, but not limited to, Mr. LizardTech's MrSID format and protocol with similar properties.

  FIG. 52 is a block diagram of a computing system 1400 that is adaptable for use with one or more embodiments of the invention. In one or more embodiments, a central processing unit (CPU) 1402 can be coupled to the bus 1404. Further, the bus 1404 may be coupled to a random access memory (RAM) 1406, a read only memory (ROM) 1408, an input / output (I / O) adapter 1410, a communication adapter 1422, a user interface adapter 1406, and a display adapter 1418. it can.

  In one or more embodiments, RAM 1406 and / or ROM 1408 may hold user data, system data, and / or programs. The I / O adapter 1410 can connect a storage device such as a hard drive 1412, a CD-ROM (not shown) or other mass storage device to the computing system 1400. Communication adapter 1422 may couple computing system 1400 to a local area network, a wide area network, or an Internet network 1424. User interface adapter 1416 may couple user input devices, such as keyboard 1426 and / or pointing device 1414, to computing system 1400. Further, the display adapter 1418 can be driven by the CPU 1402 to control the display on the display device 1420. The CPU 1402 can be any general purpose CPU.

  Any known processor, programmable digital device or device that operates to execute standard digital circuits, analog circuits, software and / or firmware programs may be used for the methods and apparatus described above and / or described later in this document. Note that any known technology can be used, such as a programmable digital system, a programmable array logic device, or any combination of the above. One or more embodiments of the present invention may also be implemented in a software program for storage on a suitable storage medium and execution by a processing unit.

  Although the invention herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present invention. Accordingly, it will be appreciated that numerous modifications can be made to these exemplary embodiments and that other arrangements can be devised without departing from the spirit and scope of the invention as defined by the appended claims. I want to be.

System and method for managing communication and / or storage of image data

[Background]
Recently developed image compression standards and image transmission standards, such as JPEG2000 / JPIP, have enabled interactive display of large images (ie, gigapixel in size) over a narrow bandwidth communication channel. However, these emerging standards and technologies enable simultaneous flexible visual interaction with a large number of images, each of which can achieve a more ambitious goal, each potentially potentially very large. Does not provide a means to

[Patent Document 5] US Patent Application No. 11/082556
[Patent Document 6] US Provisional Patent Application No. 60/617485
[Patent Document 7] US Provisional Patent Application No. 60 / 619,118 Specification
[Patent Document 8] US Patent Application No. 11/141958
[Patent Document 9] US Provisional Patent Application No. 60 / 619,053
[Disclosure of the Invention]
[Problems to be solved by the invention]
Accordingly, there is a need in the art for improved systems and methods for improved systems and methods for transmitting and / or storing image data.

[Means for solving problems]
According to one aspect, the present invention establishes communication between a first computer and a second computer via a communication link, wherein the second computer stores an image collection in the form of compressed image data. Selecting a plurality of images in a collection for communication to the first computer stored therein and a second computer from the second computer before transmitting full resolution image data of any of the selected images. Transmitting all the low resolution image data of the selected image to a computer.

  Other aspects, features, benefits, etc. will become apparent to those skilled in the art when the description of the preferred embodiments of the invention herein is taken in conjunction with the accompanying drawings.

  While the presently preferred form is shown in the drawings for purposes of illustrating various aspects of the invention, it is to be understood that the invention is not limited to the precise arrangements and instrumentalities shown.

[Best Mode for Carrying Out the Invention]
FIG. 53 is a block diagram illustrating a system 170 that can be connected to enable communication of image data between multiple computers according to one or more embodiments of the present invention. System 170 preferably includes a client computer 182 connected to display 184 and data storage device 186. The system 170 also preferably includes a server computer 188 that can connect to the data storage device 190. Server computer 188 can also be connected to the Internet 180.

  In one or more embodiments, image data is communicated between multiple computers 182, 188 to produce a large collection of potentially large images using a relatively low bandwidth connection between them. Can be allowed to see. For example, the desired viewing and navigation of images stored on the server computer 188 can be achieved by transmitting selected portions of the image data stored on the server computer 188 at a controllable level of resolution. . The selectivity of the image data 114 can be, for example, selecting a specific image at a high resolution, or even selecting a selected portion of a specific image at a high resolution.

  This specification changes the type of device used as client computer 182 and server 188, the type of image data 114 transmitted between them, and transmits selected image data at a specified level of resolution. Various embodiments are described, including various applications of capabilities.

  FIG. 54 is a block diagram of an image 250 having at least two regions of interest 252, 254 therein, according to one or more embodiments of the invention. The image 250 can be a subset of the image data 114. Alternatively, depending on the image data requested by the client computer 182, the image data 114 can represent a subset of the image 250.

  In one or more embodiments, the image 250 may be stored in a compressed form on the server computer 188 or in the storage device 190. When stored in this manner, data of multiple resolution levels in various regions of the image 250 can preferably be stored and requested by the client computer 182 for download.

  In one or more embodiments, the resolution level at which a particular image or region of an image is stored on the client computer 182 can be easily raised or lowered. If a previous download results in storage of an area or image at a first resolution level (which may be lower than full resolution), this first resolution level is replaced with data representing the next higher level resolution. The addition can preferably be done without having to discard the data representing the first resolution, thereby avoiding redundancy and increasing the efficiency of the image data communication contemplated herein. Conversely, the resolution level of the area or image stored at client 182 is discarded without discarding the data corresponding to the lower level resolution of the same area or image by discarding the highest resolution stored therein. Can be lowered. Such resolution reduction can be practiced at client 182 to clear the data storage space required for one or more regions or images other than those for which data is about to be discarded.

  Related image compression can be provided, for example, by JPEG 2000 or another discrete wavelet transform based image compression scheme. However, the present invention is not limited to the use of any particular compression format or image data representation. Other formats can be used, including image formats whose size in bytes is not substantially smaller than uncompressed image data. It is simply preferred that the selected image format allows multi-scale representation and storage of image data.

  In one or more embodiments, the client computer 182 can attempt to download one or more regions of the image 250, where such regions can be part of the image 250. it can. Only one or more regions of interest 252, 254 may be regions of interest that the client computer 182 seeks to download. Alternatively, the client computer (client) 182 may simply attempt to download one or more selected regions at a resolution that is higher than the resolution at which the rest of the image 250 is downloaded. In either case, the client 182 identifies both the specified area of the image 250 to be downloaded and the resolution level at which the specified area is supplied by the server computer (server) 188, You can request a download.

  In the example of FIG. 54, client 182 preferably requests a full download of image 250 at a lower resolution (the exact resolution level at which the majority of image 250 is downloaded is not relevant to this discussion). However, client 182 tries to download region of interest 1 252 at a higher resolution or even at full resolution. Thus, client 182 preferably specifies server 188 with the coordinates of region of interest 1 252 and the desired resolution level. Thus, in addition to downloading a large portion of image 250 (including portions outside region of interest 1 252) at a lower resolution, client 182 downloads region of interest 1 252 at a specified higher resolution. Is preferred. In other situations, the client 182 may attempt to download only the region of interest and skip the remaining downloads of the image 250.

  In this manner, the user of the client computer 182 can view the region of interest 1 252 at that high resolution without having to download the entire image 250 at the high resolution. Thus, the relatively low bandwidth data communication link between the client 182 and the server 188 can still transmit the entire image 250 while at the same time selecting a particular region of interest (region of interest 1 252 in this example). ) At a high resolution, which gives the viewer the same viewing experience regarding the region of interest that should occur when the client 182 downloads the entire image 250 at a high resolution. The 250 total download option requires significantly longer download times and significantly more data storage space at the client computer 182 or data storage device 186.

<Shift of interest>
In one or more embodiments, the user of client computer 182 may desire to pan across image 250. Typically, panning from one region of interest 252 to another region of interest 254 involves having the client 182 download both regions at the level of resolution at which those regions are seen. Further, in general, all image territories between region of interest 1 252 and region of interest 2 254 are stored at client computer 182 to allow the described panning to occur. As will be described below, in one or more embodiments of the present invention, viewing such regions of interest 252, 254 downloads much less data than the techniques described above, and the client computer. This can be achieved by using less storage space at 182.

  In one or more embodiments, client 182 can shift from a high resolution view of region of interest 1 252 to region of interest 2 254. Image data corresponding to the low resolution representation of region of interest 2 254 is preferably already present in client computer 182 from the download of image 250 described above. In this case, what is needed is additional image data that describes a higher level of resolution relevant to the existing image data of region of interest 2 254 in order to reach the high resolution rendition of region of interest 2 254 at client computer 182. It is only to augment. If necessary, the image data representing the higher resolution level of region of interest 1 252 can be discarded or overwritten to provide space or other in the data storage device 186 for additional image data downloaded for region of interest 2 254 Data storage space.

  In one or more embodiments, the view shift from region of interest 1 252 to region of interest 2 254 is accomplished progressively to allow viewers who view display 184 to view the entire image 250 downloaded at high resolution. Having a viewing experience that can closely simulate the experience that can be obtained on a computer. Specifically, the level of resolution at which region of interest 1 252 is displayed can be progressively lowered to a resolution level at which most of image 250 is represented. The view on display 184 may then represent a progressive pan across the low resolution territory between region of interest 1 252 and region of interest 2 254. Finally, when the region of interest 2 254 is reached, the view on the display 184 is displayed either after completing panning across the image 250 or simultaneously with a later portion of this panning operation. The resolution can be increased toward rendition. Upon completion of the described process, region of interest 2 254 is preferably stored in client computer 182 at a high resolution and can be displayed on display 184 at this high resolution.

  FIG. 55 is a block diagram of a “virtual book” 350 that uses aspects of the technology disclosed herein, according to one or more embodiments of the invention. Virtual book 350 may include display 352, reverse cache 354, and forward cache 356. Although caches 354 and 356 are each illustrated as having two pages stored therein, any number of pages can be stored in either cache 354 or 356.

  In one or more embodiments, the virtual book 352 uses the ability to present selected image data at a controllable level of resolution for this particular case of the virtual book 350. Within the virtual book 350, each image can be a page in the display 352 of the virtual book 350. Display 352 may correspond to display 184 of FIG. 53 or may be a special purpose display that addresses certain features of virtual book 350. Virtual book 350 may correspond to client computer 182 of FIG. 53, or may be a special purpose computer substantially limited to communication, storage, and display of book pages.

  In one or more embodiments, the virtual book 350 stores various resolutions, both the only page stored and / or displayed at full resolution, and both before and after in the sequence of displayed pages. And other pages of resolution.

  In one or more embodiments, the page currently displayed on display 184, ie, the active page, is displayed at full resolution, which is “Page 10” in FIG. In such embodiments, other pages can be displayed with progressively lower resolution with increasing distance in page units from the active page. More specifically, the resolution at which each page is stored is the power of 2 equal to the resolution of the active page displayed on display 356 equal to the number of pages between each stored page and the active page. Can be equal to the value divided by. Thus, applying this approach, page 11 (in forward cache 356) and page 9 (in reverse cache 354) each occupy half the amount of data storage space occupied by active pages in display 352. Can do. Continuing with this approach, page 12 (in forward cache 356) and page 8 (in reverse cache 354) each occupy 1/4 of the amount of data storage space occupied by active pages in display 352. Can do.

  In the above discussion, the amount of data storage space allocated to each page differs by a factor of 2 with respect to the immediately adjacent pages, but those skilled in the art may use a value greater than 2 or less than 2 as a division factor. You will understand what you can do. Further, arithmetic expressions other than the division of active page data storage space by a constant can be used to determine the allocation of data storage space to a series of pages stored in caches 354 and 356.

  In one or more embodiments, a new active page can be selected instead of the page 10 shown and shown in FIG. The new selected page can be the page directly adjacent to page 10 (either page 9 or page 10), but it need not be. That is, any page from the first page to the last page in the relevant book (or all other types of publications with discrete pages) can be the new active page.

  In one or more embodiments, a transition between a currently active page and a new active page is preferably made upon selection of a new active page. This transition to a new active page may include acquiring additional image data for the new active page to allow the new active page to be stored and / or displayed at full resolution. If the new active page is “page 11” and the “double” embodiment described above is used, the amount of data storage space allocated to page 11 is preferably doubled. Continuing with the application of the “twice” embodiment, the data storage space allocated to page 10 is preferably halved as part of the transition away from page 10 toward page 11 as the active page. Data of the active version of the page 10 that is not included in the page 10 after the transition can be discarded (this discarding can include overwriting it). Instead, however, the “extra” data for this page 10 can be stored in another cache. Such caching of excess data on page 10 may provide efficiency if the transition to page 10 occurs immediately after the transition from page 10 (ie, within a moderate number of page transitions). it can.

  In one or more embodiments, the transition from page 10 to page 11 (or other new active page) includes a gradual fade-out from page 10 and a gradual fade-in to page 11. The user of the virtual book 350 can be given a satisfying experience and / or an experience to extend physical page transitions. Optionally, a sequence of images can be provided that shows folding and turning the old active page so that the virtual page transition appears to further extend the physical turning of the page.

  56 is a diagram of a three-dimensional version of the virtual book of FIG. 55 according to one or more embodiments of the invention. The embodiment of FIG. 56 shows how an alpha channel for partial transparency (coarse edges) can be stored as image information in addition to the red, green, and blue color components. Although the color components are described above, for convenience, only the black and white rendition of the image of FIG. 56 is provided herein.

  In one or more embodiments, hardware-accelerated texture mapping can be used to support the alpha channel. Another feature that can be practiced in connection with either the two-dimensional or three-dimensional embodiment of the virtual book is the dynamic deformation of the image, eg, when turned, as shown in FIG. Bending a book page.

<Management of image data on one or more portable devices>
This section describes several mechanisms for storing and interacting with digital images based on progressive and interactive visual collection transmission. In one or more embodiments of the present invention, a variation based on the method disclosed herein is a desktop computer, mobile device, or other device of a large collection of images stored on a second mobile device. Enables near-instant viewing, using remote storage to augment the mobile device's local memory to view images, and browsing large image collections from the mobile device. The various permutations enabled by one or more embodiments of the present invention may rely on a common client / server imaging and collection representation architecture.

  One or more embodiments of the present invention provide a collection of digital images or other visual objects on a server, establish communication between a client and the server, and a visual resident on the server. Enabling a multi-scale navigation by a client of a collection of objects can be provided.

  In this disclosure, the term “digital image data” may include digital photographs, digital images, visual documents, or other forms of visual content. As used herein, the term “image” generally corresponds to the term “digital image”, and both of these terms can correspond to “digital photographs”. As used herein, the term “client” generally corresponds to the term “client side” and the term “client device”. As used herein, the terms “portable device”, “portable camera device”, and “camera device” generally refer to a digital image capturing device and / or a digital image storage device. In this specification, “digital image capturing device” refers to a digital camera, a camera-compatible mobile phone (sometimes referred to as a camera-compatible mobile phone), a mobile information terminal, and / or a digital video recorder capable of recording a digital still image. It can include, but is not limited to. A “digital image capturing device” can include devices that can record such data by optically receiving and recording the image data directly (eg, using a standard digital camera), wired or wireless. Devices that can receive image data over other Internet or other network connections can also be included.

  One or more embodiments of the methods described herein address the issue of storing, synchronizing, browsing, and organizing a collection of digital image data that can be a visual document using a multi-resolution approach. Can be dealt with. Digital photographs that can be represented as an array of color pixels at a certain resolution (eg, 1024 × 768 pixels = 0.75 megapixels, 2592 × 1944 pixels = about 5 megapixels, etc.) can be used by end users among other devices. A common visual document type that can be created in large quantities using digital cameras, camera-enabled cell phones, and digital video recorders.

  One or more of the methods described herein may include road map or other vector data from Applicant Reference Document 489 / 17NP, or text from Applicant Reference Document 489/13. It can also be applied to visual data objects other than images, such as data (both documents are identified in more detail at the beginning of this document, both documents are incorporated herein by reference) .

  A problem faced by users of existing systems is that the camera device can quickly create a large number of potentially large visual documents. However, these devices typically do not have sufficient memory or visual browsing capabilities to allow satisfactory archiving, viewing, or organization of these documents.

  Digital photos or other digital image data stored in a camera or other portable device is typically periodically downloaded to a desktop or notebook computer to allow the camera to take more photos. Cleared from memory, organized and / or viewed on a desktop or notebook computer. The digital photos can then be shared with friends by posting a selection of digital photos to one or more Internet sites.

<Traditional methods for managing image data on portable devices>
When using conventional techniques for managing image data on portable devices, the following steps can be followed. First, a mobile device, which can be a digital camera or other digital image data capturing device, takes a picture. The photos are then downloaded to the camera user's PC (personal computer) and deleted from the camera device, potentially after some sort of those photos. The local storage of the camera device may be limited and this conventional approach only holds the image temporarily until it is securely stored on the PC.

  A PC can permanently store any subset of digital photos in its memory (eg, a hard disk drive or other non-volatile storage). Users can upload certain further selected subsets of these images to a web server, usually at a reduced resolution, which shall be owned by the web publishing service Can do. Uploaded images are publicly viewable by using a web browser on a PC or other device, by all third parties, or by a subset of these users with restricted access rights Can be able to.

  Limitations of existing approaches can include lengthy download times from the camera device to the PC. There is also usually poor management of persistent storage on camera devices. Camera devices usually have a small color display on which viewers can theoretically store permanently stored images (family members of the same type that people typically carry in their wallets. And images of pets) and photos associated with the caller or their contacts on a PDA (personal digital assistant). However, the limitations on persistent storage in existing camera devices make it difficult to accomplish the above tasks.

  In addition, existing camera devices impose other limitations. In existing camera devices, navigation of images stored in the camera device is generally awkward and difficult. Existing camera devices lack a unified visual interface to image collections that should give the user a consistent experience either on the camera device or on the PC. Existing camera devices tend to place very restrictive limits on the number of photos that can be stored on a device before it needs to be downloaded. Thus, when using existing techniques, a lengthy sequence of steps is generally involved in making an image available to a third party.

<Management of image data according to one or more embodiments of the present invention>
FIG. 57 is a block diagram of a system 550 that manages image data communication between one or more portable devices 562, 572 and one or more other computers according to one or more embodiments of the invention. It is. The system 550 can include a client side 560 and a server side 570. However, in an alternative embodiment, the client grouping and server status of the device grouping shown in FIG. 57 can be reversed.

  In one or more embodiments, system 550 includes portable device 1 562, portable device 2 572, personal computer 182 (which may be substantially identical to client computer 182 of FIG. 53), server 188 (FIG. 53 server computers 188) and / or additional computers 574 may be included. Each of devices 562, 572 and computers 182, 188, and 574 preferably include a memory and one or more displays along with them. Alternatively or additionally, the device and computer of FIG. 57 can communicate with memory and / or a display.

  FIG. 57 illustrates various possible data paths that can be used in accordance with one or more embodiments of the present invention. One or more embodiments may use fewer data paths than all of the data paths shown in FIG. The available data paths shown in FIG. 57 may have one or more of the following features in common: 1) The data path is server side 570 (image data originator) and client side. 560 (image data recipient) can be used. 2) Bi-directional data paths (illustrated by lines with arrows at both ends) are either the client function or the server function indicated by the device pointed to by these arrows. 3) Connections can be hardwired networks (eg Universal Serial Bus (USB), Firewire®, or Ethernet) or wireless networks (eg for nearby devices) Bluetooth (registered trademark), WiFi or wireless wide area networking protocols) can be used for more remote connections and / or 4) the connections shown may or may not be ad hoc.

  In one or more embodiments, both client side 560 and server side 570 include one or more digital computing devices and / or including but not limited to camera devices, personal computers, and personal digital assistants. Digital storage devices can be included.

  In one or more embodiments, a client device (client) can have one or more displays. The client is described in Applicant Reference Document 489 / 15P (Patent Document 7 entitled “Method for Efficiently Interacting with Dynamic, Remote Photo Albums with Large Numbers of Potentially Large Images” incorporated herein by reference). One or more of efficient multi-resolution browsing methods can be used to browse a collection of documents residing on the server. These methods allow a large collection of large images or other visual documents to be efficiently navigated over a low bandwidth connection. Such image collection zooming, panning, and dynamic relocation are described in this referenced document.

  In one or more embodiments, one of the properties of this navigation method is that the display content can be progressively focused as information is transmitted from the server to the client. The speed at which this information is focused can be controlled by the ratio of connection bandwidth to display pixel. When the user zooms, pans, or rearranges the document on the client side 560 so that the new content is visible, this content still appears blurred and then in focus.

<Virtual display>
In one or more embodiments, the client “display” does not necessarily have to be physical or visible to the end user. In one or more embodiments, the display can be an “virtual display”, ie, an abstract model of a display having a specified resolution. Such a “virtual display” can be represented as an array of pixel values in the client's memory, regardless of whether the pixel values are rendered on the screen. The virtual display can include wavelet data that at least partially describes one or more images. The wavelet data is preferably capable of representing an image with a range of possible resolutions. In one or more embodiments, the wavelet data can correspond to that using JPEG2000. In one or more embodiments, the virtual display can include sufficient wavelet data to fully describe the one or more images.

  For example, if it is desirable for a device to obtain all thumbnails of images in a collection at a specified resolution, this device creates an appropriately sized “virtual display” and connects to the server. You can establish and request a view of the entire collection. The full set of thumbnails can then be sent to this “virtual display” where it can be rendered. If transmission is interrupted before all of the relevant data is sent from the server to the client, the client's virtual display still does not have all of the thumbnail images in full focus. It should be. However, it should be preferred that all of the requested thumbnail images be stored in the client's virtual display at a resolution sufficient to allow rendering of a visible version of these images on the screen. An image rendered in the manner described will generally have a lower visual quality than if the transmission of the image ended without interruption. Thus, some image degradation may be present in an image rendered using data from an incomplete interrupted transmission.

  Nevertheless, the described degradation is preferred over the prior art approach of transmitting a set of thumbnails across the network (a complete image of each thumbnail is transmitted in turn). Under this prior art approach, premature interruption of connectivity results in some thumbnails being fully usable (ie, full resolution) and other thumbnails being completely unusable. It should bring FIG. 58 shows this difference. FIG. 58A shows the result of incomplete image data download using existing techniques, and FIG. 58B shows the result of incomplete image data download according to one or more embodiments of the present invention.

  FIG. 55A shows a prior art scenario where all three thumbnail data (illustrated using a square shape) have been received and the remaining nine thumbnails (illustrated using X) have not been received at all. Show. FIG. 55B shows that all 12 thumbnails (shown as hatched square shapes) have been received at a certain level of resolution, and this level of resolution is preferably satisfactory for viewing, but is not completely interrupted. Fig. 4 illustrates a situation that may arise from using one or more embodiments of the present invention that are likely to be less than the resolution that would be obtained after completion of data transmission.

  In one or more embodiments, the client may have a client-side cache that caches recently viewed visual content. A standard-recently-used (MRU) cache can be used for the caching needs of one or more embodiments of the present invention. However, the cache disclosed in U.S. Patent No. 6,057,058 (client reference document 489 / 10NP), which is incorporated herein by reference and entitled "Efficient Data Cache", enables more sophisticated client-side caching. Can be used beneficially. In either case, a given amount of client-side memory can be directed to the cache. Thus, navigation back to a recently viewed image may allow the image data stored in the cache to be used rather than requiring this image data to be retransmitted from the server.

  One client can have multiple displays. A given display can be a physical display or a virtual display. A given display can be driven directly by user input or can be driven programmatically by software in a client computer, such as computer 182. The total size in pixels of all displays can be fixed or limited by some limit, which can define the minimum amount of client-side memory required for visual content. This client side memory is preferably separate from the storage space allocated to the cache memory.

  An embodiment using both a physical display and a virtual display is described below. The physical display in the client device is visible to the user and preferably allows zooming and panning navigation through a collection of digitally stored images and relocation of this collection. The user can also select one or more images from this collection and send them to a “holding pen” that can serve as a place to store the user's selected images. The holding pen can be visualized in some way on the physical display. Adding an image to the holding pen preferably causes the image to be placed on a virtual display that may be invisible to the user. As the image is added to the holding pen, the virtual display representing the holding pen is progressively filled.

  The virtual display can increase to a certain limit in size (measured in number of pixels), and after reaching this limit, the size can remain fixed at this limit. The virtual display may be too small to display all of the images in the holding pen at full resolution. In this case, the data storage space required for images residing in the virtual display is preferably reduced as necessary to fit the image on the virtual display. Thus, the off-screen view (virtual display) is preferably augmented when placing an image visible to the user on the holding pen. This off-screen view augmentation can be made invisible to the user.

  A method of browsing is disclosed in US Pat. No. 6,089,059 (Applicant Reference Document 489 / 2NP) entitled “System and Method for Exact Rendering in a Zooming User Interface”, which is incorporated herein by reference. The method disclosed in that document that determines the order in which information is transferred from the server to the client based on the client's view can be modified for multi-display scenarios. The 489/2 NP document discloses that visual information can be decomposed into tiles, each tile covering a region in space at a given resolution. Low resolution tiles can occupy a large physical area, high resolution tiles can occupy a smaller physical area, and the amount of information in each tile can be substantially the same.

  The 489 / 2NP document discloses a method for ordering tiles using the criteria described below. One criterion may be tile resolution and tile position on the display. The sorting of the tiles is lexicographic, so that lower resolution tiles always precede higher resolution tiles, and the spatial position only plays a role in resolving order within a certain resolution ( The lexicographic sort is referred to herein as a generalized tuple meaning, for example a set of triples {(1,2,3), (0,3,1), (4,0,0 ), (0, 0, 1), (0, 3, 2)} are (0, 0, 1), (0, 3, 1), (0, 3, 2), (1 , 2, 3), (4, 0, 0).

  Alternatively, non-lexicographic sort criteria can be used. For example, tiles can be sorted using a linear combination of properties. Such properties can include, but are not limited to, resolution (which can be expressed in logarithmic units) and tile distance from the center of the display. In this specification, the term “sort key” corresponds to the term “sort criterion”.

  In this embodiment, lower resolution tiles can be sent in preference to higher resolution tiles, tiles closer to the center of the display can be sent in preference to tiles closer to the periphery, These properties can be traded off against each other.

  It is preferable that minor changes can be implemented to adapt the above scheme to a multiple display scenario. In one embodiment, the display number can be added as an extra lexicographic sort key. Thus, the first display can be fully refined (according to other sort keys) before any tiles are transmitted in connection with the second display.

  In another embodiment, the display number can be an additional variable included in the linear combination, whereby the display number trades off in some way for resolution and proximity to the center of the display. It becomes possible. In another embodiment, the display numbers can coexist in an imaginary “common space” and the resolution sort key and the center proximity sort key can be used as before. This “common space” is a conceptual space that establishes an imaginary spatial relationship between multiple displays as if they were areas of a single larger display. By defining this imaginary spatial relationship, all the parameters necessary to prioritize tiles within multiple displays are determined.

  FIG. 59 is a block diagram of a “common space” 740 that may include one physical display (screen) 742 and two virtual displays 744, 746 according to one or more embodiments of the present invention. The physical display 742 is preferably a normal size at the center of the “common space” 740. Virtual displays V1 744 and V2 746 are preferably offset laterally, and V2 is preferably scaled down so that the pixel is half the linear size of a physical display pixel. . It is preferred that, assuming a purely lexicographic tile sort order, the content at each resolution level in V1 746 is sent from the server to the client after the corresponding resolution of the physical display (V1 is the physical display Means farther from the center of this space than any of the above points). The resolution in V2 746 can be transmitted after all tiles with twice the resolution are transmitted for both physical display 742 and V1 744. Note that the “common space” 740 need not correspond to any actual larger display or memory address space. The “common space” 740 is merely a conceptual convenience for establishing a relationship between tile priorities across different displays.

  Obviously many trade-offs are possible. These trade-offs, like the lexicographic example above, give top priority to the refinement of the physical display 742, while at the same time excess time and bandwidth that is not necessary to focus on the physical display. All can have the result of using them to continue the refinement of the virtual displays 744, 746. These trade-offs can instead begin to refine the virtual display after fully focusing the physical display rather than fully. After fully focusing on the physical display 742, the physical and virtual displays 744, 746 can share bandwidth resources to match and refine.

  If the images in the collection are JPEG2000 images, all subsets of the data for a given image itself can contain JPEG2000 image files. During image navigation, the client can progressively download image data from the server, which augments the quality of the client subset of the image and makes JPEG2000 files an increasingly accurate approximation of the full image. Give clients the ability to create.

  If the client has navigated to all locations in the image, or has viewed the entire image in full resolution long enough for all of the image data to be transmitted, the client can view the original JPEG2000 for that image. The entire file can be recreated. If a client zooms close to just a portion of a large image, the client can still create a JPEG2000 file, but lacks details everywhere except where the client zoomed in. Should be. This property of JPEG2000 can be extended to other multi-resolution document types. If the client has never zoomed in beyond a given resolution, information about that image content that exceeds that given resolution should not be available. In this case, the version of the JPEG 2000 image that can be created and / or stored by the client may have a lower overall resolution than the original version of the image.

  One application of the virtual display scenario described above is to remedy the problem of long download times of images from cameras. In one or more embodiments, the camera or camera-enabled mobile device can operate as a server and the PC can operate as a client.

  In one or more embodiments, instead of initiating a time-consuming batch download of all images to the PC, when the camera and PC are connected, the PC You can browse the complete set quickly. During navigation, a group of images can be selected and placed on the holding pen. Note that if all images on the camera have to be downloaded in their entirety to a PC, the total time required to achieve the transfer remains the same as in the prior art. However, similar to the closely related issue of thumbnail transmission, this method can provide several advantages over conventional serial download of images, listed and described below. The present invention is not limited to the features listed below.

  Image download and user navigation of a full image set on a camera or other mobile device can be simultaneous and coordinated in its bandwidth usage (in effect, navigation is simply a tile sent from the server to the client. Affects the order in which they are made).

  If the PC display is larger than the mobile device display, which image to download, which image to leave on the mobile device, and which image to download without incurring the delay of downloading the entire set before making a decision A better choice can be made as to whether to discard.

  The experience of browsing on a PC and a mobile device (assuming this mobile device also has a display) is respectively simple and empirically similar, which preferably increases usability.

  If a lower resolution version of the image in the holding pen is desired, it is preferably simple to appropriately limit the details of the downloaded data by reducing the size of the items on the virtual display. Note that this reduction in image size requires a smaller amount of space on the PC while speeding up the download at a higher magnification, ie, 4x for each discarded resolution level. ).

  As desired, the amount of memory allocated to photos on the PC can be limited by limiting the size of the virtual display and reducing the number of images in it. Also, different constraints can be imposed on different photos, so space can be allocated based on novelty or one or more other criteria.

  In one or more embodiments, premature loss of connectivity results in degradation of the quality of some or all of the downloaded images rather than complete removal from the download operation of some images (some are Note that most of the image data volume is very high resolution details, some of which are camera noise, all of which are less critical than the coarser image structure for normal viewing Therefore, it is preferable to transmit high resolution image data of all images after all lower resolution image data of the image has been completely transmitted). For example, hybrid prioritization of image data is also possible, which favors complete download of a subset of photos before proceeding to a second set of refinements beyond thumbnail details.

  In one or more embodiments, one or more methods disclosed herein are resilient to intermittent connectivity. This means that all JPEG2000 objects can continue to be augmented with additional information at any time, while still browsing and interacting with that visual data no matter what visual data has already been received This is because it becomes possible.

  With regard to the above references to a) reducing the size of items on the physical display and b) limiting the amount of memory allocated to photos on the PC, the normal home user will discard any of the images. Note that it may not be desirable (after the initial screening of such images). If such users continue to add enough storage to their PCs, of course, it should not be necessary to destroy any content. Adding storage itself can increase the maximum size of the virtual display. Thus, the above features (a) and (b) can be omitted if a sufficiently large virtual display size can be created (ie, there is sufficient client-side storage available).

  Some form of visual display of completion is desirable because when the "holding pen" image finishes downloading, it may be obscured by the client-side user. As an example, a check mark or green dot can be displayed next to an image when the image has finished downloading. When all the images in the “holding pen” contain green dots, the connection can be disconnected without loss.

  Actions such as using a client computer (which can be a PC) to request that the camera destroy a portion of the image exceed what was contemplated in Applicant Reference Document 489 / 15P, Can benefit from some additional communication from the server to the server. In one or more other embodiments, the client side activates its own client side on the server side (which can be a mobile device such as a digital camera or mobile phone) and receives content from the PC. Can also be told to create their own view.

  This is similar to the “push” method developed in the context of the World Wide Web. The PC can render a “view” of the camera / cell phone of the content on the PC, and thus displays (for example) the green completion dot described above for images uploaded from the PC to that camera. be able to. Each of the reciprocal arrows of FIG. 57 can be implemented using either a “push” or “pull” arrangement. Specifically, viewport settings, placement, and other navigation settings can be controlled from either the client side 560 (“pull”) or the server side 570 (“push”). Users interacting with one device can be interconnected to another device, which allows both “push” and “pull” to occur simultaneously.

  We will now enumerate the potential client-server connections shown in FIG. 57 and briefly explain how they can be used and why they are useful.

  A mobile device 562, which can be a camera or a camera-enabled mobile phone, can provide content to a user's PC (personal computer) 182. This connection can usually be made via a USB cable or a Bluetooth® ad hoc wireless network. Profit explained above.

  The PC 182 can supply the content back to the mobile device 562. This can be particularly useful for the following applications.

  “Photos in the wallet” can be sent from the PC to the camera or mobile device even if these photos were not taken by the mobile device.

  The PC can be a home appliance without a display and the mobile device can be used as the main visual interface to the archived visual material. The mobile device in this context can be a digital camera, a camera-enabled mobile phone, a PDA, or a mobile tablet PC with a display.

  The first mobile device can be directly connected to another mobile device (“guest”) or form an ad hoc network with the “guest”. The two mobile devices can display and share each other's photos.

  The PC can upload the image (via push) to the remote server. The server can be a photo sharing service, thus implementing the type of space constraints envisaged in the above process of reducing the size of items on the physical display and limiting the amount of memory allocated to photos on the PC can do. The remote server can then supply the collection to one or more additional PCs. Usually this should be a broadband connection. However, other connection types can be used.

  The remote server can also provide the collection to mobile device users. Typically this should be a mobile wireless wide area network.

  A mobile device can upload its image to a remote server via “push” (ie, under the control of the mobile device). In one or more embodiments, this upload can be automatic so that the mobile device is free to transfer the content to the server and localize the content when the transfer is complete. By deleting it, the apparent storage space can be expanded transparently.

  In relation to the last two items above, local caching on the mobile device 562 is very large using only local storage, even if the mobile device 562 has limited local storage. Note that it may be possible to support browsing a thumbnail collection. Zooming in on the details of recently viewed images may also be possible if the relevant information is still in the mobile device's local cache.

  Zooming in on an image whose details are only available on the remote server can result in an image with no blur details. However, if the mobile device is on a network that includes a remote server 188, it can progressively refine the blurred image as more and more detailed image data is downloaded to the mobile device 562. it can. If the mobile device is not connected to a network that can supply additional image data, the image cannot be presented with higher details than are available in the initial thumbnail image.

<Montage of low resolution image>
One or more embodiments of the present invention can be used in various configurations to implement the downloading of selected images and / or image regions at a controllable level of resolution for various applications. Computed steps and interactive rendering algorithms can be defined. Many of these applications (focusing on a region of interest, virtual book, etc.) can involve user interaction with the “universe” of the image.

  Thus, in one or more embodiments, the starting point for pre-computation can be a file name, a URL, or a list of other strings that reference individual images. When a user zooms out far enough to see all of these images at the same time, there can be a very large number of image files, so either the client or server traverses all of the image files That is impractical. For example, there may be tens of thousands or hundreds of thousands of images in a view, with each image occupying 2 × 2 = 4 pixels on-screen. Even if these images support efficient low-resolution access, simply opening and closing 100,000 files involves significant overhead and is not achievable on an interactive timescale. It may be practical. Therefore, it may be desirable to use a cached representation of the low resolution versions of these images, referred to herein as “montages”.

  In one or more embodiments, the montage can be a mosaic or collage of all images rendered at low resolution and efficiently packed into a rectangular area, as shown in FIG. . Auxiliary metadata that can be embedded in the montage image file or stored separately can identify rectangular regions on the montage image that accompany the particular image file.

  In one embodiment, the montage image itself can be navigated using a zoom and pan interface. When the user zooms in enough to run out of the resolution available in the montage version of one or more images in the montage, that image's metadata is sent to the client one or more individual image files. Can be referenced, and the client can use images from these image files to render the image at a higher resolution.

  In one or more embodiments, the overall size of the montage in pixels is limited to zooming in to a stage where only a few images, sometimes referred to herein as “sets” of images, are visible at the same time. You can choose to use up that resolution. Thus, access to more than a few high resolution images is preferably unnecessary at any given time. During subsequent zooms and pans, the image stream can be opened and closed as needed to limit the number of high resolution images that are open at any given time.

  The above approach to high resolution multiple image navigation suffers from the following limitations: The montage layout is preferably designed for clogging efficiency, but the user may want a different arrangement of images on-screen. Furthermore, the user may want to be able to dynamically rearrange the layout of the image on the screen.

  In one or more embodiments, a graphics rendering technique known as “texture mapping” can be utilized to allow such relocation, which is implemented in software. In general, modern personal computers are hardware accelerated. Texture mapping allows a “texture” or part of the source image to be drawn on the display, optionally re-scaling, rotating, and / or performing 3D perspective transformations. Other hardware accelerated transformations are often supported and include color correction or color change, full or partial transparency, lighting, occlusion, and coordinate remap. A low-resolution version of the montage can be used as a “texture” so that when the user zooms out, the individual images in the montage are in any form, as shown in FIG. Can be remapped dynamically. Multiple texture maps can be used, where each texture map can be a montage containing a subset of images. Transitions between placements may or may not be animated. Relocation can be done while the user is zoomed in, but the new image is initially very new because relocation may result in a new zoomed-in view of the image that was previously offscreen. Note that it can be blurry.

  In another embodiment, texture mapping techniques can only be used during dynamic image relocation. When image placement is static, software compositing can be used to assemble all or part of the on-screen higher definition relocated montage. This software synthesis method is particularly valuable in combination with the multi-resolution rendering technique described in US Pat. No. 6,089,086 (Applicant Reference Document 489 / 2NP) previously identified in detail in this disclosure. This method can effectively create a new “display montage” by rearranging the images of the original montage.

  Texture mapping can also be used to display high-resolution images, but in this case, textures that contain tiles of individual images are used, rather than textures that contain montages of multiple images. Is done. This technique is also described in Patent Document 1 (Applicant Reference Document 489 / 2NP).

  In one or more embodiments, montage relocation can be used to support image reorganization that does not rely on texture mapping.

  In one or more other embodiments, texture mapping, software rendering, or any combination of the two can be used to render an image in three dimensions rather than a one-dimensional plane. Dynamic relocation in three dimensions is also possible. Three-dimensional applications can include virtual gallery or other walk-through environments as well as virtual books. A virtual book is described in US Pat. No. 6,099,056, described herein and further incorporated herein by reference.

  FIG. 62 is a block diagram of a computing system 1000 that can be adapted for use with one or more embodiments of the invention. In one or more embodiments, a central processing unit (CPU) 1002 can be coupled to the bus 1004. Further, the bus 1004 may be coupled to a random access memory (RAM) 1006, a read only memory (ROM) 1008, an input / output (I / O) adapter 1010, a communication adapter 1022, a user interface adapter 1006, and a display adapter 1018. it can.

  In one or more embodiments, the RAM 1006 and / or the ROM 1008 may hold user data, system data, and / or programs. The I / O adapter 1010 can connect a storage device, such as a hard drive 1012, a CD-ROM (not shown), or other mass storage device, to the computing system 1000. Communication adapter 1022 can couple computing system 1000 to a local area network, a wide area network, or an Internet network 1024. User interface adapter 1016 can couple user input devices, such as keyboard 1026 and / or pointing device 1014, to computing system 1000. Further, the display adapter 1018 can be driven by the CPU 1002 to control the display on the display device 1020. The CPU 1002 can be any general purpose CPU. Any known processor, programmable digital device or device that operates to execute standard digital circuits, analog circuits, software and / or firmware programs may be used for the methods and apparatus described above and / or described later in this document. Note that any known technology can be used, such as a programmable digital system, a programmable array logic device, or any combination of the above. One or more embodiments of the present invention may also be implemented in a software program for storage on a suitable storage medium and execution by a processing unit.

  Although the invention herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present invention. Accordingly, it will be appreciated that numerous modifications can be made to these exemplary embodiments and that other arrangements can be devised without departing from the spirit and scope of the invention as defined by the appended claims. I want to be.

Method and apparatus for using image navigation techniques to promote commercial transactions

[Technical field]
The present invention is directed to a method and apparatus for application of image navigation techniques in promoting commercial transactions, for example by providing a new environment for advertising and purchasing products and / or services.

[Background]
Digital cartography applications and geospatial applications are fast growing industries. These have attracted rapidly increasing investments from candidates from a number of different market companies, such as FedEx Express, clothing stores, and fast food chains. Over the past few years, cartography has become one of the few software applications on the web (so-called “killer applications”) that have generated great interest, along with search engines, web-based email, and marriage brokerage.

[Patent Document 10] US Patent Application No. 10/803010
[Patent Document 11] US Provisional Patent Application No. 60/474313
[Non-Patent Document 4] www.w3data.com
[Disclosure of the Invention]
[Problems to be solved by the invention]
Mapping must in principle be very visual, but at present, the usefulness of mapping for the end user is almost entirely in the generation of driving directions. A map image with a driving direction is always rendered incompletely, conveys little information, cannot be navigated conveniently, and is a decorative decoration. Clicking on the pan control or zoom control causes a long delay, during which time the web browser becomes unresponsive and then a new map image appears with little visual relationship to the previous image. In principle, a computer must be able to navigate a digital map more efficiently than a paper map book, but in fact, visual navigation of a map by a computer is still inferior.

[Best Mode for Carrying Out the Invention]
The present invention is intended to be used in combination with a new technology that allows continuous and quick visual navigation of a map (or any other image), even when over a low bandwidth connection. . This technology relates to a new technique for continuously rendering a map in a pan and zoom environment. This technology makes it possible to draw road networks (1D curves) and dot marking points (0D points) at all scales, giving the illusion of a continuous physical zoom while simultaneously displaying the “visual density” of the map. Is an application of fractal geometry to the rendering of lines and points. Related technologies apply to text labels and icon context. This new approach to rendition prevents effects such as the sudden appearance and disappearance of the path during zooming, which are typical adverse effects in digital map drawing. Details of this navigation technology can be found in US Pat. No. 6,057,059 (Attorney Docket No. 489/9) entitled “METHODS AND APPARATUS FOR NAVIGATING AN IMAGE”, filed on the same day as this application, the entire disclosure of which is incorporated herein by reference. ) Can be seen. This navigation technique may be referred to herein as “Voss”.

  The use of Voss technology enables multiple new commercially valuable business models for Internet mapping. These models are both based on Yahoo! The proven success of businesses such as (registered trademark) Map and MapQuest (registered trademark) is adopted as the starting point. But our approach takes advantage of new technology's ability to go well beyond advertising and add substantial value to both the company and the end user. The essential idea is to allow companies and people to rent “real estate” on a map that can embed zoomable content, usually at their physical address. This content appears as an icon shape when viewed on a large-scale map, that is, a golden arch in the case of McDonald's®, but smoothly into any kind of web-like content when viewed nearby. Deforms continuously. Thus, by integrating our mapping application with dynamic user content and company / residential data, we can enable certain types of “geographical world wide web”.

  In addition to a huge horizontal market for both consumers and retailers, collaboration with existing geographic information service (GIS) providers and related industries includes car navigation systems, mobile phones and PDAs, real estate rental or sales, You will get departmental advertising, as well as more. Possible business relationships in these areas include technology licensing, strategic partnerships, and direct vertical sales.

  The functionality of the new navigation technique of the present invention is described in detail in the aforementioned US patent application. With regard to this patent application, the most relevant aspects of the basic technology are as follows.

-Smooth zoom and pan through a 2D world with perceptual continuity and advanced bandwidth management;
-An infinite precision coordinate system that allows visual content to be nested without restrictions;
-The ability to nest content stored on a number of different servers, making spatial inclusion equivalent to hyperlinks.

  With regard to the last two elements, additional details can be found in “SYSTEM AND METHOD FOR INFINITE PRECISION COORDINATES IN A ZOOMING USER INTERFACE” filed May 30, 2003, the entire disclosure of which is incorporated herein by reference. It can be seen in the patent document 11 entitled.

  The map consists of multiple layers of information, and ultimately the Voss map application allows the user to switch most of these layers on and off, making the map highly customizable. The layers include the following:

1. Road and
2. Waterways,
3. Administrative divisions,
4). Orthographics based on aerial photos (aerial photos digitally “undistorted” to fully tile the map),
5. Topographic maps,
6). For example, the location of public infrastructure, such as schools, churches, public telephones, and bathrooms,
7). Each of the above labels,
8). With cloud cover, rainfall, and other weather conditions,
9. Traffic conditions,
10. With advertisement 11. Personal user content and commercial user content.

  The most prominent layers from the point of view of normal users are 1-4 and 7. The advertisement / user content layers 10-11 are particularly important in the present application, but are also very important. Many of the map layers, including 1-7, are already available from the US federal government with high quality and negligible cost. Value-added layers such as 8-9 (and other layers) can be enabled at any time during development or even after deployment.

  Several companies provide enhanced road data with annotations indicating one-way street, highway entrance and exit ramps, and other features important to generate driving directions. It is not about visual geography. The most relevant commercial geographic information service (GIS) offering for this application is the geographic classification, which allows the conversion of addresses to precise latitude / longitude coordinates. The availability of regional classification services was determined not to be prohibitive.

  In addition to map data, national yellow page / white page data may also be valuable in the implementation of the present invention. This information can also be licensed. National yellow / white page data can be used in combination with regional classifications to enable geographic user search or filtering (eg, “Highlight all restaurants in Manhattan”) for a company. Perhaps most importantly, directory listings combined with geographic classifications greatly simplify the association of companies and personal users with geographic locations and lend or transfer "real estate" via online transactions. Enabling the need for large sales staff.

  National telephone databases and national address databases designed for telemarketing can be obtained inexpensively on CDs, but these are not always high quality and their coverage is usually very partial, and these databases Is often old. Several companies offer robust directory servers with APIs designed for software-oriented businesses similar to ours. Among the best is W3Data (see, for example, Non-Patent Document 4), which starts at $ 0.10 / hit with a minimum of $ 500 / month and $ 250,000 / month for $ 0.05 / month. Hits, offering a what they call “near-time” national phone listings, using an XML-based API, at a rate that drops to $ 0.03 / hit for a volume of 1000000 cases / month. The entire United States and Canada are covered. It is also possible to look up the name by reverse querying, i.e. by providing a phone number. “Near Time” data is updated at least every 90 days. Combined with 90-day caching at our end of already acquired entries, this is a very economical form of obtaining a high quality national listing. “Real time” data that is updated every night is also available, but is more expensive ($ 0.20 / hit). Real-time data is the same as that used by the 411 operator (phone number guidance).

  There are also service providers that can create national business listings by category with comparable business and pricing models similar to W3Data.

  “Media player” models with classic ad-based business models and proprietary data formats and downloadable plug-ins (such as Flash®, Acrobat®, and Real Player) are typically Face the egg problem. The place of advertising will only be worth advertising if people are already watching it, and the plug-in will be downloaded only if there is useful content to watch (even if it is free) And content will only be attractive to investment in creation if there is already an installed user base ready to view it.

  The Voss mapping application requires downloadable client software and at the same time generates revenue via advertising, but does not suffer from the disadvantages of a classic advertising-based business model. Even before substantial commercial space is “rented”, the present invention provides a useful and visually compelling form of displaying maps and searching for addresses, ie, the functionality of existing mapping applications. Provides a similar but significantly improved visual interface. Furthermore, the inventive approach provides a limited but valuable service to non-commercial users free of charge and attracts a user base. This limited service consists of hosting a small amount (5-15 MB) of server space per user at the user's geographic location, usually at home. The client software can include a simple authoring function that allows users to drag and drop images and text into their “physical address”, and this image and text can be used with any client software It can be seen by other authorized users (password protection can be enabled). The zooming user interface approach is obviously useful for navigating digital photo collections, especially over limited bandwidth, so the possibility of photo album sharing alone can attract a significant number of users. Additional server space can be made available at a reasonable annual fee. This very horizontal market is likely to be a great source of income.

  Thus, the usual chicken and egg problem is avoided by providing valuable services from the outset, ie by techniques that have worked well for search engines and other useful (currently profitable) web services. This is in sharp contrast to, for example, an online dating service that is not useful until a user base is accumulated.

  In accordance with various aspects of the present invention, revenue sources can include:

1. Commercial “rental” of the space on the map corresponding to the physical address,
2. Fees for “plus service” (defined below) for commercial users,
3. "Plus service" charges for non-commercial users,
4). Professional zoomable content authoring software,
5. Licensing or partnerships with vendors and service providers such as PDAs, mobile phones, car navigation systems,
6). information.

  A basic commercial rental of the space on the map can be priced using a combination of the following variables:

1. The number of sites on the map,
2. Map area per square meter (“footprint”),
3. The desirability of real estate based on aggregate viewing statistics,
4). Server space required to host content in MB units.

  Plus services for commercial users are for franchises, companies that want to do e-commerce or other more sophisticated uses of the web, and companies that want to increase advertising attention.

  1. Larger visible height-can provide multiple levels, visible but unobstructed icons or “flags” from business sites farther away (more zoomed out) than otherwise visible It becomes possible to indicate the position.

  2. Focus priority-Voss focuses on areas of the image when data becomes available during navigation. By default, all visual content is treated equally and focusing proceeds out of the center of the screen. Focusing priorities allow commercial content to be focused more quickly than without it, making the content more noticeable in the user's “peripheral vision”. This feature is tailored to give commercial value without compromising the user's navigation experience.

  3. Including conventional web hyperlinks in zoomable content—these are clearly marked (eg, using conventional underlined blue text) and open a web browser when the user clicks. We can either charge for including such hyperlinks, or charge per click, such as Google (R).

  4). Let the rented geographic area refer to an external commercial server (which itself hosts zoomable content of any type and size)-this is a more elaborate version of # 3, a map Allows any kind of e-business to be conducted. We can still either charge a flat fee or charge for each user connected to an external server (the latter no longer requires a click, just zoom in) I need it).

  5. Billboard—Similar to real life, many high attention areas of the map have substantially empty space. The company can purchase this space and insert hyperlinks and content containing “hyperjumps” that, when clicked, map the user through the space when clicked. Jump to a commercial site in another location. In contrast to conventional commercial space, billboard space does not need to be rented at a fixed location, which can be generated on-the-fly during user navigation.

  This last service raises issues of "ecology" or visual aesthetics and map usability. While it is desirable that the maps remain attractive and remain a true service to the user, this implies limitations on advertising and tasteful “partition regulation”. If the map becomes too confused with billboards or other content that does not reflect real world geography, the user will be distracted and the value of the map as a place for advertising and e-commerce will decline.

  Many values of these commercial “plus services” can be made quantitatively demonstrable when we use statistics are collected. Quantitative evidence of competitive advantage must increase sales of these additional features.

  Plus services for non-commercial users consist of parts of the same product that are scaled, priced and marketed differently.

  Limited authoring of zoomable Voss content is possible from within the free client. This includes inserting text, dragging and dropping digital photos, and setting passwords. Professional authoring software can be a modified version of the client designed to facilitate the creation of hyperlinks and hyperjumps and the insertion of custom applets while allowing more flexible zoomable content creation.

  The use of the present invention can generate a large amount of aggregate and individual information regarding spatial attention density, navigation routes, and other patterns. These data have commercial value.

1 is a block diagram illustrating a communication system including a proxy server and a client device according to one or more embodiments of the present invention. 1 is a front view of a front side of a client device, which can be a mobile phone, customized according to one or more embodiments of the present invention. FIG. FIG. 6 is a front view of the back of a client device, which can be a mobile phone, customized according to one or more embodiments of the present invention. FIG. 6 is a front view of the back of a client device, which can be a mobile phone, customized according to one or more embodiments of the present invention. 4 is a flow diagram illustrating a method for navigating image data by a client device according to one or more embodiments of the invention. FIG. 6 illustrates a web page column as seen on a client device according to one or more embodiments of the present invention. FIG. 5 illustrates the web page of FIG. 4 after an approximately 2 second load at a rate of 20 kilobits per second, in accordance with one or more embodiments of the present invention. FIG. 5 illustrates the web page of FIG. 4 after a load of about 10 seconds at a rate of 20 kilobits per second, in accordance with one or more embodiments of the present invention. FIG. 5 illustrates the top of the web page of FIG. 4 after loading data at 20 kilobits per second for approximately 2 seconds according to one or more embodiments of the present invention. FIG. 5 illustrates the top of the web page of FIG. 4 after loading data at 20 kilobits per second for approximately 10 seconds, in accordance with one or more embodiments of the present invention. FIG. 7 is a zoomed-in view of a portion of the web page of FIG. 4 captured immediately after capturing the view of FIG. 6, in accordance with one or more embodiments of the present invention. 5 is a portrait view showing a portion of the web page of FIG. 4 in accordance with one or more embodiments of the present invention. 5 is a landscape view showing a portion of the web page of FIG. 4 in accordance with one or more embodiments of the present invention. FIG. 7 is a block diagram illustrating a computer system that is adaptable for use with one or more embodiments of the invention. FIG. 2 shows a LOD pyramid (in this example, the base of the pyramid representing the highest resolution representation is a 512 × 512 sample image, and successive reductions of this image are shown at 2 ×). FIG. 4 shows a flow chart used in an exemplary embodiment of the invention. 3 is another flow diagram illustrating how the system displays a final image after zooming. FIG. 2 is a diagram illustrating the LOD pyramid of FIG. 1 with the addition of grid lines indicating subdivisions into rectangular tiles of equal size for each LOD sample unit. 4 is another flow diagram illustrating the process of displaying rendered tiles on a display used in connection with the present invention. FIG. 3 illustrates a concept referred to as irrational tiling described in more detail herein. As described more fully above, it is a diagram illustrating a composite tile and the tiles that make up the composite tile. FIG. 5 illustrates one of the successive stages of rendering in a forbidden order of the second layer of the LOD pyramid of FIG. 4 in accordance with one or more embodiments of the present invention. FIG. 5 illustrates one of the successive stages of rendering in a forbidden order of the second layer of the LOD pyramid of FIG. 4 in accordance with one or more embodiments of the present invention. FIG. 5 illustrates one of the successive stages of rendering in a forbidden order of the second layer of the LOD pyramid of FIG. 4 in accordance with one or more embodiments of the present invention. FIG. 5 illustrates one of the successive stages of rendering in a forbidden order of the second layer of the LOD pyramid of FIG. 4 in accordance with one or more embodiments of the present invention. It is a figure which shows the visual content on a display. It is a figure which shows the image of the visual content of FIG. 18 of a different detail level. It is a figure which shows embodiment of this invention. FIG. 4 illustrates an exemplary embodiment of the present invention showing multiple nodes on a display. FIG. 22 shows a tree diagram corresponding to the exemplary embodiment shown in FIG. 21. It is a figure which shows the block diagram corresponding to a part of tree diagram of FIG. FIG. 6 shows an image taken from the MapQuest website at a zoom level of 5; FIG. 6 shows an image taken from the MapQuest website at zoom level 6; FIG. 6 shows an image taken from the MapQuest website at zoom level 7; FIG. 6 shows an image taken from the MapQuest website at a zoom level of 9; FIG. 4 illustrates an image of a long island made at a zoom level of about 334 meters / pixel according to one or more aspects of the present invention. FIG. 6 shows an image of a long island made at a zoom level of about 191 meters / pixel according to one or more further aspects of the present invention. FIG. 5 shows an image of a long island made at a zoom level of about 109.2 meters / pixel according to one or more further aspects of the present invention. FIG. 6 shows an image of a long island made at a zoom level of about 62.4 meters / pixel according to one or more further aspects of the present invention. FIG. 6 shows an image of a long island made at a zoom level of about 35.7 meters / pixel according to one or more further aspects of the present invention. FIG. 6 shows an image of a long island made at a zoom level of about 20.4 meters / pixel according to one or more further aspects of the present invention. FIG. 6 shows an image of a long island made at a zoom level of about 11.7 meters / pixel according to one or more further aspects of the present invention. 6 is a flow diagram illustrating process steps that may be performed to provide smooth and continuous navigation of images according to one or more further aspects of the present invention. 6 is a flow diagram illustrating additional process steps that can be performed to smoothly navigate an image in accordance with various aspects of the invention. 6 is a log-log graph showing line width in pixels vs. zoom level in pixels / pixel, illustrating physical scaling and non-physical scaling, according to one or more further aspects of the present invention. FIG. 38 is a log-log graph showing a variation of the physical and non-physical scaling of FIG. FIG. 4 shows an anti-aliased vertical line whose endpoints are accurately centered on pixel coordinates. FIG. 6 shows a tilted anti-aliased line that is not positioned so that the endpoints are in exact pixel coordinates. The line width versus zoom of FIG. 37 including a horizontal line indicating incremental line width and a vertical line spaced so that the line width across the interval between two adjacent vertical lines does not change more than two pixels It is the log-log graph which shows a level. 4 is a graph illustrating the data storage size of individual cache objects in the cache as a function of the freshness of use of the cache objects in the cache, according to one or more embodiments of the invention. 6 is a graph showing the cumulative sum of data storage space occupied by cache objects in a cache plotted against the number of cache objects “N” to be summed, according to one or more embodiments of the invention. FIG. 3 is a block diagram illustrating a data cache including a plurality of cache objects according to one or more embodiments of the invention. FIG. 45 is a block diagram illustrating the data cache of FIG. 44 in which the cache object has been resized according to one or more embodiments of the present invention. FIG. 46 is a block diagram illustrating the data cache of FIG. 45 in which the accessed cache object has been resized and restored to the cache in accordance with one or more embodiments of the present invention. 3 is a flow diagram illustrating a method for accessing a cache object in a data cache and updating the data cache, according to one or more embodiments of the invention. 1 is a block diagram illustrating a computing system that is adaptable for use in accordance with one or more embodiments of the present invention. FIG. 6 illustrates a “Ulysses” full-text text image using raw ASCII encoding according to one or more embodiments of the present invention, where white has a numeric value of 0 and black has a numeric value of 255. FIG. 6 illustrates a first five chapter text image of Ulysses encoded using raw ASCII according to one or more embodiments of the present invention. FIG. 6 illustrates a first five chapter text image of Ulysses encoded using frequency-order encoding according to one or more embodiments of the present invention. FIG. 6 is a block diagram illustrating a computing system that can be adapted for use with one or more embodiments of the present invention. FIG. 2 is a block diagram illustrating a system that can be connected to enable communication of image data for multiple computers according to one or more embodiments of the present invention. FIG. 6 is a block diagram illustrating an image having at least two regions of interest therein according to one or more embodiments of the invention. FIG. 3 is a block diagram illustrating a “virtual book” using aspects of the technology disclosed herein, according to one or more embodiments of the invention. FIG. 56 illustrates a three-dimensional version of the virtual book of FIG. 55 according to one or more embodiments of the present invention. 1 is a block diagram illustrating a system for managing image data communication between one or more portable devices and one or more other computers in accordance with one or more embodiments of the invention. FIG. It is a figure which shows the result of incomplete image data download using the existing method. FIG. 4 illustrates incomplete image data download results according to one or more embodiments of the invention. FIG. 6 is a block diagram illustrating a “common space” that may include one physical display (screen) and two virtual displays, according to one or more embodiments of the invention. FIG. 3 illustrates a collection of over 1000 images (a collection of digitized maps of various sizes) packed into a montage according to one or more embodiments of the present invention. FIG. 6 shows a snapshot of about 3000 images dynamically rearranged in a random configuration in accordance with one or more embodiments of the present invention. FIG. 7 is a block diagram illustrating a computer system that can be adapted for use with one or more embodiments of the present invention.

Claims (28)

Accessing internet sites through a proxy server;
Converting image data from the Internet site into a multi-resolution representation;
Transmitting the image data of the multi-resolution representation to a client device. The method of claim 1, further comprising navigating a web page by the client device. The navigating is
(A) navigating image data of the Internet site by the proxy server;
(B) navigating the multi-resolution image data by the client device, wherein the navigating in step (a) and the navigating in step (b) occur substantially simultaneously. The method of claim 2, comprising: The navigating is
Storing at least substantially the entire multi-resolution image data of the web page at the proxy server;
The method of claim 2, comprising navigating the stored multi-resolution image data by the client device. The converting step includes:
3. The method of claim 2, comprising: continuously converting the Internet site image data that needs to be navigated by the client device. The converting step includes:
Pre-converting the selected image data by the proxy server before the selected image data from an internet site is needed for the client device navigation;
3. The method of claim 2, comprising storing the pre-converted image data at the proxy server. The method of claim 6, further comprising: transmitting the stored image data to the client device as needed by navigation of the client device. The method of claim 1, further comprising: ensuring that the stored pre-converted image data is not stale before sending the stored pre-converted image data to the client device. 6. The method according to 6. The guarantee is that
Comparing one of the time stamp and checksum of the pre-converted image data with a respective one of the time stamp and checksum of the corresponding image data originating from the Internet site. 9. The method of claim 8, wherein: The step of converting in advance includes
Converting image data of at least one web page by the proxy server;
7. The method of claim 6, comprising storing the converted image data of the at least one web page at the proxy server. The method of claim 2, further comprising enabling interactive communication between the client device and the Internet site. The interactive communication is:
The method of claim 11, comprising sending a navigation command from the client device to the proxy server. The method of claim 2, further comprising: emulating dynamic internet graphics for the client device by the proxy server. The emulating step comprises:
Dynamic HTML (Hypertext Markup Language),
The method of claim 13, comprising emulating at least one of an applet. The navigating is
The method of claim 2, comprising: selecting at the client device a resolution level for viewing at least a portion of the web page. The navigating is
The method of claim 2, comprising selecting an area of the web page for display on the client device. The navigating is
The method of claim 16, further comprising: selecting a resolution for viewing the region of the web page. The navigating is
The method of claim 2, comprising selecting a plurality of portions of the web page that are viewed on the client device. The navigating is
The method of claim 18 further comprising: selecting a plurality of respective resolutions for viewing the plurality of web page portions. Navigating the multi-resolution image data,
The method of claim 2, comprising panning the multi-resolution image data. Navigating the multi-resolution image data,
The method of claim 2, comprising zooming in the multi-resolution image data. A client device;
A proxy server that communicates with the client device and an Internet site and acts as an intermediate between the client device and the Internet site, the proxy server converting image data from the Internet site into a multi-resolution visual data format An apparatus comprising: a proxy server operable to convert.   The apparatus of claim 22, wherein the multi-resolution visual data format is JPEG2000. The client device is a mobile phone, and the mobile phone
The apparatus of claim 2, including at least one mechanism for navigating image data in the multi-resolution visual data format.   25. The apparatus of claim 24, wherein the mechanism is a touchpad, and the touchpad allows panning of image data within the multi-resolution visual data format.   The apparatus of claim 22, wherein the mechanism allows zooming in on image data within the multi-resolution visual data format.   27. The apparatus of claim 26, wherein the mechanism for zooming includes a roller that is rotatable by a user of the client device.   23. The client device according to claim 22, wherein the client device is one of a group consisting of a mobile phone, a PDA (Personal Digital Assistant), a notebook computer, a desktop computer, a tablet computer, and a television set. apparatus.

Images