KR20110043775A - Methods and systems for content processing - Google Patents

Abstract

Mobile phones and other portable devices may be equipped with a variety of technologies, whereby existing functionality may be improved and new functionality may be provided. Some aspects relate to visual search capability and determining appropriate actions in response to different image inputs. Other aspects relate to the processing of image data. Still other aspects relate to metadata generation, processing and provision. Still other aspects relate to user interface improvements. Other aspects relate to imaging architectures, wherein the image sensor of the mobile phone is one of a chain of stages that operate continuously on packetized instructions / data for capturing an image and later processing. Still other aspects relate to the distribution of processing tasks between the mobile device and remote resources (“cloud”). Although elementary image processing (eg, simple filtering and edge detection) may be performed on the mobile phone, other operations may be referenced to remote service providers. Remote service providers can be selected using techniques such as reverse auctions, through which they contend for processing tasks. Many other features and arrangements are also described above.

Description

METHODS AND SYSTEMS FOR CONTENT PROCESSING

Introduction

Certain aspects of the techniques described herein above are introduced in FIG. The user's mobile phone captures the image (in response to user commands or automatically) and the objects in the scene are recognized. Information associated with each object is identified and made available to the user via a scene-registered interactive visual "bauble" that is graphically overlaid on the image. The bauble may provide the information on its own, or it may be a simple indicia in which the user may tap on a marked location to obtain a longer listing of relevant information or to launch a related function / application.

In the illustrated scene, the camera recognized the face of the background as "Bob" and thus annotated the image. A billboard promoting a Godzilla movie was recognized, and a bauble saying "run times" was blitted on the display-prompting the user to tap for information screening.

The phone recognized the user's car from the scene and also identified another vehicle-by manufacture and age-in the image. Both are represented by overlaid text. The restaurant was also identified, and an initial review ("Jane's review: Very good!") Is seen from the collection of reviews. Tapping brings up more reviews.

In one particular arrangement, this scenario is implemented as a cloud-side service supported by local device object aware core services. Users can leave comments for both fixed and mobile objects. Tapped baubles can trigger other applications. Social networks can keep track of opposite relationships-forming a virtual "web of objects".

In the initial roll-out, the class of recognizable objects is limited but useful. Object identification events will primarily fetch and associate social-web connections and public domain information for baubles. Applications that utilize bar codes, digital watermarks, facial recognition, OCR, and the like can help support the early deployment of the technology.

Later, the arrangement is expected to develop into an auction market, where payment companies want to place their own baubles (or associated information) on highly targeted demographic user screens. User profiles along with input visual stimuli (promoted by GPS / magnet data in some cases) are fed to a Google-esque mix-master in the cloud, allowing mobile users to request baubles. Match buyers of device-screen real estate.

After all, this feature can be ubiquitous enough to fit into a common vocabulary, such as "I'll try to get a Bauble on that" or "See what happens if you Viewgle that scene."

background

Digimarc's patent 6,947,571 illustrates a system in which a cell phone camera captures content (eg, image data) and processes it to derive an identifier associated with the image. This derived identifier is provided to a data structure (eg, a database) that represents the corresponding data or operations. The cell phone then displays the response information or takes a response action. This sequence of operations may be referred to as a "visual search".

Related technologies are described in patent publications, 20080300011 (Digimarc), 7,283,983 and WO07 / 130688 (Evolution Robotics), 20070175998 and 20020102966 (DSPV), 20060012677, 20060240862 and 20050185060 (Google), 20060056707 and 20050227674 (Nokia), 20060026140 (ExBiblio), 6,491,217, 20020152388, 20020178410 and 20050144455 (Philips), 20020072982 and 20040199387 (Shazam), 20030083098 (Canon), 20010055391 (Qualcomm), 20010001854 (AirClic), 7,251,475 (Sony), 7,174,293 (Iceberg), Organan, 06065, (Evryx Technologies), 6,993,573 and 6,199,048 (Neomedia), 6,941,275 (Tune Hunter), 6,788,293 (Silverbrook Research), 6,766,363 and 6,675,165 (BarPoint), 6,389,055 (Alcatel-Lucent), 6,121,530 and 6,121,946 (Sberoda) Is shown.

It is an object of the present invention to provide methods and systems for content processing.

Aspects of the presently described technique relate to improvements to these techniques-aimed at the purpose of intuitive computation: to see / view or hear and infer the user's desires in the sensed context.

0 illustrates an example embodiment incorporating certain aspects of the techniques described herein above.
1 is a top view of an embodiment incorporating aspects of the present technology.
FIG. 2 illustrates a portion of an application that a user may request to run a camera-mounted cell phone.
3 is an illustration of some of the commercial entities in an embodiment incorporating aspects of the present technology.
4, 4A and 4B conceptually illustrate how pixel data and derivatives are applied to different tasks and packaged in packet form.
5 illustrates how different tasks may have certain image processing operations in common.
6 illustrates how common image processing operations can be identified and used to configure cell phone processing hardware to perform these operations.
FIG. 7 illustrates how a cell phone can transmit certain pixel-related data over an internal bus for local processing and other pixel-related data over a communication channel for processing in the cloud.
FIG. 8 illustrates how the cloud processing of FIG. 7 allows even more " intelligence " to be applied to the task desired by the user.
9 illustrates in detail how keyvector data is distributed to different external service providers, who executes services in exchange for compensation, and which is handled in an enhanced manner for the user.
10 illustrates an embodiment incorporating aspects of the present technology that note how cell phone-based processing is suitable for simple object identification tasks such as template matching, while cloud-based processing is suitable for complex tasks such as data association. Figure.
10A illustrates an embodiment incorporating aspects of the present technology that note that the user experience is optimized by performing visual keyvector processing as close as possible to the sensor and managing the traffic to the cloud as low as possible the communication stack.
11 shows a second group in which tasks related to external processing can be routed to a first group of service providers who routinely execute specific tasks for a cell phone, or compete dynamically based on processing tasks from the cell phone. Illustrating what may be routed to service providers in the network.
12 is an enlarged view of the concepts of FIG. 11 illustrating, for example, how the bead filter and broadcast agent software module may investigate reverse auction processing.
13 is a top block diagram of a processing arrangement incorporating aspects of the present technology.
14 is a top block diagram of another processing arrangement incorporating aspects of the present technology.
FIG. 15 illustrates an example range of image types that may be captured by a cell phone camera.
16 illustrates a particular hardware implementation incorporating aspects of the present technology.
17 illustrates aspects of a packet used in an exemplary embodiment.
18 is a block diagram illustrating an implementation of a SIFT technique.
19 is a block diagram illustrating, for example, how packet header data can be changed during processing through the use of memory.
19A illustrates a prior art architecture from a robotic player project.
FIG. 19B illustrates how various factors can affect how different operations can be handled. FIG.
20 illustrates an arrangement in which a cell phone camera and a cell phone projector share a lens.
20A illustrates a reference platform architecture that may be used in embodiments of the present technology.
21 shows an image of a desktop phone captured by a cell phone camera.
FIG. 22 illustrates a collection of similar images found in a repository of public images, with reference to features identified from the image of FIG. 21.
23-28A and 30-34 are flow charts describing methods of incorporating aspects of the present technology.
29 is an art shot of the Eiffel Tower captured by a cell phone user.
35 is another image captured by the cell phone user.
36 is an image of the underside of a telephone found using methods in accordance with aspects of the present technology.
37 illustrates a portion of one style of physical user interface of a cell phone.
37A and 37B show different linking topologies.
FIG. 38 is an image captured by a cell phone user depicting trail markers of the Appalachian Trail. FIG.
39-43 illustrate detailed methods of incorporating aspects of the present technology.
44 illustrates one style of user interface of a cell phone.
45A and 45B illustrate how different dimensions of commonality can be explored through the use of user interface control of a cell phone.
46A and 46B are detailed views of a particular method of incorporating aspects of the present technology, as keywords such as Prometheus and Paul Manship are automatically determined from the cell phone image.
47 illustrates some of the different data sources that may be referenced in a processed image in accordance with aspects of the present technology.
48A, 48B, and 49 illustrate different processing methods in accordance with aspects of the present technology.
50 identifies some of the different processes that can be performed on image data, in accordance with aspects of the present technology.
FIG. 51 illustrates an example tree structure that may be utilized in accordance with certain aspects of the present technology. FIG.
FIG. 52 illustrates a network of wearable computers (eg, cell phones) that may cooperate with one another, for example in a peer-to-peer network.
53-55 are detailed views of how a glossary of symbols can be identified by a cell phone and used to trigger different actions.
56 illustrates aspects of a digital camera technique of the prior art.
57 is a detailed view of an embodiment incorporating aspects of the present technology.
58 illustrates how a cell phone can be used for scene and display affine parameters.
59 illustrates certain state machine aspects of the present technology.
FIG. 60 shows how time or motion aspects may be included even in the case of “still” images. FIG.
FIG. 61 illustrates some metadata that may be relevant to an implementation that incorporates aspects of the present technology.
FIG. 62 illustrates an image that may be captured by a cell phone camera user. FIG.
63-66 are detailed views of how the image of FIG. 62 can be processed to convey semantic metadata.
67 illustrates another image that may be captured by a cell phone camera user.
68 and 69 are detailed views of how the image of FIG. 67 can be processed to convey semantic metadata.
70 illustrates an image that may be captured by a cell phone camera user.
FIG. 71 illustrates details of how the image of FIG. 70 may be processed to convey semantic metadata. FIG.
FIG. 72 is a chart illustrating aspects of the human visual system. FIG.
73 illustrates different low, medium and high frequency components of the image.
74 shows a newspaper.
FIG. 75 shows a layout of the FIG. 74 page as set by the layout software. FIG.
FIG. 76 is a detail view of how user interaction with an image captured from printed text can be enhanced. FIG.
FIG. 77 illustrates how semantic delivery of metadata may have a progressive aspect similar to JPEG2000 and the like.
78 is a block diagram of a thermostat of the prior art.
FIG. 79 is an external view of the thermostat of FIG. 78;
80 is a block diagram of a thermostat that utilizes certain aspects of the present technology (“ThingPipe”).
81 is a block diagram of a cell phone utilizing certain aspects of the present technology.
FIG. 82 is a block diagram in which specific operations of the thermostat of FIG. 80 are described.
FIG. 83 illustrates a cell phone display depicting an image captured from a thermostat overlaid with specific touch-screen targets that a user may touch to increase or decrease the thermostat temperature.
FIG. 84 is similar to FIG. 83 but illustrates a graphical user interface for use on a phone without a touch screen;
85 is a block diagram of an alarm clock utilizing aspects of the present technology.
86 illustrates a screen of an alarm clock user interface that may be provided on a cell phone, in accordance with an aspect of the technology.
FIG. 87 illustrates a screen of a user interface detailing nearby devices that can be controlled through the use of a cell phone.

This disclosure details the variety of techniques assembled over an extended time period to meet a variety of different purposes. Yet they are related together in various ways and are therefore provided in batches in this single document.

This modified and interrelated subject matter is not in itself suitable for direct provision. As such, the description sometimes proceeds in a non-linear fashion among classified topics and techniques, and therefore should be understood by the reader.

Each part of this specification details a technique that preferably incorporates the above described technical features in other parts. Thus, it is difficult to identify "beginning" where this disclosure should logically begin. That is, we just go on hard.

Mobile with Distributed Network Services device Object  Awareness and interaction

There is currently a huge disconnect between the innumerable amount of information contained in high quality image data streaming from a mobile device camera (eg, at a cell phone) and the mobile device's ability to process whatever data ends up. “Off device” processing of visual data may help to handle the fire hose of such data, especially when multiple visual processing tasks may be desirable. These issues become even more important when "real-time object recognition and interaction" is considered, in which case the user of the mobile device virtually instantaneous results and augmentation on the mobile device screen as the user touches the camera on the scene or object. Expected realistic graphic feedback.

According to one aspect of the present technology, a distributed network of pixel processing engines serves these mobile device users and generally feeds back much smaller than one second to meet the most qualitative "human real-time interactivity" requirements. The implementation preferably provides certain basic features for the mobile device, including a slightly close relationship between the underlying communication channel available to the mobile device and the output pixels of the image sensor. Routing instructions to pixel data as specified by the user's intentions and reservations following the basic specific levels of "content filtering and classification" of pixel data on the local device may be processed by the mobile device and one or more "cloud-based" pixel processing. Induce an interactive session between services. The keyword "session" also refers to fast responses sent back to the mobile device, and for some services marketed, such as "real time" or "interactive," the session essentially represents a packet-based duplex and Several outgoing "pixel packets" and several incoming response packets (which may be updated pixel data along with processed data) may occur every second.

Business factors and good old contention are at the heart of distributed networks. Users can subscribe to or tap on any external services they choose. The local device itself and / or the carrier service provider for that device may be configured as the user chooses to route the filtered and suitable pixel data to specified object interaction services. The charging mechanisms for these services can plug directly into existing cell and / or mobile device charging networks, being billed to users and paid to service providers.

However, let's back up for a while. The addition of camera systems to mobile devices has led to a surge of applications. The native application would simply snap quick visual aspects of their environment among common people and certainly share such pictures with friends and family.

The panning out of applications from that starting point almost certainly depends on the set of core plumbing features inherent in mobile cameras. Briefly (and of course not quickly identified), these features include: a) high quality pixel capture and low level processing; b) better local device CPU and GPU resources for pixel processing on the device with subsequent user feedback; c) "cloud" structured connectivity; And importantly d) specific traffic monitoring and charging infrastructure. 1 is one graphical view of some of these plumbing features of what may be called visual intelligent networks. (Typical details such as cell phones such as microphones, A / D converters, modulation and demodulation systems, IF stages, cellular transceivers, etc. are not shown for clarity of the drawings.)

It is good to get better CPUs and GPUs and more memory on mobile devices. However, cost, weight and power considerations have the potential to help the "cloud" do as much as possible "intelligence".

In this regard, there is a potential to be a common element set of "device-side" operations for visual data serving all cloud processes, including specific formatting, elemental graphics processing, and other mechanical operations. Similarly, there is a possibility that there should be a standardized default header and addressing scheme for back and forward communication traffic (usually packetized) resulting from the cloud.

This conceptualization is similar to the human visual system. The eye performs baseline operations such as colorimeters and optimizes the necessary information for transmission along the optic nerve of the brain. The brain does the real work of perception. It also feeds back the feedback-the brain sends information that controls muscle movement-where the eye is headed, scanning rows of books, controlling iris (brightness), and so on.

FIG. 2 shows an illustrative list that is not exhaustive of visual processing applications for mobile devices. Again, it is not difficult to find similarities between this list and the basics of the human visual system and how the human brain operates. It is a well-researched university area to deal with how "optimized" the human visual system is involved in any given object recognition task, and how efficient it is for an eye-retinal-optic-cortical system to serve a vast array of recognition needs. It is a common consensus that the paper is surprisingly pretty darn. Aspects of this technology relate to how similarly efficient and widely available elements can be made of mobile devices, mobile device connections, and network services, all of which include the applications and technology dance shown in FIG. It is intended to serve these new applications that can be shown as it continues.

Perhaps, the main difference between human analogy and mobile device networks should be a clear focus on the basic concept of a "market" where buyers buy better things as long as the business knows how to make a profit. Is the point. Any technique for serving the applications listed in FIG. 2 should assume that not thousands or hundreds of business entities will develop important details of specific commercial offerings, and one way or another may be expected to benefit from these offers. Get Yes, some behemoths will dominate the main lines of cash flows across the entire mobile industry, but there are the same certainties that niche players will continue to develop niche applications and services. Thus, this disclosure describes how a market for visual processing services can be developed and thereby has what business interests across the spectrum will gain. 3 attempts to roughly categorize some of the business interests applicable to a global business ecosystem operating at this filing time.

4 shows an inference of the introduction of the technical aspect currently under consideration. Here, we found a very abstract bit of information derived from some group of photons that affected some form of electronic image sensor, with many of its low bit atmospheric consumers. 4A then quickly introduces an intuitively well-known concept that single bits of visual information are of little value outside their role in both spatial and temporal groups. This core concept is well utilized in modern video compression standards such as MPEG7 and H.264.

The "visual" character of the bits can be removed very far from the visual domain by a particular process (eg, consider vector strings representing eigenface data). Thus, we sometimes refer to the term "keyvector data" (or "keyvector string" to collectively represent raw sensor / stimulation data (eg pixel data) and / or processed information and associated derivatives. ”). The keyvector may take the form of a container (such as a packet data structure) in which this information is conveyed. The tag or other data may be included to identify the type of information (eg, JPEG image data, or eigenface data), or the data type may be apparent from the data or from the context. One or more instructions or actions may be associated with keyvector data-which is explicitly specified or implied in the keyvector. For certain types of keyvector data, the operation may be implied in a default manner (eg, for JPEG data, "image may be stored"; for eigenface data, "this eigenface template matches"). Can be). Or the nested action may depend on the context.

4A and 4B also introduce a central player to this disclosure: an address-labeled pixel packet packaged into a body with keyvector data embedded therein. The keyvector data can be a single patch or a collection of patches, or a time series of patches / collections. Pixel packets can be less than kilobytes, or the size can be much larger. It can convey information about a separate patch of pixels extracted from a larger image, or it can convey the large-scale Photosynth of Notre Dame Cathedral.

(As currently represented, the pixel packet is an application layer structure. However, if you really deal with the network wildly, it can be broken into smaller parts-because the transmission layer constraints of the network may be required).

5 is a segue diagram that still shows an abstract level but indicates specificity. The list of user-defined applications as shown in FIG. 2 will map to a state-of-the-art list of pixel processing methods and methods that can achieve each and every application. These pixel processing methods can be broken down into common and less common component subtasks. Object-aware textbooks are populated with a wide variety of ways and terms, which at first glance cause scenes of the order that can appear to be an embarrassing array of "unique requirements" related to the application shown in FIG. (In addition, multiple computer vision and image processing libraries, such as OpenCV and CMVision-described below, have been created by identifying and rendering functional operations, which can be considered "atomic" functions within object recognition paradigms.) However, FIG. 5 attempts to show that there is actually a set of shared steps and processes shared between visual processing applications. Differently formed pie slices are intended to illustrate that certain pixel operations may be of a particular class and may have differences in low level variables or optimizations. The size of the whole pie (think in algebraic scene, for example a pie that is twice the size of another pie can represent 10 times more flops) and the size ratio of the slices represent degrees of commonality.

6 sacrifices the simplicity of processing and specifically takes major steps. Here, we can see that it is labeled "Unique Calling Visual Processing Services" at the top, which lists all the possible lists of applications from FIG. 2 that a given mobile device can be aware of or thoroughly enabled to run. Express. The concept is not that all these applications need to be active all the time, so some sub-set of services are actually "turned on" at any given moment. As a one-time configuration activity, turn-on applications negotiate to identify common component tasks labeled "public processes classifier"-first, these elementary image processing routines (eg, FFT, filtering, edge detection). Generate a common list of pixel processing routines available for processing on selected devices from a library of resampling, color histograms, log-polarity conversions, and the like. Generation of the corresponding flow gate configuration / software programming information is followed by the actual loading of library elements into places aligned with the field programmable gate array set-up, or by a suitable processor to perform the necessary component tasks. Configure.

6 also includes drawings of a general purpose pixel segmenter following the image sensor. This pixel segmenter is divided into manageable spatial and / or temporal blobs of a large stream of images from the sensor (eg MPEG macroblocks, wavelet transform blocks, 64 x 64 pixel blocks, etc.). Similar). After the rapids of pixels are broken into breakable chunks, they are fed into a newly programmed gate array (or other hardware), which performs the elementary image processing tasks associated with the selected applications. (These arrangements are further described below in an example system that utilizes "pixel packets.") Various output results are elementally processed (eg, internally and / or externally) into other resources for further processing (eg, For example, key vector data). This additional processing is typically more complex than what has already been done. Examples include making associations, deriving inferences, pattern and template matching, and the like. This additional treatment may be a high degree of special use.

(Consider an advertising game from Pepsi, which invites the public to participate in Treasure Hunt at State Park. Based on the Internet-distributed clues, people try to find a hidden box of soda to earn a $ 500 prize. Participants Pepsi- You need to download a specific application from the dot-com website (or Apple App Store), which distributes the clues (which may be published on Twitter) .The downloaded application also has a prize verification component, which A hidden paper box processes the image data captured by the user's cell phones to identify the uniquely marked special pattern SIFT object recognition is used (described below), and the SIFT feature descriptors for the special package are downloaded to the downloaded application. If an image match is found, the cell phone immediately Report wirelessly to the city The winner is the user whose cell phone first reported the detection of a specially-marked paper box In the arrangement of Figure 6, some of the component tasks of the SIFT pattern matching operation are elements in the configured hardware. Executed by an image processing; the rest represent more specialized processing internally or externally.)

FIG. 7 is a top view of the picture for a general distributed pixel services network diagram, with some symmetry on how local device pixel services and “cloud based” pixel services operate. In Fig. 7 the router notes that any given packaged pixel packet is sent to the appropriate pixel processing location, whether local or remote (the style of the charging pattern indicates different component processing functions; to possible visual processing services). Only a few processing functions are required). Some of the data shipped in cloud-based pixel services may first be processed by local device pixel services. The prototypes indicate that the routing function can have components in the cloud-nodes, which serve to distribute the tasks to active service providers and collect the results for sending back to the device. In some implementations, these functions can be executed at the edge of the wireless network, for example, by modules of wireless service towers to ensure the fastest operation. Results collected from active external service providers and active local processing stages are fed back to the pixel service manager software, which then interacts with the device user interface.

FIG. 8 is an enlarged view of the bottom right of FIG. 7, the moment Dorothy's gods turn red and why the distributed pixel services provided by the cloud-both as opposed to the local device-will be fine but are the most common object recognition tasks Express

Rich forms of object recognition are based on visual association rather than strict template matching rules. If we all learned that the pre-historic form, where the basic letter "A" never changes, would always follow the universal template image, if this is allowed, then the prescribed form "A" would be the camera. In order to get this to be sure to read the base A when appearing, very obvious and locally normative methods may be suitable for a mobile image device. 2D and 3D barcodes also follow a templated approach to object recognition in many cases, and for the included applications that relate these objects, local processing services can get jobs in large quantities. However, even in the case of barcode examples, there is a desire for an architecture that does not force "code upgrades" to countless devices whenever there is any advance in the field of symbolic, where the flexibility of growth and evolution of apparent visual coding targets is apparent.

At the other end of the spectrum, for example, any complex task of predicting a suspicious typhoon caused by the flapping of the wings of a butterfly in the middle around the world-if the application requires it-to consult the network of supercomputers. Can be thought of. Oz beckons.

8 attempts to illustrate this basic additional dimensionality of pixel processing in the cloud, as opposed to the local device. This goes without saying (or without images) virtually, but FIG. 8 is also the granular diagram of FIG. 9 where Dorothy goes back to Kansas and is happy about it.

9 is all about cash, cash flow, and happy humans using cameras on mobile devices to get very important results from their visual inquiries throughout their monthly payments. This proves that Genie comes out of the bottle at the Google "AdWords" auction. Hundreds and thousands of micro-judgments, pixel routings, result comparisons, and whether they know, for a very good product they're looking for "truthfully" behind scenes of an instant visual scandal from a mobile user in an immediate visual environment, and There is a micro-auction channel for mobile device users. This final point is deliberately blatant in that any kind of search is inherently unlimited and magical at some level, and the enjoyable part of the search in the first place is surprisingly new associations are part of the results. Search users then find out what they are really looking for. The system shown in FIG. 9 as a carrier-based financial tracking server now facilitates proper results to be sent back to the user throughout the monitoring of the use of services, in order to bill monthly and send benefits to the appropriate entities. We can see the addition of our Networked Pixel Services module and its role.

(As further described elsewhere, the flow of funds may not be exclusive to remote service providers. For example, in order to induce or compensate for certain operations, different flows of money, such as users or other third parties, may be used. Increases.)

FIG. 10 focuses on the functional division of processing, showing how tasks similar to template matching can be executed on the cell phone themselves, while more complex tasks (similar to data associations) are preferably referenced in the cloud for processing. .

The elements of the above have been extracted in FIG. 10A to illustrate the implementation of aspects of the technology as (generally) the physical work of software components. The two ovals in the figure represent symmetric pairs of software components that correlate the setup, data associations and visual query results of a "human real-time" visual recognition session between the mobile device and the general cloud or service providers. Emphasize. The ellipse on the left represents "keyvectors", more specifically "visual keyvectors". As noted, this term may include everything from simple JPEG compressed blocks in any manner through log-polarized transformed facial feature vectors and everything in between or beyond them. The key to the keyvector is that the raw information inherent in any given visual recognition task is optimally pre-processed and packaged (and possibly compressed). The left oval assembles these packets and typically inserts some addressing information to be routed (final addressing may not be possible, as the packet may ultimately be routed to remote service providers-details about this). Are not yet known.) Preferably, this process is performed as close to the raw sensor data as possible, such as by processing circuitry integrated on the same substrate as the image sensor, which is provided from another end in packet form or stored in memory. Respond to software instructions.

The right ellipse manages remote processing of keyvector data, for example trying to configure appropriate services, directing traffic flow, and so on. Preferably, such software processing is implemented as low as possible on the communication stack, generally in "cloud side" devices, access points, cell towers, and the like. (When real-time visual keyvector packets are streamed through a communication channel, the lower the communication stack they are identified and routed to, the smoother it will be to see and feel the “human real-time” that will be a given visual recognition task.) Necessary to support this arrangement The remaining high level processing is included in FIG. 10A for the context and can generally be executed via basic mobile and remote hardware capabilities.

11 and 12 show that some cloud-based pixel processing services may be pre-established in a pseudo-static manner, while other providers periodically for the privilege of processing the user's keyvector data through anti-auction participation. The concept of contention is shown. In many implementations, these latter providers contend whenever a packet is available for processing.

Consider a user snapping a cell phone image of an unfamiliar car that wants to learn the make and model. Various service providers can compete for this business. The start-up vendor may offer to run the recognition at no cost to create the brand and collect collector data. The images presented in this service simply return information indicating the make and model of the car. Consumer reports can provide manufacturing and model data, as well as alternative services that provide technical specifications for the car. However, they may charge 2 cents for the service (or may be based on bandwidth, eg 1 cent per megapixel). Edmunds or JD Powers provide the same data as consumer reports but can provide another service that the user pays for the privilege of the data provided. In exchange, the vendor is given the right to have one of its partners send a text message for user advertising products or services. The payment may take the form of credit for the user's monthly cell phone voice / data service billing.

Using criteria specified by the user, stored preferences, context, and other rules / discovery teaching methods, query routers and response managers (in cell phones, in the cloud, distributed, etc.) ensure that data packets requiring processing are stable. Determine whether it should be handled by one of the service providers of static waits or whether it should be provided to providers based on the auction, in which case the outcome of the auction is adjusted.

The static standby service can be identified when the phone is initially programmed and can only be reconfigured when the phone is reprogrammed. (For example, Verizon can specify that all FFT operations for phones are routed to the server providing for this purpose.) Alternatively, the user may periodically identify good providers through the configuration menu for specific tasks. Or may specify that certain tasks should be referenced for auction. Some applications may appear where static service providers are popular; The work may be too common, or the services of one provider may be too unparalleled, so there is no good reason for contention for the provision of services.

In the case of services referenced in the auction, some users may raise the price above all other considerations. Other users can force domestic data processing. Other users may want service providers to strive to meet "green", "ethical" or other standards of integration practice. Other users may prefer richer data output. Weights of different criteria may be applied by the query router and response manager in making the decision.

In some circumstances, one input to the query router and response manager may be at the user's location such that when the user is at home in Oregon, a different service provider may be selected than when the user is on vacation in Mexico. . In other cases, the required turnaround time is specified, which may determine some vendors as ineligible and further contend with other vendors. In some examples, the query router and response manager need not determine at all if the stored results identifying, for example, the service provider selected at the previous auction are still available and do not exceed the "freshness" threshold.

Pricing provided by vendors may vary with processing load, bandwidth, date and time, and other considerations. In some embodiments, the providers may be informed of the quotes presented by the contenders (using known trusted devices that ensure data integrity) and may be given an opportunity to make them more attractive. This bidding war may continue until there are no bidders to change the requirements offered. The query router and response manager (or in some implementations, the user) then select.

Description For convenience and visual clarity, FIG. 12 shows a software module labeled “Bead Filter and Broadcast Agent”. In most implementations, this forms part of the query router and response manager module. The bead filter module determines if some vendors-from a number of possible vendors-should be given the opportunity to bid for this processing task. (User's preference data or historical experience may indicate that certain service providers are ineligible.) The broadcast agent module then communicates with selected bidders to notify them of user actions for processing, and they Provide the information needed to

Preferably, the bead filter and the broadcast agent do some of their work in advance of at least some of the data available for processing. That is, as soon as a prediction is made about an action that the user may be likely to request soon, these modules begin working to identify the provider to run the service expected to be required. After a few hundred milliseconds, the user keyvector data may actually be available for processing (if the prediction turns out to be correct).

Sometimes, like Google's provided AdWords systems, service providers are not referenced in each user transaction. Instead, each provides bidding parameters, which are stored and consulted whenever a transaction is considered, to determine which service provider wins. These stored parameters can be updated from time to time. In some implementations, the service provider puts updated parameters in the bead filter and broadcast agent whenever it is available. (The bead filter and broadcast agent may serve many demographic users, such as all Verizon subscribers in area code 503, all subscribers to ISPs in the community, or all users of the domain well-dot-com, etc.). May serve; or more localized agents may be utilized, such as one for each cell phone tower.)

If there is a break in traffic, the service provider may discount the services for the next moment. The service provider will therefore send (or mail) a message stating that in the Unix era, it will perform eigenvector extraction for image files up to 10 megabytes for two cents by 1244754176 international standard time, after which time the price will return to three cents. )can do. The bid filter and broadcast agent thus update the table with the stored bidding parameters.

(The reader assumes that the reverse auction arrangements used by Google to place ads by advertisers on web search results pages are familiar. May 22, 2009, "Secret of Googlenomics: Data-Fueled Recipe Brews Profitability," Example technology is provided in Wired Magazine.

In other implementations, the broadcast agent polls the bidders-communicating the relevant parameters and requesting bid responses whenever a transaction is provided for processing.

Once the prevailing bidder is determined, the data is available for processing, and the broadcast agent sends keyvector data (and other parameters as it may be suitable for the particular task) to the winning bidder. The bider then executes the requested operation and returns the processed data to the query router and response manager. This module logs the processed data and participates in any necessary accounting (eg, trusting the service provider for a reasonable fee). The response data is then sent back to the user device.

In a variant arrangement, one or more competing service providers actually perform some or all of the requested processes, but “join” the user (or query router and response manager) by providing only partial results. With a sneak peek of what is available, the user (or query router and response manager) may be induced to make different choices than the relevant criteria / discovery teaching methods otherwise indicated.

Naturally, function calls sent to external service providers do not have to provide the ultimate result the consumer seeks (eg, identify a car or translate a menu listing into French in English). They may be component operations, such as calculating an FFT, performing a SIFT procedure or log-polar transformation, calculating histograms or eigenvectors, identifying edges, and the like.

Sooner or later, it is expected that a rich ecosystem of expert processors will emerge-serving countless processing requests from cell phones and other thin client devices.

Importance of money flow

Incentive payments spent by service providers themselves in exchange for user information (e.g., viewership) or in exchange for actions taken by the user, such as completing a survey, visiting a particular place, tracking the location of a store, etc. Additional business models of remote services may be possible that relate to subsidization.

Services may likewise be awarded incentives by third parties, such as coffee shops that derive value by providing differentiated services to consumers in the form of free / discounted use of remote services while they are sitting in the store. have.

In one arrangement, an economy is possible in which calls of remote processing credits are created and exchanged between users and remote service providers. It is entirely transparent to the user and can be managed, for example, as part of the service plan with the user's cell phone or data service provider. Or as a very explicit aspect of certain embodiments of the present technology. Service providers and other providers may grant credits to users who are part of a frequent-user program to take actions and make loyalty with particular providers.

For other currencies, users may be selected to explicitly donate, store, exchange, or generally barter credits as needed.

Taking these emphasis into greater detail, the service may pay a user to participate in an audience rating panel. For example, the Nielsen company can provide services to the public, such as identification of television programming from audio or video samples presented by consumers. These services are provided free of charge to consumers who agree to share some of Nielsen's media consumption data (such as acting as anonymous members for the city's viewership panel) and to other consumers on a fee-based basis. Can be. Nielsen may, for example, provide 100 units of credit-micropayments or other values-to consumers who participate each month, or provide credit each time a user presents information to Nielsen.

In another example, the consumer may be rewarded to accept advertisements or advertisement impressions from the company. When consumers go to the Pepsi Center in Denver, they can be rewarded for the Pepsi-branded experience each consumer encounters. The amount of the micropayment may be proportional to the amount of time the consumer has interacted with different Pepsi-brand objects (including audio and images) on the stage.

In addition, large brand owners can provide credits individually. Credits can be routed to friends and social / business knowledges. To illustrate, a Facebook user may share credit (which is redeemable or redeemable for goods / services) from his Facebook page-which attracts others to visit or enjoy. In some cases, credit may be available only to a person who navigates a Facebook page in a particular way, such as linking to the page from the user's business card or from another launch page.

As another example, benefited, paid, or otherwise, on certain services-such as the download of songs from iTunes, or music recognition services, or the identification of clothes that match a particular shoe (images thereof are shown). Consider a Facebook user who has received credit that may apply. These services can be associated with a particular Facebook page so that friends can call the services from that page-in particular, consuming the host's credit (again with the proper permission or invitation by the hosting user). . Similarly, friends can present images to a facial recognition service accessible through an application associated with the user's Facebook page. Images presented in this way are analyzed for the faces of friends of the host, and the identification information is returned to the presenter, for example, via a user interface provided on the original Facebook page. Again, the host can be evaluated for each such operation, but only authorized friends can allow such service itself to be used free of charge.

Credits and payments can also be routed to charities. After a particularly poignant movie about poverty in Bangladesh, spectators leaving the theater can capture images of associated movie posters, which serve as a portal for donations to charities that help the poor in Bangladesh. When recognizing a movie poster, the cell phone can provide a graphical / touch user interface that allows the user to dial to specify the amount of contributions the user can make, and at the end of the transaction, the charity in the financial account associated with the user. Is transferred to the financial account associated with it.

Specific hardware On the arrangement  More about

As noted in the above and cited patent documents, there is a need for general object recognition by mobile services. Some approaches to specialized object recognition have emerged, and they have provided an augmentation for certain data processing approaches. However, no architecture has been proposed that goes beyond generalized object recognition toward general object recognition.

Visually, a general object recognition arrangement requires access to good raw visual data, preferably without device quirks, scene changes, user changes, and the like. Developers of systems built around object identification will be the most successful, serving their users by focusing on object identification tasks in the near future, not on the myriad existing obstacles, resource sinks, and third party dependencies that they currently have to face. something to do.

As noted, virtually all object identification techniques may use a pipe for "cloud"-or may depend.

The "cloud" can include everything outside the cell phone. An example is a nearby cell phone, or a plurality of phones on a distributed network. Unused processing power on these other phone devices may be made available for use (or free of charge) whenever needed. Cell phones of the implementations detailed herein can collect processing power from these other cell phones.

This cloud may be an ad hoc, for example other cell phones within the Bluetooth range of the user phone. These other phones can be extended by ad hoc network by also extending the local cloud to other phones that are not reachable by the user but can be reached by Bluetooth.

The "cloud" also includes other computing platforms, such as set-top boxes, processors in cars, thermostats, HVAC systems, wireless routers, local cell phone towers and other wireless network edges (software-defined). And processing hardware for the wireless device). Such processors can be used with more conventional cloud computing resources-as provided by Google, Amazon, and the like.

(In view of the particular users' concerns about privacy, the phone preferably has a user-configurable option that indicates whether the phone can query the cloud resources for processing. In one arrangement, this option Has a default value of "no", which not only limits functionality and reduces battery life, but also limits privacy concerns. In other arrangements, this option has a default value of "yes".)

Preferably, image-response techniques should produce a “result or response” for a short period of time, which generally requires some level of interaction with the user—in one second fractions for actually interacting applications, or Measured in fractions of a second or a minute for recent "I have patience waiting" applications.

For those objects, they are: (1) general manual (clues to basic searches), (2) geographic manual (you at least know where you are and can connect to geo-specific resources), (3) "cloud supported" manuals, such as "identified / enumerated objects" and their associated sites, and (4) active / controllable la ThingPipe (WiFi-equipped thermostats and parking) Meters can be divided into various categories, including the techniques detailed above.

The object recognition platform should not be thought of as a classic "local device and local resources only" software intelligence, but it is likely. However, it can be thought of as a local device optimization problem. In other words, the software on the local device and its processing hardware should be designed in consideration of the interaction with off-device software and hardware. Compared to device off, the balance and interaction of both control functions, pixel high speed processing functions, and application software / GUI provided on the device are the same. (In many implementations, certain databases useful for object identification / recognition will exist remotely from the device.)

In a particularly preferred arrangement, this processing platform utilizes image processing near the sensor-on the same chip as the best, and at least some processing tasks are preferably carried out by dedicated special purpose hardware.

Considering FIG. 13, the architecture of the cell phone 10 in which the image sensor 12 supplies two processing paths is shown. One 13 is adapted for the human visual system and includes processing such as JPEG compression. The other 14 is adapted for object recognition. As discussed, some of these processes may be executed by the mobile device and other processes may be referenced to the cloud 16.

14 takes an application-centricity of the object recognition processing path. Some applications are entirely on the cell phone. Other applications are entirely external to the cell phone-simply taking keyvector data such as stimuli, for example. Hybrids are more common, such as when some processing is done at the cell phone, other processing is done externally, and application software that coordinates the processing is present at the cell phone.

To illustrate another discussion, FIG. 15 shows a range 40 of a portion of different types of images 41-46 that can be captured by a user's cell phone. Some simple (incomplete) comments on some processing that can be applied to each image are provided in the following paragraphs.

Image 41 depicts a thermostat. A steganographic digital watermark 47 is textured or printed on the case of the thermostat. (The watermark is shown visually in FIG. 15 but is typically not recognizable to the viewer.) The watermark conveys the information intended for the cell phone, allowing the user to provide a graphical user interface that can interact with the thermostat. do. Bar codes or other data carriers may alternatively be used. This technique is further detailed below.

Image 42 depicts an item that includes a barcode 48. This barcode carries Universal Product Code (UPC) data. Different barcodes can carry different information. Barcode payloads are not primarily intended to be read by the user cell phone (as opposed to watermark 47) but may nevertheless be used by the cell phone to help determine the appropriate response to the user.

Image 43 shows a product that can be identified without reference to any high speed machine readable information (such as a barcode or watermark). In order to distinguish the apparent image object from the apparent background, a segmentation algorithm can be applied to the edge-detected image data. Image objects can be identified through their shape, color and texture. Image fingerprinting can be used to identify reference images with similar labels, and metadata associated with these other images can be collected. SIFT techniques (discussed below) may be utilized for such pattern-based recognition tasks. Mirrored reflections of low texture areas may attempt to indicate that the image object is made of glass. Optical character recognition can be applied for other information (reading visible text). All these clues can be used to identify the depicted item and help determine the appropriate response to the user.

Additionally (or alternatively), a similar-image search system such as Google Similar Images and Microsoft Live Search can be utilized to find similar images, after which their metadata can be collected. (As in this record, these services do not directly support uploading user images to find similar web images. However, the user can mail images to Flickr (using Flickr's cell phone upload feature), This will soon be discovered and managed by Google and Microsoft.)

Image 44 is a snapshot of friends. Face detection and recognition can be utilized (ie, to identify certain faces to indicate that there are faces in the image, and thus maintain users maintained by, for example, Apple's iPhoto service, Google's Picasa service, Facebook, etc.). To annotate the image as metadata with reference to the associated data). Some face recognition applications may be trained for non-human faces, such as cats including avatars, dogs animated characters, and the like. Geographic location and data / time information from the cell phone may also provide useful information.

People wearing sunglasses are challenging some face recognition algorithms. Identification of these individuals can be aided by association with people whose identities can be more easily determined (eg by conventional facial recognition). That is, by identifying other groups of images in iPhoto / Picasa / Facebook / etc., Including one or more of the latter individuals, other individuals depicted in those photos may also be present in the subject image. These candidates form much smaller possibilities than those normally offered by unrestricted iPhoto / Picasa / Facebook / etc. Face vectors recognizable from sunglasses wearing faces in the subject image can then be compared against these smaller possibilities to determine the best match. In the general case of face recognition, a score of 70 or 80 may be sufficient in retrieving this group-limited set of images if a score of 90 is required to be considered a match (from any best matching score of 100). . (When two people are depicted without sunglasses, as in image 44, the appearance of both of these individuals in the picture with one or more other individuals is such an analysis implemented by, for example, increasing the weighting factor in the matching algorithm. May increase the relevance of

Image 45 shows a portion of the statue of Prometheus at Rockefeller Center, NY. The identification may follow the disclosures described elsewhere in this specification.

Image 46 depicts a landscape depicting the Maroon Bells Mountain region of Colorado. This image object may be recognized with reference to geographic location data from the cell phone along with geographic information services such as GeoNames or Yahoo! 'S GeoPlanet.

(Note that the well-known techniques, together with the processing of one of the images 41-46 of Figure 15, can also be applied to the others of the images as well. Moreover, in some aspects, the depicted images are characterized by the identification and It should be noted that although ordered according to the ease of formulating the response, it is not from other perspectives, for example, while landscape image 46 is depicted to the far right, but its geographical position data is strongly correlated with the metadata "Maroon Bells". Thus, this particular picture provides a case that is much easier than that provided by many other images.)

In one embodiment, this processing of the image occurs automatically-without using fast user instructions every hour. Subjects, information about power constraints and network connections can be collected continuously from such processing and used to process subsequent-captured images. For example, the initial image of the sequence including photo 44 shows the members of the depicted group without wearing sunglasses-simplifying the identification of those who later wore sunglasses.

16 and so on , Implementation

16 belongs to the heart of a particular implementation incorporating certain of the features discussed earlier. (Other discussed features can be implemented by a technician within this architecture, based on the disclosure provided.) In this data driven arrangement 30, the operation of the cell phone camera 32 is controlled by the setup module 34. It is dynamically controlled in accordance with the packet data transmitted by the control processor, and then controlled by the control processor module 36. (Control processor module 36 may be the primary processor or secondary processor of the cell phone, or this function may be distributed.) Packet data specifies the operations to be performed by a certain chain of processing stages 38.

In one particular implementation, the setup module 34 depicts-on a frame-by-frame basis-the parameters utilized by the camera 32 when collecting the exposure. The setup module 34 also specifies the type of data the camera outputs. These command parameters are passed in the first field 55 of the header portion 56 of the data packet 57 corresponding to that frame (FIG. 17).

For example, per frame, the setup module 34 may issue a packet 57 where the first field 55 instructs the camera about, for example, the length of exposure, aperture size, lens focus, field depth, and the like. have. The module 34 sums the sensor charges so that the sensor reduces the resolution (e.g., generates a frame of 640 x 480 data from a 1280 x 960 capable sensor) and outputs data only from the red-filtered sensor cells. Field 55 may be further specified to specify output only from the horizontal line of cells across the middle of the sensor, output only data from the 128 × 128 patch of cells from the center of the pixel data, and the like. The camera command field 55 may further specify the exact time at which the camera captures data—eg, to allow for desirable synchronization with, for example, ambient lighting (as described later below).

Each packet 56 issued by the setup module 34 may include different camera parameters in the first header field 55. Thus, the first packet may cause the camera 32 to capture the full frame image with an exposure time of 1 millisecond. The next packet can cause the camera to capture a full frame image with an exposure time of 10 milliseconds, and the third can specify an exposure time of 100 milliseconds. (These frames can later be combined and processed to produce a high dynamic range image.) The fourth packet instructs the camera to down-sample the data from the image sensor and outputs a 4 x 3 array of grayscale luminance values. The signals can be combined from differently color-filtered sensor cells to make them. The fifth packet may instruct the camera to output data only from an 8 × 8 patch of pixels at the center of the frame. The sixth packet may instruct the camera to output only five lines of image data from the top, bottom, middle and mid-top and mid-bottom rows of the sensor. The seventh packet may instruct the camera to output data only from the blue-filtered sensor cells. The eighth packet ignores any auto-focus instructions, but can instruct the camera to capture the entire frame at infinite focus instead. Etc.

Each such packet 57 is provided from setup module 34 via a bus or other data channel 60 to a camera controller module associated with the camera. (The details of a digital camera, including an array of photosensor cells, associated analog-to-digital converter and control circuitry, etc., are well known to technicians and are not discussed in detail.) The camera 32 is a header field 55 of a packet. Capture digital image data and stuff the resulting image data into the body 59 of the packet. This also deletes camera instructions 55 from the packet header (or marks the header field 55 in a manner that allows it to be ignored by subsequent processing stages).

If the packet 57 was written by the setup module 34, it also included a series of other header fields, each specifying how the corresponding consecutive post-sensor stage 38 processes the captured data. As shown in FIG. 16, there are several such post-sensor processing stages 38.

The camera 32 outputs an image-stuffed packet generated by the camera (pixel packet) on the bus or other data channel 61, which is passed to the first processing stage 38.

Stage 38 examines the header of the packet. Since the camera deletes (or is marked to be ignored) the command field 55 that passed camera instructions, the first header field encountered by the control of the stage 38 is field 58a. This field details the parameters of the operation to be applied by the stage 38 to the data in the body of the packet.

For example, field 58a specifies the parameters of the edge detection algorithm to be applied by stage 38 (or simply what such algorithm should be applied) to the image data of the packet. It may also specify that stage 38 is to replace the original image data in the body of the packet with the resulting edge-detected set of data. (Replacement of data rather than attachment may be indicated by the value of a single bit flag in the packet header.) Stage 38 performs the requested operation (this may involve configuring programmable hardware in specific implementations). have). The first stage 38 then deletes (or marks them to be ignored) from the packet header 56 and outputs the processed pixel packet for operation by the next processing stage.

The control of the next processing stage (including stages 38a and 38b discussed later here) examines the header of the packet. Since field 58a has been deleted (or marked to be ignored), the first field encountered is field 58b. In this particular packet, field 58b does not perform any processing on the data in the body of the packet, but instead instructs the second stage to simply delete field 58b from the packet header and pass the packet to the next stage. can do.

The next field of the packet header may instruct the third stage 38c to perform 2D FFT operations on image data found in the packet body based on 16 × 16 blocks. It is also used for the Internet interface to the air interface to address 216.239.32.10 executed by specified data (e.g., detailing the operations to be performed on the FFT data received by the computer at that address, such as texture classification). The stage may be instructed to hand off the processed FFT data. It is also in the stage to hand off a single 16 x 16 block of FFT data corresponding to the center of the image captured on the same or different air interface for transmission to address 12.232.235.27 again executed by corresponding instructions on usage. (Eg, search for a match in the archive of stored FFT data and return information if a match is found; also store this 16 x 16 block in an archive with an associated identifier). Finally, the header created by setup module 34 may instruct stage 38c to replace the body of the packet with a single 16 × 16 block of FFT data dispatched to the air interface. As before, the stage also edits the packet header to delete (or mark) the instructions that have been answered, so that the header instruction field for the next processing stage is encountered first.

In other arrangements, the addresses of the remote computers are not hard-coded. For example, the packet can include a pointer to a database record or memory location (either in the phone or in the cloud), which includes the destination address. Alternatively, stage 38c may be directed to the query router and response manager (eg, FIG. 7) to hand off the processed pixel packet. This module then looks at the pixel packet to determine what type of processing is required and routes it to the appropriate provider (either in the cell phone or in the cloud if the resources allow it-between stable static providers). To, or through auctioned providers). The provider returns the requested output data (e.g., texture classification information, and information about any matching FFT in the archive) and continues processing for each instruction of the next item in the pixel packet header.

The data flow continues through as many functions as specific operations may be required.

In the particular arrangement illustrated, each processing stage 38 removes the operating instructions from the packet header. These instructions are specified in the header in the sequence of processing stages, and this removal allows each stage to examine the first instructions remaining in the header for indication. Other arrangements can of course also be used alternatively. For example, the module may insert new information into the header based on the processing results—front, tail, or elsewhere in the sequence. This calibrated header then controls the packet flow and hence processing. do.)

In addition to outputting data for the next stage, each stage 38 may further include an output 31 that provides data back to the control processor module 36. For example, the processing undertaken by one of the local stages 38 should be adjusted to optimize the suitability of the upcoming frame of captured data for a particular type of processing (eg, object identification). This focus / exposure information can be used as predictive setup data for the camera and next time a frame of the same or similar form is captured. Control processor module 36 may set up a frame request using a filtered or time series prediction sequence of focus information from previous frames or a subset of those frames.

Error and status reporting functions can also be achieved using the outputs 31. Each stage may also have one or more other outputs 33-locally in the cell phone or remotely (“in the cloud”) to provide data to other processes or modules. The data (in packet form, or in other formats) may be directed to such outputs according to instructions in packet 57 or elsewhere.

For example, the processing module 38 may make a data flow selection based on some processing result that has been performed. For example, if the edge detection stage identifies a distinct contrast image, the outgoing packet can be routed to an external service provider for FFT processing. The provider can return the resulting FFT data to other stages. However, if the image has bad edges (such as out of focus), the system may not want the FFT to be performed on the data and subsequent processing. Thus, processing stages may cause branches in the data flow depending on the parameters of the processing (such as identified image features).

Instructions specifying these conventional branches may be included in the header of the packet 57 or they may be provided. 19 shows one arrangement. Instructions 58d specify a condition in the original packet 57, specify a location in memory 79, and subsequent instructions 58e'-58g 'can be read from the replacement and the condition is met. Can be replaced in the packet header. If the condition is not met, execution proceeds according to the header instructions already in the packet.

In other arrangements, other variations may be utilized. For example, all possible conditional instructions can be provided in a packet. In other arrangements, the packet architecture may still be used, but one or more header fields do not contain explicit instructions. Rather, they simply point to a memory location where, for example, the corresponding instructions (or data) are retrieved by the corresponding processing stage 38.

Memory 79 (which may include cloud components) may also facilitate adaptation of the processing flow when conditional branching is not utilized. For example, the processing stage may produce output data that determines parameters of a filter or other algorithm to be applied by later stages (eg, convolution kernel, time delay, pixel mask, etc.). These parameters may be identified (eg, determined / calculated, and stored) by the preprocessing stage of the memory and may be recalled for use by later stages. In FIG. 19, for example, processing stage 38 generates parameters that are stored in memory 79. Subsequent processing stage 38c later retrieves these parameters and uses them in the execution of its assigned operation. (Information in memory may be labeled to identify the module / provider in which they originated or in which they arrived <if known>, or other addressing arrangements may be used.) Thus, the processing flow may be controlled processor module 36. ) May be adapted to unknown environments and parameters when originally directed to the setup module 34 to compose the packet 57.

In one particular embodiment, each of the processing stages 38 includes hardware circuitry dedicated to a particular task. The first stage 38 can be a dedicated edge-detection processor. The third stage 38c may be a dedicated FFT processor. Different stages can be dedicated to other processes. These include DCT, Wavelet, Haar, Hough and Fourier-Melin transformation processors, different kinds of filters (e.g. Wiener, Low Pass, Bandpass, High Pass), and Face Recognition, Optical Character Recognition, Eigen Values Calculations, feature extraction, color and texture feature data, barcode decoding, watermark decoding, object segmentation, pattern recognition, age and gender detection, emotional classification, direction determination, compression, decompression, log-polar mapping, convolution, interpolation Decimation / down-sampling / anti-aliasing; All of the contributors such as correlation, square root and square operations, matrix multiplication, perspective transformation, butterfly operations (combining the results of smaller DFTs into a larger DFT, or a larger DCT decomposing into sub-transforms) Or stages for executing some.

These hardware processors may be field-configurable instead of dedicated. Thus, each of the processing blocks of FIG. 16 may be dynamically reconfigurable as long as the environment is justified. At one moment, the block may be configured as an FFT processing module. At the next moment, it may be configured as a filter stage or the like. At one point, the hardware processing chain can be configured as a barcode reader; Next, it may be configured as a face recognition system or the like.

This hardware reconfiguration information can be downloaded from the cloud or from services such as the Apple App Store. And, the information does not need to reside statically on the phone once downloaded-it can be called from the cloud / app store whenever needed.

Assuming broadband availability and speed increase, hardware reconfiguration data may be downloaded to the cell phone whenever the cell phone is turned on or initialized, or whenever a particular function is initialized. It is a dilemma to be placed on the market at a given time when a large number of different versions of an application are updated by different users who have downloaded last, depending on the problems companies are facing supporting heterogeneous versions of the product in the field. Each time the device or application is initialized, all or selected features of the last version are downloaded to the phone. And this works not only for the overall system functionality, but also for components such as hardware drivers, software for hardware layers, and the like. At each initialization, the hardware is newly constructed with the latest version of the applicable instructions. (For the code used during initialization, it can be downloaded for use in the next initialization.) Only when some updated code can be downloaded and certain applications need it-the hardware of FIG. 6 for specialized functions. As for constructing-it can be loaded dynamically. The instructions can also be adapted to specific platforms, for example, an iPhone device can utilize different accelerometers than an Android device, and application instructions can change accordingly.

In some embodiments, the respective use processors may be connected in a fixed order. The edge detection processor may be first, the FFT processor may be third, and so on.

Alternatively, processing modules may be interconnected by one or more buses (and / or crossbar arrangement or other interaction architecture) that allows any stage to receive data from any stage and output data to any stage. Can be. Another interconnect method is a network on a chip (effectively a packet-based LAN; similar to a crossbar in adaptability, but programmable by network protocols). Such arrangements may also support one or more stages to iteratively process the data-taking output as input to perform other processing.

One iterative processing arrangement is shown by stages 38a / 38b in FIG. 16. The output from the stage 38a can be taken as an input to the stage 38b. The stage 38b may be commanded to apply it back to the input of the stage 38a rather than to process the data. This can be looped as many times as desired. Once the iteration process by stage 38a is complete, its output can be passed to the next stage 38c in the chain.

In addition to merely acting as a pass-through stage, stage 38b may execute its own type of processing on the data processed by stage 38a. The output can be applied to the input of the stage 38a. Stage 38a may be instructed to reapply or pass through the processing on the data generated by stage 38b. Any series of combinations of stages 38a / 38b processing may thus be achieved.

The roles of the stages 38a and 38b in the above can also be reversed.

In this way, the stages 38a and 38b (1) apply one or more stage 38a processing to the data; (2) apply one or more stages 38b processing to the data; (3) apply a combination and sequence of stages 38a and 38b processes to the data; (4) can be operated to simply pass the input data to the next stage without processing.

The camera stage can be integrated into an iterative processing loop. For example, to obtain focus-locking, a packet can be passed from a camera to a processing module that evaluates focus. Examples may include an FFT stage—finding high frequency image components—finding strong edges, etc. Sample edge detection algorithms may include Canny, Sobel, and differential. Edge detection is useful for or tracking an object. The output from this processing module can be looped back to the controller module of the camera and the focus signal can change. The camera captures subsequent frames with the altered focus signal, and the resulting image is provided back to the processing module to evaluate the focus. This loop continues until the processing module is reported within the threshold range achieved. (Parameters in the packet header or memory may specify an output iteration limit, for example specifying that the iteration should end and output an error signal if the focus that meets the specified requirement is not met within 10 iterations. Can be specified.)

Although this discussion focuses on a series of data processing, images or other data can be processed in two or more parallel paths. For example, the output of stage 38d can be applied to two subsequent stages, each of which starts each branch of the fork in processing. These two chains can then be processed independently, or the data resulting from this processing can be combined-or used together-at a later stage. (Each of these processing chains may be branched.)

As is well known, forks will appear much earlier in the chain. That is, in most implementations, the parallel processing chain will be utilized to generate an image of human consumption-as opposed to a machine. Thus, parallel processing can branch to immediately follow the camera sensor 12, as shown by the mating point 17 of FIG. Processing for the human visual system 13 includes operations such as noise reduction, white balance and compression. In contrast, the process for object identification 14 may include the operations detailed in this specification.

When the architecture branches or involves other parallel processes, different modules may end their processing at different times. They may output data when processes terminate-asynchronously when a pipeline or other interconnect network is allowed. When the pipeline / network is free, the next module can deliver the finished results. Flow control can involve some arbitration, such as giving higher priority to one path or data. Packets may carry priority data that determines the ranking when arbitration is needed. For example, many image processing operations / modules use Fourier domain data, as generated by the FFT module. The output from the FFT module may thus be provided with a higher priority and higher priority than others in mediating data traffic so that Fourier data that may be needed by other modules may be made available with minimal delay.

In other implementations, some or all of the processing stages are not general purpose processors, but general purpose microprocessors programmed by software. In yet other implementations, the processors are hardware-reconfigurable. For example, some or all may be field programmable gate arrays, such as Xilinx Virtex series devices. Alternatively, they may be digital signal processing cores, such as Texas Instruments TMS320 series devices.

Other implementations may include PicoChip devices such as PC302 and PC312 multicore DSPs. These programming models allow each core to be coded independently (eg in C) and then communicate with others via an internal interconnect mesh. Associated tools particularly provide for the use of such processors in cellular devices.

Other implementations can utilize configurable logic on the ASIC. For example, the processor may include an area of configuration logic-mixed with dedicated logic. This allows for configurable logic into pipelines and pipelines with dedicated pipeline or bus interface circuits.

The implementation may also include one or more modules with a small CPU and RAM, programmable code space for firmware and a workspace for processing, essentially a dedicated core. Such a module can perform quite a wide range of calculations, which are then configurable as needed for processing using hardware.

All such devices may be placed in a bus, crossbar or other interconnect architecture that allows any stage to receive data from any stage and again output data to it. (FFTs or other transform processors implemented in this manner can be dynamically reconfigured to handle blocks such as 16 x 16, 64 x 64, 4096 x 4096, 1 x 64, 32 x 128, etc.)

In certain implementations, some processing modules are replicated-allowing parallel execution on parallel hardware. For example, several FFTs can be processed simultaneously.

In a variant arrangement, the packet carries instructions that serve to reconfigure the hardware of one or more processing modules. When a packet enters the module, the header causes the module to reconfigure the hardware before the image-related data is accepted for processing. Thus, the architecture is constructed in operation by packets (which may or may not carry image related data). Packets can similarly deliver firmware to be loaded into a module with a CPU core or into an application- or cloud-based layer; Similarly, use software instructions.

Module configuration instructions may be received wirelessly or via another external network; It does not always need to reside on the local system. If the user requests an operation for which no local instructions are available, the system can request reconfiguration data from a remote source.

Instead of conveying the configuration data / instructions themselves, the packet may simply carry an index number, a pointer, or other address information. This information can be used by the processing module to access the corresponding memory store where the required data / instructions can be retrieved. In the case of a cache, if local memory storage is not found to contain the data / instructions needed, they may be requested from another source (eg, access to an external network).

Such arrangements degrade the dynamic routing capability to the hardware layer-reconfiguring the module as data arrives.

Parallelism is widely used in graphics processing units (GPUs). Many computer systems utilize GPUs as coprocessors to process operations such as graphics rendering. Cell phones gradually include GPU chips to allow phones to serve as game platforms; These may be utilized to take advantage of certain implementations of the present technology. (Not limiting in the manner of example, a GPU may be used to perform bilinear and bicubic interpolation, projection transforms, filtering, and the like.

According to another aspect of the present technology, a GPU is used to correct lens aberrations and other optical distortions.

Cell phones often display optical nonlinearities such as barrel distortions, parameter focus variations, and the like. This is especially problematic when decoding digital watermark information from the captured image. Using the GPU, the image can be treated as a texture map and applied to the correction surface.

Typically, texture mapping is used to put images of walls or stone walls on the surface of a prison, for example. Texture memory data is referenced and mapped onto a plane or polygon when it is drawn. In this context, it is an image applied to a surface. The surface is formed so that the image is drawn arbitrarily by correcting the transformation.

Steganographic correction signals of the digitally watermarked image are used to identify the distortion to which the image is transformed. (See, eg, Digimarc's patent 6,590,996.) Each patch of a watermarked image can be characterized by affine transformation parameters such as translation and scale. The error function for each position of the captured frame can thereby be derived. From this error information, the corresponding surface can be devised-when the distorted image is projected by the GPU, the surface causes the image to appear in the anti-distorted original form.

The lens can be characterized in this way with a reference watermark image. Once the associated correction surface has been devised, it can be reused in other images captured through that optical system (since the associated distortion is fixed). Another image can be projected onto this correction surface by the GPU to correct lens distortion. (Different focal depths and apertures may require different correction functions, as the light path through the lens may be different.)

When a new image is captured, it can be initially rectified to remove the keystone / trapezoidal perspective effect. Once straightened (eg, re-squared with respect to camera lenses), local distortions can be corrected by mapping the straightened image with the GPU onto a correction surface.

Thus, the correction model is essentially on a polygonal surface, where the tilts and elevations correspond to focal irregularities. Each area of the image has a local transform matrix that allows correction of that piece of the image.

The same arrangement can likewise be used to correct distortion of the lens in the image projection system. Before projection, the image is mapped onto the synthesized correction surface-like a texture-to counteract lens distortion. When the processed image is projected through the lens, the lens distortion acts in opposition to the previously applied correction surface distortion, causing the corrected image to be projected from the system.

Reference was made to the depth of field as one of the parameters that could be utilized by camera 32 to aggregate the exposures. Although the lens can be accurately focused at only one distance, the decrease in sharpness is progressive over both sides of the focused distance. (The depth of field depends on the point spread function of the optics-including lens focal length and aperture.) As long as the captured pixels yield useful information for the intended operation, they do not have to be in perfect focus. do.

Sometimes focus algorithms track focus but fail to achieve-wasting cycles and battery life. In some examples it is better to simply grab frames in a series of different focus settings. A search tree of focal depths or field depths can be used. This is particularly useful when the image contains a number of objects of potential interest, each in a different plane. The system can capture a frame that is 6 inches in focus and another frame that is focused in 24 inches. Different frames may indicate that there are two objects of interest within the field of view, one better captured in one frame and the other better captured in the other frame. Or a 24-inch-focused frame is found to have no useful data, but a 6-inch-focused frame may contain enough identifiable frequency content to know that there are two or more target image planes. have. Based on the frequency content, one or more frames with different focus settings can then be captured. Or an area of a 24-inch-focused frame may have one of the Fourier properties, and an identical area of a 6-inch-focused frame may have a different setting of Fourier properties, between two frames From the difference of, the following experimental focus settings can be identified (eg at 10 inches), and another frame at that focus setting can be captured. Feedback is applied-it is not necessary to obtain a complete focus lock, but rather a search criterion to determine about other captures that may represent additional useful details. The search may branch and branch the information depending on the number of objects identified and the associated Fourier, etc., until satisfactory information about all objects is fathered.

A related approach is to capture and buffer a plurality of frames when the camera lens system makes adjustments to the intended focus setting. Analysis of the frame finally captured at the intended focus may suggest that the intermediate focus frames represent useful information about objects that are not initially present or that are not important, for example. One or more frames initially captured and buffered may then be recalled and processed to provide information whose importance was not initially recognized.

Camera control may also be responsive to spatial coordinate information. By using geographic location data and direction (eg, a magnetic field), the camera can verify that it captures the intended target. The camera set-up module may request images of specific objects or locations as well as specific exposure parameters. When the camera is in the correct position to capture a particular object (which may have been previously user specified or identified by computer processing), one or more frames of image data may be automatically captured. (In some arrangements, the orientation of the camera can be controlled by stepper motors or other electromechanical arrangements, such that the camera automatically sets the azimuth and altitude to capture image data from a particular orientation to capture the desired object. Electronic or fluid adjustment of the lens direction may also be utilized.)

As noted, the camera setup module can instruct the camera to capture a sequence of frames. In addition to advantages such as compositing a high dynamic range image, these frames can be aligned and combined to obtain super-resolution images. (As known in the art, super-resolution can be achieved by other kinds of methods. For example, the frequency content of the images can be analyzed, correlated with each other by linear transformation, and accurate Can be affine-converted into an alignment, and then overlayed and combined .. In addition to other applications, this can be used to decode digital watermark data from an image. If you are too far from the camera, this can be doubled by these super-resolution techniques to obtain the higher resolution needed for successful watermark decoding.)

In an exemplary embodiment, each processing stage substituted the processing results for input data contained in the packet when received. In other arrangements, the processed data may be added to the packet body while maintaining the data that originally exists. In this case, the packet grows during processing-because more information can be added. This may be disadvantageous in some contexts, but may also provide advantages. For example, it can avoid the need to branch the processing chain to two packets or two threads. Occasionally, both original and processed data may be useful at subsequent stages. For example, the FFT stage may add frequency domain information to the pixel containing the original pixel domain image. Both of these can be used to perform sub-pixel alignment for subsequent stages, for example super-resolution processing. Similarly, focus metrics can be extracted from the image and used by subsequent stages-depending on the image data.

It will be appreciated that the arrangements described above can be used to control the camera to generate different types of image data based on a frame-by-frame basis, and to control subsequent stages of the system to process each such frame differently. Thus, the system captures the first frame under conditions selected to optimize green watermark detection, the second frame under conditions selected to optimize barcode reading, and the third frame under conditions selected to optimize face recognition. To capture it, and so on. Subsequent stages may be instructed to process each of these frames differently in order to best extract the found data. All frames can be processed to detect lighting variations. Every other frame can be processed to evaluate the focus, for example, by calculating 16 x 16 pixel FFTs at nine different locations within the image frame. (Or there may be a fork that allows all frames to be evaluated for focus, and may be disabled or reconfigured to serve for other purposes when the focus branch is not needed.)

In some implementations, frame capture can be tuned to capture steganographic correction signals present in the digital watermark signal, regardless of successful self decoding of the watermark payload data. For example, the captured image data may be of low resolution-sufficient to identify the calibration signal, but insufficient to identify the payload. Or the camera may expose the image irrespective of human perception, for example, overexposing the image highlights to fade, or underexposing so that other parts of the image are indistinguishable. This exposure may be sufficient to capture the watermark direction signal. (Feedback can of course be utilized to capture one or more subsequent image frames-alleviating one or more defects of the previous image frame.)

Some digital watermarks are embedded in specific color channels (eg blue) rather than across colors as a modulation of image brightness (see, eg, co-owned patent application 12 / 337,029 to Reed). When capturing a frame containing such a watermark, the exposure can be selected to produce the maximum dynamic range in the blue channel, regardless of the exposure of other colors in the image (eg, 0-255 in an 8-bit sensor). ). One frame may be captured to maximize the dynamic range of one color, such as blue, and later frames may be captured to maximize the dynamic range of another color channel, such as yellow (ie, along the red-green axis). These frames are then aligned and a blue-yellow difference is determined. The frames can have completely different exposure times depending on the illumination, object, and the like.

Preferably, the system may have an operating mode that captures and processes the image even when the user does not intend to "snap" the photo. If the user presses the shutter button, otherwise-scheduled image capture / processing operations may be interrupted and the consumer photography mode may take the lead. In this mode, capture parameters and processes designed to enhance human visual system aspects of the image may be utilized instead.

(It will be appreciated that the particular embodiment shown in Figure 16 generates packets before any image data is collected. In contrast, Figure 10A and the associated discussion do not represent packets that exist before the camera. Both arrangements are both embodiments. That is, in FIG. 10A, packets may be established prior to the capture of image data by the camera, in which case the visual keyvector processing and packaging module uses pixel data-or, more typically, pixel data. Sub-sets or super-sets of-serve to insert into previously formed packets Similarly, in Figure 16, packets need not be generated until after the camera has captured image data.)

As noted earlier, one or more processing stages are remote from the cell phone. One or more pixel packets may be routed to (or through) the cloud for processing. The results may be returned to the cell phone or may be sent to other cloud processing stages (or both). Once back at the cell phone, one or more other local operations may be performed. The data can then be sent back out of the cloud. Processing can thus alternate between cell phones and the cloud. Finally, the resulting data can generally be provided back to the user at the cell phone.

Applicants expected different vendors to provide competing cloud services for specialized processing tasks. For example, Apple, Google, and Facebook may each offer cloud-based facial recognition services. The user device transmits the processed data packet for processing. The header of the packet may indicate the user, the requested service and-optionally-micro payment instructions. (Again, the header may be an index that serves to make the desired transaction look up in the cloud database, or to construct a sequence of actions for an action or some transaction such as purchase, mail on Facebook, face- or object-aware action, or Other identifiers may be passed in. Once this indexed transaction arrangement is initially configured, it can be easily invoked simply by sending a packet to the cloud containing the identifier and image-related data representing the desired behavior.)

In the Apple service, for example, the server examines the incoming packet, looks up the user's iPhoto account, accesses facial recognition data for the user's friends from that account, and the image data delivered with the packet. Facial recognition features can be calculated from, determining the best match, and returning the resulting information (eg, the name of the person depicted) back to the original device.

At the IP address for the Google service, the server can undertake a similar operation, but references the user's Picasa account. The same is true for Facebook.

Identifying one face among the faces for dozens or hundreds of known friends is easier than identifying the faces of strangers. Other vendors can provide the latter kind of services. For example, L-1 Identity Solks, Inc. maintains a database of images from government-issued certificates, such as driver's licenses. Appropriate permissions can be used to provide facial recognition services extracted from these databases.

Other processing operations can similarly be operated remotely. One is a barcode processor, which takes the processed image data sent from the mobile phone and applies a decoding algorithm specific to the type of barcode present. The service can support one, several, or dozens of different types of barcodes. The decoded data may be returned to the phone, or the service provider may access other data indexed by the decoded data, such as product information, instructions, purchase options, and the like, and return this other data to the phone. (Or both may be provided.)

Another service is digital watermark reading. Another service is optical character recognition (OCR). The OCR service provider may further provide transaction services, for example a service that transforms the processed image data into ASCII symbols and then provides ASCII words to the translation engine so that they are rendered in different languages. Other services are sampled in FIG. (Practicality provides a list of countless other services and also component operations that can be provided.)

Output from the remote service provider is often returned to the cell phone. In many cases, the remote service provider will return processed image data. In some cases, it may return ASCII or other such data. However, from time to time, the remote service provider may generate other forms of output, including audio (eg MP3) and / or video (eg MPEG4 and Adobe Fresh).

Video returned from the remote provider to the cell phone can be provided on the cell phone display. In some implementations, such video provides a user interface screen, inviting the user to touch or make a gesture within the displayed offer to select information or action or issue a command. The software of the cell phone may receive such user input and undertake response actions or provide response information.

In still other arrangements, the data provided back to the cell phone from the remote service provider may include JavaScript or other such instructions. When executed by the cell phone, JavaScript provides a response associated with the processed data to be queried to the remote provider.

Remote processing services may be provided under a variety of different financial models. The Apple iPhone service plan can be bundled with various remote services, such as iPhoto-based facial recognition, at no additional cost. Other services may be billed for each use, monthly subscription, or other usage plans.

You will not doubt that some services are very advanced and marketed. Others can compete for quality; Others can be priced.

As noted, the stored data may represent good providers for different services. These may be explicitly identified (eg, send all FFT operations to the Fraunhofer Institute service), or they may be specified by other attributes. For example, a cell phone user may be designated so that all remote service requests are routed to the fastest ranked providers in the provider's (eg, by consumer combination) periodically updated survey. The cell phone may periodically check the published results for this information, or may dynamically check when a service is requested. The other user may specify that service requests should be routed to service providers with the highest consumer satisfaction scores-again, with reference to the online rating resource. Another user may specify that routing should be made to providers with the highest consumer satisfaction scores-even if the service is provided free of charge; Otherwise it is routed to the lowest cost provider. Combinations of these arrangements and others are of course also possible. The user may, in certain cases, specify a particular service provider-trump any selection made by the stored profile data.

In still other arrangements, the user's request for a service may be mailed outwards and various service providers may express interest in performing the requested operation. Or the request can be sent to various specific service providers for proposals (eg, to Amazon, Google and Microsoft). Responses from different providers (price pricing, other conditions, etc.) can be provided to the user, and the user can select from them, or the selection can be made automatically-based on previously stored rules. In some cases, one or more competing service providers may be provided with user data and use it to execute the target operation entirely before they begin execution or the service provider selection is finally made—these providers respond to their response. You are given the opportunity to speed up time and encounter additional real data. (See also an initial discussion of remote service providers including auction-based services, for example in conjunction with FIGS. 7-12.)

As indicated elsewhere, certain external services may pass through a public hub (module), which is responsible for distributing those requests to appropriate service providers. Equivalently, results from certain external service requests can similarly be routed through a public hub. For example, payloads decoded by different service providers from different digital watermarks (or payloads decoded from different barcodes or fingerprints calculated from different content objects) are referred to the public hub. It can compile statistics and aggregate information (similar to Nelson's monitoring services-investigating different data and consumer encounters). In addition to coded watermark data (barcode data, fingerprint data), the hub may also be provided with (or alternatively) a quality or reliability metric associated with each decoding / calculation operation. This can help to indicate packaging problems, printing problems, media error problems, etc. that need to be considered.

Pipe manager

In the FIG. 16 implementation, communications to and from the cloud are facilitated by the pipe manager 51. This module (which can be realized as the cell phone side portion of the query router and answer handler in FIG. 7) performs various functions related to communicating over the data pipe 52. (Pipe 52 will know a data structure that cannot contain various communication channels.)

One function by the pipe manager 51 is to negotiate the necessary communication resources. The cell phone may utilize various communication networks and advertising data carriers such as cellular data, WiFi, Bluetooth, etc.—some or all of those that may be utilized. Each can have its own protocol stack. In one aspect, the pipe manager 51 interacts with respective interfaces for these data channels-determining the availability of bandwidth for different data payloads.

For example, the pipe manager alerts the cellular data carrier local interface and network that there is payload ready for transmission starting at about 450 milliseconds. It may further specify the size of the payload (eg two megabits), its character (eg block data), and the required quality of service (eg data throughput rate). It can also specify a priority level for the transmission, so that the interface and network can service this transmission in advance of low priority data exchanges in the event of a collision.

The pipe manager knows the expected size of the payload due to the information provided by the control processor module 36. (In the illustrated embodiment, the control processor module specifies the specific process for calculating the payload, so that it can estimate the size of the resulting data). The control processor module may also predict the character of the data, such as whether it will be available in bursts as a fixed block or intermittently, for example, the rate to be provided for transmission. The control processor module 36 may also predict the time for which data is ready for transmission. Priority information is also known by the control processor module. In some examples, the control processor module automatically sets the priority level. In other examples, the priority level is specified by a person or by the particular application being serviced.

For example, a user may explicitly signal via the cell phone's graphical user interface, or certain applications may regularly require that image-based operations be processed immediately. This may be the case, for example, when another action from the user is expected based on the results of the image processing. In other cases, the user may explicitly signal, or allow a particular application to be executed when the image-based operation is generally convenient (eg, when required resources may have low or empty utilization). can do. This may be the case, for example, when a user likes to post a snapshot to a social networking site such as Facebook, and an image annotated with the names of individuals depicted through facial recognition processing. For example, intermediate prioritization (expressed by the user or by the application), which is a process within one minute, ten minutes, one hour a day, or the like, may also be utilized.

In the illustrated arrangement, the control processor module 36 notifies the pipe manager of the expected data size, character, timing and priority, so that the pipe manager can use them in negotiations for the desired service. (In other embodiments, some information may be provided.)

If the carrier and the interface can meet the pipe manager's request, another data exchange can prepare for data transmission and the remote system can continue to prepare for the expected operation. For example, a pipe manager can establish a secure socket connection with a particular computer in the cloud that receives a particular data payload and identifies a user. If the cloud computer is executing a facial recognition operation, it may prepare for the operation by retrieving facial recognition features and associated names for the designated user's friends from Apple / Google / Facebook.

Thus, in addition to preparing the channel for external communication, the pipe manager enables pre-warming of the remote computer to prepare for the expected service request. (Service may be requested and may not be followed.) In some examples, the user can operate the shutter button, and the cell phone does not know what action to follow. Is the user request a facial recognition gesture? Is it a barcode decoding operation? Is the image mailed to Flickr or Facebook? In some cases, the pipe manager-or control processor module-can pre-warm several processes. Or it may predict, based on past experience, which action will be undertaken and warm the appropriate resource. (For example, if the user executed facial recognition operations follow the last three shutter operations, there is a good chance that the user will request facial recognition again.) The cell phone is actually loaded with various possible functions before anything is selected. May start executing component actions on-in particular, the results of which may be useful for various functions.

Pre-warming may also include resources within the cell phone: configuring processors, loading caches, and the like.

The situation reconsidered considerations that desired resources were ready to handle the expected traffic. In other situations, the pipe manager may report that the carrier is unavailable (eg due to a worsened wireless service state user). This information is reported to the control processor module 36 and can change the schedule of image processing, buffer results, or take other corresponding actions.

If other, conflicting data transmissions are in progress, the carrier or interface may respond to the pipe manager that the requested transmission cannot be accommodated, for example, at the requested time or with the requested quality of service. In such a case, the pipe manager may report this to the control processor module 36. The control processor module may trigger a 2-megabit data service requirement and stop rescheduled processing for later. Alternatively, the control processor module can determine that a 2 mega bit payload can be generated as originally scheduled and the results can be locally buffered for transmission when the carriers and interfaces can do so. Or other actions may be taken.

Consider a business meeting where participants gather for group photos before dinner. The user may want all faces in the picture to be recognized immediately, so that they can quickly search to avoid the difficulty of not recalling a colleague's name. Even before the user operates the user shutter button of the cell phone, the control processor module causes the system to process the frames of image data, thereby ellipsoidal shapes with two apparent eyes in the field of view (eg, expected locations). Identify faces that appear in These can be highlighted with rectangles on the cell phone's viewfinder (screen) display.

While current cameras may have taking modes based on lens / exposure profiles (eg, close-up, night, beach, landscape, snowy scenes, etc.), imaging devices may add different image-processing modes. Or (or alternatively). One mode may be selected for the user to obtain names of the person depicted in the picture (eg via face recognition). Another mode may be selected to effect optical character recognition of the text found in the image frame. The other can trigger actions related to purchasing the depicted item. Same as selling the depicted item. Same as obtaining information about the depicted object, scene or person (eg from Wikipedia, social networks, the manufacturer's website), and the like. It is the same as establishing a ThinkPipe session with an entry or related system.

These modes can be pre-activated or later selected by the user. In other arrangements, a plurality of shutter controls (physical or GUI) are provided to the user-each calling a different available operation. (In still other embodiments, it is inferred which action (s) are likely to be required rather than explicitly indicated to the user.)

If the user of the business meeting takes a group picture depicting 12 individuals and makes an immediate request based on names, the pipeline manager 51 informs the control processor module (or application software) that the requested service cannot be provided. You can report it again. Due to a bottleneck or other constraints, the manager 51 may report that only the identification of three of the faces depicted within the quality of service parameters considered to constitute the "on the fly" basis can be accepted. The other three faces can be recognized within 2 seconds and the recognition of the entire set of faces can be expected after 5 seconds. (This may be due to constraints by the remote service provider rather than the carrier in nature.

Control processor module 36 (or application software) may respond to this report in accordance with an algorithm or with reference to a set of rules stored in a local or remote data structure. The algorithm or rule set concludes that for face recognition operations, the delayed service must be accepted even if the clauses are available and the user is warned (via the device GUI) that there will be about N seconds of delay before the full results are available. Can be built. Optionally, the reported cause of the expected delay may be exposed to the user. Other service exceptions may be handled differently-in some cases the operation may be suspended, rescheduled or routed to a less desirable provider and / or not alerted to the user.

In addition to taking into account the capabilities of the local device interface to the network and the capabilities of the network / carrier, to handle predictive data traffic (within specified parameters), the pipeline manager also has query resources outside of the cloud − To ensure that services can run anything as requested (within specified parameters). These cloud resources can include, for example, data networks and remote computers. If anything responds negatively or responds with a service level condition, it can also be reported back to the control processor module 36 so that appropriate action can be taken.

In response to any communication from pipe manager 51 indicating the possibility of difficulty in servicing the expected data flow, control process 36 issues instructions corresponding to pipe manager and / or other modules as needed. can do.

In addition to the tasks just described above that pre-negotiate for the necessary services and set up appropriate data connections, the pipe manager can also act as a flow control manager-coordinating the transfer of data from different modules in the cell phone, and controlling the processor. Errors are reported back to module 36.

Although the above discussion has focused on outgoing data traffic, there is a similar flow coming back inside the cell phone. Pipe managers (and control processor modules) can help manage this traffic as well-providing services complementary to those discussed with respect to outgoing traffic.

In some embodiments, there may be a pipe manager correspondence module 53 outside of the cloud-cooperating with the pipe manager 51 of the cell phone in the execution of the functions described above.

Control handler and pipe Manager  software Example

Research in the field of autonomous robotics shares some similar challenges with the scenarios described herein, and in particular enables a system of sensors to communicate data to local and remote processes that cause locally taken actions. Share that In the case of robotics, it involves moving the robot in an inconvenient way; The most common focus in this case is on providing the desired experience based on the image, sound, etc. encountered.

In contrast to performing simple operations such as obstacle avoidance, aspects of the present technology desire to provide richer experiences based on higher levels of semantics and thus sensor input. The user who makes the camera point to the post does not want to know the distance to the wall; The user is much more inclined towards wanting to know about the content on the poster, if the poster is related to the movie, the show, reviews, what their friends think, and so on.

Despite these differences, architectural approaches from robotic toolkits can be adapted for use in the present context. One such robotic toolkit is something like a player project-a set of source free software tools and sensor applications available as open source from sourceforge-dot-net.

An example of the player project architecture is shown in FIG. 19A. Mobile robots (usually having relatively low performance processors) communicate with fixed servers (relatively higher performance processors) using wireless protocols. Various sensor peripherals are coupled to the mobile robot (client) processor via respective drivers and APIs. Similarly, services can be invoked by server handlers from software libraries via other APIs. (CMU CMVision library is shown in FIG. 19A.)

In addition to the basic tools for interfacing robotic devices to service libraries and sensors, the player project simulates the population of mobile robots moving in a 2D environment with various sensors and processing-including visual blow detection. Includes "rating" software rating "Gazebo" extends the stage model to 3D.)

With this system architecture, new sensors can be utilized quickly-by providing driver software to interface with the robotic API. Similarly, new sensors can be easily plugged in via the server API. Two player project APIs provide standardized abstractions so that drivers and services do not need to be associated with a particular configuration of a server or robot (or vice versa).

(FIG. 20A, discussed below, also provides an abstraction layer between locally available operations, externally available operations, and sensors.)

Certain embodiments of the present technology are a local processing & remote processing paradigm similar to that of a player project (eg named pipes, sockets, etc.) connected by packet networks and inter-process & intra-process communication structures familiar to the technicians. It can be implemented using. The communication minutiae is a protocol by which different processes can communicate; This may take the form of a message passing paradigm and message queue, or a more network-centric way in which collisions of keyvectors are handled after that fact (retransmission, interruption in fact timely, etc.).

In such embodiments, data from sensors on a mobile device (eg microphone, camera) may be packaged in keyvector form with associated instructions. Instruction (s) associated with the data may not be represented; These may be implicit (such as Bayer conversion) or session specific based on context or user needs (in the photographing mode, face recognition may be thought of).

In a particular arrangement, keyvectors from each sensor are generated and packaged by device driver software processes that extract hardware specific embodiments of the sensor and provide a fully formed keyvector that is faithful to the selected protocol.

The device driver software can then place a keyvector formed on an output queue unique to that sensor, or in a shared message queue shared by all sensors. Regardless of the manner, local processes can consume keyvectors and perform the necessary operations before placing the resulting keyvectors back on the queue. These keyvectors processed by the remote services are then placed in packets and sent directly to a remote service-similar to a router-that distributes the keyvectors or to remote processes for further processing. It is apparent to the reader that the instructions for initializing or setting up any sensors and processes in the system can be distributed in a similar manner from the control protocol (eg, box 36 of FIG. 16).

Branch  prediction; Commercial Incentives

The technique of branch prediction has gradually emerged to meet the needs of complex processor hardware; This allows a handler with long pipelines to fetch data and instructions (and, in some cases, to execute instructions) without waiting for conditional branches to be resolved.

Similar science can be applied in this context-predicting what actions a human user takes. For example, as discussed above, the system just described may "pre-warm" certain processors or communication channels in anticipation of certain data or processing operations coming.

What does the user want to do when the user removes the iPhone from his wallet (the sensor is exposed to increased light) and raises it to eye level (detected by the accelerometer)? References are made to past behavior to make predictions. In particular, the relevance was what the user did with the phone camera at the last time they were used; What the user did with the phone camera at the same time as yesterday (and a few weeks ago at that time); And what the user last did around the same location. Corresponding actions may be taken in prediction.

If the user longitude / latitude corresponds to a location in the video rental store, it helps. It can be expected to perform image recognition on the artwork from the DVD box. In order to speed up the possible recognition, perhaps SIFT or other feature recognition, reference data must be downloaded for the candidate DVDs and stored in the cell phone cache. Recent releases are good predictions (except age-limited (G-grade) movies or highly violent movies-stored profile data indicates that the user has no history of viewing these movies). Thus, the same is true of movies that viewers have seen in the past (as indicated in historical rental records-also available on phones).

If the user's location corresponds to a city street, and the magnetic field and other location data indicate that she is looking north, tilted up from the horizontal, what might you be interested in? Even without image data, a quick reference to online resources such as Google Streetview can suggest that she is looking at business signage along Fifth Avenue. Perhaps feature recognition reference data for this geography should be downloaded to the cache for quick matching of the image data to be obtained.

To speed up execution, the cache must be loaded in a reasonable way-so the most likely object is considered first. Google Streetview, whose location contains metadata indicating Fifth Avenue avenue, has signs for Starbucks, Nordstrom stores, and Thai restaurants. Stored profile data for users reveals daily visits to Starbucks (she has a brand loyalty card); She is a customer of regular clothing (using Macy's rather than Nordstrom's credit cards); She never eats in Thai restaurants. Perhaps the cache should be loaded to identify the Thai restaurant most quickly following the Starbucks sign, followed by Nordstrom.

Low resolution images captured for presentation on the viewfinder fail to trigger the camera's feature (eg, for exposure optimization) to highlight likely faces. It helps. There is no need to pre-warm the complex processing associated with facial recognition.

She touches the virtual shutter button to capture a frame of high resolution image, and image analysis proceeds-in an attempt to recognize what is in view, the camera application can overlay graphical links related to the objects in the captured frame. Can be. (Or this may happen without user action-the camera may be watching actively.)

In one particular arrangement, visual "baubles" (Figure 0) are overlaid on the captured image. Tapping on any of the baubles pulls up a screen of information, such as a ranked list of links. Unlike Google Web Search, which ranks search results in order based on aggregated user data, the camera application attempts a customized rank in the user's profile. If a Starbucks code or logo is found in the frame, the Starbucks link is placed on top of this user.

If signs for Starbucks, Nordstrom and Thai restaurants are all found, the links are generally provided in that order (per user preferences inferred from profile data). However, cell phone applications may have capitalist preferences and may wish to promote links by one or two (possibly not the best) locations if the circumstances are justified. In this case, the cell phone routinely sends IP packets to web servers at the addresses associated with each of the links, warning them that the iPhone user has recognized corporate synergy from a particular longitude / latitude. (If privacy considerations and user permissions are allowed, other user data may also be provided.) The Thai restaurant server responds immediately-providing the next two regulars any one item 25% lower (the restaurant's sale point). The system only displays four tables and no pending orders; no cooking). The restaurant server may provide 3 cents if the phone provides a discount offer to the user in the presentation of the search results, or 5 cents if it facilitates a link from the ranked list to the second place, or so provided in the results list. 10 cents if you have a discount. (Starbucks also responds to incentives as well as to attract). Cell phones quickly accept the provision of restaurants and payments are made quickly-more likely to the user (eg pay monthly phone bills) or to a phone carrier (eg AT & T). Links to Starbucks, Thai Restaurant, and Nordstrom are provided in that order, and links to the restaurant indicate discounts for the next two regulars.

Google's AdWord technology is well known. It determines which advertisements are to be provided as advertiser links close to the results of a Google web search, based on factors including the auction determined payment. Google has adapted this technology to provide advertisements for third party web sites and blogs based on the specific content of sites called Service AdSense.

According to another aspect of the present technology, the AdWord / AdSense technology extends to visual image search onto cell phones.

Consider a user located in small bookstores that snapped images of Warren Buffett Electric Snowball . The book is quickly recognized (which may also result from a regular Google search) rather than providing a corresponding Amazon linked at the top of the list, and the cell phone recognizes that the user is located in an independent bookstore. Context-based rules are first described as a result to provide a non-commercial link. Top ranked in this type is the Wall Street Journal book review and proceeds to the top of the list of links provided. However, courtesy is to this. The cell phone passes the book title or ISBN (or the image itself) to Google AdSense or AdWords, which identifies advertiser links to be associated with that object. (Google can independently run its own image analysis on any given image. In some cases, this cell phone-presented image can be paid-Google can utilize data from other kinds of resources. Google, Barnes and Noble have top advertiser positions, followed by alldiscountbooks-dot-net. The cell phone application may present these advertiser links in a graphically clear manner to indicate their origin (eg, in different parts of the display or in different colors), or alternately with them, along with the non-commercial search results, For example, it can be inserted in positions 2 and 4. AdSense revenue collected by Google may again be shared with the user or with the carrier of the user.

In some embodiments, the cell phone (or Google) again pings the servers of the companies for which the links are to be presented-helping to track their physical world-based online visibility. Pings include the location of the user and the identification of the object that facilitates the ping. When alldiscountbooks-dot-net receives a ping, it can check the inventory and see that it has a significant excess of Snowball inventory. As in the example provided earlier, additional payments can be provided for any additional promotion (eg, including "We have 732 bindings-cheap!" On the provided link).

In addition to providing incentives for more prominent search listings (e.g., higher in the list, or expanding to additional information), the company may also provide additional bandwidth to serve information to customers. For example, a user may want to capture a video image from an electronic bulletin board and download it to show a copy to friends. The user's cell phone identifies the content as a popular clip in the user-provided content (e.g., with reference to the encoded watermark), and the clip available from several sites-MySpace is the most popular following YouTube. Find it. In order to encourage users to link to MySpace, MySpace can offer to upgrade their baseline wireless service from 3 megabits per second to 10 megabits per second, so that the video will be downloaded in 1/3 hour. This upgraded service is just for video download or may be longer. The link provided on the screen of the user's cell phone can be calibrated to highlight the availability of faster services. (Again, MySpace can make associated payments.)

Sometimes, it is required to open a bandwidth throttle on the cell phone end of the radio link to mitigate network bottlenecks. Or a bandwidth service change must be requested or granted by the cell phone. In this case, MySpace can inform the cell phone application to take the necessary steps for higher bandwidth service, and MySpace will refund the additional associated costs to the user (or to the carrier for the benefit of the user's account). .

In some arrangements, the quality of service (eg, bandwidth) is managed by the pipe manager 51. Instructions from myspace may request the pipe manager to request increased quality of service and begin setting up the expected high bandwidth session even before the user selects myspace.

In some scenarios, vendors can negotiate an optional bandwidth for that content. MySpace can, for example, transact with AT & T that all myspace content delivered to AT & T phone subscribers is delivered at 10 megabits per second-most subscribers usually only receive 3 megabits of service per second. Higher quality service may be highlighted to the user at the provided link.

From the foregoing, in certain embodiments, the information provided by the mobile phone in response to the visual stimulus may be based on the user's demographic profile, both (1) the user's preferences, and (2) the third party contention. It will be recognized by everyone. Demographically identical but users with different tastes are likely to be served with different baubles or associated information when restaurants are looking at dense streets. Users with the same taste and different preference information—but with different demographic factors (eg, age, gender) —likewise have different demographic baubles because they want vendors, etc., to pay differently for different eyes. / Information may be provided.

Of user behavior modelling

With the help of knowledge of specific physical environments, specific places and times, and expected user behavior profiles, simulation models of the physical world and human computer interactions are distributed, such as robotics and viewership surveys, tools and techniques from the fields. Can be based on. This example includes the number of mobile devices expected in the museum at a particular time; Certain sensors that such devices are likely to be used; What stimuli are expected to be captured by these sensors (eg, where they are pointing at the camera, what the microphone is listening to, etc.). Additional information may also include assumptions about social relationships between users: are they likely to share common interests? Do they exist in common social circles that may want to share content, share experiences, or create location-based experiences such as wiki-maps (Mobile-HCI, 2009, “Map-Based Wikis” as Contextual and Cultural Mediators, "

In addition, modeling has historically been used for more advanced predictive models based on inherent human behavior (eg, people are more likely to capture images from the scoreboard during overtime than during game breaks). Based on generalized heuristic teaching derived from observations at events (eg, how many people used cell phone cameras to capture images from the scoreboard of the Portland Trail Blazers during the basketball game, etc.) Can be.

Such models may inform many aspects of the experience for the users, in addition to the business entities involved in preparing and measuring the experience.

These latter entities consist of typical value chain participants involved in the event product, and arrangements related to measuring and monetizing interactions. Event planners, producers, and technicians on the generating side and associated rights are loyalty side awards and communities (ASCAP, Directors Guild of America, etc.). In the measurement view, two sampling-based techniques and census-driven techniques from users and devices determined to subscribe may be utilized. Metrics for more static environments are based on unit revenue (RPU) generated by digital traffic generated on a digital service provider network, for more evolved models of Click Through Rates (CTR) for specific sensor stimuli. Per Unit) (how much bandwidth is being consumed).

For example, the Mona Lisa painting of the Louvre is likely to have a much higher CTR than other paintings in the museum, so that content preparation, for example, does not preload itself on the mobile device when the user accesses or enters the museum. For example, content related to Mona Lisa informs such things as priority for content preparation that should be cached as close to the edge of the cloud as possible. (Of course, the same importance is the role of the CTR in monetizing experiences and the environment.)

Consider a school group entering a sculpture museum with a garden with a collection of Rodin works. The museum can provide content related to Rodin and his works on servers or infrastructure (eg router caches) serving the garden. Moreover, because visitors include pre-established social groups, the museum can anticipate some social connectivity. Thus museums can enable sharing of capabilities (eg ad hoc networking) that may not otherwise be used. If a student queries the museum's online content to learn more about a particular Rodin sculpture, the system can achieve the delivery of detailed information by immediately inviting the student to share this information with the rest of the organization. The museum server may suggest a particular "friend" of the student with whom such information may be shared-if such information is publicly accessible from Facebook or other social networking data sources. In addition to friends 'names, this social networking data source can also provide device identifiers, IP addresses, profile information, etc. to students' friends-leveraged to help disseminate educational materials to the rest of the organization. have. Since this was of interest to other students in their group, these other students could find this relevant specific information-even if the original student's name was not identified. If the original student is identified along with the information conveyed, this can raise the attention of the information to the rest of the group.

(Detection of socially linked entities can be inferred from a review of the museum's network traffic. For example, a device sends packets of data to another device, and the museum's network sends two ends of the communication-dispatch and delivery. In the case of processing-there is an association between the two devices in the museum, if the devices are not devices with historical patterns of network usage, e.g. employees, the system has two visitors to the museum socially. If a web of such communications is detected-associating multiple unfamiliar devices, visitors of social organizations can be identified, the size of the organization being determined by the number of different participants in this network traffic. Can be surveyed. Demographic information about an organization Can be inferred from external addresses being exchanged; middle school students can have a high frequency of myspace traffic; college students can communicate with external addresses in the college domain; older citizens can argue different traffic profiles. All this information can be used to automatically adapt the information and services provided to visitors-not only to provide useful information for the management of the museum, but also to market department stores.)

Consider other situations. One is the U.S. U.S. It is a break of football superball. The show can have hundreds of fans capture the events of pictures or audio-video. Another context with predictable mass behavior is the end of the NBA Championship basketball game. Fans may want to celebrate the excitement of the final buzzer: scoreboards, streamers, and colored paper falling from the ceiling. In such cases, actions should be taken that may be taken to prepare or optimize, or deliver, the content or experience. Examples include granting rights to associated content; Render virtual worlds and other synthesized content, downtime routinely non-influenced network traffic, queuing advertising resources that can be called when a person purchases commemorative books / music from Amazon (caching pages, letting users access financial sites) To other), propagating links to post-game interviews (some pre-written / edited and ready to go), caching star players' Twitter feeds, video from the city center showing hometown crowds watching on Jumbotron displays Buffering-such as spewing pleasure from the buzzer; It includes everything related to experience or subsequent actions that need to be pre-cached / cached where possible.

Stimulus (audio, visual, tactile, smell, etc.) for sensors that are most likely to induce user behavior and attention is far more valuable from an advertising perspective than stimuli that are less likely to cause such behavior (Google's Adwords ad-serving system Similar to these underlying economic principles). These factors and metrics directly affect advertising models through auction models that are well understood by those skilled in the art.

A number of delivery mechanisms exist for the delivery of advertisements by third parties, leveraging known protocols such as VAST. Digital Video Ad Serving Template (VAST) is a standard issued by the Interactive Advertising Bureau that establishes not only associated XML schemas, but also reference communication protocols between ad servers and scriptable video rendering systems. For example, VAST generally standardizes the service of video ads on independent web sites based on bits of JavaScript contained in web page code-code that also helps in tracking traffic and managing cookies. To help replace the old banner ads. VAST may also insert promotional messages in pre-roll and post-roll viewing of other video content delivered by the web site. The website owner does not relate to selling or running ads, but the website owner receives payment based on viewership / impressions at the end of each month. In a similar manner, the physical stimulus provided to the user in the real world and sensed by the mobile technology may be the basis for payments to interested parties.

Dynamic environments in which stimuli are provided to users and their mobile devices are controlled (such as video displays in contrast to static posters), providing new opportunities for the measurement and utilization of metrics such as CTR.

Background music, content on digital displays, lighting, and the like can be modified to maximize CTR and build traffic. For example, illumination for a particular synergy may be increased or may flash as the targeted individual passes through. Similarly, when an airplane from Japan lands at the airport, digital signage, music, etc. may be used to explicitly or (securely change the advertisements to the expected audience's interest) or secretly (in Japanese websites) to maximize the CTR. Can be modified).

Mechanisms can likewise be introduced to combat false or unauthorized sensor stimuli. Within the confined spaces of the business park site, stimuli (posters, music, digital signage, etc.) that are not faithful to the property owner's intentions or policies-or entities responsible for the domain-may need to be managed. This can be achieved through the use of simple geo-specific simple blocking mechanisms (not different from region coding on DVD), so that all attempts within a particular GPS coordinate to route a keyvector to a specific place in the cloud is managed by the domain owner. Indicates that it should be arbitrated by a gateway or routing service.

Other options include filtering the resulting experience. Are you old? Does Denver Nuggets run against pre-existing advertising or branding arrangements, such as Coca-Cola advertisements delivered to users inside the Pepsi Center during the game?

This is done by sticking to MovieLabs content recognition rules related to conflicting media content (compare www.movielabs-dot-com / CRR), parental controls provided by carriers to the device, or DMCA automatic take down knots, It can likewise be achieved on the device through the use of content rules.

Under various rights management paradigms, licenses serve as the key to determining how content is consumed, shared, modified, etc. The result of extracting meaning from the stimulus provided to the location and / or the user (and the user's mobile device) provided the stimulus may be the issuance of a license for the desired content or experiences (games, etc.) by a third party. have. To illustrate, consider a user at a rock concert on stage. A user may be granted a temporary license to pre-use and listen to all music tracks by a performing artist (and / or others) on iTunes. However, such a license can be maintained only during the concert, only from when the doors are opened until the headline operation starts performing, or while the user is on stage. The license then terminates.

Similarly, passengers dropping from international flights are translating services or navigation services (e.g., camera-captured scenes on their mobile devices) for their mobile devices while at the airport 90 minutes after their arrival through customs. Location-based or time-limited licenses for augmented real-world systems that overlay directions for finding, restrooms, and the like.

Such arrangements can serve as metaphors and filtering mechanisms for experiences. One embodiment in which sharing of experiences is triggered by sensor stimulation is via broadcast social networks (eg, Twitter) and syndication protocols (eg, RSS web feeds / channels). Other users, entities, or devices may use such broadcasts / feeds as the basis for subsequent communications (social, information retrieval, etc.) upon logging of measurements (viewing rates, etc.) or activities (eg, a daily journal of a person). You can join them. Traffic associated with such networks / feeds can also be measured by devices at a particular location-allowing users to traverse time to know who was communicating at any particular point in time. This makes it possible to retrieve and gather additional information, for example, was my friend here last weekend? Was anyone in my peer group here? What content was consumed? This traffic also makes it possible to monitor in real time which users share experiences. "Tweets" monitoring of the performer's song selection during the concert allows the performer to change the songs to be played for the remainder of the concert. The same applies to brand management. For example, if users share opinions about a vehicle during vehicle display, live keyword filtering on traffic may allow brand owners to relocate specific products for maximum effectiveness (e.g., a new model of Corvette spinning Spend more time on the platform, etc.)

Addition to optimization

Predicting the user's behavior or intention is a form of optimization. Another form involves configuring the process to improve performance.

To illustrate one particular arrangement, again consider the common services classifier of FIG. 6. What keyvector operations should be run locally or remotely, or what kind of hybrid? In what order should keyvector operations be executed? Etc. The mix of expected operations and their scheduling should be configured in a suitable manner, environments and context for the processing architecture used.

One step of the process is to determine what actions should be taken. This determination may be based on explicit requests from the user, historical patterns of use, context and status, and the like.

Many operations are high level functions, which involve the operations of multiple components-executed in a particular order. For example, optical character recognition may require template pattern matching following edge detection followed by region of interest segmentation. Facial recognition may involve skin tone detection, Hough transforms (to identify elliptical regions), identification of feature locations (pupils, mouth, nose), eigenface calculations, and template matching.

The system can identify component operations that may need to be executed, and the order in which their respective results are required. Rules and heuristics can be applied to help determine whether these actions should be executed locally or remotely.

For example, at one extreme, rules may specify that simple actions, such as color histograms and thresholding, should generally be executed locally. At the other extreme, complex operations can generally be defaulted to external providers.

Scheduling may be determined based on which operations are prerequisites for other operations. This may also affect whether the operation should be executed locally or remotely (local execution may provide faster results-allowing subsequent operations to start with less delay). The rules seek to identify the action whose output (s) are to be used by the maximum number of subsequent actions, and may execute this action first (allowing its respective precedent (s)). Actions that are prerequisites for a small number of successive operations are subsequently executed sequentially. The operations and their sequence can be drawn as a tree structure-the most globally important ones are executed first and the lower relevance operations to other operations are later executed.

However, these decisions may be adjusted (or dependent) by other factors. One is power. If the cell phone battery is low, or the operation will involve significant drain on a low capacity battery, this may tip the balance to allow the operation to be performed remotely.

Another factor is response time. In some examples, the limited processing power of the cell phone may mean that local processing is slower than remote processing (eg, a more powerful, parallel architecture may enable operation). In other examples, delays in establishing communication with a remote server and establishing a session can make local execution of the operation faster. Depending on the user request and the needs of other operation (s), the speed at which results are returned may or may not be important.

Another factor is user preferences. As noted elsewhere, the user can set parameters that affect where and when operations are performed. For example, a user may specify that an operation may be referenced for remote processing by a domestic service provider, but if nothing is available, the operation should be executed locally.

Routing constraints are another factor. Occasionally, a cell phone will be in WiFi or other service area (eg at a concert stage), where the local network provider places restrictions or conditions on remote service requests that can be accessed over that network. In concerts where photography is prohibited, for example, the local network may be configured to block access to external image processing service providers for the duration of the concert. In this case, services that are normally routed for external execution should run locally.

Another factor is the specific hardware on which the cell phone is mounted. If dedicated FFT processing is available at the phone, execution of intensive FFT operations is used locally. If only a weak general purpose CPU is available, it is most likely that intensive FFT operations are externally referenced for external execution.

A related factor is current hardware utilization. Even if the cell phone is equipped with well-configured hardware for a particular task, it may be too busy to be backlogged by the system to refer to an external source to complete this kind of next task.

Another factor may be the risk of stall and length of the local processing chain. Pipeline processing architectures can be stalled for intervals when waiting for data required to complete an operation. This stall can cause all other subsequent operations to be similarly delayed. The risk of possible stall is assessed (eg, by historical patterns, or by knowledge that requires availability of other data—such as results from other external processing—where availability is not guaranteed when the completion of the operation is appropriate). If this is large enough, the operation can be referenced to external processing to avoid stalling the local processing chain.

Another factor is the connection state. Is a reliable high-speed network connection established? Or are the packets dropped, or is the network slow (or completely unavailable)?

Different kinds of geographic considerations can also be factors. One is a network close to the service provider. The other is whether the cell phone has limited access to the network (as in the home area) or a pay-per-use arrangement (as when roaming to another country).

Information about the remote service provider (s) can also be a factor. Does the service provider provide immediate turnaround or are the requested actions placed in a long queue later than other users waiting for service? When the provider is ready to handle a job, what speed is expected to run? Along with other attributes of importance to the user (eg, whether the service provider meets the "green" standards of environmental responsibility), the costs can also be key factors. Very many other factors may also be considered when suitable in certain contexts. Sources for such data may include external resources as well as the various elements shown in example block diagrams.

A conceptual diagram of the above is provided in FIG. 19B.

Based on various factors, a determination is made whether operations should be executed locally or remotely. (The same factors can be evaluated to determine the order in which actions should be executed.)

In some embodiments, different factors may be quantized by the scores, which may be combined in a polynomial manner to yield an overall score indicating how the operation should be processed. This overall score serves as a metric indicating the relevant suitability of the action for remote or external processing. (Similar scoring methods can be utilized to select among different service providers.)

Depending on changing environments, a given action can be executed locally at one moment and remotely at a later moment (or vice versa). Or the same operation can be performed simultaneously on two sets of keyvector data-one locally and one remotely.

Although described in the context of determining whether an operation should be executed locally or remotely, the same factors can likewise affect others. For example, they can also be used when determining what information is conveyed by keyvectors.

Consider an environment where cell phones run OCR on captured images. Using a set of factors, raw pixel data from the captured image can be sent to the remote service provider to make this determination. Under different sets of factors, the cell phone can perform initial processing such as edge detection, then package edge-detected data in keyvector form and route it to an external provider to complete the OCR operation. have. Under another set of factors, the cell phone executes OCR operations of all components until the final (template matching) and transmits data only for this final operation. (Under another set of factors, the OCR operation may be completed completely by the cell phone, or the operation of the different components may be alternately executed by the cell phone and the remote service provider (s), etc.)

Reference is made to routing constraints as one possible factor. This is a specific example of a more general factor-external business rules. Consider an early example of a user who attended an event at the Pepsi Center in Denver. The Pepsi Center can provide wireless communication services to customers via its own WiFi or other network. Naturally, the Pepsi Center is not happy with the network resources that will be used in the interests of competitors such as Coca-Cola. Thus, the host network may affect cloud services that may be utilized by its heads (eg, by making something inaccessible or low priority to data traffic with certain types or specific destinations). By providing it). The domain owner can exercise control over what actions the mobile device can execute. This control may affect the type of data carried in keyvector packets, as well as local / remote decisions.

Another example is a gym where you may want to stop using cell phone cameras, for example, by disrupting access to remote image providers as well as photo sharing sites such as Flickr and Picasa. Another example is a school that may want to discontinue facial recognition of students and stakeholders for privacy reasons. In such a case, access to facial recognition service providers may be blocked, or may be granted on a case-by-case basis. While on stages it can be seen that it is difficult for individuals to stop using a cell phone camera-or to use them for specific purposes-they can take various actions to prevent such use (eg, using such a use). By denying services that facilitate or facilitate).

The following outlines identify other factors that may be involved in determining which operations should be executed where and in what sequence:

1. Scheduling optimization of keyvector processing units based on a number of factors:

Operation mix, operations consist of similar atomic instructions (MicroOps, Pentium II, etc.)

Stall states, actions will create stalls for the following reasons:

Wait for external keyvector processing

Poor connectivity

User input

Change of user focus

o Cost of action based on:

Published costs

Estimated cost based on auction status

Battery status and power mode

Power profile of operation (expensive?)

Past history of power consumption

Provide opportunity cost, current status of the device, for example, what other processes should be prioritized, such as voice calls, GPS navigation, etc.

User preferences, i.e. I want a "green" provider or an open source provider

Legal uncertainties (e.g., certain providers may be at greater risk of patent infringement responsibilities, eg due to the use of claimed patent methods)

o Domain owner impact:

The privacy implications of certain physical stages, such as no facial recognition in schools

Predetermined content based on rules that prohibit specific actions for a particular stimulus

Voiceprint matching of broadcast songs highlighting the use of other singers (milli-vanilli's grammys are invalid when managers know that the actual vocals at the time of recording were performed by other singers)

o the ability to run out of order of keyvectors based on all the above mentioned impact scheduling and the optimal path to the desired result

Uncertainty in long chain operations that makes it difficult to predict the need for subsequent keyvector operations (similar to the deep pipeline in processors & branch predictions)-there may be difficulties due to weak metrics on keyvectors have

Past behavior

Position (GPS indicates that the device is moving quickly) & pattern of GPS movements

There are patterns of exposure to stimuli, such as a user walking through an airport terminal repeatedly exposed to the CNN provided at each gate.

Proximity sensors indicating the presence of a device in the pocket, etc.

Other approaches, such as Least Recently Used (LRU), can be used to track how often the desired keyvector behavior that results in or contributes to the desired effect (song recognition, etc.) is not frequent.

Other related pipelined or other time-consuming operations, certain embodiments may undertake some conformance testing before associating processing resources for what may be more than a threshold number of clock cycles. A simple conformance test is to ensure that image data is potentially useful for its intended purpose, in contrast to data that can be quickly disqualified from the analysis. For example, it is all black (e.g., frames captured in the user's pocket). Proper focus can also be quickly identified before being accommodated in extended operation.

(It will be appreciated that certain aspects of this technique discussed above have visible precedents when considered late. For example, considerable work is devoted to instruction optimization for pipelined processors. Allowing user configuration of power settings, such as user-selectable deactivation of power-deficient GPUs on certain Apple notebooks to extend battery life.)

The above discussed decision of proper instruction mixing (eg, by the common service classifier of FIG. 6) specifically took into account certain problems arising in pipelined architectures. Different principles may apply to embodiments in which one or more GPUs are available. These devices typically have hundreds or thousands of scalar processors that are adapted for parallel execution, so the costs of execution (time, stall risk, etc.) are small. Branch prediction can be handled without making predictions; Instead, the GPU processes in parallel all of the potential results of the branch, and the system uses whatever output is known that corresponds to the actual branch condition.

To illustrate, consider face recognition. The GPU-mounted cell phone can invoke instructions-when the camera is activated in user photo-shoot mode-making up 20 clusters of scalar processors on the GPU. (These clusters are sometimes referred to as "stream processors.") In particular, each cluster is configured to perform a Hough transform on a small tile from the captured image frame-finding one or more ellipse shapes that can be candidate faces. Thus, the GPU processes the entire frame in parallel by twenty simultaneous Hough transforms. (Many stream processors are probably not found, but the processing speed doesn't get worse.)

Once these GPU Hough transform operations are complete, the GPU can be reconstructed with fewer stream processors-analyzing each candidate ellipse shape to determine the positions, nose positions, and mouth-to-mouth distances of the pupils of the eye. Dedicated For any ellipse that yielded useful candidate face information, the associated parameters are packaged in keyvector form and sent to the cloud service, for example, the keyvectors of the analyzed face parameters for known templates of the user's Facebook friends. Check them. (Or such verification may be performed by another processor on the GPU or cell phone.)

Such face recognition (such as others described herein above) may comprise tens, hundreds, or thousands of bytes of volume in an original captured image, for example, from millions of pixels (bytes). It is of interest to extract to the keyvectors that are present in. These smaller pieces of information with more dense information content are routed more quickly for processing-sometimes externally. Ability to occur through the channel-to maintain cost-effectiveness and practicality of implementation.)

Contrast with the just described GPU implementation of face detection for such an operation as can be implemented on a scalar processor. Performing huff-transform-based elliptic detection over the entire image frame is suppressed in terms of processing time-more effort is worthless, and delays other tasks assigned to the processor. Instead, this implementation typically causes the processor to examine the pixels as they come out of the camera-looking for those with colors within the expected "skin tone" range. Only when a region of skin tone pixels is identified, a Hough transform is attempted for that excerpt of the image data. In a similar manner, attempts to extract facial parameters from detected ellipses are done in a series of difficult ways-often producing undesirable results.

Ambient light

Many artificial light sources do not provide consistent illumination. Most show temporary variations in intensity (luminance) and / or color. These variants generally follow the AC power frequency (50/60 or 100/120 Hz) but sometimes not. For example, fluorescent tubes can emit infrared light that changes at a -40 KHz rate. The emitted spectra depend on the specific illumination technique. Organic LEDs for home and industrial lighting can sometimes use separate color blends (eg, blue and amber) to make white. Others may utilize more conventional red / green / blue clusters or blue / UV LEDs with phosphors.

In one particular implementation, processing stage 38 monitors the average intensity, red, green, or other natural color of the image data, for example, in the bodies of the packets. This intensity data can be applied to the output 33 of that stage. Using the image data, each packet can carry a timestamp indicating the specific time (based on an absolute value or a local clock) at which the image data was captured. This time data can also be provided to the output 33.

Synchronization processor 35 coupled to such output 33 may examine the variation of frame-to-frame intensity (or color) as a function of timestamp data to identify its periodicity. Moreover, such a module can predict the next time instant at which the intensity (or color) has a maximum, minimum, or other specific state. The phase-locked loop can control the oscillator to be synchronized to reflect the periodicity of aspects of the illumination. More typically, the digital filter calculates a time interval-optionally with software interrupts-used to set or compare against timers. Digital phase-locked loops or delay-locked loops may also be used. (Kalman filters are commonly used for this type of phase lock.)

The control processor module 36 may poll the synchronization module 35 to determine when the lighting condition is expected to have the desired state. Using this information, control processor module 36 may instruct setup module 34 to capture a frame of data under preferred lighting conditions for a particular use. For example, if the camera is imaging an object that is assumed to have a digital watermark encoded in the green channel, processor 36 may capture camera 32 to capture a frame of the image at the moment when the green light is expected to be at its maximum. And processing stages 38 to process the frame for detection of such a watermark.

The camera phone can typically be equipped with a plurality of LED light sources that are also operated in series to produce a flash of white light illumination for the subject. However, operated individually or in different combinations, they can cast different colors of light to the object. The phone processor may individually control component LED sources to capture frames with non-white illumination. If you capture an image that is read to decode the green-channel watermark, only green illumination can be applied when the frame is captured. Or the camera can capture a plurality of consecutive frames-different LEDs illuminate the subject. One frame can be captured at 1 / 250th second with the corresponding period of red single illumination; Subsequent frames can be captured at the 1 / 100th second with the corresponding duration of the green only illumination. These frames can be analyzed individually or combined, for example to analyze in groups. Or a single image frame can be captured over an interval of 1 / 100th second, a red LED is activated at its entire interval, and a red LED is activated for 1 / 250th of its 1 / 100th interval. Instantaneous ambient light can be detected (or predicted as above), and component LED color light sources can be operated in each manner (eg, by adding blue light from a blue LED to tungsten light). To work against the orange).

Other notices; Projectors

Although a packet-based, data driven architecture is shown in FIG. 16, various other implementations are naturally possible. These alternative architectures are easy for those skilled in the art based on the details given.

Those skilled in the art will appreciate that the arrangements and details described above are arbitrary. The actual choices of arrangements and details will depend on the particular applications being served and most likely will be different from the well known. (This is a trivial example, but for the sake of quoting, the FFTs may not be performed on 16 x 16 blocks, but on 64 x 64, 256 x 256, full image, etc.)

Similarly, it will be appreciated that the body of a packet may carry an entire frame of data or only excerpts (eg, 128 x 128 blocks). Thus, image data from a single captured frame spans a series of multiple packets. Different excerpts in the common frame may be processed differently depending on the packet in which they are delivered.

Moreover, processing stage 38 may be instructed to divide one packet into multiple packets, such as by splitting the image data into 16 tiled smaller sub-images. Thus, more packets may be provided at the end of the system than generated at the beginning.

In the same way, a single packet can be a series of different images (e.g., images taken sequentially with different focus, aperture or shutter settings: a specific example is the depth of field bracket-overlapping, arranging or tearing down-or focusing). A set of focus areas from five images taken with a bracket). This set of data can then be processed by later stages-as a set or through the processing, the processing selects one or more excerpts of the packet payload that meet the specified criteria (eg, focus sharpness metric). .

In the specific example described above, each processing stage 38 generally substitutes the result of processing the originally received data of the body of the packet. In other arrangements, this need not be the case. For example, the stage may output the processing result to a module outside the depicted processing chain, for example to output 33. (Or, as noted, the stage may retain the originally received data-in the body of the output packet and augment it with other data-as its processing result (s).)

Reference was made to determine focus by referring to DCT frequency spectra or edge detected data. Many consumer cameras perform a simpler form of focus confirmation-simply by determining the intensity difference (contrast) between pairs of adjacent pixels. This difference is peaked at the correct focus. Such an arrangement can naturally be used in the arrangements described above. (Again, benefits can accumulate from performing this process on the sensor chip.)

Each stage typically performs a handshaking exchange with an adjacent stage-each time data is passed to or received from the adjacent stage. Such handshaking is routine to those skilled in the art familiar with digital system design and, therefore, is not discussed in detail here.

The above-described arrangements considered a single image sensor. However, in other embodiments, multiple image sensors may be used. In addition to enabling conventional stereoscopic processing, two or more image sensors enable or enhance many other operations.

One function that benefits from multiple cameras is identifying objects. To quote a simple example, a single camera cannot identify a human face from an image of a face (eg, as it can be found in a magazine, on a billboard, or on an electronic display screen). Using space-spaced sensors, in contrast, the 3D aspect of the image can be easily distinguished, allowing the image to be identified by a person. (Depending on the implementation, it may be a 3D aspect of a person that is actually distinct.)

Another feature that benefits from multiple cameras is the refinement of the geographic location. From the differences between the two images, the processor can determine the distance of the device from landmarks whose location can be known accurately. This allows for improvement of other geographic location data available to the device (eg WiFi node identification, GPS, etc.).

Since a cell phone may have one or two (or more) sensors, such a device may also have one, two (or more) projectors. Individual projectors are being deployed in cell phones by CKing (N70 model distributed by China Vision) and Samsung (MPB200). LG and others have shown prototypes. (These projectors are understood to use Texas Instruments electronically adjustable digital micro-mirror arrays with LED or laser lighting.) Microvision provides a PicoP display engine, which is a micro-electro-mechanical scanning mirror (laser sources And optical combiner) to integrate the various devices to calculate projector capabilities. Other suitable projection techniques include DisplayTech's ferroelectric LCOS systems and 3M liquid crystals on silicon (LCOS).

The use of two projectors or two cameras provides differentials of projection or viewing and provides additional information about the subject. In addition to the stereo features, it also enables local image correction. For example, consider two cameras that image a digital watermarked object. One camera view of an object provides a measure of the transformation that can be identified from the surface of the object (eg, by encoded calibration signals). This information can be used by other cameras to correct the view of the object. And vice versa. The two cameras can be repeated, yielding a comprehensive feature of the object surface. (One camera can see a better example area of the surface and see other edges that other cameras can't see. Thus, one view can represent information that no other view represents.)

When a reference pattern (eg, a grid) is projected onto the surface, the shape of the surface is revealed by the distortions of the pattern. 16 may be extended to include a projector, which projects a pattern onto an object for capture by a camera system. (The operation of the projector can be synchronized with the operation of the camera, for example by the control processor module 36-since it imposes an important battery drain, it is only activated when the project is needed.) Modules 38 (local or The processing of the resulting image by remote provides information about the surface topology of the project. This 3D topology information can be used as a clue to identify the object.

In addition to providing information about the 3D configuration of the object, the shape information allows the surface to be virtually remapped to any other configuration, for example a plane. This remapping serves as a kind of normalization behavior.

In one particular arrangement, the system 30 operates the projector to project the reference pattern into the field of view of the camera. Once the pattern is projected, the camera captures a frame of image data. The resulting image is processed to detect the reference pattern, from which the 3D shape of the image project is characterized. Subsequent processing follows thereafter based on the 3D shape data.

(With respect to these arrangements, the reader refers to Google Book-Scanning Patent 7,508,978, which utilizes related principles. The patent details reference patterns which are particularly useful among the relevant disclosures.)

If the projector uses aimed laser light (such as the PicoP display engine), the pattern will be focused regardless of the distance to the object on which the pattern is projected. This can be used to help the cell phone camera's focus adjust to any object. Since the projected pattern is known in advance by the camera, the captured image data can be processed to optimize the detection of the pattern-as by correction. (Or patterns can be selected to facilitate detection-such as a checkerboard that appears strongly at a single frequency in the image frequency domain when properly focused.) Once the camera is at the optimal focus of a known aimed pattern Once adjusted, the projected pattern can be discontinuous and the camera can then capture a properly focused image of the subject to which the pattern was projected.

Simultaneous detection can also be utilized. The pattern can be projected during the capture of one frame and then off for the next capture. The two frames can then be removed. The common image in the two frames is generally erased-leaving the projected pattern of a much higher signal-to-noise ratio.

The projected pattern can be used to determine the correct focus for various objects in the field of view of the camera. The child can pose in front of the Grand Cannon. The laser-projected pattern allows the camera to focus the eye in the first frame of the second frame and the background in the second frame. These frames can then be synthesized-taking the appropriately focused portion from each.

If a lens arrangement is used in the projector system of the cell phone, the camera system of the cell phone may also be used. The mirror may be controllably moved to adjust the camera or projector to the lens. Or beamsplitter arrangement 80 is used (FIG. 20). The body of the cell phone 81 here incorporates a lens 82 that provides light to the beam-splitter 84. Some of the illumination is routed to the camera sensor 12. The other part of the light path goes to the micro-mirror projector system 86.

Lenses used in cell phone projectors are typically larger in aperture than those used in cell phone cameras, so that the camera allows significant performance advantages (eg, shorter exposures) by the use of such a shared lens. Can be obtained). Or mutually, the beam splitter 84 may be asymmetrical-without equally favoring the two light paths. For example, the beam-splitter may be a partially silver device that externally couples a smaller fraction of incident light (eg, 2%, 8%, or 25%) to the sensor path 83. Thus, the beam-splitter may serve to externally combine a larger fraction of the illumination (eg, 98%, 92%, or 75%) from the micro-mirror projector for projection. By this arrangement, the camera sensor 12 receives light of the intensity (in spite of the larger aperture lenses), which is conventional-for a cell phone camera, but the light output from the projector is only slightly by the lens sharing arrangement. Cloudy

In other arrangements, the camera head is detached from the cell phone body-or detachable. The cell phone body is carried in the user's pocket or purse, but the camera head is adapted to look beyond the user's pocket (e.g., in the form of a pen-like factor, with pocket clips, with batteries in the pen barrel) ). The two communicate by Bluetooth or other wireless arrangement and capture image data sent from the camera head and instructions sent from the phone body. This configuration allows the camera to constantly look at the scene in front of the user-without the cell phone having to be removed from the user's pocket / wallet.

In a related arrangement, the strobe light for the camera is separated from the cell phone body-or detachable. Light (which may incorporate LEDs) may be placed near the image object, providing illumination from the desired street and distance. The strobe can be started by radio commands issued by the cell phone camera system.

(A person skilled in the art of optical system design will know a number of alternatives to specially known arrangements.)

Some of the advantages resulting from having two cameras can be realized by having two projectors (with a single camera). For example, two projectors may project alternating or otherwise distinguishable patterns (eg, simultaneous but different colors, patterns, polarities, etc.) into the field of view of the camera. By noting how the two patterns-projected from different points-are provided on the object and seen by the camera, the stereoscopic information can be identified again.

Many usage models are made possible through the use of projectors, including new shared models (see "View & Share: Exploring Co-Present Viewing and Sharing of Pictures using Personal Projection" by Greaves in Mobile Interaction with the Real World in 2009). These models utilize self-generated images by the projector as triggers to initiate a sharing session, apparently via a commonly understood symbol (“open” sign), to hide machine readable triggers. Sharing can also occur over ad hoc networks utilizing peer to peer applications or server hosted applications.

Other outputs from mobile devices can be similarly shared. Consider keyvectors. One user's phone can process the image with Hough transform and other eigenface extraction techniques, and then share the keyvectors resulting from the eigenface data with others in the user's social cycle (the same By putting something or allowing them to pool it). One or more of these social-subscribed devices may then perform face template matching that yields an identification of a previously unrecognized face in the image captured by the original user. This arrangement takes the personal experience and makes it a shared experience. Moreover, the experience can be a viral experience with keyvector data shared with the majority of others-essentially without boundaries.

Other selected Arrangements

In addition to the above-described arrangements earlier, other hardware arrangements suitable for use with certain implementations of the present technology utilize the Mali-400 ARM graphics multiprocessor architecture, which is dedicated to the different types of image processing tasks referenced in this document. A plurality of fragment processors may be included.

Standard group Khronos has published OpenGL ES2.0, which defines hundreds of standardized graphics function calls for systems that include multiple CPUs and multiple GPUs (the direction in which cell phones are migrating). . OpenGL ES2.0 seeks to route different operations to different processing units-such details are transparent to the application software. Thus, it provides a software API that is consistent with all manner of GPU / CPU hardware available.

According to another aspect of the present technology, the OpenGL ES2 standard extends not only to different CPU / GPU hardware but also to provide a standardized graphics processing library across different cloud processing hardware-again, such details are transparent to the calling application. .

Increasingly, Java service requests (JSRs) have been defined to standardize certain Java-implemented tasks. JSRs are increasingly designed for efficient implementations on top of OpenGL ES2.0 grade hardware.

According to another aspect of the present technology, some or all of the image processing operations (face recognition, SIFT processing, watermark detection, histogram processing, etc.) well known in this specification may be implemented as JSRs-other kinds of hardware platforms Provides standardized implementations that are suitable for

In addition to supporting cloud-based JSRs, the extended standard specification can also support the query router and response manager functionality described earlier-both including static and auction-based service providers.

OpenGL is similar to OpenCV-a computer vision library available under an open source license allows coders to call various functions-regardless of the specific hardware utilized to execute the same. (O'Reilly Books, Learning OpenCV , Documents in a wide range of languages.) The counterpart NokiaCV provides similar functionality standardized for the Symbian operating system (eg Nokia cell phones).

OpenCV provides support for a wide range of operations, including high-level tasks such as face recognition, gesture recognition, motion tracking / understanding, segmentation, etc., as well as a large collection of more atomic and elementary vision / image processing operations. do.

CMVision is a computer vision tool of another package that can be utilized in certain embodiments of the present technology, which was compiled by researchers at Carnegie Mello University.

Another hardware architecture uses a field programmable object array (FPOA) arrangement, where hundreds of different types of 16-bit "objects" are arranged in a grid node fashion, each of which is a neighboring device over very high bandwidth channels. You can exchange data with them. (PicoChip devices referenced earlier are of this class.) Each function is programmable like FPGAs. Again, the difference in image processing operations can be implemented by the difference in FPOA objects. These tasks can be executed as needed (eg, an object can execute SIFT processing in one state; FFT processing in another state; log-polarity processing in another state; etc.) during operation. Can be redefined.

(Although many grid arrangements of logic devices are based on a "closest neighbor" interconnect, additional flexibility can be achieved by using a "partial crossbar" interconnect. For example, Patent 5,448,496 (Quickturn). Design Systems).

Also in the area of hardware, certain embodiments of the present technology utilize "extended depth of field" imaging systems (see, for example, patents 7,218,448, 7,031,054 and 5,748,371). Such arrangements may include a mask in the imaging path that modifies the light transfer function of the system to be insensitive to the distance between the object and the imaging system. Image quality is poorly uniform over the depth of field. Digital post processing of the image compensates for mask modifications, recovering image quality but maintaining increased field depth. Using this technique, the cell phone camera captures the image by focusing all objects closer and farther (ie, higher frequency detail)-no longer needing exposure-as generally required. (Longer exposures exacerbate problems such as hand-jitter and object movements.) In the arrangements described herein, shorter exposures can withstand the temporary delay created by optical / mechanical focusing elements, or images Allowing a higher quality image to be provided to the image processing functions without requiring input from the user as to which elements of the element should be focused. This provides a much more intuitive experience when the user can simply point the imaging device to the desired target without worrying about focus or field depth settings. Similarly, the image processing functions are all expected to be in the same focus, thus leveraging all the pixels contained in the captured image / frame. In addition, new metadata about identified objects or groups of pixels related to depth within a frame simply generates “depth map” information to capture and display 3D video using emerging standards for the transmission of depth information. A stage can be set up for the storage of video streams.

In some implementations, the cell phone can have the ability to perform a given action locally, but can decide to have it executed by the cloud resource instead. The decision of whether to handle locally or remotely includes "costs" including bandwidth costs, external service provider costs, power costs for cell phone batteries, intangible costs of consumer (unsatisfied) by delaying processing, and the like. On the " For example, if the user is running on low battery power and is far from the cell tower (so the cell phone runs the RF amplifier at full output at the time of transmission), sending a large block of data for remote processing is the rest of the battery. It can consume a significant part of its life. In this case, the phone may decide to process the data locally, or may decide to send it for remote processing when the phone is close to the cell site or the battery is recharged. The set of stored rules can be applied to relevant variables to establish a pure "cost function" for different ways (eg locally processed, remotely processed, postponed), and these rules Depending on the states of these variables may represent different results.

An attractive "cloud" resource is the processing power found at the edges of wireless networks. For example, cellular networks utilize processors to perform some or all of the operations typically performed by digitally transmitting and receiving wireless circuits, such as mixers, filters, demodulators, etc., digitally, to a large portion of the network. Tower stations, which are software-defined radios. Smaller cell stations, so-called "femtocells", typically have powerful signal processing hardware for such processes. Initially known PicoChip processors and other field programmable object arrays are widely deployed in these applications.

Wireless signal processing and image signal processing have many commonalities, for example, utilizing FFT processing to convert sampled data into the frequency domain, applying various filtering operations, and the like. Cell station equipment including processors are designed to meet peak consumer needs. This means that significant processing power often remains unused.

According to other aspects of the present technology, this preliminary wireless signal processing capability (and other edges of wireless networks) of a cellular tower station is repurposed along with image (and / or audio or other) signal processing for consumer wireless devices. do. Since the FFT operation is the same-whether to process sampled radio signals or image pixels-repurpose is often easy; Often configuration data for hardware processing cores need not be changed as much as needed. And because 3G / 4G networks are so fast, processing can be quickly dispatched from the consumer device to the cell station processor, and results are returned at similar rates. In addition to the speed and computational power that such repurpose of cell station processors provides, other advantages are to reduce the power consumption of consumer devices.

Before sending image data for processing, the cell phone can quickly inquire with the cell tower station it is communicating with to confirm that it has sufficient unused capacity to undertake the intended image processing operation. Such a query may include the packager / router of FIG. 10; The local / remote router of FIG. 10A, the query router and response manager of FIG. 16 may be transmitted by the pipe manager 51 of FIG.

Alerting the cell tower / base station of upcoming processing requests and / or bandwidth requirements allows the cell site to better allocate its processing and bandwidth resources in anticipation of meeting those needs.

Cell sites are at risk of becoming bottlenecks in undertaking service operations that consume their processing or bandwidth capacity. When this occurs, they can only be degraded by unexpectedly retightening the processing / bandwidth provided to one or more users, so that others can be served. This abrupt service change is undesirable because the channel changes the originally established parameters (eg, the bit rate at which video can be delivered) causing the data services using the channel to reconfigure their respective parameters. (Eg, require ESPN to provide a low quality video feed). By renegotiating these details, if channels and services were originally set up, they always introduce minor defects, such as video delivery stuttering, interrupted troubles in phone calls, and the like.

To avoid these unexpected bandwidth slowdowns and the need for consequent service impairments, cell sites tend to adopt a conservative strategy-stray allocation of bandwidth / processing resources-to conserve capacity for possible peak demands. have. However, this approach deteriorates the quality of service normally provided-at the expense of ordinary service at unexpected expectations.

According to this aspect of the present technology, the cell phone sends alerts to the cell tower station that anticipated bandwidth or processing needs are approaching. In fact, the cell phone asks to preserve some future service capacity. The tower station also has a fixed capacity. However, if one knows what a user needs, for example a bandwidth of 8 Mbit / s for 3 seconds starting at 200 milliseconds, he or she will allow the cell site to consider these expected needs when serving other users.

Consider a cell site with an excess (allocated) channel capacity of 15 Mbit / s that normally allocates 10 Mbit / s channels to new video service users. If the site knows that a cell camera user has requested a reservation for an 8 Mbit / s channel starting at 200 milliseconds, and on the other hand, a new video service user requests a service, the site will receive a 10 Mbit / s typical for a new video service user. Rather, a 7 Mbit / s channel can be allocated. By initially setting the channel of the new video service user to a slower bit rate, service impairments associated with cutting back bandwidth during an ongoing channel session are avoided. The capacity of the cell site is the same, but is now allocated in a way that reduces the need to reduce the bandwidth of existing channels, mid-transmission.

In other situations, the cell site may determine that it has exceeded its current capacity, but expects to be more heavily burdened in 1/2 second. In this case, the current excess capacity can be used to increase throughput to one or more video subscribers, for example, those who have collected several packets of video data ready for delivery of the buffer memory. These video packets can now be transmitted over the extended channel in anticipation of slowing the video channel by 1/2 second. Again, this is practical because the cell site has useful information about future bandwidth requirements.

The service reservation message sent from the cell phone may also include a priority indicator. This indicator can be used by the cell site to determine the relative importance of satisfying the request for stated perspectives when mediation between conflicting service requests is required.

Expected service requests from these cell phones may also allow the cell site to provide a higher quality consistent service than normally assigned.

It is understood that cell sites utilize statistical models of usage patterns and thus allocate bandwidth. Allocations are typically set conservatively in anticipation of a realistic worst case of usage scenarios, including, for example, scenarios that occur 99.99% of the time. (Some theoretically possible scenarios are unlikely to be negligible in bandwidth allocations. However, in rare cases where such unlikely scenarios occur—thousands of subscribers from cell phone images from Washington, DC, during the Obama Inauguration. As with sending messages, some subscribers may simply not receive a service.)

Statistical models based on site bandwidth allocations are understood to treat subscribers-in part-as unexpected actors. Whether a particular subscriber requests service in the coming seconds (and what specific service is requested) has a random aspect.

The greater the randomness in the statistical model, the greater the tendency to be extreme. If predictions of reservations or future requests are routinely presented, for example with 15% of the subscribers, the behavior of these subscribers is no longer random. In the worst case, the peak bandwidth demand on the cell site involves only 85%, not 100% of randomly operating subscribers. Actual booking information may be utilized for 15% of the others. Thus, virtual limits of peak bandwidth utilization are adequate.

Using lower peak usage scenarios, more general allocations of current bandwidth can be granted to all subscribers. In other words, if some of the users send alerts to a site that conserves future capacity, the site can predict that the actual peak demand that is coming soon is still a site that is in unused capacity. In such a case, the camera cell phone user may be allowed to grant 12 Mbit / s channels-in place of the 8 Mbit / s channels stated in the reservation request, and / or usually 15 Mbit / s channels instead of the 10 Mbit / s channels. You can authorize the user. Thus, this usage prediction may allow the site to grant higher quality services than usual, since the bandwidth preserves the need to be maintained for fewer unexpected actors.

Expected service requests can also be communicated to other cloud processes that are expected to be related to the requested services from the cell phone (or cell site), allowing them to similarly allocate their resources as expected. These anticipated service requests may also serve to change cloud processing to pre-warming associated processing. The additional information configures the FPOA to serve as encryption keys, image dimensions (e.g., FFT processors for 1024 x 768 images to be processed in 16 x 16 tiles, and output coefficients for 32 spectral frequency bands. May be provided from the cell phone or elsewhere for this purpose.

The cloud resource can now warn the cell phone of any information that it expects to be requested from the phone upon execution of the expected action or an action that the cell phone can request to execute so that the cell phone has its own upcoming action. You can expect them and prepare accordingly. For example, cloud processing may evaluate, under certain conditions, whether the originally provided data is not sufficient for its intended use (e.g., the input data may not have sufficient focus resolution or sufficient contrast or an image requiring additional filtering). May request another set of input data. In advance, knowing that cloud processing may request such additional data, it may allow a cell phone to take this possibility into account in its own operation, for example, in a particular filter manner, unless otherwise. Maintaining the configured processing modules, preserving intervals of sensor time to capture alternate images as possible.

Expected service requests (or likelihood of conditional service requests) generally relate to events that can initiate in tens or hundreds of milliseconds, sometimes in seconds. Situations where the action starts in tens or hundreds of seconds in the future will be sparse. However, although the warning period in advance can be short, significant advantages can be derived: if the randomness of the next second is reduced-each second, the system randomness can be significantly reduced. Moreover, the events to which requests are related can be themselves in a longer duration-such as sending a large image file that can take more than 10 seconds.

Regarding pre-setup (pre-warming), preferably, any action that can take more than a threshold time interval to complete (eg, hundreds of milliseconds, milliseconds, 10 microseconds, etc.)-Implementation If possible, be prepared as expected. (In some instances, of course, the expected service is never requested, in which case such preparation may be worthless.)

In other hardware arrangements, the cell phone processor can selectively activate a Peltier device or other thermoelectric cooler coupled to the image sensor in environments where thermal image noise (Johnson noise) is a potential problem. For example, if the cell phone detects low light conditions, it can activate the cooler on the sensor to try and improve the image signal-to-noise ratio. Or image processing stages may examine the captured image for artifacts associated with thermal noise, and if such artifacts exceed a threshold, the cooling device may be activated. (One method captures a patch of an image at twice the rapid succession, such as a 16 x 16 pixel area. In the absence of random factors, the two patches should be identical-preferably correlated. Of correlation from 1.0 Variation is a measure of noise-perhaps thermal noise.) Alternate images can be captured at short time intervals after the cooling device is activated-intervals dependent on the thermal response time to the cooler / sensor. Similarly, once cell phone video is captured, the cooler can be activated because the increased switching activity by the circuitry for the sensor increases the temperature and thus its thermal noise. (Whether to activate the cooler may also be application dependent; for example, the cooler may be activated when capturing an image from which watermark data is read, but active when capturing an image from which barcode data may be read. Not.)

As noted, the packets of the FIG. 16 arrangement may carry various instructions and data—both header and packet body—in both. In other arrangements, the packet may additionally or alternatively include a pointer to a record or cloud object in the database. The cloud object / database record may include information such as object attributes useful for object recognition (eg, fingerprint or watermark attributes for a particular object).

Once the system has read the watermark, the packet may include a watermark payload and the header (or body) may include one or more database references whose payload may be associated with related information. The watermark payload read from the business card can be looked up in one database; The watermark decoded from the picture can be looked up in another database. The system can apply multiple different watermark decoding algorithms to a single image (eg, MediaSec, Digimarc ImageBridge, Civolution, etc.). Depending on which application has a specific decoding operation performed, the resulting watermark payload can be sent to the corresponding destination database. (The same is true for different barcodes, fingerprint algorithms, eigenface techniques, etc.) The destination database address may be included in an application or configuration database. (In general, addressing can be done indirectly with intermediate data storage including the address of the final database, allowing relocation of the database without changing each cell phone application.)

The system performs an FFT on the captured image data to obtain frequency domain information and then supplies that information to several watermark decoders operating in parallel, each applying a different decoding algorithm. When one of the applications extracts valid watermark data (e.g., indicated by ECC information calculated from the payload), the data is sent to a database corresponding to the format / description of the watermark. A plurality of such database pointers may be included in the packet and may be used conditionally-depending on the watermark decoding operation (or barcode reading operation, or fingerprint calculation, etc.) yielding useful data.

Similarly, the system may send a face image for the intermediate cloud service in a packet that includes the user's identifier (but not including the user's Apple iPhoto, or Picasa, or Facebook username). The intermediate cloud service can take the provided user identifier and use it to access a database record from which user names on these other services can be obtained. The intermediary cloud service then sends the face image data to Apple's server-with the user's iPhoto username; Picasa's service to your Google username; And route the Facebook username of the user to a Facebook server. Each of these services may then perform face recognition on the image and return the names of people identified from the user's iPhoto / Picasa / Facebook accounts (either directly or through an intermediate service to the user). Intermediate cloud services, which can serve the majority of users, do not cause each cell phone to attempt to maintain this data in an updated manner, rather than related servers (and alternative nearby servers if the user is away from home). Notification of the current addresses).

Facial recognition applications can be used to identify people as well as to identify relationships between individuals depicted in an image. For example, data maintained by iPhoto / Picasa / Facebook may refer to terms representing facial recognition features and associated names, as well as relationships between names' faces and account holders (eg, father, male Friends, siblings, pets, roommates, etc.). Thus, for example, instead of simply searching the user's image collection for all the photos of "David Smith", the user's collection may also be searched for all the pictures depicting "Brother and Sister".

The application software in which the pictures are reviewed provides differently colored frames around differently recognized faces-depending on the associated relationship data (eg blue for siblings, red for boyfriends, etc.).

In some arrangements, the user's system can access this information stored in accounts maintained by the user's network "friends." Faces that cannot be recognized by the facial recognition data associated with the user's account in Picasa may be recognized by referring to the Picasa facial recognition data associated with the account of the user's friend "David Smith". The relationship data indicated by the David Smith "account can be similarly used to provide and organize the user's photos. An initially unrecognized face is therefore labeled with an indicator indicating that the person is David Smith's roommate. This essentially remaps the relationship information (eg, mapping "roommate" from user's account to "David Smith's roommate" as indicated in David Smith's account).

The above-described embodiments are generally described in the context of a single network. However, a plurality of networks may generally be available to the user's phone (eg WiFi, Bluetooth, possible different cellular networks, etc.). The user can choose between these children or the system can automatically apply stored rules to do so. In some examples, a service request may be issued (or results returned) in parallel across several networks.

Reference platform architecture

The hardware of cell phones was originally introduced for special purposes. For example, microphones were only used for voice transmissions over cellular networks; A / D converters were supplied to provide modulators in the phone's wireless transceiver. The camera was used only to capture snapshots. Etc.

Additional applications resulted from utilizing this hardware, and each application had to be developed to talk to the hardware in its own way. Different kinds of software stacks have arisen-each specialized for a particular application could interact with a particular piece of hardware. This takes the implementation to application development.

 This problem is exacerbated when cloud services and / or specialized processors are added to the mix.

To alleviate these difficulties, some embodiments of the present technology may utilize an intermediate software layer that provides a standard interface with and through which hardware and software can interact. This arrangement is shown in FIG. 20A where the intermediate software layer is labeled “Reference Platform”.

In this figure, the hardware elements are shown in dotted boxes containing processing hardware on the bottom and peripherals on the left. The box "IC HW" is "intuitive computing hardware", and the hardware discussed earlier that supports different processing of image related data, such as modules 38 of FIG. 16, configurable hardware of FIG. 6, and the like. It includes. The DSP is a general purpose digital signal processor, which can be configured to perform specialized operations; CPU is the phone's main processor; GPU is a graphics processing unit. OpenCL and OpenGL are APIs in which graphics processing services (running on the CPU and / or GPU) can be called.

Different specialized techniques are intermediate, such as one or more digital watermark decoders (and / or encoders), barcode reading software, optical character recognition software, and the like. Cloud services are shown on the right and applications are shown on top.

The reference platform establishes a standard interface (eg, by API calls) where different applications interact with hardware, exchange information, and request services. Similarly, the platform establishes a standard interface through which different technologies can be accessed and in which they can send and receive data to other of the system components. Like the cloud services, the reference platform can also process the details that identify the service provider-by reverse auctions, heuristics and the like. In cases where a service is available both from the technology of the cell phone and from a remote service provider, the reference platform may also execute weighting the costs and benefits of different options and determining which should handle a particular service request. have.

By this arrangement, different system components do not have to relate to the details of other parts of the system themselves. The application can request that the system read the text from the object in front of the cell phone. There is no need to correlate with the specific control parameters of the image sensor or the image format requirements of the OCR engine. The application may request the reading of a person's emotions in front of the cell phone. The corresponding call passes whatever the phone's technology supports this function and the results are returned in a standardized form. When the improved technology becomes available, it can be added to the phone, and through the reference platform the system takes advantage of its improved capabilities. Thus, growing / changing collections of sensors and growing / developing sets of service providers are tasked to derive meaning from input stimuli (visual as well as audio, eg speech recognition) through the use of this adaptive architecture. Can be set.

Arasan Chip Systems, Inc. provides a stacked kernel-level stack to the mobile industrial processor interface Unipro software stack to enable integration of specific technologies into cell phones. The arrangement can be extended to provide the functionality described above. (The Arasan protocol focuses primarily on transport layer problems, but likewise involves layers under hardware drivers. The mobile industry processor interface alliance is a large industry organization (works on advanced cell phone technologies).

For example, an existing image for metadata Of collections Leveraging

Collections of publicly available images and other content are more generally done. Flickr, YouTube, Photobucket (MySpace), Picasa, Zooomr, Facebook, web shots and Google images are just a few of them. Often, these resources can serve as sources of metadata-such obviously inferred or deduced from a file such as file names, descriptions, etc. Sometimes geographic-location data is also available.

Exemplary embodiments according to one aspect of the present technology operate as follows. Cell Phone Picture Captures of an Object or Scene—A Desk Phone, as shown in FIG. 21. (Images can also be obtained in other ways as well, such as sent from another user or downloaded from a remote computer.)

As a preliminary operation, known image processing operations can be applied, for example, to the captured image, to correct color or contrast, to perform ortho-normalization, and so on. Known image object segmentation or classification techniques may also be used to identify the apparent target area of the image and separate it for other processing.

Image data is then processed to determine the characterized features useful for pattern matching and recognition. Color, shape and texture metrics are commonly used for this purpose. Images can also be grouped based on layout and eigenvectors (the latter being particularly popular for face recognition). As is well known elsewhere in this specification, it is obvious that many other techniques may be used.

(In faces, images, video, audio and other patterns, the use of vector characterizations / classifications and other image / video / audio metrics are well known and suitable for use with certain embodiments of the present technology. See, for example, patent publications 20060020630 and 20040243567 (Digimarc), 20070239756 and 20020037083 (Microsoft), 20070237364 (Fuji Photo Film), 7,359,889 and 6,990,453 (Shazam), 20050180635 (Corel), 6,430,306, 6,681,032 and 20030059124 (L-1) Corp.), 7,194,752 and 7,174,293 (Iceberg), 7,130,466 (Cobion), 6,553,136 (Hewlett-Packard), and 6,430,307 (Matsushita), and the academic references cited at the end of this disclosure. When used with the recognition of the same entertainment content, these features are sometimes referred to as content "fingerprinters" or "hashes."

After the feature metrics for the image have been determined, a search can be made through one or more publicly accessible image stores for images with similar metrics, thereby making it possible to clearly identify similar images. (As part of image collection processing, Flickr and other such repositories are eigenvectors, color histograms, keypoint descriptors, FFTs, or images when they are uploaded by users and collect the same index for public search. Other classification data can be calculated.) The search can result in a collection of apparently similar telephone images found in the flicker shown in FIG.

Metadata is then obtained from Flickr for each of these images, and technical terms are analyzed and ranked by frequency of occurrence. In the set of depicted images, for example, descriptors obtained from such an action and their frequency of occurrence may be as follows:

Cisco (18)

Phone (10)

Phone (7)

VOIP (7)

IP (5)

7941 (3)

Pawns (3)

Phone (3)

7960 (2)

7920 (1)

7950 (1)

Best Buy (1)

Desk (1)

Ethernet (1)

IP-Phone (1)

Office (1)

Pricey (1)

Sprint (1)

Telecommunications (1)

Uninet (1)

Action (1)

From this condensed set of inferred metadata, one can assume that terms with the highest count values (eg, the least frequently occurring terms) are the terms that most accurately characterize the user's FIG. 21 image. .

Inferred metadata can be augmented or enhanced if desired by known image recognition / rating techniques. This technique seeks to provide automatic recognition of the objects depicted in the image. For example, by recognizing the touchtone keypad layout and coil code, this classifier can label the FIG. 21 image using the terms telephone and facsimile machine terms.

If not yet present in the inferred metadata, the terms returned by the image classifier may be added to the list or given a count value. (A random number, for example 2, may be used, or a value that depends on the reported confidence of the classifier in the distinguished identification may be utilized.)

If the classifier results in one or more terms already present, the position of the term (s) in the list may be elevated. One way to raise the position of the term is to increase the count value by a percentage (eg 30%). Another method is to increase the count value by one greater than the next-parent term not distinguished by the image classifier. (Since the classifier returned the term "telephone" but did not return the term "cisco", this latter approach could rank the term telephone with a count value of "19"-1 above Cisco.) Inferred metadata Is derived from the image classifier and various other techniques for augmentation / enhancement are easy to implement.

The revised list of metadata resulting from the above may be as follows:

Phone (19)

Cisco (18)

Phone (10)

VOIP (7)

IP (5)

7941 (3)

Pawns (3)

Technology (3)

7960 (2)

Fax machines (2)

7920 (1)

7950 (1)

Best Buy (1)

Desk (1)

Ethernet (1)

IP-Phone (1)

Office (1)

Expensive (1)

Sprint (1)

Telecommunications (1)

Uninet (1)

Action (1)

The list of inferred metadata may be limited to terms with the highest clear confidence, eg, count values. For example, a subset of the list may be used that includes the top N terms, or the terms of the top Mth percentile of the ranking list. This subset can be associated with the FIG. 21 image in the metadata repository for that image as inferred metadata.

In this example, if N = 4, the terms, phone, Cisco, phone and VOIP are associated with the FIG. 21 image.

Once the list of metadata is assembled for the FIG. 21 image (by the procedures or otherwise described above), various actions can be undertaken.

TinEye, iVisit's SeeScan, Evolution Robotics' ViPR, IQ Engine's oMoby, 및 Digimarc Mobile이 몇몇의 여러 상업적으로 이용 가능한 서비스들이며, 이들은 미디어 콘텐트를 캡처하고 대응 응답을 제공한다; 다른 것들은 초기 인용된 특허 공개들에 상술되어 있다. 메타데이터와 콘텐트 데이터를 수반함으로써, 서비스 제공자는 이용자의 제시에 어떻게 응답되어야 하는지에 관한 더욱 충분한 판단을 할 수 있다. One option is to capture the captured content or images derived from the captured content (e.g., images such as Figure 21 images, eigenvectors, color histograms, keypoint descriptors, FFTs, machine readable data decoded from images, etc.). Feature) present metadata to a service provider that operates on the presented data and provides responses to the user. Shazam, Snapnow (now LinkMe Mobile), ClusterMedia Labs, Snaptell (now part of Amazon's A9 search service), Mobot, Mobile Acuity, Nokia Point & Find, Kooaba,

TinEye, iVisit's SeeScan, Evolution Robotics' ViPR, IQ Engine's oMoby, and Digimarc Mobile are some of several commercially available services that capture media content and provide corresponding responses; Others are detailed in the earlier cited patent publications. By involving metadata and content data, the service provider can make more informed decisions about how to respond to the user's presentation.

The service provider-or the user's device-can obtain one or more other services, such as Google, for example, to obtain a richer set of assistance information that can help better distinguish / infer / intuitive what is desired by the user. You can present metadata descriptors to a web search engine. Or information obtained from Google (or other such database resource) may be used to augment / improve the response delivered to the user by the service provider. (In some cases, metadata-possibly accompanied by auxiliary information received from Google-can allow the service provider to make an appropriate response to the user without requiring image data.)

In some cases, one or more images obtained from Flickr are replaced for the user's image. This can be done, for example, if the flicker image is of higher quality (using sharpness, illumination histogram or other measurements), and if the image metrics are quite similar. (Similarity can be determined by distance measurement appropriate to the metrics being used. One embodiment checks whether the distance measurement is lower than a threshold. When several alternating images pass through this screen, the closest image is obtained. Or alternatives may be used in other circumstances. The replaced image may then be used instead of (or in addition to) the image captured in the arrangements detailed herein.

In one such arrangement, the replacement image data is presented to the service provider. In addition, data for several alternative images are presented. In addition, original image data, along with one or more alternative sets of image data, is presented. In the latter two cases, the service provider may use redundancy to help reduce the chance of error-assuming that an appropriate response is provided to the user. (Or the service provider can handle each presented set of image data individually and provide a plurality of responses to the user. The client software on the cell phone then evaluates the different responses and picks between them ( For example, by bowing arrangement), or by combining user responses.)

Instead of substitution, one director's associated public image (s) may be composited or merged with the user's cell phone image. The resulting hybrid image can then be used in the different contexts detailed in this disclosure.

Another option is to use clearly similar images collected from Flickr to notify the user of an improvement in the image. Examples include color correction / matching, contrast correction, glare reduction, foreground / background object removal, and the like. By such an arrangement, for example, such a system can distinguish that the FIG. 21 image has foreground components (clearly post-it notations) on the phone that should be masked or ignored. The image data of the user can thus be improved, and the enhanced image data is then used.

 In this regard, the image of the user may have to suffer from certain obstacles, for example, depicting the object from fractional perspective, poor lighting, or the like. This disorder may cause the user's image not to be recognized by the service provider (ie, the image data presented by the user is unlikely to match any image data in the database being retrieved). In response to this failure, or in advance, data from similar images identified from Flickr can be presented to the service provider as alternatives-hope they work better.

Another way-leaving many other possibilities open-is to retrieve one or more images with similar image metrics from Flickr and collect the metadata described herein (eg, phone, Cisco, phone, VOIP). Flicker is then, on the basis of meta data is searched twice. Multiple images with similar metadata can thereby be identified. Thereafter, data for these other images (including images of various different perspectives, different lighting, etc.) may be presented to the service provider-although they may be "looked" differently from the user's cell phone image. and.

When doing metadata-based searches, the identity of the metadata may not be needed. For example, in a second search of Flickr just referenced, metadata of four terms may be associated with the user's image: phone, Cisco, phone and VOIP. Matching can be considered an example where a subset of these terms (eg 3) is found.

Another way is to rank the matches based on the rankings of shared metadata terms. Thus, images tagged with phone and Cisco are ranked with better matching than images tagged with phone and VOIP. One adaptive way of ranking "matching" sums the counts for metadata descriptors for the user's image (eg, 19 + 18 + 10 + 7 = 54), and then shared in the flicker image It is to record the count values for the terms (eg 35, when the flicker image is tagged with Cisco, Phone and VOIP). The ratio can then be calculated (35/54) and compared to a threshold (eg 60%). In this case, "matching" is found. Various other adaptive matching techniques can be devised by the technician.

The above examples retrieved images from Flickr based on the similarity of the image metrics, and optionally the similarity of the (meaning) metadata of the text. Geographic location data (eg, GPS tags) may also be used to obtain metadata toe-hold.

If the user captures an artistic abstract shot of the Eiffel Tower (eg FIG. 29) from the center of a metalwork or other rarely advantageous point, it cannot be recognized from the image metrics-the Eiffel Tower. However, the GPS information captured with the image identifies the location of the image object. Public databases (including Flickr) can be utilized to retrieve metadata of text based on GPS descriptors. By inputting GPS descriptors for the picture, the descriptors of text Paris and Eiffel are generated.

Google images or other databases may be queried with the terms Eiffel and Paris to search for other, more likely conventional images of the Eiffel Tower. One or more of these images may be presented to a service provider to drive the processing. (Alternatively, GPS information from a user image retrieves images from Flickr from the same location; creates an image of the Eiffel Tower that can be presented to the service provider.)

Although GPS is obtained from camera-metadata-deployment, most of the current images in Flickr and other public databases are missing geographic location information. However, GPS information can be a collection of images that share visible features (by image metrics such as eigenvectors, color histograms, keypoint descriptors, FFTs, or other classification techniques), or have metadata matching. Can be propagated automatically.

To illustrate, if a user takes a cell phone image of an urban fountain and the image is tagged with GPS information, a process may be presented that identifies matching the flicker / Google images of that fountain based on feature-recognition. For each of these images, the process may add GPS information from the user's image.

A second level of search may also be utilized. From the set of fountain images identified from the first search based on the similarity of appearance, metadata may be obtained and ranked as above. Flickr may then retrieve the images twice with matching metadata within a certain threshold (eg, as discussed above). For these images, GPS information can also be added from the user's image.

Alternatively, or in addition, a first set of images in flicker / Google similar to images of the user of the fountain can be identified-by pattern matching as well as by GPS (or both). Metadata can be obtained and ranked from these GPS-matched images. Flickr can be retrieved twice for a second set of images with similar metadata. For this second set of images, GPS information can be added from the user's image.

Another way for a geographic location image is to search the flicker for an image with similar image characteristics (eg gist, eigenvectors, color histograms, keypoint descriptors FFTs, etc.) Evaluate geographic location data on the identified images to infer location. For example, 2008 Proc. of the IEEE Conf. See IM2GPS: Estimating geographic information from a single image by Hays et al. on Computer Vision and Pattern Recognition. The techniques detailed in the Hays article are suitable for use with certain embodiments of the present technology (including the use of probability functions to quantify the uncertainty of inferential techniques).

When geographic location data is captured by the camera, it is very reliable. In addition, metadata (such as location) authored by the owner of the image is generally reliable. However, uncertainty and other problems arise when metadata descriptors (geographic location or meaning) are inferred or estimated or when an image is authored by a stranger.

Preferably, this inherent uncertainty should be remembered in some way so that later users (human or machine) can take this uncertainty into account.

One way is to separate the uncertain metadata from device-authored or producer-written metadata. For example, different data structures can be used. Or different tags may be used to distinguish between classes of such information. Or each metadata descriptor may have its own sub-metadata indicating the author, date of creation, and source of data. The author or source field of the sub-metadata may have a data string indicating that the descriptor is inferred, estimated, deduced, or the like, or such information may be a separate sub-metadata tag.

Each uncertain descriptor may be provided with a confidence metric or rank. Such data can be clearly or speculatively determined by the public. An example assumes that when a user views an image on Flickr, she believes that she is in Yellowstone and adds a "yellowstone" location tag with a 95% trust tag (her estimate of certainty about the contributed location metadata). This is the case. She can add an alternate location meta tag that represents "Montana" with a corresponding 50% trust tag. (Trust tags do not need to be a sum of 100%. Only one tag can be contributed-with less than 100% trust or multiple tags can be contributed-as in the case of Yellowstone and Montena. It may be wrapped.)

If multiple users contribute to the same type of metadata for an image (eg, location metadata), the combined contributions can be evaluated to generate collective data. This information, for example, 5 of 6 users who contributed to the metadata tagged yellowstone in the image with an average of 93% confidence; 1 of 6 users who contributed to the metadata tagged Montana to the image with an average of 50% confidence, 2 of 6 users tagged to Glacier National Park with an average of 15% confidence, and so on. have.

When explicit estimates made by contributors are not available, or routinely, inferential determination of metadata reliability may be performed. This example is the case of the FIG. 21 photograph, where the metadata occurrence counts are used to determine the relevant merit of each item of metadata (eg, phone = 19 or 7, depending on the mathematics used). Similar methods can be used to rank the reliability when several metadata contributors provide descriptors for a given image.

Crowd-sourcing techniques are known for distributing image-identifying tasks to online workers to collect the results. However, prior art arrangements are understood in order to seek a simple short term agreement on identification. Even better is to quantize different kinds of opinions collected about image contents (and optionally, information about variations over time and dependent sources), and about images, their values, their relevance, their use, etc. It seems to use richer data to enable automated systems to make more subtle differences.

To illustrate, known crowd-sourcing image identification techniques may identify the FIG. 35 image with identifiers “football ball” and “dog”. These are terms agreed from one or several viewers. However, information regarding the long tail of alternative descriptors such as summer, labrador, football, tongue, lunch, dinner, morning, fescue, etc. may be ignored. In addition, demographics and other information about metadata identifiers, people (or processes) serving as environments of their assessments may be ignored. A richer set of metadata can associate each descriptor with a set of sub-metadata that details this other information.

The sub-metadata may indicate, for example, that the tag “Football” was contributed by a 21 year old male in Brazil on June 18, 2008. In addition, the tags “lunch”, “evening” and “morning” are automated images at the University of Texas, where these panstages were made on July 2, 2008, for example based on the angle of illumination for the objects. It may indicate that it has been contributed by the classifier. These three descriptors may also be stored in the possibilities assigned by the classifier, for example 50% for afternoon, 30% for evening, and 20% for morning (each of these percentages being stored as sub-metatags). May be associated). One or more metadata terms contributed by the classifier may have another sub-tag that points to an on-line glossary that helps to understand the assigned terms. For example, such a sub-tag can provide the URL of a computer resource that associates the term "afternoon" with a regulation or synonym indicating that the term means from noon to 7pm. The glossary also has a probability density function indicating that the mean time implied by "afternoon" is 3:30 pm, the median time is 4:15 pm, and the term has a Gaussian function of meaning over time intervals from noon to 7pm. Can be represented.

The expertise of metadata contributors can also be reflected in the sub-metadata. The term “fescue” may have sub-metadata indicating that it was contributed by a 45 year old plant seed farmer in Oregon. An automated system can conclude that this metadata term has been contributed by someone with rare expertise in the relevant knowledge domain, and thus can treat the descriptor as highly reliable (although not highly related). This confidence can be added to the metadata collection, so that other reviewers of the metadata can benefit from the evaluation of the automated system.

An assessment of the contributor's expertise may also be made by the contributor itself. Alternatively, it may be made by reputation rankings using collected third party assessments of the contributor's metadata contributions. (These reputation rankings are known, for example, from public ratings of sellers on eBay and Amazon's book reviewers.) The ratings can be field-specific, so that a person has a lot of knowledge about plant types, With regard to these, it may be judged that there is not much knowledge (or self-judgment). Again, all such information is preferably stored in sub-metatags (including sub-sub-metatags when the information relates to the sub-metatags).

More information about crowdsourcing, including the use of contributor expertise and the like, is found in Digimarc's published patent application 20070162761.

Returning to the case of geographic location descriptors (which may be associated with numbers, for example longitude / latitude or text), the image may accumulate a long catalog of contributing geographic descriptors-over time. An automated system (eg, a server at Flickr) can periodically review the contributed geographic tag information and extract it to facilitate public use. For information related to the number, the process may apply known clustering algorithms to identify clusters of similar coordinates, and average them to generate an average position for each cluster. For example, a hot spring photo can be tagged by someone with latitude / longitude coordinates in Yellowstone, and by others by latitude / longitude coordinates at Healthgate Park in New Zealand. Thus, these coordinates form two separate clusters that can be averaged separately. If 70% of the contributors are placed in the coordinates in Yellowstone, the extracted (averaged) value can be given 70% confidence. External data may be maintained, but a low probability is assumed to correspond to its external state. This extraction of data by the owner may be stored in metadata fields that are publicly readable but not writable.

The same or another way can be used with metadata of the added text-for example, to provide a sense of related trust, the frequency of occurrence can be accumulated and ranked based on this.

The technique described herein in detail finds a number of applications in contexts involving watermarking, bar-coding, fingerprinting, OCR-decoding, and other ways to obtain information from an image. Again, consider the cell phone picture of the desk phone of FIG. Flicker may be retrieved based on image metrics to obtain a collection of object-like images (eg, as described above). The data extraction process (e.g., watermark decoding, fingerprint calculation, barcode- or OCR-reading) can be applied to some or all of the resulting images, and the information gathered by the metadata for the FIG. 21 image. And / or image data may be presented to the service provider (relative to FIG. 21 image and / or relative images).

From the collection of images found in the first search, text or GPS metadata can be collected and a second search can be done on similarly-tagged images. From the text tags Cisco and VOIP, for example, a search of Flickr can find a picture of the underside of the user's phone-as shown in FIG. 36-OCR-readable data. Again, the extracted information can be added to the metadata for the FIG. 21 image and / or presented to the service provider to enhance the response that can be provided to the user.

As just shown, the cell phone user may be provided with the ability to look under and around corners of objects-by using one image as a portal to a large collection of related images.

User interface

44 and 45A, cell phones and associated portable devices 110 typically include a display 111 and a keypad 112. In addition to keypads associated with numbers (or alphabets), multifunction controller 114 may be common. One popular controller has a center button 118 and four peripheral buttons 116a, 116b, 116c and 116d (also shown in FIG. 37).

An exemplary usage model is as follows. The system responds to the image 128 (optionally captured or wirelessly received) by displaying a collection of relevant images to the user on the cell phone display. For example, a user captures an image and presents it to a remote service. The service determines image metrics for the presented image (if possible, after pre-processing as described above), and retrieves visually similar images (eg flicker). These images are sent to the cell phone (eg, by service or directly from Flickr) and buffered for display. The service may prompt the user to repeatedly press the right-arrow button 116b on the 4-way controller (or press and hold) to view a sequence of pattern-like images, for example by instructions provided on the display. May be (FIGs. 45A, 130). Each time the button is pressed, the other of the buffered apparently-like images is displayed.

By techniques such as those described earlier, or otherwise, the remote service can also retrieve images whose geographical location is similar to the presented image. They can also be sent to the cell phone and buffered. The instructions may press the left-arrow button 116d of the controller to review these GPS-like images (FIGS. 45A, 132).

Similarly, the service can retrieve images that are similar in metadata to the presented image (eg, identified by pattern matching or GPS matching, based on metadata of text inferred from other images). Again, these images can be sent to the phone and buffered for immediate display. Instructions may recommend pressing the up arrow button 116a of the controller to view these metadata-like images (FIGS. 45A, 134).

Thus, by pressing the right, left, and up buttons, the user can view images similar to the image captured in the appearance, location, or metadata descriptors.

Each time such a review shows an image of particular interest, the user can press down button 116c. This operation identifies the picture that is currently-reviewed to the service provider, and can then repeat the process with the picture that is currently viewed as the base image. The process is then repeated with a user-selected image as a base, and button presses enable review of images that are similar to the base image and appearance 16b, location 16d or metadata 16a.

This process can continue indefinitely. At some point, the user can press the center button 118 of the four way controller. This action presents an image displayed at the service provider for another action (e.g., triggering a corresponding response, for example, as disclosed in earlier cited documents). This operation may involve different service providers from providing all alternative images, or they may be the same. (In the latter case, the finally-selected image does not need to be sent to the service provider because the service provider knows all the images buffered by the cell phone and can track which image is currently being displayed.)

The dimensions of the information browsing just described above (similar-appearance images; similar-position images; similar-position images; similar-metadata images) may be different in other embodiments. For example, consider an embodiment that takes an image of a house as input (or latitude / longitude) and returns the following sequences of images: (a) a house for sale at a location closest to the input-imaged house; (b) the home for sale at the price closest to the input-imaged home; And (c) houses for sale of features closest to the input-imaged home (eg, bedrooms / bathrooms). (The displayed home's paper may be restricted, for example, by zip code, metropolitan area, school district, or other modifiers.)

Another example of this user interface technology is the provision of search results from eBay for auctions listing Xbox 360 game consoles. One dimension can be a price (eg, pressing button 116b creates a sequence of screens showing Xbox 360 auctions starting at the lowest-priced auctions); Others may be the geographic proximity of the seller to the user (shown from the closest to the farthest by pressing button 116d); The other can be the time until the end of the auction (provided from the shortest time to the longest time by pressing button 116a). Pressing the middle button 118 can load the entire web page of the auction being displayed.

A related example is a user-captured image of a vehicle by identifying the vehicle, searching for similar vehicles on eBay and Craigslist, and providing the results on the screen (using image features and associated database (s)). Is a system that responds to. When button 116b is pressed, information about vehicles provided for sale (e.g., based on similarity (first model year / same color, then closest model year / colors) to the input image, nationwide, For example, images, seller location and price). Pressing button 116d creates such a sequence of screens, but is not limited to the user's condition (or metropolitan area or the user's location in a 50 mile radius, etc.). Pressing button 116a again creates a sequence of such geographically limited screens, but this is provided in time in ascending order of prices (rather than the closest model year / color). Again, pressing the middle button loads the entire web page (eBay or Craigslist) of the last-displayed vehicle.

Another embodiment is an application that helps people to recall names. The user sees a familiar person at the party, but cannot remember his name. Secretly, the user snaps the person's picture and the image is sent to the remote service provider. The service provider extracts face recognition parameters and retrieves similar-appearing faces in a separate database that includes face recognition parameters for social network sites or images on those sites (eg, Facebook, MySpace). (Linked-in). (The service may provide users with user's sign-on credentials, allowing for retrieval of information, otherwise not publicly accessible.) Names and other information about similar appearing people found through are returned to the user's cell phone-to help remind the user's memory.

Various UI procedures are considered. When data is returned from the remote service, the user can press button 116b to scroll the matches in the most similar order regardless of geography. Thumbnails of individuals that match the associated name and other profile information may be displayed, or only full screen images of the person may be provided-with overlapping names. Once a familiar person is recognized, the user can press button 118 to load the entire Facebook / MySpace / Linked-In page for that person. Alternatively, instead of providing images with names, only a list of texts of names may be provided, for example, all on a single screen-in order of similarity of face-matching; SMS text messaging may be satisfied with this final arrangement.

When button 116d is pressed, their residence, such as within a user's current location or within a particular geographic proximity of the user's reference location (eg, home) (eg, the same metropolitan area, same state, same campus, etc.) You can scroll the matches in order of the nearest-similarity of the listinger. When the button 116a is pressed, a similar display can be generated, but limited to those who are "friends" of the user in the social network (or those who are friends of friends, or those who are within another specified degree of separation of the user).

Related arrangements are law enforcement tools that allow officials to capture an image of a person and present it in a database that includes government driver license records and / or other source face portrait / unique information. Pressing button 116b causes the screen to display a sequence / electrical documents of images about people nationwide with the closest face matches. Pressing button 116d causes the screen to display a similar sequence, but limited to those within the state of the civil service. Button 116a creates such a sequence, but is limited to those in the metropolitan area in which the civil servant is working.

Instead of three dimensions of information browsing (e.g., buttons 116b, 116d, 116a for similar-appearing images / similarly-located images / similar metadata-tagged images) Can be utilized. 45B shows browsing screens in two dimensions. (Pressing the right button produces a first sequence of information screens 140; pressing the left button produces a different sequence of information screens 142.)

Instead of two or more individual buttons, a single UI control can be utilized to navigate in the available dimensions of the information. Joysticks are one such device. The other is a roller wheel (or scroll wheel). The portable device 110 of FIG. 44 has on its side a roller wheel 124 that can roll up or roll down. This can also be pressed in to select (for example, similar to the buttons 116c or 118 of the controller discussed earlier). Similar controls are available on many mice.

In most user interfaces, opposing buttons (eg, left button 116b and right button 116d) can navigate through the same dimension of information-only in opposite directions (eg, Forward / reverse). In the particular interface discussed above, it will be appreciated that this is not the case (though in other implementations it may be). If the right button 116b is pressed and then the left button 116d is pressed, the system does not return to the original state. Instead, pressing the right button provides, for example, a first similarly- appearing image, and pressing the left button provides, for example, a first similarly- located image.

Sometimes it is desirable to navigate through the same sequence of screens, but in the reverse order of just-reviewed. Various interface controls can be utilized to do this.

One is the "Reverse" button. The device 110 of Figure 44 includes various buttons that have not yet been discussed (eg, buttons 120a-120f around the periphery of the controller 114. Either-if pressed-in scrolling order). Serving in the reverse direction, for example, by pressing button 120a, the scrolling (providing) direction associated with nearby button 116b can be reversed, so that button 116b is usually in an order of increasing cost. Activating the button 120a may cause the function of the button 116b to switch to, for example, providing the items in order of decreasing cost. Upon reviewing, if the user wishes to "overshoot" and reverse the direction, she can press the screens 120a and then the button 116b again. In order - the current begins on the screen.

Alternatively, the operation of such a button (eg, 120a or 120f) may cause the opposite button 116d to scroll backward through the screens provided by the activation of the button 116b in reverse order.

The prompting of a text or symbol can be overlaid on the display screen in all these embodiments-notifying the user of the dimension and direction (eg, browsing by cost: increase) of the information being browsed.

In still other arrangements, a single button can execute multiple functions. For example, pressing button 116b may cause the system to begin providing a sequence of screens showing, for example, images of houses for sale closest to the user's location-providing every 800 milliseconds (user The interval set by the preference data entered by Pressing button 116b twice can cause the system to abort the sequence-displaying the static screen of the house for sale. Pressing button 116b three times may cause the system to present the screens in reverse order starting from the static screen and proceeding backward through the initially provided screens. Repeated operation of the buttons 116a, 116b, etc. may likewise operate (but control different information sequences, such as, for example, the houses of the closest price and the houses of the closest features).

In arrangements in which the information provided is rotated from a process in which the information provided is applied to a base image (e.g., a picture snapped by the user), this base image can be provided via the display-for example, a thumbnail at the display corner. As shown. Or a button on the device (eg, 126a or 120b) may be operable to immediately call the base image back to the display.

As in products available from Apple and Microsoft (e.g., in Apple's patent publications 20060026535, 20060026536, 20060250377, 20080211766, 20080158169, 20080158172, 20080204426, 20080174570 and Microsoft's patent publications 20060033701, 20070236470 and Touch interfaces are gaining popularity. These techniques can be used to enhance and extend just-reviewed user interface concepts—allowing greater degrees of flexibility and control. Each button press noted above may have a corresponding gesture in the vocabulary of the touch screen system.

For example, different touch-screen gestures may invoke the display of different types of image feeds just reviewed. For example, the brushing gesture to the right may provide right-scroll series of image frames 130 of the image with similar visual content (initial scrolling speed depends on the speed of the user gesture and scrolling speed is time). Slowed over-or not slowed down). The brushing gesture to the left may provide a similar left-scroll display of the image 132 with similar GPS information. The upward brushing gesture may present the image to an up-scroll display of the image 134 with similar metadata. At any point, the user can tap on one of the displayed images to make it the base image, and the process repeats.

Other gestures can invoke further actions. One such operation is to display an overhead image corresponding to the GPS location associated with the selected image. The image can be zoomed in / out along with other gestures. The user may have photographic images, map data, different times of day or different dates / seasons, and / or data from various overlays (topographical, places of interest, and other data as known from Google Earth), etc. Can be selected to display. Icons or other graphics may be provided on the display depending on the contents of the particular image. One such arrangement is detailed in Digimarc's published application 20080300011.

"Curbside" or "street-level" images are also displayed-rather than overhead images.

It will be appreciated that certain embodiments of the present technology include a shared general structure. An initial set of data (e.g., metadata such as image or descriptors or geographic code information, or image metrics such as unique values) is provided. From this, a second set of data (eg, image, or image metrics or metadata) is obtained. From the second set of data, the third set of data is compiled (eg, images or image metrics or metadata with similar image metrics or similar metadata). Items from data from the third set may be used as a result of the processing, or for example, by using the third set of data to determine the fourth data (eg, the set of descriptive metadata is third May be compiled from the images of the set) processing may continue. This may, for example, continue to determine a fifth set of data from the fourth data set (eg, identify a collection of images with metadata terms from the fourth data set). The sixth set of data may be obtained from five sets of data and the like (eg, identifying clusters of GPS by which images are tagged in the five sets of data).

The sets of data may be images or may be data of other borrows (eg, image metrics, metadata of text, geographic location data, decoded OCR-, barcode-, watermark-data, etc.).

Any data can be served as a seed. Processing may begin with image data, or may include image metrics, metadata of text (similar to meaningful metadata), geographic location information (eg, GPS coordinates), decoded OCR / barcode / watermark data, and the like. You can start with other information. From a first type of information (image metrics, semantic metadata, GPS information, decoded information), a first set of information-like images can be obtained. From that first set, second different types of information (image metrics / meaning metadata / GPS / decoded information, etc.) may be gathered. From that second type of information, a second set of information-like images can be obtained. From that second set, third different types of information (image metrics / meaning metadata / GPS / decoded information, etc.) may be gathered. From that third type of information, a third set of information-like images can be obtained. Etc.

Thus, although the illustrated embodiments generally start with an image and then are processed with reference to image metrics, completely different combinations of operations are also possible. The seed may be a payload from the product barcode. This may create a first collection of images depicting the same barcode. This can lead to a set of public metadata. This may result in a second collection of images based on the metadata. Image metrics are calculated from this second collection, and the most prevalent metrics can be used to retrieve and identify the third collection of images. The images thus identified may be provided to the user using the above known arrangements.

Certain embodiments of the present technology may be considered to use iterative, circular processing, whereby information about a set of images (a single image in many initial cases) may be used to identify a third set of images. It is used to identify a second set of images that can be used. The functionality with which each set of images is then associated with a particular class of image information, such as image metrics, semantic metadata, GPS, decoded information, and the like.

In other contexts, the relationship between one set of images and the next set of images is a function of not only one class of information, but also two or more classes of information. For example, the seed user image may be examined for both image metrics and GPS data. From these two classes of information, a collection of images can be determined-images that are similar in both visual appearance and position of some aspect. Other pairs of relationships, three pairs, etc. may naturally be utilized-in the determination of any of the successive sets of images.

Other discussion

Some embodiments of the present technology analyze consumer cell phone images and heuristically determine information about the subject of the pictures. For example, is it a person, a place or a thing? From this high level of determination, the system can better formulate what type of response can be pursued by the consumer—making the operation more intuitive.

For example, if the subject of the picture is a person, the consumer may be interested in adding a person depicted as a Facebook "friend." Or send a text message to the person. Alternatively, you can publish an annotated version of the photo on your website. Or simply learn who the person is.

If the object is a place (eg, Times Square), the consumer may be interested in local geography, maps, and popular things nearby.

If the object is an object (e.g., Liberty Bell or a beer bottle), the consumer may be interested in information about the object (e.g., its history, other things using it), or buying or selling the object.

Based on the image type, the example system / service may identify one or more operations that the consumer expects to find the most appropriate response to the cell phone image. One or all of these may be undertaken and cached on the consumer's cell phone for review. For example, scrolling the thumbwheel on the side of the cell phone can provide a series of different screens-each responding to an image object with different information. (Or the screen may be provided to query the consumer about which of the several possible actions is desired.)

In use, the system can monitor which of the available partners is selected by the consumer. The use history of soba can be utilized to refine the base model of consumer interests and wishes, so that future responses can be customized to better suit the user.

These concepts will be clearer by way of example (eg, the aspects depicted in FIGS. 46 and 47).

Processing a Set of Sample Images

Suppose a traveler snaps a picture of a Prometheus statue at Rockefeller Center in New York using a cell phone or other mobile device. At first, it is only one bunch of pixels. What will you do?

Assume that an image is geocoded with location information (e.g., latitude / longitude of XMP- or EXIF- metadata).

From the geographic code data, a search of Flickr for the first set of images can be undertaken-taken from the same (or nearby) location. Perhaps there will be 5 or 500 images in this first set.

Metadata from this set of images is collected. The metadata may be of various types. One is words / phrases from the title given to the image. The other is the information of the meta tags assigned to the image-generally by the photographer (eg naming the photographic subject and certain attributes / keywords), but additionally, the capture device (eg the camera model). By identifying the date / time, location, etc. of the photo. Others are words / phrases in the narrative description of photography authored by the photographer.

Some metadata terms may be repeated over different images. Descriptors common to two or more images can be identified (clustered) and the most popular words can be ranked. (This listing is shown at "A" in Figure 46A. Here, and in other metadata listings, only partial results are given for ease of explanation.)

From metadata, and from other analysis, it may be possible to determine which images in the first set are person-centric, which images are place-centric, and which images are object-centric.

Consider metadata that can be tagged in 50 image sets: some of the terms are related to places. Some are related to the people depicted in the images. Some are related to things.

Place-based processing

Terms relating to a place may be identified using various techniques. One is to look up location descriptors near a given geographic location using a database with geographic information. Yahoo's GeoPlanet service, for example, queries the latitude / longitude of Rockefeller Center, "Rockefeller Center", "10024" (zip code), "Midtown Manhattan", "New York", "Manhattan" "New York" Returns a hierarchy of descriptors such as "and" United States ".

The same services are available on request, for example, "10017", "10020", "10036", "Theater District", "Carnegie Hall", "Grand Central Station", "American Folk Art Museum", etc. It may return the names of neighbors / features between neighboring / siblings.

Given a set of latitude / longitude coordinates or other location information, nearby street names can be obtained from various mapping programs.

The glossary of nearby place-descriptors can be compiled this way. The metadata obtained from the set of flicker images can then be analyzed with reference to the glossary to identify terms related to the place (eg, matching the terms in the glossary).

Subsequently, considerations turn to the use of these place-related metadata in a reference set of images collected from Flickr.

Some images may have data that is not place-related metadata. These images are more likely to be person-centric or object-centric than place-centric.

Other images may have exclusively place-related metadata. These images are more likely place-centric than person-centric or object-centric.

Between the two are images with two place-related metadata and other metadata. Various rules can be devised and utilized to assign the relative relevance of an image to a place.

One rule looks at the number of metadata descriptors associated with an image and determines the fractions found in the glossary of place-related terms. This is one metric.

The other sees where the place-related descriptors appear in the metadata. If they appear in the image titles, they are more relevant than they would appear at the end of a long descriptive description of the photograph. The place of place-related metadata is another metric.

Consideration may also be given to the specificity of the location-related descriptor. The descriptor "New York" or "USA" may indicate less that the image is place-centric than more specific descriptors, such as "Rockefeller Center" or "Grand Central Station". This may generate a third metric.

The relevant fourth metric takes into account the frequency of occurrences (or unlikely to occur) of terms-within collected metadata or within a superset of that data. From this point of view the "RCA Building" is more relevant than the "Rockefeller Center" because it is much less frequent.

These and other metrics can be combined such that each image in the setting is assigned a place score that indicates its potential place-centricity.

The combination can be a series of four factors, each in the range of 0 to 100. However, more particularly, some metrics will be weighted more heavily. The following equation using the metrics Ml, M2, M2 and M4 can be utilized to yield a score S, the factors A, B, C, D and the exponents W, X, Y and Z are experimentally based on the base technique. Determined by:

People-centric processing

It can be utilized to estimate the person-centricity of each image of the set obtained from Flickr.

As in the example just given, a glossary of related terms can be compiled-these terms are associated with a person. In contrast to place name glossaries, person name glossaries can be global—rather than associated with a particular site. (However, different glossaries may be appropriate for different countries.)

This glossary can be compiled from various sources, including phone directories, lists of the most popular names, and other reference tasks in which names appear. The list can begin with: "Aaron, Abigail, Adam, Addison, Adrian, Aidan, Aiden, Alex, Alexa, Alexander, Alexandra, Alexis, Allison, Alyssa, Amelia, Andrea, Andrew, Angel, Angelina, Anna, Anthony, Antonio, Ariana, Arianna, Ashley , Aubrey, Audrey, Austin, Autumn, Ava, Avery ... "

The first names may be considered alone, or the last names may also be considered. (Some names may be place names or person names. Searching for adjacent first / last names and / or adjacent place names can help distinguish ambiguous cases. For example, Elizabeth Smith is a person; Elizabeth NJ is a place.)

Personal pronouns may also be included in this glossary (eg, he, she, him, to her, him, her, his, our, her, me, me, myself, us, Them, to them, mine, their). Nouns that identify people and personal relationships may also be included (eg, uncle, older sister, daughter, grandfather, boss, student, employee, wedding, etc.).

As they may be names of attributes and objects that are commonly associated with a person (e.g., t-shirts, backpacks, sunglasses, tanned, etc.), adjectives and adverbs that generally apply to a person may also be used in a person-term glossary. May be included (eg happy, boring, blond, etc.). Verbs associated with humans may also be utilized (eg surfing, drinking).

In this final group, as in some others, there are certain terms that can also be applied to object-centric images (rather than person-centric). The term “sunglasses” may appear in metadata for an image that depicts sunglasses alone; "Happy" can appear in the metadata for an image depicting a dog. There are also some cases where a person-term may also be a place-term (eg, boring, Oregon). In more sophisticated embodiments, glossary terms may be associated with respective confidence metrics, whereby any results based on these terms may be discounted or acknowledged to have different degrees of uncertainty. have.)

As before, if the image is not associated with any person-related metadata, it may be determined that the image is unlikely to be person-centric. Conversely, if all metadata is person-related, then the image is likely person-centric.

For other cases, metrics such as those reviewed above may be based on, for example, the relative person of each image, based on the number, place, specificity and / or frequency / inability of the person-related metadata associated with the image. -Can be evaluated and combined to yield a score indicating centrality.

While the analysis of the metadata provides useful information as to whether the image is person-centric, other techniques may also be utilized-alternatively, or in conjunction with metadata analysis.

One technique is to find successive areas of skin tone colors and analyze the image. These features characterize many of the people-centric images, but are less frequently found in the images of places and objects.

A related technique is face recognition. This science is the point where even digital cameras that only need to look and shoot cheaply can quickly and reliably identify faces in an image frame (for example, to focus or expose an image based on these objects). Progressed to

(Face retrieval techniques are described, for example, in patents 5,781,650 (Central Florida University), 6,633,655 (Sharp), 6,597,801 (Hewlett-Packard) and 6,430,306 (L-1 Corp.), and in January 2002 IEEE Transactions on Pattern Analysis and Detecting Faces in Images: A Survey, by Yang et al. In Machine Intelligence, Vol. 1, No. 24, pp. 1, 34-58, and Face Recognition: A Literature Survey, by Zhao et al., 2003, ACM Computing Surveys, pp. 399-458. It is detailed in.)

Face recognition algorithms may be applied to a set of reference images obtained from flicker to identify those with obvious faces and to identify locations of images corresponding to the faces.

Naturally, many photos have faces depicted by chance within an image frame. All images with faces can be identified as person-centric, but most embodiments utilize other processing to provide a further refined assessment.

Another form of processing is to determine the percentage area of the image frame occupied by the identified face (s). The higher the percentage, the more likely the image is person-centric. This is a different metric than can be used to determine the person-centric score of the image.

Another form of other processing is to (1) find the presence of one or more faces in an image, along with (2) the person-descriptors of metadata associated with the image. In such cases, facial recognition data may be used as a "plus" factor to increase the person-centric score of the image based on metadata or other analysis. ("Plus") can take various forms. For example, the score (on a scale of 0-100) can be increased by 10, or by 10%, or by half of the remaining distance to 100. , Etc.)

Thus, for example, a picture tagged with "Elizabeth" metadata is more likely a person-centric picture when the face recognition algorithm finds a face within the image than when no face is found.

(On the contrary, the absence of any face in an image can be used as a "plus" factor to increase the confidence that the image object is of a different type, for example a place or an object. Images without faces increase the likelihood that the image is related to a place-named Elizabeth or an object-named Elizabeth-like a pet.)

If the face recognition algorithm recognizes a face as a female and the metadata includes a female name, it can also be assumed that it is more trusted in the decision. Naturally, this arrangement requires that the glossary-or other data structure-have data that associates at least certain names and genders.

(More sophisticated arrangements can be implemented. For example, the age of the depicted person (s) can be estimated using automated techniques (eg, as detailed in Patent 5,781,650 of Central Florida University). Names found in image metadata can also be processed to estimate the age of the person (s) of this name, which can be done using public domain information about the statistical distribution of names as a function of age (eg For example, from published social security management data, and from Web sites detailing the most popular names from birthday records.) Thus, the names Mildred and Gertrude may be associated with an age distribution that peaks at 80 years old. , Madison, and Alexis can be associated with an age distribution that peaks at 8. The statistically possible correspondence between the metadata name and the estimated human age If found, the person-centric score for the image can be further increased, a statistically impossible correspondence can be used to reduce the person-centric score (the estimated information about the subject's age in the consumer's image is also It may be information about the subject's gender and may be used to tailor the intuitive response (s).)

Since the detection of a face in an image can be used as a "plus" factor based on metadata, the presence of the person-centered metadata as a "plus" factor to increase the person-centric score based on face recognition data. Can be used.

Naturally, if no face is found in the image, this information can be used to monitor the person-centric score for the image (perhaps down to zero).

Stuff-centric processing

The object-centric image is a third type of image that can be found in the set of images obtained from flicker in this example. There are various techniques by which the object-centric score for an image can be determined.

One technique relies on metadata analysis using the same principles as those described above. The glossary of nouns can be compiled and ranked by the frequency of occurrences-from bulk flicker metadata or from some other corpus (eg, WordNet). Nouns associated with places and people may be removed from the glossary. The glossary can be used in the above identified ways to perform analyzes of the metadata of the image to yield a score for each.

Another approach uses pattern matching to identify object-centric images that each match against a library of known object-related images.

Another approach is based on initially determined scores for person-centered and place-centric. The object-centric score may be assigned inversely to the other two scores (ie, if the image scores are low for person-centric and low for object-centric, a high score for object-centric may be assigned). Can be).

These techniques may be combined or used separately. In any event, a score is generated for each image-tending to indicate whether the image is more or less likely to be object-centric.

Other processing of the sample set of images

The data generated by the techniques described above is for each image in the set representing roughly trust / probability / likelihood that the image is (1) person-centric, (2) place-centric, or (3) object-centric. Three scores can be generated. These scores need not be added to 100% (although it may be). Occasionally, an image may have a high score in two or more categories. In such cases, the image may be considered to be of multiple relevance, for example, both depicting people and objects.

The set of images downloaded from Flickr may then be separated into groups of A, B and C, for example, depending on whether they are identified primarily as person-centric, place-centric, or object-centric. However, since some images may separate probabilities (eg, an image may have some place-centric indicator, some indicators may be person-centric), for example, an image as a whole with a high score. Identifying ignores useful information. It is preferable to calculate the weighted score for the set of images-taking into account the respective score of each image of the three categories.

A sample of images from Flickr-all taken near the Rockefeller Center-can suggest that 60% are place-centric, 25% are person-centric, and 15% are object-centric.

This information provides useful insight (except for its geographic coding) to the traveler's cell phone image-without regard to the contents of the image itself. That is, it is good opportunities that the image is less likely to be place-centric, person-centric, and less likely to be object oriented. (This order can be used to determine the order of subsequent stage systems of processing-allowing the system to more quickly provide the responses most likely to be appropriate.)

This type-evaluation of the cell phone photo can be used-alone-to help determine the automated action provided to the traveler in response to the image. However, other processing may better evaluate the contents of the image, thereby allowing a more specifically tailored action to be intuitive.

Similarity Assessments and Metadata Weighting

Within a set of co-located images collected from Flickr, place-centric images will try to have a different appearance than person-centric or object-centric images, but will have some similarity within the place-centric group. Place-centric images can be characterized by straight lines (eg, architectural edges). Or by repetitive patterns (windows). Or by large areas of similar color and uniform texture near the top of the image (sky).

Person-centric images will also try to have different appearances than images of the other two grades, but will have common attributes within the person-centric grade. For example, human-centered images will generally have faces-characterized by an elliptical shape with two eyes and a nose, areas of skin tones, and the like.

While object-centric images are likely to be the most different kind, images from any given geography may tend to unify properties or features. Geographically coded pictures on a horse track will describe the horses at any frequency; Geographically coded photographs from Independence National Historical Park in Philadelphia will regularly attempt to describe the species of freedom.

By determining whether the cell phone image is more similar to place-centric or person-centric, or object-centric images in the set of flicker images, more trust can be achieved in the object of the cell phone image (and, moreover, The correct response can be intuitive and provided to the consumer).

A fixed set of image evaluation criteria can be applied to distinguish images in these categories. However, the embodiment described above adaptively determines this criterion. In particular, this embodiment examines the set of images and which image features / characteristics / metrics are most reliably grouped together (1) with the same-categorized image (similarity); And (2) whether different categorized images are distinguished from each other (difference). Among the attributes that can be measured and identified for similarity / difference behavior in a set of images, prevailing color; Color diversity; Color histogram; Predominant textures; Texture diversity; Texture histogram; Edginess; Wavelet-domain transform coefficient histograms, and dominant wavelet coefficients; Frequency domain transmission coefficient histograms and dominant frequency coefficients (which can be calculated in different color channels); Eigenvalues; Keypoint descriptors; Geometry class probabilities; Symmetry; The percentage of the image area identified as a face; Image autocorrelation; Low-level "points" of the image, etc. (Combinations of such metrics are more reliable than individual characteristics.)

 One way to determine which metrics are most prominent for these purposes is to calculate various different image metrics for the reference images. If results in a category of images for a particular metric are clustered (eg, for place-centric images, a color histogram is clustered near specific output values), and images in other categories are clustered. If there are few output values near the result, the metric appears to be well suited to use as an image evaluation criterion. (Clustering is generally performed using an implementation of the k-means algorithm.)

 In the set of images from the Rockefeller Center, the system checks whether an edgeness score of> 40 is reliably associated with the images scored as high as place-centric; Facial area score of> 15% is reliably associated with images scored as high as person-centered; And a color histogram with a local peak of gold tones, with frequency content for yellow that peaks at low image frequencies, may be somewhat associated with images scored as object-centric as high.

The analysis techniques found to be most useful for grouping / dividing images of different categories can then be applied to the user's cell phone image. The results can then be analyzed for proximity-in terms of distance measurement (eg multidimensional space), with characteristic features associated with different categories of images. (This is the first time a cell phone image has been processed in this particular embodiment.)

Using these techniques, the cell phone image can be scored as 60 for object-centric, 15 for place-centric, and 0 for person-centric (for a scale of 0-100). This is a second, better set of scores that can be used to classify cell phone images (first is the statistical distribution of co-located photos found in Flickr).

The similarity of the user's cell phone image can then be compared to the individual images in the reference set. Initially identified similarity metrics may be used or different measurements may be applied. The time or processing devoted to this task can be distributed across three different image categories based on the just determined scores. For example, the process may not spend time determining similarity with reference images classified as 100% person-centric, but instead determines similarity with reference images classified as object- or place-centric. Can concentrate (more effort is applied to the former than the latter-for example 4 times as hard). Similarity scores are generated for most of the images in the reference set (including those evaluated at 100% person-centered).

After that, the consideration is returned to the metadata. Metadata from the reference images is assembled again-this time is weighted according to each similarity of each image to the cell phone image. (Weights can be linear or exponential.) Since metadata from similar images is weighted more than metadata from dissimilar images, the resulting set of metadata is more likely to correspond to cell phone images. This is so tailored.

From the resulting set, the top N (e.g. 3) metadata descriptors can be used. Or descriptors based on weighting, including a metadata set of total M%, may be used.

In the given example, the metadata so identified can include "Rockefeller Center", "Prometheus" and "Skating Link", each with scores of 19, 12 and 5 (see "B" in FIG. 46B). .

Using this weighted set of metadata, the system can begin to determine which responses are best for the consumer. However, in the exemplary embodiment, the system continues by further improving the evaluation of the cell phone image. (The system may also begin to determine appropriate responses while embarking on another process.)

Processing a second set of reference images

At this point, the system is better informed about the cell phone image. As well as location; You will also know the possible types (stuff-centric) and the most likely related metadata. This metadata can be used to obtain a second set of reference images from Flickr.

In an exemplary embodiment, Flickr is queried for images with identified metadata. The query may be geographically limited to the geographic location of the cell phone, or wider (or unrestricted) geography may be retrieved. (Or the query can be executed twice, so that half of the images are co-located with the cell phone image, the rest are remote, etc.)

The search can find all identified metadata and tagging images first. In this case 60 images are found. If more images are desired, Flickr can be searched for metadata terms in different pairs or individually. (In these latter cases, the distribution of the selected images may be selected such that the metadata correspondence of the results corresponds to each score of different metadata terms, eg 19/12/5.)

Metadata from this second set of images may be collected, clustered, and ranked (“C” in FIG. 46B). (Unnecessary words ("and, of, or", etc.) can be eliminated.) Descriptive words of the type of camera or dedicated words of the camera can also be ignored (e.g. "Nikon", "D80", "HDR"). "," Black and white ", etc.) Month names can also be removed.)

The analysis performed initially-by each image in the first set of images being classified as person-centric, place-centric or object-centric-can be repeated for the images in the second set of images. Can be. Suitable image metrics may be identified (or initial measurements may be utilized) to determine similarity / difference within and between the grades of this second image set. These measurements are then applied to generate improved scores for the user's cell phone image as being person-centric, place-centric or object-centric as before. Referring to the second set of images, the cell phone image can be scored as 65 for object-centric, 12 for place-centric, and 0 for person-centric. (These scores can be combined with initially determined scores, if desired, for example, by averaging.)

As before, the similarity between the user's cell phone image and each image of the second set may be determined. The metadata from each image can then be weighted according to the corresponding similarity measure. The results can then be combined to yield a set of weighted metadata according to image similarity.

Part of the metadata, often including some highly ranked terms, will be a relatively low value in determining image-suitable responses to provide to the consumer. "New York" and "Manhattan" are some examples. In general, relatively rare metadata descriptors are more useful.

The measurement of "unusualness" can be either flicker image tags (globally or within a geographically located area), or image tags by photographers where each image is presented, or the encyclopedia or Google of a website. It can be calculated by determining the frequency of different metadata terms in the relevant corpus, such as words in the index of. Terms in the weighted metadata list may be further weighted according to their novelty (ie, second weighting).

This successive processing result can produce a list of metadata shown at " D " in FIG. 46B (each with each score). This information (optionally with a tag indicating a person / place / stuff decision) allows the responses to the consumer to correlate well with the cell phone photo.

This set of inferred metadata for the user's cell phone photo is entirely by automated processing of other images obtained from public sources such as Flickr along with other public resources (e.g., listing of names and places). You will see that it has been compiled. Inferred metadata can naturally be associated with the user's image. However, more importantly for the present application, it can assist the service provider in determining the best way to respond to the presentation of the user's image.

Determination of appropriate responses to the consumer

Referring to FIG. 50, the system just described may appear as one particular application of an "image juicer" that receives image data from a user, and collects, calculates, and / or information that may be associated with an image. Apply different forms of processing to infer.

When the information is identified, it can be sent by routers to different service providers. These providers process different types of information (eg semantic descriptors, image texture data, keypoint descriptors, eigenvalues, color histograms, etc.), or images of different grades (eg friends' photos). , Photos of soda cans, etc.). Outputs from these service providers are sent to one or more devices (eg, user's cell phone) for provision or for later reference. This discussion now considers how these service providers determine which responses may be appropriate for a given set of input information.

One way is to establish a taxonomy of image objects and corresponding responses. A tree structure can be used, and the image is first classified into one of several high level groups (eg, person / place / stuff), after which each group is divided into different subgroups. In use, the image is evaluated through different branches of the tree until the limitations of the information available do not allow for further progress. The actions associated with the terminal leaf or node of the tree are then taken.

Part of the simple tree structure is shown in FIG. 51. (Each node makes three branches, but this is for illustration only; some branches can be used as well.)

If the subject of the image is inferred to be a food item (eg, if the image is associated with food-related metadata), the information of three different screens may be cached in the user's phone. One initiates online purchases of the item depicted at the online vendor (vendor selection, and payment / shipment details can be obtained from the user profile data.) The second screen shows nutritional information about the product. The third screen provides a near map-identifying the stores that sell the depicted product. The user uses the roller wheel 124 on the side of the phone to switch between these responses (FIG. 44).

If the subject is inferred to be a picture of a family member or friend, one screen provided to the user provides the option to post a copy of the picture on the user's Facebook page, and the possible name (s) of the person (s) are annotated. Run (Determining the names of the people depicted in the photos can be done by presenting the photos to the user's account in Picasa. Picasa performs facial recognition operations on the presented user images, and the individual names and faces provided by the user. Correlate the eigenvectors and thereby compile a user-specific database of face recognition information for friends and others depicted in the user's previous images .. Picasa's face recognition is based on the technique detailed in patent 6,356,659 to Google. Apple's iPhoto software and Facebook's photo finder software include similar facial recognition functions.) Another screen launches a text message to an individual, with the user whose address information indexed by a Picasa-determined identity. Is obtained from an address book. The user can continue any or all of the options provided by switching between associated screens.

If the subject appears to be a stranger (eg not recognized by Picasa), the system will initially initiate the person's attempted recognition using publicly available facial recognition information. (This information can be extracted from photos of known people. VideoSurf is a vendor with a database of facial recognition features for actors and others. L-1 Corp. To maintain a database of driver's license photos and associated data that can be utilized-using suitable safeguards. The screen (s) provided to the user not only show reference photos of matched persons ("matching"). Along with the scores), can show relevant documents of associated information compiled from the Web and other databases. Another screen provides the user with the option to send a "friend" invitation to the recognized person on MySpace or another social networking site where the recognized person is found to be present. Another screen details the degree of separation between the user and the recognized person. (For example, my brother David has classmate Steve, and he is the son of the person depicted.) These relationships can be determined from publicly relevant information published on social networking sites.

Naturally, each option considered for different sub-groups of image objects may meet most user needs, but some users will want others. Thus, at least one alternative response to each image may be unlimited—for example allowing a user to navigate different information or specifying a desired response—image / metadata processed information is used. You can use anything you can.

One such non-limiting way is to present the double-weighted metadata noted above (eg, “D” in FIG. 46B) to the universal search engine. Google does not necessarily have to be the best for this feature, as Google search now requires that all search terms be found in the results. Better is a search engine that does fuzzy search and responds differently-weighted keywords-without having to find everything. The results can indicate a seemingly different relationship depending on where the keywords are found, where they are found, and so on. (Results that include "Prometheus" but lack "RCA Building" rank more relevant than results that include the latter but lack the former.)

The results from this search can be clustered by other concepts. For example, some results may be clustered because they share the subject "art deco". Others can be clustered because they address the history of RCA and GE collaboration. Others can be clustered because they relate the works of architect Raymond Hood. Others may be clustered as they relate to 20th century American sculptures or Paul Manships. Other concepts discovered to create individual clusters may include John Rockefeller, Mitsubishi Group, Columbia University, Radio City Music Hall, Rainbow Room Restaurant, and the like.

Information from these clusters may be provided to the user on successive UI screens, for example, after prescribed information / operations are provided on the screen. The order of these screens may be determined by keyword-determined relevance or sizes of information clusters.

Another response is to provide the user with a Google search screen-pre-placed with twice the weighted metadata as search terms. The user can then delete terms that are irrelevant to his or her interests and add other terms to quickly execute a web search leading to the action or information required by the user.

In some embodiments, the system response may depend on people with whom the user has a "friend" relationship in the social network or some other indicator of trust. For example, if little is known about User Ted, but there is a rich set of information available about Ted's friend Alice, that rich set of information can be used to determine how to respond to Ted with a given content stimulus. have.

Similarly, if User Ted is a user Alice's friend and Bob is Alice's friend, information related to Bob can be used to determine a proper response to Ted.

Assuming that there is another basis for implicit trust, the same principles can be used when Ted and Alice are strangers. When basic profile similarity is one possible basis, it is better to share rare attributes (or better, several). Thus, for example, if Ted and Alice both share Dennis Kucinich's ardent supporters of the president and are enthusiasts of ginger pickles, information about them can be used to determine the appropriate response to be provided to others.

The arrangements just described offer powerful new features. However, the "intuiting" of responses that the user may wish to rely heavily on system designers. They take into account different types of images that can be encountered and can indicate responses (or choices of responses) that we believe will best meet the user's possible desires.

In this regard, the above-described arrangements allow Yahoo! to manually search for human-generated taxonomics of information that people can search for and web resources that can satisfy different search results. Similar to the initial indexes of the web like teams.

In the end, the web overwhelmed such manual efforts in the organization. Google's founders were among those who recognized that the untapped richness of information about the web could be obtained by examining the links between pages and the actions of users when navigating these links. Thus, the understanding of the system came from the data within the system rather than from the outside.

In the same way, manually created trees of image classifications / responses are likely to appear to be an early stage in the development of image-response techniques later. In the end, these approaches will be obscured by arrangements that depend on machine understanding derived from the system itself and its use.

One such technique simply examines which response screen (s) are selected by the users in specific contexts. Since these usage patterns are obvious, the most popular responses can be moved earlier in the sequence of screens presented to the user.

Likewise, if the patterns become evident in the use of the unlimited search query option, this behavior may be a standard response and may be moved higher in the provision queue.

Usage patterns can be tailored in the context of various dimensions. Men in New York, ages 40 to 60, can demonstrate interest in responses different from women in Beijing, ages 13-16, after capturing snapshots of statues by 20th-century sculptors. Most people who snap photos of food handlers in the weeks before Christmas may be interested in finding the cheapest online vendor of the product; Most people who snap photos of the same object the week before Christmas may be interested in listings for sale on eBay or Craigslist. Preferably, usage patterns are tracked with as many demographics and other descriptors as possible to predict the majority of user behavior.

More sophisticated techniques derived from abundant sources of currently and explicitly available speculatively linked data sources can also be applied. These include, for example, cell phone bills, credit card statements, shopping data from Amazon and eBay, Google search history, browsing history, cached web pages, cookies, email archives, as well as web and personal profile information. , Phone message archives from Google Voice, travel reservations on Expedia and Orbitz, music collections on iTunes, cable television subscriptions, Netflix movie selections, GPS tracking information, social network data and activities, Flickr and Picasawa Activities and postings on photo sites such as YouTube and YouTube, date and time these records were recorded (our "digital life log"), all the way we touched and left traces Include other digital data. Moreover, this information is potentially available not only for the user but also for the user's friends / family, for others with demographic similarities with the user, and ultimately for everyone else (appropriate anonymous and / Or with privacy protectors).

The network of correlations between these data sources is smaller than the network of web links analyzed by Google, but perhaps richer in diversity and types of links. From this, it is possible to extract a wealth of inferences and insights that can help inform a particular user what he or she wants to process with a particular snapped image.

Artificial intelligence techniques can be applied to data-gathering tasks. One class of these technologies is natural language processing (NLP), and the science has evolved significantly.

One example is a semantic map compiled by Cognition Technologies, Inc., which is a database that can be used to analyze words in a context to distinguish these meanings. This function can be used, for example, to resolve homonym ambiguities in the analysis of image metadata (e.g., "bow" is part of a ship, or a ribbon ornament, or a thank you from the practitioner, or of an arrow). The closeness of terms like "Carnival cruise", "satin", "Carnegie Hall", or "hunting" can provide a probable answer. have). Patent 5,794,050 to FRCD Corp. details basic techniques.

An understanding of the meaning obtained through NLP techniques can be used to augment image metadata with other related descriptors-which can be used as additional metadata in the embodiments detailed herein. For example, a close-up image tagged with the descriptor "hibiscus stamens" can be further tagged with the term "flower"-through NLP techniques. (With this record, Flickr has 460 images tagged as "Habiscus" and "Surgery" but omitting "Flower.")

Patent 7,383,169 (Microsoft) details how dictionaries and other large tasks of the language can be handled by NLP techniques to compile lexical knowledge bases that serve as fictitious sources of this "common sense" information about the world. . This common sense knowledge can be applied to the metadata processing detailed herein. (Wikipedia is another reference source that can serve as the basis for this knowledge base. Our digital lifelog is another-generating insights that are unique to us individually.)

When applied to our digital lifelog, NLP techniques understand subtle differences about our historical interests and actions-information that can be used to model (predict) our current interests and upcoming actions. Can be reached. This understanding can be used to dynamically determine what information should be provided or what action should be undertaken, in response to a particular user (or other stimulus) capturing a particular image. After that you will actually come to an intuitive calculation.

Other opinions

Although the image / metadata processing described above has taken and described many words, it does not need to take much time to execute. Indeed, processing of a large number of reference data, compilation of glossaries, etc. can be done off-line before any input image is provided to the system. Flickr, Yahoo! Or other service providers are readily available when they need to compile periodically and pre-process reference sets of data for various sites to answer image queries.

In some embodiments, other processing activities will begin in parallel with those described above. For example, if initial processing of the first set of reference images suggests that the snapped image is place-centric, the system may request likely-useful information from other resources before processing of the user image is terminated. have. To illustrate, the system may immediately request a street map of the surrounding area along with satellite view, street view, large scale transport map, and the like. Similarly, a page of information about surrounding restaurants can be compiled, along with another page detailing surrounding movies and show-times, and another page of local weather forecasts. They can all be sent to the user's phone and cached for later display (eg by scrolling the thumbwheel on the side of the phone).

These operations can likewise be undertaken before any image processing takes place-simply based on geographic code data accompanying the cell phone image.

Although geographic coding data involving cell phone images has been used in specially described arrangements, this is not essential. Other embodiments may select sets of reference images based on other criteria such as image similarity. (This may be determined by various metrics as described above and below. Known image classification techniques may also be used to determine one of several grades of images that include the input image, such that similarly classified images Another criterion is the IP address from which the input image is uploaded. Other images uploaded from the same-or geographically close-IP addresses can be sampled to form reference sets.

Despite the absence of geographic code data for the input image, the reference sets of the image can be compiled based on the location. Location information for the input image can be inferred from various indirect techniques. The wireless service provider to which the cell phone image is relayed may identify the particular cell tower from which the traveler's transmission was received. (If the transmission was through another wireless link, such as WiFi, the location is also known.) The traveler may have used his credit card an hour earlier at the Manhattan hotel, so that the system (with appropriate privacy guards) Allow to infer that the picture was taken somewhere near Manhattan. Sometimes, the features depicted in an image are symbolic, so a quick search for similar images in Flickr can find the user (eg, in the Eiffel Tower, or in the Statue of Liberty).

GeoPlanet is cited as one source of geographic information. However, many other geographic information databases may alternatively be used. GeoNames-dot-org is one. (Note that the "-dot-" switch and the omission of the usual http preamble are used to prevent playback by the Office of patents from being displayed as live hyperlinks to this text.) Place names for a given latitude / longitude ( In addition to providing neighboring, city, state, and country levels, and providing parent, child, and sibling information for geographic divisions, Geonames' free data (available as a web service) is also the nearest intersection. It provides features such as finding, finding the nearest post office, finding surface elevation, and more. Another option is Google's GeoSearch API, which allows interacting with and searching for data from Google Earth and Google Maps.

It will be appreciated that archives of aerial imagery are growing exponentially. Part of this image is a straight line view, but the off-axis of the image gradually becomes oblique. From two or more different oblique views of the position, a 3D model can be generated. As the resolution of this image increases, a fairly rich set of data-for some locations-is available so that a view of the scene, such as taken from the ground level, can be synthesized. Such views can be matched with street level photos and metadata from one can augment the metadata for the other.

As shown in FIG. 47, the embodiment described above utilized a variety of resources, including Flickr, a database of person names, a word frequency database, and the like. There are some of many different information sources that can be utilized in such arrangements. Other social networking sites, shopping sites (eg, Amazon, eBay), weather and traffic sites, online thesaurus, cache of recently visited web pages, browsing history, cookie collection, Google, other digital repositories (As detailed herein), and the like can all provide a wealth of additional information that can be applied to the intended tasks. Some of this data reveals information about the user's interests, habits and preferences-data that can better infer the contents of the snapped picture and better tailor the intuitive response (s).

Likewise, although FIG. 47 shows several lines interconnecting different items, these are merely exemplary. Different interconnections may naturally be utilized.

The arrangements detailed in this specification are some of the myriad of which can be utilized. Most embodiments will be different from those described above. Some operations will be omitted, some will execute in different orders, some will execute in parallel rather than in series (or vice versa), some additional operations may be included, and the like.

One additional operation is to improve the processing just described, for example by receiving user-related input after processing the first set of flicker images. For example, the system has identified "Rockefeller Center", "Prometheus" and "Skating Link" as relevant metadata for user-snapped images. The system can query the user as to which of these terms is most relevant (or at least related) to his / her particular interest. Other processing (eg, other searches, etc.) may thus be focused.

Within the image provided on the touch screen, the user can touch the area to represent an object of a particular relevance in the image frame. Thereafter, image analysis and subsequent operations can be focused on the identified object.

Some of the database searches may be recursive / rotary. For example, the results from one database search can be combined with the original search inputs and used as inputs to another process.

It will be appreciated that most of the above described processes are blurred in boundaries. Most data does not have absolute meaning, but may be in terms of metrics that are only related to a different range than other metrics. Many such different stochastic factors can be evaluated and then combined-statistical stew. Those skilled in the art will appreciate that the particular implementation suitable for a given situation may be predominantly arbitrary. However, through experiences and base techniques, more inferred ways of weighting and using different factors can be identified and eventually used.

If the flicker archive is large enough, the first set of images in the arrangement described above may be selectively selected to be more likely to be similar to the target image. For example, Flickr can retrieve images taken at about the same time of day. The lighting states will be approximately similar, for example, to avoid matching the night scene to the day scene, and the shadow / shading states will be similar. Similarly, images taken in the same season / month can be retrieved from Flickr. Thus, problems such as the ice skating rink at the Rockefeller Center and the seasonal loss of snow on the winter landscape can be mitigated. Similarly, if the camera / phone is equipped with a magnetic field, an internal sensor, or other technology that allows the bearing to be determined (and / or azimuth / altitude), shots with this degree of similarity in flicker can also be retrieved.

Moreover, the sets of reference images collected from Flickr preferably include images from many different sources (photographers)-so they do not tend to use the same metadata descriptors.

Images collected from Flickr can be screened for appropriate metadata. For example, images without metadata (possibly excluding any number of images) may be removed from the reference set (s). Similarly, images with fewer than two (or 20) metadata terms or no descriptive description can be ignored.

Flickr is often mentioned in this specification, but collections of other content may naturally be used. Images in Flickr generally have designated license rights for each image. These include various Creative Commons licenses, as well as "all rights envisioned," allowing the public to use images for different terms. The systems described herein above may restrict retrieval through Flickr for images that meet specified license criteria (eg, ignore images marked as "all rights envisioned").

Other image collections are good in some respects. For example, images. The database at google-dot-com looks better on ranking images based on meta-relevance than Flickr.

Flickr and Google maintain publicly accessible image archives. Many other image archives are secret. Embodiments of the present technology may find an application with both-including some hybrid contexts in which both public and owner image collections are used (e.g., Flickr is used to find an image based on a user image, The flicker image is presented in a secret database to find a match and determine the corresponding response to the user.)

Similarly, although referenced to services such as Flickr to provide data (eg, images and metadata), other sources may of course be used.

One alternative resource is an ad hoc peer-to-peer (P2P) network. In one such P2P arrangement, there may optionally be a central index, which allows peers to communicate when searching for the desired content, detailing the content they are available for sharing. The index may include metadata and metrics for the images, along with pointers to the nodes where the images themselves are stored.

Peers may include cameras, PADs, and other portable devices from which image information may be available almost immediately after being captured.

In the course of the methods described herein, certain relationships are found between images (eg, similar geographic location; similar image metrics; similar metadata, etc.). If these data are generally interactive, and the system finds that during the processing of Image A its color histogram is similar to that of Image B, this information can be stored for later use. If the later processing involves image B, then the initially-stored information may be consulted to find that image A has a similar histogram-without analyzing image B. These relationships are similar to virtual links between images.

In order for this relationship information to maintain its utility over time, it is desirable that the images be identified in a continuous manner. If the relationship is Image A on the user's PDA and Image B is on some desktop, Image A identifies Image A even after it is sent to the user's MySpace account, and Image B is preserved on an anonymous computer in the cloud network. Means for tracking image B should be provided.

Images may be assigned digital object identifiers (DOI) for this purpose. The DOI Foundation has implemented the CNRI Handle System so that such resources can be determined in their current location through the website doi-dot-org. Another alternative is that images are assigned and digitally watermarked with identifiers tracked by the Digimarc For Images service.

If an image or other information is retrieved from several different repositories, it is often desirable to be adapted to use a query for a particular database. For example, different face recognition databases may use different face recognition parameters. To search across multiple databases, techniques such as those described in Digimarc's published patent applications 20040243567 and 20060020630 may be utilized to ensure that each database is examined with a properly tailored query.

Although frequent references to images are made, in many cases other information may be used instead of the image information itself. In different applications, image identifiers, characterization of eigenvectors, color histograms, keypoint descriptors, FFTs, associated metadata, decoded barcode or watermark data, etc. can be used in essence instead of an image (e.g., For example, as a data proxy).

Although the initial example talked about geographic coding by longitude / latitude data, in other arrangements, the cell phone / camera used the location data in one or more other reference systems, such as Yahoo's GeoPlanet ID (WOEID). Can provide.

Location metadata may be used to identify other resources in addition to the similarly-located image. Web pages may have, for example, geographic associations (eg, a blog may be associated with the author's neighborhood; a restaurant's web page is associated with a particular physical address). The web service GeoURL-dot-org is a location-to-URL reverse directory that can be used to identify web sites associated with particular geography.

GeoURL supports a variety of location tags, including their own ICMB meta tags as well as geo tags. Other systems that support geo tagging include RDF, Geo microformats, and GPSLongitude / GPSLatitude tags, which are commonly used in XMP- and EXIF-camera meta information. Flickr uses the syntax established by Geobloggers, for example:

geotagged

geo: lat = 57.64911

geo: lon = 10.40744

In metadata processing, as referenced above, it is sometimes helpful to clean up data prior to analysis. Metadata may also be searched for the prevailing language, and if it is not English (or an implementation of another particular language), metadata and associated images may be removed from consideration.

Initially the above-described embodiments were sought to identify an image object as being one of people / places / objects so that correspondingly different actions could be taken, but analysis / identification of images in other classes may naturally be utilized. Some examples of countless other grade / type groups include animal / vegetable / mineral; Golf / tennis / football / baseball; Male / female; Detected wedding rings / undetected wedding rings; Urban / countryside; Rain / sunny; Day / night; Children / adults; Summer / autumn / winter / spring; Vehicle / truck; Consumer products / non-consumer products; Cans / boxes / bags; Natural / artificial; Suitable for all ages / parental advice for children under 13 / parental advice for children under 17 / adult only; And the like.

Sometimes different analysis engines may be applied to the user's image data. These engines can operate sequentially or in parallel. For example, FIG. 48A shows an arrangement which is then referenced to two different engines-when the image is identified as person-centric. One identifies a person as a family, friend or stranger. The other identifies the person as a child or adult. The latter two engines work in parallel after the first has completed its work.

Sometimes engines may be utilized without any certainty with which they are applicable. For example, FIG. 48B shows engines for performing family / friend / familiar and child / adult analyzes-at the same time person / place / object undertakes analysis. If it is determined that the latter engine may be a place or an object, the results of the first two engines are likely not to be used.

(Specialized online services can be used for distinguishing / identifying certain types of images. For example, one website can provide aircraft recognition services: if an image of the aircraft is uploaded to the site, the identification of the plane. (This technique is described, for example, by Sun in JCIS-2008 Proceedings, The Features Vector Research on Target Recognition of Airplane; and 2003 by Tien, Vol. Using Invariants to Recognize Airplanes in Inverse Synthetic Aperture Radar Images.) The arrangements described herein can refer to these sites an image that appears to be an aircraft and can use the returned identification information. Or all input images can be referenced to these sites; most of the returned results are ambiguous or available Will not be.)

49 shows that different analysis engines can provide their outputs for different response engines. Often different analysis engines and response engines can be operated by different service providers. The outputs from these response engines can then be integrated / adjusted for presentation to the consumer. (This integration can be performed by the user's cell phone-assembling inputs from different data sources; or such work can be performed by a processor elsewhere.)

One example of the technology detailed herein is a home builder that takes a cell phone image of a drill in need of spare parts. The image is analyzed, the drill is identified by the system as the Black and Decker DR250B, and the user is provided with various information / action options. They review photos of drills with similar appearances, reviews photos of drills with similar descriptors / features, review the user's manual for the drill, view the parts list for the drill, find new or ebay from Amazon Buying a drill used from, listing a builder's drill on eBay, buying parts for the drill, and so on. The builder selects the “buy parts” option and proceeds to order the required parts (FIG. 41).

Another example is people who shop at home. She snaps a picture of the house. The system references images in both a secret database of MLS information and a public database such as Google. The system reviews the photos of the closest houses provided for sale; Reviewing the photos of the listed homes for sale that are closest in value to the house in the image and within the same postal code; Reviewing the photos of the listed homes for sale whose home and features are most similar and within the same postal code; Respond with various options, including neighborhood and school information, etc. (FIG. 43).

In another example, the first user snaps an image of Paul Simon at a concert. The system automatically posts the image to the user's Flickr account-along with the metadata inferred by the procedures described above. (The artist's name can be found in Google's search for the user's geographic location; for example, the ticketmaster web page indicates that Paul Simon is performing on stage that night.) Later, they are encountered by a system that processes photos of a second concert-gore of the same event from different advantageous locations. The second user sees the photo of the first user as one of the responses of the system to the second photo. The system may also warn the first user that if he presses a particular button twice, another picture of the same event-from a different point in time-is available for review on his cell phone.

In many such arrangements, one will recognize that "content is a network". Associated with each photo or each object depicted in the photo (or any other item of digital content or information represented therein) acts as an explicit-or explicit-link to actions and other content. Set of data and attributes. The user can navigate from one node to the next-navigating between nodes on the network.

Television shows are rated by the number of viewers, and school newspapers are judged by the number of later quotes. Expressed at a higher level, it will be appreciated that this "viewing rate" for physical- or virtual-content is an entity survey of links that associate it with other physical- or virtual-content.

While Google is limited to the analysis and development of links between digital content, the techniques described herein also allow for the analysis and development of links between physical content (and between physical and electronic content) as well.

Known cell phone cameras and other imaging devices typically have a single "shutter" button. However, the device may be equipped with different actuator buttons-each invoking a different operation with the captured image information. By such an arrangement, the user-upon launch-identifies faces in the image per type of intended behavior (e.g., Picasa or VideoSurf information, posts to my Facebook page; or depicted) Try and identify a person, and send a "requested friend" to that person's MySpace account.

The function of a single actuator button, rather than multiple actuator buttons, may be controlled in accordance with other UI controls on the device. For example, repeated pressing of the function selection button can cause differently intended actions to be displayed on the screen of the UI (as familiar consumer cameras have different photo modes such as close-up, beach, night, portrait, etc.). If the user then presses the shutter button, the selected operation is invoked.

One common response (which may not need to be verified) is posting an image on Flickr or social network site (s). Metadata inferred by the processes detailed herein may be stored with an image (possibly eligible for its trust).

In the past, a "click" of a mouse has been served to trigger a user-desired action. The operation identified X-Y-position coordinates on a virtual landscape (eg, a desktop screen) that showed the user's clear intent. Further, this role will be served gradually by the "snap" of the shutter-capturing the actual landscape from which the user's intention is deduced.

Business roles can dictate the responses appropriate to a given situation. These roles and responses can be determined using intelligent routing, with reference to data collected by web indexers such as Google.

Crowdsourcing is generally not suitable for real-time implementations. However, inputs that interfere with the system and produce no corresponding action (or produce actions that the user selects nothing) can be referenced offline for crowdsource analysis-as a result it is provided next time and better. Can be processed.

Image-based navigation systems provide a different topology that is familiar from web page-based navigation systems. 57A shows that web pages on the Internet are related in a point-to-point manner. For example, web page 1 can be linked to web pages 2 and 3. Web page 3 can be linked to page 2. Web page 2 may be linked to page 4. Etc. 57B shows a contrasting network associated with image-based navigation. The individual images are linked to a central node (eg router), which is then linked to other nodes (eg response engines) in accordance with the image information.

Here, the "router" does not simply route the input packet to the destination determined by the address information carried with the packet-as in the case of familiarity with Internet traffic routers. Rather, the router takes the image information and decides what to do with it, for example, which response system should infer the image information.

Routers can be standalone nodes on the network, or they can be integrated with other devices. (Or the functionality may be distributed between such locations.) A wearable computer may have a router portion (eg a set of software instructions)-it takes image information from the computer and how it should be processed. Decide (For example, if the image information is recognized to be an image of a business card, it may be an OCR name, phone number and other data, which is entered into the contact database.) The specific response to different types of input image information is yes. For example, it may be determined by a kind or other registry database maintained by the computer's operating system.

Similarly, the response engines may be standalone nodes on the network, but they may also be integrated with other devices (or their functions may be distributed). A wearable computer is one that takes action on information provided by the router portion. Or may have several different responses.

52 illustrates an arrangement utilizing several computers A-E, some of which may be wearable computers (eg, cell phones). Computers include common components such as processors, memory, storage, input / output, and the like. Storage or memory may include content such as images, audio and video. Computers may also include one or more routers and / or response engines. Standalone routers and response engines may also be coupled to the network.

The computers are networked and schematically illustrated by the link 150. This connection is software on at least some specific computers, including the Internet and / or wireless links (WiFi, WiMax, Bluetooth, etc.), a peer-to-peer (P2P) client, wherein at least some of the resources of the computer It may be known by any known networking arrangement, including the software that makes it available to other computers and allows the computer to interoperate certain specific resources of other computers.

Through the P2P client, computer A can obtain image, video and audio content from computer B. The sharing parameters on computer B may be set to determine what content is shared and with whom. Dating on computer B, for example, keeps some content confidential; Some content may be shared with known ones (eg, groups of social network “friends”); The remaining content may specify that it can be freely shared. (Other information, such as geographical location information, may also be shared-subject to these parameters.)

In addition to setting sharing parameters based on the party, the sharing parameters may also specify sharing based on content age. For example, content / information older than one year may be freely shared, and content older than one month may be shared with a group of friends (or according to other rule-based restrictions). In other arrangements, fresher content may be of the type that is most freely shared. For example, content captured or stored within a past time, day or week may be freely shared, and content from past months or years may be shared with friends.

The exclusion list may identify content that is treated differently from the rules described above (eg, never shared or always shared) —or one or more ratings of content.

In addition to sharing content, computers can also share their respective router and response engine resources across the network. Thus, for example, if computer A does not have a response engine suitable for a particular type of image information, it can pass the information to computer B for processing by the response engine.

It will be appreciated that this distributed architecture has a number of advantages in terms of reduced cost and increased reliability. Also, "peer" groups can be geographically defined, such as computers that find themselves within a particular spatial environment (eg, an area served by a particular WiFi system). Thus, a peer can dynamically establish ad hoc subscriptions for content and services from nearby computers. If the computer leaves the environment, the session ends.

Some researchers predict that all our experiences will be captured in digital form. In fact, Gordon Bell at Microsoft compiled his digital presence's digital archive through his technologies Cyber All, SenseCam and MyLifeBits. Bell's archive includes records of all phone calls, daily videos, captures of all TV and video watched, archives of all web pages visited, map data of all places visited, polysomnograms during his sleep apnea, and more. have. (For other information, see, for example, A Digital Life by Bell at Scientific American in March 2007; Gemmell, MyLifeBits: A Personal Database for Everything, Microsoft Research Technical Report MSR-TR-2006-23; Gemmell, Passive Capture. and Ensuing Issues for a Personal Lifetime Store, Proceedings of The First ACM Workshop on Continuous Archival and Retrieval of Personal Experiences (CARPE '04), pp. 48-55; May 27, 2007 at The New Yorker, Remember by Wilkinson See also Gordon Bell's Microsoft Research web page and other references cited on the ACM Specific Interest Group web page on CARPE (Capture, Archival & Retrieval of Personal Experiences).

Certain embodiments incorporating aspects of the present technology are well suited for use with such empirical digital content—as input to a system (ie, the system is responsive to a user's current experience), or as metadata, habits and As a resource from which other attributes can be mined (including services in the role of the flicker archive in the embodiments described earlier).

In embodiments that utilize a personal experience as input, it is desirable to initially trigger the system and to respond only when the user wants-rather than running freely limitedly (currently forbidden in terms of processing, memory and bandwidth issues). Do.

The user's desires can be expressed by the intended action by the user, for example, pressing a button or making a gesture with the head or hand. The system takes a date from the current empirical environment and provides candidate responses.

Perhaps more interesting are systems that determine the user's interest through biological sensors. Electroencephalography can be used, for example, to generate a signal that triggers a system's response (or, for example, triggers one of several different responses in response to a different stimulus in the current environment). ). Skin conductivity, pupil dilation and other autonomic physiological responses can also be selectively or electrically sensed and provide a triggering signal to the system.

Eye tracking techniques can be utilized to identify which objects in the field of view captured by the heuristic-video sensor are of interest to the user. If Tony is sitting at the bar and his eyes touch a rare beer bottle in front of a nearby woman, the system can identify his focus interest and focus his own processing efforts on the pixels corresponding to the bottle. Using a signal from Tony, such as two rapid eye-flashes, the system can undertake the effort to provide candidate responses based on the beer bottle-collected from the environment as well as Tony's own personal profile data. You may also be notified by other data (date, date, ambient audio, etc.). (Speech recognition and related technologies are disclosed in, for example, Apple Patent Publication 20080211766.)

The system can quickly identify the beer as Doppelbock by, for example, pattern matching from an image (and / or OCR). Using that identifier, it finds that other resources representing beer originate from Bavaria, which is brewed by Monks of Fran Street in Paola. Its 9% alcohol content is also characteristic.

By checking the personal empirical archives that friends made available to Tony, the system learns that his friend Geoff likes Doppelbock and most recently drank a bottle of alcohol at Dublin's buffs. Oblique encounters with Tony's illness are logged in his own empirical archive, and Geoff can later encounter the same. The oblique encounter can be related to Geoff in Prague in real time, helping to provide on-going data on the activities of his friends.

The bar can also supply an empirical data server, and Tony gives it wirelessly authorized access. The server maintains an archive of digital data captured in the bar and contributed by customers. The server can also be primed with relevant metadata & information, and management can determine which brand will be released in a few weeks, or on what day, his clients, such as the Wikipedia page on Monks' brewing methods in Paul Street. May be considered of interest to (Per user preference, some users require their data to be cleared when they leave the bar; others allow the data to be retained.) Tony's system uses local to know what odd bits of information can be found. You can routinely check your environment's empirical data server. This time, the woman with the female Doppelbock on bar stool 3 shows that Tom of her friends has an encoded final name. Tony's system recognizes that Geoff's friends' cycles (Geoff can make his friends available) contain the same Tom.

A few seconds after his two blinks, Tony's cell phone vibrates on his belt. Flip it back and rotate the roll wheel on the side, and Tony reviews a series of screens that provide information gathered by the system-the information that seems most useful to Tony first.

With knowledge of this Tony-Geoff-Tom connection (typically closer than six degrees of separation) and primed with the trivial things about her Doppelbock beer, Tony walks around the bar with his glass. (Additional details that can be utilized in these arrangements, including user interfaces and visualization techniques, can be found in Dunekacke's "Localized Communication with Mobile Devices" at 2009 MobileHCI.)

While P2P networks such as BitTorrent allow sharing audio, image and video content, the same arrangements as shown in FIG. 52 allow networks to share a richer set in the context of empirical content. The basic concept of P2P networks is that despite the techniques of gathering long tails of content, the majority of users are interested in similar content (score of tonight's NBA game, current episode of lost, etc.), sufficient bandwidth and protocols. Is given, it is the most efficient mechanism for delivering similar content to users by combining content together based on what "neighbors" you have on the network, rather than transmitting individual streams. This same mechanism can be used to provide metadata related to enhancing the experience, such as drinking Dopplebock in a bar or watching the highlights of tonight's NBA game on the phone while in the bar. The protocol used in the ad-hoc network described above may leverage the P2P protocols in actual P2P form or to experience servers that provide peer registration services (similar to the initial P2P networks), All devices advertise what experiences they have available (metadata, content, social connections, etc.), whether free or paid, or bartering the same kind of information. Apple's Bonjour software is well suited for this kind of application.

Within this infrastructure, Tony's cell phones can simply retrieve information about the Dopplebock by posting questions to the peer network and receive a wealth of information from various devices or experience servers in the bar without knowing the source. . Similarly, the experience server can also act as a data-recorder, recording their experiences in an ad-hoc network and providing persistence for the experience at time and place. Geoff can visit the same bar at some point in the future, find out which threads made communications or connections with his friend Tony two weeks ago, or the next time he visits Tony to search for future times in the bar. There is a possibility to leave the notation for.

The ability to collect social threads represented by traffic on the network also allows bar owners to enhance the customer's experiences by coordinating interactions or introductions. This can include people in the form of a game or with shared interests, singles, etc. by allowing people to participate in theme-based games, where customers can unlock secrets (similar to board game clues) or have someone at the bar. Combine the clues together to discover the true identity of the. Finally, demographic information related to audience ratings would be a valuable ingredient for owners when they consider which beers will be stocked next and where they will advertise.

Another discussion

Some portable devices, such as the Apple iPhone, provide a single button for accessing predefined functions. Review pieces of your favorite stocks among these, review weather forecasts, and review a general map of your location. Although additional functions are available, the user must undertake a series of additional operations, for example, to reach his favorite web site and the like.

Embodiments of certain aspects of the present technology allow their other manipulations to be facilitated by capturing unusual images. Capturing an image of the user's hand-linking the user to the BabyCam back home-delivering a real-time video of the baby's manger. An image of a wrist watch can be captured to load a map showing traffic conditions along some portion of the route to the user's driving home, and the like. This function is shown in FIGS. 53-55.

The user interface for the portable device includes a set-up / training step that allows the user to associate different functions with different visual signs. The user is prompted to capture an image and enter a URL and action name associated with the depicted object. (URL is a type of response; others can also be used-such as launching a JAVA application.)

The system then features the snapped image by deriving a set of feature vectors from which similar images can be recognized (eg, via pattern / template matching). Feature vectors are stored in the data structure (Figure 55) in association with the function name and the associated URL.

In this initial training phase, the user can capture several images of the same visual sign-perhaps from different distances and views and with different lighting and backgrounds. The feature extraction algorithm processes the collection to extract a feature set that captures shared similarities of all training images.

Extraction of image features and storage of the data structure may be performed at the portable device or at the remote device (or in a distributed manner).

In later operation, the device verifies each image captured by the device for correspondence with one of the stored visual signs. If something is recognized, the corresponding action can be undertaken. Otherwise, the device responds with other functions available to the user when capturing a new image.

In another embodiment, the portable device is equipped with two or more shutter buttons. Manipulation of one button captures the image and executes the operation-based on the closest match between the captured image and the stored visual code. Operation of other buttons captures the image without undertaking this action.

The device UI may include control to provide the user with signs of the visual term dictionary, as shown in FIG. 54. When activated, thumbnails of different visual signs are provided on the device display in association with the names of the initially stored functions-reminding the user of the defined vocabulary of the signs.

The control of launching a glossary of these symbols can-by itself-be an image. One image that is appropriate for this feature is typically a featureless frame. All dark frames can be achieved by covering the lens and operating the shutter. All bright frames can be achieved by operating the shutter with the lens facing the light source. Another substantially featureless frame (of medium density) can be achieved by imaging a patch of skin or wall or sky. (In order to be substantially non-featured, the frame should be characterized less closely than the matching of one of the other stored visual codes. In other embodiments, a "featureless" means that the image has a texture metric lower than the threshold. It can be concluded that with eggplant.)

(The concept of triggering an operation by capturing all bright frames can be extended to any device function. In some embodiments, all repeated bright exposures alternately toggle on and off the function. All dark and The same is true for frames of medium density, the threshold being set by the user with UI control, or by the manufacturer, to establish how such a frame should be "bright" or "dark" in order to be interpreted as a command. For example, 8-bit (0-255) pixel values from a million pixel sensor may be summed up, if the sum is less than 900,000, the frames may all be considered dark. If large, the frames can all be considered bright.

One of the other featureless frames may trigger another special response. It may allow the portable device to launch all stored functions / URLs (or some particular five or ten, for example) in the glossary. The device caches the resulting frames of information and provides them continuously when the user operates one of the phone controls or makes some particular gesture on the touch screen, such as the button 116b or scroll wheel 124 of FIG. can do. (This function can likewise be called by other controls.)

Third featureless frames (ie, dark, white, or medium-density) may send the device's location to the map server, which may then send back multiple map views of the user's location. have. These views may include aerial views and street map views at different zoom levels, along with the surrounding street-level image. Each of these frames can be cached at the device and quickly reviewed by rotating the scroll wheel or other UI control.

The user interface preferably includes controls for deleting visual signs and editing the name / function assigned to each. URLs can be defined by typing on the keypad or otherwise navigating to a desired destination and then storing the destination as a response corresponding to a particular image.

Training of the pattern recognition engine can continue through use, and successive images of different visual codes each serve to refine the template model in which the visual code is defined.

It will be appreciated that a variety of different visual codes can be defined, generally using the resources available to the user. The hand may define different codes using a finger arranged in different positions (fist, one through five fingers, thumb-forefinger OK sign, open hand, thumb up, US sign language codes, etc.). Clothing and its components (eg shoes, buttons) can also be used when they can be used as ornaments. Features of common peripherals (eg telephone) may also be used.

In addition to launching certain favorite operations, these techniques can be used as user interface technology in other operations. For example, a software program or web service can provide a list of options for a user. For example, rather than manipulating the keyboard to enter selection # 3, the user can capture an image of three fingers-visually sign the selection. The software recognizes the three finger symbols as meaning digit 3 and inputs the values to the process.

If desired, the visual codes may form part of the authentication procedures, for example to access a social-networking website or bank. For example, after entering a name code or password on a site, a user can view the stored image (to confirm that the site is authenticated) and then be prompted to present a particular visual type of image. (As initially defined by the user, but now not explicitly prompted by the site). The website checks the features extracted from the just captured image for correspondence with the expected response before allowing the user to access the website.

Other embodiments may respond to a sequence of snapshots within a certain time period (eg 10 seconds) —grammar of the image. An image sequence of "watch", "four fingers", "three fingers" can set the alarm clock function on the portable device to ring at seven.

In still other embodiments, the visual signs may be gestures that include motion-captured as a sequence (eg, video) of frames by the portable device.

Context data (e.g., indicating the user's geographic location, date, month, etc.) can also be used to tailor the response. For example, when the user is at work, the response to the particular visual code may be to fetch an image from the security camera from the user's home. At home, the answer to the same sign may be to fetch an image from a security camera at work.

In this embodiment, as in others, the response need not be visual. Audio or other outputs (eg, tactile, olfactory, etc.) may of course be utilized.

The technique just described allows the user to define a glossary of visual signs and corresponding custom responses. The intended response can be called quickly by imaging the readily available object. The captured image may be of low quality (eg, overexposed, blurry) as it must be classified between and must be distinguished from a relatively small range of alternatives.

Visual Intelligence Pre-Processing

Another aspect of the present technology is to perform one or more visual intelligence pre-processing operations on image information captured by a camera sensor. These operations can be executed without user request and before other image processing operations that the camera habitually performs.

56 is a schematic diagram illustrating certain specific processes performed in an exemplary camera such as a cell phone camera. The illumination impinges on an image sensor that includes an array of photodiodes. (CCD or CMOS sensor technologies are commonly used.) The resulting analog electrical signals are amplified and converted into digital form by D / A converters. The outputs of these D / A converters provide image data in most raw or "natural" form.

The above-described operations are generally performed by circuitry formed on a common substrate, i.e., "on-chip." One or more other processes are generally executed before other processes access the image data.

One such other operation is Bayer interpolation (mosaic-release). Photodiodes of the sensor array typically only capture light of a single color, respectively: red, green or blue (R / G / B) due to the color filter array. This array consists of a tiled 2 × 2 pattern of filter elements: one red, one blue diagonally opposite, and two other greens. Bayer interpolation effectively "fills" the blanks of the resulting R / G / B mosaic pattern by providing a red signal with a blue filter present.

Another common operation is white balance correction. This process adjusts the contrast of the component R / G / B colors to render certain colors (especially intermediate colors) correctly.

Other operations that can be performed include gamma correction and edge enhancement.

Finally, the processed image data is typically compressed to reduce storage requirements. JPEG compression is most commonly used.

The processed, compressed image data is then stored in the buffer memory. Only at this point image information is generally available to the cell phone's services and other processes (eg, by calling a system API).

One such process, commonly called with this processed image data, is to present the image to the user on the screen of the camera. The user can then evaluate the image, for example: (1) whether to store it in the camera's memory card, (2) whether to send it in a video message, (3) whether to delete it, etc. Determine.

The image remains in the buffer memory until the user instructs the camera (eg, via a button-based user interface or graphical control). Without other instructions, the use of the processed image data is only to display it on the screen of the cell phone.

57 illustrates an example embodiment of a currently discussed aspect of the technology. After converting the analog signals into natural digital form, one or more other processes are performed.

One such process is to perform a Fourier transform (e.g., FFT) on natural image data. This converts the spatial-domain representation of the image into a frequency-domain representation.

Fourier-domain representations of natural image data may be useful in a variety of ways. One is to screen images for likely barcode data.

One familiar 2D barcode is a checkerboard-like array of bright- and dark-squares. The size of the components are squares, so their repetitive spacing provides a pair of prominent peaks in the Fourier-domain representation of the image at the corresponding frequency. (Peaks may be phase-spaced 90 degrees in the UV plane if the pattern is regenerated at the same frequency in both the vertical and horizontal directions.) These peaks extend significantly above the other image components at the surrounding image frequencies. Peaks often have a size two to five times or ten times (or more) of the surrounding image frequencies. If the Fourier transform is performed on patches tiled from an image (e.g., patches of 16 x 16 pixels, or 128 x 128 pixels, etc.), then certain particular patches that are entirely present in the barcode portion of the image frame are characterized by this feature. It can be seen that there is essentially no signal energy except for the normal frequency.

As shown in FIG. 57, Fourier transform information may be analyzed for teltile codes associated with the image of the barcode. Template-type approaches may be used. The template may include a set of parameters for which the Fourier transform information is tested-to see if the data has an indicator associated with a barcode-like pattern.

If the Fourier data matches the image depicting the 2D barcode, the corresponding information can be routed for other processing (eg, sent from the cell phone to the barcode-response service). This information may include natural image data and / or Fourier transform information derived from the image data.

In the former case, the entire image data does not need to be sent. In some embodiments, a down sampled version of the image data, eg, a fourth resolution in both horizontal and vertical directions can be transmitted. Or patches of image data most likely to depict portions of the barcode pattern can be sent. Or conversely, patches of image data that are least likely to depict a barcode can not be sent. (These may be patches that have no peak at characteristic frequencies or that have a lower amplitude there than at ambient.)

The transmission can be prompted by the user. For example, the camera UI may ask the user if information should be directed for barcode processing. In other arrangements, the transmission is dispatched immediately upon determining that the image frame matches a template representing possible barcode data. User actions are not called.

The Fourier transform data can likewise be tested for signs of other image objects. The D barcode can be characterized, for example, by the highest amplitude component at high frequencies-("over the pickets" and over other highest amplitude spikes at low frequencies-along the pickets. As described above, the amplitudes of the surrounding frequencies are averaged again two or more times.) Other image contents can also be characterized with reference to their Fourier domain representation, and corresponding templates can be devised. It is generally used to calculate fingerprints used for automated recognition of media content.

The Fourier-Meline (F-M) transformation is also useful for characterizing various image objects / components-including the barcodes noted above. F-M conversion has the advantage of being robust to scale and rotation (scale / rotation intrusion) of the image object. In an exemplary embodiment, if the scale of the subject increases (such as by moving the camera closer), the F-M conversion pattern moves up; If the scale is reduced, the F-M pattern moves down. Similarly, if the object is rotated clockwise, the F-M pattern moves to the left. (The specific directions of the movements can be tailored depending on the implementation.) These properties make F-M data important in recognizing patterns that can be affine-transformed, such as face recognition, character recognition, object recognition, and the like.

The arrangement shown in FIG. 57 applies a Melin transform to the output of the Fourier transform process to generate F-M data. Thereafter, the F-M may be screened for attributes associated with different image objects.

For example, text is characterized by a plurality of symbols of approximately similar size, consisting of strokes in the foreground color as opposed to a larger background field. Vertical edges tend to predominate (even slightly tilted into italics)-significant energy is also found in the horizontal directions. The spaces between the strokes are generally in a fairly narrow range.

These properties are self evident as features that tend to reliably within certain boundaries in the F-M transform space. Again, F-M data can define tests that are screened to indicate the likely presence of text in the captured natural image data. If the image decides to include a likely-text, it can be dispatched to a service dealing with this type of editor (eg optical character recognition or OCR, engine). Again, an image (or variants of the image) can be sent, transform data can be sent, or some other data can be sent.

Just as the text itself is apparent with a particular set of characteristic attributes in F-M, so do faces. F-M data output from the Melin transformation can be tested against different templates to determine the likely presence of a face with the captured image.

Similarly, F-M data may be examined for Tel-tile codes in which image data carries a watermark. Watermark orientation signals are unique signals present in some watermarks that may serve as the sign in which the watermark exists.

In the example just given, as in others, templates can be compiled by testing with known images (eg, "training"). By capturing images of many different textual provisions, the resulting transform data can be examined for attributes that match across the sample set, or are (most likely) within bounded ranges. These attributes can then be used as a template in which images containing likely text are identified. (Likewise, for other types of faces, barcodes and image objects.)

57 illustrates that various different transformations may be applied to image data. They are generally shown to be executed in parallel, but one or more may be executed sequentially-all of which operate on the same input image data, or one transform is executed using the output of the previous transform (as with the Mellin transform). together). Although not all shown (for the sake of clarity of illustration), outputs from each of the other transform processes may be examined for features suggesting the presence of a particular image type. If found, the relevant data is sent to the appropriate service for that type of image information.

In addition to Fourier transform and melin transform processes, eigenface calculation, image compression, cropping, fine distortion, filtering, DCT transform, wavelet transform, Gabor transform and other signal processing operations may be applied. (All regarded as transformations). Others are well known elsewhere in this specification and are incorporated herein by reference. The outputs from these processes are then tested for features that indicate that the chance that the image depicts a particular grade of information is greater than the random chance.

Outputs from some processes may be input to other processes. For example, the output from one of the ETC labeled boxes in FIG. 57 is provided as input to the Fourier transform process. This ETC box can be a filtering operation, for example. Sample filtering operations may include median, laplacian, winner, sobel, high-pass, low-pass, bandpass, gabor, signum, and the like. (Digimarc's patents 6,442,284, 6,483,927, 6,516,079, 6,614,914, 6,631,198, 6,724,914, 6,988,202, 7,013,021 and 7,076,082 show various such filters.)

Sometimes a single service may handle data passing through different data types or different screens. In FIG. 57, for example, the face recognition service may receive F-M converted data or eigenface data. Or it may receive image information that has passed through one of several different screens (eg, its F-M transform through one screen, or its eigenface representation through a different screen).

In some cases, data may be sent to two or more different services.

Although not essential, it is preferred that some or all of the processing shown in FIG. 57 be performed by circuitry integrated on the same substrate as the image sensors. (Some of the operations may be performed by programmable hardware-on or off board-in response to software instructions.)

Although the operations described above have been described immediately after the conversion of analog sensor signals to digital form, in other embodiments, such operations can be performed after other processing operations (eg, Bayer interpolation, white balance correction, JPEG compression, etc.). ).

Some of the services for which information is transmitted may be provided locally at the cell phone. Or they may be provided by a remote device, which uses the cell phone to establish a link that is at least partially wireless. Or this process can be distributed among the various devices.

(While described in the context of conventional CCD and CMOS sensors, this technique is applicable regardless of sensor type. Thus, for example, Foveon and monochromatic image sensors may alternatively be used. High dynamic range sensor Or sensors using Kodak's TrueSense color filter pattern (adding monochromatic sensor pixels to a typical Bayer array of red / green / blue sensor pixels). For example, sensors that output infrared image data (in addition to visible or invisible image data) are used to identify faces and other image objects with temperature differences—frame It helps to segment the image objects within.)

Devices that utilize the architecture of FIG. 57 will recognize that, in essence, have two parallel processing chains. One processing chain generates data for rendering in perceptual form for use by human viewers. This typically includes at least one of a mosaic-releasing processor, a white balance module, a JPEG image processor, and the like. The second processing chain generates data for analysis by one or more machine-implemented algorithms, and illustrative examples include Fourier transform processors, eigenface processors, and the like.

Such processing architectures are further detailed in the previously cited application 61 / 176,739.

By arrangements as described above, one or more suitable image-response services may begin to formulate candidate responses to the visual stimulus before the user even decides what to do with the captured image.

Additional comments on visual intelligence pre-processing

Although static image pre-processing has been discussed in conjunction with FIG. 57 (and FIG. 50), such processing may also include temporal aspects such as motion.

Movement is most commonly associated with video, and the techniques described herein may be used when capturing video content. However, motion / time implications are also provided with a "stop" image.

For example, some image sensors are read sequentially from the top row to the bottom row. During a read operation, the image object may move within the image frame (ie, due to camera movement or object movement). An aggregated view of this effect is shown in FIG. 60 and depicts the imaging "E" captured when the sensor moves to the left. The vertical stroke of the letter is further from the left edge of the image frame at the bottom than at the top due to the movement of the sensor when the pixel data is clocking out.

This phenomenon also occurs when the camera assembles data from several frames to produce a single "still" image. Although often unknown to the user, many consumer imaging devices quickly capture multiple frames of image data and combine different aspects of the data together (eg, software provided by FotoNation, Inc., now Tessera Technologies, Inc.). Using). For example, the device can take three exposures-one optimizes the appearance of the faces depicted in the image frame, the other is exposed according to the background, and the other is exposed according to the foreground. These are mixed together to create interesting montages. (In another example, the camera captures a burst of frames, and in each, determines whether people are laughing or blinking. Then, different faces can be selected from different frames to produce the final image. )

Thus, the distinction between video and still images is no longer simply a device form, but a user form.

Motion detection can be accomplished in the spatial domain (eg, by referring to the motion of feature pixels between frames) or in the transform domain. Fourier transform and DCT data are exemplary. The system can extract the transform domain signature of the image component and track the motion across different frames-identifying the motion. One exemplary technique deletes the lowest N frequency coefficients, for example-leaving very high frequency edges and the like. (The highest M frequency coefficients can be ignored as well.) Threshold operation is performed on the magnitude of the remaining coefficients-zero below some value (such as 30% of the mean). The resulting coefficients serve as signatures for that image area. (The transformation may be based on tiles of 8 x 8 pixels, for example.) When a pattern corresponding to this signature is found at a position around in another (or the same) image frame (a known similarity test such as correlation) Use), the movement of the image area can be identified.

Image passing of semantic information

In many systems, it is desirable to execute a set of processing steps (such as those described above) that extract information about incoming content (eg, image data) in a scalable (eg, distributed) manner. This extracted information (metadata) is then preferably packaged to facilitate subsequent processing (this may be application specific, more computationally intensive, and may be executed within the originating device or by a remote system). .

The rough analogy is user interaction with Google. Naked search terms aren't sent to the Google mainframe as they come from a missing device. Instead, the user's computer formats the query as an HTTP request, including the originating computer's Internet protocol address (indicating the location), and makes available cookie information for which user language preferences, desired safe search filtering, etc. can be identified. . This structure of relevant information acts as a predecessor to Google's search process, allowing Google to execute the search process more intelligently-providing users with faster and better results.

61 illustrates a portion of metadata that may be relevant in an example system. The types of information in the leftmost column can be calculated directly from the natural image data signals taken from the image sensor. (As noted, some or all of these may be calculated using processing arrangements integrated with the sensor on a common substrate.) The additional information is shown by the information types of the second column, Can be derived with reference to these basic data types. This other information may be generated by processing at the cell phone, or an external service may be utilized (eg, the OCR recognition service shown in FIG. 57 may be in the cell phone, may be a remote server, etc.); Similar to the operations shown in FIG. 50)

How can this information be packaged to facilitate subsequent processing? One alternative is to convey this in the "alpha" channel of common image formats.

Most image formats represent an image by data conveyed in byte-planes or in multiple channels. In RGB, for example, one channel carries red luminance, the second channel carries green luminance, and the third channel carries blue luminance. The same is true for YUV, similar to CMYK (channels carry cyan, magenta, yellow and black information, respectively)-generally video (luma, or brightness, channel: Y, and two color channels: U and V ) And LAB (or brightness with two color channels).

These imaging structures are generally extended to include additional channels: alpha. An alpha channel is provided to convey ambiguity-background objects represent the visible range through the image.

In general, although supported by image processing file structures, software and systems, alpha channels are not used very much (most notably computer generated images and radiation). Certain implementations of the present technology use an alpha channel to transmit information derived from image data.

Different channels of image formats may generally have the same size and bit-depth. In RGB, in general, the red channel can carry 8-bit data for each pixel in a 640 x 480 array (allowing values of 0-255 to be represented). The same is true for the green and blue channels. The alpha channel of this arrangement is also generally 8 bits, and is co-range with the image size (eg 8 bits x 640 x 480). Thus, every pixel has a red value, green value, blue value, and alpha value. (Composite image representation is commonly known as RGBA.)

Some of the many ways in which an alpha channel can be used to convey information derived from image data are shown in FIGS. 62-71 and discussed below.

62 illustrates an image that a user can snap to a cell phone. The processor of the cell phone (on the sensor substrate or elsewhere) may apply an edge detection filter (eg, Sobel filter) on the image data to calculate the edge map. It is determined whether each pixel of the image is part of an edge. So this edge information can only be conveyed in one bit plane of the eight bit planes available in the alpha channel. This alpha channel payload is shown in FIG.

The cell phone camera can also apply known techniques to identify faces within an image frame. Red, green and blue image data corresponding to facial regions can be combined to produce a grey-scale representation, for example in an aligned correspondence with faces identified in the RGB image data. May be included in the alpha channel. An alpha channel carrying two edge information and grayscale faces is shown in FIG. 64. (8-bit grayscale is used for faces in the illustrated embodiment, but shallower bit-depths, such as 6- or 7-bits, are available in different arrangements-freeing other bit planes for other information. Can be.)

The camera may also perform operations to find the positions of the eyes and mouth in each detected face. Markers can be transmitted in the alpha channel-indicating the scale and positions of these detected features. The simple form of the marker is a “smiley face” bit mapped icon, and the icons of eyes and mouth are located at the positions of the detected eyes and mouth. The scale of the face may be indicated by the length of the icon mouth or by the size surrounding the ellipse (or the space between the eye and the markers). The tilt of the face may be indicated by the angle of the mouth (or the angle of the line between the eyes or the tilt surrounding the ellipse).

If cell phone processing yields the determination of the names of the people depicted in the image, this too can be represented in an additional image channel. For example, an ellipse line that delineates the depicted face of a woman can be made in dashed lines or in other patterns. The eyes may be represented by crosshairs or Xs or the like instead of darkened circles. The ages of the depicted people may also be approximated and displayed similarly. The treatment may also classify each person's emotional state by visual facial cues, and an indication such as surprise / happy / sadness / angry / ambiguity may be expressed (eg, Proceedings, Queensland, 2007, 2007). See “A simple approach to facial expression recognition” on pages 456-461 of the 2007 Int'l Conf on Computer Engineering and Applications, see also Patent Publications 20080218472 (Emotiv Systems, Pty), and 20040207720 (NTT DoCoMo). do.)

When a decision has some uncertainty (such as guessing gender, age group, or emotion), the confidence metric output by the analysis process is also represented in an iconic manner, such as by the width or selection of line width or pattern elements. Can be.

65 illustrates different pattern elements that can be used to display different information in the auxiliary image plane, including gender and confidence.

The portable device may also perform peaked operations in optical character recognition of alphanumeric symbols and strings depicted in the image data. In the illustrated example, the device can recognize the string "LAS VEGAS" in the picture. This decision can be stored by a PDF417 2D barcode added to the alpha channel. The barcode may be at the location of the OCR'd text in the image frame or elsewhere.

(PDF417 is exemplary only. Other barcodes-such as ID, Aztec, Datamatrix, high capacity color barcodes, Maxicode, QR code, Semacode, and ShotCode-or other machine-readable data symbols-OCR fonts and data patterns Patterns can be used both to convey arbitrary data and also to form halftone image descriptions, in this regard, patents 6,419,162 to Xerox and IEEE Computer Magazine 3rd 2001 See "Printed Embedded Data Graphical User Interfaces" by Hecht, Vol. 34, pp. 47-55.)

66 shows an alpha channel representation of a portion of the information determined by the device. All these information are organized in a manner that allows them to be conveyed within a single bit plane (of 8 bit planes) of the alpha channel. Information derived from another of the processing operations (eg, the analyzes shown in FIGS. 50 and 61) can be conveyed in this same bit plane or in other bit planes.

62-66 illustrate various information that can be conveyed in the alpha channel and its different representations, more are shown in the example of FIGS. 67-69. These involve the new GMC truck and the owner's cell phone burn.

Among other processes, the cell phone of this example can recognize the model, age and color of the truck, recognize the text on the truck grill and the owner's t-shirt, recognize the owner's face, and recognize the full and sky areas. Image data was processed.

The sky is a region of the top of the frame, by a color histogram within the expected distance of norms, by a weak spectral configuration at certain frequency coefficients (e.g., substantially "flat" region). A) was recognized. Pools were recognized by textures and colors. Other techniques for recognizing these features are described by Batlle in, for example, May 2000, Image and Vision Computing, Volume 18, pages 6-7, 515-530, in "A review on strategies for recognizing natural objects in color images of outdoor scenes ”;“ Fast Labeling of Natural Scenes Using Enhanced Knowledge ”by Hayashi in Pattern Analysis & Applications Volume 1, Volume 4, pages 20-27, March 2001; and IEEE Int'l Conf. on Multimedia and July 2005 Expo is disclosed by Boutell in “Improved semantic region labeling based on scene context.” See also Patent Publications 20050105776 and 20050105775 (Kodak). The trees could be recognized similarly.

The human face of the image was detected using the same arrangements as those commonly used in consumer cameras. Optical character recognition was performed on the data set resulting from Fourier and Mellyn transforms after application of the edge detection algorithm following the input image. (The texts GMC and LSU TIGERS were found, but the algorithm did not identify other text on the t-shirt and text on the tires. With additional processing time, some of this missing text could be decoded.)

The truck was first classified as a vehicle, then as a truck and then finally, by pattern matching, as a Dark Crimson Metallic 2007 GMC Sierra Z-71 with an extended steering wheel. (This above-described identification was obtained through the use of known reference truck images from resources such as the fan site: IMCDB-dot-com, faithful to identifying vehicles on the GM trucks website, flicker and Hollywood motion pictures.) And other ways of model recognition in 2009 Proc. This is described in detail in "Localized Contourlet Features in Vehicle Make and Model Recognition" by Zafar in SPIE, vol.

FIG. 68 shows an example graphical, bitonal representation of differentiated information when added to the alpha channel of the FIG. 67 image. (Figure 69 shows the different planes of the composite image: red, green, blue and alpha.)

Portions of the image are detected when the depicted pool is represented by a uniform image of the points. The image region depicting the sky is represented as a grid of lines. (If the trees were specifically identified, they could be labeled using one of the same patterns, but they could be of different sizes / spacings / etc. Or completely different patterns could be used.)

The identification of the truck with the Dark Crimson Metallic 2007 GMC Sierra Z-71 with an extended steering wheel is encoded in the PDF417 2D barcode-scaled to the size of the truck and masked by its shape. Because PDF417 encodes information redundantly with error-correcting features, the portions of the rectangular barcode that are damaged do not prevent the encoded information from being recovered.

Face information is encoded in a second PDF417 bar code. This second barcode is oriented 90 degrees relative to the truck barcode and scaled differently to help distinguish two separate symbols for downstream decoders. (Other different orientations could be used, in some cases, for example, 30 degrees, 45 degrees, etc. is preferred.)

Face barcodes are oval shaped and can be outlined with an elliptic rim (this is not depicted). The center of the barcode is placed at the midpoint of the human eyes. The width of the barcode is twice the distance between the eyes. The height of the ellipse barcode is four times the distance between the lines joining the mouth and eyes.

The payload of the face barcode carries the information identified from the face. In embodiments, the barcode simply indicates the presence of a face. In more sophisticated embodiments, the eigenvectors calculated from the face image may be encoded. If a particular face is recognized, information identifying the person can be encoded. The processor determines about the likely gender of the subject and this information can also be conveyed in the barcode.

Those who appear in the images captured by consumer cameras and cell phones are not random: significant proportions are reoccurring objects, ie owner children, spouses, friends, users themselves. There are often many previous images of these reoccurring objects distributed between devices owned or used by the owner, such as a PDA, cell phone, home computer, network storage, and the like. Many of these images are annotated with the names of the people depicted. From these reference images, settings characterizing face vectors can be calculated and used to identify objects in new photos. (As noted, Google's Picasa service works on this principle to identify people in a user's photo collection; facebook and iPhoto are the same.) This library of reference face vectors is shown in the photo of FIG. The identification may be verified to attempt and identify the person depicted, and the identification may be represented in a barcode. (The identification may include the person's name and / or other identifier (s), whereby the matched face may be, for example, an index number, phone number, Facebook user name, etc., in a database or contact list. Known.)

The text recognized from the regions of FIG. 67 is added to the corresponding regions of the alpha channel frame and provided in a reliably decodable OCR font. (OCR-A is depicted, but other fonts may be used.)

Various other information may be included in the FIG. 68 alpha channel. For example, there are positions in the frame where the processor suspects text, but OCR did not successfully decode alphanumeric symbols (probably on tires or other characters on a person's shirt), and the corresponding visual cue. By adding (eg, a pattern of diagonals). The contour of the person (rather than the representation of his face) can also be detected by the processor and indicated by the corresponding border or filling pattern.

Although the examples of FIGS. 62-66 and 67-69 illustrate various different ways of representing semantic metadata in an alpha channel, more techniques are shown in the examples of FIGS. 70 and 71. Here, the user captured a snapshot of the playing child (FIG. 70).

The child's face is turned away from the camera and captured with poor contrast. However, even with this limited information, the processor makes a likely identification with reference to the user's previous images: the user's first born child Matthew Doe (likely to be found in the preserved photos of countless users).

As shown in FIG. 71, the alpha channel of this example conveys an edge-detected version of the user's image. What is imposed on the child's head is the child's face replaced image. This alternate image can be selected for its construction (eg, depicting two eyes, nose and mouth) and better contrast.

In some embodiments, each person known to the system has an icon face image that acts as a visual proxy for the person in different contexts. For example, some PDAs store contact lists that include face images of the contacts. The user (or contacts) provide facial images that are easily recognized-with an icon. These icon face images can be scaled to match the head of the person depicted in the image and added to the alpha channel at the corresponding face position.

In addition, the alpha channel depicted in FIG. 71 includes a 2D barcode. This barcode can convey or be available the rest of the information distinguished from the processing of the image data (e.g. the child's name, color histogram, exposure metadata, how many faces were detected in the image, the ten largest DCTs) Or other transform coefficients, etc.).

In order to make the 2D bar code as powerful as possible to compression and other image processing operations, the size is not fixed but rather can be dynamically scaled based on circumstances-such as image characteristics. In the depicted embodiment, the processor analyzes the edge map to identify areas with uniform edgeness (ie, within a critical range). The largest such area is selected. The barcode is then scaled and placed to occupy a central area of this area. (In subsequent processing, the edgeness with the barcode replaced can be largely recovered by averaging the edgeness at the center points adjacent to the four barcode sides.)

In another embodiment, the area size is adjusted with edgeness to determine where to place the barcode: low edgeness is good. In this alternative embodiment, smaller areas of lower edgeness may be selected through larger areas of higher edgeness. The subtracted edgeness of the scaled value in each candidate region can serve as a metric to determine which region should host the barcode. This is the arrangement used in FIG. 71, leading to the placement of the barcode in the left region of Matthew's head-rather than the region for the larger but more edged right.

While Figure 70 is relatively "edge" (as opposed to, for example, the Figure 62 photo), most edgeness may be irrelevant. In some embodiments, the edge data is filtered so that only major edges (eg, edges indicated by continuous line contours) are preserved. Within the blank areas of the resulting filtered edge map, the processor can convey additional data. In one arrangement, the processor inserts a pattern to indicate a particular color histogram bin with the person's image colors. (In a 64-bin histogram that requires 64 different patterns, bin 2 has a red channel value of 0-63, a green channel value of 0-63, a blue channel value of 64-127, and so on. May include colors). Other image metrics may be conveyed similarly.

Instead of using different patterns to represent different data, blank areas of the filtered edge map can be filtered with a noise-like signal-steganographically to convey histograms (or other information) as digital watermark data. Is encoded. (Suitable watermarking techniques are detailed in Digimarc's patent 6,590,996.)

It will be appreciated that some information in the alpha channel-when presented visually to humans in graphical form-conveys useful information. From FIG. 63, a person can identify a male who embraced a female before a sign that says "WELCOME TO Fabulous LAS VEGAS NEVADA". From FIG. 64, a person can see the grayscale faces and the outline of the scene. From FIG. 66, a person may additionally identify a barcode carrying some information and identify two smiley face icons showing the locations of the face.

Similarly, in FIG. 68, a viewer, in which a frame of graphical information can be rendered, can identify the outline of a person, read LSU TIGERS from a person's t-shirt, and find out what appears in the outline of the truck (of the truck). Assisted by clues in GMC text with draw).

From the provision of the alpha channel data of FIG. 71, a person can identify a child sitting on the floor playing with toys.

The bar code of FIG. 71, like the bar code of FIG. 66, is prominently displayed to the person investigating the existence of the information, but the content thereof is not shown.

The rest of the graphic content in the alpha channel may not be beneficial to humans upon investigation. For example, if a child's name is encoded steganographically as a digital watermark in the noise-like signal in FIG. 71, the presence of information that may be noisy may not be detected by the person.

The examples described above detail some of the diversity of semantic information that can be stuffed into an alpha channel and the diversity of representational structures that can be utilized. Of course, this is just a small sampling; An artist can quickly adapt these disclosures to the needs of specific applications, creating many other different embodiments. Thus, for example, any information that can be extracted from an image can be stored in an alpha channel using arrangements similar to those disclosed herein.

It will be appreciated that image related information may be added to the alpha channel at different locations by different processors at different times. For example, a sensor chip in a portable device may perform on-chip processing to perform specific analyzes and add the resulting data to the alpha channel. The device may have another processor-for the image data and / or for the results of the initial analyzes-that performs additional processing and adds a representation of these additional results to the alpha channel. (These additional results may be based in part on data obtained wirelessly from a remote source. For example, a consumer camera may be linked by Bluetooth to a user's PDA to obtain facial information from the user's contact files. have.)

The composite image file can be sent from the portable device to an intermediate network node (e.g., to a carrier such as Verizon, AT & T, or T-Mobile, or to another service provider), which executes further processing and outputs the result to alpha. Add to channel (With more capable processing hardware, these intermediate network nodes perform more complex, resource-intensive processing-such as more sophisticated face recognition and pattern matching.) With higher-bandwidth network access, these nodes Can utilize various remote resources to augment the alpha channel with additional data, such as links to Wikipedia entries-or information from Wikipedia content itself, phone database and image database lookups.) The image may then be sent to an image query service provider (eg, SnapNow, MobileAcuity, etc.), which may continue processing and / or instruct a response operation based on the information so provided.

The alpha channel can thus convey an icon view of what all ongoing processing has identified the image and learned about it. Each subsequent processor can easily access this information and contribute more. All of this is within the constraints of existing workflow channels and long established file formats.

In some embodiments, the source of some or all of the distinguished / inferred data is indicated. For example, the stored data can be the OCR generating the specific text, the MAC address of 01-50-F3-83-AB-CC or PDX-LA002290.corp.verizon-dot at 8:35 pm on August 28, 2008. It may indicate that execution was performed by a Verizon server having a unique identifier, such as a network identifier of -com. Such information may be stored in the alpha channel, in the header data, in remote storage or the like provided with a pointer.

Different processors may contribute to different bit-planes of the alpha channel. The capture device can write the information to bit plane # 1. The intermediate node may store its contributions in bit plane # 2. Certain bit planes may be available for shared use.

Or different grades or types of semantic information may be assigned to different bit planes. Information related to faces or people in the image can always be recorded in bit plane # 1. It can always be written to information bit plane # 2 related to the places. Edge map data can always be found in bit plane # 3 along with color histogram data (eg, represented in the form of a 2D barcode). Other content labeling (eg grass, sand, sky) can be found in bit plane # 4 with OCR'd text. Information of the text, such as content of the text obtained from the web or related links, may be found in bit plane # 5. (ASCII symbols can be included as bit patterns, for example each symbol takes 8 bits in the plane. Robustness for subsequent processing is improved by assigning two or more bits to the image plane for each bit of ASCII data. Convolutional coding and other error correction techniques may be utilized for some or all of the image plan information, again with error correction barcodes.)

The index to the information carried in the alpha channel can be compiled, for example, in the EXIF header associated with the image, allowing subsequent systems to quickly interpret and process this data. The index utilizes XML-type tags that specify the types of data conveyed in the alpha channel and optionally other information (eg, their locations).

The locations can be specified as the location of the most significant bit (or most significant bit) in the bit-plane array, eg, X-, Y-coordinates. Alternatively, a rectangular bounding box may be specified with reference to two corner points (eg, designated by X, Y coordinates)-detailing the region in which the information is represented.

In the example of FIG. 66, the index may convey information as:

<MaleFace1> AlphaBitPlane1 (637,938) </ MaleFace1>

<FemaleFace1> AlphaBitPlane1 (750,1012) </ FemaleFace1>

<OCRTextPDF417> AlphaBitPlane1 (75,450)-(1425,980) </ OCRTextPDF417>

<EdgeMap> AlphaBitPlane1 </ EdgeMap>

This index is thus found in bit plane # 1 of the alpha channel where the top pixel is at positions 637 and 938; The female face is similarly represented in the upper pixel located at 750, 1012; OCR'd text encoded as a PDF417 barcode is found in bit plane # 1 of a rectangular region with corner points 75,450 and 1425,980, which bit plane # 1 also includes an edge map of the image.

Some information may naturally be provided. Different types of indexes with less information can be specified, for example:

<AlphaBitPlane1> Face, Face, PDF417, EdgeMap </ AlphaBitPlane1>

This form of index may simply indicate that bit plane # 1 of the alpha channel includes two faces, a PDF417 barcode and an edge map.

The index with more information is the rotation angle and scale factor for each face, the LAS VEGAS payload of the PDF417 barcode, the angle of the LAS VEGAS barcode, the confidence factors for subjective decisions, the names of recognized people, the alpha channel. Vocabulary or glossary (e.g., patterns in FIG. 65 and graphic labels used in the sky and grass in FIG. 68) detailing the significance significance of each pattern used in the For example, data may be specified, such as the face of a duplicated child in FIG. 71, or remote reference image data served as the basis for the conclusion that the truck is Sierra Z71 in FIG. 67.

As can be seen, the index can carry information that is also conveyed in the bit planes of the alpha channel. In general, different forms of representation are used in the index versus the graphical representations of the alpha channel. For example, in the alpha channel, the femininity of the second face is represented by '+' to express eyes; In the index, femininity is represented by the XML tag <FemaleFace1>. The redundant representation of the information serves as a check for data integrity.

Sometimes, header information such as EXIF data may be separated from the image data (eg, when the image is delivered in a different format). Instead of conveying index information in the header, the bit plane of the alpha channel may serve to convey index information, for example bit plane # 1. One such arrangement encodes index information as a 2D barcode. The barcode can be scaled to fill the frame to provide maximum robustness to possible image degradation.

In some embodiments, some or all of the index information is replicated in different data stores. For example, both may be delivered in the form of an EXIF header and as a barcode in bit plane # 1. Some or all of the data may also be maintained remotely, such as by Google or other web storage "in the cloud." The address information carried by the image can serve as a pointer to this remote store. A pointer (which can be a URL, but more generally-when required-is a UID or index into the database that returns the current address of the data sought) can be included in the index and / or in one or more bit planes of the alpha channel. . Alternatively, the pointer may be steganographically encoded in pixels (in some or all of the composite image planes) of the image data using digital watermarking techniques.

In still other embodiments, some or part of the information described above as stored in the alpha channel can additionally or alternatively be remotely stored or encoded in image pixels as a digital watermark. (The image itself may or may not be duplicated in remote storage, by any device in the processing chain, with or without an alpha channel.)

Some image formats may include more than the four planes described above. Geospatial imagery and other mapping techniques typically represent data in formats that extend to more than half a dozen or more information planes. For example, multispectral space-based images may be applied to (1) red, (2) green, (3) blue, (4) near infrared, (5) mid-infrared, (6) far infrared, and (7) thermal infrared. It can have individual image planes engrossed. The techniques described above can convey derived / inferred image information using one or more auxiliary data planes available in these formats.

As the image moves between processing nodes, some of the nodes may overwrite the data inserted by the initial processing. Although not required, the overwrite processor can copy the overwritten information to the remote storage and include a link or other reference to it in the image or index or alpha channel-necessary in the latter case.

When presenting information in an alpha channel, consideration may be given to the degradations that this channel may experience. JPEG compression, for example, generally discards high frequency details that do not contribute significantly to human perception of the image. However, this revocation of information based on the human visual system can serve as disadvantages when applied to information that exists for other purposes (although a human view of the alpha channel is clearly possible and useful in some cases).

In an effort to eliminate this degradation, the information in the alpha channel may be represented by features that are unlikely to be considered visually irrelevant. Different types of information can be represented by different features, so the most important one persists even through strict compression. Thus, for example, the presence of faces in FIG. 66 is indicated by bold ovals. The positions of the eyes may be less relevant and are represented by smaller features. The patterns shown in FIG. 65 may not be reliably distinguished after compression, so they may be reserved to represent secondary information where loss is less important. Using JPEG compression, the top bit-planes are best preserved, while the lower top bit-planes are progressively error-prone. Thus, the most important metadata is carried in the most significant bit planes of the alpha channel-to improve viability.

If the kind of technique shown by Figs. 62-71 becomes a common language for carrying metadata, image compression will evolve to take into account its presence. For example, JPEG compression may be applied to the red, green and blue image channels, while lossless (or low loss) compression may be applied to the alpha channel. Because the alpha channel of the various bit planes can carry different information, they can be compressed separately-rather than as bytes of 8-bit depth. (If compressed separately, lossy compression may be more acceptable.) Each bit-plane carries only non-tonal information, including modified Huffman, modified READ, run length encoding, and ITU-T T.6. Compression schemes known from facsimile techniques can be used. Thus, hybrid compression techniques are well suited for these files.

Alpha channel delivery of metadata may be configured to progressively transmit and decode correspondingly to the associated image features, using compression arrangements such as JPEG 2000. That is, since the alpha channel provides semantic information in the visual domain (eg, with an icon), it can be expressed to decompress layers of semantic detail at the same rate as the image.

In JPEG 2000, wavelet transform is used to generate data representing an image. JPEG 2000 packages and processes this transformed data in a way that produces progressive transmission and decoding. For example, when rendering a JPEG 2000 image, the overall details of the image appear first, followed by more fine details. The same is true for transmission.

Consider the truck and male image of FIG. 67. The JPEG 2000 version of this renders the first rendering of a low frequency thick line truck. Then, the shape of the male appears. Next, features such as GMC letters on the truck grille and logos on men's t-shirts are distinguished. Finally, male facial features, grass, details of trees, and other high frequency finishers complete the rendering of the image. The same is true for transmission.

This progress is shown in the pyramid of FIG. 77A. Initially a relatively small amount of information is represented by providing details of the overall shape. Gradually, the image is filled inside-finally ending with a relatively large amount of small detailed data.

The information of the alpha channel may be similarly configured (FIG. 77B). Information about the truck can be represented by large, low frequency (shape-dominant) symbols. The presence and position of the male may be encoded in the next-most-dominant expression. The information corresponding to the GMC letters on the truck grille and the letters on the men's shirts can be represented in the alpha channel in fine detail. The finest degree of detail in the image, for example the facial facial of a male, can be represented in the finest detail in the alpha channel. (As may be noted, the exemplary alpha channel of FIG. 68 does not follow this model very much.)

If the alpha channel conveys its information in the form of machine-readable symbols (eg, barcodes, digital watermarks, glyphs, etc.), the order of the alpha channel decoding can be deterministically controlled. Features with the largest features are decoded first; Features with the finest features are decoded last. Thus, the alpha channel can carry barcodes at several different sizes (all in the same bitframe, eg, located side by side or distributed between bit frames). Or the alpha channel may carry a plurality of digital watermark signals, for example one in overall resolution (eg, corresponding to ten watermark elements or "waxels" in inches, others Continuously at finer resolutions (eg 50, 100, 150 and 300 wax cells per inch) The same is true for data glyphs: a range of larger and smaller sizes of glyphs can be used, which are relatively more It will be decoded early or later.

(JPEG2000 is the most common compression scheme that exhibits gradual behavior, but there are others. Some laborious JPEGs can behave similarly. These concepts are applicable whenever such graduality exists.)

Such arrangements make corresponding metadata available when image features are decoded for provision—or when transmitted (eg, by media delivery streaming).

It will be appreciated that the results contributed to the alpha channel by the various distributed processing nodes are readily available for each subsequent reception of the image. Thus, a service provider receiving the processed image may, for example, depict a male and a female in FIG. 62 depicting Las Vegas; 63 shows a male and his GMC truck; It is quickly understood that Figure 70 shows a child named Matthew Doe. Edge maps, color histograms, and other information conveyed with these images provide a headstart to the service provider in the processing of the image, for example to augment it, recognize its content, and initiate an appropriate response.

Receiving nodes may also use the transferred data to enhance stored profile information related to the user. The node receiving the metadata of FIG. 66 may mark Las Vegas as a location of potential interest. The system for receiving the metadata of FIG. 68 may infer that the GMC Z71 trucks are related to the user and / or the person depicted in the photo. These associations can serve as launch points for tailored user experiences.

The metadata also allows images with certain attributes to be quickly identified in response to user queries. (For example, find photos showing GMC Sierra Z71 trucks.) Preferably, web-indexing crawlers can see the alpha channels of the images they find on the web and make the images easier to identify to searchers. You can add information from the alpha channel to the compiled index.

As noted, an alpha channel-based approach is not necessary for the use of the techniques detailed herein. Another alternative is a data structure indexed by the coordinates of the image pixels. The data structure can be passed with the image file (such as EXIF header data, for example) or stored on a remote server.

For example, one entry in the data structure corresponding to pixels 637 and 938 in FIG. 66 may indicate that the pixels form part of a male's face. The second entry for this pixel may point to a shared sub-data structure in which the eigenface values for this face are stored. (The shared sub-data structure may also list all pixels associated with that face.) A data record corresponding to pixels 622,970 may indicate that the pixel corresponds to the left eye of the male's face. The data records indexed by pixels 155 and 780 may indicate that the pixels form part of the text that is recognized as spelled "L" (by OCR) and are also in color histogram bin 49. The source of each datum of information can also be recorded.

(Instead of identifying each pixel by X- and Y-coordinates, each pixel may be assigned a referenced sequential number.)

Instead of several pointers pointing to a common sub-data structure from data records of different pixels, the entries may form a linked list, where each pixel has a common attribute (eg, associated with the same face). Contains a pointer to the next pixel. The record for the pixel may include a plurality of different sub-data structures or pointers to a plurality of other pixels—to associate the pixel with a plurality of different image features or data.

If the data structure is stored remotely, a pointer to remote storage may be included with the image file, for example, steganographically encoded in the image data and represented as EXIF data. If any watermarking arrangement is used, the origin of the watermark (see Digimarc's patent 6,307, 949) may be used as the basis for which pixel references are specified as offsets (eg, instead of using the upper left corner of the image). This arrangement allows the pixels to be correctly identified despite errors such as cropping or rotation.

Like alpha channel data, metadata recorded in remote storage is preferably available for retrieval. A web crawler that encounters an image may identify a corresponding repository of metadata and add a metadata to the index terms for the image from that repository (even if found at different locations), such as a steganographically encoded watermark or You can use pointers in EXIF data.

By means of the arrangements described above, it will be appreciated that existing image standards, workflows and ecosystems-originally designed to support the pixel image data-are utilized here in the support of metadata as well.

(Of course, the alpha channel and the other ways described above in this section are not essential to other aspects of the present technology. For example, derived from processes such as those shown in FIGS. 50, 57 and 61 or Inferred information can be dispatched as packetized data using, for example, WiFi or WiMax, transmitted from the device using Bluetooth, transmitted as SMS short text or MMS multimedia messages, or a low power peer-to-peer Shared with other nodes in a wireless network, communicated with a wireless cellular transmission other transmission or wireless data service, or the like).

Texting  Etc

U.S. Patents 5,602,566 (Hitachi), 6,115,028 (Silicon Graphics), 6,201,554 (Ericsson), 6,466,198 (Innoventions), 6,573,883 (Hewlett-Packard), 6,624,824 (Sun) and 6,956,564 (British Telecom), and published 63 PCTPhilips WO98P ) Discloses that portable computers can be equipped with devices in which tilting can be sensed and used for different purposes (eg, scrolling through menus).

According to another aspect of the present technology, a tip / tilt interface is used in connection with a typing operation, such as constructing text messages sent by a simple message service (SMS) protocol from a PDA, cell phone or other portable wireless device.

In one embodiment, the user activates the tip / tilt text input mode using any of a variety of known means (eg, pressing a button, entering a gesture, etc.). The scrollable user interface appears on the device screen providing a series of icons. Each icon has the appearance of a cell phone key, such as a button depicting the number "2" and the letter "abc". The user tilts the device left and right to scroll backward or forward through a series of icons to reach the desired button. The user then tips the devices towards or away from themselves to navigate between the three letters associated with the icon (eg, tipping to navigate away from “a”; corresponding to “b”). No tipping; tipping to navigate towards "c"). After navigating to a desired letter, the user takes an action to select that letter. This action may press a button on the device (eg with a user's thumb), or another action may signal a selection. The user then proceeds as described for selecting subsequent letters. By this arrangement, the user enters a series of text without the constraints of large fingers on small buttons or UI features.

Many variations are of course possible. The device does not need to be a phone; It can be a watch, keyfob, or other small form factor.

The device may have a touch-screen. After navigating to the desired character, the user can tap the touch screen to make a selection. When tipping / tilting the device, the corresponding letters may be displayed on the screen in an enlarged manner to indicate the user's progress in navigation (eg, overlaid on or over an icon representing the button). ).

Accelerometers or other physical sensors have been utilized in certain embodiments, while others use 2D optional sensors (eg, cameras). The user may direct the sensor to the floor, to the knees, or to another object, and the device then detects the relevant physical movement by sensing the movement (up / down; left and right) of the features in the image frame. In such embodiments, the image frame captured by the camera need not be provided on the screen; The symbol selection UI may be displayed alone. (Or the UI may be provided as an overlay on the background image captured by the camera.)

In camera-based embodiments, as in embodiments utilizing physical sensors, other dimensions of motion may also be sensed: up / down. This may provide an additional degree of control (eg shifting to uppercase letters, shifting from characters to numbers, or selecting the current symbol, etc.).

In some embodiments, the device has several modes: one for entering text; The other for entering numbers; The other for inputting symbols; Etc. The user can switch between these modes using mechanical controls (eg buttons) or via controls of the user interface (eg touches or gestures or voice commands). For example, tapping the first area of the screen can select the currently displayed symbol and tapping the second area of the screen can toggle the mode between character input and numeric input. Or one tap in this second area can switch to character input (default); Two taps in this area can switch to numeric input; And three taps in this area can switch to entry of other symbols.

Instead of selecting between individual symbols, this interface may also include common words or phrases (eg, signature blocks), in which the user can tip / tilt navigate and then You can choose. There may be several lists of words / phrases. For example, the first list can be standardized (pre-programmed by the device vendor) and statistically include common words. The second list may include words and / or phrases associated with a particular user (or a particular class of users). The user can enter these words in this list, or the device can compile the list during operation-to determine which words are most commonly entered by the user. (The second list may or may not exclude words found on the first list.) Again, the user can switch between these lists as described above.

Preferably, the sensitivity of the tip / tilt interface is adjustable by the user to accommodate different user preferences and techniques.

Although the embodiments described above have considered tilts / tips of limited grammar, more extended grammars can be devised. For example, tilting the screen relatively slowly to the left can cause the icons to scroll in a given direction (left or right depending on the implementation), and sudden tilting of the screen in that direction can cause breaks in the line (or paragraph) in the text. As you insert-you can perform different actions. A sharp tilt in the other direction can cause the device to send a message.

Instead of the speed of tilt, the angle of tilt may correspond to different actions. For example, tilting the device at 5 to 25 degrees can cause icons to scroll, but tilting the device above 30 degrees can insert a line break (if left) or have a message delivered (if right). ).

Different tip gestures may likewise trigger different actions.

The arrangements just described are only a few of the many different possibilities. Those skilled in the art that employ this technology are expected to modify and adapt these disclosures as appropriate for their particular applications.

Affine  Capture parameters

According to another aspect of the present technology, a portable device may capture and provide geometric information related to the location (or location of the object) of the device.

Digimarc's published patent application 20080300011 discloses various arrangements in which a cell phone can respond in response to "seeing", including overlaying graphical features on top of certain imaging objects. The overlay can be wrapped according to the perceived affine distortion of the object.

Steganographic correction signals for which the affine distortion of an imaging object can be accurately quantized are described, for example, in Digimarc's patents 6,614,914 and 6,580,809; And patent publications 20040105569, 20040101157, and 20060031684. Digimarc's patent 6,959,098 discloses how distortion can be characterized by such watermark correction signals along with visible image features (eg, edges of a rectangular object). From this affine distortion information, the 6D position of the watermarked object relative to the imager of the cell phone can be determined.

There are various ways in which 6D location can be described. One is by three position parameters: x, y, z, and three angle parameters: tip, tilt, rotation. The other is by rotation and scale parameters along with the 2D metrics of the four elements defining the linear transformation (eg shear mapping, translation, etc.). The matrix transforms the position of any pixel x, y into the resulting position after the linear transformation has occurred. (The reader refers to references to shear mapping, for example Wikipedia, for information about the matrix mass.)

58 illustrates how a cell phone can display affine parameters (eg, derived from an image or elsewhere). The camera may be placed in this mode through UI controls (eg, tapping physical buttons, making touchscreen gestures, etc.).

In the depicted arrangement, the rotation of the device from the (obvious) horizontal direction is provided on top of the cell phone screen. The cell phone processor can make this determination by analyzing image data for one or more generally parallel long edge features, averaging them to determine an average, and assuming it is horizontal. If the camera is typically aligned with the horizontal, this average line will be horizontal. The divergence of this line from the horizontal indicates the camera's rotation. This information may be provided in text (eg, "12 degrees right"), and / or a graphical representation showing divergence from the horizontal may be utilized.

(Other means for sensing angular orientation may be utilized. For example, many cell phones include accelerometers or other tilt detectors, which output data in which the cell phone processor can distinguish the angular orientation of the device. do.

In the illustrated embodiment, the camera captures a sequence of image frames (eg video) when in this mode of operation. The second datum represents the angle at which features in the image frame have been rotated since image capture began. Again, this information can be collected by analysis of the image data and provided in text and / or graphically. (The graphic can include a circle-or an arrow-with a line through the center showing real-time angular movement to the left or right of the camera.)

In a similar manner, the device can track changes in the apparent size of the edges, objects and / or other features in the image to determine the amount of scale change since image capture began. This shows how far the camera has been moved towards or away from the object and how much has been moved. Again, the information can be provided in text and graphically. The graphical representation may include two lines: a reference line, and a second parallel line whose length changes in real time as the scale changes (larger than the reference line for movement of the camera closer to the subject, and at a distance away from it). Smaller).

Although not specifically shown in the example embodiment of FIG. 58, other such geometric data may also be derived or provided, for example, translation, different scaling, tip angle (ie, forward / reverse), and the like.

The above-described determinations can be simplified when the camera field of view includes a digital watermark with steganographic correction / orientation data of the kind described above in the referenced patent documents. However, the information can also be derived from other features in the image.

Of course, in still other embodiments, data from other position sensing arrangements in one or more accelerometers or devices may be used to generate information provided alone or in combination with image data.

In addition to providing such geometric information on the device screen, this information can also be used, for example, in the detection of gestures made by the user by the user, in the context of providing a context in which the remote system can be customized, and the like.

Camera-Based Environment and Behavior machine

According to another aspect of the present technology, a cell phone functions as a state machine, for example changing aspects of its functionality based on previously acquired image-related information. The image-related information is related to the natural behavior of the camera user, the typical environments in which the camera operates, the physical properties of the camera itself, the structure and dynamic properties of the scenes imaged by the camera, and many other such categories of information. Can be focused. Changes resulting from camera functionality can be directed towards improving image analysis programs that are remotely located on any image-analysis server or reside on a camera-device. Image analysis covers the full range of analysis, from digital watermark reading to object and face recognition, 2-D and 3-D barcode reading and optical character recognition, in all ways through scene categorization analysis and beyond. It is interpreted very broadly.

Some simple examples will illustrate what is expected to be an important aspect of future mobile devices.

Consider the problem of object recognition. Most objects have different appearances depending on the angle at which they are viewed. Given some information about the view in which an object is viewed, the machine version object-recognition algorithm can make a more accurate (and faster) guess of what the object is.

People are creatures of habit, including the use of cell phone cameras. This extends to the hands they usually hold the phone and how to tilt it while taking pictures. After the user establishes a history with the phone, the usage patterns can be distinguished from the captured images. For example, the user may try to take the photos of the subject slightly to the right rather than straight. This right-beveling tendency in the view can generally be due to the fact that the user is holding the camera in his right hand, so that the exposures are taken slightly from the right center.

(Right-tilt can be sensed in various ways by, for example, the lengths of the vertical parallel edges in the image frames. If the edges no longer want to be on the right sides of the image, this is right-tilted Try to show that the images were taken from the view, differences in illumination across the foreground objects can also be used-brighter illumination on the right side of the objects suggests that the right side is closer to the lens.

Similarly, in order to easily manipulate the shutter button of the phone while holding the device, this particular user may habitually adopt a grip of the photo which tilts the top of the camera 5 degrees (ie to the left) towards the user. have. In captured image objects these results are generally oblique with a clear rotation of 5 degrees.

These reoccurring biases can be distinguished by examining the collection of images captured by the cell phone and its user. Once identified, data storing these characteristics can be stored in memory and used to optimize image recognition processes performed by the device.

Thus, the device may generate a first output (e.g., tentative object identification) from a given image frame at one time, but a second different output (e.g., different object identification) from the same image frame at a later time. )-Due to the intervening use of the camera.

The characteristic pattern of jitter of the user's hand can also be inferred by experimentation of a plurality of images. For example, by examining images of different exposure periods, one may find that the user has jitter with a frequency of 4 hertz predominant in the left-right (horizontal) direction. Sharp filters tailored to the jitter behavior (and also dependent on the length of exposure) can then be applied to enhance the resulting image.

In a similar manner, through the use, the device is generally illuminated with spectral features of fluorescence of images captured by the user during weekly hours of 9:00-5:00, whereby a rather sharp white-balancing operation is attempted and Should be applied for compensation. Using prior knowledge of this trend, the device can expose the photos captured during those times differently from the baseline exposure parameters-expect fluorescent illumination and allow a better white balance to be achieved.

Over time, the device derives information that models some aspect of the user's habitual behavior or environmental variables. Thereafter, the device adapts according to some aspect of the operation.

The device may also adapt to its own features or degradations. These include nonuniformities of the photodiodes of the image sensor, dust on the image sensor, flaws on the lens, and the like.

Again, over time, the device can detect the reoccurrence pattern: (a) one pixel provides an average output signal that is 2% lower than adjacent pixels; (b) consecutive groups of pixels tend to output signals that are about three digital numbers lower than the indicated averages; (c) Certain areas of the photosensor are unlikely to capture high frequency details-images in those areas are consistently blurry bits, etc. From this reoccurrence, the device is for example (a) having a low gain for the amplifier serving this pixel; (b) dust or other foreign object is blocking these pixels; (c) Lens flaws can be inferred from preventing the light in this area of the photosensor from being properly focused. Thereafter, appropriate compensations can be applied to mitigate these defects.

Common aspects of the object or “imaged scenes” are at least early-stage image processing steps or other rich for subsequent image analysis routines that assist later image analysis routines by optimally filtering and / or transforming pixel data. Source information. For example, a given user using these cameras for only three basic interests: digital watermark reading, barcode reading, and visual logging of experimental setups in the laboratory may become apparent over the days and weeks of camera use. A histogram can be developed over time showing what "end result" operation some given camera use caused, followed by an increase in processing cycles focused on initial detections of both watermark and barcode basic characteristics. . By drilling the bit deeper here, the Fourier-transformed set of image data may be preferentially routed to fast 2-D barcode detection, otherwise it may be out of priority. The same is true for digital watermark reading, where the Fourier transformed data can be shipped to a specialized pattern recognition routine. The partial summary scheme for viewing this state-machine change is that there is only a fixed amount of CPU and image-processing cycles available to the camera device, and choices must be made as to which modes of analysis get what parts of those cycles. do.

An oversimplified representation of these embodiments is shown in FIG. 59.

With the arrangements just discussed, the operation of the imager mounted device is developed through continuous operation.

Focus issues, improved print-to-web based on page layout Linking

Cameras mounted on most cell phones and other portable PDA-type devices generally do not have adjustable focal points. Rather, the optics are configured in a compromise manner-for the purpose of obtaining a matching image under conventional portrait snapshots and landscape environments. Imaging at close distances generally produces lower results-losing high frequency detail. (This is improved by the "extended depth of field" image sensors just discussed, but widespread deployment of such devices has not yet occurred.)

Human visual systems have different sensitivity to the image at different spectral frequencies. Different image frequencies convey different impressions. Low frequencies provide global information about the image, such as orientation and general shape. High frequencies provide fine details and edges. As shown in FIG. 72, the sensitivity of the human visual system peaks at frequencies of about 10 cycles / mm on the retina and drops sharply on the side. (Perception also depends on the contrast between the features to be distinguished-the vertical axis.) Image features with spatial frequencies and contrast in the shaded region of the parallel are generally not perceived by humans. 73 shows an image with low and high frequencies (left and right) depicted separately.

Digital watermarking of print media, such as newspapers, can be done by shading (before, during or after painting) the page with a non-nasty background pattern that conveys secondary payload data to steganographic. Different columns of text may be encoded with different payload data, for example, allowing each news story to link to a different electronic resource (see, for example, Digimarc's patents 6,985,600, 6,947,571 and 6,724,912).

According to another aspect of the present technology, the near-focus defects of portable imaging devices can be overcome by embedding a low frequency digital watermark (eg, in a spectral configuration above the curve centered on the left side of FIG. 72). Instead of encoding different watermarks in different columns, the page is marked with a single watermark that applies to the page-encoding the identifier for that page.

When a user snaps an image of a newspaper story of interest (the image can capture text / graphics from the desired story / advertisement, or can likewise span other content), the watermark of the page is decoded (on the device Locally, remotely by different devices, or in a distributed manner).

The decoded watermark serves to provide on the display screen to index the data structure that returns information to the device. The display provides a map of the newspaper page layout with different articles / ads shown in different colors.

74 and 75 illustrate one particular embodiment. The original page is shown in FIG. The layout map displayed on the user device screen is shown in FIG. 75.

To link additional information about any of the stories, the user simply touches a portion of the display map that corresponds to the story of interest. (If the device is not equipped with a touch screen, the map of FIG. 75 may be provided with indicators identifying different map regions, for example 1, 2, 3 ... or A, B, C .... May then manipulate the device's numeric or alphabetical user interface (eg, keypad) to identify the article of interest.)

The user's selection is sent to a remote server (which can be the same one or the other serving layout map data to the portable device), and then consults the stored data to identify the information in response to the user's selection. For example, if the user touches the area on the bottom right of the page map, the router system can instruct the server of buick-dot-com to send a page with more information about Buick Lucerne for presentation on the user device. have. Or the remote system can send a link to the page to the user device, which can then load the page. Or the remote system listens to the associated podcast, for example; See early stories on the same subject; Indicate reprints; For a new article, where the options may be given to the user for downloading the article to a word file, etc., the user device may be provided with a menu of options. Alternatively, the remote system can send a link to the menu page or web page to the user by e-mail so that the user can review it later. (A variety of these different responses to user-presented choices may be provided as known in the art cited herein.)

Instead of the map of FIG. 75, the system allows the user device to display a screen showing a reduced scale version of the newspaper page itself—as shown in FIG. 74. Again, the user can simply touch the article of interest to trigger the associated response.

Or, instead of providing a graphical layout of the page, the remote system may provide titles of all content on the page (eg, "Banks Owe Billions ...", "McCain Pins Hopes ...", "Buick Lucerne"). Can return These titles are provided in the form of a menu on the device screen, and the user touches the desired item (or enters the corresponding number / letter selection).

Layout maps for printed newspaper and magazine pages, respectively, are typically generated by a publishing company as part of its layout process using automated software from vendors, such as, for example, Quark, Impress, and Adobe. Thus, existing software knows which articles and advertisements appear in which spaces on each printed page. These same software tools or others take this layout map information, associate corresponding links or other data for each story / advertisement, and web-access where portable devices can access a web-accessible server. It can be adapted to store the resulting data structure on a possible server.

The layout of newspaper and magazine pages provides orientation information that may be useful for watermark decoding. The columns are vertical. Headlines and lines of text are horizontal. Even at very low spatial image frequencies, such shape orientation can be distinguished. A user capturing an image of a printed page cannot capture content “squarely”. However, these powerful vertical and horizontal components of the image are easily determined by algorithmic analysis of the captured image data, allowing the rotation of the captured image to be distinguished. This knowledge simplifies and speeds up the watermark decoding process (since the first step of many watermark decoding operations is to distinguish the rotation of the image from the original encoded state).

In another embodiment, the transfer of the page map from the remote server to the user device is unnecessary. Again, the area of the page that spans several items of content is encoded into a single watermark payload. Again, the user captures an image containing the content of interest. The watermark identifying the page is decoded.

In this embodiment, the captured image is displayed on the device screen and the user touches a particular area of content of interest. Coordinates of user selections in the captured image area are recorded.

76 is illustrative. The user captured an image from an excerpt from a watermarked newspaper page and then used an Apple iPhone, a T-Mobile Android phone, etc. to touch the article of interest (represented by an oval). The position of the touch in the image frame is known to the touch screen software, for example as an offset from the upper left corner measured in pixels. (The display can have a resolution of 480 x 320 pixels). The touch may be at pixel location 200, 160.

The watermark spans the page and is shown in FIG. 76 by dotted diagonal lines. The watermark (as described in, for example, Digimarc's patent 6,590,996) has an origin, but the origin is not in the image frame captured by the user. However, from the watermark, the watermark decoder software knows the scale of the image and its rotation. It also knows the offset of the image frame captured from the origin of the watermark. Based on this information and information about the scale at which the original watermark was encoded (information can be conveyed with the watermark, accessed from a remote repository, hard-coded to the detector, etc.) It can be determined whether the upper left corner of the captured image frame corresponds to a point 1.6 inches below the top left corner of the originally printed page and 2.3 inches right (assuming the watermark origin is at the top left corner of the page). From the decoded scale information, the software can identify that the 480 pixel width of the captured image corresponds to a 12 inch wide area of the original printed page.

The software finally determines the location of the user's touch as an offset from the top left corner of the originally printed page. It has a corner of the captured image offset from the top left corner of the printed page (1.6 ", 2.3") and the touch is 5 "more to the right, for the final position in the original printed page of (6.6", 6.3 "). It can be seen that it is farther away (200 pixels x 12 "/ 480 pixels) and 4" further down (160 pixels * 12 "/ 480 pixels).

The device then sends these coordinates with the payload of the watermark (identifying the page) to the remote server. The server looks up the layout map of the identified page (from the appropriate database stored by the page layout software) and consults the coordinates to determine where the user's touch is in articles / advertisements. The remote system then returns the corresponding information related to the displayed article to the user device, as noted above.

Returning to focus, the near-focus handicap of the PDA camera can actually turn into an advantage in decoding watermarks. The watermark information is not retrieved from the inked areas of the text. Subtle modulations of luminance on which most watermarks are based are lost in areas where full black is printed.

Once the page substrate is painted with a watermark, useful watermark information is recovered from areas of the unprinted page, such as from "white space" such as between columns, between lines, the end of paragraphs, and the like. Inked characters are the "noise" that is best ignored. The blurring of the printed portions of the page introduced by the focal defects of the PDA cameras can be used to define a mask-identifying areas of large amounts of ink. These parts can be ignored when decoding the watermark data.

More particularly, blurred image data may be thresholded. Any image pixels with values darker than the threshold may be ignored. Alternatively, only image pixels with values brighter than the threshold are input to the watermark decoder. The "noise" contributed by the characters in the ink is thus filtered out.

In imaging devices capturing clearly focused text, similar advantages can be created by processing text with a blurry kernel-and by extracting the areas found to be dominant by this printed text.

With arrangements such as those described above, the shortcomings of portable imaging devices are corrected and improved print-to-web linking based on page layout data is enabled.

Image search, feature extraction, pattern matching  Etc

Inc.(Toronto, ON)로부터의 Pixsimilar 이미지 검색 소프트웨어 및/또는 비주얼 검색 개발자의 키트(SDK)를 이용하여 구현될 수 있다. 이미지에 대한 설명적 주석들을 자동으로 생성하는 도구는 특허 7,394,947 (Penn State)에 상술된 바와 같이 ALIPR(Automatic Linguistic Indexing of Pictures)이다. All of the image retrieval functions of the specific embodiments described above

Pixsimilar image search software from Inc. (Toronto, ON) and / or Visual Search Developer's Kit (SDK). A tool for automatically generating descriptive annotations for an image is Automatic Linguistic Indexing of Pictures (ALIPR) as detailed in Patent 7,394,947 (Penn State).

In the embodiments described above content-based image retrieval (CBIR) may also be used. As is familiar to those skilled in the art, CBIR inherently involves (1) extracting the characterization of an image—generally mathematically—and (2) using this characterization to assess similarity between images. Two documents examining these fields are described in IEEE Trans. Pattern Anal. Mach. "Content-Based Image Retrieval at the End of the Early Years" by Smeulders et al. In Intell, Vol. 12, no. 22, pp. 1349-1380, and by Datta et al. In ACM Computing Surveys, Vol. 40, April 2008. Image Retrieval: Ideas, Influences and Trends of the New Age.

Identifying identical looking images from large image databases is a familiar operation in the issuance of driver's licenses. That is, the image captured from the new applicant is generally verified against a database of all previous driver's license photos to check whether the applicant has already issued a driver's license (possibly under another name). Methods and systems known from the driver's license field may be utilized in the arrangements described herein above. (Examples include Identix patent 7,369,685 and L-1 Corp. patent 7,283,649 and 7,130,454.)

In many embodiments herein, image feature extraction algorithms known as CEDD and FCTH are useful. The former was published in Chatzichristofis et al. In "CEDD: Color and Edge Directivity Descriptor-A Compact Descriptor for Image Indexing and Retrieval" at the 6th International Conference in Advanced Research on Computer Vision Systems ICVS 2008, May 2008; The latter is detailed in the May 9th International Workshop on Image Analysis for Multimedia Interactive Services, Proceedings: IEEE Computer Society, in "FCTH: Fuzzy Color And Texture Histogram-A Low Level Feature for Accurate Image Retrieval" by Chatzichristofis et al.

Open-source software that implements these techniques is available; See the web page savvash.blogspot-dot-com / 2008/05 / cedd-and-fcth-are-now-open-dot-html. DLLs that implement that functionality can be downloaded. Classes for input image data (e.g., file.jpg) can be invoked as follows:

double [] CEDDTable = new double [144];

double []

FCTHTable = new double [144];

Bitmap ImageData = new Bitmap ("c: /file.jpg");

CEDD

GetCEDD = new CEDD ();

FCTH GetFCTH = new FCTH ();

CEDDTable = GetCEDD.Apply (ImageData);

FCTHTable =

GetFCTH.Apply (ImageData, 2);

CEDD and FCTH can be combined to produce improved results using the joint compound descriptor file available from the web page just cited.

Chatzichristofis made the open source program "img (Finder)" available (see savvash.blogspot-dot-com / 2008/07 / image-retrieval-in-facebook-dot-html)-using CEDD and FCTH Content based on an image search desktop application that retrieves and indexes images from a Facebook social networking site. In use, the user connects Facebook with their personal account data, and the application downloads information from the user's friends' image albums as well as the user's images, indexing these images for search with CEDD and FCTH features. . The index can then be queried by the sample image.

Chatzichristofis also uses the service online retrieval service "img (Anaktisi)"-with image metrics including CEDD and FCTH-in which the user uploads photos and the service retrieves similar images from one of eleven different image archives. Made available. See orpheus.ee.duth-dot-gr / anaktisi /. (Image archives include flicker). In the related description of the Anaktisi search service, Chatzichristofis is described as follows:

The rapid growth of digital images through the widespread popularity of computers and the Internet has led to the development of critical and efficient image retrieval techniques. CBIR Known as Content Trait-based image retrieval extracts several features describing image content and maps them to a new space called the feature space of the visual content of the images. Feature space values for a given image are stored in a descriptor that can be used to retrieve similar images. The key to a successful retrieval system is to select the appropriate features that represent the images as accurately and uniquely as possible. The selected features must be unique and sufficient to describe the objects present in the image. To achieve these goals, CBIR systems use three basic types of features: color features, texture features and spatial features. It is very difficult to achieve satisfactory search results using only one of these types of features.

To date, many proposed retrieval techniques employ methods in which more than one feature type is involved. For example, color, texture and shape features are used for both IBM's QBIC and MIT's photobook. QBIC uses color histograms, instant-based shape features and descriptors of text. The photobook uses appearance features, texture features and 2D shape features. Other CBIR systems include SIMBA, CIRES, SIMPLIcity, IRMA, FIRE, and MIRROR. The cumulative body of the search provides extraction methods for these feature types.

In most search systems that combine two or more feature types, such as color and texture, separate vectors are used to describe each kind of information. While it is possible to achieve very good search scores by increasing the size of descriptors of images with high dimensional vectors, this technique has several drawbacks. If the descriptor has hundreds or even thousands of beans, it is not practically available because the retrieval procedure is considerably delayed. Also, Of descriptor  Increasing the size increases the storage requirements, which can have significant penalties for databases containing millions of images. Many provided methods limit the length of the descriptor to a smaller number of bins, leaving the possible factor values in undetermined quantized form.

Moving Picture Experts Group (MPEG) defines a standard for content-based access to multimedia data of the MPEG-7 standard. This standard identifies a set of image descriptors that maintain a balance between the size of a feature and the quality of the search results.

In this web-site, a new set of feature descriptors is provided to the search system. These descriptors are designed with special attention to size and storage requirements, keeping them as small as possible without compromising discernment. These descriptors combine color and texture information into one histogram while maintaining sizes between 23 and 74 bytes per image.

High search scores in content-based image retrieval systems can be achieved by employing relevant feedback mechanisms. These mechanisms require the user to classify the quality of query results by marking the retrieved images as being related or not. The search engine then uses this sorted information in subsequent queries to better meet the needs of the user. Note that although relevant feedback mechanisms were first introduced in the information related field, they are now of considerable interest in the CBIR field. Many of the relevant feedback techniques proposed in the references are based on modifying the values of the search parameters, which better express the concept that the user has in mind. The search parameters are calculated as a function of the relevant values assigned by the user for all the images retrieved so far. For example, relevant feedback is frequently formulated in terms of modification of the query vector and / or in terms of adaptive similarity metrics.

Also in this web-site, an Auto Relevance Feedback (ARF) technique is introduced based on the proposed descriptors. The goal of the proposed automatic related feedback (ARF) algorithm is best re-adapted in the initial search results based on user preferences. During this procedure, the user selects one from the primary retrieved images as related to his initial search expectations. Information from these selected images is used to change the initial query image descriptor.

Another open source content-based image retrieval system is the GIFT (GNU image search tool) created by researchers at Geneva University. One of the tools allows a user to index directory trees containing images. The GIFT server and its client (SnakeCharmer) can then be used to retrieve indexed images based on image similarity. The system is further described on the web page gnu-dot-org / software / gift / gift-dot-html. The latest version of the software can be found on the ftp server ftp.gnu-dot-org / gnu / gift.

Another open source CBIR system is Fire, written by Tom Deselaers et al at RWTH Aachen University and available for download from the web page -i6.informatik.rwth-aachen-dot-de / ~ deselaers / fire /. Fire uses, for example, the technique described in "Features for Image Retrieval: An Experimental Comparison" by Deselaers, March, 2008, in Scringer, Netherlands, Vol. 11, No. 11, pp. 77-107.

Embodiments of the present invention generally relate to objects depicted in an image rather than entire frames of image pixels. Recognition (sometimes called computer vision) of objects in an image is a large science that is considered familiar to the reader. Edges and centroids are image features that can be used to help recognize objects in images. The other is shape contexts (see Matching with Shape Contexts by Belongie et al. In the 2000 IEEE Workshop on Content Based Access of Image and Video Libraries). Robustness to affine transformations (eg, scale invariance, rotational invariance) is an advantageous feature of certain object recognition / pattern matching / computer vision techniques. Methods based on Hough transform and Fourier melin transform exhibit rotation-invariant properties. SIFT (discussed below) is an image recognition technique with this and other advantageous properties.

In addition to object recognition / computer vision, the image processing (as opposed to metadata associated processing) discussed herein may utilize a variety of other techniques, which may proceed by various names. Image analysis, pattern recognition, feature extraction, feature detection, template matching, face recognition, eigenvectors, and the like. (All these terms are generally used interchangeably herein.) Interested readers refer to Wikipedia with articles on each of the topics just listed, including individual descriptions and citations of related information.

Image metrics of the described kind are sometimes considered as metadata, ie "content-dependent metadata". This is in contrast to "content-descriptive metadata"-more familiar in terms of the term metadata being used.

Communication With devices  Interaction

Most of the examples described above involve imaging objects that do not have a means to communicate. This section considers more particularly those techniques applied to objects that may or may not be equipped to communicate. Simple examples are WiFi-equipped thermostats and parking meters, Ethernet-linked phones and hotel bedside watches equipped with Bluetooth.

Consider a user driving into the user's office downtown. When she finds an empty parking space, she points her cell phone to the parking meter. The virtual user interface (UI) appears almost immediately on the cell phone screen-allowing the user to purchase two hours from the meter. Inside the office building, a woman finds the conference room cool and points the cell phone to the thermostat. After a while, a different virtual user interface appears on the cell phone-allowing her to change the settings of the thermostat. The cell phone will ring 10 minutes before the parking meter is about to run out of time and again provide the UI for the parking meter. The user buys another hour-from her office.

For industrial users and other applications where security of interaction is important or for applications where anonymity is important, various levels of security and access privileges can be integrated into the interaction session between the object being imaged and the user's mobile device. . The first level involves simply explicitly or covertly encoding contact instructions in the surface feature of the object, such as an IP address; The second level includes providing the public-key information to the device either explicitly through explicit symbols or more subtly via digital watermarking; And the third level can be obtained only by unique patterns or digital watermarking actively taking a picture of the object.

The interface provided on the user's cell phone may be adapted to the user's preferences and / or to facilitate specific operational interactions with the device (e.g., while the office staff may stop controlling the temperature setting. You can stop the "debug" interface to the stats).

These costs may be unnecessary where there is a physical object or device incorporating elements such as displays, buttons, dials, or other such features intended for physical interaction with the object or device. Instead, the functionality may be overlapped by the mobile device actively and virtually interacting with the object or device.

By integrating a wireless chip into a device, manufacturers can effectively enable a mobile GUI for that device.

According to one aspect, such techniques include using a mobile phone to obtain identification information corresponding to a device. With reference to the obtained identification information, application software corresponding to the device is then identified and downloaded to the mobile phone. This application software is then used to facilitate user interaction with the device. By this arrangement, the mobile phone acts as a multifunctional controller-adapted to control a particular device through the use of the identified application software with reference to the information corresponding to that device.

According to another aspect, such techniques include using a mobile phone to sense information from a housing of a device. Through the use of this sensed information, other information is encrypted using the public key corresponding to the device.

According to another aspect, this technique includes using a mobile phone to sense analog information from a device. The sensed analog information is converted into digital form and the corresponding data is transmitted from the cell phone. This transmitted data is used to confirm user proximity to the device before allowing the user to interact with the device using the mobile phone.

According to another aspect, this technique includes using a user interface on a user's cell phone to receive instructions related to control of the device. This user interface is combined with the cell phone-captured image of the device to provide a cell phone on the screen. While the information corresponding to the command is signaled to the user in a first manner, the command is pending; And, in the second manner, the command was successfully executed once.

According to another aspect, the present technology includes initiating a transaction with the device using a user interface provided on the screen of the user cell phone when the user is in proximity to the device. Later, cell phones are used for non-device related purposes. Later on, the user interface is recalled and used to associate in another transaction with the device.

According to another aspect, such technology includes a mobile phone that includes a processor, a memory, a sensor, and a display. The instructions in the memory configure the processor to perform the following operations: sense information from the first device in proximity; Referring to the sensed information, download first user interface software corresponding to the first device; Interact with the first device by user interaction with the downloaded first user interface software; Recall from the memory, second user interface software initially downloaded to the mobile phone corresponding to the second device; Operations interacting with the second device by user interaction with the recalled second user interface software, whether or not the user is in close proximity to the second device.

According to another aspect, such technology includes a mobile phone that includes a processor, a memory, and a display. The instructions in the memory allow the processor to provide a user interface that allows the user to select between several different device-specified user interfaces stored in memory for interacting with a plurality of different external devices using the mobile phone. Configure your handler.

These arrangements are more particularly detailed with reference to FIGS. 78-87.

78 and 79 show a prior art WiFi-mounted thermostat 512. This includes a temperature sensor 514, a processor 516 and a user interface 518. The user interface includes various buttons 518, an LCD display screen 520, and one or more indicator lights 522. Memory 524 stores programming and data for the thermostat. Finally, WiFi transceiver 526 and antenna 528 allow communication with remote devices. (The depicted thermostat 512 is available as a model CT80 from a US wireless thermostat company. The WiFi transceiver includes a GainSpan GS1010 System on Chip (SoC) device.)

80 shows a similar thermostat 530, but implements principles in accordance with certain aspects of the present technology. Like the thermostat 512, the thermostat 530 includes a temperature sensor 514 and a processor 532. Memory 534 may store the same programming and data as memory 524. However, this memory 534 contains slightly more software to support the functions described below. (For convenience of description, a name is given to the software associated with this aspect of the present technology: ThingPipe software. The thermostat memory thus has a ThingPipe code, which is used to implement the functions described above-other devices-such as cell phones-. Work with other code on the desktop.

The thermostat 530 may include the same user interface 518 as the thermostat 512. However, significant savings can be achieved by omitting many associated parts, such as LCD displays and buttons. The depicted thermostat may thus only include indicator lights 522, which may also be omitted.

 The thermostat 530 also includes an arrangement whose identity can be detected by the cell phone. WiFi emissions from the thermostat may be utilized (eg, by the device's MAC identifier). However, other means such as an indicator that can be detected by the camera of the cell phone are desirable.

Steganographic digital watermarks are one such indicator that can be detected by a cell phone camera. Digital watermark technology is detailed in Assignee's patents, including 6,590,996 and 6,947,571. The watermark data may be encoded in a texture pattern on the exterior of the thermostat, on an attachment label, on a pseudo wood-grain trim on the thermostat, and the like. (The steganographic encoding is hidden and therefore not depicted in FIG. 80.)

Other suitable indicators are 1D or 2D barcodes or clear symbols, such as barcode 536 shown in FIG. This can be printed on the thermostat housing applied by an attachment label or the like.

Another means for identifying a thermostat, such as an RFID chip 538, may be utilized. The other is a short range wireless broadcast or network service discovery protocol (eg Bonjour) of the Bluetooth identifier, such as by Bluetooth. Object recognition by means such as scale-invariant feature transformation such as SIFT or image fingerprinting may also be used. Other identifiers can also be used-manually entered by the user or identified through navigating the directory structure of possible devices. The skilled person will know many other alternatives.

81 shows an example cell phone 540, such as an Apple iPhone device. This includes typical elements including processor 542, camera 544, microphone, RF transceiver, network adaptor, display and user interface. The user interface includes physical controls as well as touch-screen sensors. (Details of the user interface and associated software are provided in Apple's patent publication 20080174570.) The phone's memory 546 includes general operating system and application software. In addition, it includes ThingPipe software for performing the functions detailed herein.

Returning to the operation of the illustrated embodiment, the user captures an image depicting the digital-watermarked thermostat 530 using cell phone camera 544. The processor 542 of the cell phone pre-processes the captured image data (eg, by applying Wiener filters or other filtering and / or compression), and wirelessly processes the processed data to the remote server 552. (Fig. 82)-with information identifying the cell phone. (This may be part of the function of the ThingPipe code in the cell phone.) The wireless communication may be by WiFi to a nearby wireless access point and then by the Internet to the server 552. Or a cell phone network may be utilized, and the like.

The server 552 applies a decoding algorithm to the processed image data received from the cell phone to extract steganographically encoded digital watermark data. This decoded data, which may include an identifier of the thermostat, is sent by the Internet to the router 554, along with information identifying the cell phone.

Router 554 receives the identifier and looks up it in namespace database 555. The namespace database 555 examines the most significant bits of the identifier and makes a query to identify the particular server responsible for that group's identifiers. The server 556 identified by this process has data belonging to the thermostat. (This arrangement is similar to domain name servers utilized in Internet routing. Patent 6,947,571 describes additional disclosures about how watermarked data can be used to identify servers that know what to do with such data. Have.)

Router 554 polls the identified server 556 for information. For example, router 554 may store current data related to the thermostat (eg, current temperature set point and ambient temperature, server 556 may be obtained from the thermostat by a link including WiFi). Claim 556. The server 556 is also required to provide information regarding a graphical user interface suitable for display on the Apple iPhone 540 to control a particular thermostat. This information may include, for example, a JavaScript application that runs on cell phone 540 and provides a suitable GUI for use with a thermostat. This information can be passed back onto the cell phone-either directly or through the server 552. The returned information may include the IP address of the server 556 so that the cell phone may then exchange data directly with the server 556.

At cell phone 540, the ThingPipe software responds to the received information by providing a graphical user interface to the thermostat 530 on its screen. This GUI may include ambient temperature and setpoint temperature for the thermostat-whether to receive from the server 556 or directly from the thermostat (such as by WiFi). The provided GUI also includes controls that the user can operate to change settings. To raise the setpoint temperature, the user touches the displayed control (eg, "increase temperature" button) corresponding to this operation. The setpoint temperature provided in the UI display immediately increases in response to the user's action-perhaps in flashing or other unusual way to indicate that the request is pending.

The user's touch also causes the ThingPipe software to transmit corresponding data from the cell phone 540 to the thermostat (transmission may include some or all of the other devices shown in FIG. 82, or directly to the thermostat-WiFi As in-can proceed). Upon receipt of this data, the thermostat increases its set temperature per user's instructions. Thereafter, a confirmation message relayed from the thermostat back to the cell phone is issued. Upon receipt of the confirmation message, flashing of the increased temperature indicator is stopped and the setpoint temperature is then displayed in static form. (Other arrangements are of course possible. For example, the confirmation message can be rendered to the user as a visible signal-such as the text "accepted", audibly ringing or "OK" in speech provided on the display. .)

In one particular embodiment, the displayed UI is provided as an overlay on the screen of the cell phone and depicts the thermostat on top of the image initially captured by the user. Features of the UI are provided in registered alignment with any corresponding physical controls (eg, buttons) shown in the captured image. Thus, if the thermostat has temperature up and temperature down buttons (eg, "+" and "-" buttons in FIG. 79), the graphical overlay displays the image displayed in a unique manner such as scrolling red dotted lines. You can outline them in. These are graphic controls that the user touches to raise or lower the setpoint temperature.

This is shown schematically in FIG. 83, where the user has captured an image 560 of a portion of the thermostat. The image includes at least a portion of the watermark 562 (shown visually for illustration). With reference to the watermark and the data obtained from the server 556 regarding the layout of the thermostat, the cell phone processor scrolls and overlays the dotted lines at the top of the image-outlines the "+" and "-" buttons. When the phone's touch-screen user interface touches in these outlined areas, the cell phone reports it to the ThingPipe software. Thereafter, these touches are interpreted as commands to increase or decrease the thermostat temperature and send these commands to the thermostat (eg, via server 552 and / or 556). On the other hand, this also increases the "SET TEMPERATURE" graphic overlaid on top of the image and causes it to flash until a confirmation message is received back from the thermostat.

Registered overlay on top of the graphical user interface that captured the image data is enabled by encoded watermark data on the thermostat housing. The calibration data in the watermark allows rotation of the scale, transformation and placement of the thermostat in the image to be precisely determined. If the watermark is reliably placed on the thermostat in a known spatial relationship with other device features (eg, buttons and displays), the locations of these features in the captured image may be determined with reference to the watermark. have. (These techniques are further detailed in Applicant's published patent application 20080300011.)

If the cell phone does not have a touch-screen, the registered overlay of the UI can still be used. However, instead of providing a screen target for the user to touch, the outlined buttons provided on the cell phone screen may represent corresponding buttons on the keypad of the phone that the user must press to activate the outlined function. For example, the outlined box around the "+" button can be flashed orange with the number "2"-the user must press the "2" button on the cell phone keypad to increase the thermostat temperature setpoint. It shows. (The number "2" is flashed on top of the "+" portion of the image-allows the user to identify the "+" marking on the captured image when the number is not flashing.) Similarly, around the "-" button The outlined box of can be flashed orange periodically with the number "8"-indicating that the user must press the "8" button on the cell phone keypad to decrease the thermostat temperature setpoint. See 572,574 in FIG. 84.

Although the overlay of the graphical user interface to the registered alignment of the thermostat onto the captured image is thought to be easiest to implement through the use of watermarks, other arrangements are possible. For example, once the size and scale of the barcode and its location on the thermostat are known, the locations of the thermostat features for the overlay can be determined geometrically. Similar to an image fingerprint-based approach (including SIFT). Once the normal appearance of the thermostat is known (eg, by server 556), the relevant locations of the features in the captured image can be distinguished by image analysis.

In one particular arrangement, the user captures a frame of the image depicting the thermostat, which is buffered for static display by the phone. The overlay is then provided in a registered alignment with this static image. As the user moves the camera, the static image persists, and the overlaid UI is similarly static. In another arrangement, the user captures a stream of images (eg, video capture), and the overlay is provided in registered alignment with the features of the image even when they move from frame to frame. In this case, the overlay may move across the screen in response to movement of the depicted thermostat within the cell phone screen. Such an arrangement may allow the user to move the camera to capture thermostats of different aspects-perhaps additional features / controls appear. Or allow the user to zoom the camera so that certain specific features (and corresponding graphic overlays) appear or appear on a larger scale on the touch screen display of the cell phone. In this dynamic overlay embodiment, the user can optionally freeze the captured image at any time, and then continue to operate with the overlaid user interface control (static) —keep the thermostat in view of the camera. Regardless of what you do.

If the thermostat 530 is of a kind without visual controls, the UI displayed on the cell phone may be in any format. If the cell phone has a touch-screen, thermostat controls may be provided on the display. If no touch-screen is present, the display may simply provide a corresponding menu. For example, the user may be instructed to press "2" to increase the temperature setpoint and to press "8" to decrease the temperature setpoint.

After the user issues a command through the cell phone, the command is preferably relayed to the thermostat as described above and a confirmation message is returned again-for rendering to the user by ThingPipe software.

The displayed user interface is the function of the device (e.g., thermostat) with which the phone is interacting, and also the functions of the capabilities of the cell phone itself (e.g. whether it has a touch-screen, the dimensions of the screen). Etc.). Instructions and data that allow the cell phone's ThingPipe software to create these different UIs can be stored in a server 556 that manages the thermostat and is delivered to the memory 546 of the cell phone with which the thermostat interacts. .

Another example of a device that can be controlled so is a WiFi enabled parking meter. The user captures an image of the parking meter with the cell phone camera (for example, by pressing a button, or the image capture can be freely performed-such as every second or several times). The processes generally occur as described above. ThingPipe software processes the image data, and router 554 identifies the server 556a responsible for ThingPipe interactions with its parking meter. The server returns status information for that meter and optionally UI interactions (eg, remaining time; maximum allowable time). These data are displayed on the cell phone UI and are overlaid on the captured image of the cell phone, for example with controls / commands to buy time.

The user interacts with the cell phone to add two hours to the meter. The corresponding payment is withdrawn, for example, from the user's credit card account, which is stored as encrypted profile information on the cell phone or remote server. (Online payment systems, including payment arrays suitable for use with cell phones, are well known and are not described in detail here.) The user interface on the cell phone confirms that the payment has been made satisfactorily, and has been purchased from the meter. Display the number of. The display at the meter on the side may also reflect the time of purchase.

The user can leave the meter to perform other tasks and use the cell phone for other purposes. The cell phone can be in a low power mode-the screen dims. However, the downloaded application software keeps track of how many remain on the meter. This can also be done by periodically querying the data at the associated server. Alternatively, the countdown time can be tracked independently. At a given point, for example, if 10 minutes remain, the cell phone sounds an alarm.

Looking at the cell phone, the user can see that the cell phone has returned to the active state and the Meter UI has been restored to the screen. The displayed UI reports the remaining time and gives the user the opportunity to purchase more time. The user purchases another 30 minutes of time. The completed purchase is confirmed on the cell phone display-showing 40 minutes of time remaining. The display on the distance side meter can be similarly updated.

Note that the user does not need to physically return to the meter to add time. The virtual link between the cell phone and the parking meter was either continued or reestablished-even though the user could walk the 12 blocks and climb the elevator over multiple floors. Parking meter control is as close as the cell phone.

(Unless specifically mentioned, the block diagram of the parking meter is similar to that of the thermostat of FIG. 80 except that it does not have a temperature sensor.)

Consider the third example-bed alarm clock in a hotel. Most travelers know the various illogical user interfaces that these clocks provide are wrong. It is late; The traveler is confusing from the long flight and now faces the chore of finding out which of the black buttons on the black clock should be manipulated in the hazy hotel room to set the alarm clock at 5:30 am. It is even better if these devices can be controlled by an interface provided on the user's cell phone-a standardized user interface that the traveler knows from repeated use is preferred.

85 illustrates an alarm clock 580 utilizing aspects of the present technology. Like other alarm clocks, this includes a display 582, a physical UI 584 (eg, buttons), and a control processor 586. However, this clock also includes a Bluetooth air interface 588 and a memory 590 in which ThingPipe and Bluetooth software are stored for execution by the processor. The clock also has means for self identification, such as a digital watermark or barcode, as described above.

As in the earlier examples, the user captures an image of the clock. The identifier is decoded from the image by the cell phone processor or by the processor of the remote server 552b. From the identifier, the router identifies another server 556b that can know about such clocks. The router passes the identifier along with the cell phone's address to another server. The server uses the decoded watermark identifier to look up a particular clock and recall instructions about its processor, display and other configuration data. It also provides instructions for a particular display of cell phone 530 to provide a standardized clock interface through which clock parameters can be set. The server packages this information in a file, which is sent back to the cell phone.

The cell phone receives this information and provides the user interface described above by the server 556b on the screen. This is a familiar interface-which appears whenever this cell phone is used to interact with the hotel alarm clock, regardless of the model or manufacturer of the clock. (In some cases, the pawn can simply recall the UI from the UI cache, for example in a cell phone, because it is frequently used.)

The UI contains the control "LINK TO CLOCK". When selected, the cell phone communicates by clock and Bluetooth. (Parameters sent from server 556b may be required to establish a session.) Once linked by Bluetooth, the time displayed on the clock is provided with a menu of options on the cell phone UI.

One of the options provided on the cell phone screen is "SET ALARM". Upon selection, the UI moves to another screen 595 (FIG. 86), prompting the user to enter the desired alarm time by pressing the digit keys on the keypad of the phone. (Other paradigms can naturally be used, such as, for example, flicking displayed numbers on a touch-screen interface to rotate them until the desired digits appear.) When the desired time is entered, the user Press the OK button on the cell phone keypad to set the time.

As before, the input user data (e.g., alarm time) remains until the device issues a confirmation signal-the data displayed at that point stops flashing-when the command is sent to the device (in this case Bluetooth). Flash).

In the clock, a command to set the alarm time to 5:30 a.m. is received by Bluetooth. The ThingPipe software in the alarm clock memory understands the format in which data is transmitted by the Bluetooth signal, and analyzes the commands to set the desired time and alarm. The alarm clock processor then sets the alarm to sound at the specified time.

Note that in this example, the cell phone and clock communicate directly—rather than through one or more intermediate computers—other computers have been referenced by the cell phone to obtain programming details about the clock, but once obtained, No contact)

In this example-unlike the thermostat-the user interface is further noted that it does not self-integrate (eg, with a registered alignment) with the image of the clock captured by the user. These improvements are omitted to provide a consistent user interface experience-independent of the particular clock being programmed.

As in the initial example, the watermark is good for identifying a particular device by the present subscriber. However, any other known identification technique can be used, including the above known identification techniques.

The optional location modules 596 for each of the devices described above have not been described yet. One such module is a GPS receiver. Another state-of-the-art technology suitable for such modules relies on wireless signals that are generally exchanged between devices (eg WiFi, cellular, etc.). Given several communication devices, the signals themselves-and the incomplete digital clock signals that control them-form a reference system from which both very accurate time and location can be extracted. Such techniques are detailed in International Publication No. WO08 / 073347.

Knowing the locations of the devices allows the improved functionality to be realized. For example, it allows devices to be identified by their location (eg, unique latitude / longitude / altitude coordinates) —rather than by an identifier (eg watermarked or otherwise). Moreover, it allows the proximity between the cell phone and other ThingPipe devices to be determined.

Consider the example of a user accessing a thermostat. Rather than capture an image of the thermostat, the user can simply launch the phone's ThingPipe software (or may already be running in the background). This software communicates the current location of the cell phone to the server 552 and requests identification of other ThingPipe-enabled devices nearby. ("Nearby" is of course implementation dependent. It may be, for example, 10 feet, 10 meters, 50 feet, 50 meters, etc. This parameter may be defined by the cell phone user, or may be the default. The value may be utilized.) The server 552 checks the database identifying the current locations of other ThingPipe-enabled devices and returns the data to the cell phone identifying those nearby. Listing 598 (FIG. 87) is provided on the cell phone screen—including distance from the user. (If the location module of the cell phone includes a magnetic field or other means for determining the direction that the device is facing, the displayed listing may also display direction clues with distance, eg "4 'to your left". May contain)

The user selects THERMOSTAT from the displayed list (for example, by touching the screen-by touching the screen, or by entering an associated digit on the keypad). The phone then establishes a ThingPipe session with the device so identified as described above. (In this example, the thermostat user interface is not overlaid on top of the thermostat's image because no image has been captured.)

In the three examples described above, there is a question of who should be allowed to interact with the device and for how long.

In the case of a hotel alarm clock, permission is not important. Anyone in the room can be considered authorized to set clock parameters (eg, current, time, alarm time, display brightness, buzz or wireless alarm, etc.) that can detect a clock identifier. However, the permission should only last if the user is in the vicinity of the clock (eg within Bluetooth range). The previous guest does not need to reprogram the alarm while the next night's guest sleeps.

In the case of a parking meter, permission must be given back to someone who approaches the meter and captures an image (or detects its identifier from a short range).

As noted, in the case of the parking meter, the user can recall the UI corresponding to Naju and associate with the device and other transactions. This is somewhat good. Perhaps 12 hours from the time of image capture is a suitable time interval within which the user can interact with the meter. (If the user adds time later in 12 hours-there is no problem when someone else is parked in the space.) Alternatively, the permission to interact with the user's device allows the new user to establish a session with the meter. May be terminated upon initiation (eg, by capturing an image of the device and initiating a transaction of the kind identified above).

The memory storing the data setting the user's permission period may be located in the meter or elsewhere, for example in the server 556a. The corresponding ID for the user may also be generally stored. This may be the user's telephone number, the MAC identifier for the phone device, or some other generally unique identifier.

In the case of a thermostat, there may be tighter controls on who is allowed to change the temperature and how long. Perhaps only the managers of the office can set the temperature. Another employee may be provided with lower rights, for example, just to see the current ambient temperature. Again, the memory storing this data may be located in the thermostat, server 556, or elsewhere.

These three examples are simple and the controlled devices are a small result. In other applications, higher security is naturally involved. The field of authentication is well developed and the technician can derive from known techniques and techniques for implementing an authentication arrangement suitable for the specific needs of any given application.

As technology becomes widespread, users may need to switch between several on-going ThingPipe sessions. The ThingPipe application may have a "Recent UI" menu option that, when selected, invokes a list of pending or recent sessions. Selecting anything recalls the corresponding UI, allowing the user to continue initial interaction with a particular device.

Physical user interfaces-such as for thermostats, etc.-are fixed. All users are provided with the same physical display, knobs, dials and the like. All interactions must forcibly adapt these same physical vocabulary controls.

Implementations of aspects of the present technology may be more diverse. Users can have stored profile settings-tailor cell phone UIs to their specific preferences-globally and / or on a per device basis. For example, a user who is color blind may be so specified, that a gray scale interface is always provided—instead of colors that may be difficult for the user to distinguish. It may be desirable for a person with primitive vision to have the information displayed in the largest possible font-regardless of aesthetics. Others may choose to have the text read from the display, such as by synthesized speech. One particular thermostat UI may generally provide text representing current data; The user may prefer that the UI not be clustered with such information, and specify that-for that UI-no data information should be shown.

The user interface can also be customized for specific task oriented interactions with the object. The technician can call the "debug" interface to the thermostat to adjust the associated HVAC system; The office worker can invoke a simpler UI that simply provides current and setpoint temperatures.

Different levels of security and access privileges may also be provided, as different interfaces may be provided to different users.

The first level of security simply includes (obviously or covertly) encoding contact instructions for the object in surface features of the object, such as an IP address. The session simply begins with a cell phone collecting contact information from the device. (It may be indirectly related; information on the device may refer to a remote repository that stores contact information for the device.)

The second level contains public-key information, which can be more subtly hidden through steganographic digital watermarking, which is explicitly provided on the device via explicit symbols and indirectly accessed or otherwise-delivered. For example, the machine-readable data on the device can provide the device's public-key-using which transmissions from the user must be encrypted. The user's transmissions can also convey the user's public key-whereby the device can identify the user and use it to encrypt the data / commands returned to the cell phone.

This arrangement allows for a secure session with the device. Thermostats in the mall can use this technique. All passers can read the public key of the thermostat. However, the thermostat can provide control rights only to a particular user-identified by their respective public keys.

The third level includes preventing control of the device unless the user presents unique patterns or digital watermarking that can only be obtained by actively taking a picture of the device. That is, it is not simple enough to send an identifier corresponding to the device. Rather, a minus that proves the physical proximity of the user's device must also be captured and transmitted. The user can acquire the necessary data only by capturing an image of the device; Image pixels must prove that the user is taking a picture nearby.

To avoid spooling, all previously presented patterns can be cached-either at the remote server or at the device-and verified with respect to the new data when new data is received. If the same pattern is presented twice, it may be disqualified-as an apparent playback attack (ie each image of the device must have some variation at pixel level). In some arrangements, the appearance of the device changes over time (e.g., by a display providing a periodically changing pattern of pixels), and the presented data is immediately preceding the time interval (e.g., 5 seconds or 5 minutes). It must correspond to the device within.

In a related embodiment, any analog information (appearance, sound and temperature, etc.) can be sensed from the device or its environment and used to establish user proximity to the device. (Incomplete representations of analog information can be used again to detect playback attacks when converted to digital form.)

One simple application of this arrangement is a scavenger hunt-taking a picture of the device provides the user's presence to the device. More practical applications are industrial settings and are related to those who attempt to remotely access devices that are not physically present.

Many variations and hybrids of such arrangements will be apparent to the skilled person from what has been described above.

SIFT

Occasionally, references are made to SIFT techniques. SIFT is an acronym for scale-invariant feature conversion and computer vision technology pioneered by David Lowe and described in various his papers. His paper is published in International Journal of Computer Vision, 60, 2 (2004), 91- "Distinctive Image Features from Scale-Invariant Keypoints" on page 110; And "Object Recognition from Local Scale-Invariant Features," International Conference on Computer Vision, Corfu, Greece (September 1999), 1150-1157, as well as patents 6,711,293.

SIFT works by identifying and describing local image features—and subsequent detection. SIFT features are local and are based on the appearance of an object at points of particular interest and are invariant to image scale, rotation and affine transformation. They are also robust to changes in lighting, noise and some changes in viewpoint. In addition to these attributes, they are unique, relatively easy to extract, allow for accurate object identification with low mismatchability, and are easy to match against a (large) database of local features. Object description by a set of SIFT features is also robust to partial occlusion; As many as three SIFT features from the object are sufficient to calculate position and pose.

The technique begins by identifying local image features in the reference image-called keypoints. This is done by winding the image with Gaussian blur filters at different scales (resolutions) and determining the differences between successive Gaussian-blurred images. Keypoints are image features with a maximum or minimum difference of Gaussians occurring at multiple scales. (Each pixel of the difference of the Gaussian frame is compared to eight neighbors at the same scale and to corresponding pixels of each of the neighboring scales (eg, nine different scales). The pixel value is compared to all these pixels. If it is the maximum or minimum from these, it is selected as the candidate keypoint.

(The procedure just described is a volve-detection method that detects the space-scale extrema of the scale-localized Laplacian transform of an image. The Gaussian difference is an approximation of this Laplacian operation expressed in the pyramid setup.)

The procedure is typically for example due to having low contrast (and therefore sensitive to noise) or having poorly determined positions along the edge (the difference function of Gaussians has a strong response along the edges, Many candidate keypoints are generated, but most of them are not robust to noise). Unreliable keypoints are screened by performing a detailed fit on candidate keypoints for nearby data for accurate position, scale, and ratio of major curvatures. This rejects keypoints that have low contrast or are poorly located along the edge.

More particularly, this process begins by interpolating nearby data to determine the keypoint location more accurately-for each candidate keypoint. This is often done by Taylor extension using keypoints as the origin to determine an improved estimate of the maximum / minimum position.

The value of the secondary taylor extension can also be used to identify low contrast keypoints. If the contrast is less than the threshold (eg 0.03), the keypoint is discarded.

In order to remove keypoints having strong edge responses but poorly localized, a modification of the corner detection procedure is applied. In brief, this involves calculating the major curvature across the edges and comparing the major curvatures along the edges. This is done by subtracting the eigenvalues of the second Hessian matrix.

If unsuitable keypoints are discarded, the remaining keypoints are evaluated for orientation by the local image slope function. The magnitude and direction of the slope is calculated for all pixels in the neighboring area around the keypoint in the Gaussian blurred image (at the scale of that keypoint). An orientation histogram with 36 bins is then compiled-each bin contains degrees of orientation. Each pixel in the neighborhood contributes to the histogram, and the contribution is weighted by the magnitude of the slope and by a Gaussian where σ is 1.5 times the scale of the keypoint. The peaks of this histogram define the predominant orientation of the keypoint. This orientation data allows SIFT to achieve rotational robustness because the keypoint descriptor can be represented for this orientation.

From the foregoing, a plurality of keypoints with different scales are identified-each with corresponding orientations. This data is invariant to image translation, scale and rotation. The 128 element descriptors are then generated for each keypoint, allowing robustness to the illumination and 3D perspective.

This operation is similar to the orientation evaluation procedure just reviewed. The keypoint descriptor is calculated as a set of orientation histograms for (4 x 4) pixel neighbors. Orientation histograms relate to keypoint orientation, and the orientation data comes from a Gaussian image that is closest to the scale of the keypoint. As before, the contribution of each pixel is weighted by the gradient magnitude, and by a Gaussian where sigma is 1.5 times the scale of the keypoint. The histograms each contain eight bins, and each descriptor contains a 4 x 4 array of 16 histograms around the keypoint. This results in a SIFT feature vector with (4 x 4 x 8 = 128 elements). This vector is normalized to improve the invariance of changes in illumination.

The procedure described above is applied to training images to compile a reference database. The unknown image is then processed as described above to generate keypoint data, and the closest matching image in the database is identified by Euclidean distance-type measurement. (The "best-bin-first" algorithm is commonly used instead of pure Euclidean distance calculation to achieve several orders of magnitude speed improvement.) To avoid false positives, the distance score for best matching is When it is close to the distance score for the next best match-for example 25%-an "no match" output is generated.

To further improve performance, images can be matched by clustering. This identifies the features belonging to the same reference image-allowing the non-clustered results to be considered fake. Hough transform can be used-identify clusters of objects that favor the same object pose.

A paper detailing a specific hardware embodiment for implementing the SIFT procedure is "Parallel Hardware Architecture for Scale and Rotation Invariant Feature Detection" by Bonato et al. In IEEE Trans on Circuits and Systems for Video Tech, Vol. 12, No. 12, 2008. . A block diagram of this arrangement 70 is provided in FIG. 18 (adapted from Bonato).

In addition to the camera 32 generating pixel data, there are three hardware modules 72-74. Module 72 receives the pixels as input from the camera and performs two types of operations: a Gaussian filter and a difference of Gaussians. The former is sent to module 73; The latter is sent to module 74. Module 73 calculates pixel orientation and gradient magnitudes. Module 74 detects keypoints and performs stability checks to ensure that the keypoints are reliable when identifying features.

Software block 75 (executed on Altera NIOS II field programmable gate arrays) generates descriptors for each feature detected by block 74 based on the pixel orientation and gradient magnitudes generated by block 73. Create

In addition to running different modules simultaneously, there is parallelism within each hardware block. An example implementation of Bonato handles 30 frames per second. The cell phone implementation may run somewhat slower at least 10 fps in initial generation.

The reader refers to the Bonato paper for other details.

An alternative hardware architecture for existing SIFT technologies was published in October 2004 in Proc. of Int. It is detailed in "Vision Based Modeling and Localization for Planetary Exploration Rovers" by Se et al. At the Astronautical Congress (IAC).

Another arrangement is detailed in "What is That? Object Recognition from Natural Features on a Mobile Phone" by Bonze, Mobile Interaction with the Real World, 2009. Henze et al. Use techniques by Nister et al. And Schindler et al. To expand the use of objects that can be recognized through the use of tree methods (eg, “Scalable” by Nister et al. In 2006 proc. Of Computer Vision and Pattern Recognition. Recognition with a Vocabulary Tree ", and" City-Scale Location Recognition "by Schindler et al. In 2007 Proc. Of Computer Vision and pattern Recognition).

Implementations described above may be utilized on cell phone platforms, or processing may be distributed between the cell phone and one or more remote service providers (or may be implemented using any image-processing execution-phone).

Published patent application WO07 / 130688 relates to a cell phone-based implementation of SIFT, wherein local descriptor features are extracted by the cell phone processor and sent to a remote database for matching to a reference library.

SIFT is probably the best known technique for generating powerful local descriptors, but there are others that may be more or less appropriate depending on the application. They GLOH (2005 years IEEE Trans Pattern Anal Mach Intell, the "Performance Evaluation of Local Descriptors" due 10 No. 27 1615-1630 Mikolajczyk side note...); And SURF (side 2006 Eur Conf on Computer Vision (1) 404-417 due Bay.. "SURF: Speeded Up Robust Features"note); In addition, 2007 Proc. of the 6th IEEE and ACM Int. Symp. "Efficient Extraction of Robust Image Features on Mobile Devices" by Chen et al in On Mixed and Augmented Reality; And October 2008 ACM Int. Conf. In the on Multimedia Information Retrieval, "Outdoors Augmented Reality on Mobile Phone Using Loxel-Based Visual Feature Organization" by Takacs et al. A survey of local descriptor features was published in 2005 in IEEE Trans. On Pattern Analysis and Machine Intelligence are provided in "A Performance Evaluation of Local Descriptors" by Mikolajczyk et al.

The Takacs paper discloses that the image matching speed is greatly increased by limiting a large number of reference images (matches derived therefrom) to those geographically close to the user's current location (eg within 30 meters). Applicants believe that it may be advantageous for the universe to be limited to specialized domains such as faces, foodstuffs, houses, etc.-by user selection or otherwise.

Additions to Audio Applications

Voice conversations on mobile devices naturally define the structure of the session, providing a significant amount of metadata that can be leveraged for prioritizing audio keyvector processing (most management information in the form of identified callers, geographic locations, etc.). ).

If a call is received without involving CallerID information, this may trigger the processing of voice pattern matching with past calls that are still in the voicemail box or where keyvector data for them is preserved. (Google Voice is a long term storage of potentially useful voice data for recognition or matching purposes.)

If the origin geography of the call can be identified but is not a familiar number (e.g., it is also a number generally received in the user's contact list), in view of the origin geography-functional blocks for speech recognition can be called. For example, when it is foreign, speech recognition in the language of that country may be initiated. When the receiver receives the call, simultaneous voice-to-text conversion to the user's native language may be initiated and displayed on the screen to assist the conversion. If geography is domestic, recall of local dialect / specific accented speech recognition libraries may allow for easier handling of southern-specific tonal tone or Boston accents.

Once the conversation has been initiated, prompts based on speech recognition can be provided on the cell phone screen (or the like). When the speaker on the far end of the connection begins discussing a particular subject, the resulting text is used to generate native language queries to reference sites such as Wikipedia, find a local user's calendar to check availability, and copy shopping lists. Can leverage.

Beyond the evaluation and processing of speech during the session, other audio can be analyzed as well. If the user on the far end of the conversation chooses not to or cannot do local processing and keyvector generation, this can be accomplished on the local user's handset, allowing remote experiences to be shared locally.

It is clear that all of the above are equally valid for video calls, and that both audio and visual information can be analyzed and processed into keyvectors.

Public image for individual processing

Most of the above discussion involved images captured by personal devices such as mobile phones. However, the principles and arrangements discussed are also applicable to other images.

Consider the problem of finding parked cars in crowded runner windows. The owner of the parking lot can set up one or more pole-mounted cameras to capture an image of the parking lot at a very advantageous point. Such an image may be made available-generally (eg by a file or page downloaded from the Internet), or locally (eg by a file or page downloaded from a local wireless network). Individuals can acquire images from these cameras and analyze them for individual uses-such as finding where their vehicle is parked. For example, a user's mobile phone can download one or more images and apply image processing (machine vision) techniques as described above to recognize a person's red Honda Civic and thus find it in a parking lot (or a perfect match). If not found, if multiple matches are found, then multiple candidate positions are identified).

In a variant arrangement, the user's mobile phone simply provides a template of data that characterizes the desired vehicle to a web service (eg, operated by the owner of the parking lot). The web service then analyzes the available images to identify candidate vehicles that match the data template.

In some arrangements, the camera may be mounted on an adjustable gimble and zoom lenses may be mounted to provide pan / tilt / zoom controls. (One such camera is the Axis 215 PTZ-E.) These controls can be made accessible to users-capturing images from specific parts of the parking lot for analysis (I know Macy's is parked). Or, if analysis of another image has been performed, allow users to adjust the camera to better distinguish between candidate matching vehicles. (To prevent abuse, camera control privileges can only be provided to authorized users. One way to establish a permit is a parking slip by the user, issued when the user's vehicle first enters the parking lot. This slip is shown on the user's mobile phone camera and printed information (eg alphanumeric, bar code, watermark, etc.) indicating that the user has parked in the parking lot within a certain period of time (eg 12 hours ago). May be analyzed (eg, by a server associated with the parking lot).

Instead of (or in addition to) a pole-mounted camera provided by the parking lot owner, a similar "find my vehicle" function can be achieved through the use of crowd-sourced images. If each of the individual users has a "find my vehicle" application that processes the image captured by their mobile phones to find their respective vehicles, the image captured in this way can be shared for the advantages of others. . Thus, user "A" may roam the aisles in search of a particular vehicle, user "B" may likewise roam elsewhere in the parking lot, and image supplies from each may be shared. Thus, user B's mobile phone can find B's vehicle in the image captured by user A's mobile phone.

This collected image can be stored in an archive accessible through the local wireless network, from which the image is removed after a set period of time, for example two hours later. Preferably, geographic location data is associated with each image so that a matched vehicle with an image captured by a different user can be physically found in the parking lot. This image archive can be maintained by the parking lot owner (or by the owner's contracted service). Alternatively, images can be sourced from elsewhere. For example, mobile phones can automatically post a captured image-or in response to user commands-for storage in one or more online archives such as Flickr, Picasa, and the like. Your "Find My Car" application is geographically stunning (such as an image captured within a user's specific distance or reference location, such as within 10 or 100 yards-or an object depicted-) and also temporarily approximated (e.g., For example, one or more such archives may be queried for an image captured within a particular previous time interval, such as within the past 10 or 60 minutes. Such analysis of the third party image may serve to find the user's vehicle.

Expanding the ranges slightly, now consider that all cameras in the world where information is now publicly accessible (the rapidly increasing highway cameras are just one example) are simply perceived as "data" of ever-changing web pages. (Images from cameras' secret networks, such as security cameras, can be made available with certain privacy guards or other cleansing of sensitive information.) In the past, "data" was dominated by "text" Triggered them, triggered search engines, triggered Google. However, using a highly distributed camera network, most ubiquitous data types become pixels. In addition to positional and temporal data, along with some data structures (eg, may include keyvectors in standardized formats—time / position) and other techniques detailed herein, a new class of search Steps are set up, where providers are sufficiently supplied with some form of keyvector classifications and time / location, both of which are the basis of new search components-compiling and / or compiling a large collection of current and past public perspectives. Classify (or index). Text search is significantly reduced, and new query paradigms-visual stimulus and keyvector properties-are emerging.

Other comments

Although the principles of our original work have been described and illustrated with reference to the illustrated examples, it will be appreciated that the technology is not so limited.

For example, although reference has been made to cell phones, it will be appreciated that the present technology finds utility with all manner of devices-both portable and fixed. PDAs, configurators, portable music players, desktop computers, laptop computers, tablet computers, notebooks, ultraportables, wearable computers, servers and the like described above The principles can all be used. In particular, cell phones contemplated include the Apple iPhone, and cell phones that conform to Google's Android specification (eg, G1 phones manufactured for T-Mobile by HTC Corp.). The term "cell phone" (and "mobile phone") should be interpreted to encompass all such devices, even if they are not strictly cellular or telephone.

(The details of the iPhone, including the touch interface, are provided in Apple published patent application 20080174570.)

The design of cell phones and other computers referenced in this disclosure is familiar to the skilled person. In general terms, each may include one or more processors, one or more memories (eg, RAM), storage (eg, disk or flash memory), user interface (eg, keypad, TFT LCD or OLED). May include software instructions to provide a graphical user interface with a display screen, touch or other gesture sensors, camera or other optical sensor, compass sensor, 3D magnetometer, 3-axis accelerometer, microphone, etc.), these elements Interfaces (e.g. buses) and interfaces to communicate with other devices (wireless like GSM, CDMA, W-CDMA, CDMA2000, TDMA, EV-DO, HSDPA, WiFi, WiMax, or Bluetooth). And / or wired, such as through an Ethernet local area network, T-1 internet connection, etc.).

The arrangements detailed herein can also be utilized in portable monitoring devices, such as phaser-sized devices that sense ambient media for viewer research purposes, such as Personal People Meters (PPMs) (eg See, eg, Nielsen Patent Publication 20090070797 and Arbitron Patents 6,871,180 and 7,222,071. The same principles can also be applied to different forms of content that can be provided to a user online. In this regard, reference is made to Nielsen patent application 20080320508, which details a network-connected media monitoring device.

At the beginning of this specification, the relevance to the transferee's previous patent applications was noted, but can be repeated. These disclosures should be interpreted as a whole and read in concert. Applicants intend that each feature be combined with the features of others. Thus, for example, the signal processing disclosed in applications 12 / 271,772 and 12 / 490,980 can be implemented using the architectures and cloud arrangements detailed herein, and crowd-sourced databases may cover flow user interfaces. And other features described above in the '772 and' 980 applications can be incorporated into implementations of the presently disclosed techniques. Etc. Accordingly, it should be understood that the methods, elements, and concepts disclosed herein are combined with the methods, elements, and concepts described above in these related applications. Some have been specifically described herein, but most are not because of the large number of substitutions and large combinations. However, implementation of all such combinations is straightforward from the disclosures provided to the skilled person.

Elements and disclosures in the different embodiments disclosed herein are also meant to be interchanged and combined. For example, the disclosures described above in the context of FIGS. 1-12 may be used in the arrangements of FIGS. 14-20 and vice versa.

The processes and system components detailed herein include microprocessors, graphics processing units (GPUs such as the nVidia Tegra APX 2600), digital signal processors (eg, Texas Instruments TMS320 series devices), and the like. Can be implemented as instructions for a computing device that includes general purpose processor instructions for various programmable processors. These instructions may be implemented as software, firmware, or the like. These instructions also include programmable logic devices, FPGAs (eg, well-known Xilinx Virtex series devices), FPOAs (eg, well-known PicoChip devices) and special purpose circuits-digital, analog and It can be implemented in various types of processor circuits, including mixed analog / digital circuits. Execution of the instructions may be distributed among processors and / or in parallel across processors within a device or across a network of devices. Conversion of content signal data may also be distributed among different processors and memory devices. References to "processors" or "modules" (such as Fourier transform processor or FFT module, etc.) should be understood to refer to functionality rather than requiring a particular form of implementation.

References to FFTs should also be understood to include inverse FFTs and associated transforms (eg, DFT, DCT, their respective inverses, etc.).

Software instructions for implementing the functions described above can be easily written by technicians from the techniques provided herein, for example C, C ++, Visual Basic, Java, Pysan, Tcl, Perl, Scheme, Ruby Or the like. Cell phones and other devices in accordance with certain implementations of the present technology may include software modules for performing different functions and operations. Known artificial intelligence systems and techniques may be utilized to make the inferences, conclusions, and other decisions noted above.

In general, each device includes operating system software that provides interfaces and general purpose functions for hardware resources, and may also include application software that may be selectively called by a user to perform particular tasks desired. have. Known browser software, communication software, and media processing software can be adapted for many of the uses detailed herein. Software and hardware configuration data / instructions are generally stored as instructions in one or more data structures carried by tangible media such as magnetic and optical disks, memory cards, ROM, and the like, that can be accessed over a network. Some embodiments may be implemented as an embedded system—a special purpose computer system in which operating system software and application software are indistinguishable to the user (eg, as is common in basic cell phones). The functionality detailed herein may be implemented in offrating system software, application software and / or embedded system software.

Different functionality can be implemented on different devices. For example, in a system in which a cell phone communicates with a server of a remote service provider, different tasks may be executed exclusively by one device or another device, or execution may be distributed among the devices. Extraction of eigenvalue data from an image is just one example of such a task. Thus, the description of operation as being performed by a particular device (eg, cell phone) is illustrative rather than limiting; It should be understood that the execution of an operation shared by or between other devices (eg, a remote server) is explicitly contemplated. (Furthermore, more than two devices may generally be utilized. For example, the service provider refers some tasks, such as image retrieval, object segmentation, and / or image classification, to servers dedicated to those tasks. )

(In the same way, the description of the data stored on the particular device is also exemplary; the data can be stored anywhere: local device, remote device, in the cloud, distributed, etc.)

The operations need not be executed exclusively by specially-identifiable hardware. Rather, some operations may be referenced to other services (eg, cloud computing) and participate in execution by more generally anonymous systems. Such distributed systems can be large in size (eg, involving computing resources on the planet) or local (eg, a portable device identifies the surrounding devices via Bluetooth communication and has one in operation). When relating the above surrounding devices-as contribution data from local geometry; in this regard, see patent 7,254,406 to Beros.)

Similarly, while specific functions have been described above as being executed by specific modules (eg, control processor module 36, pipe manager 51, query router and response manager of FIG. 7, etc.), in other implementations, Such functions may be executed by other modules or by application software (or all together).

The reader is cautioned that certain discussions consider arrangements where most image processing is performed on the cell phone. In such arrangements, external resources are used more as resources for data (eg, Google) than for image processing tasks. Such arrangements can be naturally implemented using the principles discussed in other sections, with some or all of the hardcore mass processing of image-related data being referenced to external processors (service providers).

Likewise, although this disclosure has described the specific order of certain combinations and operations of elements in the illustrated embodiments, other contemplated methods may reorder the operations, and other contemplated combinations may omit some elements and other You will see that you can add elements.

Although disclosed as complete systems, subcombinations of the above-described arrangements are also considered separately.

In exemplary embodiments, reference was made to the Internet. In other embodiments, other networks-including secret networks of computers-can also be utilized instead.

The reader will appreciate that different names are sometimes used when referring to similar or identical components, processes, and the like. This is in part due to the development of this patent specification over a nearly year-long process-using terms that have changed over time. Thus, for example, "visual query packet" and "keyvector" may both indicate the same thing. Similar for other terms.

In some modes, cell phones utilizing aspects of the present technology can be considered as observation state machines.

Although primarily described above in the context of systems that perform image capture and processing, corresponding arrangements are equally applicable to systems that capture and process audio, or capture and process both image and audio.

Some processing modules in an audio-based system may naturally be different. For example, audio processing generally relies on threshold band sampling (per human auditory system). Capstrum processing (DCT of power spectrum) is also frequently used.

An exemplary processing chain may include a band pass filter that filters the audio captured by the microphone to remove low and high frequencies, for example to leave the band 300-3000 Hz. The decimation station may follow (eg, reduce the sampling rate from 40K samples / second to 6K samples / second). The FFT can then follow. Power spectral data can be calculated by squared output coefficients from the FFT (they can be grouped to perform critical band segmentation). Thereafter, the DCT may be executed to generate capstrum data. Outputs from any of these stages can be sent to the cloud for application processing such as speech recognition, language translation, anonymization (returning the same utterances with different voices), and the like. Remote systems may also respond to commands captured by the microphone and spoken by the user, for example, to control other systems, to supply information for use by other processes, and the like.

It will be appreciated that the above described processing of content signals involves the conversion of these signals in various physical forms. Images and video (in the forms of electromagnetic waves depicting physical objects and moving through physical space) may be captured from physical objects using cameras or other capture device, or may be generated by computing devices. Similarly, audio pressure waveforms traveling through a physical medium can be captured using an audio transducer (eg, a microphone) or converted into an electrical signal (digital or analog form). While these signals are typically processed in electronic and digital form to implement the components and processes described above, they are also captured, processed, transmitted and stored in other physical forms, including electronic, optical, magnetic and electromagnetic forms. Can be. The content signals are transformed for various uses in various forms while processing and generating various data structure representations of the signals and related information. Now, data structure signals in the memory are converted for manipulation during retrieval, sorting, reading, writing and retrieval. The signals are also converted for capture, transmission, storage and output through a display or audio converter (eg, speakers).

In some embodiments, an appropriate response to the captured image can be determined with reference to data stored on the device-without reference to any external resources. (The registry database used in many operating systems is where response-related data for specific inputs can be specified.) Alternatively, information can be sent to the remote system-this is to determine the response. .

The drawings, not particularly identified above, illustrate aspects of the details or example embodiments of the disclosed technology.

The information transmitted from the device may be raw pixels, may be an image in compressed form, may be a converted copy of the image, features / metrics extracted from the image data, and so forth. All can be regarded as image data. The receiving system can recognize the data type or can be explicitly identified to the receiving system (e.g., bitmap, eigenvectors, Fourier-Meline transform data, etc.) and the system inputs to determine how to process. One of the data types can be used.

If the transmitted data is full image data (raw or compressed form), then there will be essentially no duplicates in the packets received by the processing system-essentially every picture is somewhat different. However, if the originating device performs processing on the entire image to extract features or metrics, etc., the receiving system may sometimes receive (or almost do) the same packet as initially encountered. In such a case, the response to that "snap packet" (also referred to as a "pixel packet" or "keyvector") may be recalled from the cache-rather than a new one being determined. (If available and applicable, response information may be modified according to user preference information.)

In certain embodiments, to ensure that a known user is operating the device, it may be desirable for the capture device to include some form of biometric authentication, such as a fingerprint reader integrated with a shutter button.

Some embodiments may capture several images of the subject from different views (eg, video clip). Thereafter, algorithms for synthesizing the 3D model of the imaged object can be applied. From this model, new views of the subject-views that may be more suitable as a stimulus to the processes described above (eg, avoiding occlusion of the foreground object) can be derived.

In embodiments using descriptors of text, it is sometimes desirable to augment descriptors with synonyms, lower words (more specific terms) and / or higher words (more general terms). These can be obtained from a variety of sources, including a WordNet database compiled by Princeton University.

While most of the embodiments described above have been in the context of a cell phone that presents image data to a service provider to trigger a corresponding response, the technique is generally more applicable-whenever processing of images and other content occurs.

The focus of this disclosure has been on the image. However, these techniques are also useful for audio and video. The technique described above is particularly useful in User Generated Content (UGC) sites such as YouTube. Videos are often downloaded with little metadata. Various techniques are applied (eg, reading watermarks; calculating fingerprints, human reviewers, etc.) with varying degrees of uncertainty to identify this, and this identification metadata is stored. Other metadata is accumulated based on the profiles of users watching the video. Still other metadata may be collected from later user comment frames posted about the video. (UGC-related arrangements intended by the applicant to be included in the present technology are disclosed in published patent applications 20080208849 and 20080228733 (Digimarc), 20080165960 (TagStory), 20080162228 (Trivid), 20080178302 and 20080059211 (Attributor), 20080109369 (Google), 20080249961 (Nielsen), and 20080209502 (MovieLabs).) With arrangements such as those described herein above, appropriate advertisement / content organizations can be collected and other improvements to the user's experience can be provided. .

Similarly, the technology can be used with audio captured by the user device and captured speech recognition. Information collected from any captured information (eg, OCR'd text, decoded watermark data, recognized speech) can be used as metadata for the purposes detailed herein.

Multi-media applications of this technology are also contemplated. For example, the image may be pattern-matched or GPS-mounted to identify a set of similar images in flicker. Metadata descriptors may be collected from a set of similar images or used to query metadata including audio and / or video. Thus, a user who captures and submits an image of a trail marker on the Appalachian Trail (FIG. 38) may trigger the download of an audio track from Aaron Copeland's “Appalachian Spring” orchestra suitable for his cell phone or home entertainment system. Can be. (See, eg, patent publication 20070195987 regarding transmitting content to different destinations that may be associated with a user.)

Repeated references to GPS data were made. It should be understood that there is a lack of work for any location-related information; It does not need to be derived from global positioning system satellite deployments. For example, another technique suitable for generating location data relies on wireless signals (eg, WiFi, cellular, etc.) that are generally exchanged between devices. Given several communication devices, the signals themselves-and the incomplete digital clock signals that control them-both form a reference system from which very accurate time and location can be extracted. Such techniques are detailed in published international patent publication WO08 / 073347. The technician is responsible for location-estimation techniques based on arrival time techniques and locations such as broadcast wireless and television towers (as provided by Rosum) and WiFi nodes (as provided by Skyhook radio and utilized in an iPhone). You will be familiar with several other location- estimation techniques, including based location- estimation techniques.

Geographic location data generally includes latitude and longitude data, but may alternatively include somewhat or different data. For example, it may include orientation information, such as compass direction provided by the magnetic field, or tilt information provided by gyroscopic or other sensors. It may also include altitude information as provided by digital altimeter systems.

A reference to Apple's Bonjour software was made. Bonjour is Apple's implementation of Zeroconf, a service discovery protocol. Bonjour looks for devices on the local network and identifies the services that each provides using multicast domain name system service records. This software is built on the Apple MAC OS X operating system and is also included in the Apple "remote" application for the iPhone-used to establish connections to iTunes libraries over WiFi. Bonjour services are implemented at the bulk application level using standard TCP / IP calls rather than in the operating system. Apple created the source code for the Bonjour multicast DNS responder-a core component of service discovery-available as a Darwin open source project. The project provides source code for creating responder daemons for a wide range of platforms including Mac OS X, Linux, * BSD, Solaris, and Windows. In addition, Apple offers not only Java libraries, but also a so-called user-installable set of services for Bonjour for Windows. Bonjour may be used in various embodiments of the present technology by relating communications between devices and systems.

(Other software may alternatively or additionally be used to exchange data between devices. Examples include device profiles for Universal Plug and Play (UPnP) and its successor web services. DPWS: Devices Profile for Web Services (DPWS), which are other protocols that implement zero-configuration networking services, through which devices can connect, identify themselves, and advertise available capabilities to other devices. And share content, etc.)

As noted earlier, artificial intelligence techniques can play an important role in embodiments of the present technology. A recent participant in this field is the Wolfram Alpha product by Wolfram Research. Alpha calculates responses and visualizations in response to input configured by referencing a knowledge base of auxiliary data. Information collected from metadata analysis or semantic search engines may be provided to the Wolfram Alpha product to provide response information back to the user, as detailed herein. In some embodiments, a user is involved in the presentation of such information, such as by constructing a query from terms and other primitives collected by the system, selecting from a menu of different queries configured by the system, and the like. Additionally or alternatively, response information from the Alpha system may be provided as input to other systems, such as Google, to identify other response information. Wolfram's patent publications 20080066052 and 20080250347 further detail aspects of the technology.

Another recent technology introduction is Google Voice (based on the initial venture's GrandCentral product), which provides a number of improvements over conventional telephone systems. Such features may be used with certain aspects of the present technology.

For example, voice-text transcription services provided by Google Voice use microphones in a user's cell phone to capture ambient audio from the speaker's environment and generate corresponding digital data (e.g., ASCII information). Can be utilized for The system can present this data to services such as Google or Wolfram Alpha to obtain relevant information, which can then provide the relevant information back to the user-either by a screen display or by voice. Similarly, speech recognition possible by Google Voice can be used to provide an interactive user interface to cell phone devices, whereby the features of the techniques described above are selectively invoked and controlled by words spoken. Can be.

In another aspect, when the user captures content (audio or visual) with the cell phone device, and the system utilizing the presently disclosed technology returns a response, the response information may be converted from text to voice and the user in Google Voice Will be forwarded to your voicemail account. The user can access this data store from any information or from any computer. The stored voicemail may be reviewed in audible form, or the user may choose, for example, instead of reviewing a copy of the text provided on the cell phone or computer screen.

(The aspects of Google Voice technology are detailed in patent application 20080259918.)

More than a century of history has made users accustomed to thinking of phones as communication devices that receive audio at point A and deliver audio at point B. However, aspects of the present technology can be utilized with very different effects. Audio-in and audio-out are becoming the old paradigm. In accordance with certain aspects of the present technology, phones also receive communication (or other stimuli) at point A to communicate text, voice, data, image, video, fragrance or other sensory experience at point B. Devices.

Instead of using the technology described above as a querying device, where a single phone acts as both an input and an output, the user can have the content delivered to one or several destination systems in response to the query-this is the originating phone It may or may not include. (Recipient (s) may be selected by known UI techniques, including keypad input, scrolling through recipients' menus, speech recognition, etc.)

A simple example of this usage model is a person using a cell phone to capture a picture of a rose flower plant. In response to the user's instructions, an image, which is augmented by the synthesized fragrance of that particular kind of rose, is delivered to the user's girlfriend. (Arrangements for mounting computer devices for propagating programmable fragrances are known, for example the techniques described in iSmell provision by Digiscents and patent documents 20080147515, 20080049960, 20060067859, WO00 / 15268 and WO00 / 15269. .) The stimuli captured by one user at one location may cause the delivery of related empirical stimuli to different users in different but different locations.

As noted, the response to the visual stimuli may include one or more graphical overlays (bobbles) provided on the cell phone screen—top image data from the cell phone camera. The overlay may be registered geometrically as features in the image data and may be affine-distorted in response to the affine distortion of the object depicted in the image. Graphical features can be used to interest the bauble, such as sparks emitted in that area or flashing / moving visual effects. Such techniques are further detailed, for example, in Digimarc's patent publication 20080300011.

Such graphical overlay may include menu features, which allows the user to interact to perform the desired functions. In addition or in the alternative, the overlay may include one or more graphical user controls. For example, several different objects can be recognized within the field of view of the camera. The overlay associated with each may be graphical, which may be touched by the user to obtain information or trigger a function associated with that respective object. Overlays can be considered as visual flags-such as by user tapping on that location of the screen, or by rotating the area with a finger or stylus, etc., to interact with such graphical features. It draws attention to the availability of information that can be accessed through. As the user changes the view of the camera, different baubles may appear—tracking the movement of different objects in the underlying reality field of view image and prompting the user to explore the associated assistance information. Again, the overlays are preferably corrected at right angles to the affine-correction projection on the associated real-world features. (The pose estimation of the objects as imaged in the real world, where appropriate spatial registration of the overlays is determined, is preferably executed locally, but can be referenced to the cloud depending on the application.)

Objects can be recognized, tracked, and fed back by the operations described above. For example, the local processor can perform object analysis and initial object recognition (eg, proto-objects in the list). The cloud processes can complete the recognition operations and create the appropriate interactive portals registered at right angles on the display scene (registration can be executed by the local processor or the cloud).

In some aspects, it will be appreciated that the present technology may operate in the real world-on a cell phone-as a graphical user interface.

In early implementations, general-purpose visual query systems of the described type would be relatively crude and not very good looking. However, by supplying a trickle (or torrent) of keyvector data back to the cloud for achievement and analysis (along with information about user behavior based on such data), they are created by templates and other training models. Data foundations can be established that allow subsequent generations of these systems to become highly intuitive and responsive when visual stimuli are provided. (The trickle sometimes catches small pieces of information about how users work with the device, what they do, what they don't do, what choices the user makes based on what stimuli, what stimuli are involved, and feed them to the cloud. May be provided by a subroutine on the local device.)

Reference has been made to touchscreen interfaces in the form of gesture interfaces. Another form of gesture interface that can be used in certain embodiments operates by sensing movement of the cell phone-by tracking the movement of features in the captured image. Other information regarding these gesture interfaces is detailed in Digimarc's patent 6,947,571.

Watermark decoding may be used in certain embodiments. Techniques for encoding / decoding watermarks are described, for example, in Digimarc's patents 6,614,914 and 6,122,403; In Nielsen's patents 6,968,564 and 7,006,555; And Arbitron's patents 5,450,490, 5,764,763, 6,862,355, and 6,845,360.

Digimarc has various other patent applications related to this subject matter. See patent publications 20070156726, 20080049971, and 20070266252, and pending application 12 / 125,840 to Sharma et al., Filed May 22, 2008.

Google's book-scanning patent 7,508,978 details certain principles useful in this context. For example, the '978 patent discloses that by projecting a reference pattern on a non-planar surface, the surface topology can be identified. An image captured from this surface can then be processed to renormalize it so that it appears to originate from a flat page. This refinement can also be used with the object recognition arrangements described herein above. Similarly, Google's patent application 20080271080, which details visions for interacting with next generation television, also details useful principles along with the techniques described above.

Examples of audio fingerprinting are detailed in patent publications 20070250716, 20070174059 and 20080300011 (Digimarc), 20080276265, 20070274537 and 20050232411 (Nielsen), 20070124756 (Google), 7,516,074 (Auditude), and 6,990,453 and 7,359,889 (both Shazam). Examples of image / video fingerprinting are detailed in patent publications 7,020,304 (Digimarc), 7,486,827 (Seiko-Epson), 20070253594 (Vobile), 20080317278 (Thomson), and 20020044659 (NEC).

While certain aspects of the techniques described above involve processing multiple images to collect information, by allowing the majority of people (and / or automated processes) to consider a single image (eg, crowd-sourcing) It will be appreciated that relevant results may be obtained. Greater information and utility can be achieved by combining these two general ways.

It is meant that the depictions are exemplary only and not intended to be limiting. For example, when a single database can be used they sometimes show multiple databases (or vice versa). Likewise, some links between depicted blocks are not shown for clarity.

Context data can be used throughout the above-described embodiments to further improve operation. For example, the processing may be whether the originating device is a cell phone or a desktop computer; Whether the ambient temperature is 30 degrees or 80 degrees; The location of the user and other information characterizing the user; And so on.

While the embodiments described above often provide candidate results / actions as a series of cached displays on a cell phone screen that the user can switch quickly, this need not be the case in other embodiments. A more conventional single-screen presentation that provides a menu of results can be used-the user can press the keypad digits or highlight the desired option to select. Or the bandwidth can be increased enough so that the same user experience can be provided without caching or buffering the data locally-rather than being delivered to the cell phone when needed.

Geographic-based database methods are described, for example, in Digimarc's patent publication 20030110185. Other arrangements for navigating and searching execution through image collection are shown in patent publications 20080010276 (Executive Development Corp.) and 20060195475, 20070110338, 20080027985, 20080028341 (Microsoft's Photosynth work).

It is not possible to explicitly list the myriad variations and combinations of the techniques described herein. Applicants have recognized and intended that the concepts of this specification may be combined, replaced, and exchanged-two of and between them, as well as with concepts known from the cited prior art. Moreover, it will be appreciated that the techniques described above may be included with other techniques—current and coming soon—to advantageously implement.

The reader is believed to be familiar with the documents (including patent documents) referenced herein. In order to provide a comprehensive disclosure without unduly elongating this specification, applicants include these references referenced above by reference. (These documents are incorporated in their entirety even when cited above with specific disclosures.) These references may be incorporated into the arrangements detailed herein, and the techniques and disclosures set forth herein herein. Disclosed are techniques and disclosures in which the contents may be incorporated.

As can be appreciated, the present specification has described a myriad of novel arrangements. (Because of actual constraints, many such arrangements have not yet been claimed at the time of the first filing of this application, but applicants try to claim these other points in subsequent applications claiming priority.) Incomplete sampling of some of the original arrangements Is reviewed in the following paragraphs:

In one arrangement, the stimuli captured by the sensor of the user's mobile device are processed, where some processing tasks can be executed on the processing hardware of the device and other processing tasks are on one processor-or multiple-remote from the device. Wherein the at least: mobile device power considerations; Required response time; Routing constraints; State of hardware resources within the mobile device; Connection status; Geographic considerations; Risk of pipeline stalls; Information about the remote processor, including its readiness, processing speed, cost, and attributes not critical to the user of the mobile device; And based on considerations of two or more different factors extracted from the set that includes the association to other processing tasks of the task, the determination as to whether the first task should be executed on the device hardware or on the remote processor is automated. And in some circumstances, the first task is executed on the device hardware, and in other environments, the first task is executed on the remote processor. In addition, in such an arrangement, the determination is based on a score that depends on the combination of parameters related to at least some of the listed considerations.

In one arrangement, the stimuli captured by the sensor of the user's mobile device are processed, where some processing tasks can be executed on the processing hardware of the device, and other processing tasks are on a processor-or a plurality of processors-remote from the device. Wherein the at least: mobile device power considerations; Required response time; Routing constraints; State of hardware resources within the mobile device; Connection status; Geographic considerations; Risk of pipeline stalls; Information about the remote processor, including its readiness, processing speed, cost, and attributes not critical to the user of the mobile device; And based on considerations of two or more different factors extracted from the set including the association to other processing tasks of the task, an order in which the set of tasks should be executed; In some circumstances, the set of tasks is executed in a first order, and in other environments, the set of tasks is executed in a second different order. In addition, in such an arrangement, the determination is based on a score that depends on the combination of parameters related to at least some of the listed considerations.

In one arrangement, the stimuli captured by the sensor of the user's mobile device are processed, where some processing tasks can be executed on the processing hardware of the device and other processing tasks are on one processor-or multiple-remote from the device. Wherein the packets are utilized to convey data between processing tasks, the contents of the packets being at least: mobile device power considerations; Required response time; Routing constraints; State of hardware resources within the mobile device; Connection status; Geographic considerations; Risk of pipeline stalls; Information about the remote processor, including its readiness, processing speed, cost, and attributes not critical to the user of the mobile device; And based on consideration of two or more different factors extracted from the set that includes the association to other processing tasks of the task; In some circumstances, packets may contain data in a first form, and in other environments, packets may contain data in a second form. In addition, in such an arrangement, the determination is based on a score that depends on the combination of parameters related to at least some of the listed considerations.

In one arrangement, the stage provides data services to users over the network, and the network is configured to inhibit the use of electronic imaging by the users while on stage. Also in such an arrangement, the suppression is done by limiting the transmission of data from user devices to specific data processing providers outside the network.

In one arrangement, the mobile communication device with image capture capability includes a pipelined processing chain for executing the first operation, the control system has a mode for testing image data by executing the second operation, and the second The operation is computationally simpler than the first operation, and the control system applies the image data to the pipelined processing chain only if the second operation produces an output of the first type.

In one arrangement, the mobile phone is equipped with a GPU for facilitating the rendering of graphics for display on the mobile phone screen, for example for games, and the GPU is also utilized for machine vision applications. Also in this arrangement, machine vision applications include face detection.

In one arrangement, a plurality of social-associated mobile devices maintained by different individuals cooperate to perform machine vision operations. Also in this arrangement, a first of the devices performs an operation for extracting facial features from the image, and a second of the devices performs template matching on the extracted facial features generated by the first device. Run

In one arrangement, a speech recognition operation is performed on incoming video or audio from a phone call to identify the caller. Also in this arrangement, the video recognition operation is performed only if the incoming call is not identified by the CallerID data. In this arrangement, the voice recognition operation also includes a reference to data corresponding to one or more initially-stored voice messages.

In one arrangement, voice from an incoming video or phone call can be recognized and text data corresponding thereto is generated when the call is processed. In addition, in such an arrangement, the incoming call may be associated with a particular geography, which may be considered in recognizing speech. Also in this arrangement, the text data is used to query the data structure for assistance information.

One arrangement is to migrate overlay baubles onto the mobile device screen so that they are derived from both local and cloud processing. Also in this arrangement, overlay baubles are tuned according to user preference information.

In one arrangement, the visual query data is processed in a distributed manner between the user's mobile device and the cloud resources to generate a response, and the associated information is preserved in the cloud so that subsequent visual query data can generate a more intuitive response. So that it is processed.

In one arrangement, the user may be (1) billed for data processing services by the vendor, or alternatively (2) may be provided free services from the vendor if the user takes certain actions in relation to it, or even You can receive credit.

In one arrangement, the user receives commercial interest in an exchange for providing promotional content-as sensed by the mobile device delivered by the user.

In one arrangement, the first user allows the second party to consume credits of the first user or to incur costs by the first user by a social networking connection between the first user and the second party. Also in this arrangement, the social networking web page is configured for the second party to interact upon the consumption of these credits, or such costs.

In an arrangement for a charity fund, the user interacts with a physical object associated with the charity organization to trigger a computer-related process that facilitates user donation to the charity.

In a portable device, it receives input from one or more physical sensors, utilizes processing by one or more local services, and also utilizes processing by one or more remote services, and the software of the device provides one or more abstraction layers. And through which the sensors, local services, and remote services interface to a device architecture to facilitate operation.

In a portable device, it receives an input from one or more physical sensors, processes the input and packages the result into a keyvector form, and transmits the keyvector form from the device. Also, in such an arrangement, the device receives another processed copy of the keyvector from the remote resource to which the keyvector was sent. Also in such an arrangement, the keyvector form is processed-on a portable device or a remote device-according to one or more instructions implied in accordance with the context.

In a distributed processing architecture for responding to physical stimuli sensed by the mobile phone, the architecture utilizes local processing on the mobile phone and remote processing on a remote computer, the two processes being linked by a packet network and an interprocess communication structure. The architecture also includes a protocol through which different processes can communicate, which protocol includes a message passing paradigm with a message queue or conflict handling arrangement. In this arrangement, the sensor data is also provided in the form of driver software packets for one or more physical sensor components, and the packets are placed on an output queue to be uniquely associated with the sensor or commonly associated with a plurality of components. and; Unless a packet is processed remotely, local processes operate on those packets, place the resulting packets back on the queue, and if the packet is processed remotely, it is directed to remote processing by the router arrangement. Lose.

In one arrangement, the network associated with a particular physical place adapts to refer to traffic on the network to automatically distinguish whether or not the set of visitors to that place have a social connection. In addition, such arrangements also include distinguishing the demographic characteristics of the groups. Also, in such an arrangement, the network facilitates ad hoc networking among visitors who are identified as not having a social connection.

In one arrangement, a network comprising computer resources in a public place is dynamically reconfigured according to a predictive model of the behavior of users visiting the place. Also in such an arrangement, network reconfiguration is based in part on the context. Also, in such an arrangement, network reconstruction includes caching specific content. Also in such an arrangement, the reconstruction includes rendering the synthesized content and storing it in one or more computer resources to make it available quickly. This arrangement also includes re-balancing time-sensitive network traffic in anticipation of a temporary increase in traffic from users.

In one arrangement, the advertisement is associated with real-world content, and billing accordingly is assessed based on surveys of exposure to the content-as indicated by sensors in the user's mobile phones. Also, in such arrangements, billing is established through the use of automated auction arrangements.

In one arrangement comprising two objects in a public place, the illumination for the objects is charged differently-based on the nature of the person's proximity to the objects.

In one arrangement, the content is provided to people at a public location, a link exists between the provided content and the supplemental content, and the linked supplemental content is billed according to the demographic attributes of the person from whom the content is provided.

In one arrangement, a temporary electronic license for specific content is provided to a person associated with a person's visit to the public place.

In one arrangement, the mobile phone includes an image sensor connected to both the human vision system processor and the machine vision processor, and the image sensor is coupled to the machine vision processor without passing through the human vision system processor. Also, in this arrangement, the human vision system processing unit includes a white balance correction module, a gamma correction module, an edge enhancement module and / or a JPEG compression module. In this arrangement, the machine vision processing unit also includes an FFT module, an edge detection module, a pattern extraction module, a Fourier-Mellin processing module, a texture classifier module, a color histogram module, a motion detection module, and / or a feature recognition module.

In one arrangement, the mobile phone includes a plurality of stages and image sensors for processing image-related data, and a data driven packet architecture is utilized. Also in this arrangement, the information in the header of the packet determines the parameters applied by the image sensor when the image data is first captured. Also in such an arrangement, the information in the header of the packet determines the processing performed by the plurality of stages for the image-related data carried in the body of the packet.

In one arrangement, the mobile phone cooperates with one or more remote processors to perform image-related processing. Also in such an arrangement, the mobile phone packages the image-the related image into packets-at least some of which contain less than a single frame of image data. Also in this arrangement, the mobile phone routes certain image-related data for processing by the processor in the mobile phone and routes the specific image-related data for processing by the remote processor.

In one arrangement, the mobile phone cooperates with a remote routing system, which collects processed image-related data from the processors for processing by different remote processors and for return to the mobile phone. Serve to distribute image-related data from the mobile phone. In addition, in one arrangement, the mobile phone includes an internal routing system that serves to distribute image-related data to one or more processors inside the mobile phone for processing or to a remote routing system for processing by remote processors. .

In one arrangement, the image-related data from the mobile phone is referenced to the remote processor for processing, and the remote processor is selected by an automated assessment involving multiple remote processors. Also in this arrangement, the evaluation includes reverse auction. Also in this arrangement, output data from the selected remote processor is returned to the mobile phone. Also in this arrangement, the image-related data is processed by the processing module at the mobile phone before being sent to the selected processor. Also in this arrangement, other image-related data from the mobile phone is referenced to a remote processor other than the selected processor.

In one arrangement, the image data is stored in at least one planar data structure of the multi-plane data structure, and the graphical representation of metadata related to the image data is stored in a data structure of another plane. Also in this arrangement, the metadata includes edge map data derived from the image data. Also in this arrangement, the metadata includes information about the faces recognized in the image data.

In one arrangement, the camera-mounted mobile phone comprises (1) rotation from horizontal; (2) rotation after initial time; And (3) data representing at least one of the scale change after the initial time, wherein the displayed data is determined with reference to information from the camera.

In one arrangement, the camera-mounted mobile phone includes first and second parallel processing units, the first processing unit processing image data to be rendered in a perceptual form for use by human viewers, and a mosaic-releasing processor. And at least one of a white balance correction module, a gamma correction module, an edge enhancement module, and a JPEG compression module, wherein the second processor analyzes the image data to derive semantic information therefrom.

In one arrangement, information of two or more dimensions related to the object is provided on the screen of the mobile phone, the operation of the first user interface control providing a sequence of screens providing information related to the object in the first dimension, and the second The operation of the user interface control provides a sequence of screens that provide information related to the object in the second dimension. Also in this arrangement, the object can be changed by manipulating user interface controls while the screen providing the object is displayed. Also, in this arrangement, the object is an image, the first dimension being similar to the image in one of (1) geographic location, (2) appearance, or (3) content description metadata, and the second dimension is the ( There is a similarity with the image in the different one of 1), (2) or (3).

In one arrangement of composing a text message on a camera-mounted portable device, the device is tilted in a first direction to scroll through a sequence of displayed icons-each representing a plurality of letters of the alphabet, and then a plurality of Choose from letters. The arrangement also includes tilting the device in a second direction to select among a plurality of letters. Also in this arrangement, the tilt is sensed with reference to the image data captured by the camera. Also in this arrangement, the tilts of the different characters are due to different meanings.

In one arrangement, the camera-mounted mobile phone functions as a state machine to change aspects of its functionality based on previously acquired image-related information.

In one arrangement, the identification information corresponding to the processor-mounted device is used to identify and identify the corresponding application software, which is then used to program the operation of the mobile phone device, the programmed mobile device being the device. It acts as a controller for. Also in such an arrangement, the device is a thermostat, parking meter, alarm clock, or vehicle. Also, in such an arrangement, the mobile phone device captures an image of the device, and the software causes the user interface for the device to be provided as a graphic overlay on the captured image (optionally, the graphic overlay may pose or pose the device in the captured image) Is provided in the corresponding position or pose).

In one arrangement, the screen of the mobile phone provides one or more user interface controls for the individual device, and the user interface controls on the screen are provided in combination with the phone-captured image of the individual device. Also, in such an arrangement, user interface control is used to issue commands related to the control of an individual device, the screen is pending while the screen signals information corresponding to the command in a first manner, and once in a second manner. Is executed successfully.

In one arrangement, the user interface provided on the screen of the mobile phone is used to initiate a transaction with an individual device when the phone is in physical proximity to the device, the mobile phone is later used for purposes not related to the individual device, Later the user interface is recalled on the screen of the mobile phone to associate in another transaction with the device. Also in this arrangement, the user interface is recalled to associate with other transactions with the device when the mobile phone is remote from the device. In this arrangement, the device also includes a parking meter, a vehicle or a thermostat.

In one arrangement, the mobile phone provides a user interface that allows selection between several user interfaces corresponding to different devices, such that the phone can be used when interacting with a plurality of individual devices.

In one arrangement, using a mobile phone to sense information from a housing of a network-connected device and encrypting the information using a key corresponding to the device, through the use of such information.

In one arrangement, a mobile phone is used to sense information from a device equipped with a wireless device and to transmit relevant information from the mobile phone, and the transmitted data serves to confirm user proximity to the device. Also in this arrangement, such proximity is required before allowing the user to interact with the device using the mobile phone. Also, in this arrangement, the sensed information is analog information.

In one arrangement utilizing a portable electronic device and reconfigurable hardware, when initialized to be ready for use, updated configuration instructions for the hardware are downloaded wirelessly from a remote source and used to configure reconfigurable hardware.

In one arrangement, the hardware processing component of the wireless system base station is utilized to process data related to wireless signals exchanged between the base station and the plurality of associated remote wireless devices, and is also used by the wireless base station for processing by the camera device. It is utilized to process offloaded image-related data. Also, in such an arrangement, the hardware processing component includes one or more field programmable object arrays and the remote wireless devices include mobile phones.

In one arrangement, an optical distortion function is characterized and used to define the geometry of the corresponding virtual correction surface on which the optically distorted image is projected, the geometry neutralizing the distortion of the projected image. Also, in one arrangement, the image is projected onto a virtual surface whose topology is formed to neutralize the distortion present in the image. Also in such arrangements, the distortion includes the distortion introduced by the lens.

In one arrangement, the wireless station receives a service reservation message from the mobile device, the message including one or more parameters of a future service requesting that the mobile device be available at a future time without immediate use; With respect to the service provided to the second mobile device-determining based at least in part on the service reservation message received from the first mobile device, the resource allocation of the wireless station is improved information regarding expected services to be provided to the first mobile device. Is improved.

In one arrangement, a thermoelectric cooling device is coupled to the image sensor of the mobile phone and selectively activated to reduce noise in the captured image data.

In one arrangement, the mobile phone includes first and second wirelessly linked portions, the first portion comprising an optical sensor and a lens assembly and adapted to be carried in a first position relative to the user's body, the second portion Includes a display and a user interface and is adapted to be carried in a second, different location. Also in this arrangement, the second part is adapted to be detachably received in the first part.

In one arrangement, the mobile phone includes first and second wirelessly linked portions and the first wirelessly linked portion includes LED lights that are assembleably detachably coupled to the second wirelessly linked portion. The second wirelessly linked portion includes a display, a user interface, an optical sensor and a lens, wherein the first wirelessly linked portion can be detached from the second wirelessly linked portion, and the second wirelessly linked portion It is arranged to illuminate the object being imaged by the light sensor.

In one arrangement, the camera-mounted mobile phone processes the image data through the selection of a plurality of processing stages, and the selection of one processing stage depends on the attributes of the processed image data output from the previous processing stage.

In one arrangement, conditional branching of different image processing stages is utilized in a camera-mounted mobile phone. Also in such an arrangement, the stages respond to packet data and conditional branching instructions are carried in the packet data.

In one arrangement, the GPU of a camera-mounted mobile phone is utilized to process image data captured by the camera.

In one arrangement, the camera-mounted mobile device detects temporary variations in illumination and takes those variations into operation. In this arrangement, the camera also predicts the future state of the temporarily changing illumination and captures the image camera when the illumination is expected to have the desired state.

In one arrangement, the camera-mounted mobile phone is equipped with two or more cameras.

In one arrangement, the mobile phone is equipped with two or more projectors. Also in such an arrangement, the projectors alternately project patterns on the surface, and the projected patterns are sensed by the camera portion of the mobile phone and used to identify the topology information.

In one arrangement, the camera-mounted mobile phone is then equipped with a project that projects a pattern onto a surface that is captured by the camera, and the mobile phone can identify information about the topology of the surface. It is also used in this arrangement to help identify the object. Also in this arrangement, the identified topology information is used to normalize the image information captured by the camera.

In one arrangement, the camera and projector portions of the mobile phone share at least one optical component. Also, in this arrangement, the camera and projector portions share a lens.

In one arrangement, a camera-mounted mobile phone utilizes a packet architecture to route image-related data between a plurality of processing modules. Also in this arrangement, the packets additionally carry instructions that the processing modules respond to. Also in this arrangement, the phone's image sensor responds to a packet carrying image capture instructions therefor.

In one arrangement, the image capture system of the camera-mounted mobile phone outputs different types of first and second sequence sets in accordance with the automated instructions provided thereto. Also, in such arrangements, the sequence sets differ in size, color or resolution.

In one arrangement, the camera-mounted mobile phone captures a sequence of visual data sets, and the parameters used to capture one of the sets depend on the analysis of the previously-captured data set.

In one arrangement, the camera-mounted mobile phone sends image-related data to one of a plurality of competing cloud-based services for analysis. Also in this arrangement, the analysis includes face recognition, optical character recognition or FFT motion. Also in this arrangement, selecting a service from a plurality of contending services based on a rule set.

In one arrangement, the camera-mounted mobile phone sends image-related data to a cloud-based service for processing and receives audio or video data, or JavaScript instructions in response.

In one arrangement, the camera-mounted mobile phone sends image-related data to a cloud-based service for processing, and the phone pre-warms the service or communication channel in anticipation of the transmission of the image-related data. Also in such an arrangement, the prewarmed service or channel is identified by prediction based on the circumstances.

In one arrangement, the camera-mounted mobile phone has a plurality of modes that the user can select, one of the modes being a face recognition mode, an optical character recognition mode, a mode associated with purchasing an imaged item, an imaged item A mode associated with selling, or a mode for determining information about an imaging item, scene or person (eg, from Wikipedia, a manufacturer's website, a social network site). Also in this arrangement, the user selects a mode before capturing the image.

In one arrangement, it defines a glossary of visual signs, which is recognized by the mobile phone to serve to trigger associated functions.

In one arrangement, a camera-mounted mobile phone is used as a help in name recognition, and the camera captures an image containing a face, which is associated with reference data determined by a remote resource such as Facebook, Picasa or iPhoto. By face recognition processing.

In one arrangement, the image of the object captured by the camera-mounted mobile phone is used to link to information related to the object, such as spare parts or manual instructions, images of objects with similar appearance, and the like.

In one arrangement, an image is stored in association with a set of data or attributes that serve as implicit or explicit links to operations and / or other content. Also in this arrangement, the user navigates from one image to the next-similar to navigating between nodes on the network. Also in such an arrangement, these links are analyzed to identify additional information.

In one arrangement, the image is processed-in accordance with information from the data store-to identify associated semantic information. Also in this arrangement, the identified semantic information is processed-in accordance with the information from the data store-to identify another associated semantic information.

One arrangement includes a plurality of mobile phones in a network cluster. Also in this arrangement, the networked cluster comprises a peer-to-peer network.

In one arrangement, the default rule governs the sharing of content in the network, and the default rule specifies that content of the range of the first time period is not shared. In addition, in this arrangement, the default rule specifies that the content of the range of the second time period can be shared. In addition, in such an arrangement, the default rule specifies that the content of the range of the second time period can be shared only if it is in the social link.

In one arrangement, the empirical data associated with a location becomes available to users at that location. Also in such an arrangement, the mobile phones at that location form an ad hoc network where empirical data is shared.

In one arrangement, an image sensor of a camera-mounted mobile phone is formed on a substrate, and also on the substrate for processing image-related data to serve automated visual queries (eg object recognition). One or more modules are formed.

In one arrangement, the image is captured by one party and made available to multiple users for analysis, such as object recognition purposes (eg, find my vehicle).

In one arrangement, the image feed from the distributed camera network is made available for public retrieval.

Also, arrangements corresponding to those described above relate to audio captured by a microphone rather than visual input captured by an image sensor (eg, including video recognition for face recognition, etc.).

It also relates to computer-readable storage media having methods, systems and subcombinations corresponding to those described above, and instructions for configuring a processing system to perform some or all of these methods.

10, 81, 530: cell phone 12: image sensor
16: cloud 32, 544: camera
34: Setup Module 35: Synchronization Processor
36: control processor module 38: processing module
51: pipeline manager 52: data pipe
72, 73, 74: hardware module
79, 524, 534, 546, 590: Memory 82: Lens
84: beam splitter
86: micro-mirror projector system 110: portable device
111, 582: Display 112: Keypad
114: controller 124: roller wheel
512, 530: thermostat 514: temperature sensor
516, 542: processor
520: LCD display screen 526: WiFi transceiver
528: antenna 532: processor
552b: remote server 554: router
556b: Server 584: Physical UI
586: control handler

Claims (53)

In a mobile phone comprising a microphone and a wireless transceiver:
And further comprising an image sensor coupled to both the human vision system processor and the machine vision processor, wherein the image sensor is coupled to the machine vision processor without passing through the human vision system processor, and wherein the human vision system processor is a white balance correction module. And a module selected from the group consisting of a gamma correction module, an edge enhancement module and a JPEG compression module. The method of claim 1,
The machine vision processor includes at least one selected from the group consisting of an FFT module, an edge detection module, a pattern extraction module, a Fourier-Mellin module, a texture classifier module, a color histogram module, a motion detection module, and a feature recognition module. A mobile phone comprising a module of, a microphone and a wireless transceiver. The method of claim 2,
Wherein the machine vision processing module comprises a microphone and a wireless transceiver utilizing a processing circuit integrated on the image sensor and a common substrate. In a mobile phone comprising a microphone and a wireless transceiver:
A mobile phone comprising a microphone and a wireless transceiver, further comprising a plurality of stages for processing an image sensor and image-related data, wherein a data driven packet architecture is utilized. The method of claim 4, wherein
Wherein the information in the header of the packet determines the parameters to be applied by the image sensor when first capturing image data. The method of claim 4, wherein
Wherein the information in the header of the packet determines a process to be executed by the plurality of stages for image-related data conveyed in the body of the packet. A system comprising a mobile phone and one or more remote processors for performing image-related processing:
The mobile phone includes a microphone, a wireless transceiver, a memory, and an image sensor operable to capture frames of image information, the mobile phone being operated to package image-related data into packets, wherein at least some of the packets are A system for performing image-related processing, comprising fewer than a single frame of image data. A system comprising a mobile phone and one or more remote processors for performing image-related processing:
The mobile phone includes a microphone, a wireless transceiver, a memory, and an image sensor operative to capture frames of image information, the mobile phone routing specific image-related data for processing by a processor in the mobile phone, Operative to route other image-related data for processing by a remote processor. In an arrangement where the mobile phone cooperates with a remote routing system:
The remote routing system serves to distribute image-related data from the mobile phone for processing by different remote processors and to collect processed image-related data for returning from the processors to the mobile phone. Arrangement in which the mobile phone cooperates with the remote routing system. And a routing system serving the mobile phone to distribute image-related data to one or more processors inside the mobile phone for processing or to a remote routing system for processing by remote processors. In a method of processing captured image data using an image sensor in a mobile phone:
Executing a first processing operation on the captured image data using the processing module of the mobile phone to generate an internally-processed image data;
Executing an automated assessment to select one of two or more external service providers;
Sending the internally-processed image data to the selected external service provider; And
Returning data from the selected external service provider to the mobile phone. The method of claim 11,
The automated assessment includes reverse auction. The method of processing image data captured using an image sensor in a mobile phone. The method of claim 11,
The module of the mobile phone generates a plurality of sets of image-related data,
The method is:
Routing one of the sets of image-related data to the selected external service provider; And
Routing another one of said sets of image-related data to a different external service provider. The method of claim 11,
Wherein said evaluation comprises an anti-auction, wherein said image data is captured using an image sensor in a mobile phone. In the image processing method:
Receiving an image, and metadata associated therewith;
Storing image data in at least one plane of the multi-plane data structure;
Generating a graphical representation of the metadata; And
And storing the metadata in another plane of the data structure. The method of claim 15,
Processing the image to derive edge map data related to the image; And
Storing the edge map data as metadata in the other plane of the data structure. The method of claim 15,
Processing the image to recognize one or more faces in the image; And
Storing information about the faces as metadata in the other plane of the data structure. The method of claim 15,
Storing the graphical representation of the image data and the metadata in JPEG2000 form, wherein the graphical representation of the metadata is metadata corresponding to the features when the features of the image are progressively decoded for provision. Is similarly configured to be progressively decoded. A mobile phone comprising a camera, a memory, a processor, a screen, and software in the memory:
The software provides the processor with display display data including at least one of (1) rotation from horizontal, (2) rotation after an initial time, and (3) scale change after an initial time. Wherein the processor comprises a camera, a memory, a processor, a screen, and software in the memory to determine the data with reference to information from the camera. A mobile phone system comprising an image data sensor, a memory, a data processing system, and a screen:
The data processing system includes two parallel processing units,
The two parallel processors are:
A first processing unit for processing image data to be rendered in a recognition form for use by human viewers, the first processing module comprising a mosaic-releasing processor, a white balance correction module, a gamma correction module, an edge enhancement module, and a JPEG The first processor including at least one of a compression module; And
A mobile phone system comprising an image data sensor, a memory, a data processing system, and a screen, the second processing portion for analyzing the image data to derive semantic information from the image data. The method of claim 20,
The second processing module includes at least one of an edge detection module, a Fourier transform processor, a discrete cosine transform processor, and an eigenface processor, wherein the mobile phone comprises an image data sensor, a memory, a data processing system, and a screen. system. In a method for presenting information on the screen of a mobile phone:
Defining two or more dimensions of information related to the subject;
Displaying in a first dimension a sequence of screens providing information about the object when a first user interface control is operated; And
Displaying in a second dimension a sequence of screens providing information about the object when a second user interface control is operated. The method of claim 22,
While the screen is displayed, in response to the operation of the third user interface control, changing the object to the object represented on one of the displayed screens, providing information on the screen of the mobile phone. Way. The method of claim 22,
The object is an image;
The first dimension is similar to the image in one of (1) geolocation, (2) appearance, or (3) content description metadata;
Said second dimension being similar to said image in one of said (1), (2) or (3). A method of composing a text message on a portable device equipped with a camera, comprising:
Tilting the device in a first direction to scroll through a sequence of displayed icons, wherein at least some of the icons each represent the device representing a plurality of letters of the alphabet until an intended icon is reached; Tilting;
Tilting the device in a second different direction to scroll through a sequence of displayed characters including the plurality of letters associated with the intended icon until an intended letter is reached; And
Selecting the intended letter to add to a text message. Obtaining, using a mobile phone, identification information corresponding to a device, the device comprising an electronic processor;
Identifying application software corresponding to the device with reference to the identification information;
Programming the mobile phone with the identified application software; And
Interacting with the device through the use of the application software,
And the mobile phone acts as a multifunction controller configured to control a particular device through the use of application software identified with reference to information corresponding to that device. The method of claim 26,
The device is a thermostat, parking meter, alarm clock, or vehicle. The method of claim 26,
The acquiring identification information includes capturing image data from the device and processing it to obtain an identifier corresponding to the device. The method of claim 26,
Capturing image data from the device, and providing a user interface for the device as a graphical overlay on the captured image data. The method of claim 29,
The graphical overlay is provided in a registered alignment with the features of the captured image data. Issuing a command regarding control of the device via a user interface on a user mobile phone, the user interface being provided on the screen of the phone in combination with a phone-captured image of the device. Issue step;
Signaling to the user information corresponding to the command in a first manner while the command is pending; And
And in a second different manner, once the command has been executed successfully, signaling information corresponding to the command to the user. When the user is in physical proximity to the device, initiating a transaction with the device using the user interface provided on the screen of the mobile phone;
Later using the mobile phone for purposes not related to the device; And
Later, recalling the user interface to associate with another transaction with the device. 33. The method of claim 32,
And the device comprises a parking meter. 33. The method of claim 32,
And the device comprises a thermostat. A mobile phone comprising a processor, a wireless interface, a memory, a sensor, and a display:
The instructions in the memory configure the processor to provide a user interface, wherein the user interface uses the mobile phone to interact with a plurality of different external devices such that the user has several different device-specific users stored in memory. A mobile phone comprising a processor, a wireless interface, a memory, a sensor, and a display allowing to select among the interfaces. Using a mobile phone, sensing information from a housing of a device including a wireless interface;
Encrypting the information using the sensed information using a public key corresponding to the device; And
Transmitting the encrypted information. Using the mobile phone, detecting analog information from a device equipped with an air interface;
Converting the sensed analog information into a digital form;
Transmitting data corresponding to the digital form from the mobile phone; And
Using the transmitted data to confirm user proximity to the device before allowing a user to interact with the device using the mobile phone. In an arrangement in which a camera-mounted mobile phone processes image data through selection of a plurality of processing stages:
An arrangement in which a camera-mounted mobile phone processes image data through selection of a plurality of processing stages, wherein the selection of one processing stage processes image data that depends on the attributes of the processed image data output from different processing stages. A method of using a portable electronic device comprising a wireless interface and reconfigurable hardware:
In response to a user action, initializing the device to prepare for use, wherein the initializing step includes downloading, via the wireless interface, updated hardware configuration instructions for the hardware from a remote source. An initialization step; And
Configuring hardware of the device according to the downloaded hardware configuration instructions,
The method dynamically reconfigures the hardware whenever the device is ready for use, and wherein the reconfiguration ensures that the device is not initialized with replaced configuration instructions for programmable hardware. A method of using a portable electronic device comprising. Utilizing a hardware processing component of the wireless system base station to process data related to wireless signals exchanged between the wireless system base station and a plurality of associated remote wireless devices; And
And using the hardware processor component of the wireless system base station to process image-related data offloaded to the wireless base station for processing by a consumer camera device. The method of claim 40,
Wherein the hardware processing component comprises a field programmable object array and the remote wireless devices comprise mobile phones. Characterizing an optical distortion function associated with the lens;
Defining a geometry of a correction surface corresponding to the optical distortion function; And
After receiving the image through the lens or before projecting the image through the lens, texture-mapping the image onto the correction surface using a GPU;
The tilts and / or elevations at the correction surface serve to compensate for the optical distortion function associated with the lenses. In the method of correcting lens distortion in an image:
Projecting the image onto a virtual surface whose topology is formed to remove the lens distortion. Receiving, at a wireless station, a service reservation message from a first mobile device, the service reservation message including one or more parameters of a future service requesting that the first mobile device be available at a future time, not immediately. Receiving the service reservation message; And
Based on the service reservation message received from the first mobile device, determining about a service provided to a second mobile device,
The allocation of resources of the wireless station is improved due to the improved information regarding the expected services to be provided to the first mobile device. 45. The method of claim 44,
Wherein the one or more parameters include a parameter indicating a future time when the future service is requested to be made available. 45. The method of claim 44,
The one or more parameters include a parameter indicative of an expected duration of the requested future service. A mobile phone comprising an image sensor, a processor, and a memory coupled to a thermoelectric cooling device:
The instructions in the memory include an image sensor, processor, and memory coupled to a thermoelectric cooling device that configure the processor to selectively activate the thermoelectric cooling device based on noise or environments detected in captured image data. Mobilephone. In a mobile phone device:
A first portion comprising an optical sensor and a lens assembly, the first portion configured to be carried by the user at a first position with respect to the user's body; And
A second portion comprising a display, a user interface, and a wireless transceiver, the second portion configured to be carried by the user at a second different location relative to the user's body; And
And a wireless communication system linking the first and second portions. 49. The method of claim 48 wherein
And the second portion is configured to removably receive the first portion. In a mobile phone device:
A first portion comprising a lighting assembly;
A second portion comprising a display, a user interface, a wireless transceiver, an optical sensor and a lens; And
A wireless communication system linking the first and second portions;
And the first portion is detached from the second portion and can be positioned to illuminate an object to be imaged by the optical sensor of the second portion. Receiving an image at a camera-mounted mobile phone device;
Recognizing a plurality of objects depicted in the image;
On the screen of the mobile phone device, graphical features are overlaid with a registered alignment with the recognized objects to provide the image; And
Detecting a user selection of one of the graphical features and thereby triggering an action related to a recognized object associated therewith. The method of claim 51 wherein
The operation includes providing a menu of links related to the recognized object. The method of claim 51 wherein
The operation includes launching an application.

Images