US20020072982A1 - Method and system for interacting with a user in an experiential environment - Google Patents

Method and system for interacting with a user in an experiential environment Download PDF

Info

Publication number
US20020072982A1
US20020072982A1 US09/734,949 US73494900A US2002072982A1 US 20020072982 A1 US20020072982 A1 US 20020072982A1 US 73494900 A US73494900 A US 73494900A US 2002072982 A1 US2002072982 A1 US 2002072982A1
Authority
US
United States
Prior art keywords
user
sample
event
events
predetermined
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US09/734,949
Inventor
Christopher Barton
Philip Inghelbrecht
Dheeraj Mukherjee
Avery Wang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Apple Inc
Original Assignee
Shazam Entertainment Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US09/734,949 priority Critical patent/US20020072982A1/en
Application filed by Shazam Entertainment Ltd filed Critical Shazam Entertainment Ltd
Assigned to SHAZAM ENTERTAINMENT LTD. reassignment SHAZAM ENTERTAINMENT LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: BARTON, CHRISTOPHER JACQUES PENROSE, INGHELBRECHT, PHILIP, MUKHERJEE, DHEERAJ SHANKAR, WANG, AVERY LI-CHUN
Priority to PCT/US2001/046986 priority patent/WO2002061652A2/en
Publication of US20020072982A1 publication Critical patent/US20020072982A1/en
Priority to US11/188,385 priority patent/US20050267817A1/en
Assigned to LANDMARK DIGITAL SERVICES LLC reassignment LANDMARK DIGITAL SERVICES LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SHAZAM ENTERTAINMENT LIMITED
Assigned to LANDMARK DIGITAL SERVICES LLC reassignment LANDMARK DIGITAL SERVICES LLC CORRECTIVE ASSIGNMENT TO RE-RECORD ASSIGNMENT PREVIOUSLY RECORDED UNDER REEL AND FRAME 016546/0733 TO CORRECT THE ADDRESS FROM 10 MUSIC SQUARE, EAST NASHVILLE, TEXAS 37203 TO 10 MUSIC SQUARE EAST, NASHVILLE, TENNESSEE 37203 Assignors: SHAZAM ENTERTAINMENT LIMITED
Priority to US12/204,526 priority patent/US8015123B2/en
Priority to US13/100,825 priority patent/US8996380B2/en
Priority to US13/193,205 priority patent/US8688600B2/en
Assigned to SHAZAM INVESTMENTS LIMITED reassignment SHAZAM INVESTMENTS LIMITED ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: LANDMARK DIGITAL SERVICES LLC
Priority to US14/230,061 priority patent/US9721287B2/en
Assigned to APPLE INC. reassignment APPLE INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SHAZAM INVESTMENTS LIMITED
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/06Buying, selling or leasing transactions
    • G06Q30/0601Electronic shopping [e-shopping]
    • G06Q30/0623Item investigation
    • G06Q30/0625Directed, with specific intent or strategy
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/02Marketing; Price estimation or determination; Fundraising
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/02Marketing; Price estimation or determination; Fundraising
    • G06Q30/0207Discounts or incentives, e.g. coupons or rebates
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/02Marketing; Price estimation or determination; Fundraising
    • G06Q30/0207Discounts or incentives, e.g. coupons or rebates
    • G06Q30/0217Discounts or incentives, e.g. coupons or rebates involving input on products or services in exchange for incentives or rewards
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/02Marketing; Price estimation or determination; Fundraising
    • G06Q30/0241Advertisements
    • G06Q30/0251Targeted advertisements
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/02Marketing; Price estimation or determination; Fundraising
    • G06Q30/0241Advertisements
    • G06Q30/0277Online advertisement
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/06Buying, selling or leasing transactions
    • G06Q30/0601Electronic shopping [e-shopping]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/06Buying, selling or leasing transactions
    • G06Q30/0601Electronic shopping [e-shopping]
    • G06Q30/0633Lists, e.g. purchase orders, compilation or processing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/06Buying, selling or leasing transactions
    • G06Q30/0601Electronic shopping [e-shopping]
    • G06Q30/0641Shopping interfaces
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q99/00Subject matter not provided for in other groups of this subclass
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10HELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H7/00Instruments in which the tones are synthesised from a data store, e.g. computer organs
    • G10H7/02Instruments in which the tones are synthesised from a data store, e.g. computer organs in which amplitudes at successive sample points of a tone waveform are stored in one or more memories

Definitions

  • the present invention relates generally to methods and apparatuses for interacting with a user, and more particularly to a method and apparatus for interacting with a user in an experiential environment.
  • the present invention is therefore directed to the problem of developing a method and system for interacting with a user in a way that takes advantage of the rich and stimulating experiential environment to which the user is exposed to simplify the processing of user requests.
  • the present invention solves these and other problems by providing a method and apparatus for interacting with a user that employs a captured sample of an experiential environment in which the user exists as a command to trigger subsequent events.
  • a method and system for interacting with the user includes capturing a sample of an experiential environment and delivering that sample to an interactive service to trigger one or more predetermined events.
  • triggered events include the delivery of information and services to the user, the execution of tasks and instructions by the service on the user's behalf, communication events, surveillance events and other control-oriented events that are responsive to the user's wishes.
  • the triggered events include transaction-oriented events, entertainment events, and events associated with enhancements to human ability or function.
  • FIG. 1 depicts a block diagram of an arrangement in accordance with the invention including an experiential environment, a recognition engine and related structures, and an event trigger.
  • FIG. 2 depicts a block diagram of the relationship of elements of an exemplary embodiment of the invention among a service provider, service user, media provider, and experiential environment, among other elements.
  • any reference herein to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention.
  • the appearances of the phrase “in one embodiment” in various places herein are not necessarily all referring to the same embodiment.
  • the present invention includes a method and system for interacting with a user that includes providing a user with an ability to capture a sample of an experiential environment and delivering that sample to an interactive service to trigger one or more predetermined events.
  • service and “service provider” and “system” as used herein include a service, service provider, and system, respectively, which employ the various principles of the present invention unless otherwise indicated.
  • events that can be triggered include communication events or surveillance events.
  • the experiential environment can be of any form where a message or information (e.g., content), or other sensory stimuli is available for the user to experience, and may come from many sources.
  • Radio and television, pre-recorded audio and video (e.g. tapes, CDs, DVDs, MP3, etc.) signals on internet and computer-based systems, telephones, live demonstrations or performances, and sensory stimuli from ambient surroundings are several examples of elements which may form, alone or in combination, a typical experiential environment.
  • a sampling device such as an ordinary mobile (or cellular) phone, video recorder, or camera, or a personal digital assistant (PDA) or palm pad, the user captures a sample of the experiential environment and transmits the sample to a service provider employing the present invention.
  • PDA personal digital assistant
  • a personal digital assistant or computer could be specially enabled to act as the interactive service itself by storing the database in its memory and performing the database query and processing without externally accessing the service.
  • the database includes song records
  • this can easily be incorporated in current capacity disk drives even for millions of songs.
  • the personal digital assistant would provide complete end-to-end capability for a user and would not require a separate communication channel to the interactive service.
  • transmission to the service provider would only be needed for additional interaction and potentially for updates of the music database, such as periodically (e.g., weekly).
  • Another embodiment consists of a hybrid form of above mentioned capturing devices and communication channel.
  • a Palm handheld device captures the experimental environment and preprocesses the characteristics of this environment before sending it on to the service provider for recognition. This reduces the bandwidth required for the communication channel and thereby improves the response time, yet does not require the very high speed, high capacity hand held device required by the above embodiments.
  • the service provider may employ the sample by itself, may derive information from the sample, may use data known about the user (e.g., the user's identity and/or user profile), may accept input from the user, or may employ a combination of all such inputs, to trigger a predetermined event that is responsive to the user's needs.
  • data known about the user e.g., the user's identity and/or user profile
  • the service provider may employ the sample by itself, may derive information from the sample, may use data known about the user (e.g., the user's identity and/or user profile), may accept input from the user, or may employ a combination of all such inputs, to trigger a predetermined event that is responsive to the user's needs.
  • such events include the delivery of information and/or services to the user, the execution of tasks and/or instructions by the service on the user's behalf, and other control-oriented events that are responsive to the user's wishes.
  • the triggered events also include transaction-oriented events, entertainment events, and events associated with enhancements to human ability or function. It is noted that combinations of triggered events, as shown in FIG. 1 and described in greater detail below, may also be advantageously used in certain applications of the invention.
  • a brief overview of one exemplary embodiment of the invention is now provided to briefly highlight the principles of the present invention.
  • a method and system is provided for triggering an information delivery event to a user.
  • a user listening to a commercial radio broadcast of popular music at home is interested in singing along with a particular song being broadcast, and thus desires to obtain songs lyrics.
  • the user dials a predetermined telephone number to access a service utilizing the principles of the present invention, and then holds the telephone handset within audible distance of the source of the audio signal (i.e., the speaker of the user's home stereo), to capture a sample of the environment.
  • the experiential environment includes the music playing on the user's home stereo.
  • a system employed by the service provider at the other end of the telephone automatically answers the phone, and using a recognition engine, compares the captured sample to music recordings in a database to identify the song and thereby trigger a desired event for the user.
  • the triggered event is the delivery of the lyric information to the user that corresponds to the song recognized from the captured sample.
  • Event fulfillment the delivery of the desired song lyrics to the user, may be accomplished in number of different ways in both real-time and delayed modes.
  • real-time mode lyric data is delivered to so that the user can read the lyrics to sing along with the song, in time, as it is being broadcast.
  • data for the song corresponding to the captured sample is read out of a database and transmitted by the service to a user device such as a computer, TTD device, SMS telephone or device, pager, or other personal communication device capable of displaying text.
  • the real-time delivery of lyrics may be transmitted in a batch arrangement where relatively large portions or the entire lyric data is downloaded at a time, or using a discrete arrangement where lyric data is automatically synchronized to the song and transmitted in pieces to the user as the song progresses.
  • the user may choose to synchronize the display of the lyrics to the song being played (so that the lyrics are displayed at the appropriate time interval in advance of the lyrics being sung or spoken by the song performer on the broadcast), or in some applications of the invention, the user device may be configured to perform synchronization of the batch transmission.
  • the service transmits the lyric data to automatically synchronize to the broadcast song without any action required by the user or user device.
  • Such automatic synchronization is implemented through a derivation of the temporal position of the captured sample within the recognized song.
  • the delayed mode of event fulfillment includes the delivery of lyrics corresponding to the sampled song sometime after the song broadcast.
  • the same delivery methods as with real-time mode may be used along with delivery methods that are characterized as being more time-consuming such as mail, courier, and facsimile transmission.
  • additional user input may be collected via voice or touch-tone (i.e., DTMF tones) to further control the lyric delivery event or trigger additional events such as transaction events.
  • voice or touch-tone i.e., DTMF tones
  • the service may provide purchase options to the user to obtain the record album containing the broadcast and identified song for which the lyrics were sought.
  • Other purchase options may also be provided to the user by the service provider.
  • the user may purchase other merchandise (which could be related to the music or not). For example, after purchasing the album containing the identified song, the user may choose to purchase tickets to an upcoming performance by the song's artist. Or, the user may inform friends or potential users of the service about the identification and/or album purchase. Such communication could be implemented by the service provider sending an email or SMS message on the user's behalf.
  • FIG. 1 a block diagram of an exemplary arrangement of the invention is depicted.
  • a sample captured from an experiential environment 101 is represented by block 105 .
  • the experiential environment 101 that may be used by the present invention includes virtually any environment where sensory stimuli are present for a user to sense or experience.
  • elements of such experiential environment 101 include, singularly or in combination, media or other information content (each singularly or in combination referred to as “media” here) delivered in a channelized stream and media delivered in unchannelized streams that are generally sensed via sight or sound.
  • media media delivered in unchannelized streams that are generally sensed via sight or sound.
  • sight and sound are not necessarily the only sensed experiences that may be used in applications of the invention.
  • Channelized media as used here means media that is delivered to a user through discrete and identifiable means through such conventional sources such as television, radio, pre-recorded audio and video (e.g. tapes, CDs, DVDs, MP3, streaming IP video and audio), computers, telephones (both wired and wireless), and messaging devices such as SMS devices and pagers.
  • channelized media encompasses both unidirectional broadcasts, and bi-directional communications, it also includes print media such as newspapers, books, and magazines and the visual arts.
  • Unchannelized media means media that is delivered to a user through means that may be indistinct, less readily identified, or involve multiple sources of media. Examples of unchannelized media include the performing arts, speeches and lectures and the sensory experiences derived from interactions with other persons, or the surrounding environment. Unchannelized media further includes sights, sounds, and other sensory stimuli that result from a user's ambient surroundings.
  • a user captures a sample of the experiential environment 101 via line 103 using a capture device 102 .
  • the capture device 102 captures a sample of experiential environment 105 and provides it in a format suitable for processing by the recognition engine 110 on line 117 .
  • Illustrative embodiments of capture devices include, but are not limited to, microphone, telephone, mobile telephone, PDA, tape recorder, digital voice recorder, answering machine, radio receiver, walkie-talkie, internet streaming socket, voice-over-IP transceiver, or other such audio sampling devices.
  • the signal capture device is incorporated into a device that the user employs in a location remote from the service.
  • Conventional devices like mobile and regular telephones, PCs, radios, and other recording and communication devices that users already own or use everyday for other purposes may conveniently be used, without modification, to sample both channelized and unchannelized media.
  • Video or image capture devices may be also be used as capture device 102 , in accordance with the invention to provide a sample of experiential environment 105 to recognition engine 110 via line 107 , as depicted.
  • camcorders, video conferencing equipment, motion and still video cameras, surveillance devices, cameras, digital cameras, image grabbers, picture telephones and other devices that can capture an image sample of an experiential environment may be conveniently used to provide a sample to recognition engine 110 .
  • User input/identification block 120 may identify the user to the recognition engine 110 as indicated by line 122 , and may optionally be operationally coupled to the event trigger block 130 via line 124 .
  • Examples of devices that generate the appropriate identification for use with user input/identification block 120 may include caller ID on a POTS (Plain Old Telephone Service) line or a mobile telephone, internet IP address of a terminal sending in the captured signal, or a cookie file stored on an internet browser on the user's terminal.
  • POTS Packe Old Telephone Service
  • user input/identification block 120 the user may be required to enter an account code, for example by keying it in on a touchtone pad on a telephone or saying a pass phrase while signing on to a service incorporating the principles of the present invention if dialing in.
  • the user may be identified by inserting an object carrying identification codes into a terminal. Examples of this include a credit card, ATM card, or Dallas Semiconductor Java Ring.
  • the user may also be identified by a biometric device to scan fingerprints, retinas, palm print, or other such physical characteristics of the user.
  • a speaker identification system to identify the user by vocal characteristics is another alternative method.
  • User input/identification block 120 is an optional component of the present arrangement, which is employed if billing and tracking of user activity is desired.
  • Experiential environment 101 is typically associated with a media operator or content provider such as radio or television broadcasters, CATV provider, internet service providers, entertainment organizations, private network or LAN operators, and the like.
  • a media operator or content provider such as radio or television broadcasters, CATV provider, internet service providers, entertainment organizations, private network or LAN operators, and the like.
  • the experiential environment may comprise a live demonstration or performance, or other unchannelized media, taking place for example, at a nightclub, bar, or discotheque.
  • the experiential environment may include sights, sounds, and other sensory stimuli from the ambient surroundings.
  • Experiential environment capture device 102 is generally associated with users, however, such association may be merely temporary, as public access devices (e.g., public telephones and internet access facilities) may be readily used, without modification, in order to realize the benefits provided by the present invention.
  • Experiential environment sample 105 represents features and functionalities that, for example, are implemented by the microphone and associated transceiver circuits in a user's mobile telephone or computer. Such features and functionalities could also be implemented by a charge coupled device (“CCD”) array that is integrated in a camera associated with a picture telephone or computer.
  • CCD charge coupled device
  • User input/ID block 120 is also associated with the user.
  • FIG. 1 The remaining elements of FIG. 1 are collectively associated as indicated in FIG. 2 with a service provider.
  • Recognition engine 110 , reporting and transaction block 137 , database 115 , and user profile/transaction history 135 represent features and functionalities of an integrated system that form key elements of an interactive service arrangement that may be particularly useful in some applications of the invention. In such interactive service arrangements, these collected elements are typically implemented in a system formed by one or more CPUs.
  • the interactive service arrangement is identified by reference numeral 250 in FIG. 2.
  • a media provider 210 utilizes signal source 201 which transmits signal 203 via media distribution network 220 which may be arranged from network and transmission elements or other channelized distribution arrangements (as for example, with copper or fiber networks for data or telephony services) or a free space/transmitter infrastructure array (as for example with radio and television broadcasts, satellite systems, and cellular/PCS/GSM wireless telephone services or networks which operate according to short-range wireless protocols such as the BluetoothTM wireless standard).
  • media distribution network 220 may be arranged from network and transmission elements or other channelized distribution arrangements (as for example, with copper or fiber networks for data or telephony services) or a free space/transmitter infrastructure array (as for example with radio and television broadcasts, satellite systems, and cellular/PCS/GSM wireless telephone services or networks which operate according to short-range wireless protocols such as the BluetoothTM wireless standard).
  • Receiver/monitor 240 is employed by user 230 to receive signal 203 and transform the signal into a format that facilitates signal 203 to be monitored by the user.
  • Receiver/monitor 240 may be a radio, television, PC, Hi-fi (i.e., stereo) with speakers or any other device that may be used to create a media experience (including audio and video) that may be monitored by the user.
  • User 230 using, for example, a mobile telephone, obtains a sample of signal 203 played on receiver/monitor 240 , where the sample includes media content of interest selected by the user, such as a portion of an image or sound.
  • receiver/monitor 240 both outputs a rendition of signal 203 to 105 and allows the user 230 to monitor signal 203 .
  • signal capture device 102 may capture a sample of signal 203 via a direct connection to media distribution network 220 (i.e., not relying on receiver/monitor 240 or similar device for signal input). In such instances, the user monitoring of signal 203 is accomplished through other means or user monitoring is not performed.
  • capture device captures a sample directly from experiential environment 101 via line 222 without relying on a media distribution network 220 .
  • a signal source transmits directly into the environment, as indicated by line 221 , or the experiential environment is itself being captured by capture device 102 .
  • the arrangement of the present invention may be advantageously used to trigger a variety of predetermined events from a sample of live performances, interaction, or the sights and sounds of the ambient surroundings.
  • the captured sample of signal 203 is relayed by the user 230 to the interactive service arrangement 250 , as shown, via communication network 260 on line 207 , as shown.
  • Communication network 260 may have a similar arrangement as distribution network 220 , or may be unitary with distribution network 220 . It should be noted, however, that in certain applications of the invention distribution network 220 may typically be characterized by unidirectional signal propagation (as in the case with broadcast radio and television or typical CATV head end systems) while communication network 260 may typically be characterized by bi-directional signal propagation (as in the case with the public switched telephone network and wireless or wired voice, data, and internet systems). Such bi-directional nature of communication network 260 is indicated by signal flow lines 207 and 208 as depicted in FIG. 2.
  • the interactive service arrangement 250 derives information or characteristics of sample of signal 203 . Such derived information may be returned to the user 230 from the interactive service arrangement 250 using the same communication network 260 or other networks.
  • the signal return path is indicated with lines 208 in FIG. 2.
  • the interactive service arrangement 250 may interact with the user and other entities. For illustrative purposes, such interaction pathways are depicted in FIG. 2 as lines 289 and 282 , input and output, respectively, via alternate network 280 .
  • Alternate network 280 may be a network of any type, however, in some applications of the invention it may be advantageous to employ private networks, dedicated lines, or other high-capacity transmission methods should high-bandwidth interactions be desired.
  • fulfillment center 286 interacts with interactive service arrangement 250 using alternate network 280 and fulfills user requests as indicated by line 288 .
  • Fulfillment center 286 may also interact with interactive service arrangement 250 using communication network 260 over interaction pathways 292 and 294 as indicated.
  • FIG. 2 may serve to additionally highlight the principles applied in the exemplary embodiment of the invention that was briefly overviewed at the beginning of the Detailed Description.
  • a radio station (media provider 210 ) employing signal source 201 broadcasts a song (signal 203 ) over the air (media distribution network 220 ), which is received on a radio (receiver/monitor 240 ) of user 230 .
  • User 230 wishes to sing along with the song.
  • User 230 places a mobile telephone call over a wireless network (communication network 260 ) to interactive service arrangement 250 .
  • User 230 positions the microphone of his wireless telephone (capture device 102 ) to capture a sample of the music being played on the radio.
  • the interactive service arrangement 250 receives the sample over the wireless network and derives the identity of the song.
  • the identity of the user may be derived from the user's mobile telephone number that is sent from the communication network 260 to the interactive service arrangement 250 typically during call set up.
  • the song lyrics are returned to back to the user's mobile telephone over the same network.
  • Further interactions between the user 230 and the interactive service arrangement 250 may occur, and if the user chooses to purchase related merchandise (such as a CD recording of the album containing the broadcast song, or hardcopy printout of the desired lyrics), the interactive service arrangement 250 can send the purchase information to the service's distribution facility (fulfillment center 286 ) via data or voice communication using its local area PC network (alternate network 280 ) or via data or voice communication over communication network 260 as discussed above.
  • the service's distribution facility fullfillment center 286
  • data or voice communication using its local area PC network (alternate network 280 ) or via data or voice communication over communication network 260 as discussed above.
  • the distribution center sends the merchandise to the user via overnight courier service (line 288 ).
  • alternate network 280 may be utilized to send an email or facsimile containing the song lyrics to the user's email account.
  • the service relies upon a profile for the user that is created in advance of the call (but may be updated during a call as described below).
  • a “user profile” may typically include general information about the user such as name, address, preferred method of payment (i.e., credit card pre-authorization), and set dollar limits on purchases.
  • service-specific information regarding the user may also be included in the profile, such as demographic and user-identified preference information, to facilitate the service tailoring the transaction to fit a particular user.
  • profiles can be built without prior user registration.
  • Age, education, residence, gender, occupation, and personal interests, likes and dislikes, among other criteria, may be employed to most effectively match transaction offers to users' interests and purchase habits.
  • one particular customer of the service may have a user profile that indicates that the user is a member of a demographic group that is music-savvy and aware of music trends. After offering to sell a recording of the song selected by the user, the service could offer to sell a recording by an artist that is “moving up the charts” in popularity.
  • a higher transaction closing rate may be realized as offers are more accurately targeted to users who may be predisposed to view the offer favorably.
  • user profiles are often dynamic and that the present invention includes an ability to update and change a user profile in response to changes in usage patterns (as described below), market and industry trends, user input, and other factors.
  • a feature of the invention related to user profile is usage tracking.
  • usage tracking By tracking the frequency and time of access, and purchase history, for example, of a user, the service can gain additional insight into factors which may influence a user. Patterns of usage may be derived which may allow predictive modeling to be utilized, for example, to enhance and refine service offerings.
  • the system of the present invention can thus differentiate between repeat users (who heavily access the service) and casual users (who occasionally or infrequently use the service) and adjust the options, offers, and interactive scripting (as described below) so that interest and enthusiasm is maintained among the service's customers. It is contemplated that the user profile and usage tracking/pattern features described here may be used in other embodiments and applications of the inventions as well.
  • User profile and usage tracking are examples of a priori information (i.e., information known by the service provider in advance of a specific interaction with a user) and external information (i.e., information known by the service provider that is determined other than by analyzing the captured sample itself. It will be recognized that other sources of a priori and/or external information may also be advantageously used in some applications of the invention.
  • the location of the user, caller ID signal, dialed number indication service (“DNIS”) and the capture device employed by the user may, alone or in combination, be used by the service provider to select or otherwise tailor the triggered events to meet user expectations or needs.
  • DNIS dialed number indication service
  • the experiential environment sample is received by recognition engine 110 on line 117 .
  • Recognition engine 110 derives characteristics of the received sample by using data stored in database 115 .
  • Recognition 110 and database 115 are operationally coupled via line 119 , as shown in FIG. 1.
  • a variety of derivation methods may be used. In the case of audio samples, the techniques described in Appendix may be used. However, it is noted that the derivation methods that may be used in this invention are not limited to such techniques. The particular derivation method chosen is only required to be able to derive sufficient characteristics from the experiential environment sample to enable a predetermined event to be triggered. Thus, the strength or resolution of the derivation is coupled, in large measure, to the level of discrimination required in selecting an event to be triggered. As the number of potentially triggered events increases, the necessity to resolve ambiguity in the sample also increases.
  • a relatively large number of characteristics about the sample may be derived and compared against stored data to be able to identify the particular song from the many such songs that may be stored. That is, as more songs are potentially identified, more lyric delivery events are potentially triggered. By comparison, in se vice offerings where are relatively small number of events are potentially triggered, fewer sample characteristics need typically be derived in order to resolve ambiguity as to which event to trigger.
  • Such service offering may include those where a binary “Yes” or “No” event may be triggered as may be the case for customer surveys and voting/polling type services.
  • Event trigger 130 may trigger one or more predetermined events according to the derived sample characteristics.
  • the triggering criteria used by event trigger 130 may be provided by the characteristics themselves or be externally provided by database 115 . Alternatively, in the case of binary type events, the mere receipt of any input signal at the event trigger may be sufficient to trigger a particular event.
  • triggering criteria means those factors, data, or inputs, which are used by the event trigger 130 to select the particular one (or group) of actions to take among the many that may be available.
  • triggering criteria are provided by recognition engine 110 as a result of the derivation of characteristics from the captured experiential environment sample
  • additional triggering criteria may also be used in combination with the derived characteristics.
  • additional triggering criteria may come from the user on line 124 as user input in the form of voice, audio or visual commands.
  • a user may provide input to the event trigger by using the touch tone keypads on a telephone, providing keyboard input on a computer, or providing captioning information over a video feed using a video camera.
  • user profile and transaction history information as described above, may be used as additional triggering criteria.
  • user profile/transaction history block 135 is operably coupled to event trigger 130 via line 131 to provide such additional triggering criteria.
  • user profile/transaction history block 135 may provide user and/or transaction data to recognition engine 110 (not shown in FIG. 1) that may be used to assist in the characteristic derivation process.
  • recognition engine 110 may, in certain applications, rely on the particular user's usage history which indicates a strong preference for a specific music genre. This usage history may help eliminate some potential identification candidates that may have some sonic correlations to stored data in database 115 , but are very dissimilar to the types of songs sought to be identified by the user in the past and stored in user profile/transaction history 135 .
  • Reporting/transaction block 137 is operably coupled to the event trigger 130 via line 139 as shown so that event usage may be tracked by the service provider for purposes of creating reports or billing transactions to the user.
  • event trigger 130 can trigger a variety of events indicated collectively by reference numeral 180 . It is noted that the depicted events are merely illustrative and are shown to highlight various categories or types of events that may be triggered using the principles of the invention.
  • Transaction events 182 are one category of events that may be triggered in accordance with the invention. These typically include the purchase and sale of merchandise.
  • a user reading the newspaper may see an advertisement for a raincoat that stimulates a buying interest in the user.
  • the user Using a video camera on the user's PC, the user holds the print advertisement near the camera to capture a still image of the advertisement.
  • the user reaches a service provider and uploads to the image to the service provider's recognition engine.
  • THe advertisement could be included in a billboard, in which case the user could use a still camera or video camera to capture it.
  • the recognition engine uses conventional text recognition algorithms, pulls the name of the advertiser and advertising copy for the raincoat from the still image. Employing an interactive script on the web-page, the user is prompted for size and color preferences.
  • the user's credit card and shipping information may already be on file with the service provider using user profile/transaction history 135 , or such information may be collected from the user on-line through the web-interface or using off-line communications such as telephone, email, or facsimile.
  • the derived characteristics namely the advertiser's name and clothing information, is combined with information about the advertiser stored on database 115 and used by event trigger 130 to trigger a purchase by the user of the desired raincoat.
  • the transaction particulars are forwarded to an event fulfillment center, depicted in FIG. 1 by reference numeral 190 , for actual physical fulfillment of the raincoat to the event recipient 195 , in this case, the user.
  • the service provider may merely function as a deal broker between the advertiser and user where the order information is provided to the advertiser and the service provider collects a service fee from the advertiser.
  • the service provider may derive revenue from charging the user a subscription fee to use the service, which facilitates the transaction as described above. In some applications of the invention, a combination of these revenue models may be advantageously used.
  • Entertainment events 184 may also be triggered in accordance with the invention.
  • entertainment-type services may be advantageously provided by the practice of the invention.
  • Entertainment is defined here to include services where interaction with the service directly provides the value to the user. This compares with some typical transaction events where the service provider furnishes a tangential broker or facilitation service, which is attendant to the purchase and sale transaction between the user and a third party.
  • An illustrative example of an entertainment event 184 provided by the present invention includes game-type events.
  • a user listening to music captures a sample of a song on a mobile telephone, which has been dialed in to a service provider.
  • the recognition engine 110 compares the sample to known songs in database 115 and derives the identity of the song.
  • a game type entertainment event is then triggered by the service to automatically send a small excerpt of the originally recorded song (i.e., not the captured sample of the song) to a pre-determined group of the user's friends via. For example, a WAV file or MP 3 attachment to an email message sent by the service provider or through a call placed by the service to the friend's wired or wireless telephone.
  • the identification data for members of the group may be stored in a user profile, for example.
  • the friends are prompted in the message or call to try to “Name that Tune” by identifying the song's title or artist from the small excerpt.
  • the friend's guesses may be collected by the service provider using a variety of methods, including for example, an interactive web-site, telephone call center, email, or conventional mail. If no one correctly identifies the song, the song excerpt may be increased in length, or a different excerpt may be furnished, in an iterative manner, until a “winner” is determined.
  • Human ability enhancement events 186 are also depicted in FIG. 1. These events are similarly triggered by captured samples of an experiential environment. In embodiments of inventions using these events, enhancements to ability and function are realized by the ability of capture devices 102 to sense the environment in manners that transcend human ability or in manners that replace ordinary senses that are undeveloped or lost through illness or accident in particular individuals.
  • capture devices such as telephones may quite easily arranged to be more sensitive to sound or to particular frequencies of sound than the human ear through the use of amplifiers and filters.
  • visual capture devices may be arranged to provide magnification of far away scenes or make small scenes appear large.
  • the recognition engine 110 could be configured to recognize the bird based on characteristics such as colors and patterns, however, in other applications, recognition engine 110 may more simply be configured to derive a smaller constellation of key characteristics. For example, the recognition engine may derive that the captured experiential environment sample represented is a color digital photograph and trigger an image archiving event for the user. Or, dominant colors may be derived to allow an image-sorting event to be triggered. In both of these events, the service provider uses the derived characteristics of the captured sample (the digital photographs) to automatically archive and arrange the photos in a manner that is consistent with a service offering in which the user participates.
  • Another exemplary embodiment of sensory enhancement is the ability for a user to implement a virtual sense in a remote location. For example, a mother may dial a predetermined number to access a service provider using a conventional telephone located in a sleeping child's room. The telephone is left off hook to constantly sample the ambient conditions in the room. The sampled environment is processed to determine whether the child has awoken by listening for the frequency and amplitude characteristics associated with crying or yelling and places a call or page to the mother to let her know of the change in circumstances.
  • the database 115 and event trigger 130 are arranged to trigger the call or page based on the sound characteristics of crying, but other events could be triggered, such as a call to fire department, using other sound characteristics such as the timbre of a smoke or carbon monoxide detector.
  • capture devices replace senses lost or undeveloped in an individual user, or augment existing senses to make up for a lost sense.
  • capture devices may be arranged to capture a sample of an experiential environment to trigger events that can provide assistance that more typical sensory aides cannot.
  • a visually impaired user may configure video equipment to monitor the entrance to the home.
  • a service receiving the video feed may sample the scene at the remote end to determine when sufficient changes have occurred in the captured scene to determine that a person is at the door.
  • the user may provide data to the service to allow particular individuals to be recognized and announced to the user through a telephone call, page, SMS message, email, and the like.
  • Block 188 indicates that the execution of task and instruction events are provided by the invention.
  • a variety of predetermined events are triggered by the receipt of a particular sample that involve a service performing actions on behalf of the user.
  • Tasks such as bill or invoice payment may be readily automated by uploading a scanned image of the bill to a service provider.
  • the service provider upon receipt of the captured image, and optionally in combination with user profile data, can derive the identity of the payee and automatically pay the bill on behalf of the user and debit the user's bank account or credit card.
  • Block 170 in FIG. 1 depicts that events for the delivery of information can be triggered in response to the receipt of particular captured samples.
  • a service provider triggers the delivery of information to the user or third parties.
  • a service provider may receive a set of sound samples from a user's microphone attached to a computer that is monitoring financial news broadcasts. Deriving the names of companies from the sampled news that the user has indicated an interest in monitoring, the service automatically downloads stock quotes and related news information to the user's computer.
  • delivery of services events may also be triggered in accordance with the invention.
  • a service provider may provide financial documentation services to the user such as the automated provision of public securities and investment analysis documents based on the companies of interest that are discussed in the monitored news broadcast.
  • the user may provide other event triggering criteria such as the minimum number of utterances of a particular company name in a particular time period before the automated provision of documents is triggered.
  • Block 174 in FIG. 1 shows that control events may also be triggered in response to a sampled experiential environment in accordance with the invention.
  • Control events are those that provide the user with an ability to control or otherwise manipulate information and data, services, or other events in a predetermined manner according to the captured sample received by a service provider.
  • a human resources recruiter may organize a data archive of job candidates and associated demographic data by engaging a service provider that automatically manipulates the data according to web-site images of potential hiring companies that are captured in a frame grabber running on the user's computer and uploaded to the service provider.
  • the candidate database can be sorted according the to captured web-document and derived by deriving preselected characteristics such as industry type, key-words in the text elements of the page, and other characteristics.
  • Communication events may be triggered in accordance with the invention as depicted by block 175 in FIG. 1.
  • Communication events include, for example, communicative interactions among users, between users and the service provider, or such interactions between users, the service provider, and third parties.
  • An exemplary embodiment of the invention involving a triggered communication event follows.
  • a user listening to music dials into a service and captures a sample of a song on his mobile telephone.
  • the recognition engine 110 compares the sample to known songs in database 115 and derives the identity of the song.
  • the service then triggers a communication event to notify a second user that the first user engaged the service to identify the song.
  • the notification may be implemented, for example, by the service sending an email or SMS message to the second user.
  • An excerpt of the original recording of the identified song may optionally be attached in the message for the second user to hear.
  • This embodiment of the invention may be of particular relevance to persons interested in music trends who want to communicate, share, or kept be abreast of the latest developments in music.
  • the particular user or group of users chosen to be notified by the user may be selected in advance, for example, by employing a user profile that stores the email address or SMS number of the users to be notified. Or, the user may interact with the service via a conventional interactive voice response unit to indicate a choice relatively contemporaneously with the sample capture process.
  • the dial-in number to the service may be used to indicate a choice by the user.
  • the user may program his mobile phone (or the phone may be preprogrammed by the service provider or partner) so that a plurality of numbers are stored, each indicating a particular user selection to the service. If, for example, the mobile phone stores three dial-in numbers, say *211, *212, and *213, each number may be used to reach the service while indicating a particular user choice. While all the numbers dial into the same service provider, the fact that a particular dialed number was employed by the user is given to the service provider by communication networks 260 (FIG. 2) via a dialed number identification service (“DNIS”).
  • DNIS dialed number identification service
  • *211 could be used to indicate a user desire to trigger a particular communication event, according to a pre-programmed user profile, for example, to notify user A that a song identification was sought by the user.
  • *212 could indicate communications with users A and B
  • *213 could indicate the user desire to notify and provide a song excerpt to a pre-defined third party.
  • the DNIS may be advantageously utilized to allow user selection of a triggerable event from a plurality of events that may be hosted by a service provider. If, for example, a single service provider hosts both shopping services (a transaction event) and game services (an entertainment event), then the DNIS may be used by the service provider (perhaps among other factors used) to disambiguate among the offered events and trigger the particular event desired by the user in response to a triggering characteristic derived from the experiential environment sample received from the user.
  • An embodiment of the invention similar to that above also involves communication events.
  • the user captures a sample of a song to trigger a communication event, which is embodied in the form of a musical greeting card.
  • Such musical greeting cards include an excerpt of the original recording of the identified song with a text or voice annotation from the user that may be personalized to a specific occasion or event.
  • the principles of the present invention may be advantageously utilized to trigger user-to-user (also referred to as “peer-to-peer”) services in response to a received sample of a captured experiential environment.
  • Surveillance and/or monitoring events are also contemplated by the present invention, as depicted by block 176 in FIG. 1.
  • a sample or series of samples of an experiential environment is captured by the user and relayed to the service provider in order to trigger events that may facilitate the detection of the presence or absence of one or more characteristics, elements, or features in the environment, or a change in such environment.
  • a chemical detection device or “sniffer” is employed by a user to sample the air in the environment.
  • sniffers are known and may employ various sensory techniques, such as gas chromatography, to determine the presence and concentration of chemical elements or compounds in the air sample.
  • a signal indicative of the elemental constituents of the sample is forwarded to the service provider whereby analysis of the signal is performed to derive characteristics of the sample used to trigger a desired event.
  • an agricultural user may desire to monitor the conditions of crops. By monitoring the emission of certain gases associated with the decomposition of fertilizer and organic matter, and other gases produced by the growing crops, the user may optimize the application of water, fertilizers and pesticides to produce the highest yield.
  • the service provider can trigger an alert to the user, via facsimile, email, mail, phone call, SMS message, or a variety of other communication methods, when a particular threshold condition has been met or exceeded.
  • the present invention separates the experiential environment sensing apparatus (i.e., the sniffer) from the analysis and event triggering apparatus (i.e., the recognition engine 110 and event trigger 130 in FIG. 1).
  • the sensing apparatus is associated with the user who can best control its deployment in the experiential environment, while the analysis and event triggering apparatus is detached from the user.
  • Such detachment allows the service provider the opportunity to host a wide variety of surveillance and monitoring events that would be impractical to implement in a portable local device.
  • a single sniffer may be used to monitor both crop condition, as described above, as well as methane levels in effluents from a milk parlor of a dairy farm.
  • the required analysis of the gas samples, and the determination of whether a alarm condition exists would be based on criteria that vary widely between these two applications, yet the service provider could readily trigger the different monitoring events due to flexibility associated with remote analysis/trigger arrangement.
  • a variation of this first embodiment includes an arrangement where a scaled down sniffer is carried by a user. While such a device would be limited in the range of elements it could detect, the flexibility of the remote analysis and triggering arrangement of the present invention still may provide many advantages. That is, a less sensitive sensing device, in accordance with the invention, may be coupled with a remotely located, flexible and powerful analysis tool to trigger useful and desirable events. For example, a user with a sensitivity to a particular environment condition, such as an allergy to a specific food, may carry a small device that has sensitivity to the chemical signature of that food. The device samples the environment periodically and sends a representative signal via a wireless transceiver to the service provider for analysis.
  • the service provider can communicate back to the user, via the same sensing device or using alternative communication pathways such as telephone, pager or fax.
  • the alert signal may be sent by the service provider to third parties including, for example, the user's doctor, emergency contact or others as conditions warrant.
  • a spectrum analyzer is configured to sample the strength of constituent wavelengths of light in an experiential environment.
  • a particular spectral sample of the environment can be compared by the service provider against stored signatures to trigger a variety of surveillance or monitoring events.
  • the level of ultraviolet radiation in a warehouse may be compared against stored data to trigger remote control of vents, skylights, or other HVAC devices in the building by the service provider.
  • Block 178 in FIG. 1 indicates that various combinations of the events shown and described in blocks 182 - 188 , and 170 - 176 are also contemplated as being advantageously used in some applications of the invention. That is, some applications may benefit by triggering multiple events, in sequence or in parallel, that may be of the same or differing types.
  • the disclosed invention is capable of recognizing an exogenous sound signal that is a rendition of a known recording indexed in a database.
  • the exogenous sound signal may be subjected to distortion and interference, including background noise, talking voices, compression artifacts, band-limited filtering, transmission dropouts, time warping, and other linear and nonlinear corruptions of the original signal.
  • the algorithm is capable of identifying the corresponding original recording from a large database of recordings in time proportional to the logarithm of the number of entries in the database. Given sufficient computational power the system can perform the identification in nearly real-time, i.e. as the sound is being sampled, with a small lag.
  • the sound database may consist of any collection of recordings, such as speech, music, advertisements, or sonar signatures.
  • each recording in the library is subjected to landmarking and fingerprinting analysis to generate an index set for each item.
  • Each recording in the database has a unique index, sound 13 ID.
  • Each sound recording is landmarked using methods to find distinctive and reproducible locations within the sound recording.
  • the ideal landmarking algorithm will be able to mark the same points within a sound recording despite the presence of noise and other linear and nonlinear distortion.
  • the landmarking method is conceptually independent of the fingerprinting process, but may be chosen to optimize performance of the latter. Landmarking results in a list of timepoints ⁇ landmark k ⁇ within the sound recording at which fingerprints should be calculated.
  • a good landmarking scheme marks about 5-10 landmarks per second of sound recording, of course depending on the amount of activity within the sound recording.
  • a simple landmarking technique is to calculate the instantaneous power at every timepoint and to select local maxima.
  • One way of doing this is to calculate the envelope by rectifying and filtering the waveform directly.
  • Another way is to calculate the Hilbert transform (quadrature) of the signal and use the sum of the magnitudes squared of the Hilbert transform and the original signal.
  • the power norm method of landmarking is especially good for finding transients in the sound signal.
  • the general Spectral Lp Norm is calculated at each time along the sound signal by calculating the spectrum, for example via a Hanning-windowed Fast Fourier Transform (FFT).
  • FFT Fast Fourier Transform
  • the Lp norm for that time slice is then calculated as the sum of the p-th power of the absolute values of the spectral components, optionally followed by taking the p-th root.
  • the landmarks are chosen as the local maxima of the resulting values over time.
  • Multi-slice landmarks may be calculated by taking the sum of p-th powers of absolute values of spectral components over multiple timeslices instead of a single slice. Finding the local maxima of this extended sum allows optimization of placement of the multislice fingerprints, described below.
  • the algorithm computes a fingerprint at each landmark timepoint in the recording.
  • the fingerprint is generally a value or set of values that summarize a set of features in the recording near the timepoint.
  • the fingerprint is a single numerical value that is a hashed function of multiple features.
  • a frequency analysis is performed to extract the top several spectral peaks.
  • a simple such fingerprint value is just the single frequency value of the strongest spectral peak.
  • the use of such a simple peak resulted in surprisingly good recognition in the presence of noise, but resulted in many false positive matches due to the non-uniqueness of such a simple scheme.
  • Using fingerprints consisting of the two or three strongest spectral peaks resulted in fewer false positives, but in some cases created a susceptibility to noise if the second-strongest spectral peak was not sufficiently strong enough to distinguish it from its competitors in the presence of noise—the calculated fingerprint value would not be sufficiently stable. Despite this, the performance of this case was also good.
  • a set of timeslices is determined by adding a set of offsets to a landmark timepoint.
  • a Salient Spectral Fingerprint is calculated.
  • the resulting set of fingerprint information is then combined to form one multitone fingerprint.
  • Each such fingerprint is much more unique than the single-time salient spectral fingerprint since it tracks temporal evolution, resulting in fewer false matches.
  • Our experiments indicate that using two or three timeslices along with the single strongest spectral peak in each timeslice results in very good performance, even in the presence of significant noise.
  • LPC analysis extracts the linearly predictable features of a signal, such as spectral peaks, as well as spectral shape.
  • LPC coefficients of waveform slices anchored at landmark positions can be used as fingerprints by hashing the quantized LPC coefficients into an index value. LPC is well-known in the art of digital signal processing.
  • Cepstral coefficents are useful as a measure of periodicity and may be used to characterize signals that are harmonic, such as voices or many musical instruments. A number of cepstral coefficients may hashed together into an index and used as a fingerprint. Cepstral analysis is well-known in the art of digital signal processing.
  • the resulting index set for a given sound recording is a list of pairs (fingerprint, landmark) of analyzed values. Since the index set is composed simply of pairs of values, it is possible to use multiple landmarking and fingerprinting schemes simultaneously. For example, one landmarking/fingerprinting scheme may be good at detecting unique tonal patterns, but poor at identifying percussion, whereas a different algorithm may have the opposite attributes. Use of multiple landmarking/fingerprinting strategies results in a more robust and richer range of recognition performance. Different fingerprinting techniques may be used together by reserving certain ranges of fingerprint values for certain kinds of fingerprints. For example, in a 32-bit fingerprint value, the first 3 bits may be used to specify which of 8 fingerprinting schemes the following 29 bits are encoding.
  • a searchable database is constructed in such a way as to allow fast (logtime) searching. This is accomplished by constructing a list of triplets (fingerprint, landmark, sound_ID), obtained by appending the corresponding sound 13 ID to each doublet from each index set. All such triplets for all sound recordings are collected into a large index list. In order to optimize the search process, the list of triplets is then sorted according to the fingerprint. Fast sorting algorithms are well-known in the art and extensively discussed in D. E. Knuth, “The Art of Computer Programming, Volume 3: Sorting and Searching,” hereby incorporated by reference.
  • High-performance sorting algorithms can sort the list in N log(N) time, where N is the number of entries in the list. Once this list is sorted it is further processed by segmenting it such that each unique fingerprint in the list is collected into a new master index list. Each entry in this master index list contains a fingerprint value and a pointer to a list of (landmark, sound 13 ID) pairs. Rearranging the index list in this way is optional, but saves memory since each fingerprint value only appears once. It also speeds up the database search since the effective number of entries in the list is greatly reduced to a list of unique values.
  • the master index list could also be constructed by inserting each triplet into a B-tree with non-unique fingerprints hanging off a linked list. Other possibilities exist for constructing the master index list.
  • the master index list is preferably held in system memory, such as DRAM, for fast access.
  • Exogenous sound is provided from any number of analog or digital sources, such as a stereo system, television, Compact Disc player, radio broadcast, telephone, mobile phone, internet stream, or computer file.
  • the sounds may be realtime or offline. They may be from any kind of environment, such as a disco, pub, submarine, answering machine, sound file, stereo, radio broadcast, or tape recorder. Noise may be present in the sound signal, for example in the form of background noise, talking voices, etc.
  • the sound stream is then captured into the recognition system either in realtime or presented offline, as with a sound file.
  • Real-time sounds may be sampled digitally and sent to the system by a sampling device such as a microphone, or be stored in a storage device such as an answering machine, computer file, tape recorder, telephone, mobile phone, radio, etc.
  • the sound signal may be subjected to further degradation due to limitations of the channel or sound capture device. Sounds may also be sent to the recognition system via an internet stream, FTP, or as a file attachment to email.
  • the sound signal is processed for recognition.
  • landmarks and fingerprints are calculated.
  • the resulting index set for exogenous sound sample is also a list of pairs (fingerprint, landmark) of analyzed values.
  • each fingerprint/landmark pair (fingerprints, landmarks) in the resulting input sound's index set is processed by searching for fingerprint k in the master index list.
  • Fast searching algorithms on an ordered list are well-known in the art and extensively discussed in Knuth, Volume 3 (ibid), incorporated by reference. If fingerprints is found then the corresponding list of matching (landmark* j , sound 13I ID j ) pairs having the same fingerprint is copied and augmented with landmark to form a set of triplets of the form(landmark k , landmark* j ,sound_ID j ). This process is repeated for all k ranging over the input sound's index set, with the all the resulting triplets being collected into a large candidate list.
  • the candidate list is compiled it is further processed by segmenting according to sound 13 ID.
  • a convenient way of doing this is to sort the candidate list according to sound 13 ID, or by insertion into a B-tree.
  • the result of this is a list of candidate sound_IDs, each of which having a scatter list of pairs of landmark timepoints, (landmark k ,landmark* j ) with the sound 13 ID stripped off.
  • landmark n is the corresponding timepoint within the exogenous sound signal
  • landmark* n is the corresponding timepoint within the library sound recording indexed by sound 13 ID
  • offset is the time offset into the library sound recording corresponding to the beginning of the exogenous sound signal.
  • the diagonal-finding problem is then reduced to finding multiple landmark pairs that cluster near the same offset value. This is accomplished easily by calculating a histogram of the resulting offset values and searching for the offset bin with the highest number of points. Since the offset must be positive if the exogenous sound signal is fully contained within the correct library sound recording, landmark pairs that result in a negative offset are excluded.
  • the winning offset bin of the histogram is noted for each qualifying sound 13 ID, and the corresponding score is the number of points in the winning bin.
  • the sound recording in the candidate list with the highest score is chosen as the winner.
  • the winning sound 13 ID is provided to an output means to signal the success of the identification.
  • a minimum threshold score may be used to gate the success of the identification process. If no library sound recording meets the minimum threshold then there is no identification.
  • the result is reported.
  • this may be done using a computer printout, email, SMS text messaging to a mobile phone, computer-generated voice annotation over a telephone, posting of the result to an internet account which the user can access later.

Abstract

A method and system for provides a user with an ability to capture a sample of an experiential environment and deliver that sample to an interactive service to trigger one or more predetermined events. In exemplary embodiments of the invention such triggered events include the delivery of information and services to the user, the execution of tasks and instructions by the service on the user's behalf, communication events, surveillance events and other control-oriented events that are responsive to the user's wishes. In other exemplary embodiments of the invention, the triggered events include transaction-oriented events, entertainment events, and events associated with enhancements to human ability or function.

Description

    BACKGROUND OF THE INVENTION
  • The present invention relates generally to methods and apparatuses for interacting with a user, and more particularly to a method and apparatus for interacting with a user in an experiential environment. [0001]
  • Methods and systems for interacting with a user exist. Examples include computer graphical user interfaces, computer pointing and clicking devices, sometimes referred to as “mice” to name just a few. These devices fail to take advantage of the experiential environment in which a user often finds oneself and therefore lack the ability to provide rich information to a computer as to the user's desires. [0002]
  • A wide variety of media is available through many delivery channels to a large audience. Rapidly evolving communication technologies such as telephony, television, pre-recorded media including CDs and DVD, radio, and the Internet in combination with traditional media such as print and visual arts operate to create a vast experiential environment for many people. Such environments are further enriched with other sensory experiences such as those coming from live performances, speeches and lectures, and routine day-to-day interactions with others. Ambient conditions like the sights and sounds of a bustling city street or the buzz of insects and chirping of birds in a more idyllic rural setting provide additional dimensions to the experiential environment. [0003]
  • In the midst of these experiential environments, users are often desirous of communicating with a computer or other automated processing device. Yet these devices cannot take advantage of the state of the user, which otherwise provides valuable information to one attempting to ascertain the needs of the user. [0004]
  • The present invention is therefore directed to the problem of developing a method and system for interacting with a user in a way that takes advantage of the rich and stimulating experiential environment to which the user is exposed to simplify the processing of user requests. [0005]
  • SUMMARY OF THE INVENTION
  • The present invention solves these and other problems by providing a method and apparatus for interacting with a user that employs a captured sample of an experiential environment in which the user exists as a command to trigger subsequent events. [0006]
  • A method and system for interacting with the user includes capturing a sample of an experiential environment and delivering that sample to an interactive service to trigger one or more predetermined events. In exemplary embodiments of the invention such triggered events include the delivery of information and services to the user, the execution of tasks and instructions by the service on the user's behalf, communication events, surveillance events and other control-oriented events that are responsive to the user's wishes. In other exemplary embodiments of the invention, the triggered events include transaction-oriented events, entertainment events, and events associated with enhancements to human ability or function. [0007]
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 depicts a block diagram of an arrangement in accordance with the invention including an experiential environment, a recognition engine and related structures, and an event trigger. [0008]
  • FIG. 2 depicts a block diagram of the relationship of elements of an exemplary embodiment of the invention among a service provider, service user, media provider, and experiential environment, among other elements.[0009]
  • DETAILED DESCRIPTION
  • At this point, it is worthy to note that any reference herein to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. The appearances of the phrase “in one embodiment” in various places herein are not necessarily all referring to the same embodiment. [0010]
  • The present invention includes a method and system for interacting with a user that includes providing a user with an ability to capture a sample of an experiential environment and delivering that sample to an interactive service to trigger one or more predetermined events. It is noted that the terms “service” and “service provider” and “system” as used herein include a service, service provider, and system, respectively, which employ the various principles of the present invention unless otherwise indicated. Some examples of events that can be triggered include communication events or surveillance events. [0011]
  • The experiential environment can be of any form where a message or information (e.g., content), or other sensory stimuli is available for the user to experience, and may come from many sources. Radio and television, pre-recorded audio and video (e.g. tapes, CDs, DVDs, MP3, etc.) signals on internet and computer-based systems, telephones, live demonstrations or performances, and sensory stimuli from ambient surroundings are several examples of elements which may form, alone or in combination, a typical experiential environment. Using a sampling device, such as an ordinary mobile (or cellular) phone, video recorder, or camera, or a personal digital assistant (PDA) or palm pad, the user captures a sample of the experiential environment and transmits the sample to a service provider employing the present invention. [0012]
  • Alternatively, a personal digital assistant or computer could be specially enabled to act as the interactive service itself by storing the database in its memory and performing the database query and processing without externally accessing the service. In those embodiments wherein the database includes song records, this can easily be incorporated in current capacity disk drives even for millions of songs. Thus, the personal digital assistant would provide complete end-to-end capability for a user and would not require a separate communication channel to the interactive service. In this embodiment, transmission to the service provider would only be needed for additional interaction and potentially for updates of the music database, such as periodically (e.g., weekly). [0013]
  • Another embodiment consists of a hybrid form of above mentioned capturing devices and communication channel. In this embodiment, a Palm handheld device, for example, captures the experimental environment and preprocesses the characteristics of this environment before sending it on to the service provider for recognition. This reduces the bandwidth required for the communication channel and thereby improves the response time, yet does not require the very high speed, high capacity hand held device required by the above embodiments. [0014]
  • The service provider may employ the sample by itself, may derive information from the sample, may use data known about the user (e.g., the user's identity and/or user profile), may accept input from the user, or may employ a combination of all such inputs, to trigger a predetermined event that is responsive to the user's needs. [0015]
  • As described in more detail in the various embodiments of the invention below, and as shown in FIG. 1, such events include the delivery of information and/or services to the user, the execution of tasks and/or instructions by the service on the user's behalf, and other control-oriented events that are responsive to the user's wishes. The triggered events also include transaction-oriented events, entertainment events, and events associated with enhancements to human ability or function. It is noted that combinations of triggered events, as shown in FIG. 1 and described in greater detail below, may also be advantageously used in certain applications of the invention. [0016]
  • A brief overview of one exemplary embodiment of the invention is now provided to briefly highlight the principles of the present invention. In this example (which is described in greater detail below), a method and system is provided for triggering an information delivery event to a user. A user listening to a commercial radio broadcast of popular music at home is interested in singing along with a particular song being broadcast, and thus desires to obtain songs lyrics. Using a conventional telephone, the user dials a predetermined telephone number to access a service utilizing the principles of the present invention, and then holds the telephone handset within audible distance of the source of the audio signal (i.e., the speaker of the user's home stereo), to capture a sample of the environment. In this example, the experiential environment includes the music playing on the user's home stereo. [0017]
  • A system employed by the service provider at the other end of the telephone automatically answers the phone, and using a recognition engine, compares the captured sample to music recordings in a database to identify the song and thereby trigger a desired event for the user. In this example, the triggered event is the delivery of the lyric information to the user that corresponds to the song recognized from the captured sample. [0018]
  • Event fulfillment, the delivery of the desired song lyrics to the user, may be accomplished in number of different ways in both real-time and delayed modes. In real-time mode, lyric data is delivered to so that the user can read the lyrics to sing along with the song, in time, as it is being broadcast. Accordingly, data for the song corresponding to the captured sample is read out of a database and transmitted by the service to a user device such as a computer, TTD device, SMS telephone or device, pager, or other personal communication device capable of displaying text. [0019]
  • Depending on the display device and user preference, the real-time delivery of lyrics may be transmitted in a batch arrangement where relatively large portions or the entire lyric data is downloaded at a time, or using a discrete arrangement where lyric data is automatically synchronized to the song and transmitted in pieces to the user as the song progresses. In the former case, the user may choose to synchronize the display of the lyrics to the song being played (so that the lyrics are displayed at the appropriate time interval in advance of the lyrics being sung or spoken by the song performer on the broadcast), or in some applications of the invention, the user device may be configured to perform synchronization of the batch transmission. In the latter case, the service transmits the lyric data to automatically synchronize to the broadcast song without any action required by the user or user device. Such automatic synchronization is implemented through a derivation of the temporal position of the captured sample within the recognized song. [0020]
  • The delayed mode of event fulfillment includes the delivery of lyrics corresponding to the sampled song sometime after the song broadcast. The same delivery methods as with real-time mode may be used along with delivery methods that are characterized as being more time-consuming such as mail, courier, and facsimile transmission. [0021]
  • In this exemplary embodiment of the invention, additional user input may be collected via voice or touch-tone (i.e., DTMF tones) to further control the lyric delivery event or trigger additional events such as transaction events. For example, by interacting with the user through the capture device or the delivery device, the telephone, and text-displaying device respectively, the service may provide purchase options to the user to obtain the record album containing the broadcast and identified song for which the lyrics were sought. [0022]
  • Other purchase options may also be provided to the user by the service provider. In addition to being able to purchase the identified music, the user may purchase other merchandise (which could be related to the music or not). For example, after purchasing the album containing the identified song, the user may choose to purchase tickets to an upcoming performance by the song's artist. Or, the user may inform friends or potential users of the service about the identification and/or album purchase. Such communication could be implemented by the service provider sending an email or SMS message on the user's behalf. [0023]
  • Additional details of the invention will now be described. Referring again to FIG. 1, a block diagram of an exemplary arrangement of the invention is depicted. A sample captured from an experiential environment [0024] 101 is represented by block 105. The experiential environment 101 that may be used by the present invention includes virtually any environment where sensory stimuli are present for a user to sense or experience. Typically, elements of such experiential environment 101 include, singularly or in combination, media or other information content (each singularly or in combination referred to as “media” here) delivered in a channelized stream and media delivered in unchannelized streams that are generally sensed via sight or sound. However, it is emphasized that sight and sound are not necessarily the only sensed experiences that may be used in applications of the invention.
  • Channelized media as used here means media that is delivered to a user through discrete and identifiable means through such conventional sources such as television, radio, pre-recorded audio and video (e.g. tapes, CDs, DVDs, MP3, streaming IP video and audio), computers, telephones (both wired and wireless), and messaging devices such as SMS devices and pagers. Thus, while channelized media encompasses both unidirectional broadcasts, and bi-directional communications, it also includes print media such as newspapers, books, and magazines and the visual arts. [0025]
  • Unchannelized media means media that is delivered to a user through means that may be indistinct, less readily identified, or involve multiple sources of media. Examples of unchannelized media include the performing arts, speeches and lectures and the sensory experiences derived from interactions with other persons, or the surrounding environment. Unchannelized media further includes sights, sounds, and other sensory stimuli that result from a user's ambient surroundings. [0026]
  • In accordance with the invention, a user (not shown in FIG. 1) captures a sample of the experiential environment [0027] 101 via line 103 using a capture device 102. The capture device 102 captures a sample of experiential environment 105 and provides it in a format suitable for processing by the recognition engine 110 on line 117. Illustrative embodiments of capture devices include, but are not limited to, microphone, telephone, mobile telephone, PDA, tape recorder, digital voice recorder, answering machine, radio receiver, walkie-talkie, internet streaming socket, voice-over-IP transceiver, or other such audio sampling devices. Typically then, the signal capture device is incorporated into a device that the user employs in a location remote from the service. Conventional devices like mobile and regular telephones, PCs, radios, and other recording and communication devices that users already own or use everyday for other purposes may conveniently be used, without modification, to sample both channelized and unchannelized media.
  • Video or image capture devices may be also be used as [0028] capture device 102, in accordance with the invention to provide a sample of experiential environment 105 to recognition engine 110 via line 107, as depicted. For example, camcorders, video conferencing equipment, motion and still video cameras, surveillance devices, cameras, digital cameras, image grabbers, picture telephones and other devices that can capture an image sample of an experiential environment may be conveniently used to provide a sample to recognition engine 110.
  • User input/[0029] identification block 120 may identify the user to the recognition engine 110 as indicated by line 122, and may optionally be operationally coupled to the event trigger block 130 via line 124. Examples of devices that generate the appropriate identification for use with user input/identification block 120 may include caller ID on a POTS (Plain Old Telephone Service) line or a mobile telephone, internet IP address of a terminal sending in the captured signal, or a cookie file stored on an internet browser on the user's terminal.
  • In another illustrative example of user input/[0030] identification block 120, the user may be required to enter an account code, for example by keying it in on a touchtone pad on a telephone or saying a pass phrase while signing on to a service incorporating the principles of the present invention if dialing in. Alternatively, the user may be identified by inserting an object carrying identification codes into a terminal. Examples of this include a credit card, ATM card, or Dallas Semiconductor Java Ring. The user may also be identified by a biometric device to scan fingerprints, retinas, palm print, or other such physical characteristics of the user. A speaker identification system to identify the user by vocal characteristics is another alternative method. User input/identification block 120 is an optional component of the present arrangement, which is employed if billing and tracking of user activity is desired.
  • As shown in FIG. 2, the elements shown and described in FIG. 1 are typically associated with entities that are independent of one another. Experiential environment [0031] 101 is typically associated with a media operator or content provider such as radio or television broadcasters, CATV provider, internet service providers, entertainment organizations, private network or LAN operators, and the like. However, it is important to emphasize that the present invention contemplates that the experiential environment may comprise a live demonstration or performance, or other unchannelized media, taking place for example, at a nightclub, bar, or discotheque. Moreover, as described above, the experiential environment may include sights, sounds, and other sensory stimuli from the ambient surroundings.
  • Experiential [0032] environment capture device 102 is generally associated with users, however, such association may be merely temporary, as public access devices (e.g., public telephones and internet access facilities) may be readily used, without modification, in order to realize the benefits provided by the present invention. Experiential environment sample 105 represents features and functionalities that, for example, are implemented by the microphone and associated transceiver circuits in a user's mobile telephone or computer. Such features and functionalities could also be implemented by a charge coupled device (“CCD”) array that is integrated in a camera associated with a picture telephone or computer. User input/ID block 120 is also associated with the user.
  • The remaining elements of FIG. 1 are collectively associated as indicated in FIG. 2 with a service provider. [0033] Recognition engine 110, reporting and transaction block 137, database 115, and user profile/transaction history 135 represent features and functionalities of an integrated system that form key elements of an interactive service arrangement that may be particularly useful in some applications of the invention. In such interactive service arrangements, these collected elements are typically implemented in a system formed by one or more CPUs. The interactive service arrangement is identified by reference numeral 250 in FIG. 2.
  • As depicted in FIG. 2, a [0034] media provider 210 utilizes signal source 201 which transmits signal 203 via media distribution network 220 which may be arranged from network and transmission elements or other channelized distribution arrangements (as for example, with copper or fiber networks for data or telephony services) or a free space/transmitter infrastructure array (as for example with radio and television broadcasts, satellite systems, and cellular/PCS/GSM wireless telephone services or networks which operate according to short-range wireless protocols such as the Bluetooth™ wireless standard).
  • Receiver/[0035] monitor 240 is employed by user 230 to receive signal 203 and transform the signal into a format that facilitates signal 203 to be monitored by the user. Receiver/monitor 240 may be a radio, television, PC, Hi-fi (i.e., stereo) with speakers or any other device that may be used to create a media experience (including audio and video) that may be monitored by the user. User 230 using, for example, a mobile telephone, obtains a sample of signal 203 played on receiver/monitor 240, where the sample includes media content of interest selected by the user, such as a portion of an image or sound.
  • Thus, as shown in FIG. 2, receiver/monitor [0036] 240 both outputs a rendition of signal 203 to 105 and allows the user 230 to monitor signal 203. However, it is noted that signal capture device 102 may capture a sample of signal 203 via a direct connection to media distribution network 220 (i.e., not relying on receiver/monitor 240 or similar device for signal input). In such instances, the user monitoring of signal 203 is accomplished through other means or user monitoring is not performed.
  • In the case of unchannelized media, capture device captures a sample directly from experiential environment [0037] 101 via line 222 without relying on a media distribution network 220. In such cases, a signal source transmits directly into the environment, as indicated by line 221, or the experiential environment is itself being captured by capture device 102. For example, as described above, the arrangement of the present invention may be advantageously used to trigger a variety of predetermined events from a sample of live performances, interaction, or the sights and sounds of the ambient surroundings.
  • The captured sample of [0038] signal 203 is relayed by the user 230 to the interactive service arrangement 250, as shown, via communication network 260 on line 207, as shown. Communication network 260 may have a similar arrangement as distribution network 220, or may be unitary with distribution network 220. It should be noted, however, that in certain applications of the invention distribution network 220 may typically be characterized by unidirectional signal propagation (as in the case with broadcast radio and television or typical CATV head end systems) while communication network 260 may typically be characterized by bi-directional signal propagation (as in the case with the public switched telephone network and wireless or wired voice, data, and internet systems). Such bi-directional nature of communication network 260 is indicated by signal flow lines 207 and 208 as depicted in FIG. 2.
  • In accordance with the invention, the [0039] interactive service arrangement 250 derives information or characteristics of sample of signal 203. Such derived information may be returned to the user 230 from the interactive service arrangement 250 using the same communication network 260 or other networks. The signal return path is indicated with lines 208 in FIG. 2. And, as described above, the interactive service arrangement 250 may interact with the user and other entities. For illustrative purposes, such interaction pathways are depicted in FIG. 2 as lines 289 and 282, input and output, respectively, via alternate network 280. Alternate network 280 may be a network of any type, however, in some applications of the invention it may be advantageous to employ private networks, dedicated lines, or other high-capacity transmission methods should high-bandwidth interactions be desired. Such bandwidth intensive interactions could occur, for example, between the interactive service arrangement 250 and fulfillment partners such as record distributors or merchandise retailers. This communication path is shown in FIG. 2 where fulfillment center 286 interacts with interactive service arrangement 250 using alternate network 280 and fulfills user requests as indicated by line 288. Fulfillment center 286 may also interact with interactive service arrangement 250 using communication network 260 over interaction pathways 292 and 294 as indicated.
  • Thus, FIG. 2 may serve to additionally highlight the principles applied in the exemplary embodiment of the invention that was briefly overviewed at the beginning of the Detailed Description. A radio station (media provider [0040] 210) employing signal source 201 broadcasts a song (signal 203) over the air (media distribution network 220), which is received on a radio (receiver/monitor 240) of user 230.User 230 wishes to sing along with the song. User 230 places a mobile telephone call over a wireless network (communication network 260) to interactive service arrangement 250. User 230 positions the microphone of his wireless telephone (capture device 102) to capture a sample of the music being played on the radio. The interactive service arrangement 250 receives the sample over the wireless network and derives the identity of the song.
  • Optionally, the identity of the user may be derived from the user's mobile telephone number that is sent from the communication network [0041] 260 to the interactive service arrangement 250 typically during call set up. The song lyrics are returned to back to the user's mobile telephone over the same network. Further interactions between the user 230 and the interactive service arrangement 250 may occur, and if the user chooses to purchase related merchandise (such as a CD recording of the album containing the broadcast song, or hardcopy printout of the desired lyrics), the interactive service arrangement 250 can send the purchase information to the service's distribution facility (fulfillment center 286) via data or voice communication using its local area PC network (alternate network 280) or via data or voice communication over communication network 260 as discussed above.
  • The distribution center sends the merchandise to the user via overnight courier service (line [0042] 288). Similarly, alternate network 280 may be utilized to send an email or facsimile containing the song lyrics to the user's email account.
  • In some embodiments of the invention, the service relies upon a profile for the user that is created in advance of the call (but may be updated during a call as described below). A “user profile” may typically include general information about the user such as name, address, preferred method of payment (i.e., credit card pre-authorization), and set dollar limits on purchases. In addition, service-specific information regarding the user may also be included in the profile, such as demographic and user-identified preference information, to facilitate the service tailoring the transaction to fit a particular user. Moreover, with automatic telephone number identification, i.e., “caller ID”, profiles can be built without prior user registration. [0043]
  • Age, education, residence, gender, occupation, and personal interests, likes and dislikes, among other criteria, may be employed to most effectively match transaction offers to users' interests and purchase habits. For example, one particular customer of the service may have a user profile that indicates that the user is a member of a demographic group that is music-savvy and aware of music trends. After offering to sell a recording of the song selected by the user, the service could offer to sell a recording by an artist that is “moving up the charts” in popularity. Thus, by employing a user profile in some applications of the invention, a higher transaction closing rate may be realized as offers are more accurately targeted to users who may be predisposed to view the offer favorably. [0044]
  • It is noted that user profiles are often dynamic and that the present invention includes an ability to update and change a user profile in response to changes in usage patterns (as described below), market and industry trends, user input, and other factors. [0045]
  • A feature of the invention related to user profile is usage tracking. By tracking the frequency and time of access, and purchase history, for example, of a user, the service can gain additional insight into factors which may influence a user. Patterns of usage may be derived which may allow predictive modeling to be utilized, for example, to enhance and refine service offerings. The system of the present invention can thus differentiate between repeat users (who heavily access the service) and casual users (who occasionally or infrequently use the service) and adjust the options, offers, and interactive scripting (as described below) so that interest and enthusiasm is maintained among the service's customers. It is contemplated that the user profile and usage tracking/pattern features described here may be used in other embodiments and applications of the inventions as well. [0046]
  • User profile and usage tracking are examples of a priori information (i.e., information known by the service provider in advance of a specific interaction with a user) and external information (i.e., information known by the service provider that is determined other than by analyzing the captured sample itself. It will be recognized that other sources of a priori and/or external information may also be advantageously used in some applications of the invention. For example, the location of the user, caller ID signal, dialed number indication service (“DNIS”) and the capture device employed by the user may, alone or in combination, be used by the service provider to select or otherwise tailor the triggered events to meet user expectations or needs. [0047]
  • Referring again to FIG. 1, the experiential environment sample is received by [0048] recognition engine 110 on line 117. Recognition engine 110 derives characteristics of the received sample by using data stored in database 115. Recognition 110 and database 115 are operationally coupled via line 119, as shown in FIG. 1. A variety of derivation methods may be used. In the case of audio samples, the techniques described in Appendix may be used. However, it is noted that the derivation methods that may be used in this invention are not limited to such techniques. The particular derivation method chosen is only required to be able to derive sufficient characteristics from the experiential environment sample to enable a predetermined event to be triggered. Thus, the strength or resolution of the derivation is coupled, in large measure, to the level of discrimination required in selecting an event to be triggered. As the number of potentially triggered events increases, the necessity to resolve ambiguity in the sample also increases.
  • For example, in the case of the exemplary embodiment where song lyrics corresponding to a broadcast song are sought by a user, a relatively large number of characteristics about the sample may be derived and compared against stored data to be able to identify the particular song from the many such songs that may be stored. That is, as more songs are potentially identified, more lyric delivery events are potentially triggered. By comparison, in se vice offerings where are relatively small number of events are potentially triggered, fewer sample characteristics need typically be derived in order to resolve ambiguity as to which event to trigger. Such service offering may include those where a binary “Yes” or “No” event may be triggered as may be the case for customer surveys and voting/polling type services. [0049]
  • The derived sample characteristics are provided to [0050] event trigger 130 on line 128, as indicated in FIG. 1. Event trigger 130 may trigger one or more predetermined events according to the derived sample characteristics. The triggering criteria used by event trigger 130 may be provided by the characteristics themselves or be externally provided by database 115. Alternatively, in the case of binary type events, the mere receipt of any input signal at the event trigger may be sufficient to trigger a particular event. As used here, triggering criteria means those factors, data, or inputs, which are used by the event trigger 130 to select the particular one (or group) of actions to take among the many that may be available.
  • While triggering criteria are provided by [0051] recognition engine 110 as a result of the derivation of characteristics from the captured experiential environment sample, additional triggering criteria may also be used in combination with the derived characteristics. As depicted in FIG. 1, such additional triggering criteria may come from the user on line 124 as user input in the form of voice, audio or visual commands. For example, a user may provide input to the event trigger by using the touch tone keypads on a telephone, providing keyboard input on a computer, or providing captioning information over a video feed using a video camera. In addition, user profile and transaction history information, as described above, may be used as additional triggering criteria. As shown in FIG. 1, user profile/transaction history block 135 is operably coupled to event trigger 130 via line 131 to provide such additional triggering criteria. However, in some applications of the invention, user profile/transaction history block 135 may provide user and/or transaction data to recognition engine 110 (not shown in FIG. 1) that may be used to assist in the characteristic derivation process. For example in a music identification embodiment of the invention, recognition engine 110 may, in certain applications, rely on the particular user's usage history which indicates a strong preference for a specific music genre. This usage history may help eliminate some potential identification candidates that may have some sonic correlations to stored data in database 115, but are very dissimilar to the types of songs sought to be identified by the user in the past and stored in user profile/transaction history 135.
  • Reporting/[0052] transaction block 137 is operably coupled to the event trigger 130 via line 139 as shown so that event usage may be tracked by the service provider for purposes of creating reports or billing transactions to the user.
  • As shown in FIG. 1, [0053] event trigger 130 can trigger a variety of events indicated collectively by reference numeral 180. It is noted that the depicted events are merely illustrative and are shown to highlight various categories or types of events that may be triggered using the principles of the invention.
  • [0054] Transaction events 182 are one category of events that may be triggered in accordance with the invention. These typically include the purchase and sale of merchandise. In an exemplary embodiment of the invention, a user reading the newspaper may see an advertisement for a raincoat that stimulates a buying interest in the user. Using a video camera on the user's PC, the user holds the print advertisement near the camera to capture a still image of the advertisement. By using a predetermined web address of the World Wide Web, the user reaches a service provider and uploads to the image to the service provider's recognition engine. THe advertisement could be included in a billboard, in which case the user could use a still camera or video camera to capture it.
  • The recognition engine, using conventional text recognition algorithms, pulls the name of the advertiser and advertising copy for the raincoat from the still image. Employing an interactive script on the web-page, the user is prompted for size and color preferences. The user's credit card and shipping information may already be on file with the service provider using user profile/[0055] transaction history 135, or such information may be collected from the user on-line through the web-interface or using off-line communications such as telephone, email, or facsimile.
  • The derived characteristics, namely the advertiser's name and clothing information, is combined with information about the advertiser stored on database [0056] 115 and used by event trigger 130 to trigger a purchase by the user of the desired raincoat. The transaction particulars are forwarded to an event fulfillment center, depicted in FIG. 1 by reference numeral 190, for actual physical fulfillment of the raincoat to the event recipient 195, in this case, the user. In this exemplary embodiment of the invention, the service provider may merely function as a deal broker between the advertiser and user where the order information is provided to the advertiser and the service provider collects a service fee from the advertiser. Alternatively, the service provider may derive revenue from charging the user a subscription fee to use the service, which facilitates the transaction as described above. In some applications of the invention, a combination of these revenue models may be advantageously used.
  • Entertainment events [0057] 184 may also be triggered in accordance with the invention. In addition to the exemplary embodiment involving the delivery of lyrics, as described above, a wide variety of entertainment-type services may be advantageously provided by the practice of the invention. Entertainment is defined here to include services where interaction with the service directly provides the value to the user. This compares with some typical transaction events where the service provider furnishes a tangential broker or facilitation service, which is attendant to the purchase and sale transaction between the user and a third party.
  • An illustrative example of an entertainment event [0058] 184 provided by the present invention includes game-type events. A user listening to music captures a sample of a song on a mobile telephone, which has been dialed in to a service provider. The recognition engine 110 compares the sample to known songs in database 115 and derives the identity of the song. A game type entertainment event is then triggered by the service to automatically send a small excerpt of the originally recorded song (i.e., not the captured sample of the song) to a pre-determined group of the user's friends via. For example, a WAV file or MP3 attachment to an email message sent by the service provider or through a call placed by the service to the friend's wired or wireless telephone. The identification data for members of the group may be stored in a user profile, for example.
  • The friends are prompted in the message or call to try to “Name that Tune” by identifying the song's title or artist from the small excerpt. The friend's guesses may be collected by the service provider using a variety of methods, including for example, an interactive web-site, telephone call center, email, or conventional mail. If no one correctly identifies the song, the song excerpt may be increased in length, or a different excerpt may be furnished, in an iterative manner, until a “winner” is determined. [0059]
  • Human [0060] ability enhancement events 186 are also depicted in FIG. 1. These events are similarly triggered by captured samples of an experiential environment. In embodiments of inventions using these events, enhancements to ability and function are realized by the ability of capture devices 102 to sense the environment in manners that transcend human ability or in manners that replace ordinary senses that are undeveloped or lost through illness or accident in particular individuals.
  • For example, in the case of sensory enhancement, capture devices such as telephones may quite easily arranged to be more sensitive to sound or to particular frequencies of sound than the human ear through the use of amplifiers and filters. Similarly, visual capture devices may be arranged to provide magnification of far away scenes or make small scenes appear large. Thus, an ability to extend one's senses through the use of technology, in accordance with the invention, provides great opportunity to trigger desired events that could otherwise not occur with unaided senses. In an exemplary embodiment of this aspect of the invention, a bird watching user may wish to keep a catalog of sightings in the neighborhood. Using a digital camera with a zoom lens, the user takes a digital picture of a distant bird and the captured image is later uploaded to a predetermined service provider's computer. The [0061] recognition engine 110 could be configured to recognize the bird based on characteristics such as colors and patterns, however, in other applications, recognition engine 110 may more simply be configured to derive a smaller constellation of key characteristics. For example, the recognition engine may derive that the captured experiential environment sample represented is a color digital photograph and trigger an image archiving event for the user. Or, dominant colors may be derived to allow an image-sorting event to be triggered. In both of these events, the service provider uses the derived characteristics of the captured sample (the digital photographs) to automatically archive and arrange the photos in a manner that is consistent with a service offering in which the user participates.
  • Another exemplary embodiment of sensory enhancement is the ability for a user to implement a virtual sense in a remote location. For example, a mother may dial a predetermined number to access a service provider using a conventional telephone located in a sleeping child's room. The telephone is left off hook to constantly sample the ambient conditions in the room. The sampled environment is processed to determine whether the child has awoken by listening for the frequency and amplitude characteristics associated with crying or yelling and places a call or page to the mother to let her know of the change in circumstances. In such cases, the database [0062] 115 and event trigger 130 are arranged to trigger the call or page based on the sound characteristics of crying, but other events could be triggered, such as a call to fire department, using other sound characteristics such as the timbre of a smoke or carbon monoxide detector.
  • In the case of sensory replacement, capture devices replace senses lost or undeveloped in an individual user, or augment existing senses to make up for a lost sense. In these cases, capture devices may be arranged to capture a sample of an experiential environment to trigger events that can provide assistance that more typical sensory aides cannot. For example, a visually impaired user may configure video equipment to monitor the entrance to the home. A service receiving the video feed may sample the scene at the remote end to determine when sufficient changes have occurred in the captured scene to determine that a person is at the door. If desired, the user may provide data to the service to allow particular individuals to be recognized and announced to the user through a telephone call, page, SMS message, email, and the like. [0063]
  • Block [0064] 188 indicates that the execution of task and instruction events are provided by the invention. In embodiment of the invention using this feature, a variety of predetermined events are triggered by the receipt of a particular sample that involve a service performing actions on behalf of the user. Tasks such as bill or invoice payment may be readily automated by uploading a scanned image of the bill to a service provider. The service provider, upon receipt of the captured image, and optionally in combination with user profile data, can derive the identity of the payee and automatically pay the bill on behalf of the user and debit the user's bank account or credit card.
  • [0065] Block 170 in FIG. 1 depicts that events for the delivery of information can be triggered in response to the receipt of particular captured samples. In embodiments of the invention utilizing this feature, a service provider triggers the delivery of information to the user or third parties. For example, a service provider may receive a set of sound samples from a user's microphone attached to a computer that is monitoring financial news broadcasts. Deriving the names of companies from the sampled news that the user has indicated an interest in monitoring, the service automatically downloads stock quotes and related news information to the user's computer.
  • In a similar manner as with the delivery of information, delivery of services events, as shown in [0066] block 172 may also be triggered in accordance with the invention. Continuing with the financial example above, a service provider may provide financial documentation services to the user such as the automated provision of public securities and investment analysis documents based on the companies of interest that are discussed in the monitored news broadcast. In such embodiments of the invention, the user may provide other event triggering criteria such as the minimum number of utterances of a particular company name in a particular time period before the automated provision of documents is triggered.
  • [0067] Block 174 in FIG. 1 shows that control events may also be triggered in response to a sampled experiential environment in accordance with the invention. Control events are those that provide the user with an ability to control or otherwise manipulate information and data, services, or other events in a predetermined manner according to the captured sample received by a service provider. For example, a human resources recruiter may organize a data archive of job candidates and associated demographic data by engaging a service provider that automatically manipulates the data according to web-site images of potential hiring companies that are captured in a frame grabber running on the user's computer and uploaded to the service provider. In such cases, the candidate database can be sorted according the to captured web-document and derived by deriving preselected characteristics such as industry type, key-words in the text elements of the page, and other characteristics.
  • Communication events may be triggered in accordance with the invention as depicted by block [0068] 175 in FIG. 1. Communication events include, for example, communicative interactions among users, between users and the service provider, or such interactions between users, the service provider, and third parties.
  • An exemplary embodiment of the invention involving a triggered communication event follows. In this embodiment, a user listening to music dials into a service and captures a sample of a song on his mobile telephone. The [0069] recognition engine 110 compares the sample to known songs in database 115 and derives the identity of the song. In accordance with the invention, the service then triggers a communication event to notify a second user that the first user engaged the service to identify the song. The notification may be implemented, for example, by the service sending an email or SMS message to the second user. An excerpt of the original recording of the identified song may optionally be attached in the message for the second user to hear. This embodiment of the invention may be of particular relevance to persons interested in music trends who want to communicate, share, or kept be abreast of the latest developments in music.
  • The particular user or group of users chosen to be notified by the user may be selected in advance, for example, by employing a user profile that stores the email address or SMS number of the users to be notified. Or, the user may interact with the service via a conventional interactive voice response unit to indicate a choice relatively contemporaneously with the sample capture process. [0070]
  • In some applications of the invention, the dial-in number to the service may be used to indicate a choice by the user. For example, the user may program his mobile phone (or the phone may be preprogrammed by the service provider or partner) so that a plurality of numbers are stored, each indicating a particular user selection to the service. If, for example, the mobile phone stores three dial-in numbers, say *211, *212, and *213, each number may be used to reach the service while indicating a particular user choice. While all the numbers dial into the same service provider, the fact that a particular dialed number was employed by the user is given to the service provider by communication networks [0071] 260 (FIG. 2) via a dialed number identification service (“DNIS”). In the music trend embodiment described above, *211 could be used to indicate a user desire to trigger a particular communication event, according to a pre-programmed user profile, for example, to notify user A that a song identification was sought by the user. Similarly, *212 could indicate communications with users A and B, and *213 could indicate the user desire to notify and provide a song excerpt to a pre-defined third party.
  • In other applications of the invention, the DNIS may be advantageously utilized to allow user selection of a triggerable event from a plurality of events that may be hosted by a service provider. If, for example, a single service provider hosts both shopping services (a transaction event) and game services (an entertainment event), then the DNIS may be used by the service provider (perhaps among other factors used) to disambiguate among the offered events and trigger the particular event desired by the user in response to a triggering characteristic derived from the experiential environment sample received from the user. [0072]
  • An embodiment of the invention similar to that above also involves communication events. In this embodiment, the user captures a sample of a song to trigger a communication event, which is embodied in the form of a musical greeting card. Such musical greeting cards include an excerpt of the original recording of the identified song with a text or voice annotation from the user that may be personalized to a specific occasion or event. Thus, as the two communication event embodiments described above demonstrate, the principles of the present invention may be advantageously utilized to trigger user-to-user (also referred to as “peer-to-peer”) services in response to a received sample of a captured experiential environment. [0073]
  • Surveillance and/or monitoring events are also contemplated by the present invention, as depicted by [0074] block 176 in FIG. 1. In such events, a sample or series of samples of an experiential environment is captured by the user and relayed to the service provider in order to trigger events that may facilitate the detection of the presence or absence of one or more characteristics, elements, or features in the environment, or a change in such environment.
  • In a first embodiment of the invention involving a triggerable surveillance or monitoring event, a chemical detection device or “sniffer” is employed by a user to sample the air in the environment. Such sniffers are known and may employ various sensory techniques, such as gas chromatography, to determine the presence and concentration of chemical elements or compounds in the air sample. A signal indicative of the elemental constituents of the sample is forwarded to the service provider whereby analysis of the signal is performed to derive characteristics of the sample used to trigger a desired event. For example, an agricultural user may desire to monitor the conditions of crops. By monitoring the emission of certain gases associated with the decomposition of fertilizer and organic matter, and other gases produced by the growing crops, the user may optimize the application of water, fertilizers and pesticides to produce the highest yield. The service provider can trigger an alert to the user, via facsimile, email, mail, phone call, SMS message, or a variety of other communication methods, when a particular threshold condition has been met or exceeded. [0075]
  • Advantageously, the present invention separates the experiential environment sensing apparatus (i.e., the sniffer) from the analysis and event triggering apparatus (i.e., the [0076] recognition engine 110 and event trigger 130 in FIG. 1). The sensing apparatus is associated with the user who can best control its deployment in the experiential environment, while the analysis and event triggering apparatus is detached from the user. Such detachment allows the service provider the opportunity to host a wide variety of surveillance and monitoring events that would be impractical to implement in a portable local device. For example, in accordance with the invention, a single sniffer may be used to monitor both crop condition, as described above, as well as methane levels in effluents from a milk parlor of a dairy farm. The required analysis of the gas samples, and the determination of whether a alarm condition exists would be based on criteria that vary widely between these two applications, yet the service provider could readily trigger the different monitoring events due to flexibility associated with remote analysis/trigger arrangement.
  • A variation of this first embodiment includes an arrangement where a scaled down sniffer is carried by a user. While such a device would be limited in the range of elements it could detect, the flexibility of the remote analysis and triggering arrangement of the present invention still may provide many advantages. That is, a less sensitive sensing device, in accordance with the invention, may be coupled with a remotely located, flexible and powerful analysis tool to trigger useful and desirable events. For example, a user with a sensitivity to a particular environment condition, such as an allergy to a specific food, may carry a small device that has sensitivity to the chemical signature of that food. The device samples the environment periodically and sends a representative signal via a wireless transceiver to the service provider for analysis. Upon derivation of characteristics indicating an alert event, the service provider can communicate back to the user, via the same sensing device or using alternative communication pathways such as telephone, pager or fax. Optionally, the alert signal may be sent by the service provider to third parties including, for example, the user's doctor, emergency contact or others as conditions warrant. [0077]
  • In a second embodiment of the invention involving a triggerable surveillance or monitoring event, a spectrum analyzer is configured to sample the strength of constituent wavelengths of light in an experiential environment. A particular spectral sample of the environment can be compared by the service provider against stored signatures to trigger a variety of surveillance or monitoring events. For example, the level of ultraviolet radiation in a warehouse may be compared against stored data to trigger remote control of vents, skylights, or other HVAC devices in the building by the service provider. [0078]
  • [0079] Block 178 in FIG. 1 indicates that various combinations of the events shown and described in blocks 182-188, and 170-176 are also contemplated as being advantageously used in some applications of the invention. That is, some applications may benefit by triggering multiple events, in sequence or in parallel, that may be of the same or differing types.
  • Other embodiments of the invention may be implemented in accordance with the claims that follow. [0080]
  • APPENDIX 1
  • Abstract [0081]
  • We disclose a method and apparatus for recognizing sound, music, and other similar signals. The disclosed invention is capable of recognizing an exogenous sound signal that is a rendition of a known recording indexed in a database. The exogenous sound signal may be subjected to distortion and interference, including background noise, talking voices, compression artifacts, band-limited filtering, transmission dropouts, time warping, and other linear and nonlinear corruptions of the original signal. The algorithm is capable of identifying the corresponding original recording from a large database of recordings in time proportional to the logarithm of the number of entries in the database. Given sufficient computational power the system can perform the identification in nearly real-time, i.e. as the sound is being sampled, with a small lag. [0082]
  • Database Construction [0083]
  • The sound database may consist of any collection of recordings, such as speech, music, advertisements, or sonar signatures. [0084]
  • Indexing [0085]
  • In order to index the sound database, each recording in the library is subjected to landmarking and fingerprinting analysis to generate an index set for each item. Each recording in the database has a unique index, sound[0086] 13 ID.
  • Landmarking [0087]
  • Each sound recording is landmarked using methods to find distinctive and reproducible locations within the sound recording. The ideal landmarking algorithm will be able to mark the same points within a sound recording despite the presence of noise and other linear and nonlinear distortion. The landmarking method is conceptually independent of the fingerprinting process, but may be chosen to optimize performance of the latter. Landmarking results in a list of timepoints {landmark[0088] k} within the sound recording at which fingerprints should be calculated. A good landmarking scheme marks about 5-10 landmarks per second of sound recording, of course depending on the amount of activity within the sound recording.
  • Power Norms [0089]
  • A simple landmarking technique is to calculate the instantaneous power at every timepoint and to select local maxima. One way of doing this is to calculate the envelope by rectifying and filtering the waveform directly. Another way is to calculate the Hilbert transform (quadrature) of the signal and use the sum of the magnitudes squared of the Hilbert transform and the original signal. [0090]
  • Spectral Lp Norms [0091]
  • The power norm method of landmarking is especially good for finding transients in the sound signal. The power norm is actually a special case of the more general Spectral Lp Norm, where p=2. The general Spectral Lp Norm is calculated at each time along the sound signal by calculating the spectrum, for example via a Hanning-windowed Fast Fourier Transform (FFT). The Lp norm for that time slice is then calculated as the sum of the p-th power of the absolute values of the spectral components, optionally followed by taking the p-th root. As before, the landmarks are chosen as the local maxima of the resulting values over time. [0092]
  • Multislice Landmarks [0093]
  • Multi-slice landmarks may be calculated by taking the sum of p-th powers of absolute values of spectral components over multiple timeslices instead of a single slice. Finding the local maxima of this extended sum allows optimization of placement of the multislice fingerprints, described below. [0094]
  • Fingerprinting [0095]
  • The algorithm computes a fingerprint at each landmark timepoint in the recording. The fingerprint is generally a value or set of values that summarize a set of features in the recording near the timepoint. In our implementation the fingerprint is a single numerical value that is a hashed function of multiple features. [0096]
  • The following are a few possible fingerprint categories. [0097]
  • Salient Spectral Fingerprints [0098]
  • In the neighborhood of each landmark timepoint a frequency analysis is performed to extract the top several spectral peaks. A simple such fingerprint value is just the single frequency value of the strongest spectral peak. The use of such a simple peak resulted in surprisingly good recognition in the presence of noise, but resulted in many false positive matches due to the non-uniqueness of such a simple scheme. Using fingerprints consisting of the two or three strongest spectral peaks resulted in fewer false positives, but in some cases created a susceptibility to noise if the second-strongest spectral peak was not sufficiently strong enough to distinguish it from its competitors in the presence of noise—the calculated fingerprint value would not be sufficiently stable. Despite this, the performance of this case was also good. [0099]
  • Multislice Fingerprints [0100]
  • In order to take advantage of the time-evolution of many sounds a set of timeslices is determined by adding a set of offsets to a landmark timepoint. At each resulting timeslice a Salient Spectral Fingerprint is calculated. The resulting set of fingerprint information is then combined to form one multitone fingerprint. Each such fingerprint is much more unique than the single-time salient spectral fingerprint since it tracks temporal evolution, resulting in fewer false matches. Our experiments indicate that using two or three timeslices along with the single strongest spectral peak in each timeslice results in very good performance, even in the presence of significant noise. [0101]
  • LPC Coefficients [0102]
  • In addition to finding the strongest spectral components, there are other spectral features that can be extracted and used as fingerprints. LPC analysis extracts the linearly predictable features of a signal, such as spectral peaks, as well as spectral shape. LPC coefficients of waveform slices anchored at landmark positions can be used as fingerprints by hashing the quantized LPC coefficients into an index value. LPC is well-known in the art of digital signal processing. [0103]
  • Cepstral Coefficients [0104]
  • Cepstral coefficents are useful as a measure of periodicity and may be used to characterize signals that are harmonic, such as voices or many musical instruments. A number of cepstral coefficients may hashed together into an index and used as a fingerprint. Cepstral analysis is well-known in the art of digital signal processing. [0105]
  • Index Set [0106]
  • The resulting index set for a given sound recording is a list of pairs (fingerprint, landmark) of analyzed values. Since the index set is composed simply of pairs of values, it is possible to use multiple landmarking and fingerprinting schemes simultaneously. For example, one landmarking/fingerprinting scheme may be good at detecting unique tonal patterns, but poor at identifying percussion, whereas a different algorithm may have the opposite attributes. Use of multiple landmarking/fingerprinting strategies results in a more robust and richer range of recognition performance. Different fingerprinting techniques may be used together by reserving certain ranges of fingerprint values for certain kinds of fingerprints. For example, in a 32-bit fingerprint value, the first 3 bits may be used to specify which of 8 fingerprinting schemes the following 29 bits are encoding. [0107]
  • Searchable Database [0108]
  • Once the index sets have been processed for each sound recording in the database, a searchable database is constructed in such a way as to allow fast (logtime) searching. This is accomplished by constructing a list of triplets (fingerprint, landmark, sound_ID), obtained by appending the corresponding sound[0109] 13 ID to each doublet from each index set. All such triplets for all sound recordings are collected into a large index list. In order to optimize the search process, the list of triplets is then sorted according to the fingerprint. Fast sorting algorithms are well-known in the art and extensively discussed in D. E. Knuth, “The Art of Computer Programming, Volume 3: Sorting and Searching,” hereby incorporated by reference. High-performance sorting algorithms can sort the list in N log(N) time, where N is the number of entries in the list. Once this list is sorted it is further processed by segmenting it such that each unique fingerprint in the list is collected into a new master index list. Each entry in this master index list contains a fingerprint value and a pointer to a list of (landmark, sound13 ID) pairs. Rearranging the index list in this way is optional, but saves memory since each fingerprint value only appears once. It also speeds up the database search since the effective number of entries in the list is greatly reduced to a list of unique values.
  • Alternatively, the master index list could also be constructed by inserting each triplet into a B-tree with non-unique fingerprints hanging off a linked list. Other possibilities exist for constructing the master index list. The master index list is preferably held in system memory, such as DRAM, for fast access. [0110]
  • Recognition System [0111]
  • Once the master index list has been built it is possible to perform sound recognition over the database. [0112]
  • Sound Source [0113]
  • Exogenous sound is provided from any number of analog or digital sources, such as a stereo system, television, Compact Disc player, radio broadcast, telephone, mobile phone, internet stream, or computer file. The sounds may be realtime or offline. They may be from any kind of environment, such as a disco, pub, submarine, answering machine, sound file, stereo, radio broadcast, or tape recorder. Noise may be present in the sound signal, for example in the form of background noise, talking voices, etc. [0114]
  • Input to the Recognition System [0115]
  • The sound stream is then captured into the recognition system either in realtime or presented offline, as with a sound file. Real-time sounds may be sampled digitally and sent to the system by a sampling device such as a microphone, or be stored in a storage device such as an answering machine, computer file, tape recorder, telephone, mobile phone, radio, etc. The sound signal may be subjected to further degradation due to limitations of the channel or sound capture device. Sounds may also be sent to the recognition system via an internet stream, FTP, or as a file attachment to email. [0116]
  • Preprocessing [0117]
  • Once the sound signal has been converted into digital form it is processed for recognition. As with the construction of the master index list, landmarks and fingerprints are calculated. In fact, it is advisable to use the very same code that was used for processing the sound recording library to do the landmarking and fingerprinting of the exogenous sound input. The resulting index set for exogenous sound sample is also a list of pairs (fingerprint, landmark) of analyzed values. [0118]
  • Searching [0119]
  • Searching is carried out as follows: each fingerprint/landmark pair (fingerprints, landmarks) in the resulting input sound's index set is processed by searching for fingerprint[0120] k in the master index list. Fast searching algorithms on an ordered list are well-known in the art and extensively discussed in Knuth, Volume 3 (ibid), incorporated by reference. If fingerprints is found then the corresponding list of matching (landmark*j, sound13IIDj) pairs having the same fingerprint is copied and augmented with landmark to form a set of triplets of the form(landmarkk, landmark*j,sound_IDj). This process is repeated for all k ranging over the input sound's index set, with the all the resulting triplets being collected into a large candidate list.
  • After the candidate list is compiled it is further processed by segmenting according to sound[0121] 13 ID. A convenient way of doing this is to sort the candidate list according to sound13 ID, or by insertion into a B-tree. The result of this is a list of candidate sound_IDs, each of which having a scatter list of pairs of landmark timepoints, (landmarkk,landmark*j) with the sound13 ID stripped off.
  • Scanning [0122]
  • The scatter list for each sound[0123] 13 ID is analyzed to determine whether it is a likely match.
  • Thresholding [0124]
  • One way to eliminate a large number of candidates is to toss out those having a small scatter list. Clearly, those having only 1 entry in their scatter lists cannot be matched. [0125]
  • Alignment [0126]
  • A key insight into the matching process is that the time evolution in matching sounds must follow a linear correspondence, assuming that the timebases on both sides are steady. This is almost always true unless the sound on one side has been nonlinearly warped intentionally or subject to defective playback equipment such as a tape deck with a warbling speed problem. Thus, the matching fingerprints yielding correct landmark pairs (landmark[0127] n, landmark*n) in the scatter list of a given sound13 ID must have a linear correspondence of the form
  • landmark*n=m*landmarkn+offset
  • where m is the slope, and should be near [0128] 1, landmarkn is the corresponding timepoint within the exogenous sound signal, landmark*n is the corresponding timepoint within the library sound recording indexed by sound13 ID, and offset is the time offset into the library sound recording corresponding to the beginning of the exogenous sound signal.
  • This relationship ties together the true landmark/fingerprint correspondences between the exogenous sound signal and the correct library sound recording with high probability, and excludes outlier landmark pairs. Thus, the problem of determining whether there is a match is reduced to finding a diagonal line with slope near I within the scatterplot of the points in the scatter list. [0129]
  • There are many ways of finding the diagonal line. A preferred method starts by subtracting m*landmark[0130] n from both sides of the above equation.
  • (landmark*n−m*landmarkn)=offset
  • Assuming that m is approximately 1, we arrive at[0131]
  • (landmarkn−landmarkn)=offset
  • The diagonal-finding problem is then reduced to finding multiple landmark pairs that cluster near the same offset value. This is accomplished easily by calculating a histogram of the resulting offset values and searching for the offset bin with the highest number of points. Since the offset must be positive if the exogenous sound signal is fully contained within the correct library sound recording, landmark pairs that result in a negative offset are excluded. [0132]
  • The winning offset bin of the histogram is noted for each qualifying sound[0133] 13 ID, and the corresponding score is the number of points in the winning bin. The sound recording in the candidate list with the highest score is chosen as the winner. The winning sound13 ID is provided to an output means to signal the success of the identification.
  • To prevent false identification, a minimum threshold score may be used to gate the success of the identification process. If no library sound recording meets the minimum threshold then there is no identification. [0134]
  • Pipelined Recognition [0135]
  • In a real-time system the sound is provided to the recognition system incrementally over time. In this case it is possible to process the data in chunks and to update the index set incrementally. Each update period the newly augmented index set is used as above to retrieve candidate library sound recordings using the searching and scanning steps above. The advantage of this approach is that if sufficient data has been collected to identify the sound recording unambiguously then the data acquisition may be terminated and the result may be announced. [0136]
  • Reporting the Result [0137]
  • Once the correct sound has been identified, the result is reported. Among the result-reporting means, this may be done using a computer printout, email, SMS text messaging to a mobile phone, computer-generated voice annotation over a telephone, posting of the result to an internet account which the user can access later. [0138]

Claims (75)

What is claimed is:
1. A method for interacting with a user comprising:
receiving from the user a captured sample of an experiential environment; and
triggering a predetermined event in response to the captured sample.
2. The method according to claim 1, further comprising determining from the signal a characteristic of the captured sample and triggering a predetermined event in response to the determined characteristic.
3. The method according to claim 1, wherein the predetermined event includes a transaction.
4. The method according to claim 3, wherein the transaction includes sales and purchase of merchandise.
5. The method according to claim 3, wherein the transaction includes an offer for sale of merchandise.
6. The method according to claim 5, wherein the offer for sale of merchandise includes an offer to sell recordings of music.
7. The method according to claim 6, wherein the recording is related to a characteristic of the captured sample.
8. The method according to claim 3, wherein the transaction includes furnishing and receiving information.
9. The method according to claim 3, wherein the transaction includes delivery of advertising or promotional offers.
10. The method according to claim 9, wherein the promotional offers include trial offers.
11. The method according to claim 9, wherein the promotional offers include offers to sell merchandise or services at discounted prices.
12. The method according to claim 3, wherein the transaction includes an exchange of information between a sales source and the user attendant to a sale of merchandise or services to the user.
13. The method according to claim 5, wherein the offer is selected in response to a profile of the user.
14. The method according to claim 5, wherein the offer is selected in response to history of transactions completed with the user.
15. The method according to claim 1, wherein the predetermined event includes a communication event.
16. The method according to claim 1, wherein the predetermined event includes an entertainment event.
17. The method according to claim 1, wherein the predetermined event includes a surveillance or monitoring event.
18. The method according to claim 1, wherein the predetermined event includes a human ability enhancement event.
19. The method according to claim 1, wherein the predetermined event includes an execution of tasks or instructions event.
20. The method according to claim 1, wherein the predetermined event includes a delivery of information event.
21. The method according to claim 1, wherein the predetermined event includes a delivery of services event.
22. The method according to claim 1, wherein the predetermined event includes a control event.
23. The method according to claim 1, wherein the predetermined event includes a combination of communication events and control events.
24. The method according to claim 5, wherein the predetermined event includes an entertainment event.
25. A method for interacting with a user, comprising:
providing a user with an ability to capture a sample of an experiential environment; and
delivering said sample to an interactive service to trigger one or more predetermined events.
26. The method according to claim 25, wherein the predetermined event includes a communication event.
27. The method according to claim 25, wherein the predetermined event includes a surveillance event.
28. A method for interacting with a user, comprising:
capturing a sample of an experiential environment; and
triggering one or more predetermined events at an interactive service upon receipt of said sample.
29. The method according to claim 28, wherein the one or more predetermined events includes at least a communication event.
30. The method according to claim 28, wherein the one or more predetermined events includes at least a surveillance event.
31. A method for receiving instructions from a user in an interactive service, comprising:
receiving a sample of an experiential environment from the user; and
triggering one or more predetermined events in the interactive service upon receipt of said sample.
32. The method according to claim 31, wherein the one or more predetermined events includes at least a communication event.
33. The method according to claim 31, wherein the one or more predetermined events includes at least a surveillance event.
34. A method for interacting with an interactive service comprising:
sampling an experiential environment; and
forwarding the sample to an interactive service to trigger one or more predetermined events.
35. The method according to claim 34, wherein the one or more predetermined events includes at least a communication event.
36. The method according to claim 34, wherein the one or more predetermined events includes at least a surveillance event.
37. The method according to claim 34, further comprising deriving information from the sample.
38. The method according to claim 37, further comprising combining information derived from the sample with information known about the user to trigger the one or more predetermined events.
39. The method according to claim 38, wherein the one or more predetermined events includes one or more of the following: delivery of information and/or services to the user, execution of tasks and/or instructions by the service on the user's behalf, a communication event, a surveillance event and one or more control-oriented events that are responsive to the user's wishes.
40. The method according to claim 39, wherein the triggered events also include one or more of the following: transaction-oriented events, entertainment events, and events associated with enhancements to human ability or function.
41. A method for triggering an information delivery event to a user, comprising:
accessing an interactive service;
capturing a sample of an audio signal to which a user is listening;
identifying a song included in the audio signal; and
delivering lyric information to the user upon identifying the song.
42. The method according to claim 41, wherein the delivery of lyric information occurs in real-time so that the user can read the lyrics and sing along with the song.
43. The method according to claim 42, wherein the lyric data is transmitted in near synchronism to the song.
44. The method according to claim 43, wherein the user further interacts with the service via a graphical user interface.
45. The method according to claim 43, wherein the user accesses the interactive service by dialing a predetermined telephone number and interacts with the interactive service via touch tones on the telephone handset.
46. A system for interacting with a user comprising:
a capture device to capture a sample of an experiential environment;
a user input/identification device to identify the user;
a database to store data representative of potential samples;
a recognition engine to process the sample of the experiential environment in combination with the stored data in the database to identify the sample;
a user profile storage device to store information regarding one or more users of the system; and
a transaction engine initiating a transaction with the user based on the identified sample and identification of the user and stored information regarding the identified user.
47. The system according to claim 46, wherein the transaction includes at least a communication event.
48. The system according to claim 46, wherein the transaction includes at least a surveillance event.
49. A method for interacting with a user to trigger a transaction with the user comprising:
capturing an image of an advertisement; and
forwarding the captured image to a predetermined web site.
50. The method according to claim 49, further comprising:
identifying the captured image; and
engaging in a transaction with the user based on the identified sample.
51. The method according to claim 50, wherein the transaction includes at least a communication event.
52. The method according to claim 50, wherein the transaction includes at least a surveillance event.
53. The method according to claim 49, wherein the advertisement is disposed in one or more of the following: a newspaper, a magazine, a billboard, a sign, a poster, a printed publication.
54. A method for enhancing human abilities of a user comprising:
capturing a sample of an experiential environment of the user;
forwarding the captured sample to an interactive service; and
triggering an enhancement event based on the captured sample.
55. A method for interacting with a user comprising:
receiving a captured sample of an experiential event from a user;
triggering a transactional event with the user based on a characteristic of the captured sample.
56. The method according to claim 55, wherein the transactional event includes storing the captured sample.
57. The method according to claim 56, wherein the transactional event includes sorting the captured sample based on a dominant characteristic inherent in the captured sample.
58. A method for remote sensing comprising:
accessing a remote sensing service;
sampling the audio environment in the location; and
triggering an event based on a characteristic in the sampled audio environment.
59. The method according to claim 58, wherein the accessing includes accessing a remote sensing service by dialing a predetermined telephone number.
60. The method according to claim 58, wherein the sampling includes leaving the telephone handset off hook in a location in which the user wishes to remotely sense activity.
61. A method for remotely sensing an environment of a user;
capturing a sample of an experiential environment; and
using the sample to trigger an event to provide assistance to the user.
62. The method according to claim 61, wherein the sample includes a video image and the assistance includes identifying a person in the image.
63. A method for paying bills comprising:
scanning a bill;
forwarding the scanned bill to a bill payment service; and
paying the bill by the bill payment service upon receiving the scanned bill in combination with user profile data.
64. The method according to claim 63, wherein paying the bill includes debiting the user's bank account.
65. A method for interacting with a user comprising:
sampling an audio component of a broadcast signal;
forwarding the sampled audio component to a service provider;
identifying one or more corporations in the sampled audio component; and
engaging in a transaction with the user involving the one or more corporations.
66. The method according to claim 65, wherein the transaction includes forwarding information to the user regarding the one or more corporations.
67. The method according to claim 65, wherein the transaction includes trading securities of the one or more corporations based on user profile data and information characteristics derived from the audio sample.
68. A method for interacting with a user comprising:
receiving a sample of an experiential event; and
initiating one or more control events in response to said sampled experiential event.
69. A method for finding suitable employers comprising:
capturing images from a plurality of web sites of a plurality of potential employers;
uploading the images to a processor;
deriving one or more characteristics from the captured images; and
sorting a database of the plurality of potential employers in accordance with the derived characteristics.
70. A portable apparatus for interacting with a user comprising:
a capture device to capture a sample of an experiential environment;
a database to store data representative of potential samples; and
a processor to process the sample of the experiential environment in combination with the stored data in the database to identify the sample and to initiate a transaction with the user based on the identified sample and identification of the user and stored information regarding the identified user.
71. The portable apparatus according to claim 70, further comprising a user input/identification device to identify the user.
72. The portable apparatus according to claim 70, wherein the portable apparatus comprises a portable computing device.
73. The portable apparatus according to claim 70, wherein the database is updatable by coupling the apparatus to an interactive service.
74. A portable apparatus for interacting with a user comprising:
a capture device to capture a sample of an experiential environment;
a communication device to communicate with an interactive service; and
a processor to process the sample of the experiential environment and forward a processed version of the sample to the interactive service via the communication device to identify the sample and to initiate a transaction with the interactive service for the user based on the identified sample and identification of the user and stored information regarding the identified user.
75. A method for interacting with a user comprising:
sampling an experiential event; and
initiating one or more control events in response to said sampled experiential event.
US09/734,949 2000-12-12 2000-12-12 Method and system for interacting with a user in an experiential environment Abandoned US20020072982A1 (en)

Priority Applications (7)

Application Number Priority Date Filing Date Title
US09/734,949 US20020072982A1 (en) 2000-12-12 2000-12-12 Method and system for interacting with a user in an experiential environment
PCT/US2001/046986 WO2002061652A2 (en) 2000-12-12 2001-12-06 Method and system for interacting with a user in an experiential environment
US11/188,385 US20050267817A1 (en) 2000-12-12 2005-07-25 Method and system for interacting with a user in an experiential environment
US12/204,526 US8015123B2 (en) 2000-12-12 2008-09-04 Method and system for interacting with a user in an experiential environment
US13/100,825 US8996380B2 (en) 2000-12-12 2011-05-04 Methods and systems for synchronizing media
US13/193,205 US8688600B2 (en) 2000-12-12 2011-07-28 Method and system for interacting with a user in an experiential environment
US14/230,061 US9721287B2 (en) 2000-12-12 2014-03-31 Method and system for interacting with a user in an experimental environment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US09/734,949 US20020072982A1 (en) 2000-12-12 2000-12-12 Method and system for interacting with a user in an experiential environment

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US11/188,385 Continuation US20050267817A1 (en) 2000-12-12 2005-07-25 Method and system for interacting with a user in an experiential environment

Publications (1)

Publication Number Publication Date
US20020072982A1 true US20020072982A1 (en) 2002-06-13

Family

ID=24953713

Family Applications (5)

Application Number Title Priority Date Filing Date
US09/734,949 Abandoned US20020072982A1 (en) 2000-12-12 2000-12-12 Method and system for interacting with a user in an experiential environment
US11/188,385 Abandoned US20050267817A1 (en) 2000-12-12 2005-07-25 Method and system for interacting with a user in an experiential environment
US12/204,526 Expired - Fee Related US8015123B2 (en) 2000-12-12 2008-09-04 Method and system for interacting with a user in an experiential environment
US13/193,205 Expired - Fee Related US8688600B2 (en) 2000-12-12 2011-07-28 Method and system for interacting with a user in an experiential environment
US14/230,061 Expired - Lifetime US9721287B2 (en) 2000-12-12 2014-03-31 Method and system for interacting with a user in an experimental environment

Family Applications After (4)

Application Number Title Priority Date Filing Date
US11/188,385 Abandoned US20050267817A1 (en) 2000-12-12 2005-07-25 Method and system for interacting with a user in an experiential environment
US12/204,526 Expired - Fee Related US8015123B2 (en) 2000-12-12 2008-09-04 Method and system for interacting with a user in an experiential environment
US13/193,205 Expired - Fee Related US8688600B2 (en) 2000-12-12 2011-07-28 Method and system for interacting with a user in an experiential environment
US14/230,061 Expired - Lifetime US9721287B2 (en) 2000-12-12 2014-03-31 Method and system for interacting with a user in an experimental environment

Country Status (2)

Country Link
US (5) US20020072982A1 (en)
WO (1) WO2002061652A2 (en)

Cited By (138)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030086341A1 (en) * 2001-07-20 2003-05-08 Gracenote, Inc. Automatic identification of sound recordings
US20030177014A1 (en) * 2002-02-13 2003-09-18 Crawford Robert E. System and method for displaying multiple language text during performance or play of a musical work
US20030229900A1 (en) * 2002-05-10 2003-12-11 Richard Reisman Method and apparatus for browsing using multiple coordinated device sets
WO2004070523A2 (en) * 2003-01-27 2004-08-19 Ciphergen Biosystems Inc. Data management system and method for processing signals from sample spots
US20040243567A1 (en) * 2003-03-03 2004-12-02 Levy Kenneth L. Integrating and enhancing searching of media content and biometric databases
US20050151669A1 (en) * 2003-11-28 2005-07-14 Craig Swallow Lone worker monitor
US20050177372A1 (en) * 2002-04-25 2005-08-11 Wang Avery L. Robust and invariant audio pattern matching
US20050193016A1 (en) * 2004-02-17 2005-09-01 Nicholas Seet Generation of a media content database by correlating repeating media content in media streams
US20050215239A1 (en) * 2004-03-26 2005-09-29 Nokia Corporation Feature extraction in a networked portable device
US20050222913A1 (en) * 2004-04-01 2005-10-06 Ian Eisenberg PR/SMS business method, system and method of conducting business
US20050267750A1 (en) * 2004-05-27 2005-12-01 Anonymous Media, Llc Media usage monitoring and measurement system and method
US20060004731A1 (en) * 2000-05-24 2006-01-05 Seibel John C Text mining system for web-based business intelligence
EP1617365A1 (en) * 2004-07-16 2006-01-18 Vodafone Holding GmbH Coordination and processing of games serving education, entertainment or competition purposes
US7082427B1 (en) 2000-05-24 2006-07-25 Reachforce, Inc. Text indexing system to index, query the archive database document by keyword data representing the content of the documents and by contact data associated with the participant who generated the document
US7096220B1 (en) * 2000-05-24 2006-08-22 Reachforce, Inc. Web-based customer prospects harvester system
US20060246412A1 (en) * 2005-05-01 2006-11-02 George Pieczenik Professor Pieczenik's instruction method and system
US20070016790A1 (en) * 1998-11-19 2007-01-18 Brundage Trent J Identification document and related methods
US20070033409A1 (en) * 2000-12-21 2007-02-08 Brunk Hugh L Methods, Apparatus and Programs for Generating and Utilizing Content Signatures
US20070196802A1 (en) * 2006-02-21 2007-08-23 Nokia Corporation Visually Enhanced Personal Music Broadcast
US20070250716A1 (en) * 2000-05-02 2007-10-25 Brunk Hugh L Fingerprinting of Media Signals
US20070286454A1 (en) * 2001-10-16 2007-12-13 Brundage Trent J Distributed Content Identification
US20080032622A1 (en) * 2004-04-07 2008-02-07 Nokia Corporation Mobile station and interface adapted for feature extraction from an input media sample
US20080208851A1 (en) * 2007-02-27 2008-08-28 Landmark Digital Services Llc System and method for monitoring and recognizing broadcast data
US7421376B1 (en) 2001-04-24 2008-09-02 Auditude, Inc. Comparison of data signals using characteristic electronic thumbprints
US20080256032A1 (en) * 2004-01-20 2008-10-16 Koninklike Philips Electronics, N.V. Hierarchical Playlist Generator
US7516074B2 (en) 2005-09-01 2009-04-07 Auditude, Inc. Extraction and matching of characteristic fingerprints from audio signals
US20090144391A1 (en) * 2007-11-30 2009-06-04 Searete Llc, A Limited Liability Corporation Of The State Of Delaware Audio sharing
US20090182622A1 (en) * 2008-01-15 2009-07-16 Agarwal Amit D Enhancing and storing data for recall and use
US20090282334A1 (en) * 2006-05-04 2009-11-12 Mobilians Co., Ltd. System and method for providing information using outside sound recognition of mobile phone, and mobile terminal for the same
US20100070501A1 (en) * 2008-01-15 2010-03-18 Walsh Paul J Enhancing and storing data for recall and use using user feedback
US20100113091A1 (en) * 2008-10-31 2010-05-06 Sharma Ravi K Histogram methods and systems for object recognition
US20100146445A1 (en) * 2008-12-08 2010-06-10 Apple Inc. Ambient Noise Based Augmentation of Media Playback
US7824029B2 (en) 2002-05-10 2010-11-02 L-1 Secure Credentialing, Inc. Identification card printer-assembler for over the counter card issuing
US20100305729A1 (en) * 2009-05-27 2010-12-02 Glitsch Hans M Audio-based synchronization to media
US20110016172A1 (en) * 2009-05-27 2011-01-20 Ajay Shah Synchronized delivery of interactive content
US20110161076A1 (en) * 2009-12-31 2011-06-30 Davis Bruce L Intuitive Computing Methods and Systems
US20110159921A1 (en) * 2009-12-31 2011-06-30 Davis Bruce L Methods and arrangements employing sensor-equipped smart phones
US20110202338A1 (en) * 2010-02-18 2011-08-18 Philip Inghelbrecht System and method for recognition of alphanumeric patterns including license plate numbers
US20110213760A1 (en) * 2006-12-01 2011-09-01 Jeffrey Scott Bardsley Methods, Systems, And Computer Program Products For Determining Availability Of Presentable Content Via A Subscription Service
US20110244784A1 (en) * 2004-02-19 2011-10-06 Landmark Digital Services Llc Method and apparatus for identification of broadcast source
USRE42870E1 (en) 2000-10-04 2011-10-25 Dafineais Protocol Data B.V., Llc Text mining system for web-based business intelligence applied to web site server logs
US20110276333A1 (en) * 2010-05-04 2011-11-10 Avery Li-Chun Wang Methods and Systems for Synchronizing Media
US20110307786A1 (en) * 2010-06-11 2011-12-15 Brian Shuster Method and apparatus for bookmarking and retrieval of video content
US8094949B1 (en) 1994-10-21 2012-01-10 Digimarc Corporation Music methods and systems
US20120010736A1 (en) * 2010-07-09 2012-01-12 Nexidia Inc. Spotting multimedia
US8170273B2 (en) 2001-04-25 2012-05-01 Digimarc Corporation Encoding and decoding auxiliary signals
US20120162513A1 (en) * 2010-12-28 2012-06-28 Tse-Ming Chang Dual-connection multimedia apparatus
US8214067B1 (en) * 2004-09-24 2012-07-03 Sztybel Robert S Interactive audio content delivery system and method
WO2012170451A1 (en) * 2011-06-08 2012-12-13 Shazam Entertainment Ltd. Methods and systems for performing comparisons of received data and providing a follow-on service based on the comparisons
US8433431B1 (en) * 2008-12-02 2013-04-30 Soundhound, Inc. Displaying text to end users in coordination with audio playback
US8554681B1 (en) * 2003-11-03 2013-10-08 James W. Wieder Providing “identified” compositions and digital-works
US8554265B1 (en) * 2007-01-17 2013-10-08 At&T Mobility Ii Llc Distribution of user-generated multimedia broadcasts to mobile wireless telecommunication network users
US8639178B2 (en) 2011-08-30 2014-01-28 Clear Channel Management Sevices, Inc. Broadcast source identification based on matching broadcast signal fingerprints
US20140122302A1 (en) * 2012-11-01 2014-05-01 At&T Mobility Ii Llc Customized Data Delivery
US20140137139A1 (en) * 2012-11-14 2014-05-15 Bank Of America Automatic Deal Or Promotion Offering Based on Audio Cues
US20140161263A1 (en) * 2012-12-10 2014-06-12 Microsoft Corporation Facilitating recognition of real-time content
US8781152B2 (en) 2010-08-05 2014-07-15 Brian Momeyer Identifying visual media content captured by camera-enabled mobile device
US20140214532A1 (en) * 2000-12-12 2014-07-31 Shazam Investments Limited Method and System for Interacting with a User in an Experimental Environment
US8825518B2 (en) 2000-12-21 2014-09-02 Digimarc Corporation Media methods and systems
US8861937B2 (en) 2011-08-31 2014-10-14 The Nielsen Company (Us), Llc Methods and apparatus to access media
US20140336797A1 (en) * 2013-05-12 2014-11-13 Harry E. Emerson, III Audio content monitoring and identification of broadcast radio stations
US20140336798A1 (en) * 2012-05-13 2014-11-13 Harry E. Emerson, III Discovery of music artist and title for syndicated content played by radio stations
US20150051967A1 (en) * 2004-05-27 2015-02-19 Anonymous Media Research, Llc Media usage monitoring and measurment system and method
US9047371B2 (en) 2010-07-29 2015-06-02 Soundhound, Inc. System and method for matching a query against a broadcast stream
US9093120B2 (en) 2011-02-10 2015-07-28 Yahoo! Inc. Audio fingerprint extraction by scaling in time and resampling
US20150222950A1 (en) * 2012-08-21 2015-08-06 Omnifone Ltd. Method of identifying media content
US9111537B1 (en) 2010-06-29 2015-08-18 Google Inc. Real-time audio recognition protocol
US9159338B2 (en) 2010-05-04 2015-10-13 Shazam Entertainment Ltd. Systems and methods of rendering a textual animation
US9167520B2 (en) 2012-03-20 2015-10-20 Qualcomm Incorporated Controlling applications in a mobile device based on environmental context
US9208225B1 (en) * 2012-02-24 2015-12-08 Google Inc. Incentive-based check-in
US9256673B2 (en) 2011-06-10 2016-02-09 Shazam Entertainment Ltd. Methods and systems for identifying content in a data stream
US9275141B2 (en) 2010-05-04 2016-03-01 Shazam Entertainment Ltd. Methods and systems for processing a sample of a media stream
US9280599B1 (en) 2012-02-24 2016-03-08 Google Inc. Interface for real-time audio recognition
US9292488B2 (en) 2014-02-01 2016-03-22 Soundhound, Inc. Method for embedding voice mail in a spoken utterance using a natural language processing computer system
US9301070B2 (en) 2013-03-11 2016-03-29 Arris Enterprises, Inc. Signature matching of corrupted audio signal
US9307337B2 (en) 2013-03-11 2016-04-05 Arris Enterprises, Inc. Systems and methods for interactive broadcast content
US9374183B2 (en) 2011-08-30 2016-06-21 Iheartmedia Management Services, Inc. Broadcast source identification based on matching via bit count
US9390167B2 (en) 2010-07-29 2016-07-12 Soundhound, Inc. System and methods for continuous audio matching
US9390170B2 (en) 2013-03-15 2016-07-12 Shazam Investments Ltd. Methods and systems for arranging and searching a database of media content recordings
US9451048B2 (en) 2013-03-12 2016-09-20 Shazam Investments Ltd. Methods and systems for identifying information of a broadcast station and information of broadcasted content
US9460201B2 (en) 2013-05-06 2016-10-04 Iheartmedia Management Services, Inc. Unordered matching of audio fingerprints
US9461759B2 (en) 2011-08-30 2016-10-04 Iheartmedia Management Services, Inc. Identification of changed broadcast media items
US20160336025A1 (en) * 2014-05-16 2016-11-17 Alphonso Inc. Efficient apparatus and method for audio signature generation using recognition history
US20160343371A1 (en) * 2012-09-10 2016-11-24 Google Inc. Answering questions using environmental context
US9507849B2 (en) 2013-11-28 2016-11-29 Soundhound, Inc. Method for combining a query and a communication command in a natural language computer system
US9560425B2 (en) 2008-11-26 2017-01-31 Free Stream Media Corp. Remotely control devices over a network without authentication or registration
US9564123B1 (en) 2014-05-12 2017-02-07 Soundhound, Inc. Method and system for building an integrated user profile
US9703947B2 (en) 2008-11-26 2017-07-11 Free Stream Media Corp. Relevancy improvement through targeting of information based on data gathered from a networked device associated with a security sandbox of a client device
US9711153B2 (en) 2002-09-27 2017-07-18 The Nielsen Company (Us), Llc Activating functions in processing devices using encoded audio and detecting audio signatures
US9716736B2 (en) 2008-11-26 2017-07-25 Free Stream Media Corp. System and method of discovery and launch associated with a networked media device
US9773058B2 (en) 2013-03-15 2017-09-26 Shazam Investments Ltd. Methods and systems for arranging and searching a database of media content recordings
US9858595B2 (en) 2002-05-23 2018-01-02 Gula Consulting Limited Liability Company Location-based transmissions using a mobile communication device
US9864958B2 (en) 2000-06-29 2018-01-09 Gula Consulting Limited Liability Company System, method, and computer program product for video based services and commerce
US9910341B2 (en) 2005-01-31 2018-03-06 The Invention Science Fund I, Llc Shared image device designation
US9942511B2 (en) 2005-10-31 2018-04-10 Invention Science Fund I, Llc Preservation/degradation of video/audio aspects of a data stream
US9961388B2 (en) 2008-11-26 2018-05-01 David Harrison Exposure of public internet protocol addresses in an advertising exchange server to improve relevancy of advertisements
US9986279B2 (en) 2008-11-26 2018-05-29 Free Stream Media Corp. Discovery, access control, and communication with networked services
US10003762B2 (en) 2005-04-26 2018-06-19 Invention Science Fund I, Llc Shared image devices
US10097756B2 (en) 2005-06-02 2018-10-09 Invention Science Fund I, Llc Enhanced video/still image correlation
US20180307808A1 (en) * 2011-11-04 2018-10-25 Christopher A. Estes Digital media reproduction and licensing
US10121165B1 (en) 2011-05-10 2018-11-06 Soundhound, Inc. System and method for targeting content based on identified audio and multimedia
US10264297B1 (en) * 2017-09-13 2019-04-16 Perfect Sense, Inc. Time-based content synchronization
US20190141414A1 (en) * 2017-09-12 2019-05-09 Irdeto B.V. Device and Method for GPU-based Watermarking
US10334324B2 (en) 2008-11-26 2019-06-25 Free Stream Media Corp. Relevant advertisement generation based on a user operating a client device communicatively coupled with a networked media device
US10339936B2 (en) 2012-11-27 2019-07-02 Roland Storti Method, device and system of encoding a digital interactive response action in an analog broadcasting message
US10366419B2 (en) 2012-11-27 2019-07-30 Roland Storti Enhanced digital media platform with user control of application data thereon
US10419541B2 (en) 2008-11-26 2019-09-17 Free Stream Media Corp. Remotely control devices over a network without authentication or registration
US10489449B2 (en) 2002-05-23 2019-11-26 Gula Consulting Limited Liability Company Computer accepting voice input and/or generating audible output
US10521830B2 (en) * 2013-03-14 2019-12-31 Facebook, Inc. Method for displaying a product-related image to a user while shopping
US10546352B2 (en) 2013-03-14 2020-01-28 Facebook, Inc. Method for selectively advertising items in an image
US10567823B2 (en) 2008-11-26 2020-02-18 Free Stream Media Corp. Relevant advertisement generation based on a user operating a client device communicatively coupled with a networked media device
US20200092592A1 (en) * 2018-09-18 2020-03-19 Free Stream Media Corporation d/b/a Samba TV Content consensus management
US10631068B2 (en) 2008-11-26 2020-04-21 Free Stream Media Corp. Content exposure attribution based on renderings of related content across multiple devices
US10868620B2 (en) * 2018-12-26 2020-12-15 The Nielsen Company (Us), Llc Methods and apparatus for optimizing station reference fingerprint loading using reference watermarks
US10880340B2 (en) 2008-11-26 2020-12-29 Free Stream Media Corp. Relevancy improvement through targeting of information based on data gathered from a networked device associated with a security sandbox of a client device
US10922720B2 (en) 2017-01-11 2021-02-16 Adobe Inc. Managing content delivery via audio cues
US10922957B2 (en) 2008-08-19 2021-02-16 Digimarc Corporation Methods and systems for content processing
US10957310B1 (en) 2012-07-23 2021-03-23 Soundhound, Inc. Integrated programming framework for speech and text understanding with meaning parsing
US10972306B2 (en) 2016-11-23 2021-04-06 Carrier Corporation Building management system having event reporting
US10977693B2 (en) 2008-11-26 2021-04-13 Free Stream Media Corp. Association of content identifier of audio-visual data with additional data through capture infrastructure
US11036781B1 (en) 2020-01-30 2021-06-15 Snap Inc. Video generation system to render frames on demand using a fleet of servers
US11049094B2 (en) 2014-02-11 2021-06-29 Digimarc Corporation Methods and arrangements for device to device communication
US11051057B2 (en) * 2019-06-24 2021-06-29 The Nielsen Company (Us), Llc Use of steganographically-encoded time information as basis to establish a time offset, to facilitate taking content-related action
US11164571B2 (en) * 2017-11-16 2021-11-02 Baidu Online Network Technology (Beijing) Co., Ltd. Content recognizing method and apparatus, device, and computer storage medium
US11165999B1 (en) * 2003-11-03 2021-11-02 Synergyze Technologies Llc Identifying and providing compositions and digital-works
US20220021948A1 (en) * 2020-07-17 2022-01-20 Playrcart Limited Media player
US11234049B2 (en) * 2019-06-24 2022-01-25 The Nielsen Company (Us), Llc Use of steganographically-encoded time information as basis to control implementation of dynamic content modification
US20220078492A1 (en) * 2019-12-13 2022-03-10 Tencent Technology (Shenzhen) Company Limited Interactive service processing method and system, device, and storage medium
US11284139B1 (en) * 2020-09-10 2022-03-22 Hulu, LLC Stateless re-discovery of identity using watermarking of a video stream
US11284144B2 (en) * 2020-01-30 2022-03-22 Snap Inc. Video generation system to render frames on demand using a fleet of GPUs
US11295730B1 (en) 2014-02-27 2022-04-05 Soundhound, Inc. Using phonetic variants in a local context to improve natural language understanding
US11356720B2 (en) 2020-01-30 2022-06-07 Snap Inc. Video generation system to render frames on demand
US11487815B2 (en) * 2019-06-06 2022-11-01 Sony Corporation Audio track determination based on identification of performer-of-interest at live event
US11496318B1 (en) 2021-07-19 2022-11-08 Intrado Corporation Database layer caching for video communications
US11589100B1 (en) * 2021-03-31 2023-02-21 Amazon Technologies, Inc. On-demand issuance private keys for encrypted video transmission
US11586938B2 (en) 2016-11-23 2023-02-21 Carrier Corporation Building management system having knowledge base
US11651539B2 (en) 2020-01-30 2023-05-16 Snap Inc. System for generating media content items on demand
WO2024016901A1 (en) * 2022-07-19 2024-01-25 北京字跳网络技术有限公司 Lyric-based information prompting method and apparatus, device, medium and product

Families Citing this family (77)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7562392B1 (en) 1999-05-19 2009-07-14 Digimarc Corporation Methods of interacting with audio and ambient music
US6505160B1 (en) * 1995-07-27 2003-01-07 Digimarc Corporation Connected audio and other media objects
US7711564B2 (en) 1995-07-27 2010-05-04 Digimarc Corporation Connected audio and other media objects
US7055034B1 (en) 1998-09-25 2006-05-30 Digimarc Corporation Method and apparatus for robust embedded data
WO2003061285A2 (en) * 2001-12-24 2003-07-24 Scientific Generics Limited Captioning system
NZ534840A (en) 2002-01-24 2005-07-29 Newport Coast Invest Llc Dynamic selection and scheduling of radio frequency communications
US7239981B2 (en) 2002-07-26 2007-07-03 Arbitron Inc. Systems and methods for gathering audience measurement data
US8959016B2 (en) 2002-09-27 2015-02-17 The Nielsen Company (Us), Llc Activating functions in processing devices using start codes embedded in audio
CA2511919A1 (en) 2002-12-27 2004-07-22 Nielsen Media Research, Inc. Methods and apparatus for transcoding metadata
US7313359B2 (en) * 2004-03-09 2007-12-25 Google Inc. Dynamic data delivery apparatus and method for same
US7496897B1 (en) 2004-03-17 2009-02-24 Timesys Corporation Multiple code sets for multiple execution contexts
US7751804B2 (en) * 2004-07-23 2010-07-06 Wideorbit, Inc. Dynamic creation, selection, and scheduling of radio frequency communications
US8694049B2 (en) * 2004-08-06 2014-04-08 Digimarc Corporation Fast signal detection and distributed computing in portable computing devices
US7363001B2 (en) 2005-03-08 2008-04-22 Google Inc. Dynamic data delivery apparatus and method for same
US8099327B2 (en) * 2005-06-01 2012-01-17 Google Inc. Auctioneer
US20110029373A1 (en) * 2005-06-01 2011-02-03 Chad Steelberg Media play optimization using make good strategies
US20070130009A1 (en) * 2005-06-01 2007-06-07 Chad Steelberg System and method for media play pricing
US7676405B2 (en) * 2005-06-01 2010-03-09 Google Inc. System and method for media play forecasting
US8099326B2 (en) * 2005-06-01 2012-01-17 Google Inc. Traffic estimator
CA2984577C (en) * 2005-06-01 2019-05-21 Google Inc. Media play optimization
US8874477B2 (en) 2005-10-04 2014-10-28 Steven Mark Hoffberg Multifactorial optimization system and method
US20070208629A1 (en) * 2006-03-02 2007-09-06 Jung Edward K Y Shopping using exemplars
US9940626B2 (en) 2006-06-23 2018-04-10 At&T Intellectual Property I, L.P. Methods and apparatus to provide an electronic agent
US8468561B2 (en) 2006-08-09 2013-06-18 Google Inc. Preemptible station inventory
US20080243923A1 (en) * 2007-03-26 2008-10-02 Gadi Mazor System and method for facilitating impulse content purchases
US7889724B2 (en) 2007-04-13 2011-02-15 Wideorbit, Inc. Multi-station media controller
US7925201B2 (en) 2007-04-13 2011-04-12 Wideorbit, Inc. Sharing media content among families of broadcast stations
US7826444B2 (en) 2007-04-13 2010-11-02 Wideorbit, Inc. Leader and follower broadcast stations
US8019683B1 (en) * 2007-11-02 2011-09-13 At&T Mobility Ii Llc Intelligent charging for services
US9323836B2 (en) 2008-02-11 2016-04-26 Popular Metrics, Inc. Internet based method and system for ranking artists using a popularity profile
US9122749B2 (en) 2009-02-04 2015-09-01 Popular Metrics, Inc. Internet based system and method for wagering on an artist
US9326099B2 (en) 2008-02-11 2016-04-26 Popular Metrics, Inc. System and method for determining audience characteristics of a music concert based on mobile phone tracking and mobile data transmissions
US8983677B2 (en) * 2008-10-01 2015-03-17 Honeywell International Inc. Acoustic fingerprinting of mechanical devices
US20100082180A1 (en) * 2008-10-01 2010-04-01 Honeywell International Inc. Errant vehicle countermeasures
US8359205B2 (en) 2008-10-24 2013-01-22 The Nielsen Company (Us), Llc Methods and apparatus to perform audio watermarking and watermark detection and extraction
US9667365B2 (en) 2008-10-24 2017-05-30 The Nielsen Company (Us), Llc Methods and apparatus to perform audio watermarking and watermark detection and extraction
US8200661B1 (en) 2008-12-18 2012-06-12 Google Inc. Dynamic recommendations based on user actions
US8161137B2 (en) * 2009-01-16 2012-04-17 At&T Intellectual Property I., L.P. Environment delivery network
CA2760677C (en) 2009-05-01 2018-07-24 David Henry Harkness Methods, apparatus and articles of manufacture to provide secondary content in association with primary broadcast media content
US20100281108A1 (en) * 2009-05-01 2010-11-04 Cohen Ronald H Provision of Content Correlated with Events
US8521779B2 (en) 2009-10-09 2013-08-27 Adelphoi Limited Metadata record generation
CN102959624B (en) 2010-06-09 2015-04-22 阿德尔福伊有限公司 System and method for audio media recognition
KR20120034378A (en) * 2010-10-01 2012-04-12 엔에이치엔(주) Advertisement information providing system through recognition of sound and method thereof
US11062615B1 (en) 2011-03-01 2021-07-13 Intelligibility Training LLC Methods and systems for remote language learning in a pandemic-aware world
US10019995B1 (en) 2011-03-01 2018-07-10 Alice J. Stiebel Methods and systems for language learning based on a series of pitch patterns
US9380356B2 (en) 2011-04-12 2016-06-28 The Nielsen Company (Us), Llc Methods and apparatus to generate a tag for media content
US9210208B2 (en) 2011-06-21 2015-12-08 The Nielsen Company (Us), Llc Monitoring streaming media content
US9209978B2 (en) 2012-05-15 2015-12-08 The Nielsen Company (Us), Llc Methods and apparatus to measure exposure to streaming media
US9262522B2 (en) * 2011-06-30 2016-02-16 Rednote LLC Method and system for communicating between a sender and a recipient via a personalized message including an audio clip extracted from a pre-existing recording
US10333876B2 (en) * 2011-06-30 2019-06-25 Audiobyte Llc Method and system for communicating between a sender and a recipient via a personalized message including an audio clip extracted from a pre-existing recording
US10200323B2 (en) * 2011-06-30 2019-02-05 Audiobyte Llc Method and system for communicating between a sender and a recipient via a personalized message including an audio clip extracted from a pre-existing recording
US10560410B2 (en) * 2011-06-30 2020-02-11 Audiobyte Llc Method and system for communicating between a sender and a recipient via a personalized message including an audio clip extracted from a pre-existing recording
US11599915B1 (en) * 2011-10-25 2023-03-07 Auddia Inc. Apparatus, system, and method for audio based browser cookies
US10006462B2 (en) 2012-09-18 2018-06-26 Regal Beloit America, Inc. Systems and method for wirelessly communicating with electric motors
WO2014078391A1 (en) * 2012-11-13 2014-05-22 Nant Holdings Ip, Llc Systems and methods for synchronizing content playback across media devices
US9881011B2 (en) * 2012-11-29 2018-01-30 Ricoh Company, Ltd. System and method for generating user profiles for human resources
US20140188592A1 (en) * 2012-12-27 2014-07-03 Magix Ag Content recognition based evaluation system in a mobile environment
US9317872B2 (en) 2013-02-06 2016-04-19 Muzak Llc Encoding and decoding an audio watermark using key sequences comprising of more than two frequency components
US9313544B2 (en) 2013-02-14 2016-04-12 The Nielsen Company (Us), Llc Methods and apparatus to measure exposure to streaming media
KR101277523B1 (en) * 2013-03-08 2013-06-21 (주) 에이노드 Local interactive platform system, and local interactive service providing method using the same, and computer-readable recording medium for the same
US20150039321A1 (en) 2013-07-31 2015-02-05 Arbitron Inc. Apparatus, System and Method for Reading Codes From Digital Audio on a Processing Device
US9711152B2 (en) 2013-07-31 2017-07-18 The Nielsen Company (Us), Llc Systems apparatus and methods for encoding/decoding persistent universal media codes to encoded audio
US10014006B1 (en) 2013-09-10 2018-07-03 Ampersand, Inc. Method of determining whether a phone call is answered by a human or by an automated device
US9053711B1 (en) 2013-09-10 2015-06-09 Ampersand, Inc. Method of matching a digitized stream of audio signals to a known audio recording
WO2015118431A1 (en) 2014-02-05 2015-08-13 Edge Innovation, Lda. Method for capture and analysis of multimedia content
JP6467887B2 (en) * 2014-11-21 2019-02-13 ヤマハ株式会社 Information providing apparatus and information providing method
US9762965B2 (en) 2015-05-29 2017-09-12 The Nielsen Company (Us), Llc Methods and apparatus to measure exposure to streaming media
US9749685B2 (en) 2015-07-23 2017-08-29 Echostar Technologies L.L.C. Apparatus, systems and methods for accessing information based on an image presented on a display
JP6801225B2 (en) 2016-05-18 2020-12-16 ヤマハ株式会社 Automatic performance system and automatic performance method
CN107657557A (en) * 2016-07-25 2018-02-02 上海佩创网络科技有限公司 A kind of experimental teaching App management systems and application method
WO2018057097A1 (en) * 2016-09-23 2018-03-29 Hewlett-Packard Development Company, L.P. Device setting based on usage history
WO2019099357A1 (en) * 2017-11-14 2019-05-23 Worldcast Live Inc. Computing technologies for musical performances
US10832673B2 (en) 2018-07-13 2020-11-10 International Business Machines Corporation Smart speaker device with cognitive sound analysis and response
US10832672B2 (en) 2018-07-13 2020-11-10 International Business Machines Corporation Smart speaker system with cognitive sound analysis and response
US11252860B2 (en) 2018-11-13 2022-02-22 Cnh Industrial America Llc Agricultural product control system for an agricultural implement
US10956490B2 (en) 2018-12-31 2021-03-23 Audiobyte Llc Audio and visual asset matching platform
US11086931B2 (en) 2018-12-31 2021-08-10 Audiobyte Llc Audio and visual asset matching platform including a master digital asset

Citations (33)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4230990A (en) * 1979-03-16 1980-10-28 Lert John G Jr Broadcast program identification method and system
US4450531A (en) * 1982-09-10 1984-05-22 Ensco, Inc. Broadcast signal recognition system and method
US4739398A (en) * 1986-05-02 1988-04-19 Control Data Corporation Method, apparatus and system for recognizing broadcast segments
US4843562A (en) * 1987-06-24 1989-06-27 Broadcast Data Systems Limited Partnership Broadcast information classification system and method
US4918730A (en) * 1987-06-24 1990-04-17 Media Control-Musik-Medien-Analysen Gesellschaft Mit Beschrankter Haftung Process and circuit arrangement for the automatic recognition of signal sequences
US5125024A (en) * 1990-03-28 1992-06-23 At&T Bell Laboratories Voice response unit
US5134719A (en) * 1991-02-19 1992-07-28 Mankovitz Roy J Apparatus and methods for identifying broadcast audio program selections in an FM stereo broadcast system
US5210820A (en) * 1990-05-02 1993-05-11 Broadcast Data Systems Limited Partnership Signal recognition system and method
US5437050A (en) * 1992-11-09 1995-07-25 Lamb; Robert G. Method and apparatus for recognizing broadcast information using multi-frequency magnitude detection
US5574962A (en) * 1991-09-30 1996-11-12 The Arbitron Company Method and apparatus for automatically identifying a program including a sound signal
US5577249A (en) * 1992-07-31 1996-11-19 International Business Machines Corporation Method for finding a reference token sequence in an original token string within a database of token strings using appended non-contiguous substrings
US5581658A (en) * 1993-12-14 1996-12-03 Infobase Systems, Inc. Adaptive system for broadcast program identification and reporting
US5661787A (en) * 1994-10-27 1997-08-26 Pocock; Michael H. System for on-demand remote access to a self-generating audio recording, storage, indexing and transaction system
US5842162A (en) * 1996-03-08 1998-11-24 Motorola, Inc. Method and recognizer for recognizing a sampled sound signal in noise
US5918223A (en) * 1996-07-22 1999-06-29 Muscle Fish Method and article of manufacture for content-based analysis, storage, retrieval, and segmentation of audio information
US5918213A (en) * 1995-12-22 1999-06-29 Mci Communications Corporation System and method for automated remote previewing and purchasing of music, video, software, and other multimedia products
US5963916A (en) * 1990-09-13 1999-10-05 Intouch Group, Inc. Network apparatus and method for preview of music products and compilation of market data
US5991737A (en) * 1996-03-11 1999-11-23 Connexus Corporation Automated consumer response to publicly broadcast information
US6018724A (en) * 1997-06-30 2000-01-25 Sun Micorsystems, Inc. Method and apparatus for authenticating on-line transaction data
US6121530A (en) * 1998-03-19 2000-09-19 Sonoda; Tomonari World Wide Web-based melody retrieval system with thresholds determined by using distribution of pitch and span of notes
US6233682B1 (en) * 1999-01-22 2001-05-15 Bernhard Fritsch Distribution of musical products by a web site vendor over the internet
US6389403B1 (en) * 1998-08-13 2002-05-14 International Business Machines Corporation Method and apparatus for uniquely identifying a customer purchase in an electronic distribution system
US6389538B1 (en) * 1998-08-13 2002-05-14 International Business Machines Corporation System for tracking end-user electronic content usage
US6496744B1 (en) * 1999-01-11 2002-12-17 David Philip Cook Method and system for custom manufacture and delivery of a data product
US6502194B1 (en) * 1999-04-16 2002-12-31 Synetix Technologies System for playback of network audio material on demand
US6553129B1 (en) * 1995-07-27 2003-04-22 Digimarc Corporation Computer system linked by using information in data objects
US6587127B1 (en) * 1997-11-25 2003-07-01 Motorola, Inc. Content player method and server with user profile
US6606744B1 (en) * 1999-11-22 2003-08-12 Accenture, Llp Providing collaborative installation management in a network-based supply chain environment
US6611812B2 (en) * 1998-08-13 2003-08-26 International Business Machines Corporation Secure electronic content distribution on CDS and DVDs
US6647417B1 (en) * 2000-02-10 2003-11-11 World Theatre, Inc. Music distribution systems
US6671818B1 (en) * 1999-11-22 2003-12-30 Accenture Llp Problem isolation through translating and filtering events into a standard object format in a network based supply chain
US6681029B1 (en) * 1993-11-18 2004-01-20 Digimarc Corporation Decoding steganographic messages embedded in media signals
US6785671B1 (en) * 1999-12-08 2004-08-31 Amazon.Com, Inc. System and method for locating web-based product offerings

Family Cites Families (46)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US571827A (en) * 1896-11-24 Cultivator
US728280A (en) * 1897-12-31 1903-05-19 Gen Fire Extinguisher Co Automatic fire-extinguisher.
US4682370A (en) * 1982-10-18 1987-07-21 Matthews Gordon H Apparatus for automatically detecting and playing desired audio segments over a broadcast receiver
AU633828B2 (en) * 1988-12-05 1993-02-11 Ricos Co., Ltd. Apparatus for reproducing music and displaying words
US5436653A (en) 1992-04-30 1995-07-25 The Arbitron Company Method and system for recognition of broadcast segments
CA2115772C (en) * 1992-06-22 2004-08-17 Roy J. Mankovitz Apparatus and method for identifying broadcast programs and accessing information relating thereto
US5465240A (en) * 1993-01-05 1995-11-07 Mankovitz; Roy J. Apparatus and methods for displaying text in conjunction with recorded audio programs
WO1994018218A1 (en) * 1993-02-01 1994-08-18 Seq, Ltd. Methods and apparatus for dna sequencing
US6288991B1 (en) * 1995-03-06 2001-09-11 Fujitsu Limited Storage medium playback method and device
US6505160B1 (en) * 1995-07-27 2003-01-07 Digimarc Corporation Connected audio and other media objects
US7562392B1 (en) * 1999-05-19 2009-07-14 Digimarc Corporation Methods of interacting with audio and ambient music
US5806031A (en) 1996-04-25 1998-09-08 Motorola Method and recognizer for recognizing tonal acoustic sound signals
JPH09293083A (en) 1996-04-26 1997-11-11 Toshiba Corp Music retrieval device and method
US5740230A (en) * 1996-05-31 1998-04-14 Octel Communications Corporation Directory management system and method
US5822729A (en) * 1996-06-05 1998-10-13 Massachusetts Institute Of Technology Feature-based speech recognizer having probabilistic linguistic processor providing word matching based on the entire space of feature vectors
US5721827A (en) * 1996-10-02 1998-02-24 James Logan System for electrically distributing personalized information
US5828314A (en) * 1996-10-16 1998-10-27 Seiko Communication Systems Pager with adaptable alarm
US5952597A (en) * 1996-10-25 1999-09-14 Timewarp Technologies, Ltd. Method and apparatus for real-time correlation of a performance to a musical score
US7228280B1 (en) * 1997-04-15 2007-06-05 Gracenote, Inc. Finding database match for file based on file characteristics
US5987525A (en) * 1997-04-15 1999-11-16 Cddb, Inc. Network delivery of interactive entertainment synchronized to playback of audio recordings
JPH1155201A (en) * 1997-07-29 1999-02-26 Sony Corp Device, method and system for information processing and transmitting medium
AU9678998A (en) 1997-10-03 1999-04-27 Peter Polash Internet based musical indexing system for radio
US5973252A (en) * 1997-10-27 1999-10-26 Auburn Audio Technologies, Inc. Pitch detection and intonation correction apparatus and method
US6941274B1 (en) * 1997-11-28 2005-09-06 Diebold, Incorporated Automated transaction machine
US20010014868A1 (en) * 1997-12-05 2001-08-16 Frederick Herz System for the automatic determination of customized prices and promotions
FR2776418A1 (en) 1998-03-17 1999-09-24 Didier Valade Sound module for attachment to video or audiocassette providing means for voice based recording of contents of tape
US6201176B1 (en) 1998-05-07 2001-03-13 Canon Kabushiki Kaisha System and method for querying a music database
DE69943228D1 (en) * 1998-07-14 2011-04-07 Sony Corp Data receiving device
JP3925586B2 (en) * 1998-07-17 2007-06-06 ソニー株式会社 Data receiving apparatus and method, and data transmission / reception system and method
US6182018B1 (en) 1998-08-25 2001-01-30 Ford Global Technologies, Inc. Method and apparatus for identifying sound in a composite sound signal
US7426480B2 (en) * 1999-06-09 2008-09-16 Catalina Marketing Corporation In-vehicle promotions system
TW495735B (en) * 1999-07-28 2002-07-21 Yamaha Corp Audio controller and the portable terminal and system using the same
US7174293B2 (en) * 1999-09-21 2007-02-06 Iceberg Industries Llc Audio identification system and method
US7194752B1 (en) * 1999-10-19 2007-03-20 Iceberg Industries, Llc Method and apparatus for automatically recognizing input audio and/or video streams
US6941275B1 (en) * 1999-10-07 2005-09-06 Remi Swierczek Music identification system
US6954859B1 (en) * 1999-10-08 2005-10-11 Axcess, Inc. Networked digital security system and methods
DE19948974A1 (en) * 1999-10-11 2001-04-12 Nokia Mobile Phones Ltd Method for recognizing and selecting a tone sequence, in particular a piece of music
US7444353B1 (en) * 2000-01-31 2008-10-28 Chen Alexander C Apparatus for delivering music and information
US6928414B1 (en) * 2000-04-12 2005-08-09 Jae Heon Kim Advertisement method using game program on the internet and method for executing the game program having the advertisement according to the advertisement method
FR2808906B1 (en) * 2000-05-10 2005-02-11 Touchtunes Music Corp DEVICE AND METHOD FOR REMOTELY MANAGING A NETWORK OF AUDIOVISUAL INFORMATION REPRODUCTION SYSTEMS
US6990453B2 (en) * 2000-07-31 2006-01-24 Landmark Digital Services Llc System and methods for recognizing sound and music signals in high noise and distortion
US7853664B1 (en) * 2000-07-31 2010-12-14 Landmark Digital Services Llc Method and system for purchasing pre-recorded music
FI20002161A (en) * 2000-09-29 2002-03-30 Nokia Mobile Phones Ltd Method and system for recognizing a melody
US7277766B1 (en) * 2000-10-24 2007-10-02 Moodlogic, Inc. Method and system for analyzing digital audio files
US20020072982A1 (en) * 2000-12-12 2002-06-13 Shazam Entertainment Ltd. Method and system for interacting with a user in an experiential environment
JP4029735B2 (en) 2003-01-28 2008-01-09 ヤマハ株式会社 Song data reproducing apparatus and program

Patent Citations (40)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4230990C1 (en) * 1979-03-16 2002-04-09 John G Lert Jr Broadcast program identification method and system
US4230990A (en) * 1979-03-16 1980-10-28 Lert John G Jr Broadcast program identification method and system
US4450531A (en) * 1982-09-10 1984-05-22 Ensco, Inc. Broadcast signal recognition system and method
US4739398A (en) * 1986-05-02 1988-04-19 Control Data Corporation Method, apparatus and system for recognizing broadcast segments
US4843562A (en) * 1987-06-24 1989-06-27 Broadcast Data Systems Limited Partnership Broadcast information classification system and method
US4918730A (en) * 1987-06-24 1990-04-17 Media Control-Musik-Medien-Analysen Gesellschaft Mit Beschrankter Haftung Process and circuit arrangement for the automatic recognition of signal sequences
US5125024A (en) * 1990-03-28 1992-06-23 At&T Bell Laboratories Voice response unit
US5210820A (en) * 1990-05-02 1993-05-11 Broadcast Data Systems Limited Partnership Signal recognition system and method
US5963916A (en) * 1990-09-13 1999-10-05 Intouch Group, Inc. Network apparatus and method for preview of music products and compilation of market data
US5134719A (en) * 1991-02-19 1992-07-28 Mankovitz Roy J Apparatus and methods for identifying broadcast audio program selections in an FM stereo broadcast system
US5581800A (en) * 1991-09-30 1996-12-03 The Arbitron Company Method and apparatus for automatically identifying a program including a sound signal
US5787334A (en) * 1991-09-30 1998-07-28 Ceridian Corporation Method and apparatus for automatically identifying a program including a sound signal
US5574962A (en) * 1991-09-30 1996-11-12 The Arbitron Company Method and apparatus for automatically identifying a program including a sound signal
US5577249A (en) * 1992-07-31 1996-11-19 International Business Machines Corporation Method for finding a reference token sequence in an original token string within a database of token strings using appended non-contiguous substrings
US5437050A (en) * 1992-11-09 1995-07-25 Lamb; Robert G. Method and apparatus for recognizing broadcast information using multi-frequency magnitude detection
US6681029B1 (en) * 1993-11-18 2004-01-20 Digimarc Corporation Decoding steganographic messages embedded in media signals
US5581658A (en) * 1993-12-14 1996-12-03 Infobase Systems, Inc. Adaptive system for broadcast program identification and reporting
US5661787A (en) * 1994-10-27 1997-08-26 Pocock; Michael H. System for on-demand remote access to a self-generating audio recording, storage, indexing and transaction system
US6553129B1 (en) * 1995-07-27 2003-04-22 Digimarc Corporation Computer system linked by using information in data objects
US5918213A (en) * 1995-12-22 1999-06-29 Mci Communications Corporation System and method for automated remote previewing and purchasing of music, video, software, and other multimedia products
US5842162A (en) * 1996-03-08 1998-11-24 Motorola, Inc. Method and recognizer for recognizing a sampled sound signal in noise
US5991737A (en) * 1996-03-11 1999-11-23 Connexus Corporation Automated consumer response to publicly broadcast information
US5918223A (en) * 1996-07-22 1999-06-29 Muscle Fish Method and article of manufacture for content-based analysis, storage, retrieval, and segmentation of audio information
US6018724A (en) * 1997-06-30 2000-01-25 Sun Micorsystems, Inc. Method and apparatus for authenticating on-line transaction data
US6587127B1 (en) * 1997-11-25 2003-07-01 Motorola, Inc. Content player method and server with user profile
US6121530A (en) * 1998-03-19 2000-09-19 Sonoda; Tomonari World Wide Web-based melody retrieval system with thresholds determined by using distribution of pitch and span of notes
US6418421B1 (en) * 1998-08-13 2002-07-09 International Business Machines Corporation Multimedia player for an electronic content delivery system
US6611812B2 (en) * 1998-08-13 2003-08-26 International Business Machines Corporation Secure electronic content distribution on CDS and DVDs
US6398245B1 (en) * 1998-08-13 2002-06-04 International Business Machines Corporation Key management system for digital content player
US6389538B1 (en) * 1998-08-13 2002-05-14 International Business Machines Corporation System for tracking end-user electronic content usage
US6574609B1 (en) * 1998-08-13 2003-06-03 International Business Machines Corporation Secure electronic content management system
US6389403B1 (en) * 1998-08-13 2002-05-14 International Business Machines Corporation Method and apparatus for uniquely identifying a customer purchase in an electronic distribution system
US6587837B1 (en) * 1998-08-13 2003-07-01 International Business Machines Corporation Method for delivering electronic content from an online store
US6496744B1 (en) * 1999-01-11 2002-12-17 David Philip Cook Method and system for custom manufacture and delivery of a data product
US6233682B1 (en) * 1999-01-22 2001-05-15 Bernhard Fritsch Distribution of musical products by a web site vendor over the internet
US6502194B1 (en) * 1999-04-16 2002-12-31 Synetix Technologies System for playback of network audio material on demand
US6671818B1 (en) * 1999-11-22 2003-12-30 Accenture Llp Problem isolation through translating and filtering events into a standard object format in a network based supply chain
US6606744B1 (en) * 1999-11-22 2003-08-12 Accenture, Llp Providing collaborative installation management in a network-based supply chain environment
US6785671B1 (en) * 1999-12-08 2004-08-31 Amazon.Com, Inc. System and method for locating web-based product offerings
US6647417B1 (en) * 2000-02-10 2003-11-11 World Theatre, Inc. Music distribution systems

Cited By (330)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8094949B1 (en) 1994-10-21 2012-01-10 Digimarc Corporation Music methods and systems
US8059858B2 (en) 1998-11-19 2011-11-15 Digimarc Corporation Identification document and related methods
US20070016790A1 (en) * 1998-11-19 2007-01-18 Brundage Trent J Identification document and related methods
US20070250716A1 (en) * 2000-05-02 2007-10-25 Brunk Hugh L Fingerprinting of Media Signals
US8121843B2 (en) 2000-05-02 2012-02-21 Digimarc Corporation Fingerprint methods and systems for media signals
US7003517B1 (en) 2000-05-24 2006-02-21 Inetprofit, Inc. Web-based system and method for archiving and searching participant-based internet text sources for customer lead data
US20060004731A1 (en) * 2000-05-24 2006-01-05 Seibel John C Text mining system for web-based business intelligence
US7096220B1 (en) * 2000-05-24 2006-08-22 Reachforce, Inc. Web-based customer prospects harvester system
US7082427B1 (en) 2000-05-24 2006-07-25 Reachforce, Inc. Text indexing system to index, query the archive database document by keyword data representing the content of the documents and by contact data associated with the participant who generated the document
US9864958B2 (en) 2000-06-29 2018-01-09 Gula Consulting Limited Liability Company System, method, and computer program product for video based services and commerce
USRE42870E1 (en) 2000-10-04 2011-10-25 Dafineais Protocol Data B.V., Llc Text mining system for web-based business intelligence applied to web site server logs
US20140214532A1 (en) * 2000-12-12 2014-07-31 Shazam Investments Limited Method and System for Interacting with a User in an Experimental Environment
US8996380B2 (en) 2000-12-12 2015-03-31 Shazam Entertainment Ltd. Methods and systems for synchronizing media
US9721287B2 (en) * 2000-12-12 2017-08-01 Shazam Investments Limited Method and system for interacting with a user in an experimental environment
US20070033409A1 (en) * 2000-12-21 2007-02-08 Brunk Hugh L Methods, Apparatus and Programs for Generating and Utilizing Content Signatures
US8077911B2 (en) 2000-12-21 2011-12-13 Digimarc Corporation Methods, apparatus and programs for generating and utilizing content signatures
US9843846B2 (en) 2000-12-21 2017-12-12 Digimarc Corporation Watermark and fingerprint systems for media
US8825518B2 (en) 2000-12-21 2014-09-02 Digimarc Corporation Media methods and systems
US20070101147A1 (en) * 2000-12-21 2007-05-03 Brunk Hugh L Methods, Apparatus and Programs for Generating and Utilizing Content Signatures
US8023773B2 (en) 2000-12-21 2011-09-20 Digimarc Corporation Methods, apparatus and programs for generating and utilizing content signatures
US20090034807A1 (en) * 2001-04-24 2009-02-05 Id3Man, Inc. Comparison of Data Signals Using Characteristic Electronic Thumbprints Extracted Therefrom
US7421376B1 (en) 2001-04-24 2008-09-02 Auditude, Inc. Comparison of data signals using characteristic electronic thumbprints
US7853438B2 (en) 2001-04-24 2010-12-14 Auditude, Inc. Comparison of data signals using characteristic electronic thumbprints extracted therefrom
US8170273B2 (en) 2001-04-25 2012-05-01 Digimarc Corporation Encoding and decoding auxiliary signals
US20030086341A1 (en) * 2001-07-20 2003-05-08 Gracenote, Inc. Automatic identification of sound recordings
US7328153B2 (en) 2001-07-20 2008-02-05 Gracenote, Inc. Automatic identification of sound recordings
US8085978B2 (en) 2001-10-16 2011-12-27 Digimarc Corporation Distributed decoding of digitally encoded media signals
US20110069861A1 (en) * 2001-10-16 2011-03-24 Brundage Trent J Distributed Decoding of Digitally Encoded Media Signals
US20070286454A1 (en) * 2001-10-16 2007-12-13 Brundage Trent J Distributed Content Identification
US7676060B2 (en) 2001-10-16 2010-03-09 Brundage Trent J Distributed content identification
US20030177014A1 (en) * 2002-02-13 2003-09-18 Crawford Robert E. System and method for displaying multiple language text during performance or play of a musical work
US7627477B2 (en) * 2002-04-25 2009-12-01 Landmark Digital Services, Llc Robust and invariant audio pattern matching
US20090265174A9 (en) * 2002-04-25 2009-10-22 Wang Avery L Robust and invariant audio pattern matching
US20050177372A1 (en) * 2002-04-25 2005-08-11 Wang Avery L. Robust and invariant audio pattern matching
US7987491B2 (en) 2002-05-10 2011-07-26 Richard Reisman Method and apparatus for browsing using alternative linkbases
US20110219419A1 (en) * 2002-05-10 2011-09-08 Richard Reisman Method and apparatus for browsing using alternative linkbases
US8898722B2 (en) 2002-05-10 2014-11-25 Convergent Media Solutions Llc Method and apparatus for browsing using alternative linkbases
US8689273B2 (en) 2002-05-10 2014-04-01 Convergent Media Solutions, LLC Method and apparatus for browsing using alternative linkbases
US8661495B2 (en) 2002-05-10 2014-02-25 Convergent Media Solutions, LLC Method and apparatus for browsing using alternative linkbases
US8646020B2 (en) 2002-05-10 2014-02-04 Convergent Media Solutions, LLC Method and apparatus for browsing using alternative linkbases
US20030229900A1 (en) * 2002-05-10 2003-12-11 Richard Reisman Method and apparatus for browsing using multiple coordinated device sets
US8161172B2 (en) 2002-05-10 2012-04-17 Teleshuttle Tech2, Llc Method and apparatus for browsing using multiple coordinated device sets
US8893212B2 (en) 2002-05-10 2014-11-18 Convergent Media Solutions Llc Method and apparatus for browsing using alternative linkbases
US8640183B2 (en) 2002-05-10 2014-01-28 Convergent Media Solutions, LLC Method and apparatus for browsing using alternative linkbases
US8914840B2 (en) 2002-05-10 2014-12-16 Convergent Media Solutions Llc Method and apparatus for browsing using alternative linkbases
US20090319672A1 (en) * 2002-05-10 2009-12-24 Richard Reisman Method and Apparatus for Browsing Using Multiple Coordinated Device Sets
US20090320073A1 (en) * 2002-05-10 2009-12-24 Richard Reisman Method and Apparatus for Browsing Using Multiple Coordinated Device Sets
US8875215B2 (en) 2002-05-10 2014-10-28 Convergent Media Solutions Llc Method and apparatus for browsing using alternative linkbases
US8631456B2 (en) 2002-05-10 2014-01-14 Convergent Media Solutions, LLC Method and apparatus for browsing using alternative linkbases
US8813125B2 (en) 2002-05-10 2014-08-19 Convergent Media Solutions Llc Method and apparatus for browsing using alternative linkbases
US8850507B2 (en) 2002-05-10 2014-09-30 Convergent Media Solutions Llc Method and apparatus for browsing using alternative linkbases
US7824029B2 (en) 2002-05-10 2010-11-02 L-1 Secure Credentialing, Inc. Identification card printer-assembler for over the counter card issuing
US8527640B2 (en) 2002-05-10 2013-09-03 Teleshuttle Tech2, Llc Method and apparatus for browsing using multiple coordinated device sets
US7899915B2 (en) 2002-05-10 2011-03-01 Richard Reisman Method and apparatus for browsing using multiple coordinated device sets
US9143839B2 (en) 2002-05-10 2015-09-22 Convergent Media Solutions Llc Method and apparatus for browsing using multiple coordinated device sets
US11182121B2 (en) 2002-05-23 2021-11-23 Gula Consulting Limited Liability Company Navigating an information hierarchy using a mobile communication device
US9996315B2 (en) * 2002-05-23 2018-06-12 Gula Consulting Limited Liability Company Systems and methods using audio input with a mobile device
US9858595B2 (en) 2002-05-23 2018-01-02 Gula Consulting Limited Liability Company Location-based transmissions using a mobile communication device
US10489449B2 (en) 2002-05-23 2019-11-26 Gula Consulting Limited Liability Company Computer accepting voice input and/or generating audible output
US9711153B2 (en) 2002-09-27 2017-07-18 The Nielsen Company (Us), Llc Activating functions in processing devices using encoded audio and detecting audio signatures
US6909981B2 (en) * 2003-01-27 2005-06-21 Ciphergen Biosystems, Inc. Data management system and method for processing signals from sample spots
WO2004070523A2 (en) * 2003-01-27 2004-08-19 Ciphergen Biosystems Inc. Data management system and method for processing signals from sample spots
US20040159783A1 (en) * 2003-01-27 2004-08-19 Ciphergen Biosystems, Inc. Data management system and method for processing signals from sample spots
WO2004070523A3 (en) * 2003-01-27 2005-03-17 Ciphergen Biosystems Inc Data management system and method for processing signals from sample spots
US7606790B2 (en) 2003-03-03 2009-10-20 Digimarc Corporation Integrating and enhancing searching of media content and biometric databases
US20040243567A1 (en) * 2003-03-03 2004-12-02 Levy Kenneth L. Integrating and enhancing searching of media content and biometric databases
US8055667B2 (en) 2003-03-03 2011-11-08 Digimarc Corporation Integrating and enhancing searching of media content and biometric databases
US8554681B1 (en) * 2003-11-03 2013-10-08 James W. Wieder Providing “identified” compositions and digital-works
US11165999B1 (en) * 2003-11-03 2021-11-02 Synergyze Technologies Llc Identifying and providing compositions and digital-works
US20050151669A1 (en) * 2003-11-28 2005-07-14 Craig Swallow Lone worker monitor
US7412264B2 (en) * 2003-11-28 2008-08-12 Connexion2 Limited Lone worker monitor
US20080256032A1 (en) * 2004-01-20 2008-10-16 Koninklike Philips Electronics, N.V. Hierarchical Playlist Generator
US7840620B2 (en) * 2004-01-20 2010-11-23 Koninklijke Philips Electronics N.V. Hierarchical playlist generator
US20050193016A1 (en) * 2004-02-17 2005-09-01 Nicholas Seet Generation of a media content database by correlating repeating media content in media streams
US20110244784A1 (en) * 2004-02-19 2011-10-06 Landmark Digital Services Llc Method and apparatus for identification of broadcast source
US8290423B2 (en) * 2004-02-19 2012-10-16 Shazam Investments Limited Method and apparatus for identification of broadcast source
US9071371B2 (en) 2004-02-19 2015-06-30 Shazam Investments Limited Method and apparatus for identification of broadcast source
US9225444B2 (en) 2004-02-19 2015-12-29 Shazam Investments Limited Method and apparatus for identification of broadcast source
US8811885B2 (en) 2004-02-19 2014-08-19 Shazam Investments Limited Method and apparatus for identification of broadcast source
US20050215239A1 (en) * 2004-03-26 2005-09-29 Nokia Corporation Feature extraction in a networked portable device
US20050222913A1 (en) * 2004-04-01 2005-10-06 Ian Eisenberg PR/SMS business method, system and method of conducting business
US8060008B2 (en) 2004-04-07 2011-11-15 Nokia Corporation Mobile station and interface adapted for feature extraction from an input media sample
US20080032622A1 (en) * 2004-04-07 2008-02-07 Nokia Corporation Mobile station and interface adapted for feature extraction from an input media sample
US10719848B2 (en) 2004-05-27 2020-07-21 Anonymous Media Research LLC Media usage monitoring and measurement system and method
US20090076812A1 (en) * 2004-05-27 2009-03-19 Anonymous Media, Llc Media usage monitoring and measurement system and method
US20050267750A1 (en) * 2004-05-27 2005-12-01 Anonymous Media, Llc Media usage monitoring and measurement system and method
US8756622B2 (en) 2004-05-27 2014-06-17 Anonymous Media Research, Llc Media usage monitoring and measurement system and method
US20090018684A1 (en) * 2004-05-27 2009-01-15 Anonymous Media, Llc Media usage monitoring and measurement system and method
US20090019466A1 (en) * 2004-05-27 2009-01-15 Anonymous Media, Llc Media usage monitoring and measurement system and method
US20090018827A1 (en) * 2004-05-27 2009-01-15 Anonymous Media, Llc Media usage monitoring and measurement system and method
US20150051967A1 (en) * 2004-05-27 2015-02-19 Anonymous Media Research, Llc Media usage monitoring and measurment system and method
US8296791B2 (en) 2004-05-27 2012-10-23 Anonymous Media Research LLC Media usage monitoring and measurement system and method
US8677389B2 (en) 2004-05-27 2014-03-18 Anonymous Media Research, Llc Media usage monitoring and measurement system and method
US10963911B2 (en) 2004-05-27 2021-03-30 Anonymous Media Research LLC Media usage monitoring and measurement system and method
US20110131340A1 (en) * 2004-05-27 2011-06-02 Anonymous Media Research, Llc Media Usage Monitoring and Measurement System and Method
US8510768B2 (en) * 2004-05-27 2013-08-13 Anonymous Media Research, Llc Media usage monitoring and measurement system and method
US10719849B2 (en) 2004-05-27 2020-07-21 Anonymous Media Research LLC Media usage monitoring and measurement system and method
US10572896B2 (en) 2004-05-27 2020-02-25 Anonymous Media Research LLC Media usage monitoring and measurement system and method
EP1617365A1 (en) * 2004-07-16 2006-01-18 Vodafone Holding GmbH Coordination and processing of games serving education, entertainment or competition purposes
US8214067B1 (en) * 2004-09-24 2012-07-03 Sztybel Robert S Interactive audio content delivery system and method
US9910341B2 (en) 2005-01-31 2018-03-06 The Invention Science Fund I, Llc Shared image device designation
US10003762B2 (en) 2005-04-26 2018-06-19 Invention Science Fund I, Llc Shared image devices
US20060246412A1 (en) * 2005-05-01 2006-11-02 George Pieczenik Professor Pieczenik's instruction method and system
US20090077578A1 (en) * 2005-05-26 2009-03-19 Anonymous Media, Llc Media usage monitoring and measurement system and method
US10097756B2 (en) 2005-06-02 2018-10-09 Invention Science Fund I, Llc Enhanced video/still image correlation
US7516074B2 (en) 2005-09-01 2009-04-07 Auditude, Inc. Extraction and matching of characteristic fingerprints from audio signals
US9942511B2 (en) 2005-10-31 2018-04-10 Invention Science Fund I, Llc Preservation/degradation of video/audio aspects of a data stream
US20070196802A1 (en) * 2006-02-21 2007-08-23 Nokia Corporation Visually Enhanced Personal Music Broadcast
US20090282334A1 (en) * 2006-05-04 2009-11-12 Mobilians Co., Ltd. System and method for providing information using outside sound recognition of mobile phone, and mobile terminal for the same
US10133780B2 (en) 2006-12-01 2018-11-20 Scenera Mobile Technologies, Llc Methods, systems, and computer program products for determining availability of presentable content
US8438145B2 (en) 2006-12-01 2013-05-07 Scenera Technologies, Llc Methods, systems, and computer program products for determining availability of presentable content via a subscription service
US20110213760A1 (en) * 2006-12-01 2011-09-01 Jeffrey Scott Bardsley Methods, Systems, And Computer Program Products For Determining Availability Of Presentable Content Via A Subscription Service
US8554265B1 (en) * 2007-01-17 2013-10-08 At&T Mobility Ii Llc Distribution of user-generated multimedia broadcasts to mobile wireless telecommunication network users
US20080208851A1 (en) * 2007-02-27 2008-08-28 Landmark Digital Services Llc System and method for monitoring and recognizing broadcast data
US8453170B2 (en) 2007-02-27 2013-05-28 Landmark Digital Services Llc System and method for monitoring and recognizing broadcast data
US20090144391A1 (en) * 2007-11-30 2009-06-04 Searete Llc, A Limited Liability Corporation Of The State Of Delaware Audio sharing
US20100070501A1 (en) * 2008-01-15 2010-03-18 Walsh Paul J Enhancing and storing data for recall and use using user feedback
US20130030853A1 (en) * 2008-01-15 2013-01-31 Agarwal Amit D Enhancing and storing data for recall and use
EP2250575A4 (en) * 2008-01-15 2012-09-26 Amazon Tech Inc Enhancing and storing data for recall and use
US20090182622A1 (en) * 2008-01-15 2009-07-16 Agarwal Amit D Enhancing and storing data for recall and use
US10922957B2 (en) 2008-08-19 2021-02-16 Digimarc Corporation Methods and systems for content processing
US20100113091A1 (en) * 2008-10-31 2010-05-06 Sharma Ravi K Histogram methods and systems for object recognition
US8004576B2 (en) 2008-10-31 2011-08-23 Digimarc Corporation Histogram methods and systems for object recognition
US10631068B2 (en) 2008-11-26 2020-04-21 Free Stream Media Corp. Content exposure attribution based on renderings of related content across multiple devices
US9854330B2 (en) 2008-11-26 2017-12-26 David Harrison Relevancy improvement through targeting of information based on data gathered from a networked device associated with a security sandbox of a client device
US10334324B2 (en) 2008-11-26 2019-06-25 Free Stream Media Corp. Relevant advertisement generation based on a user operating a client device communicatively coupled with a networked media device
US10142377B2 (en) 2008-11-26 2018-11-27 Free Stream Media Corp. Relevancy improvement through targeting of information based on data gathered from a networked device associated with a security sandbox of a client device
US10977693B2 (en) 2008-11-26 2021-04-13 Free Stream Media Corp. Association of content identifier of audio-visual data with additional data through capture infrastructure
US10074108B2 (en) 2008-11-26 2018-09-11 Free Stream Media Corp. Annotation of metadata through capture infrastructure
US10791152B2 (en) 2008-11-26 2020-09-29 Free Stream Media Corp. Automatic communications between networked devices such as televisions and mobile devices
US10032191B2 (en) 2008-11-26 2018-07-24 Free Stream Media Corp. Advertisement targeting through embedded scripts in supply-side and demand-side platforms
US9986279B2 (en) 2008-11-26 2018-05-29 Free Stream Media Corp. Discovery, access control, and communication with networked services
US9967295B2 (en) 2008-11-26 2018-05-08 David Harrison Automated discovery and launch of an application on a network enabled device
US9961388B2 (en) 2008-11-26 2018-05-01 David Harrison Exposure of public internet protocol addresses in an advertising exchange server to improve relevancy of advertisements
US10419541B2 (en) 2008-11-26 2019-09-17 Free Stream Media Corp. Remotely control devices over a network without authentication or registration
US9866925B2 (en) 2008-11-26 2018-01-09 Free Stream Media Corp. Relevancy improvement through targeting of information based on data gathered from a networked device associated with a security sandbox of a client device
US10567823B2 (en) 2008-11-26 2020-02-18 Free Stream Media Corp. Relevant advertisement generation based on a user operating a client device communicatively coupled with a networked media device
US9848250B2 (en) 2008-11-26 2017-12-19 Free Stream Media Corp. Relevancy improvement through targeting of information based on data gathered from a networked device associated with a security sandbox of a client device
US10771525B2 (en) 2008-11-26 2020-09-08 Free Stream Media Corp. System and method of discovery and launch associated with a networked media device
US10986141B2 (en) 2008-11-26 2021-04-20 Free Stream Media Corp. Relevancy improvement through targeting of information based on data gathered from a networked device associated with a security sandbox of a client device
US10880340B2 (en) 2008-11-26 2020-12-29 Free Stream Media Corp. Relevancy improvement through targeting of information based on data gathered from a networked device associated with a security sandbox of a client device
US9838758B2 (en) 2008-11-26 2017-12-05 David Harrison Relevancy improvement through targeting of information based on data gathered from a networked device associated with a security sandbox of a client device
US9560425B2 (en) 2008-11-26 2017-01-31 Free Stream Media Corp. Remotely control devices over a network without authentication or registration
US10425675B2 (en) 2008-11-26 2019-09-24 Free Stream Media Corp. Discovery, access control, and communication with networked services
US9716736B2 (en) 2008-11-26 2017-07-25 Free Stream Media Corp. System and method of discovery and launch associated with a networked media device
US9703947B2 (en) 2008-11-26 2017-07-11 Free Stream Media Corp. Relevancy improvement through targeting of information based on data gathered from a networked device associated with a security sandbox of a client device
US9706265B2 (en) 2008-11-26 2017-07-11 Free Stream Media Corp. Automatic communications between networked devices such as televisions and mobile devices
US9686596B2 (en) 2008-11-26 2017-06-20 Free Stream Media Corp. Advertisement targeting through embedded scripts in supply-side and demand-side platforms
US9591381B2 (en) 2008-11-26 2017-03-07 Free Stream Media Corp. Automated discovery and launch of an application on a network enabled device
US8433431B1 (en) * 2008-12-02 2013-04-30 Soundhound, Inc. Displaying text to end users in coordination with audio playback
US8977974B2 (en) * 2008-12-08 2015-03-10 Apple Inc. Ambient noise based augmentation of media playback
US20100146445A1 (en) * 2008-12-08 2010-06-10 Apple Inc. Ambient Noise Based Augmentation of Media Playback
US10073670B2 (en) 2008-12-08 2018-09-11 Apple Inc. Ambient noise based augmentation of media playback
US9135520B2 (en) 2009-05-19 2015-09-15 Digimarc Corporation Histogram methods and systems for object recognition
US9811757B2 (en) 2009-05-19 2017-11-07 Digimarc Corporation Histogram methods and systems for object recognition
US8767084B2 (en) 2009-05-19 2014-07-01 Digimarc Corporation Histogram methods and systems for object recognition
US8539106B2 (en) 2009-05-27 2013-09-17 Spot411 Technologies, Inc. Server for aggregating search activity synchronized to time-based media
US20110202524A1 (en) * 2009-05-27 2011-08-18 Ajay Shah Tracking time-based selection of search results
US20110208334A1 (en) * 2009-05-27 2011-08-25 Glitsch Hans M Audio-based synchronization server
US20110208333A1 (en) * 2009-05-27 2011-08-25 Glitsch Hans M Pre-processing media for audio-based synchronization
US20110016172A1 (en) * 2009-05-27 2011-01-20 Ajay Shah Synchronized delivery of interactive content
US20110209191A1 (en) * 2009-05-27 2011-08-25 Ajay Shah Device for presenting interactive content
US8751690B2 (en) 2009-05-27 2014-06-10 Spot411 Technologies, Inc. Tracking time-based selection of search results
US8521811B2 (en) 2009-05-27 2013-08-27 Spot411 Technologies, Inc. Device for presenting interactive content
US8718805B2 (en) * 2009-05-27 2014-05-06 Spot411 Technologies, Inc. Audio-based synchronization to media
US20100305729A1 (en) * 2009-05-27 2010-12-02 Glitsch Hans M Audio-based synchronization to media
US8489777B2 (en) 2009-05-27 2013-07-16 Spot411 Technologies, Inc. Server for presenting interactive content synchronized to time-based media
US8489774B2 (en) 2009-05-27 2013-07-16 Spot411 Technologies, Inc. Synchronized delivery of interactive content
US20110202687A1 (en) * 2009-05-27 2011-08-18 Glitsch Hans M Synchronizing audience feedback from live and time-shifted broadcast views
US8789084B2 (en) 2009-05-27 2014-07-22 Spot411 Technologies, Inc. Identifying commercial breaks in broadcast media
US20110202156A1 (en) * 2009-05-27 2011-08-18 Glitsch Hans M Device with audio-based media synchronization
US20110159921A1 (en) * 2009-12-31 2011-06-30 Davis Bruce L Methods and arrangements employing sensor-equipped smart phones
US9197736B2 (en) 2009-12-31 2015-11-24 Digimarc Corporation Intuitive computing methods and systems
US20110161076A1 (en) * 2009-12-31 2011-06-30 Davis Bruce L Intuitive Computing Methods and Systems
US9609117B2 (en) 2009-12-31 2017-03-28 Digimarc Corporation Methods and arrangements employing sensor-equipped smart phones
US9143603B2 (en) 2009-12-31 2015-09-22 Digimarc Corporation Methods and arrangements employing sensor-equipped smart phones
US20110202338A1 (en) * 2010-02-18 2011-08-18 Philip Inghelbrecht System and method for recognition of alphanumeric patterns including license plate numbers
US8816179B2 (en) 2010-05-04 2014-08-26 Shazam Entertainment Ltd. Methods and systems for disambiguation of an identification of a sample of a media stream
US10003664B2 (en) 2010-05-04 2018-06-19 Shazam Entertainment Ltd. Methods and systems for processing a sample of a media stream
US20140360343A1 (en) * 2010-05-04 2014-12-11 Shazam Entertainment Limited Methods and Systems for Disambiguation of an Identification of a Sample of a Media Stream
US8686271B2 (en) * 2010-05-04 2014-04-01 Shazam Entertainment Ltd. Methods and systems for synchronizing media
US9159338B2 (en) 2010-05-04 2015-10-13 Shazam Entertainment Ltd. Systems and methods of rendering a textual animation
EP3418917A1 (en) * 2010-05-04 2018-12-26 Shazam Entertainment Ltd. Methods and systems for synchronizing media
US20110276333A1 (en) * 2010-05-04 2011-11-10 Avery Li-Chun Wang Methods and Systems for Synchronizing Media
US9251796B2 (en) * 2010-05-04 2016-02-02 Shazam Entertainment Ltd. Methods and systems for disambiguation of an identification of a sample of a media stream
US9275141B2 (en) 2010-05-04 2016-03-01 Shazam Entertainment Ltd. Methods and systems for processing a sample of a media stream
US20110307786A1 (en) * 2010-06-11 2011-12-15 Brian Shuster Method and apparatus for bookmarking and retrieval of video content
US9111537B1 (en) 2010-06-29 2015-08-18 Google Inc. Real-time audio recognition protocol
US20120010736A1 (en) * 2010-07-09 2012-01-12 Nexidia Inc. Spotting multimedia
US8832320B2 (en) 2010-07-16 2014-09-09 Spot411 Technologies, Inc. Server for presenting interactive content synchronized to time-based media
US9047371B2 (en) 2010-07-29 2015-06-02 Soundhound, Inc. System and method for matching a query against a broadcast stream
US10055490B2 (en) 2010-07-29 2018-08-21 Soundhound, Inc. System and methods for continuous audio matching
US10657174B2 (en) 2010-07-29 2020-05-19 Soundhound, Inc. Systems and methods for providing identification information in response to an audio segment
US9563699B1 (en) 2010-07-29 2017-02-07 Soundhound, Inc. System and method for matching a query against a broadcast stream
US9390167B2 (en) 2010-07-29 2016-07-12 Soundhound, Inc. System and methods for continuous audio matching
US8781152B2 (en) 2010-08-05 2014-07-15 Brian Momeyer Identifying visual media content captured by camera-enabled mobile device
US20120162513A1 (en) * 2010-12-28 2012-06-28 Tse-Ming Chang Dual-connection multimedia apparatus
US9093120B2 (en) 2011-02-10 2015-07-28 Yahoo! Inc. Audio fingerprint extraction by scaling in time and resampling
US10121165B1 (en) 2011-05-10 2018-11-06 Soundhound, Inc. System and method for targeting content based on identified audio and multimedia
US10832287B2 (en) 2011-05-10 2020-11-10 Soundhound, Inc. Promotional content targeting based on recognized audio
CN103797482A (en) * 2011-06-08 2014-05-14 沙扎姆娱乐有限公司 Methods and systems for performing comparisons of received data and providing follow-on service based on the comparisons
WO2012170451A1 (en) * 2011-06-08 2012-12-13 Shazam Entertainment Ltd. Methods and systems for performing comparisons of received data and providing a follow-on service based on the comparisons
US9256673B2 (en) 2011-06-10 2016-02-09 Shazam Entertainment Ltd. Methods and systems for identifying content in a data stream
US9014615B2 (en) 2011-08-30 2015-04-21 Iheartmedia Management Services, Inc. Broadcast source identification based on matching broadcast signal fingerprints
US11095380B2 (en) 2011-08-30 2021-08-17 Iheartmedia Management Services, Inc. Source identification using parallel accumulation and comparison of broadcast fingerprints
US9374183B2 (en) 2011-08-30 2016-06-21 Iheartmedia Management Services, Inc. Broadcast source identification based on matching via bit count
US10763983B2 (en) 2011-08-30 2020-09-01 Iheartmedia Management Services, Inc. Identification of unknown altered versions of a known base media item
US9461759B2 (en) 2011-08-30 2016-10-04 Iheartmedia Management Services, Inc. Identification of changed broadcast media items
US9203538B2 (en) 2011-08-30 2015-12-01 Iheartmedia Management Services, Inc. Broadcast source identification based on matching broadcast signal fingerprints
US9960868B2 (en) 2011-08-30 2018-05-01 Iheartmedia Management Services, Inc. Identification of broadcast source associated with unknown fingerprint
US11394478B2 (en) 2011-08-30 2022-07-19 Iheartmedia Management Services, Inc. Cloud callout identification of unknown broadcast signatures based on previously recorded broadcast signatures
US10530507B2 (en) 2011-08-30 2020-01-07 Iheartmedia Management Services, Inc. Identification of broadcast source associated with unknown fingerprint
US9860000B2 (en) 2011-08-30 2018-01-02 Iheartmedia Management Services, Inc. Identification of changed broadcast media items
US10461870B2 (en) 2011-08-30 2019-10-29 Iheartmedia Management Services, Inc. Parallel identification of media source
US11575454B2 (en) 2011-08-30 2023-02-07 Iheartmedia Management Services, Inc. Automated data-matching based on fingerprints
US8639178B2 (en) 2011-08-30 2014-01-28 Clear Channel Management Sevices, Inc. Broadcast source identification based on matching broadcast signal fingerprints
US8861937B2 (en) 2011-08-31 2014-10-14 The Nielsen Company (Us), Llc Methods and apparatus to access media
US9400984B2 (en) 2011-08-31 2016-07-26 The Nielsen Company (Us), Llc Methods and apparatus to access media
US9779426B2 (en) 2011-08-31 2017-10-03 The Nielsen Company (Us), Llc Methods and apparatus to access media
US10860691B2 (en) * 2011-11-04 2020-12-08 Media Chain LLC Digital media reproduction and licensing
US20180307808A1 (en) * 2011-11-04 2018-10-25 Christopher A. Estes Digital media reproduction and licensing
US10885154B2 (en) * 2011-11-04 2021-01-05 Media Chain, Llc Digital media reproduction and licensing
US11210371B1 (en) * 2011-11-04 2021-12-28 Media Chain, Llc Digital media reproduction and licensing
US10650120B2 (en) * 2011-11-04 2020-05-12 Media Chain, Llc Digital media reproduction and licensing
US10657226B2 (en) * 2011-11-04 2020-05-19 Media Chain, Llc Digital media reproduction and licensing
US11210370B1 (en) * 2011-11-04 2021-12-28 Media Chain, Llc Digital media reproduction and licensing
US10242378B1 (en) * 2012-02-24 2019-03-26 Google Llc Incentive-based check-in
US9280599B1 (en) 2012-02-24 2016-03-08 Google Inc. Interface for real-time audio recognition
US9208225B1 (en) * 2012-02-24 2015-12-08 Google Inc. Incentive-based check-in
US9167520B2 (en) 2012-03-20 2015-10-20 Qualcomm Incorporated Controlling applications in a mobile device based on environmental context
US20140336798A1 (en) * 2012-05-13 2014-11-13 Harry E. Emerson, III Discovery of music artist and title for syndicated content played by radio stations
US9418669B2 (en) * 2012-05-13 2016-08-16 Harry E. Emerson, III Discovery of music artist and title for syndicated content played by radio stations
US10996931B1 (en) 2012-07-23 2021-05-04 Soundhound, Inc. Integrated programming framework for speech and text understanding with block and statement structure
US10957310B1 (en) 2012-07-23 2021-03-23 Soundhound, Inc. Integrated programming framework for speech and text understanding with meaning parsing
US11776533B2 (en) 2012-07-23 2023-10-03 Soundhound, Inc. Building a natural language understanding application using a received electronic record containing programming code including an interpret-block, an interpret-statement, a pattern expression and an action statement
US20150222950A1 (en) * 2012-08-21 2015-08-06 Omnifone Ltd. Method of identifying media content
US9576576B2 (en) * 2012-09-10 2017-02-21 Google Inc. Answering questions using environmental context
US9786279B2 (en) 2012-09-10 2017-10-10 Google Inc. Answering questions using environmental context
US20160343371A1 (en) * 2012-09-10 2016-11-24 Google Inc. Answering questions using environmental context
US20140122302A1 (en) * 2012-11-01 2014-05-01 At&T Mobility Ii Llc Customized Data Delivery
US9027048B2 (en) * 2012-11-14 2015-05-05 Bank Of America Corporation Automatic deal or promotion offering based on audio cues
US20140137139A1 (en) * 2012-11-14 2014-05-15 Bank Of America Automatic Deal Or Promotion Offering Based on Audio Cues
US10366419B2 (en) 2012-11-27 2019-07-30 Roland Storti Enhanced digital media platform with user control of application data thereon
US10339936B2 (en) 2012-11-27 2019-07-02 Roland Storti Method, device and system of encoding a digital interactive response action in an analog broadcasting message
US20140161263A1 (en) * 2012-12-10 2014-06-12 Microsoft Corporation Facilitating recognition of real-time content
US9301070B2 (en) 2013-03-11 2016-03-29 Arris Enterprises, Inc. Signature matching of corrupted audio signal
US9307337B2 (en) 2013-03-11 2016-04-05 Arris Enterprises, Inc. Systems and methods for interactive broadcast content
US9451048B2 (en) 2013-03-12 2016-09-20 Shazam Investments Ltd. Methods and systems for identifying information of a broadcast station and information of broadcasted content
US10546352B2 (en) 2013-03-14 2020-01-28 Facebook, Inc. Method for selectively advertising items in an image
US11227343B2 (en) 2013-03-14 2022-01-18 Facebook, Inc. Method for selectively advertising items in an image
US10521830B2 (en) * 2013-03-14 2019-12-31 Facebook, Inc. Method for displaying a product-related image to a user while shopping
US9390170B2 (en) 2013-03-15 2016-07-12 Shazam Investments Ltd. Methods and systems for arranging and searching a database of media content recordings
US9773058B2 (en) 2013-03-15 2017-09-26 Shazam Investments Ltd. Methods and systems for arranging and searching a database of media content recordings
US10540395B2 (en) 2013-05-06 2020-01-21 Iheartmedia Management Services, Inc. Unordered matching of audio fingerprints
US9460201B2 (en) 2013-05-06 2016-10-04 Iheartmedia Management Services, Inc. Unordered matching of audio fingerprints
US10146866B2 (en) 2013-05-06 2018-12-04 Iheartmedia Management Services, Inc. Unordered matching of audio fingerprints
US11328011B2 (en) 2013-05-06 2022-05-10 Iheartmedia Management Services, Inc. Unordered matching of audio fingerprints
US10810254B2 (en) 2013-05-06 2020-10-20 Iheartmedia Management Services, Inc. Unordered matching of audio fingerprints
US10459973B2 (en) 2013-05-06 2019-10-29 Iheartmedia Management Services, Inc. Unordered matching of audio fingerprints
US11630859B2 (en) 2013-05-06 2023-04-18 Iheartmedia Management Services, Inc. System for matching media files
US20140336797A1 (en) * 2013-05-12 2014-11-13 Harry E. Emerson, III Audio content monitoring and identification of broadcast radio stations
US9507849B2 (en) 2013-11-28 2016-11-29 Soundhound, Inc. Method for combining a query and a communication command in a natural language computer system
US9292488B2 (en) 2014-02-01 2016-03-22 Soundhound, Inc. Method for embedding voice mail in a spoken utterance using a natural language processing computer system
US9601114B2 (en) 2014-02-01 2017-03-21 Soundhound, Inc. Method for embedding voice mail in a spoken utterance using a natural language processing computer system
US11049094B2 (en) 2014-02-11 2021-06-29 Digimarc Corporation Methods and arrangements for device to device communication
US11295730B1 (en) 2014-02-27 2022-04-05 Soundhound, Inc. Using phonetic variants in a local context to improve natural language understanding
US11030993B2 (en) 2014-05-12 2021-06-08 Soundhound, Inc. Advertisement selection by linguistic classification
US10311858B1 (en) 2014-05-12 2019-06-04 Soundhound, Inc. Method and system for building an integrated user profile
US9564123B1 (en) 2014-05-12 2017-02-07 Soundhound, Inc. Method and system for building an integrated user profile
US9942711B2 (en) 2014-05-16 2018-04-10 Alphonso Inc. Apparatus and method for determining co-location of services using a device that generates an audio signal
US10575126B2 (en) 2014-05-16 2020-02-25 Alphonso Inc. Apparatus and method for determining audio and/or visual time shift
US9698924B2 (en) * 2014-05-16 2017-07-04 Alphonso Inc. Efficient apparatus and method for audio signature generation using recognition history
US9583121B2 (en) 2014-05-16 2017-02-28 Alphonso Inc. Apparatus and method for determining co-location of services
US10278017B2 (en) 2014-05-16 2019-04-30 Alphonso, Inc Efficient apparatus and method for audio signature generation using recognition history
US9590755B2 (en) 2014-05-16 2017-03-07 Alphonso Inc. Efficient apparatus and method for audio signature generation using audio threshold
US20160336025A1 (en) * 2014-05-16 2016-11-17 Alphonso Inc. Efficient apparatus and method for audio signature generation using recognition history
US9584236B2 (en) 2014-05-16 2017-02-28 Alphonso Inc. Efficient apparatus and method for audio signature generation using motion
US9641980B2 (en) 2014-05-16 2017-05-02 Alphonso Inc. Apparatus and method for determining co-location of services using a device that generates an audio signal
US11586938B2 (en) 2016-11-23 2023-02-21 Carrier Corporation Building management system having knowledge base
US10972306B2 (en) 2016-11-23 2021-04-06 Carrier Corporation Building management system having event reporting
US10922720B2 (en) 2017-01-11 2021-02-16 Adobe Inc. Managing content delivery via audio cues
US11410196B2 (en) 2017-01-11 2022-08-09 Adobe Inc. Managing content delivery via audio cues
US10805693B2 (en) * 2017-09-12 2020-10-13 Irdeto B.V. Device and method for GPU-based watermarking
US20190141414A1 (en) * 2017-09-12 2019-05-09 Irdeto B.V. Device and Method for GPU-based Watermarking
US11109078B2 (en) * 2017-09-13 2021-08-31 Perfect Sense, Inc. Time-based content synchronization
US11711556B2 (en) * 2017-09-13 2023-07-25 Perfect Sense, Inc. Time-based content synchronization
US10645431B2 (en) 2017-09-13 2020-05-05 Perfect Sense, Inc. Time-based content synchronization
US10264297B1 (en) * 2017-09-13 2019-04-16 Perfect Sense, Inc. Time-based content synchronization
US11164571B2 (en) * 2017-11-16 2021-11-02 Baidu Online Network Technology (Beijing) Co., Ltd. Content recognizing method and apparatus, device, and computer storage medium
US20200092592A1 (en) * 2018-09-18 2020-03-19 Free Stream Media Corporation d/b/a Samba TV Content consensus management
US10771828B2 (en) * 2018-09-18 2020-09-08 Free Stream Media Corp. Content consensus management
US10868620B2 (en) * 2018-12-26 2020-12-15 The Nielsen Company (Us), Llc Methods and apparatus for optimizing station reference fingerprint loading using reference watermarks
US11469841B2 (en) * 2018-12-26 2022-10-11 The Nielsen Company (Us), Llc Methods and apparatus for optimizing station reference fingerprint loading using reference watermarks
US20230089158A1 (en) * 2018-12-26 2023-03-23 The Nielsen Company (Us), Llc Methods and apparatus for optimizing station reference fingerprint loading using reference watermarks
US11784737B2 (en) * 2018-12-26 2023-10-10 The Nielsen Company (Us), Llc Methods and apparatus for optimizing station reference fingerprint loading using reference watermarks
US11487815B2 (en) * 2019-06-06 2022-11-01 Sony Corporation Audio track determination based on identification of performer-of-interest at live event
US11051057B2 (en) * 2019-06-24 2021-06-29 The Nielsen Company (Us), Llc Use of steganographically-encoded time information as basis to establish a time offset, to facilitate taking content-related action
US11736746B2 (en) * 2019-06-24 2023-08-22 The Nielsen Company (Us), Llc Use of steganographically-encoded time information as basis to establish a time offset, to facilitate taking content-related action
US11470364B2 (en) * 2019-06-24 2022-10-11 The Nielsen Company (Us), Llc Use of steganographically-encoded time information as basis to establish a time offset, to facilitate taking content-related action
US20230336796A1 (en) * 2019-06-24 2023-10-19 The Nielsen Company (Us), Llc Use of Steganographically-Encoded Time Information as Basis to Establish a Time Offset, to Facilitate Taking Content-Related Action
US20220103895A1 (en) * 2019-06-24 2022-03-31 The Nielsen Company (Us), Llc Use of Steganographically-Encoded Time Information as Basis to Control Implementation of Dynamic Content Modification
US11234049B2 (en) * 2019-06-24 2022-01-25 The Nielsen Company (Us), Llc Use of steganographically-encoded time information as basis to control implementation of dynamic content modification
US20230007320A1 (en) * 2019-06-24 2023-01-05 The Nielsen Company (Us), Llc Use of Steganographically-Encoded Time Information as Basis to Establish a Time Offset, to Facilitate Taking Content-Related Action
US20230171463A1 (en) * 2019-06-24 2023-06-01 The Nielsen Company (Us), Llc Use of Steganographically-Encoded Time Information as Basis to Control Implementation of Dynamic Content Modification
US11863817B2 (en) * 2019-06-24 2024-01-02 The Nielsen Company (Us), Llc Use of steganographically-encoded time information as basis to control implementation of dynamic content modification
US11212560B2 (en) 2019-06-24 2021-12-28 The Nielsen Company (Us), Llc Use of steganographically-encoded time information as basis to establish a time offset, to facilitate taking content-related action
US11589109B2 (en) * 2019-06-24 2023-02-21 The Nielsen Company (Us), Llc Use of steganographically-encoded time information as basis to control implementation of dynamic content modification
US20220078492A1 (en) * 2019-12-13 2022-03-10 Tencent Technology (Shenzhen) Company Limited Interactive service processing method and system, device, and storage medium
US11736749B2 (en) * 2019-12-13 2023-08-22 Tencent Technology (Shenzhen) Company Limited Interactive service processing method and system, device, and storage medium
US11729441B2 (en) 2020-01-30 2023-08-15 Snap Inc. Video generation system to render frames on demand
US11831937B2 (en) * 2020-01-30 2023-11-28 Snap Inc. Video generation system to render frames on demand using a fleet of GPUS
US11284144B2 (en) * 2020-01-30 2022-03-22 Snap Inc. Video generation system to render frames on demand using a fleet of GPUs
US11356720B2 (en) 2020-01-30 2022-06-07 Snap Inc. Video generation system to render frames on demand
US20230088471A1 (en) * 2020-01-30 2023-03-23 Snap Inc. Video generation system to render frames on demand using a fleet of gpus
US11263254B2 (en) 2020-01-30 2022-03-01 Snap Inc. Video generation system to render frames on demand using a fleet of servers
US11651539B2 (en) 2020-01-30 2023-05-16 Snap Inc. System for generating media content items on demand
US11651022B2 (en) 2020-01-30 2023-05-16 Snap Inc. Video generation system to render frames on demand using a fleet of servers
US11036781B1 (en) 2020-01-30 2021-06-15 Snap Inc. Video generation system to render frames on demand using a fleet of servers
US20220021948A1 (en) * 2020-07-17 2022-01-20 Playrcart Limited Media player
US11595736B2 (en) * 2020-07-17 2023-02-28 Playrcart Limited Media player
US11877038B2 (en) 2020-07-17 2024-01-16 Playrcart Limited Media player
US11284139B1 (en) * 2020-09-10 2022-03-22 Hulu, LLC Stateless re-discovery of identity using watermarking of a video stream
US11589100B1 (en) * 2021-03-31 2023-02-21 Amazon Technologies, Inc. On-demand issuance private keys for encrypted video transmission
US11849167B1 (en) * 2021-03-31 2023-12-19 Amazon Technologies, Inc. Video encoding device for use with on-demand issuance private keys
US11496777B1 (en) * 2021-07-19 2022-11-08 Intrado Corporation Database layer caching for video communications
US11496776B1 (en) 2021-07-19 2022-11-08 Intrado Corporation Database layer caching for video communications
US11496318B1 (en) 2021-07-19 2022-11-08 Intrado Corporation Database layer caching for video communications
US20230015758A1 (en) * 2021-07-19 2023-01-19 Intrado Corporation Database layer caching for video communications
US20230020715A1 (en) * 2021-07-19 2023-01-19 Intrado Corporation Database layer caching for video communications
WO2024016901A1 (en) * 2022-07-19 2024-01-25 北京字跳网络技术有限公司 Lyric-based information prompting method and apparatus, device, medium and product

Also Published As

Publication number Publication date
US20140214532A1 (en) 2014-07-31
US8015123B2 (en) 2011-09-06
US8688600B2 (en) 2014-04-01
WO2002061652A8 (en) 2003-02-13
WO2002061652A2 (en) 2002-08-08
US20090012849A1 (en) 2009-01-08
US20050267817A1 (en) 2005-12-01
US20110314132A1 (en) 2011-12-22
US9721287B2 (en) 2017-08-01

Similar Documents

Publication Publication Date Title
US9721287B2 (en) Method and system for interacting with a user in an experimental environment
US8725829B2 (en) Method and system for identifying sound signals
US11671193B2 (en) Distributed audience measurement systems and methods
JP6060155B2 (en) Method and system for performing a comparison of received data and providing subsequent services based on the comparison
CN102959544B (en) For the method and system of synchronized multimedia
US7174293B2 (en) Audio identification system and method
US20080235018A1 (en) Method and System for Determing the Topic of a Conversation and Locating and Presenting Related Content
US20100131558A1 (en) System and method for managing copyright licensing
US20110069937A1 (en) Apparatus, system and method for identifying advertisements from a broadcast source and providing functionality relating to the same
CN105659230A (en) Query response using media consumption history
US11120077B2 (en) Electronic media signature based applications
CN101361301A (en) Detecting repeating content in broadcast media
CN110083714A (en) Acquisition, recovery and the matching to the peculiar information from media file-based for autofile detection
TW201419194A (en) Coupon management system, terminal device and program
US20050216512A1 (en) Method of accessing a work of art, a product, or other tangible or intangible objects without knowing the title or name thereof using fractional sampling of the work of art or object
CN108140056A (en) Media program moment guide
WO2020176057A1 (en) Music analysis system and method for public spaces
US11830043B2 (en) Apparatus, system, and method for audio based browser cookies
KR20040110291A (en) Method and System of Music Information Identification
Selvakumar et al. Content recognition using audio finger printing
EP1281128B1 (en) Mobile personal digital assistant

Legal Events

Date Code Title Description
AS Assignment

Owner name: SHAZAM ENTERTAINMENT LTD., GREAT BRITAIN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BARTON, CHRISTOPHER JACQUES PENROSE;INGHELBRECHT, PHILIP;MUKHERJEE, DHEERAJ SHANKAR;AND OTHERS;REEL/FRAME:011574/0188;SIGNING DATES FROM 20010212 TO 20010213

AS Assignment

Owner name: LANDMARK DIGITAL SERVICES LLC, TEXAS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SHAZAM ENTERTAINMENT LIMITED;REEL/FRAME:016546/0733

Effective date: 20050826

AS Assignment

Owner name: LANDMARK DIGITAL SERVICES LLC, TENNESSEE

Free format text: CORRECTIVE ASSIGNMENT TO RE-RECORD ASSIGNMENT PREVIOUSLY RECORDED UNDER REEL AND FRAME 0165;ASSIGNOR:SHAZAM ENTERTAINMENT LIMITED;REEL/FRAME:016551/0062

Effective date: 20050826

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION

AS Assignment

Owner name: SHAZAM INVESTMENTS LIMITED, UNITED KINGDOM

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:LANDMARK DIGITAL SERVICES LLC;REEL/FRAME:027274/0799

Effective date: 20111121

AS Assignment

Owner name: APPLE INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SHAZAM INVESTMENTS LIMITED;REEL/FRAME:053689/0704

Effective date: 20200507