EP1784711A4 - Detecting and measuring exposure to media content items - Google Patents

Detecting and measuring exposure to media content items

Info

Publication number
EP1784711A4
EP1784711A4 EP05793408A EP05793408A EP1784711A4 EP 1784711 A4 EP1784711 A4 EP 1784711A4 EP 05793408 A EP05793408 A EP 05793408A EP 05793408 A EP05793408 A EP 05793408A EP 1784711 A4 EP1784711 A4 EP 1784711A4
Authority
EP
European Patent Office
Prior art keywords
media item
signature
target
target media
server
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
EP05793408A
Other languages
German (de)
French (fr)
Other versions
EP1784711A2 (en
Inventor
Tom Zito
Mark D Klein
Allan E Alcorn
Gary S Fletcher Jr
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Astro West LLC
Original Assignee
Integrated Media Measurement Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US11/216,543 external-priority patent/US7623823B2/en
Application filed by Integrated Media Measurement Inc filed Critical Integrated Media Measurement Inc
Publication of EP1784711A2 publication Critical patent/EP1784711A2/en
Publication of EP1784711A4 publication Critical patent/EP1784711A4/en
Ceased legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/02Marketing; Price estimation or determination; Fundraising
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/60Information retrieval; Database structures therefor; File system structures therefor of audio data
    • G06F16/63Querying
    • G06F16/632Query formulation
    • G06F16/634Query by example, e.g. query by humming
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/60Information retrieval; Database structures therefor; File system structures therefor of audio data
    • G06F16/68Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/60Information retrieval; Database structures therefor; File system structures therefor of audio data
    • G06F16/68Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/683Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04HBROADCAST COMMUNICATION
    • H04H60/00Arrangements for broadcast applications with a direct linking to broadcast information or broadcast space-time; Broadcast-related systems
    • H04H60/56Arrangements characterised by components specially adapted for monitoring, identification or recognition covered by groups H04H60/29-H04H60/54
    • H04H60/58Arrangements characterised by components specially adapted for monitoring, identification or recognition covered by groups H04H60/29-H04H60/54 of audio
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04HBROADCAST COMMUNICATION
    • H04H60/00Arrangements for broadcast applications with a direct linking to broadcast information or broadcast space-time; Broadcast-related systems
    • H04H60/61Arrangements for services using the result of monitoring, identification or recognition covered by groups H04H60/29-H04H60/54
    • H04H60/63Arrangements for services using the result of monitoring, identification or recognition covered by groups H04H60/29-H04H60/54 for services of sales
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/24Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
    • H04N21/2407Monitoring of transmitted content, e.g. distribution time, number of downloads
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/266Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
    • H04N21/26603Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel for automatically generating descriptors from content, e.g. when it is not made available by its provider, using content analysis techniques
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81Monomedia components thereof
    • H04N21/812Monomedia components thereof involving advertisement data
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04HBROADCAST COMMUNICATION
    • H04H2201/00Aspects of broadcast communication
    • H04H2201/90Aspects of broadcast communication characterised by the use of signatures

Definitions

  • the present invention relates generally to tracking of media consump ⁇ tion, and more particularly to detecting and measuring exposure to media items without requiring user action or knowledge.
  • the system of the present invention detects and identifies user expo ⁇ sure to audible media.
  • a mobile client device carried by a user digitally samples the audio environment of the consumer on a regular basis. These samples are transformed into a stream of data signatures and transmitted to a network opera ⁇ tions center (“NOC").
  • NOC network opera ⁇ tions center
  • the user's exposure to media content items is tracked and recorded, and can later be aggregated with information about other users so as to provide effective statistical reporting as to overall media exposure and consumption.
  • the system can also record specific characteristics of the consumer and of the cir ⁇ cumstances of his or her exposure to the media content item. For example, the loca ⁇ tion at which the media was encountered, the activity the user was engaging in, and the like.
  • Media exposure information can also be correlated with purchase informa ⁇ tion so as to make inferences as to the relationship between advertisements and pur ⁇ chasing behavior.
  • MCD 101 may be built into a consumer device with some other utility to the user; examples include a mobile phone, PDA, wristwatch, or the like. In alter ⁇ native embodiments, MCD 101 can take any other form, such as a standalone device that is carried by or attached to the user.
  • Embedding the functionality of the present invention in a device such as a mobile phone or wristwatch makes it more conven ⁇ ient for a user to carry MCD 101, and also encourages the user to keep MCD 101 in their possession at all times.
  • MCD 101 operates passively and requires no user in ⁇ put.
  • Fig. 1 is a block diagram depicting an overall architecture for imple ⁇ menting the present invention according to one embodiment.
  • Fig s - 2A through 2C are block diagrams depicting alternative method ⁇ ologies for practicing the present invention.
  • Fig. 3 is a block diagram depicting a conceptual architecture for im ⁇ plementing the present invention according to one embodiment.
  • Fig.4 is a flowchart depicting a method for practicing the present in ⁇ vention according to one embodiment.
  • Fig. 1 there is shown a block diagram depicting an overall architecture for implementing the present invention according to one em ⁇ bodiment.
  • Fig. 3 there is shown a block diagram depicting a con ⁇ ceptual architecture for implementing the present invention according to one em ⁇ bodiment.
  • Fig. 4 there is shown a flowchart depicting a method for implementing the present invention according to one embodiment.
  • media receiv ⁇ ers 111 receive broadcast audio; this audio (or a sample of it) is recorded 401 and transformed into signatures 402. These signatures are stored 403 along with appro ⁇ priate indexing mechanisms to facilitate retrieval and comparison.
  • Mobile client de ⁇ vices 101 detect user exposure to media content sources 102, for example by picking up audio at a microphone of a cell phone. This audio is recorded 404 and trans ⁇ formed into signatures 405. These signatures are also stored 406.
  • NOC Network Op ⁇ erations Center
  • the MCD signatures are compared 407 with signatures derived from the broadcast audio, so as to detect and identify media items to which the user has been exposed.
  • the present invention is able to deter ⁇ mine, with great specificity, which media items a user has been exposed to, and the particulars of such exposure (including number of repetitions, location of exposure, correlation to buying behavior, and the like). GPS or other location data can also be used in analyzing the media exposure.
  • the present invention thus provides a mechanism for determining degrees of penetration and effectiveness for media content items such as advertisements.
  • the system of the present invention is able to detect exposure at any location, include within the home and outside the home.
  • Media content source 102 is any source to which a user may be ex ⁇ posed. Examples include television, radio, CDs, movies, public address announce ⁇ ments, and the like. According to the techniques described herein, the present in ⁇ vention tracks user exposure to various media content items that may come from any number of sources 102.
  • Mobile client device (MCD) 101 is a device capable of detecting and re ⁇ ceiving audio from source 102. Any number of MCDs 101 can be provided; for ex ⁇ ample, in one embodiment each user being tracked has an MCD 101.
  • each MCD 101 is a device (or a component of a device) carried by a user (consumer).
  • MCD 101 may be built into a consumer device with some other utility to the user, such as a mobile phone, personal digital assistant (PD A), wristwatch, handheld computer, or the like.
  • MCD 101 includes a GPRS platform for transmitting data, and runs an operating system such as Micro ⁇ soft Windows Mobile or J2ME.
  • MCD 101 can take any other form, such as a standalone device that is carried by or attached to the user. Embedding the functionality of the present invention in a device such as a mobile phone or wristwatch makes it more convenient for a user to carry MCD 101, and also encourages the user to keep MCD 101 in his or her possession at all times. [0029] Although the description provided herein makes use of the term
  • MCD it will be recognized by one skilled in the art that the detection devices need not be mobile; in other words, the present invention can be implemented using sta ⁇ tionary devices that perform essentially the same function as described herein.
  • MCD 101 makes use of already-present compo ⁇ nents (such as a microphone in a cell phone) to implement the operations described herein.
  • MCD 101 operates passively and requires no user input. In one em ⁇ bodiment, MCD 101 digitally samples the audio environment of the user on a regu ⁇ lar basis; in another embodiment, MCD 101 performs such sampling when it detects that meaningful audio has been received at MCD 101. MCD 101 transforms the au ⁇ dio samples to a data signature stream that can be digitally transmitted and/ or stored.
  • the audio content items received by MCD 101 are referred to herein as target media items.
  • signature transformation is performed at MCD
  • the MCD 101 samples 10 seconds of audio data per 30 seconds received. Such a ratio is particularly effective for detect ⁇ ing exposure to commercials (advertisements), since many such commercials are 30 seconds long.
  • MCD 101 creates a raw audio file (such as a .WAV file) from the sam ⁇ pled data, and performs a signature transformation to generate a signature file from the raw audio file.
  • the system of the present invention uses a signature transformation algorithm such as Shazam, described in Wang et al. and available from Shazam Entertainment Ltd., of London, England.
  • the signature transformation algorithm generates a 4k file that is spooled (temporar ⁇ ily stored) at MCD 101.
  • MCD 101 erases the raw audio file once the signature file has been created; in another embodiment, raw audio is saved for some period of time for testing purposes.
  • the system of the present invention uses a signature transformation algorithm that meets design constraints of MCD 101 (processing power, battery life, available memory) and the transmission channel (bandwidth, availability, and the like). For example, the system may be configured to skip or postpone uploads if the network is unavailable. The transformation algorithm retains enough data to facilitate high- accuracy determinations of what media the user is listening to.
  • MCD 101 places itself into a quiescent (“sleeping") state to lessen battery drain when the detected audio level drops below a threshold. During this sleeping state, MCD 101 periodically "wakes up” and determines if the audio level is sufficient to resume sampling.
  • MCD 101 continually samples for N seconds every M seconds (such as 10 seconds every 30 seconds) and then processes the audio content to make a judgment via frequency analysis/ power levels as to its useful au ⁇ dio content before passing it on to the audio fingerprint process. If the audio content is judged to be not sufficient relevant given current power levels, then the sample is discarded so as to save processing and transmission time and thus conserving bat ⁇ tery usage.
  • MCD 101 does not do any transforming; rather it merely sends raw data.
  • MCD 101 transmits data to data signature stream collection server(s)
  • this transmis ⁇ sion takes place via wireless data service provider 103 which operates communica ⁇ tion towers 104 that receive signals from MCD 101 and relay the sampled audio data via the Internet 105 to one or more data signature stream collection servers 107 run ⁇ ning at NOC 106.
  • this data transmission can take place using any known wireless (or wired) communication method, and that such transmission can take place in real-time, or in a batched mode at periodic intervals, or in response to NOC 106 queries to MCD 101.
  • such transmission can take place via GPRS, TCP/IP, or other mechanisms, or any combi ⁇ nation thereof.
  • the user "docks" or connects MCD 101 to a computer or other device (not shown) in order to transmit data to NOC 106.
  • signature files are transmitted to data signature stream collection server (s) 107 on a periodic basis (for example, every five minutes). However, if a connection cannot be made, or if power at MCD 101 is low, transfers of signature files can be delayed as long as is necessary.
  • individ ⁇ ual MCDs 101 are capable of spooling (temporarily storing) some quantity of signa ⁇ ture data so as to account for temporary inability to transmit to server(s) 107.
  • MCD 101 can temporarily stop col ⁇ lecting data. Alternatively, MCD 101 can discard old data in favor of new data. In one embodiment, once data has been transferred to server 107, MCD 101 clears its local storage (spool) in order to make room for new data.
  • MCD 101 does not perform any transfers of target media items when its battery power is less than some threshold amount such as 50%, unless a) it is being charged; or b) its spool space is close to being full. In one em ⁇ bodiment, MCD 101 stops receiving and sampling data from media content sources 102 when its battery power is less than some second threshold amount such as 10%.
  • a single data signature stream collection server 107 is used. In another embodiment, a plurality of servers 107 are used, and transmis ⁇ sions of data from MCDs 101 are directed to an appropriate server 107 for receipt, based on current load, geographic location, and/ or other factors.
  • data signature stream collection server 107 receives data from MCD 101 and stores it in data signature stream store 114 (also referred to as a dynamic database).
  • Correlator server 115 correlates the data signature stream against a set of data signature streams transformed from candidate media sources 113 to determine which candidate media source, if any, the user is listening to at any given time.
  • correlator server 115 uses a correlation algorithm as described in Avery Li-chun Wang, "An Industrial-Strength Audio Search Algo ⁇ rithm,” October 2003, and Avery Li-Chun Wang and Julius O.
  • media receivers monitor media sources for broadcast candidate media content items 121 (also referred to as reference media items).
  • Each media receiver 111 can be implemented, for example, as a personal computer (also referred to as a media monitor) with a number of tuner cards that can pick up broadcasts.
  • each media receiver 111 includes four tuner cards, each capable of receiving AM, FM, or television audio signals.
  • An example of the type of tuner card that can be used for implementing the present invention is the ASI8712 or ASI8713 eight-tuner broadcast adapter available from AudioScience, Inc. of New Castle, Delaware.
  • media receivers 111 are pro ⁇ vided, running in different locations so as to be able to pick up different mar ⁇ kets/stations, and also to provide improved reliability and redundancy.
  • Media re ⁇ DCvers 111 can be configured, for example, to simultaneously receive 32 channels in parallel, taking audio components audio only, and to convert the received audio into digital form via sampling.
  • media receivers 111 are located in a location that is remote with respect to NOC 106 (for example, in a location suitable for receiving candidate media 121); media receivers 111 then transmit signals to NOC 106 via the Internet or by other means.
  • media receiv ⁇ ers 111 are located at NOC 106.
  • Transformation server 112 transforms detected candidate media con ⁇ tent items 121 to candidate data signature streams, which are then stored at data sig ⁇ nature stream store 114 (or at a different stream store, not shown). In one embodi ⁇ ment, only audio is transformed, although one skilled in the art will recognize that the present invention can also be used in connection with video, and that video transforms can thus be applied as well. The transformation converts the raw sam ⁇ ples into data files (referred to as signature files or signature streams) that can be compared against other signature files to determine matches. [0046] In addition, in one embodiment, transformation server 112 also trans ⁇ forms candidate media content items from non-broadcast reference media 113 such as audio CDs, video game sound tracks, movie sound tracks, and the like.
  • non-broadcast reference media 113 such as audio CDs, video game sound tracks, movie sound tracks, and the like.
  • servers 112 (or 112A) and 107 are implemented as a single server for collecting data from both MCDs 101 and media receivers 111.
  • servers 112 or 112A
  • 107 are implemented as a single server for collecting data from both MCDs 101 and media receivers 111.
  • the present invention can be implemented using separate dedicated servers for these two functions, or using a single server that per ⁇ forms both functions.
  • raw audio files are stored in addition to signature files. These can be stored at media receivers 111 or at a storage location associated with server 112 or 112A.
  • reference media signature files are broken up into fixed-time increments (such as five-minute increments) for ease of indexing, han ⁇ dling, and comparison.
  • individual signature files are stored in stream store 114, each signature file representing five minutes of data for one audio channel.
  • the raw audio files are also broken up into fixed-time increments (such as five-minute increments).
  • fixed-time divisions some other form of intelligent time-based division can be used; for example, blank areas can be detected and interpreted as indicating breaks be ⁇ tween commercials, and files can be divided up according to such commercial breaks.
  • media receivers 111 transmit data
  • server 112 or 112A (either in raw form or in signature form) to server 112 or 112A on a periodic basis.
  • Data can be transmitted, for example, in five-minute increments, so that one file is transferred in each transmission.
  • a number of files can be collected at receivers 111 and then transmitted to server 112 or 112A in batch form.
  • media receivers 111 retain raw audio files for some period of time (such as 3-5 days) and then discard them.
  • server 112 or 112A retains signature files for some period of time (such as 30 days) and then discards them.
  • the present invention is able to detect such activity and can report that the show was recorded and at what time it was watched.
  • Candidate media sources can include any type of media that has an audio component detectable by MCD 101. Examples include television (broadcast, cable, satellite, etc.), radio (broadcast, cable and satellite, etc.), recorded music (CD, mp3, etc.), videogame audio, audio track of a DVD and other media sources.
  • the present invention can detect user exposure to visual media such as billboards, for example by determining, based on a GPS reading on a user's location, that the user is driving past a billboard.
  • Such me ⁇ dia exposure events can be tracked and correlated with purchases in the same man ⁇ ner as exposure to audio media items, as described herein.
  • such exposure can be tracked along with exposure to audio media items, so as to obtain a complete overview of the effectiveness of an advertising campaign that includes billboard, radio advertisements, and the like.
  • server 112 On a periodic basis (for example, every three hours), server 112 or
  • server 112A sends signature data from stream store 114 to correlator server 115 (which may include a single server or any number of servers).
  • server 115 makes a periodic request for data from server 112 or 112A, and from data signature stream collection server(s) 107.
  • server 112 or 112A in response to the request, sends signature files representing media items collected by receivers 111, as well as data from MCDs 101 collected by data signature stream collection server(s) 107.
  • Correlator server 115 identifies user exposure to candidate media items including broadcast items and non-broadcast items. In one embodiment, time stamps stored with the data signatures in stream store 114 aid in the correlation. [0056] In one embodiment, location information is collected by location track ⁇ ing server 109 and used to assist in the correlation. For example, correlator server 115 can recognize that there is a lower probability of TV viewing while moving. Some behaviors that can be inferred using location information include: driving in a car (using speed range and route tracking against a road map), riding in a bus (using bus routes with frequent stops), visits to retail locations (using coordinates of retail establishments), presence at home, and presence at the workplace. Some locations influence the correlation algorithm. For example, radio and CDs in the database are checked before television if the user is moving; television is checked first while the user is at home.
  • purchasing track ⁇ ing server 117 can collect purchase information from sources 116 for use in assisting correlation server 115. In one embodiment, this is accomplished by tracking the use of a particular credit card that is in the possession of the user.
  • the present invention is also able to track exposure to entertainment
  • Exposure to promotional advertisements can be correlated with exposure to movies and the like.
  • the present invention can also help to determine which promotional channels are most effective in reaching users and which channels are less effective.
  • advertisements are given a unique
  • advertisements are assigned one or more attributes describing the goods or services being advertised at some level of specificity ranging from narrow (“Ford Mustang”) to more broad (“Ford”) to even more broad (“automobile”).
  • a Ford Mustang advertisement will have all three of these attributes. Attributes may also describe the target audience, such as "professional.”
  • Tallies are kept for each attribute during a sliding window of time, for example 30 days.
  • Purchase information can be acquired from the use of a credit card issued to panel members, through retailer reporting, through survey, or from other sources.
  • the tracking methods provided by the present invention facilitate measurement of the effectiveness of advertisements in attracting consumers who otherwise would purchase competing brands, as well as attracting consumers who otherwise would not buy the product or product type at all.
  • the present invention is able to measure the effect an advertisement has on consumption of brands other than the advertised brand.
  • Using the correlation between purchase behavior and media exposure may show, for example, that people exposed to Ford Mustang commercials have a higher propensity to buy Ford Thunderbirds if they are not exposed to many non- Ford automobile ads.
  • database queries can be performed to reveal causal relationships and to test advertis ⁇ ing hypotheses.
  • Bluetooth transceivers can be installed in certain loca ⁇ tions, and location tracking is performed by detection of unique Bluetooth trans ⁇ DCV codes.
  • the system of the present invention can determine that a user watches news at home and listens to music in the car, or can infer purchasing behavior such as a pat ⁇ tern where the user visits a movie theater after listening to an ad for one of the mov ⁇ ies playing at that theater.
  • correlator server 115 interacts with additional components, as shown in Fig. 3, to correlate data from MCDs 101 with data from media receivers 111.
  • a series of nodes 303 can be provided for storing signature data.
  • any number (for example 14) of 4GB nodes 303 can be provided, each running at a separate computer; alternatively, other arrange- merits and node sizes can be provided.
  • Slice factory 302 controls the distribution of data from stream store 114 among nodes 303.
  • a utility server (not shown) controls the operation of slice factory 302.
  • nodes 303 run signature matching software 301, such as for example the SongMon function of software provided by Shazam Enter ⁇ tainment Ltd. and described more fully in Wang et al.
  • this software 301 is located at correlator server 115.
  • Software 301 identifies matches, re ⁇ turning a tag indicating what media content item from media receivers 111 matches a media content item picked up by an MCD 101. Additional information, such as time codes, specifics as to which media receiver 111 and/ or channel matched, and the like, can also be returned.
  • Match data is stored, for example in matches database 305 for further analysis and reporting.
  • signature matching software 301 runs through candidate data twice.
  • a preliminary run provides quick estimates of matches and can be run on demand.
  • a production run performed at regular intervals such as once every three hours, provides more accurate matches.
  • slice factory 302 and/ or utility server run signature matching software 301 to aggregate and index signature files from stream store 114 using a hash methodology. From this processed data, a disk image containing indexed signature file data for a particular length of time (such as 18 hours) is formed; this disk image is then stored at each of nodes 303.
  • each node 303 contains an identical image with identical data so as to improve throughput by processing data at multiple nodes 303 operating in parallel with one another. In one embodiment, for example, each node 303 processes a maximum of approximately 50 MCDs so as not to become over ⁇ loaded. Correlator server 115 dispatches MCD files to available nodes 303 based on load and other factors.
  • each node processes approximately 18 hours of a large number of channels (for example 70 channels, totaling over 1,000 hours of con ⁇ tent) in rolling update fashion.
  • Target media items are compared with these refer- ence media items so as to determine whether the user has been exposed to the media represented by the reference media items.
  • the aggregation and indexing is re-run so as to delete the oldest three hours and add the newest three hours, thus maintaining a set amount of data (for example, 18 hours' worth) at all times.
  • 32 channels of data are stored.
  • a binary representation of the reference media signature data is stored in each node 303. Nodes 303 compare the signature data from MCDs 101 against this reference data set. In one embodiment, multiple nodes 303 are utilized to improve matching throughput.
  • correlator server(s) 115 sends samples from stream store 114 to an appropriate node 303 based on load and other factors.
  • the present invention is scalable in two dimensions: additional numbers of MCDs 101 can be handled by adding more nodes vertically. Greater amounts of data can be stored by adding more nodes horizontally, or by providing nodes having higher storage ca ⁇ pacities.
  • data from MCDs 101 includes additional informa ⁇ tion about the circumstances and conditions of the media exposure, including for ex ⁇ ample time, day, battery level, telephone number, GPS data, and the like; all of this information can be stored at stream store 114 and used in analysis, matching, and/ or reporting.
  • certain candidate signatures are stored in static database 304. For example, if an advertiser would like to determine how many users have been exposed to a particular advertisement, a signature for that advertisement is stored in static database 304. The signature need not be collected via media re ⁇ DCvers 111, but can be provided by direct input or via correlator servers 115. This stored signature can then be compared with signatures received from MCDs 101. In one embodiment, these candidate signatures are also passed to nodes 303 to facilitate comparison with incoming signatures from MCDs 101. In one embodiment, the con ⁇ tent in static database 304 (“static content”) is combined with the content collected as described above (“dynamic content”) to create the reference data set loaded into each node 303.
  • static content is combined with the content collected as described above (“dynamic content”) to create the reference data set loaded into each node 303.
  • static content is processed to generate a ref ⁇ erence data set; this reference data set is then loaded into a node 303 that is separate from the dynamic content for analysis of matches against the static content only.
  • these candidate signatures are not aged using a sliding window as described above for data collected from media receivers 111. In another embodiment, these candidates are aged but at a different rate than other data.
  • data in static database 304 can be com ⁇ pared against data from media receivers 111.
  • an advertiser can de ⁇ termine whether his or her advertisement ran and was detected by a media receiver 111, regardless of whether it was picked up by an MCD 101.
  • FIG. 1 there are shown additional components that can be used in generating reports on media exposure and consumption.
  • a time- based history of user exposure to media items is stored in consumer tracking data ⁇ base 118. Location information is also stored, if available.
  • Analytical reporting server 119 uses consumer tracking data from da ⁇ tabase 118 to generate reports 120. Reports can include, for example: [0082] - advertisement play rates;
  • reports are generated using standard relational- database queries. Results of these queries can be place in tabular or graphical format for presentation.
  • live media source data signatures may be discarded.
  • Figs. 2A through 2C provide examples of different con ⁇ figurations for performing these functions. These Figures also provide a description of the overall method of operation of the present invention according to various em ⁇ bodiments.
  • the term “candidate data signature stream” is equivalent to reference media items; the term “consumer data signature stream” is equivalent to target me ⁇ dia items.
  • Fig. 2A there is shown a configuration where trans ⁇ formation takes place at MCD 101 and matching takes place at NOC 106.
  • MCD 101 collects audio samples, transforms them to a signature stream, and transmits the signature stream to NOC 106.
  • NOC 106 identifies matches between signature stream received from MCD 101 and candidate streams derived from candidate media 113 and 121.
  • Fig. 2A depicts the candidate data signature data stream being created at NOC 106; however, one skilled in the art will recognize that the candidate signature data stream can instead be generated at media receivers 111 and/ or provided directly to NOC 106 from some other source.
  • Fig. 2A depicts the candidate data signature data stream being created at NOC 106; however, one skilled in the art will recognize that the candidate signature data stream can instead be generated at media receivers 111 and/ or provided directly to NOC 106 from some other source.
  • MCD 101 collects audio samples and transforms them to a signature stream.
  • NOC 106 generates candidate data signature streams from candidate media, and transmits these candidate streams to MCD 101.
  • the candidate signature data stream can instead be gen ⁇ erated at media receivers 111 and/ or provided to NOC 106 from some other source; NOC 106 then sends them to MCD 101.
  • MCD 101 identifies matches between the signature stream generated from incoming audio data and the candidate data signature streams received from NOC 106.
  • MCD 101 then sends match data to NOC 106. This configuration is particularly useful when the number of candidate audio sources is relatively small and known in advance.
  • such a configuration can be used for monitoring a user's exposure to a limited and expected set of adver- tisements.
  • this configuration reduces the amount of data that is continually transmitted from MCD 101 to NOC 106; once the set of candidates is provided to MCD 101, only match data need be transmitted, which typically requires less bandwidth than transmission of signatures from MCD 101 to NOC 106.
  • Such a configuration is also a more distributed processing paradigm that can serve to re ⁇ cute processor load at NOC 106, since NOC 106 need not perform matching opera ⁇ tions for a large number of MCDs; rather MCD 101s do their own matching. [0093] Referring now to Fig.
  • MCD 101 collects audio samples and transmits the raw data to NOC 106.
  • NOC 106 transforms the raw data to a signature stream; alternatively, the candidate signature data stream can instead be generated at media receivers 111 and/ or provided directly to NOC 106 from some other source. NOC 106 then identifies matches between the signature stream and the candidate data signature streams. This configuration reduces the processing load on MCD 101.
  • NOC 106 can optionally inform MCD
  • MCD 101 stores feature-extracted sam ⁇ ples. When MCD 101 detects (hears) an advertisement or other sought-for audio, it reports back to NOC 106 that it heard the item. In one variation of this embodiment, MCD 101 does not need to transmit any transformed audio back to NOC 106, but simply reports and identifies the item that was heard. In another variation of this embodiment, MCD 101 transmits additional information about the detected audio, such as time and place where it was detected. In yet another variation, MCD 101 transmits the transformed audio (or some subset of it) to NOC 106, so that additional information can be derived from the detected audio.
  • the invention operates at a variable sample rate depending on the amount of usage that is detected.
  • a default, lower sample rate is used when the usage pattern is continuous and/ or relatively stable.
  • a higher sample rate is used when changes in usage pattern are detected.
  • MCD 101 switches automatically between these rates in re ⁇ sponse to changing conditions. Any number of different sample rates, or continuous variation within a defined range, can be used.
  • the present invention performs audio data signa ⁇ ture transformation according to any of a number of well-known algorithms.
  • an algorithm is used that meets the processing power, memory size, battery life, and bandwidth constraints of MCD 101, and also meets a minimum accuracy requirement.
  • the audio data signature transformation algorithm finds matching au ⁇ dio streams in broadcast audio signals, known to be transmitted at a certain time, and asynchronous audio signals such as music tracks and video game sound tracks.
  • the algorithm can be based in the time-domain, based in the fre ⁇ quency-domain, or based in a hybrid of the two.
  • the audio data signature transformation algorithm correlates a consumer data signature stream (target media items) against a poten ⁇ tially large number of candidate data signature streams (reference media items). Once a match is found, it can be presumed the match continues for some period of time. In one embodiment, only the candidate data signature stream is correlated against until there is no longer a match. In other words, when a match is found, it is "locked on to" and no other candidate data signature streams are considered until the match fails.
  • the system of the present invention uses a signa ⁇ ture transformation algorithm as described in Avery Li-chun Wang, "An Industrial- Strength Audio Search Algorithm," October 2003, and Avery Li-Chun Wang and Julius O. Smith, III, WIPO publication WO0211123 A3, 7 February 2002, "Method for Search in an Audio Database.”
  • the parameters of the audio acquisition can be adjusted dynami- cally by MCD 101 and/ or by NOC 106.
  • These adjustments may be a function of loca ⁇ tion information downloaded to MCD 101 from NOC 106 in advance or in near real ⁇ time based on current location. These adjustments are performed, for example, to increase matching accuracy, minimize data transmission, minimize MCD 101 battery drain, and for other system performance optimizations. For example, if uploading of data signatures can be carried out close to real-time, and NOC 106 has "locked on" to a matching signal, MCD 101 may be instructed to lower its sampling duty cycle or to suspend sampling for some period of time.
  • the audio sampling can be continuous or periodic; in one embodiment,
  • location of MCD 101 is determined by built-in or added-in GPS, by triangulation with wireless data provider transceiver sites, by closest tower identification, by wireless data network registration (Bluetooth, WiFi / 802.11), or by other means.
  • Location data can be analyzed and compared with loca ⁇ tion databases to determine entry into a store or other location, time spent in the lo ⁇ cation, speed of travel, presence at a public venue (movie theater, concert hall, sta ⁇ dium), and other attributes that may be of commercial value when combined with media consumption.
  • location information is used to detect when a user takes a car for a test drive by tracking the entry into a car deal ⁇ ership, an average wait time, and a circuit of automobile-speed motion ending back at the dealership.
  • the present invention provides a mechanism for correlating exposure to media items with purchases. Users are given credit cards to be used for making purchases. Purchase behavior, at least to the resolution of store and amount, is available from the partner affinity card issuer. Other methods for col ⁇ lecting this data may include use of an RFID tag and/ or a barcode on MCD 101. For example, many merchants, most notably grocery stores with affinity cards, store de ⁇ tailed purchase history information about their customers. The media measurement system can access this data in order to provide purchase information to the system of the present invention.
  • a bar code or other identifying indicia can be provided on MCD 101 itself, so that the user can simply hand the device to the grocery store clerk for "swiping"; alternatively, MCD 101 can be detected via RFID or other mechanisms, and the code detected thereby can be linked to the user's affinity program record.
  • MCD 101 includes sensors to help determine if it is in the possession of a person. Sensors may include: vibration, state of the Hosting Device (on/ off, usage, key presses, etc.), temperature (to detect whether MCD 101 is being carried, since it will be proximate to or in contact with the person's body), and others. Information from these sensors is processed to assist in determining whether a user is carrying MCD 101 as needed for accurate results. If data is received at a time period when evidence indicates the device is not being carried, the received data may be discarded or treated as having a lower degree of reliability, or it may be otherwise flagged.
  • Sensors may include: vibration, state of the Hosting Device (on/ off, usage, key presses, etc.), temperature (to detect whether MCD 101 is being carried, since it will be proximate to or in contact with the person's body), and others. Information from these sensors is processed to assist in determining whether a user is carrying MCD 101 as needed for accurate results. If data is received at a time period when evidence indicates the device is
  • the present invention also relates to an apparatus for performing the operations herein.
  • This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer.
  • a computer pro ⁇ gram may be stored in a computer readable storage medium, such as, but is not lim- ited to, any type of disk including floppy disks, optical disks, CD-ROMs, and mag ⁇ netic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.
  • modules, features, attributes, methodologies, and other aspects of the invention can be implemented as software, hardware, firmware or any combina ⁇ tion of the three.
  • a component of the present invention is im ⁇ plemented as software, the component can be implemented as a standalone pro ⁇ gram, as part of a larger program, as a plurality of separate programs, as a statically or dynamically linked library, as a kernel loadable module, as a device driver, and/ or in every and any other way known now or in the future to those of skill in the art of computer programming.
  • the present invention is in no way limited to implementation in any specific operating system or environment.

Abstract

User exposure to media items is detected and measured. A mobile client device (101) carried by a user digitally samples the audio environment of the user on a regular basis. These samples are transformed into a stream of data signatures and compared with reference media items to detect user exposure to the reference items. Purchase behavior following exposure to selected media content items can be detected and tracked as well, so as to gauge effectiveness of media items such as advertisements.

Description

DETECTING AND MEASURING EXPOSURE TO MEDIA CONTENT
ITEMS
Cross-Reference to Related Applications
[0001] The present application claims priority from U.S. Provisional patent application serial no. 60/606,229, for "Measuring Media Consumption and Purchase Behavior Using Client-Device Audio Sampling, Client-Device Location Tracking, and Individual Purchase Information to Derive Marketing Effectiveness," filed Au¬ gust 31, 2004, attorney docket number 9468, and U.S. Utility patent application serial no. (not yet known), for "Detecting and Measuring Exposure to Media Content Items," filed August 30, 2005, attorney docket number 24518-10389, the disclosure of which is incorporated herein by reference.
[0002] The present application further claims priority from U.S. Provisional patent application serial no. 60/655,305, for "Personal Music Preference Determina¬ tion Based on Listening Behavior," filed February 22, 2005, attorney docket number 10056, the disclosure of which is incorporated herein by reference.
Field of the Invention
[0003] The present invention relates generally to tracking of media consump¬ tion, and more particularly to detecting and measuring exposure to media items without requiring user action or knowledge.
Background of the Invention
[0004] Producers and distributors of media content, as well as advertisers, are very interested in accurately measuring the size of their audiences. Such measure¬ ments ideally indicate the number of people that are exposed to, or that consume, the media content. There is also considerable interest in the demographics and other characteristics of media consumers. [0005] Existing audience measurement techniques overlook many types of media consumption, consumer exposure to purchased music (on CD or mp3), video games, and movies. Initial sales data is available, but usage data is not. In other words, there may be information as to how many copies of a CD were purchased, but there is no information as to how often each consumer has listened to the CD (i.e., how often the consumer was exposed to the media item). [0006] In addition, television viewing and radio listening are typically meas¬ ured independently from each other and independently of other media consump¬ tion. Thus, there is no easy way to determine overall exposure to (consumption of) a particular content item such as an advertisement across various media, nor to obtain an overview of aggregate exposure to various media items and types of media items. [0007] Another disadvantage of conventional media measurement techniques is that the effectiveness of advertising is typically measured in very broad terms. In particular, the effectiveness of advertising targeted at a particular demographic mar¬ ket segment is not easily determined, nor is it easy for advertisers to obtain quick feedback to confirm that their advertising has reached its intended target market. [0008] Yet another disadvantage of conventional media measurement tech¬ niques is that such techniques are typically associated with the playback device (such as a particular television set, radio, or the like) and not directly associated with the consumer him- or herself (the viewer of the television set, listener to the radio, or the like). In addition, conventional measurement techniques do not provide a mechanism for tracking exposure both inside the home and outside the home (for example at a bar, or a friend's house, or the like).
[0009] Yet another disadvantage of conventional media measurement tech¬ niques is that they require active participation by the consumer and can often be quite burdensome. For example, in a "Nielsen family", users must interact with the measuring device (or fill out a paper diary) to identify which family members are watching the television at any given time. Such requirements result in inaccuracy, lowered and/ or unreliable participation, skewed results, and the like. [0010] Finally, existing media measurement techniques do not correlate user exposure to media items with store visits or purchases.
[0011] What is needed, therefore, is a media measurement technique that ad¬ dresses the above-described limitations of the prior art.
[0012] What is further needed is a media measurement technique that reliably tracks user exposure to media content, regardless of how the content is delivered to the user. What is further needed is a technique that does so without unduly burden¬ ing the user and without requiring action on his part.
Summary of the Invention
[0013] The system of the present invention detects and identifies user expo¬ sure to audible media. A mobile client device ("MCD") carried by a user digitally samples the audio environment of the consumer on a regular basis. These samples are transformed into a stream of data signatures and transmitted to a network opera¬ tions center ("NOC").
[0014] At the NOC, the user's exposure to media content items is tracked and recorded, and can later be aggregated with information about other users so as to provide effective statistical reporting as to overall media exposure and consumption. The system can also record specific characteristics of the consumer and of the cir¬ cumstances of his or her exposure to the media content item. For example, the loca¬ tion at which the media was encountered, the activity the user was engaging in, and the like. Media exposure information can also be correlated with purchase informa¬ tion so as to make inferences as to the relationship between advertisements and pur¬ chasing behavior.
[0015] Using the recorded media exposure information and related tracking data, the present invention, in one embodiment, employs collaborative filtering or other cohort behavior analysis to identify fine-grained psychographic classifications of users and to measure the effectiveness of marketing on various groups. The re¬ sulting aggregated reports and interpretations facilitate better targeting of marketing resources so that they tend to reach users who have a high propensity to buy. [0016] MCD 101 may be built into a consumer device with some other utility to the user; examples include a mobile phone, PDA, wristwatch, or the like. In alter¬ native embodiments, MCD 101 can take any other form, such as a standalone device that is carried by or attached to the user. Embedding the functionality of the present invention in a device such as a mobile phone or wristwatch makes it more conven¬ ient for a user to carry MCD 101, and also encourages the user to keep MCD 101 in their possession at all times. MCD 101 operates passively and requires no user in¬ put.
Brief Description of the Drawings
[0017] The accompanying drawings illustrate several embodiments of the in¬ vention and, together with the description, serve to explain the principles of the in¬ vention.
[0018] Fig. 1 is a block diagram depicting an overall architecture for imple¬ menting the present invention according to one embodiment. [0019] Figs- 2A through 2C are block diagrams depicting alternative method¬ ologies for practicing the present invention.
[0020] Fig. 3 is a block diagram depicting a conceptual architecture for im¬ plementing the present invention according to one embodiment. [0021] Fig.4 is a flowchart depicting a method for practicing the present in¬ vention according to one embodiment.
[0022] One skilled in the art will recognize that these Figures are merely ex¬ amples of the operation of the invention according to one embodiment, and that other architectures and modes of operation can be used without departing from the essential characteristics of the invention.
Detailed Description of the Embodiments
[0023] For purposes of the following description, the terms "user", "con¬ sumer", and "individual" are synonymous and are used interchangeably. The terms "consumption" and "exposure" are also synonymous and are used interchangeably. [0024] The present invention is now described more fully with reference to the accompanying Figures, in which several embodiments of the invention are shown. The present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather these embodiments are provided so that this disclosure will be complete and will fully convey the invention to those skilled in the art.
[0025] Referring now to Fig. 1, there is shown a block diagram depicting an overall architecture for implementing the present invention according to one em¬ bodiment. Referring also to Fig. 3, there is shown a block diagram depicting a con¬ ceptual architecture for implementing the present invention according to one em¬ bodiment. Referring also to Fig. 4, there is shown a flowchart depicting a method for implementing the present invention according to one embodiment. [0026] According to one embodiment of the present invention, media receiv¬ ers 111 receive broadcast audio; this audio (or a sample of it) is recorded 401 and transformed into signatures 402. These signatures are stored 403 along with appro¬ priate indexing mechanisms to facilitate retrieval and comparison. Mobile client de¬ vices 101 detect user exposure to media content sources 102, for example by picking up audio at a microphone of a cell phone. This audio is recorded 404 and trans¬ formed into signatures 405. These signatures are also stored 406. At a Network Op¬ erations Center (NOC) 106, the MCD signatures are compared 407 with signatures derived from the broadcast audio, so as to detect and identify media items to which the user has been exposed. In this manner, the present invention is able to deter¬ mine, with great specificity, which media items a user has been exposed to, and the particulars of such exposure (including number of repetitions, location of exposure, correlation to buying behavior, and the like). GPS or other location data can also be used in analyzing the media exposure. From this correlation and analysis, reports are generated 408 and output. The present invention thus provides a mechanism for determining degrees of penetration and effectiveness for media content items such as advertisements. In addition, since users carry MCDs with them, the system of the present invention is able to detect exposure at any location, include within the home and outside the home. [0027] Media content source 102 is any source to which a user may be ex¬ posed. Examples include television, radio, CDs, movies, public address announce¬ ments, and the like. According to the techniques described herein, the present in¬ vention tracks user exposure to various media content items that may come from any number of sources 102.
[0028] Mobile client device (MCD) 101 is a device capable of detecting and re¬ ceiving audio from source 102. Any number of MCDs 101 can be provided; for ex¬ ample, in one embodiment each user being tracked has an MCD 101. In one em¬ bodiment, each MCD 101 is a device (or a component of a device) carried by a user (consumer). For example, MCD 101 may be built into a consumer device with some other utility to the user, such as a mobile phone, personal digital assistant (PD A), wristwatch, handheld computer, or the like. In one embodiment, MCD 101 includes a GPRS platform for transmitting data, and runs an operating system such as Micro¬ soft Windows Mobile or J2ME. In alternative embodiments, MCD 101 can take any other form, such as a standalone device that is carried by or attached to the user. Embedding the functionality of the present invention in a device such as a mobile phone or wristwatch makes it more convenient for a user to carry MCD 101, and also encourages the user to keep MCD 101 in his or her possession at all times. [0029] Although the description provided herein makes use of the term
"MCD", it will be recognized by one skilled in the art that the detection devices need not be mobile; in other words, the present invention can be implemented using sta¬ tionary devices that perform essentially the same function as described herein. [0030] In one embodiment, MCD 101 makes use of already-present compo¬ nents (such as a microphone in a cell phone) to implement the operations described herein.
[0031] MCD 101 operates passively and requires no user input. In one em¬ bodiment, MCD 101 digitally samples the audio environment of the user on a regu¬ lar basis; in another embodiment, MCD 101 performs such sampling when it detects that meaningful audio has been received at MCD 101. MCD 101 transforms the au¬ dio samples to a data signature stream that can be digitally transmitted and/ or stored. The audio content items received by MCD 101 are referred to herein as target media items.
[0032] In one embodiment, signature transformation is performed at MCD
101. Since the signature transformation results in a compressed representation of the audio data, performing the transformation at MCD 101 reduces the quantity of data to be transmitted to NOC 106, thus reducing bandwidth requirements and system load.
[0033] For example, in one embodiment, the MCD 101 samples 10 seconds of audio data per 30 seconds received. Such a ratio is particularly effective for detect¬ ing exposure to commercials (advertisements), since many such commercials are 30 seconds long. MCD 101 creates a raw audio file (such as a .WAV file) from the sam¬ pled data, and performs a signature transformation to generate a signature file from the raw audio file. In one embodiment, the system of the present invention uses a signature transformation algorithm such as Shazam, described in Wang et al. and available from Shazam Entertainment Ltd., of London, England. This algorithm is also described in Avery Li-chun Wang, "An Industrial-Strength Audio Search Algo¬ rithm," October 2003, and Avery Li-Chun Wang and Julius O. Smith, III, WIPO pub¬ lication WO0211123A3, 7 February 2002, "Method for Search in an Audio Database." The signature transformation algorithm generates a 4k file that is spooled (temporar¬ ily stored) at MCD 101. In one embodiment, MCD 101 erases the raw audio file once the signature file has been created; in another embodiment, raw audio is saved for some period of time for testing purposes.
[0034] In one embodiment, some information is lost during the transforma¬ tion, so that the transformation is a one-way process; raw data cannot be recon¬ structed from the transformed data signature. Transmitting transformed data, as opposed to raw data, thus provides a measure of privacy and confidentiality. [0035] Any of a number of signature algorithms can used. In one embodi¬ ment, the system of the present invention uses a signature transformation algorithm that meets design constraints of MCD 101 (processing power, battery life, available memory) and the transmission channel (bandwidth, availability, and the like). For example, the system may be configured to skip or postpone uploads if the network is unavailable. The transformation algorithm retains enough data to facilitate high- accuracy determinations of what media the user is listening to. [0036] In one embodiment, MCD 101 places itself into a quiescent ("sleeping") state to lessen battery drain when the detected audio level drops below a threshold. During this sleeping state, MCD 101 periodically "wakes up" and determines if the audio level is sufficient to resume sampling.
[0037] In another embodiment, MCD 101 continually samples for N seconds every M seconds (such as 10 seconds every 30 seconds) and then processes the audio content to make a judgment via frequency analysis/ power levels as to its useful au¬ dio content before passing it on to the audio fingerprint process. If the audio content is judged to be not sufficient relevant given current power levels, then the sample is discarded so as to save processing and transmission time and thus conserving bat¬ tery usage.
[0038] In another embodiment, MCD 101 does not do any transforming; rather it merely sends raw data.
[0039] MCD 101 transmits data to data signature stream collection server(s)
107 at Network Operations Center (NOC) 106. In one embodiment, this transmis¬ sion takes place via wireless data service provider 103 which operates communica¬ tion towers 104 that receive signals from MCD 101 and relay the sampled audio data via the Internet 105 to one or more data signature stream collection servers 107 run¬ ning at NOC 106. One skilled in the art will recognize that this data transmission can take place using any known wireless (or wired) communication method, and that such transmission can take place in real-time, or in a batched mode at periodic intervals, or in response to NOC 106 queries to MCD 101. In one embodiment, such transmission can take place via GPRS, TCP/IP, or other mechanisms, or any combi¬ nation thereof. In one embodiment, the user "docks" or connects MCD 101 to a computer or other device (not shown) in order to transmit data to NOC 106. [0040] In one embodiment, signature files are transmitted to data signature stream collection server (s) 107 on a periodic basis (for example, every five minutes). However, if a connection cannot be made, or if power at MCD 101 is low, transfers of signature files can be delayed as long as is necessary. In one embodiment, individ¬ ual MCDs 101 are capable of spooling (temporarily storing) some quantity of signa¬ ture data so as to account for temporary inability to transmit to server(s) 107. If a prolonged period of time takes place when data cannot be transferred, so that an MCD 101 cannot spool additional incoming data, MCD 101 can temporarily stop col¬ lecting data. Alternatively, MCD 101 can discard old data in favor of new data. In one embodiment, once data has been transferred to server 107, MCD 101 clears its local storage (spool) in order to make room for new data.
[0041] In one embodiment, MCD 101 does not perform any transfers of target media items when its battery power is less than some threshold amount such as 50%, unless a) it is being charged; or b) its spool space is close to being full. In one em¬ bodiment, MCD 101 stops receiving and sampling data from media content sources 102 when its battery power is less than some second threshold amount such as 10%. [0042] In one embodiment, a single data signature stream collection server 107 is used. In another embodiment, a plurality of servers 107 are used, and transmis¬ sions of data from MCDs 101 are directed to an appropriate server 107 for receipt, based on current load, geographic location, and/ or other factors. [0043] At NOC 106, data signature stream collection server 107 receives data from MCD 101 and stores it in data signature stream store 114 (also referred to as a dynamic database). Correlator server 115 correlates the data signature stream against a set of data signature streams transformed from candidate media sources 113 to determine which candidate media source, if any, the user is listening to at any given time. In one embodiment, correlator server 115 uses a correlation algorithm as described in Avery Li-chun Wang, "An Industrial-Strength Audio Search Algo¬ rithm," October 2003, and Avery Li-Chun Wang and Julius O. Smith, III, WIPO pub¬ lication WO0211123A3, 7 February 2002, "Method for Search in an Audio Database." [0044] In one embodiment, media receivers monitor media sources for broadcast candidate media content items 121 (also referred to as reference media items). Each media receiver 111 can be implemented, for example, as a personal computer (also referred to as a media monitor) with a number of tuner cards that can pick up broadcasts. In one embodiment, each media receiver 111 includes four tuner cards, each capable of receiving AM, FM, or television audio signals. An example of the type of tuner card that can be used for implementing the present invention is the ASI8712 or ASI8713 eight-tuner broadcast adapter available from AudioScience, Inc. of New Castle, Delaware. In one embodiment, several media receivers 111 are pro¬ vided, running in different locations so as to be able to pick up different mar¬ kets/stations, and also to provide improved reliability and redundancy. Media re¬ ceivers 111 can be configured, for example, to simultaneously receive 32 channels in parallel, taking audio components audio only, and to convert the received audio into digital form via sampling. In one embodiment, media receivers 111 are located in a location that is remote with respect to NOC 106 (for example, in a location suitable for receiving candidate media 121); media receivers 111 then transmit signals to NOC 106 via the Internet or by other means. In another embodiment, media receiv¬ ers 111 are located at NOC 106.
[0045] Transformation server 112 transforms detected candidate media con¬ tent items 121 to candidate data signature streams, which are then stored at data sig¬ nature stream store 114 (or at a different stream store, not shown). In one embodi¬ ment, only audio is transformed, although one skilled in the art will recognize that the present invention can also be used in connection with video, and that video transforms can thus be applied as well. The transformation converts the raw sam¬ ples into data files (referred to as signature files or signature streams) that can be compared against other signature files to determine matches. [0046] In addition, in one embodiment, transformation server 112 also trans¬ forms candidate media content items from non-broadcast reference media 113 such as audio CDs, video game sound tracks, movie sound tracks, and the like. [0047] In an alternative embodiment, as shown in Fig. 3, individual media re¬ ceivers 111 transform audio into signature files, and transmit the signature files to server 112A. [0048] In one embodiment, servers 112 (or 112A) and 107 are implemented as a single server for collecting data from both MCDs 101 and media receivers 111. One skilled in the art will recognize that the present invention can be implemented using separate dedicated servers for these two functions, or using a single server that per¬ forms both functions.
[0049] In one embodiment, raw audio files (for example in .WAV format) are stored in addition to signature files. These can be stored at media receivers 111 or at a storage location associated with server 112 or 112A.
[0050] In one embodiment, reference media signature files are broken up into fixed-time increments (such as five-minute increments) for ease of indexing, han¬ dling, and comparison. Thus, individual signature files are stored in stream store 114, each signature file representing five minutes of data for one audio channel. In one embodiment where raw audio is stored, the raw audio files are also broken up into fixed-time increments (such as five-minute increments). As an alternative to fixed-time divisions, some other form of intelligent time-based division can be used; for example, blank areas can be detected and interpreted as indicating breaks be¬ tween commercials, and files can be divided up according to such commercial breaks.
[0051] Accordingly, in one embodiment, media receivers 111 transmit data
(either in raw form or in signature form) to server 112 or 112A on a periodic basis. Data can be transmitted, for example, in five-minute increments, so that one file is transferred in each transmission. Alternatively, a number of files can be collected at receivers 111 and then transmitted to server 112 or 112A in batch form. In one em¬ bodiment, media receivers 111 retain raw audio files for some period of time (such as 3-5 days) and then discard them. In one embodiment, server 112 or 112A retains signature files for some period of time (such as 30 days) and then discards them. By retaining signature files, the present invention enables detection of user exposure to time-shifted media content items. For example, if a user is watching a television show at a time other than the broadcast time (for example, if the show was recorded on a TiVo or other video recording device), the present invention is able to detect such activity and can report that the show was recorded and at what time it was watched.
[0052] Candidate media sources can include any type of media that has an audio component detectable by MCD 101. Examples include television (broadcast, cable, satellite, etc.), radio (broadcast, cable and satellite, etc.), recorded music (CD, mp3, etc.), videogame audio, audio track of a DVD and other media sources. [0053] In an alternative embodiment, the present invention can detect user exposure to visual media such as billboards, for example by determining, based on a GPS reading on a user's location, that the user is driving past a billboard. Such me¬ dia exposure events can be tracked and correlated with purchases in the same man¬ ner as exposure to audio media items, as described herein. In one embodiment, such exposure can be tracked along with exposure to audio media items, so as to obtain a complete overview of the effectiveness of an advertising campaign that includes billboard, radio advertisements, and the like.
[0054] On a periodic basis (for example, every three hours), server 112 or
112A sends signature data from stream store 114 to correlator server 115 (which may include a single server or any number of servers). In one embodiment, server 115 makes a periodic request for data from server 112 or 112A, and from data signature stream collection server(s) 107. In one embodiment, in response to the request, server 112 or 112A sends signature files representing media items collected by receivers 111, as well as data from MCDs 101 collected by data signature stream collection server(s) 107.
[0055] Correlator server 115 identifies user exposure to candidate media items including broadcast items and non-broadcast items. In one embodiment, time stamps stored with the data signatures in stream store 114 aid in the correlation. [0056] In one embodiment, location information is collected by location track¬ ing server 109 and used to assist in the correlation. For example, correlator server 115 can recognize that there is a lower probability of TV viewing while moving. Some behaviors that can be inferred using location information include: driving in a car (using speed range and route tracking against a road map), riding in a bus (using bus routes with frequent stops), visits to retail locations (using coordinates of retail establishments), presence at home, and presence at the workplace. Some locations influence the correlation algorithm. For example, radio and CDs in the database are checked before television if the user is moving; television is checked first while the user is at home.
[0057] Additional useful correlations and analyses can also be performed, for example to ascertain particular listening behaviors. For example, purchasing track¬ ing server 117 can collect purchase information from sources 116 for use in assisting correlation server 115. In one embodiment, this is accomplished by tracking the use of a particular credit card that is in the possession of the user. [0058] The present invention is also able to track exposure to entertainment
(such as movies), whether such exposure takes place in a movie theater, at home, or elsewhere. Exposure to promotional advertisements can be correlated with exposure to movies and the like. The present invention can also help to determine which promotional channels are most effective in reaching users and which channels are less effective.
[0059] For example, in one embodiment, advertisements are given a unique
ID. In addition, advertisements are assigned one or more attributes describing the goods or services being advertised at some level of specificity ranging from narrow ("Ford Mustang") to more broad ("Ford") to even more broad ("automobile"). A Ford Mustang advertisement will have all three of these attributes. Attributes may also describe the target audience, such as "professional."
[0060] Tallies are kept for each attribute during a sliding window of time, for example 30 days. Purchase information can be acquired from the use of a credit card issued to panel members, through retailer reporting, through survey, or from other sources.
[0061] When a user makes a purchase of an item being tracked, related tallies for that user over some period of time (such as the past 30 days) is examined. With a sufficiently large set of users, correlations are made between purchase behavior and media exposure. This is done, for example, by comparing the media exposure of the purchaser of product A with the media exposure of the purchaser of competing product B.
[0062] The tracking methods provided by the present invention facilitate measurement of the effectiveness of advertisements in attracting consumers who otherwise would purchase competing brands, as well as attracting consumers who otherwise would not buy the product or product type at all. In addition, the present invention is able to measure the effect an advertisement has on consumption of brands other than the advertised brand.
[0063] Using the correlation between purchase behavior and media exposure may show, for example, that people exposed to Ford Mustang commercials have a higher propensity to buy Ford Thunderbirds if they are not exposed to many non- Ford automobile ads.
[0064] Given the raw data of purchase behavior and attribute exposure tallies, database queries can be performed to reveal causal relationships and to test advertis¬ ing hypotheses.
[0065] Other sources of data that can be used and stored include an RFID tag,
GPS tracking information, and/ or a barcode on MCD 101, so as to assist in location tracking. In one embodiment, Bluetooth transceivers can be installed in certain loca¬ tions, and location tracking is performed by detection of unique Bluetooth trans¬ ceiver codes.
[0066] From these various types of data, patterns can be deduced. For exam¬ ple, the system of the present invention can determine that a user watches news at home and listens to music in the car, or can infer purchasing behavior such as a pat¬ tern where the user visits a movie theater after listening to an ad for one of the mov¬ ies playing at that theater.
[0067] In one embodiment, correlator server 115 interacts with additional components, as shown in Fig. 3, to correlate data from MCDs 101 with data from media receivers 111. For example, a series of nodes 303 can be provided for storing signature data. In one embodiment, any number (for example 14) of 4GB nodes 303 can be provided, each running at a separate computer; alternatively, other arrange- merits and node sizes can be provided. Slice factory 302 controls the distribution of data from stream store 114 among nodes 303. In one embodiment, a utility server (not shown) controls the operation of slice factory 302.
[0068] In one embodiment, nodes 303 run signature matching software 301, such as for example the SongMon function of software provided by Shazam Enter¬ tainment Ltd. and described more fully in Wang et al. In another embodiment, this software 301 is located at correlator server 115. Software 301 identifies matches, re¬ turning a tag indicating what media content item from media receivers 111 matches a media content item picked up by an MCD 101. Additional information, such as time codes, specifics as to which media receiver 111 and/ or channel matched, and the like, can also be returned. Match data is stored, for example in matches database 305 for further analysis and reporting.
[0069] In one embodiment, signature matching software 301 runs through candidate data twice. A preliminary run provides quick estimates of matches and can be run on demand. A production run, performed at regular intervals such as once every three hours, provides more accurate matches.
[0070] In one embodiment, slice factory 302 and/ or utility server (not shown) run signature matching software 301 to aggregate and index signature files from stream store 114 using a hash methodology. From this processed data, a disk image containing indexed signature file data for a particular length of time (such as 18 hours) is formed; this disk image is then stored at each of nodes 303. [0071] In one embodiment, each node 303 contains an identical image with identical data so as to improve throughput by processing data at multiple nodes 303 operating in parallel with one another. In one embodiment, for example, each node 303 processes a maximum of approximately 50 MCDs so as not to become over¬ loaded. Correlator server 115 dispatches MCD files to available nodes 303 based on load and other factors.
[0072] In one embodiment, each node processes approximately 18 hours of a large number of channels (for example 70 channels, totaling over 1,000 hours of con¬ tent) in rolling update fashion. Target media items are compared with these refer- ence media items so as to determine whether the user has been exposed to the media represented by the reference media items.
[0073] At periodic intervals (for example, every three hours), the aggregation and indexing is re-run so as to delete the oldest three hours and add the newest three hours, thus maintaining a set amount of data (for example, 18 hours' worth) at all times. In one embodiment, 32 channels of data are stored. [0074] In one embodiment, a binary representation of the reference media signature data is stored in each node 303. Nodes 303 compare the signature data from MCDs 101 against this reference data set. In one embodiment, multiple nodes 303 are utilized to improve matching throughput.
[0075] In one embodiment, correlator server(s) 115 sends samples from stream store 114 to an appropriate node 303 based on load and other factors. The present invention is scalable in two dimensions: additional numbers of MCDs 101 can be handled by adding more nodes vertically. Greater amounts of data can be stored by adding more nodes horizontally, or by providing nodes having higher storage ca¬ pacities.
[0076] In one embodiment, data from MCDs 101 includes additional informa¬ tion about the circumstances and conditions of the media exposure, including for ex¬ ample time, day, battery level, telephone number, GPS data, and the like; all of this information can be stored at stream store 114 and used in analysis, matching, and/ or reporting.
[0077] In one embodiment, certain candidate signatures are stored in static database 304. For example, if an advertiser would like to determine how many users have been exposed to a particular advertisement, a signature for that advertisement is stored in static database 304. The signature need not be collected via media re¬ ceivers 111, but can be provided by direct input or via correlator servers 115. This stored signature can then be compared with signatures received from MCDs 101. In one embodiment, these candidate signatures are also passed to nodes 303 to facilitate comparison with incoming signatures from MCDs 101. In one embodiment, the con¬ tent in static database 304 ("static content") is combined with the content collected as described above ("dynamic content") to create the reference data set loaded into each node 303. In another embodiment, static content is processed to generate a ref¬ erence data set; this reference data set is then loaded into a node 303 that is separate from the dynamic content for analysis of matches against the static content only. [0078] In one embodiment, these candidate signatures are not aged using a sliding window as described above for data collected from media receivers 111. In another embodiment, these candidates are aged but at a different rate than other data.
[0079] In yet another embodiment, data in static database 304 can be com¬ pared against data from media receivers 111. In this manner, an advertiser can de¬ termine whether his or her advertisement ran and was detected by a media receiver 111, regardless of whether it was picked up by an MCD 101.
[0080] Referring again to Fig. 1, there are shown additional components that can be used in generating reports on media exposure and consumption. A time- based history of user exposure to media items is stored in consumer tracking data¬ base 118. Location information is also stored, if available.
[0081] Analytical reporting server 119 uses consumer tracking data from da¬ tabase 118 to generate reports 120. Reports can include, for example: [0082] - advertisement play rates;
[0083] - program ratings;
[0084] - metrics of marketing effectiveness;
[0085] - psychographic classifications;
[0086] - and the like.
[0087] In one embodiment, reports are generated using standard relational- database queries. Results of these queries can be place in tabular or graphical format for presentation.
[0088] After all correlations are complete, live media source data signatures may be discarded.
[0089] Using the above-described techniques, the system of the present inven¬ tion is able to measure media consumption both in and out of the home. Alternate configurations
[0090] One skilled in the art will recognize that the transformation and match¬ ing steps can be performed in many different ways and at different components within the overall system. Figs. 2A through 2C provide examples of different con¬ figurations for performing these functions. These Figures also provide a description of the overall method of operation of the present invention according to various em¬ bodiments. The term "candidate data signature stream" is equivalent to reference media items; the term "consumer data signature stream" is equivalent to target me¬ dia items.
[0091] Referring now to Fig. 2A, there is shown a configuration where trans¬ formation takes place at MCD 101 and matching takes place at NOC 106. As de¬ scribed above, MCD 101 collects audio samples, transforms them to a signature stream, and transmits the signature stream to NOC 106. NOC 106 identifies matches between signature stream received from MCD 101 and candidate streams derived from candidate media 113 and 121. Although Fig. 2A depicts the candidate data signature data stream being created at NOC 106; however, one skilled in the art will recognize that the candidate signature data stream can instead be generated at media receivers 111 and/ or provided directly to NOC 106 from some other source. [0092] Referring now to Fig. 2B, there is shown a configuration where trans¬ formation and matching both take place at MCD 101. Here, MCD 101 collects audio samples and transforms them to a signature stream. NOC 106 generates candidate data signature streams from candidate media, and transmits these candidate streams to MCD 101. Alternatively, the candidate signature data stream can instead be gen¬ erated at media receivers 111 and/ or provided to NOC 106 from some other source; NOC 106 then sends them to MCD 101. MCD 101 then identifies matches between the signature stream generated from incoming audio data and the candidate data signature streams received from NOC 106. MCD 101 then sends match data to NOC 106. This configuration is particularly useful when the number of candidate audio sources is relatively small and known in advance. For example, such a configuration can be used for monitoring a user's exposure to a limited and expected set of adver- tisements. In such a situation, this configuration reduces the amount of data that is continually transmitted from MCD 101 to NOC 106; once the set of candidates is provided to MCD 101, only match data need be transmitted, which typically requires less bandwidth than transmission of signatures from MCD 101 to NOC 106. Such a configuration is also a more distributed processing paradigm that can serve to re¬ duce processor load at NOC 106, since NOC 106 need not perform matching opera¬ tions for a large number of MCDs; rather MCD 101s do their own matching. [0093] Referring now to Fig. 2C, there is shown a configuration where trans¬ formation and matching both take place at NOC 106. Here, MCD 101 collects audio samples and transmits the raw data to NOC 106. NOC 106 transforms the raw data to a signature stream; alternatively, the candidate signature data stream can instead be generated at media receivers 111 and/ or provided directly to NOC 106 from some other source. NOC 106 then identifies matches between the signature stream and the candidate data signature streams. This configuration reduces the processing load on MCD 101.
[0094] In any of these configurations, NOC 106 can optionally inform MCD
101 to only sample audio at specific time periods. In this way, MCD 101 sample pe¬ riod may be limited to commercial time periods, or other periods of interest. [0095] In an alternative embodiment, MCD 101 stores feature-extracted sam¬ ples. When MCD 101 detects (hears) an advertisement or other sought-for audio, it reports back to NOC 106 that it heard the item. In one variation of this embodiment, MCD 101 does not need to transmit any transformed audio back to NOC 106, but simply reports and identifies the item that was heard. In another variation of this embodiment, MCD 101 transmits additional information about the detected audio, such as time and place where it was detected. In yet another variation, MCD 101 transmits the transformed audio (or some subset of it) to NOC 106, so that additional information can be derived from the detected audio.
[0096] In another alternative embodiment, the invention operates at a variable sample rate depending on the amount of usage that is detected. A default, lower sample rate is used when the usage pattern is continuous and/ or relatively stable. A higher sample rate is used when changes in usage pattern are detected. In one varia¬ tion of this embodiment, MCD 101 switches automatically between these rates in re¬ sponse to changing conditions. Any number of different sample rates, or continuous variation within a defined range, can be used. Data Signature Algorithm
[0097] In one embodiment, the present invention performs audio data signa¬ ture transformation according to any of a number of well-known algorithms. Pref¬ erably, an algorithm is used that meets the processing power, memory size, battery life, and bandwidth constraints of MCD 101, and also meets a minimum accuracy requirement. The audio data signature transformation algorithm finds matching au¬ dio streams in broadcast audio signals, known to be transmitted at a certain time, and asynchronous audio signals such as music tracks and video game sound tracks. [0098] The algorithm can be based in the time-domain, based in the fre¬ quency-domain, or based in a hybrid of the two.
[0099] In one embodiment, the audio data signature transformation algorithm correlates a consumer data signature stream (target media items) against a poten¬ tially large number of candidate data signature streams (reference media items). Once a match is found, it can be presumed the match continues for some period of time. In one embodiment, only the candidate data signature stream is correlated against until there is no longer a match. In other words, when a match is found, it is "locked on to" and no other candidate data signature streams are considered until the match fails.
[0100] In one embodiment, the system of the present invention uses a signa¬ ture transformation algorithm as described in Avery Li-chun Wang, "An Industrial- Strength Audio Search Algorithm," October 2003, and Avery Li-Chun Wang and Julius O. Smith, III, WIPO publication WO0211123 A3, 7 February 2002, "Method for Search in an Audio Database."
[0101] In one embodiment, the parameters of the audio acquisition (sampling rate, sampling duty cycle, quiescent time between sampling periods, volume, filter parameters, and the like), and even the algorithm in use, can be adjusted dynami- cally by MCD 101 and/ or by NOC 106. These adjustments may be a function of loca¬ tion information downloaded to MCD 101 from NOC 106 in advance or in near real¬ time based on current location. These adjustments are performed, for example, to increase matching accuracy, minimize data transmission, minimize MCD 101 battery drain, and for other system performance optimizations. For example, if uploading of data signatures can be carried out close to real-time, and NOC 106 has "locked on" to a matching signal, MCD 101 may be instructed to lower its sampling duty cycle or to suspend sampling for some period of time.
[0102] The audio sampling can be continuous or periodic; in one embodiment,
10 seconds are sampled every 30 seconds. Location Information
[0103] In one embodiment, location of MCD 101 is determined by built-in or added-in GPS, by triangulation with wireless data provider transceiver sites, by closest tower identification, by wireless data network registration (Bluetooth, WiFi / 802.11), or by other means. Location data can be analyzed and compared with loca¬ tion databases to determine entry into a store or other location, time spent in the lo¬ cation, speed of travel, presence at a public venue (movie theater, concert hall, sta¬ dium), and other attributes that may be of commercial value when combined with media consumption. For example, in one embodiment, location information is used to detect when a user takes a car for a test drive by tracking the entry into a car deal¬ ership, an average wait time, and a circuit of automobile-speed motion ending back at the dealership. Purchase Tracking Credit Card
[0104] In one embodiment, the present invention provides a mechanism for correlating exposure to media items with purchases. Users are given credit cards to be used for making purchases. Purchase behavior, at least to the resolution of store and amount, is available from the partner affinity card issuer. Other methods for col¬ lecting this data may include use of an RFID tag and/ or a barcode on MCD 101. For example, many merchants, most notably grocery stores with affinity cards, store de¬ tailed purchase history information about their customers. The media measurement system can access this data in order to provide purchase information to the system of the present invention. In one embodiment, a bar code or other identifying indicia can be provided on MCD 101 itself, so that the user can simply hand the device to the grocery store clerk for "swiping"; alternatively, MCD 101 can be detected via RFID or other mechanisms, and the code detected thereby can be linked to the user's affinity program record. User Compliance
[0105] In one embodiment, MCD 101 includes sensors to help determine if it is in the possession of a person. Sensors may include: vibration, state of the Hosting Device (on/ off, usage, key presses, etc.), temperature (to detect whether MCD 101 is being carried, since it will be proximate to or in contact with the person's body), and others. Information from these sensors is processed to assist in determining whether a user is carrying MCD 101 as needed for accurate results. If data is received at a time period when evidence indicates the device is not being carried, the received data may be discarded or treated as having a lower degree of reliability, or it may be otherwise flagged.
[0106] In the above description, for purposes of explanation, numerous spe¬ cific details are set forth in order to provide a thorough understanding of the inven¬ tion. It will be apparent, however, to one skilled in the art that the invention can be practiced without these specific details. In other instances, structures and devices are shown in block diagram form in order to avoid obscuring the invention. [0107] In particular, one skilled in the art will recognize that other architec¬ tures and analysis and processing techniques and mechanisms may be used, and that the present invention can be implemented using mechanisms other than those described above.
[0108] Reference in the specification to "one embodiment" or "an embodi¬ ment" means that a particular feature, structure, or characteristic described in con¬ nection with the embodiment is included in at least one embodiment of the inven- tion. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment. [0109] Some portions of the detailed description are presented in terms of al¬ gorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self -consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.
[0110] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely conven¬ ient labels applied to these quantities. Unless specifically stated otherwise as appar¬ ent from the discussion, it is appreciated that throughout the description, discussions utilizing terms such as "processing" or "computing" or "calculating" or "determin¬ ing" or "displaying" or the like, refer to the action and processes of a computer sys¬ tem, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmis¬ sion or display devices.
[0111] The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer pro¬ gram may be stored in a computer readable storage medium, such as, but is not lim- ited to, any type of disk including floppy disks, optical disks, CD-ROMs, and mag¬ netic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. [0112] The algorithms and modules presented herein are not inherently re¬ lated to any particular computer or other apparatus. Various general-purpose sys¬ tems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatuses to perform the method steps. The required structure for a variety of these systems will appear from the de¬ scription below. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of pro¬ gramming languages may be used to implement the teachings of the invention as described herein. Furthermore, as will be apparent to one of ordinary skill in the relevant art, the modules, features, attributes, methodologies, and other aspects of the invention can be implemented as software, hardware, firmware or any combina¬ tion of the three. Of course, wherever a component of the present invention is im¬ plemented as software, the component can be implemented as a standalone pro¬ gram, as part of a larger program, as a plurality of separate programs, as a statically or dynamically linked library, as a kernel loadable module, as a device driver, and/ or in every and any other way known now or in the future to those of skill in the art of computer programming. Additionally, the present invention is in no way limited to implementation in any specific operating system or environment.
[0113] It will be understood by those skilled in the relevant art that the above- described implementations are merely exemplary, and many changes can be made without departing from the true spirit and scope of the present invention. Therefore, it is intended by the appended claims to cover all such changes and modifications that come within the true spirit and scope of this invention.

Claims

Claims
What is claimed is:
1. A method for detecting user exposure to media items, comprising: receiving a plurality of reference media items from at least one media source; receiving a representation of at least one target media item from at least one client device; identifying one of the reference media items as corresponding to the target media item; and generating output signaling user exposure to the identified reference media item.
2. The method of claim 1, wherein each client device comprises a mobile cli¬ ent device.
3. The method of claim 1, wherein each mobile client device is carried by a user.
4. The method of claim 1, wherein each mobile client device comprises one selected from the group consisting of: a mobile telephone; a personal digital assistant; a handheld computer; a laptop computer; a wristwatch; a portable game player; and a portable music player.
5. The method of claim 1, wherein each reference media item comprises au¬ dio content, and the target media item comprises audio content.
6. The method of claim 1, further comprising: generating a signature representing each reference media item; and generating a signature representing each target media item; and wherein identifying one of the reference media items as corresponding to the target media item comprises detecting a match between a refer¬ ence media item signature and a target media item signature.
7. The method of claim 6, wherein the steps of : generating a signature representing each reference media item is performed at a media receiver associated with a network operations center; generating a signature representing each target media item is performed at a network operations center; and identifying one of the reference media items are performed a network operations center.
8. The method of claim 1, wherein each received representation of a target media item comprises a signature representing the target media item, the method further comprising: generating a signature representing each reference media item; and and wherein identifying one of the reference media items as corresponding to the target media item comprises detecting a match between a refer¬ ence media item signature and a target media item signature.
9. The method of claim 8, wherein: each signature representing a target media item is generated at a client device; generating a signature representing each reference media item is performed at a media receiver associated with a network operations center; and identifying one of the reference media items as corresponding to the target media item is performed at a network operations center.
10. The method of claim 1, wherein each received representation of a target media item comprises a signature representing the target media item, and wherein each received representation of a reference media item comprises a signature repre¬ senting the reference media item; and wherein identifying one of the reference media items as corresponding to the target media item comprises detecting a match between a refer¬ ence media item signature and a target media item signature. 11. The method of claim 1, wherein each received representation of a refer¬ ence media item comprises a signature representing the reference media item, the method further comprising: generating a signature representing each target media item; and wherein identifying one of the reference media items as corresponding to the target media item comprises detecting a match between a refer¬ ence media item signature and a target media item signature.
12. The method of claim 11, wherein: each signature representing a reference media item is generated at a media re¬ ceiver associated with a network operations center; generating a signature representing each target media item is performed at a client device; and identifying one of the reference media items as corresponding to the target media item is performed at a client device.
13. The method of claim 12, further comprising: transmitting, from the client device to the network operations center, an indi¬ cation of at least one match between a reference media item and a target media item.
14. The method of claim 1, wherein each reference media item comprises a sample of received reference media.
15. The method of claim 1, wherein each target media item comprises a sam¬ ple of received target media.
16. The method of claim 1, wherein receiving a plurality of reference media items comprises receiving a reference media stream and sampling the reference me¬ dia stream.
17. The method of claim 1, wherein receiving a plurality of reference media items comprises receiving a reference media stream and dividing the reference me¬ dia stream into a plurality of files. 18. The method of claim 1, wherein receiving a representation of at least one target media item comprises receiving a target media stream and sampling the target media stream.
19. The method of claim 1, wherein receiving a representation of at least one target media item comprises receiving a target media stream and dividing the target media stream into a plurality of files.
20. The method of claim 1, wherein the client device comprises a microphone having a use other than recording target media items, and wherein the client device receives the at least one target media item via the microphone.
21. The method of claim 1, further comprising detecting a location for the cli¬ ent device, and wherein identifying one of the reference media items as correspond¬ ing to the target media item comprises correlating the detected location with a loca¬ tion of at least one geographically identifiable sponsor of a reference media item.
22. The method of claim 21, wherein detecting a location for the client device comprises detecting the location via at least one selected from the group consisting of:
GPS tracking; RFID tracking; Bluetooth tracking.
23. The method of claim 1, wherein receiving a plurality of reference media items comprises receiving at least one broadcast.
24. The method of claim 1, wherein receiving a plurality of reference media items comprises receiving at least one media item read from a fixed medium.
25. The method of claim 1, wherein receiving a plurality of reference media items comprises reading at least one media item from a non-broadcast source.
26. The method of claim 1, wherein receiving a plurality of reference media items comprises reading at least one broadcast, and wherein the representation of at least one target media item indicates user exposure at a time other than the time of the broadcast. 27. The method of claim 1, wherein generating output signaling user expo¬ sure to the identified reference media item comprises generating a report.
28. The method of claim 1, further comprising receiving consumer purchas¬ ing information, and wherein generating output signaling user exposure to the iden¬ tified reference media item comprises generating a report correlating user purchas¬ ing info with user exposure to the identified reference media item.
29. The method of claim 28, wherein receiving consumer purchasing informa¬ tion comprises receiving at least one indication of user use of a credit card.
30. The method of claim 1, wherein receiving a plurality of reference media items is performed by at least one media receiver.
31. The method of claim 1, wherein receiving a representation of at least one target media item from at least one client device is performed by at least one data signature stream collection server.
32. The method of claim 1, wherein receiving a representation of at least one target media item from at least one client device comprises: selecting, based on at least one of load and geographic proximity to the client device, one of a plurality of data signature stream collection servers; and receiving the representation of at least one target media item at the selected data signature stream collection server.
33. The method of claim 1, wherein the target media item comprises an ad¬ vertisement.
34. The method of claim 1, wherein at least one client device is positioned to detect audio from a media play device.
35. A method for detecting broadcast of a target media item, comprising: receiving a plurality of reference media items from at least one broadcast me¬ dia source; receiving a representation of a target media item; identifying one of the reference media items as corresponding to the target media item; and generating output signaling that the target media item has been broadcast.
36. A system for detecting user exposure to media items, comprising: a server, for receiving a plurality of reference media items from at least one media receiver; at least one stream collection server, for receiving a representation of at least one target media item from at least one client device; a stream store, coupled to the server and to the at least one stream collection server, for storing a representation of at least a subset of the re¬ ceived reference media items and for storing a representation of the at least one target media item; a correlator server, coupled to the stream store, for identifying one of the ref¬ erence media items as corresponding to the target media item; and an output device, coupled to the correlator server, for generating output sig¬ naling user exposure to the identified reference media item.
37. The system of claim 36, wherein each client device comprises a mobile cli¬ ent device.
38. The system of claim 36, wherein each mobile client device is carried by a user.
39. The system of claim 36, wherein each mobile client device comprises one selected from the group consisting of: a mobile telephone; a personal digital assistant; a handheld computer; a laptop computer; a wristwatch; a portable game player; and a portable music player.
40. The system of claim 36, wherein each reference media item comprises au¬ dio content, and the target media item comprises audio content.
41. The system of claim 36, further comprising: a transformation server, for generating a signature representing each refer¬ ence media item and for generating a signature representing each target media item; and wherein the correlator server identifies one of the reference media items as corresponding to the target media item by detecting a match be¬ tween a reference media item signature and a target media item signature.
42. The system of claim 41, wherein the transformation server and the corre¬ lator server are located at a network operations center.
43. The system of claim 36, wherein each received representation of a target media item comprises a signature representing the target media item, the system fur¬ ther comprising: a transformation server, for generating a signature representing each refer¬ ence media item; and and wherein the correlator server identifies one of the reference media items as corresponding to the target media item by detecting a match be¬ tween a reference media item signature and a target media item signature.
44. The system of claim 43, wherein: each signature representing a target media item is generated at a client device; and wherein the transformation server and the correlator server are located at a network operations center.
45. The system of claim 36, wherein each received representation of a target media item comprises a signature representing the target media item, and wherein each received representation of a reference media item comprises a signature repre¬ senting the reference media item; and wherein the correlator server identifies one of the reference media items as corresponding to the target media item by detecting a match be¬ tween a reference media item signature and a target media item signature. 46. The system of claim 36, wherein each received representation of a refer¬ ence media item comprises a signature representing the reference media item, the system further comprising: a transformation mechanism, for generating a signature representing each target media item; and wherein the correlator server identifies one of the reference media items as corresponding to the target media item by detecting a match be¬ tween a reference media item signature and a target media item signature.
47. The system of claim 36, wherein each reference media item comprises a sample of received reference media.
48. The system of claim 36, wherein each target media item comprises a sam¬ ple of received target media.
49. The system of claim 36, wherein the server receives a reference media stream and samples the reference media stream.
50. The system of claim 36, wherein the server receives a reference media stream and divides the reference media stream into a plurality of files.
51. The system of claim 36, wherein the stream collection server receives a target media stream and samples the target media stream.
52. The system of claim 36, wherein the stream collection server receives a target media stream and divides the target media stream into a plurality of files.
53. The system of claim 36, wherein each client device comprises a micro¬ phone having a use other than recording target media items, and wherein the client device receives the at least one target media item via the microphone.
54. The system of claim 36, further comprising a location tracking server for detecting a location for the client device, and wherein the correlator server correlates the detected location with a location of at least one geographically identifiable spon¬ sor of a reference media item.
55. The system of claim 54, wherein the location tracking server detects the location via at least one selected from the group consisting of: GPS tracking; RFID tracking; Bluetooth tracking.
56. The system of claim 36, wherein the plurality of reference media items comprises at least one broadcast.
57. The system of claim 36, wherein the plurality of reference media items comprises at least one media item read from a fixed medium.
58. The system of claim 36, wherein the plurality of reference media items comprises at least one media item from a non-broadcast source.
59. The system of claim 36, wherein the plurality of reference media items comprises at least one broadcast, and wherein the representation of at least one tar¬ get media item indicates user exposure at a time other than the time of the broadcast.
60. The system of claim 36, wherein the output device generates a report sig¬ naling user exposure to the identified reference media item.
61. The system of claim 36, further comprising a purchasing tracking server, for receiving consumer purchasing information, and wherein the output device gen¬ erates a report correlating user purchasing info with user exposure to the identified reference media item.
62. The system of claim 61, wherein the purchasing tracking server receives at least one indication of user use of a credit card.
63. The system of claim 36, wherein the at least one stream collection server comprises a plurality of stream collection servers, the system further comprising: a stream collection server selector, for selecting, based on at least one of load and geographic proximity to the client device, one of a plurality of data signature stream collection servers to receive the representa¬ tion of at least one target media item.
64. The system of claim 36, wherein the target media item comprises an ad¬ vertisement.
65. The system of claim 36, wherein at least one client device is positioned to detect audio from a media play device. 66. A system for detecting broadcast of a target media item, comprising: a server, for receiving a plurality of reference media items from at least one broadcast media source; at least one stream collection server, for receiving a representation of a target media item; a correlator server, for identifying one of the reference media items as corre¬ sponding to the target media item; and an output device, coupled to the correlator server, for generating output sig¬ naling that the target media item has been broadcast.
67. A system for detecting user exposure to media items, comprising: a server, for receiving a plurality of reference media items from at least one media receiver; a transformation server, for generating a signature for each received reference media item, the transformation server being located at a network operations center; a client device interface, for transmitting at least a subset of the generated sig¬ natures to at least one client device and for receiving from a client device an indication as to whether any target media items match any generated signature; and an output device, coupled to the client device interface, for generating output signaling user exposure to the identified reference media item.
EP05793408A 2004-08-31 2005-08-31 Detecting and measuring exposure to media content items Ceased EP1784711A4 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US60622904P 2004-08-31 2004-08-31
US11/216,543 US7623823B2 (en) 2004-08-31 2005-08-30 Detecting and measuring exposure to media content items
PCT/US2005/031250 WO2006026736A2 (en) 2004-08-31 2005-08-31 Detecting and measuring exposure to media content items

Publications (2)

Publication Number Publication Date
EP1784711A2 EP1784711A2 (en) 2007-05-16
EP1784711A4 true EP1784711A4 (en) 2008-10-01

Family

ID=36000743

Family Applications (1)

Application Number Title Priority Date Filing Date
EP05793408A Ceased EP1784711A4 (en) 2004-08-31 2005-08-31 Detecting and measuring exposure to media content items

Country Status (4)

Country Link
EP (1) EP1784711A4 (en)
JP (1) JP5026269B2 (en)
BR (1) BRPI0514559A (en)
WO (1) WO2006026736A2 (en)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2435367A (en) * 2006-02-15 2007-08-22 Intime Media Ltd User interacting with events in a broadcast audio stream, such a a quizz, by comparing patterns in the stream to a stored signature.
US10489795B2 (en) 2007-04-23 2019-11-26 The Nielsen Company (Us), Llc Determining relative effectiveness of media content items
US8302120B2 (en) 2008-02-19 2012-10-30 The Nielsen Company (Us), Llc Methods and apparatus to monitor advertisement exposure
US20110185382A2 (en) * 2008-10-07 2011-07-28 Google Inc. Generating reach and frequency data for television advertisements
GB2465141B (en) 2008-10-31 2014-01-22 Media Instr Sa Simulcast resolution in content matching systems
KR101799443B1 (en) * 2011-05-02 2017-11-20 삼성전자주식회사 Method for surveying watching of video content, Broadcasting receiving apparatus and Server thereof
US8352981B1 (en) 2011-12-01 2013-01-08 Google Inc. Television advertisement reach and frequency management
EP2600545A1 (en) * 2011-12-02 2013-06-05 Waterford Institute Of Technology Research data measurement system and method
US9426525B2 (en) 2013-12-31 2016-08-23 The Nielsen Company (Us), Llc. Methods and apparatus to count people in an audience
US9900636B2 (en) 2015-08-14 2018-02-20 The Nielsen Company (Us), Llc Reducing signature matching uncertainty in media monitoring systems
US11553054B2 (en) 2020-04-30 2023-01-10 The Nielsen Company (Us), Llc Measurement of internet media consumption

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5574962A (en) * 1991-09-30 1996-11-12 The Arbitron Company Method and apparatus for automatically identifying a program including a sound signal
WO2000079709A1 (en) * 1999-06-18 2000-12-28 Apel Steven G Audience survey system, and systems and methods for compressing and correlating audio signals

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5594934A (en) * 1994-09-21 1997-01-14 A.C. Nielsen Company Real time correlation meter
JP2002041569A (en) * 2000-05-19 2002-02-08 Nippon Telegr & Teleph Corp <Ntt> Method and system for distributing retrieval service, method and device for retrieving information, information retrieving server, retrieval service providing method, program therefor, and recording medium the program recorded thereon
JP4558891B2 (en) * 2000-05-24 2010-10-06 株式会社MetaMoJi Marketing survey system and method, apparatus and recording medium
WO2002035516A1 (en) * 2000-10-23 2002-05-02 Ntt Communications Corporation Musical composition recognition method and system, storage medium where musical composition program is stored, commercial recognition method and system, and storage medium where commercial recognition program is stored
JP2002183152A (en) * 2000-12-18 2002-06-28 Jinyama Shunichi Device and method for music retrieval and recording medium with recorded software for music retrieval
JP2002196769A (en) * 2000-12-22 2002-07-12 Nec Corp Music title determination system
JP2002215632A (en) * 2001-01-18 2002-08-02 Nec Corp Music retrieval system, music retrieval method and purchase method using portable terminal
JP4506004B2 (en) * 2001-03-01 2010-07-21 ソニー株式会社 Music recognition device
JP2002268651A (en) * 2001-03-07 2002-09-20 Matsushita Electric Ind Co Ltd Portable radio terminal having music retrieval function
JP2002341875A (en) * 2001-05-11 2002-11-29 Fujitsu Ten Ltd Delivery method for music data
JP4171962B2 (en) * 2001-09-17 2008-10-29 ソニー株式会社 Detailed information providing system, detailed information providing apparatus, portable communication terminal, detailed information providing method of detailed information providing apparatus, and detailed information acquiring method of portable communication terminal
JP4031933B2 (en) * 2002-01-11 2008-01-09 株式会社エヌケービー Advertising system

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5574962A (en) * 1991-09-30 1996-11-12 The Arbitron Company Method and apparatus for automatically identifying a program including a sound signal
WO2000079709A1 (en) * 1999-06-18 2000-12-28 Apel Steven G Audience survey system, and systems and methods for compressing and correlating audio signals

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
JAAP HAITSMA ET AL: "A Highly Robust Audio Fingerprinting System With an Efficient Search Strategy", JOURNAL OF NEW MUSIC RESEARCH, 1 June 2003 (2003-06-01), Lisse, pages 211 - 221, XP055259740, Retrieved from the Internet <URL:http://www.cs.northwestern.edu/~pardo/courses/casa2009/casa_papers_2009/papers/audio fingerprint - haitsma.pdf> [retrieved on 20160318], DOI: 10.1076/jnmr.32.2.211.16746 *

Also Published As

Publication number Publication date
EP1784711A2 (en) 2007-05-16
WO2006026736A3 (en) 2007-11-22
BRPI0514559A (en) 2008-06-17
JP2008511937A (en) 2008-04-17
JP5026269B2 (en) 2012-09-12
WO2006026736A2 (en) 2006-03-09

Similar Documents

Publication Publication Date Title
US8358966B2 (en) Detecting and measuring exposure to media content items
US20070016918A1 (en) Detecting and tracking advertisements
EP1784711A2 (en) Detecting and measuring exposure to media content items
US11625755B1 (en) Determining targeting information based on a predictive targeting model
US20230262367A1 (en) Gathering research data
US20220129916A1 (en) Determining relative effectiveness of media content items
US8185351B2 (en) Methods and systems for testing ability to conduct a research operation
US20080281687A1 (en) Method for determining user interest in products and services for targeted advertising
US20110313849A1 (en) Method and system for effective market research
US20120203363A1 (en) Apparatus, system and method for activating functions in processing devices using encoded audio and audio signatures
US20140236737A1 (en) Wearable computers as media exposure meters
JP5519431B2 (en) Mobile object feature description device, mobile object feature description method, mobile object feature description program, and recording medium
KR20160127139A (en) Method and apparatus for selecting an advertisement for display on a digital sign
US20150365714A1 (en) Personal measurement devices for media consumption studies
JP2012098598A (en) Advertisement providing system, advertisement provision management device, advertisement provision management method, and advertisement provision management program
CN104303539A (en) Estimating demographic statistics of media viewership via context aware mobile devices
US9301019B1 (en) Media correlation by feature matching
Page et al. Measuring Audience Reach Of Outdoor Advertisements: Using Bluetooth Technology To Validate Measurement
WO2006062771A2 (en) System and method for obtaining consumer related statistics
US20110066700A1 (en) Behavior monitoring system
US20220084066A1 (en) System and method for managing advertising and information content, intended for positioning on the means of displaying information, with the possibility of evaluating the effectiveness of the displayed content
CA2907168C (en) System and method for monitoring mobile device activity
CN101288252A (en) Detecting and measuring exposure to media content items
US20210142355A1 (en) Advertisement effectiveness determination
US10129595B2 (en) Methods and apparatus for defining an audience for a particular location by surveying nearby receivers and/or passively monitoring content consumption

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20070221

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL BA HR MK YU

DAX Request for extension of the european patent (deleted)
R17D Deferred search report published (corrected)

Effective date: 20071122

RIC1 Information provided on ipc code assigned before grant

Ipc: H04H 9/00 20080101AFI20080415BHEP

A4 Supplementary search report drawn up and despatched

Effective date: 20080901

RIC1 Information provided on ipc code assigned before grant

Ipc: G06F 17/30 20060101AFI20080826BHEP

17Q First examination report despatched

Effective date: 20081211

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: ASTRO WEST LLC

REG Reference to a national code

Ref country code: DE

Ref legal event code: R003

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN REFUSED

18R Application refused

Effective date: 20161002