EP2973034A1 - Methods and systems for arranging and searching a databased of media content recordings - Google Patents
Methods and systems for arranging and searching a databased of media content recordingsInfo
- Publication number
- EP2973034A1 EP2973034A1 EP14720780.7A EP14720780A EP2973034A1 EP 2973034 A1 EP2973034 A1 EP 2973034A1 EP 14720780 A EP14720780 A EP 14720780A EP 2973034 A1 EP2973034 A1 EP 2973034A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- media content
- data file
- content recording
- representation
- recordings
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 68
- 238000013507 mapping Methods 0.000 claims abstract description 17
- 230000006870 function Effects 0.000 claims description 22
- 238000013500 data storage Methods 0.000 claims description 4
- 238000009877 rendering Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 4
- 230000002441 reversible effect Effects 0.000 description 4
- 230000003190 augmentative effect Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000002123 temporal effect Effects 0.000 description 3
- 230000007613 environmental effect Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000003213 activating effect Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000001174 ascending effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/60—Information retrieval; Database structures therefor; File system structures therefor of audio data
- G06F16/61—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/60—Information retrieval; Database structures therefor; File system structures therefor of audio data
- G06F16/68—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/683—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
Definitions
- TITLE Methods and Systems for Arranging and Searching a Database of Media Content
- a client device may capture a media sample recording of a media stream (such as radio), and may then request a server to perform a search of media recordings (also known as media tracks) for a match to identify the media stream.
- a media sample recording such as radio
- the sample recording may be passed to a content identification server module, which can perform content identification of the sample and return a result of the identification to the client device.
- a recognition result may then be displayed to a user on the client device or used for various follow-on services, such as purchasing or referencing related information.
- Other applications for content identification include broadcast monitoring, for example.
- Existing procedures for ingesting target content into a database index for automatic content identification include acquiring a catalog of content from a content provider or indexing a database from a content owner. Furthermore, existing sources of information to return to a user in a content identification query are obtained from a catalog of content prepared in advance.
- a method comprises receiving a sample of media content, and performing, by a computing device, a content recognition of the sample of media content using a data file including a concatenation of representations for each of a plurality of media content recordings.
- a non-transitory computer readable medium having stored therein instructions, that when executed by a computing device, cause the computing device to perform functions.
- the functions may comprise receiving a sample of media content, and performing, by a computing device, a content recognition of the sample of media content using a data file including a concatenation of representations for each of a plurality of media content recordings.
- a system comprising at least one processor, and data storage configured to store instructions that when executed by the at least one processor cause the system to perform functions.
- the functions may comprise receiving a sample of media content, and performing, by a computing device, a content recognition of the sample of media content using a data file including a concatenation of representations for each of a plurality of media content recordings.
- another method comprises receiving media content recordings, determining a representation for each media content recording, concatenating, by a computing device, the representation for each media content recording as a data file, and storing, by the computing device, a mapping between an identifier for a respective media content recording and a global position in the data file that corresponds to the representation of the respective media content recording.
- a non-transitory computer readable medium having stored therein instructions, that when executed by a computing device, cause the computing device to perform functions.
- the functions may comprise receiving media content recordings, determining a representation for each media content recording, concatenating, by a computing device, the representation for each media content recording as a data file, and storing, by the computing device, a mapping between an identifier for a respective media content recording and a global position in the data file that corresponds to the representation of the respective media content recording.
- a system comprises at least one processor, data storage configured to store instructions that when executed by the at least one processor cause the system to perform functions.
- the functions may comprise receiving media content recordings, determining a representation for each media content recording, concatenating, by a computing device, the representation for each media content recording as a data file, and storing, by the computing device, a mapping between an identifier for a respective media content recording and a global position in the data file that corresponds to the representation of the respective media content recording.
- Figure 1 illustrates one example of a system for identifying content within a data stream and for determining information associated with the identified content.
- Figure 2 shows a flowchart of an example method for performing content recognitions.
- Figure 3 illustrates a diagram of an example method to form a concatenation of representations of media content recordings.
- Figure 4 shows a flowchart of an example method for providing a database of concatenated media content recordings.
- Figure 5 shows a flowchart of an example method for performing a content recognition of a received sample of media content.
- Figure 6 is a diagram that conceptually illustrates performing a content recognition.
- Figure 1 illustrates one example of a system for identifying content within a data stream and for determining information associated with the identified content. While Figure 1 illustrates a system that has a given configuration, the components within the system may be arranged in other manners.
- the system includes a media or data rendering source 102 that renders and presents content from a media stream in any known manner.
- the media stream may be stored on the media rendering source 102 or received from external sources, such as an analog or digital broadcast.
- the media rendering source 102 may be a radio station or a television content provider that broadcasts media streams (e.g., audio and/or video) and/or other information.
- the media rendering source 102 may also be any type of device that plays or audio or video media in a recorded or live format.
- the media rendering source 102 may include a live performance as a source of audio and/or a source of video, for example.
- the media rendering source 102 may render or present the media stream through a graphical display, audio speakers, a MIDI musical instrument, an animatronic puppet, etc., or any other kind of presentation provided by the media rendering source 102, for example.
- a client device 104 receives a rendering of the media stream from the media rendering source 102 through an input interface 106.
- the input interface 106 may include antenna, in which case the media rendering source 102 may broadcast the media stream wirelessly to the client device 104.
- the media rendering source 102 may render the media using wireless or wired communication techniques.
- the input interface 106 can include any of a microphone, video camera, vibration sensor, radio receiver, network interface, etc. The input interface 106 may be preprogrammed to capture media samples continuously without user intervention, such as to record all audio received and store recordings in a buffer 108.
- the buffer 108 may store a number of recordings, or may store recordings for a limited time, such that the client device 104 may record and store recordings in predetermined intervals, for example, or in a way so that a history of a certain length backwards in time is available for analysis.
- capturing of the media sample may be caused or triggered by a user activating a button or other application to trigger the sample capture.
- the client device 104 can be implemented as a portion of a small-form factor portable (or mobile) electronic device such as a cell phone, a wireless cell phone, a personal data assistant (PDA), tablet computer, a personal media player device, a wireless web-watch device, a personal headset device, an application specific device, or a hybrid device that include any of the above functions.
- the client device 104 can also be implemented as a personal computer including both laptop computer and non- laptop computer configurations.
- the client device 104 can also be a component of a larger device or system as well.
- the client device 104 further includes a position identification module 110 and a content identification module 112.
- the position identification module 110 is configured to receive a media sample from the buffer 108 and to identify a corresponding estimated time position (Ts) indicating a time offset of the media sample into the rendered media stream (or into a segment of the rendered media stream) based on the media sample that is being captured at that moment.
- the time position (Ts) may also, in some examples, be an elapsed amount of time from a beginning of the media stream.
- the media stream may be a radio broadcast, and the time position (Ts) may correspond to an elapsed amount of time of a song being rendered.
- the content identification module 112 is configured to receive the media sample from the buffer 108 and to perform a content identification on the received media sample.
- the content identification identifies a media stream, or identifies information about or related to the media sample.
- the content identification module 112 may be configured to receive samples of environmental audio, identify a content of the audio sample, and provide information about the content, including the track name, artist, album, artwork, biography, discography, concert tickets, etc.
- the content identification module 112 includes a media search engine 114 and may include or be coupled to a database 116 that indexes reference media streams, for example, to compare the received media sample with the stored information so as to identify tracks within the received media sample.
- the database 116 may store content patterns that include information to identify pieces of content.
- the content patterns may include media recordings such as music, advertisements, jingles, movies, documentaries, television and radio programs. Each recording may be identified by a unique identifier (e.g., sound ID).
- the database 116 may not necessarily store audio or video files for each recording, since the sound lDs can be used to retrieve audio files from elsewhere.
- the database 116 may yet additionally or alternatively store representations for multiple media content recordings as a single data file where all media content recordings are concatenated end to end to conceptually form a single media content recording, for example.
- the database 116 may include other information (in addition to or rather than media recordings), such as reference signature files including a temporally mapped collection of features describing content of a media recording that has a temporal dimension corresponding to a timeline of the media recording, and each feature may be a description of the content in a vicinity of each mapped timepoint.
- reference signature files including a temporally mapped collection of features describing content of a media recording that has a temporal dimension corresponding to a timeline of the media recording, and each feature may be a description of the content in a vicinity of each mapped timepoint.
- the database 116 may also include information associated with stored content patterns, such as metadata that indicates information about the content pattern like an artist name, a length of song, lyrics of the song, time indices for lines or words of the lyrics, album artwork, or any other identifying or related information to the file. Metadata may also comprise data and hyperlinks to other related content and services, including recommendations, ads, offers to preview, bookmark, and buy musical recordings, videos, concert tickets, and bonus content; as well as to facilitate browsing, exploring, discovering related content on the world wide web.
- Metadata may also comprise data and hyperlinks to other related content and services, including recommendations, ads, offers to preview, bookmark, and buy musical recordings, videos, concert tickets, and bonus content; as well as to facilitate browsing, exploring, discovering related content on the world wide web.
- the system in Figure 1 further includes a network 118 to which the client device
- a server 120 is provided coupled to the network 118, and the server 120 includes a position identification module 122 and a content identification module 124.
- Figure 1 illustrates the server 120 to include both the position identification module 122 and the content identification module 124, either of the position identification module 122 and/or the content identification module 124 may be separate entities apart from the server 120, for example.
- the position identification module 122 and/or the content identification module 124 may be on a remote server connected to the server 120 over the network 118, for example.
- the server 120 may be configured to index target media content rendered by the media rendering source 102.
- the content identification module 124 includes a media search engine 126 and may include or be coupled to a database 128 that indexes reference or known media streams, for example, to compare the rendered media content with the stored information so as to identify content within the rendered media content.
- the database 128 (similar to database 116 in the client device 104) may additionally or alternatively store multiple media content recordings as a single data file where all the media content recordings are concatenated end to end to conceptually form a single media content recording. A content recognition can then be performed by compared rendered media content with the data file to identify matching content using a single search. Once content within the media stream have been identified, identities or other information may be indexed in the database 128.
- the client device 104 may capture a media sample and may send the media sample over the network 118 to the server 120 to determine an identity of content in the media sample.
- the server 120 may identify a media recoding from which the media sample was obtained based on comparison to indexed recordings in the database 128. The server 120 may then return information identifying the media recording, and other associated information to the client device 104.
- Figure 2 shows a flowchart of an example method 200 for performing content recognitions.
- Method 200 shown in Figure 2 presents an embodiment of a method that, for example, could be used with the system shown in Figure 1 , for example, and may be performed by a computing device (or components of a computing device) such as a client device or a server or may be performed by components of both a client device and a server.
- Method 200 may include one or more operations, functions, or actions as illustrated by one or more of blocks 202-204. Although the blocks are illustrated in a sequential order, these blocks may also be performed in parallel, and/or in a different order than those described herein. Also, the various blocks may be combined into fewer blocks, divided into additional blocks, and/or removed based upon the desired implementation.
- each block may represent a module, a segment, or a portion of program code, which includes one or more instructions executable by a processor for implementing specific logical functions or steps in the process.
- the program code may be stored on any type of computer readable medium or data storage, for example, such as a storage device including a disk or hard drive.
- the computer readable medium may include non- transitory computer readable medium or memory, for example, such as computer-readable media that stores data for short periods of time like register memory, processor cache and Random Access Memory (RAM).
- the computer readable medium may also include non- transitory media, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, compact-disc read only memory (CD-ROM), for example.
- the computer readable media may also be any other volatile or non-volatile storage systems.
- the computer readable medium may be considered a tangible computer readable storage medium, for example.
- each block in Figure 2 may represent circuitry that is wired to perform the specific logical functions in the process.
- Alternative implementations are included within the scope of the example embodiments of the present disclosure in which functions may be executed out of order from that shown or discussed, including substantially concurrent or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art.
- the method 200 includes receiving a sample of media content.
- a computing device may receive the sample of media content from an ambient environment of the computing device, such as via a microphone, receiver, etc., and may record and store the sample.
- the computing device may receive the sample of media content from another computing device (e.g., one computing device records the sample and sends the sample to a server).
- the method 200 includes performing a content recognition of the sample of media content using a data file including a concatenation of representations for each of a plurality of media content recordings.
- the concatenation may include a plurality of respective representations (e.g., fingerprints or set of fingerprints) per media content recording and arranged in sequential time order per media content recording in the data file.
- a representation for a given media content recording may include a set of fingerprints determined or extracted at respective landmark positions within the given media content recording, and each fingerprint corresponds to a global position within the data file.
- the data file also may have associated identifiers per groupings of representations (e.g., per sets of fingerprints) for each of the plurality of media content recordings.
- the identifiers may include any of a title of a song, an artist, genre, etc.
- the content recognition can be performed by determining a representation in the data file that matches to a portion of the sample of media content, and then to identify a mapping between the matching portion in the data file and an identifier for a respective media content recording.
- the mapping may be between a global position of the representation in the data file and the identifier.
- the content recognition may be performed by identifying within the data file a substantially matching representation to a respective representation of the sample of media content, and then determining a global position in the data file corresponding to the substantially matching representation.
- the representations for each of the plurality of media content recordings in the data file have associated global starting positions within the data file so as to segment a global timeline of the data file according to the plurality of media content recordings.
- a global starting position in the data file associated with the substantially matching representation at the determined global position can also be identified.
- the method 200 may also include determining a local position within a given media content recording corresponding to the sample of media content based on the global position and the global starting position.
- a large database of media recordings may be searched using a single bucket (instead of separate buckets indexed by a sound ID) to obtain enhanced recognition performance with simplified data processing structures.
- Existing search techniques may process search data by separating matching data into different buckets, and each bucket corresponds to a distinct target object.
- it may be more efficient not to distribute data into separate buckets, but rather to process un-separated data in a single bulk operation.
- By performing a single search operation of a received sample of media content into a database for content recognition overhead processing due to distribution and tracking of bucket indices and iterating over buckets may be removed.
- a method of aggregating searches in which one bulk operation carried out on a single concatenated media content recording may be more efficient than a number of small operations.
- a reference database of media content recordings to use to identify unknown media content may include a concatenation of representations of all known media content recordings into a single concatenated media recording file that has a single concatenated timeline, in which associated identifiers may not be directly referenced in the file.
- Each media content recording can be represented as being located along the concatenated timeline at a given position, and boundaries of the recordings can be stored to translate an identified position in the file to an identifier.
- the representations of the media content recordings may be any number or type of data.
- the representations may include a set of fingerprints for each media content recording.
- Figure 3 illustrates a diagram of an example method to form a concatenation of representations of media content recordings.
- media content can be identified by computing characteristics or fingerprints of a media sample and comparing the fingerprints to previously identified fingerprints of reference media files. Particular locations within the sample at which fingerprints are computed may depend on reproducible points in the sample. Such reproducibly computable locations are referred to as "landmarks.”
- One landmarking technique known as Power Norm, is to calculate an instantaneous power at many time points in the recording and to select local maxima. One way of doing this is to calculate an envelope by rectifying and filtering a waveform directly.
- Figure 3 illustrates a media content recording being input to a fingerprint extractor 302 (or fingerprint generator) that is configured to determine fingerprints of the media content recording.
- An example plot of dB (magnitude) of a sample vs. time is shown, and the plot illustrates a number of identified landmark positions (Li to Lg).
- the fingerprint extractor 302 is configured to compute a fingerprint at or near each landmark time point in the recording.
- the fingerprint is generally a value or set of values that summarizes a set of features in the recording at or near the landmark time point.
- each fingerprint is a single numerical value that is a hashed function of multiple features.
- Other examples of fingerprints include spectral slice fingerprints, multi-slice fingerprints, LPC coefficients, cepstral coefficients, and frequency components of spectrogram peaks.
- the fingerprint extractor 302 may generate a set of fingerprints each with a corresponding landmark and provide the fingerprint/landmark pairs for each media content recording to a database 304 for storage.
- the fingerprints are then represented in the database 304 as key- value pairs where the key is the fingerprint and the value is a corresponding landmark.
- a value may also have an associated sound ID within the database 304, for example.
- Media recordings can be indexed with sound ID from 0 to N-l, where N is a number of media recordings.
- a concatenator 306 may retrieve the fingerprint/landmark pairs for each media content recording and maintain the fingerprints per recording in time order based on the landmarks for that recording so as to create a time ordered fingerprint set for each recording. The concatenator 306 then joins the fingerprint sets for all recordings end to end into a single data file 308 that has a conceptual timeline or global time.
- a mapping can be created between each sound ID and a corresponding global position in the data file 308.
- a list of global starting positions for each original media recording within the concatenated media recording data file is stored to create a reverse mapping from each global position to a corresponding local position in an original media recording indexed by a sound ID.
- the global starting positions thus segment the global timeline according to the original media recordings.
- a global position in the timeline as well as a global start position of the media recording is determined according to Equation 1.
- a global position in the timeline as well as a global start position of the media recording is determined according to Equation 1.
- the global start positions are searched for an interval containing the global position, i.e., find a sound ID
- a mapping can be created between each sound ID and a corresponding global position in the data file 308.
- a list of global starting positions for each original media recording within the concatenated media recording data file is stored to create a reverse mapping from each global position to a corresponding local position in an original media recording indexed by a sound lD.
- the global starting positions thus segment the global timeline according to the original media recordings.
- the data file 308 conceptually represents a K-V database where each key K is a fingerprint and the value V comprises a global position corresponding to a landmark position of the fingerprint.
- a buffer e.g., blank space of several seconds worth of timeline
- Figure 4 shows a flowchart of an example method 400 for providing a database of concatenated media content recordings.
- Method 400 shown in Figure 4 presents an embodiment of a method that, for example, could be used with the system shown in Figure 1, for example, and may be performed by a computing device (or components of a computing device) such as a client device or a server or may be performed by components of both a client device and a server.
- a computing device or components of a computing device
- the method 400 includes receiving media content recordings.
- Media content recordings may include a number of songs, television programs, or any type of audio and/or video recordings.
- the method 400 includes determining a representation for each media content recording.
- fingerprints of a respective media content recording can be determined at respective positions within the respective media content recording, and the representation can be defined as the fingerprints.
- the representation may include additional or alternative information describing the media content recording such as any type of characteristic of the media content recording.
- the method 400 includes concatenating the representation for each media content recording as a data file.
- the data file has a concatenated or global timeline, and a given media content recording is represented as being located within the data file along the concatenated timeline at a given position or global position.
- a buffer is provided between adjacent representations of media content recordings within the data file.
- the method 400 includes storing a mapping between an identifier for a respective media content recording and a global position in the data file that corresponds to the representation of the respective media content recording.
- the global position may thus correspond to a sound identifier of the given media content recording and a local landmark position of the fingerprint within the given media content recording.
- the method 400 also includes storing a list of global starting positions for media content recordings within the concatenated data file to segment a global timeline of the data file according to the media content recordings. Additionally, a list of boundaries between each representation of media content recording within the concatenated data file can be stored as well.
- a sample of media content is received, and a content recognition is performed by searching for matching content within the data file of concatenated media recordings.
- Any number of content identification matching methods may be used depending on a type of content being identified.
- an example video identification algorithm is described in Oostveen, J., et al., "Feature Extraction and a Database Strategy for Video Fingerprinting", Lecture Notes in Computer Science, 2314, (Mar. 11, 2002), 117-128, the entire contents of which are herein incorporated by reference.
- a position of the video sample into a video can be derived by determining which video frame was identified.
- frames of the media sample can be divided into a grid of rows and columns, and for each block of the grid, a mean of the luminance values of pixels is computed.
- a spatial filter can be applied to the computed mean luminance values to derive fingerprint bits for each block of the grid.
- the fingerprint bits can be used to uniquely identify the frame, and can be compared or matched to fingerprint bits of a database that includes known media. Based on which frame the media sample included, a position into the video (e.g., time offset) can be determined.
- fingerprints of a received sample of media content can be matched to fingerprints of known media content by generating correspondences between equivalent fingerprints in the concatenated data file to locate a media recording that has a largest number of linearly related correspondences, or whose relative locations of characteristic fingerprints most closely match the relative locations of the same fingerprints of the recording.
- Figure 5 shows a flowchart of an example method 500 for performing a content recognition of a received sample of media content.
- Method 500 shown in Figure 5 presents an embodiment of a method that, for example, could be used with the system shown in Figure 1, for example, and may be performed by a computing device (or components of a computing device) such as a client device or a server or may be performed by components of both a client device and a server.
- a computing device or components of a computing device
- the method 500 includes determining fingerprints in the data file that substantially match to one or more fingerprints of the sample of media content. Fingerprints of the received sample of media content are created by processing a query media sample into a set of sample landmark and fingerprint pairs. The sample fingerprints are then used to retrieve matching KV pairs in the KV data file of concatenated media content, where the key K is a fingerprint and the value V is the payload, which in this case is a concatenated global position value.
- the method 500 includes pairing corresponding global positions of the substantially matching fingerprints with corresponding respective landmark positions of the one or more fingerprints in the sample of media content to provide global position-landmark position pairs.
- a retrieved global position value is paired with the sample landmark value.
- a time offset between the two positions may then be determined, for each global position-landmark position pair, by subtracting the global position value from the sample landmark value for matching fingerprints.
- time offset pair differences generated by subtracting corresponding time offsets from matching sample versus reference fingerprints
- all time offset differences can be stored in a single bucket.
- the method 500 includes sorting the global position-landmark position pairs.
- the method 500 may include sorting the time offset differences generated from the global position-landmark position pairs.
- a radix sorting method may be used. Radix sorting algorithms are known in the art and discussed in D. E. Knuth, The Art of Computer Programming, Volume 3: Sorting and Searching, Reading, Mass.: Addison- Wesley, 1998, herein incorporated by reference.
- the radix sort includes a non-comparison linear-time sort that sorts data with integer keys by grouping keys by the individual digits which share the same significant position and value.
- the radix sort method may be conveniently implemented using commodity computational hardware and algorithms. For a large scale sorting of the entire set of time offset differences into one bucket, the radix sort may be economically advantageous over standard sorts on many small buckets, for example using conventional quicksort or heapsort methods. Following the sort, the time offset differences will be organized in order of ascending global position.
- the method 500 includes determining clusters of the global position-landmark position pairs that are substantially linearly related (or have some associated temporal correspondence).
- a histogram scan can be performed to search for a significant peak in the sorted time offset difference data (e.g., number of data points occurring within a predetermined window width or number of points in a histogram bin). A presence of a peak in the number of points above a threshold within a window or bin can be interpreted as evidence for a match.
- Each occurrence of a significant peak in the long concatenated timeline of time offset differences indicates a candidate match, and candidate matches may be further processed individually to ascertain whether the candidates matches are exact, possibly using a different algorithm to verify a match.
- the time offset differences may be filtered using a predetermined window width of a few milliseconds.
- the method 500 includes identifying a matching media content recording to the sample of media content as a media content recording having a cluster with a largest number of global position-landmark position pairs that are substantially linearly related.
- the candidate match that has the most time offset differences within a predetermined window width can be deemed the winning matching file, for example.
- a buffer (e.g., blank space of several seconds worth of timeline) may be inserted between adjacent recordings in the concatenated data file to make it less ambiguous which media content recording was a match in case a sample offset into a particular song was negative, e.g., if the sample started before the song started then an offset mapping would put the recognized offset point in the previous song of the concatenated data file.
- the method 500 may further include determining a sound identifier of the matching media content recording based on the corresponding global position of the substantially matching fingerprints in the data file. For example, global positions of representations of the given media content recording in the data file can be associated or mapped to respective sound identifiers, and the mapping may be referenced when a winning global position is identified.
- FIG. 6 is a diagram that conceptually illustrates performing a content recognition.
- fingerprint and landmark pairs (Fi/Li, F 2 /L 2 , F n /L n ) can be determined and the fingerprints can be used to find matching fingerprints within the concatenated data file of known media content recordings.
- Global positions within the data file can be paired with landmarks in the sample for matching fingerprints.
- a scatter plot of landmarks of the sample and global positions of the known reference files can be determined After generating a scatter plot, clusters of landmark pairs having linear correspondences can be identified, and the clusters can be scored according to the number of pairs that are linearly related.
- a linear correspondence may occur when a statistically significant number of corresponding sample locations and reference file locations can be described with a linear equation, within an allowed tolerance, for example.
- An X-intercept of the linear equation may be a global time offset of the beginning of a matching media recording, and may be used for position detection, as well as for content identification.
- the file of the cluster with the highest statistically significant score, i.e., with the largest number of linearly related landmark pairs, is the winning file, and may be deemed the matching media file.
- a histogram of offset values can be generated.
- the offset values may be differences between landmark time positions and the global positions where a fingerprint matches.
- Figure 6 illustrates an example histogram of offset values.
- the entire concatenated data file may be processed in this manner using a single bulk operation to determine histogram peaks and a score for each peak, and the media content recording corresponding to the global position resulting in the highest score may be determined to be a match to the sample.
- the Hough transform or RANSAC algorithms may be used to determine or detect a linear or temporal correspondence between time differences.
- multiple simultaneous searches of the concatenated data file may be performed to determine a content recognition for multiple samples at the same time.
- the time offset pair differences between landmarks and global positions for matching fingerprints, per sample can be augmented by adding extra bits to the representation to indicate a sub-search index.
- extra bits For data representations of the time differences of up to 30 bits, an extra 2 high bits may be added to make the data representation an even 32 bits. The extra 2 bits may then index up to 4 separate searches.
- MSBs most significant bits
- each search may be processed with time offset pair differences put into the single bucket, and augmented with a unique identifier using the upper k MSBs.
- the single bucket may thus be filled with data for up to 2 A k searches over a large number of songs, and thus, buckets for many songs and sessions can be collapsed into one.
- a single sort operation can be performed to sort all the augmented time differences in the bucket.
- a histogram peak scan is carried out, as before, and the peaks are determined, and locations of the peaks may be interpreted as follows: the upper k bits of a peak indicate which sub-search the peak belongs to, and the lower bits indicate which song the sample matched.
- a content recognition of a received sample of media content can be performed using a single array of data representing all known reference media content.
- Reference to media content identifiers can be removed from the searching process, and determined based on mappings to positions in the single array of data.
- a single or bulk global sort can be performed for simpler and more efficient sorting so that rather than minimizing an amount of material to sort (per sort), an amount of material is maximized.
- a number of items searched can be increased by batching multiple queries, using bits to index batch entry, and a single sort then accomplishes separation of batches.
- a histogram scan or other peak determination methods can be performed to identify a winning match on a continuous timeline, and a media content identifier is retrieved after histogram searching by using reverse lookup, e.g. a binary search on an offset table.
Abstract
Description
Claims
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/837,711 US9773058B2 (en) | 2013-03-15 | 2013-03-15 | Methods and systems for arranging and searching a database of media content recordings |
PCT/US2014/025575 WO2014151365A1 (en) | 2013-03-15 | 2014-03-13 | Methods and systems for arranging and searching a databased of media content recordings |
Publications (2)
Publication Number | Publication Date |
---|---|
EP2973034A1 true EP2973034A1 (en) | 2016-01-20 |
EP2973034B1 EP2973034B1 (en) | 2018-01-03 |
Family
ID=50628943
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP14720780.7A Active EP2973034B1 (en) | 2013-03-15 | 2014-03-13 | Methods and systems for arranging and searching a database of media content recordings |
Country Status (4)
Country | Link |
---|---|
US (1) | US9773058B2 (en) |
EP (1) | EP2973034B1 (en) |
CA (1) | CA2905654C (en) |
WO (1) | WO2014151365A1 (en) |
Families Citing this family (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9275682B1 (en) * | 2014-09-26 | 2016-03-01 | A9.Com, Inc. | Video content alignment |
GB201508963D0 (en) * | 2015-05-26 | 2015-07-01 | Geo Track Identifier Gmbh | Audio identification method |
WO2017039693A1 (en) * | 2015-09-04 | 2017-03-09 | Costabile Michael J | System for remotely starting and stopping a time clock in an environment having a plurality of distinct activation signals |
US10083353B2 (en) * | 2016-10-28 | 2018-09-25 | Intuit Inc. | Identifying document forms using digital fingerprints |
US10922720B2 (en) | 2017-01-11 | 2021-02-16 | Adobe Inc. | Managing content delivery via audio cues |
GB2578082A (en) * | 2018-05-23 | 2020-04-22 | Zoo Digital Ltd | Comparing Audiovisual Products |
US10860647B2 (en) | 2018-09-06 | 2020-12-08 | Gracenote, Inc. | Systems, methods, and apparatus to improve media identification |
EP3937503A1 (en) * | 2020-07-07 | 2022-01-12 | Open Products, Networks & Software | Audio data process |
US11763591B2 (en) * | 2020-08-20 | 2023-09-19 | The Nielsen Company (Us), Llc | Methods and apparatus to determine an audience composition based on voice recognition, thermal imaging, and facial recognition |
US11553247B2 (en) | 2020-08-20 | 2023-01-10 | The Nielsen Company (Us), Llc | Methods and apparatus to determine an audience composition based on thermal imaging and facial recognition |
Family Cites Families (69)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4450531A (en) | 1982-09-10 | 1984-05-22 | Ensco, Inc. | Broadcast signal recognition system and method |
US4843562A (en) | 1987-06-24 | 1989-06-27 | Broadcast Data Systems Limited Partnership | Broadcast information classification system and method |
US5134719A (en) | 1991-02-19 | 1992-07-28 | Mankovitz Roy J | Apparatus and methods for identifying broadcast audio program selections in an FM stereo broadcast system |
US6850252B1 (en) * | 1999-10-05 | 2005-02-01 | Steven M. Hoffberg | Intelligent electronic appliance system and method |
US5333275A (en) | 1992-06-23 | 1994-07-26 | Wheatley Barbara J | System and method for time aligning speech |
US5437050A (en) | 1992-11-09 | 1995-07-25 | Lamb; Robert G. | Method and apparatus for recognizing broadcast information using multi-frequency magnitude detection |
US5465240A (en) | 1993-01-05 | 1995-11-07 | Mankovitz; Roy J. | Apparatus and methods for displaying text in conjunction with recorded audio programs |
IL108497A0 (en) | 1993-02-01 | 1994-05-30 | Seq Ltd | Methods and apparatus for dna sequencing |
JP2986345B2 (en) | 1993-10-18 | 1999-12-06 | インターナショナル・ビジネス・マシーンズ・コーポレイション | Voice recording indexing apparatus and method |
US5740230A (en) | 1996-05-31 | 1998-04-14 | Octel Communications Corporation | Directory management system and method |
US5918223A (en) | 1996-07-22 | 1999-06-29 | Muscle Fish | Method and article of manufacture for content-based analysis, storage, retrieval, and segmentation of audio information |
US5952597A (en) | 1996-10-25 | 1999-09-14 | Timewarp Technologies, Ltd. | Method and apparatus for real-time correlation of a performance to a musical score |
US6166314A (en) | 1997-06-19 | 2000-12-26 | Time Warp Technologies, Ltd. | Method and apparatus for real-time correlation of a performance to a musical score |
JPH1155201A (en) | 1997-07-29 | 1999-02-26 | Sony Corp | Device, method and system for information processing and transmitting medium |
CN100382498C (en) | 1998-07-14 | 2008-04-16 | 索尼公司 | Data transmission control method, data transmission method, data transmitter, and receiver |
JP3925586B2 (en) | 1998-07-17 | 2007-06-06 | ソニー株式会社 | Data receiving apparatus and method, and data transmission / reception system and method |
TW495735B (en) | 1999-07-28 | 2002-07-21 | Yamaha Corp | Audio controller and the portable terminal and system using the same |
CN100379273C (en) * | 1999-09-20 | 2008-04-02 | 提维股份有限公司 | Closed caption tagging system |
US7174293B2 (en) | 1999-09-21 | 2007-02-06 | Iceberg Industries Llc | Audio identification system and method |
US7194752B1 (en) | 1999-10-19 | 2007-03-20 | Iceberg Industries, Llc | Method and apparatus for automatically recognizing input audio and/or video streams |
US7444353B1 (en) | 2000-01-31 | 2008-10-28 | Chen Alexander C | Apparatus for delivering music and information |
US6990453B2 (en) | 2000-07-31 | 2006-01-24 | Landmark Digital Services Llc | System and methods for recognizing sound and music signals in high noise and distortion |
ES2414650T3 (en) | 2000-08-23 | 2013-07-22 | Gracenote, Inc. | Procedure and system for obtaining information |
FI20002161A (en) | 2000-09-29 | 2002-03-30 | Nokia Mobile Phones Ltd | Method and system for recognizing a melody |
US7277766B1 (en) | 2000-10-24 | 2007-10-02 | Moodlogic, Inc. | Method and system for analyzing digital audio files |
US20020072982A1 (en) | 2000-12-12 | 2002-06-13 | Shazam Entertainment Ltd. | Method and system for interacting with a user in an experiential environment |
EP1362485B1 (en) | 2001-02-12 | 2008-08-13 | Gracenote, Inc. | Generating and matching hashes of multimedia content |
PT1504445E (en) | 2002-04-25 | 2008-11-24 | Landmark Digital Services Llc | Robust and invariant audio pattern matching |
US6766523B2 (en) | 2002-05-31 | 2004-07-20 | Microsoft Corporation | System and method for identifying and segmenting repeating media objects embedded in a stream |
US7461392B2 (en) | 2002-07-01 | 2008-12-02 | Microsoft Corporation | System and method for identifying and segmenting repeating media objects embedded in a stream |
JP4658598B2 (en) | 2002-07-01 | 2011-03-23 | マイクロソフト コーポレーション | System and method for providing user control over repetitive objects embedded in a stream |
US9818136B1 (en) * | 2003-02-05 | 2017-11-14 | Steven M. Hoffberg | System and method for determining contingent relevance |
CN1742492B (en) * | 2003-02-14 | 2011-07-20 | 汤姆森特许公司 | Automatic synchronization of audio and video based media services of media content |
US20040266337A1 (en) | 2003-06-25 | 2004-12-30 | Microsoft Corporation | Method and apparatus for synchronizing lyrics |
US20050078822A1 (en) * | 2003-10-08 | 2005-04-14 | Eyal Shavit | Secure access and copy protection management system |
US7788696B2 (en) | 2003-10-15 | 2010-08-31 | Microsoft Corporation | Inferring information about media stream objects |
JP4933899B2 (en) | 2004-02-19 | 2012-05-16 | ランドマーク、ディジタル、サーヴィセズ、エルエルシー | Method and apparatus for broadcast source identification |
US20060112812A1 (en) | 2004-11-30 | 2006-06-01 | Anand Venkataraman | Method and apparatus for adapting original musical tracks for karaoke use |
CN1845591A (en) | 2005-04-06 | 2006-10-11 | 上海渐华科技发展有限公司 | Kara-Ok receiver |
JP5329968B2 (en) | 2005-11-10 | 2013-10-30 | サウンドハウンド インコーポレイテッド | How to store and retrieve non-text based information |
EP1963958B1 (en) * | 2005-12-21 | 2019-04-24 | Digimarc Corporation | Rules driven pan id metadata routing system and network |
WO2007072468A1 (en) * | 2005-12-22 | 2007-06-28 | Digiprove Limited | Establishing proof of existence and possession of digital content |
US20070166683A1 (en) | 2006-01-05 | 2007-07-19 | Apple Computer, Inc. | Dynamic lyrics display for portable media devices |
US7831531B1 (en) | 2006-06-22 | 2010-11-09 | Google Inc. | Approximate hashing functions for finding similar content |
US8363882B2 (en) | 2006-07-25 | 2013-01-29 | Adobe Systems Incorporated | User discernible watermarking |
US8005666B2 (en) | 2006-10-24 | 2011-08-23 | National Institute Of Advanced Industrial Science And Technology | Automatic system for temporal alignment of music audio signal with lyrics |
TWI330795B (en) | 2006-11-17 | 2010-09-21 | Via Tech Inc | Playing systems and methods with integrated music, lyrics and song information |
US20080187188A1 (en) * | 2007-02-07 | 2008-08-07 | Oleg Beletski | Systems, apparatuses and methods for facilitating efficient recognition of delivered content |
US20080196575A1 (en) | 2007-02-16 | 2008-08-21 | Recordare Llc | Process for creating and viewing digital sheet music on a media device |
US7979464B2 (en) | 2007-02-27 | 2011-07-12 | Motion Picture Laboratories, Inc. | Associating rights to multimedia content |
US20090083281A1 (en) | 2007-08-22 | 2009-03-26 | Amnon Sarig | System and method for real time local music playback and remote server lyric timing synchronization utilizing social networks and wiki technology |
US20090063277A1 (en) * | 2007-08-31 | 2009-03-05 | Dolby Laboratiories Licensing Corp. | Associating information with a portion of media content |
WO2009100093A1 (en) * | 2008-02-05 | 2009-08-13 | Dolby Laboratories Licensing Corporation | Associating information with media content |
JP5204893B2 (en) * | 2008-03-31 | 2013-06-05 | ドルビー ラボラトリーズ ライセンシング コーポレイション | Distributed media fingerprint repository |
EP2141689A1 (en) * | 2008-07-04 | 2010-01-06 | Koninklijke KPN N.V. | Generating a stream comprising interactive content |
US8143508B2 (en) | 2008-08-29 | 2012-03-27 | At&T Intellectual Property I, L.P. | System for providing lyrics with streaming music |
US20100132122A1 (en) | 2008-12-02 | 2010-06-03 | Dan Hollingshead | Bed-Mounted Computer Terminal |
US9390167B2 (en) | 2010-07-29 | 2016-07-12 | Soundhound, Inc. | System and methods for continuous audio matching |
US8291328B2 (en) | 2009-03-24 | 2012-10-16 | Disney Enterprises, Inc. | System and method for synchronizing a real-time performance with a virtual object |
US8548613B2 (en) | 2009-04-20 | 2013-10-01 | Disney Enterprises, Inc. | System and method for an interactive device for use with a media device |
US8886531B2 (en) | 2010-01-13 | 2014-11-11 | Rovi Technologies Corporation | Apparatus and method for generating an audio fingerprint and using a two-stage query |
US9280598B2 (en) | 2010-05-04 | 2016-03-08 | Soundhound, Inc. | Systems and methods for sound recognition |
US9965771B2 (en) * | 2010-07-02 | 2018-05-08 | Dialogic Corporation | System and method of tracking video advertisements |
US9047371B2 (en) | 2010-07-29 | 2015-06-02 | Soundhound, Inc. | System and method for matching a query against a broadcast stream |
US8977207B2 (en) * | 2011-09-15 | 2015-03-10 | Nokia Corporation | Methods, apparatuses and computer program products for providing automatic maintenance of a geoposition system |
BR102012019954A2 (en) * | 2012-08-09 | 2013-08-13 | Connectmix Elaboracao De Programas Eireli | real-time audio monitoring of radio and tv stations |
US8838657B1 (en) * | 2012-09-07 | 2014-09-16 | Amazon Technologies, Inc. | Document fingerprints using block encoding of text |
US9317471B2 (en) * | 2012-11-02 | 2016-04-19 | Google Inc. | Bookmarking prospective media content on computer network |
US10447826B2 (en) * | 2013-03-14 | 2019-10-15 | Google Llc | Detecting user interest in presented media items by observing volume change events |
-
2013
- 2013-03-15 US US13/837,711 patent/US9773058B2/en active Active
-
2014
- 2014-03-13 WO PCT/US2014/025575 patent/WO2014151365A1/en active Application Filing
- 2014-03-13 EP EP14720780.7A patent/EP2973034B1/en active Active
- 2014-03-13 CA CA2905654A patent/CA2905654C/en active Active
Non-Patent Citations (1)
Title |
---|
See references of WO2014151365A1 * |
Also Published As
Publication number | Publication date |
---|---|
US9773058B2 (en) | 2017-09-26 |
EP2973034B1 (en) | 2018-01-03 |
CA2905654C (en) | 2019-05-21 |
CA2905654A1 (en) | 2014-09-25 |
WO2014151365A1 (en) | 2014-09-25 |
US20140280127A1 (en) | 2014-09-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CA2905654C (en) | Methods and systems for arranging and searching a database of media content recordings | |
CA2905385C (en) | Methods and systems for arranging and searching a database of media content recordings | |
US20160132600A1 (en) | Methods and Systems for Performing Content Recognition for a Surge of Incoming Recognition Queries | |
US20140278845A1 (en) | Methods and Systems for Identifying Target Media Content and Determining Supplemental Information about the Target Media Content | |
KR101578279B1 (en) | Methods and systems for identifying content in a data stream | |
EP2234024B1 (en) | Context based video finder | |
US9451048B2 (en) | Methods and systems for identifying information of a broadcast station and information of broadcasted content | |
JP5907511B2 (en) | System and method for audio media recognition | |
KR101648204B1 (en) | Generating metadata for association with a collection of content items | |
JP2004505328A5 (en) | ||
US20140161263A1 (en) | Facilitating recognition of real-time content | |
EP3132363A1 (en) | Methods, systems, and media for presenting music items relating to media content | |
CN102411578A (en) | Multimedia playing system and method | |
EP2676208A1 (en) | Methods and systems for identifying content in a data stream by a client device | |
KR100916310B1 (en) | System and Method for recommendation of music and moving video based on audio signal processing | |
CN112732972B (en) | Audio fingerprint generation system and method | |
CN107784128B (en) | Method and system for storing and retrieving digital audio fingerprint big data | |
CN112435688A (en) | Audio recognition method, server and storage medium | |
KR20170097908A (en) | Apparatus and method for generating or matching hash based on user information | |
CN112380382A (en) | Audio classification method and device and storage medium | |
CN117573918A (en) | Method, apparatus and computer readable medium for identifying audio clips | |
WOODHEAD et al. | Patent 2965360 Summary | |
CN113129855A (en) | Audio fingerprint extraction and database building method, and audio identification and retrieval method and system | |
Aruna et al. | SS International Journal of Multidisciplinary Research |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20150921 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAX | Request for extension of the european patent (deleted) | ||
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
INTG | Intention to grant announced |
Effective date: 20170309 |
|
GRAJ | Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted |
Free format text: ORIGINAL CODE: EPIDOSDIGR1 |
|
RAP1 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: SHAZAM INVESTMENTS LIMITED |
|
INTC | Intention to grant announced (deleted) | ||
GRAR | Information related to intention to grant a patent recorded |
Free format text: ORIGINAL CODE: EPIDOSNIGR71 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
INTG | Intention to grant announced |
Effective date: 20171122 |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP Ref country code: AT Ref legal event code: REF Ref document number: 960907 Country of ref document: AT Kind code of ref document: T Effective date: 20180115 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602014019377 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 5 |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20180103 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 960907 Country of ref document: AT Kind code of ref document: T Effective date: 20180103 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180403 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180503 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180403 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180404 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602014019377 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Ref document number: 602014019377 Country of ref document: DE Free format text: PREVIOUS MAIN CLASS: G06F0017300000 Ipc: G06F0016000000 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 |
|
26N | No opposition filed |
Effective date: 20181005 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180313 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180331 Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180331 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180313 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180103 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20140313 Ref country code: MK Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180103 |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: 732E Free format text: REGISTERED BETWEEN 20200924 AND 20200930 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R082 Ref document number: 602014019377 Country of ref document: DE Representative=s name: BARDEHLE PAGENBERG PARTNERSCHAFT MBB PATENTANW, DE Ref country code: DE Ref legal event code: R081 Ref document number: 602014019377 Country of ref document: DE Owner name: APPLE INC., CUPERTINO, US Free format text: FORMER OWNER: SHAZAM INVESTMENTS LIMITED, LONDON, GB |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: IT Payment date: 20210211 Year of fee payment: 8 |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: PD Owner name: APPLE INC.; US Free format text: DETAILS ASSIGNMENT: CHANGE OF OWNER(S), ASSIGNMENT; FORMER OWNER NAME: KORSTEN MARIUS Effective date: 20201020 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: IE Payment date: 20230110 Year of fee payment: 10 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20230119 Year of fee payment: 10 Ref country code: DE Payment date: 20230117 Year of fee payment: 10 Ref country code: BE Payment date: 20230216 Year of fee payment: 10 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230526 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20231229 Year of fee payment: 11 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: IE Payment date: 20240103 Year of fee payment: 11 |