Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/40—Scenes; Scene-specific elements in video content
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/40—Scenes; Scene-specific elements in video content
  • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
  • G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
  • G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
  • G06F16/7847—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using low-level visual features of the video content
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/40—Extraction of image or video features
  • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
  • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]

Definitions

• Described embodiments relate generally to product annotation in a video and specifically to in-video product annotation using web information mining.
• Video annotation also widely known as video concept detection or high-level feature extraction
• video concept detection or high-level feature extraction which aims to automatically assign descriptive concepts to video content
• high level concepts such as events (e.g., airplane crash and running), scenes (e.g., sundown and beach) and object categories (e.g., car and screen)
• object categories e.g., car and screen
• Annotation of product concepts is of great importance to many applications such
• the second challenge is the effective visual representation. Bag of Visual Words
• Scale Invariant Feature Transform SIFT
• a BoVW histogram is generated to describe the product image.
• the descriptors of an image are about the whole image and not the product parts contained in the image and contain a lot noise for product annotation.
• Embodiments of the invention provide product annotation in a video to one or more users using product training images from web mining.
• a computer system provides services of product annotation in a video to one or more users.
• the system receives a video from a user, where the video includes multiple video frames.
• the system extracts multiple key frames from the video and generates a visual representation of the key frame.
• the system compares the visual representation of the key frame with a plurality of product visual signatures, where each visual signature identifies a product.
• the system collects multiple training images comprising multiple of expert product images obtained from an expert product repository, each of which is associated with multiple product images obtained from multiple web resources. Based on the comparison of the visual representation of the key frame and a product visual signature, the system determines whether the key frame contains the product identified by the visual signature of the product.
• FIG. 1 is a block diagram of a computing environment configured to provide in- video product annotation service to clients.
• FIG. 2 is a block diagram of an in- video product annotation module to generate product visual signatures and annotate products detected in a video stream.
• FIG. 3 is an example of collecting training images for in- video product annotation process according to one embodiment of the invention.
• FIG. 4 is an example of product images for collectively generating product visual signatures.
• FIG. 5 is a flow chart of a process for generating a visual signature of a product according to one embodiment of the invention.
• FIG. 6 is a flow chart of a process for detecting a product and annotating the detected product in one or more video frames of a videos stream according to one embodiment of the invention.
• FIG. 7 is an example of an in- video product annotation system according to one embodiment of the invention.
• FIG. 8 is an example result of an in- video product annotation process according to one embodiment of the invention.
• FIG. 1 is a block diagram of a computing environment 100 configured to provide in-video product annotation service to clients 1 10.
• Multiple users/viewers use clients 1 10A-N to provide video streams to an in-video product annotation service 120 and request the in-video product annotation service 120 to annotate the products contained in the video frames of the video streams.
• the product annotation service 120 stores the video streams and responds to the requests with product detection and annotation results to the clients 1 10.
• Each client 110 executes a browser 1 12 for browsing the video streams and product annotation results from the product annotation service 120.
• Other embodiments can have different configurations.
• each client 1 10 is used by a user to use services provided by the in- video product annotation service 120.
• a user uses a client 1 10 to browse a video, request annotation of a product contained in the video and receive the product detection and annotation results from the product annotation service 120.
• the client 110 can be any type of computer device, such as a personal computer (e.g., desktop, notebook, and laptop) computer, as well as devices such as a mobile telephone or personal digital assistant that has the capability to record video content.
• the client 110 typically includes a processor, a display device (or output to a display device), a local storage, such as a hard drive or flash memory device, to which the client 1 10 stores data used by the user in performing tasks, and a network interface for coupling to the in-video product annotation service 120 via the network 130.
• the network 130 enables communications between the clients 1 10 and the in- video product annotation service 120.
• the network 130 is the Internet, and uses standardized internetworking communications technologies and protocols, known now or subsequently developed that enable the clients 110 to communicate with the in- video product annotation service 120.
• the network 130 is a cloud computing network and includes one or more components of the in- video product annotation service 120.
• the visual signature generation stage includes three components: collection of high-quality visual examples of products from a repository, e.g., AMAZONTM, expansion of the collected visual examples with Internet product images search results, and generation of visual signature from training examples comprising the high-quality visual examples of products and their
• the visual signatures of a variety of known products are stored in a product visual signature file.
• the runtime video processing stage includes two components, feature extraction and product annotation.
• the product annotation service 120 For an input video stream, the product annotation service 120 identifies a set of key frames of the video stream, and for each key frame, the product annotation service 120 extracts visual features (e.g., Scale Invariant Feature Transform (SIFT) descriptors) and generates a visual representation of the extracted features (e.g., the Bag of Visual Words (BoVW) histograms).
• SIFT Scale Invariant Feature Transform
• BoVW Bag of Visual Words
• the product annotation service 120 performs production annotation by comparing the visual signature of each product stored in the visual signature file with the BoVW histogram of each key frame of the input video.
• the in-video product annotation service In the embodiment illustrated in FIG. 1 , the in-video product annotation service
• the 120 has an in-video product annotation module 102, a video server 104 and a product images database 106.
• the in- video product annotation module 102 comprises a product visual signature generation module 200 for product visual signature generation and a video processing module 300 for processing input videos from clients 110.
• the video server 104 stores the videos streams received from the clients 110 and annotated video frames of the videos streams.
• the product images database 106 comprises two sub-databases: database 1 (106A) and database 2 (106B), to store high-quality product images obtained from one or more online product merchants, such as AMAZONTM, and related product images collected through Internet search.
• the product images from a known product merchant generally have high visual quality, but the number of them for a given product can be limited.
• the number of related images of the product through Internet search using various search engines, such as GOOGLETM can be large but noisy (e.g., containing textual information unrelated to the product).
• the product annotation service 120 filters the related product images obtained from the Internet search results based on the high quality product images to generate product visual signatures, and uses the product visual signatures to detect and annotate products in a video stream.
• the high-quality product images from a known merchant are referred to as "expert product images,” and for a given expert product image, its associated images obtained from Internet search are referred to as “extended product images.”
• extended product images IN-VIDHO PRODUCT ANNO TA TION VISUAL SIGNA TURE GENERATION
• FIG. 2 is a block diagram of an in- video product annotation module 102 to generate product visual signatures and annotate products detected in a video stream according to one embodiment.
• the product annotation module 102 comprises a product visual signature generation module 200 and a video processing module 300.
• the product visual signature generation module 200 includes an expert product images module 210, an extended product images module 220 and a visual signature generation module 230.
• the video processing module 300 includes a frame extraction module 310, a feature extraction and quantization module 320 and a product annotation module 330.
• the product visual signature generation module 200 is configured to generate product visual signatures.
• the expert product images module 210 is configured to collect high- quality visual examples of products (e.g., expert product images in different views, such as frontal, side and back views). In one embodiment, the expert product images module 210 collects the expert product images from AMAZONTM for a variety of consumer products, such as digital cameras, cars and digital phones.
• the expert product images of a given product are often too few for constructing a good visual signature for the product.
• numbers of expert product images collected from AMAZONTM for a product vary from 1 to 8.
• the extended product images module 220 is configured to collect, from the Internet, associated images of a product that has one or more expert product images.
• the product name is used as a search query for associated product images in the Internet using GOOGLETM search engine. The process is to expand the expert product images using the web product image database.
• the images from the Internet search contain a lot of noise, e.g., text information
• the signature generation module 230 Before the signature generation module 230 generates visual signatures of products, the signature generation module 230 re-ranks the extended product images from the Internet search results based on the expert product images. For each expert product image, a pre-determined number of extended product images that are near to the expert product image are selected as the result of the filtering. For a given product, the expert product images and the filtered extended product images form a set of positive training images for the product, from which the signature generation module 230 generates a visual signature for the product.
• the collection of training images of known products can be automated to improve the in-video product annotation system performance.
• the signature generation module 230 extracts the visual features of the expert product image and its associated extended product images.
• the visual features of the product images are Bag of Visual Words (BoVW) features.
• the signature generation module 230 extracts one or more SIFT descriptors on several detected key points or by densely sampling patches of each product image and quantizes the SIFT descriptors into multiple visual words.
• a BoVW histogram is generated from the quantized SIFT descriptors to describe each image.
• the signature generation module 230 uses a visual feature detection and extraction method, e.g., Difference-of-Gaussian method, to extract 128-dimension SIFT features from a product image and groups the SIFT features into 160, 000 clusters with hierarchical K-means.
• the product image is represented by a 160,000-dimensional BoVW histogram.
• the signature generation module 230 selects a predetermined a number of nearest neighbors from the extended product images associated with the expert product image based on a similarity measure defined in Equation (1) below:
• the signature generation module 230 obtains kn positive training images for a given product, where k is the number of expert product images and n is the pre-determined nearest neighbors (i.e., the extended product images) of the expert product image.
• FIG. 3 provides an example of training data collection process for digital camera Canon 40D.
• the product visual signature generation module 200 collects five expert product images 302 of the camera in different views from online merchant AMAZONTM. For each expert product image, the product visual signature generation module 200 searches the Internet using GOOGLETM search engine to collect a number of related product images 304. Because the product images obtained from the Internet search can be noisy (e.g., containing text unrelated to the product), the product visual signature generation module 200 filters the related product images based on the expert product images.
• the product visual signature generation module 200 applies a correlative sparsification described below to reduce the noise by selecting a pre-determined number of nearest neighbors of the product images from the Internet search.
• the selection of related product image is based on a similarity measure between the related product image and its corresponding expert product image.
• the product visual signature generation module 200 obtains a set of training examples 306 for digital camera Canon 40D, where the product visual signature generation module 200 generates a visual signature for digital camera Canon 40D.
• the signature generation module 230 To effectively annotate a product contained in a product image represented in a high dimensional feature space, the signature generation module 230 generates a template for annotation by averaging positive training images of the product. In one embodiment, the signature generation module 230 merges the visual representation of multiple training images of a product to generate an accumulated histogram for the product. Since there are many noises caused by the descriptors from image background, there are actually many noisy bins in the accumulated histogram.
• Equation (2) which fits the LI -regularized least square optimization problem, where indicate the 2-norm and 1-norm, respectively.
• the parameter ⁇ modulates
• Equation (2) keeps the obtained signature to be close to the original one, while the second term minimizes the 1 -norm value of the obtained visual signature, which makes the signature sparse.
• the signature generation module 230 generates visual signatures of products collectively.
• the signature generation module 230 modifies the visual signature generation defined in Equation (2) by adding a graph Laplacian term to Equation (2) as follows:
• Equation (3) can be solved using an optimization approach. Assuming that all the visual signatures except v, are fixed. The problem described by Equation (3) can be rewritten as Equation (4) below:
• the visual signature v is defined by Equation
• the signature generation module 230 uses an interior-point method to solve the problem defined in Equation (5).
• the visual signature of a product represents the ground truth of the product, which can be used to determine whether a video frame of a video stream contains the product at runtime.
• the similarity between two sets of product images is defined by Equation (6) as: where the number of images for image sets indicates the &-th product in the set P t , and sim(.,) is the similarity of an image pair from different sets.
• Equation (6) The similarity measure defined in Equation (6) has the following properties:
• Wij Wji,: the similarity is symmetry
• the similarity sim(.,.) of an image pair from different sets is calculated from the histogram intersection for the image pair described in Equation (1).
• the similarity of two products belonging to two different subcategories of a product e.g., video games and portable audio/video products under the same class of product "electronics" is set to zero.
• FIG. 4 is an example of three sets of product images for collectively generating product visual signatures according to one embodiment of the invention.
• the example in FIG. 4 contains three sets of product images for three products: product image set 410 is for digital camera Canon 40D; product image set 420 is for digital camera Nikon D90; and product image set 430 is for video game console Xbox. It is noted that the products Canon 40D and Nikon D90 have very close appearances because they belong to same class of product. Generating visual signatures of products collectively enables the visual signatures of products to reflect the closeness of images of products of a same class.
• the signature generation module 230 can derive an iterative process to solve each v ; by repeatedly updating each v, .
• An example pseudo code for iteratively updating the visual signature v is the following:
• FIG. 5 is a flow chart of a process for generating a visual signature of a product according to one embodiment of the invention.
• the product visual signature generation module 200 searches 510 for expert product images from a repository for a product and stores 520 the expert product images in a storage place (e.g., the database 1 of the product images database 106 of FIG. 1).
• the product visual signature generation module 200 collects 530 multiple related product images through web mining (e.g., the Internet search).
• the product visual signature generation module 200 filters the related product images based on its corresponding expert product image and generates 540 training sample product images from the filtering.
• the product visual signature generation module 200 uses the training sample product images, the product visual signature generation module 200 generates 550 a visual signature for the product.
• the product visual signature generation module 200 compiles 560 a visual signature file containing the visual signatures for a variety of products.
• the in-video product annotation module 102 has a video processing module 300 to annotate product in one or more video frames of a video stream.
• the video processing module 300 receives a video stream from a client 110 and processes one or more selected video frames of the video stream. For each selected video frame, the video processing module 300 determine whether the video frame contains a known product using the product visual signatures provided by the product visual signature generation module 200.
• the video processing module 300 includes a video frame extraction module 310, a feature extract and quantization module 320 and a product annotation module 330.
• the video frame extraction module 310 receives a video stream consisting of multiple video frames and extracts a number of key frames from the video stream.
• One way to extract the key frames from the video stream is to select a video frame at a fixed point of the video stream, e.g., extracting a video frame every 5 seconds of the video stream.
• Other embodiments of the frame extraction module 310 can use different methods, e.g., selecting the first frame of every group of pictures (GOPs) of the video stream, to obtain key frames.
• GOPs group of pictures
• the feature extract and quantization module 320 is for extracting visual features from the key frames of the video stream and quantized the extracted visual features to generate a visual representation of each key frame.
• the feature extraction and quantization module 320 uses a Difference-of-Gaussian method to detect keypoints in a key frame and from each keypoiiit, the module 320 extracts 128-dimensional SIFT features.
• the module 320 groups the SIFT features into a number of clusters (e.g., 160,000 clusters) with hierarchical k-means.
• a key frame is represented by a multi-dimensional bag of visual words histogram (e.g., a 160,000-dimensiaonl BoVW histogram).
• the product annotation module 330 determines whether a key frame of a video stream contains a known product by comparing the product visual signatures with the visual representation (e.g., the 160,000-dimensiaonl BoVW histogram) of the key frame.
• the comparison between a product visual signature and the visual representation of the key frame is measured by a product relevance measure defined in Equation (7):
• the product annotation module 330 determines whether the key frame contains a known product. In one embodiment, the estimated product relevance measure is compared with a threshold value to determine whether the key frame contains a known product.
• FIG. 6 is a flow chart of a process for detecting product and annotating the detected product in one or more video frames of a videos stream according to one embodiment of the invention.
• the video processing module 300 receives 610 a video stream from a client 110 and extracts 620 multiple key frames from the video stream. For each key frame, the video processing module 300 extracts 630 visual features (e.g., SIFT features) of the key frame and generates 620 a visual presentation of the key frame (e.g., multi-dimensional BoVW histogram).
• the video processing module 300 compares 650 the visual representation of the key frame with each visual signature of known products. Based on the comparison, the video processing module 300 determines 660 whether the key frame contains a known product.
• FIG. 7 is an example of an in-video product annotation system according to one embodiment of the invention.
• the in-video product annotation system 700 has a product visual signature generation sub-system 701 A for generating a visual signature of digital camera Canon G9 702 off-line and a video processing sub-system 70 IB for processing selected video frames of a video stream 712 for product annotation at runtime.
• the product visual signature generation sub-system 701 A collects one or more expert product images at different views of Canon G9 from AMAZONTM 704 as product visual examples 706. For each expert product image of Canon G9, and the product visual signature generation sub-system 701 A collects multiple associated product images from GOOGLETM 710.
• the product visual signature generation sub-system 701 A filters the product images from the Internet search by a correlative sparsification method 708 to reduce the noise.
• the filtered product images and the associated expert images form a set of training images of Canon G9 and the product visual signature generation sub-system 701 A generates a visual signature for Canon G9.
• the video processing sub-system 70 IB receives a video stream 712 and extracts
• the video processing sub-system 70 IB compares the visual representation of the key frame with the visual signature of Canon G9, and based on the comparison, the video processing sub-system 101B determines whether the key frame contains the digital camera Canon G9. For illustration purpose, assuming that the product visual signature file compiled by the product visual signature generation 701 A contains only the visual signature of Canon G9. If the product visual signature file contains visual signatures of more products, the video processing sub-system 701B compares the visual presentation of a key frame to each of the visual signatures to determine whether the key frame contains the product identified by the visual signature.
• a visual representation of the key frame e.g., a BoVW histogram
• a Difference-of-Gaussian method was used to detect keypoints and from each keypoint 128-dimensional SIFT features were extracted.
• the SIFT features were grouped into 160,000 clusters with hierarchical k-means.
• Each product image is represented by a 160,000-dimensional BoVW histogram.
• image search engine There are 300 images used for each product.
• the parameters ⁇ and ⁇ 2 are empirically set to 5 and 0.05 respectively.
• the performance results (e.g., MAP results) of different combination of training data sources and annotation algorithms are demonstrated in Table I below.
• FIG. 8 is an example of a video frame processed by variations of product annotation algorithm and their corresponding visual words in a video frame.
• the left column contains the visual signatures (81 OA, 820A and 830A by different annotation methods.
• the visual signature 81 OA is generated by "No-sparse” method.
• the visual signature 820A is generated by "1-norm sparsification” method and the visual signature 830A is generated by "Correlative sparsification” method.
• the right column shows the corresponding visual words (810B, 820B and 830B) in a video frame.
• the examples in FIG. 8 show that sparsification methods are able to remove several noisy bins and thus the obtained visual signatures are better.
• in-video product annotation Another embodiment of in-video product annotation described above is multimodal product annotation, which makes use of both text information associated with a video stream.
• the text information associated with a video includes the video title, description and tags.
• the in- video product annotation performance can be further improved.
• the experimental results show that pure text-based method for in- video product annotation achieves an MAP measurement of 0.23 and it is worse than the MAP 0.39 achieved by only using visual information.
• the MAP measure can be boosted to 0.55.
• the embodiments of the invention advantageously provide in-video product annotation using correlative sparsification of product visual signatures.
• Sparse visual signatures of products are generated using expert product images and associated product images from web mining.
• the noise of the associated product images are reduced by the correlative sparsification.
• the extended training images of products from the web mining along with the expert product images enable the embodiments of the invention to generate product visual signatures, which are used to efficiently identify products contained in a video frame of a video stream at run time.
• the performance of the embodiments of the invention is computationally efficient. For example, after feature extraction, the annotation of a product for a video frame actually scales with the number of non-zeros bins of the visual signature. When annotating a large dataset, an inverted structure can be built by investigating the sparsity of the product visual signatures. Therefore, the sparsification of the product visual signatures not only improves annotation performance also reduces computational cost.
• Reference in the specification to "one embodiment” or to "an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment of the invention. The appearances of the phrase “in one embodiment” or "a preferred embodiment” in various places in the specification are not necessarily all referring to the same embodiment.
• Certain aspects of the invention include process steps and instructions described herein in the form of a method. It should be noted that the process steps and instructions of the invention can be embodied in software, firmware or hardware, and when embodied in software, can be downloaded to reside on and be operated from different platforms used by a variety of operating systems.
• the invention also relates to an apparatus for performing the operations herein.
• This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer.
• a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.
• the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Multimedia (AREA)
• General Physics & Mathematics (AREA)
• Physics & Mathematics (AREA)
• Library & Information Science (AREA)
• Data Mining & Analysis (AREA)
• General Engineering & Computer Science (AREA)
• Databases & Information Systems (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Bioinformatics & Computational Biology (AREA)
• Evolutionary Biology (AREA)
• Evolutionary Computation (AREA)
• Bioinformatics & Cheminformatics (AREA)
• Artificial Intelligence (AREA)
• Life Sciences & Earth Sciences (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Image Analysis (AREA)
• Electrically Operated Instructional Devices (AREA)
• Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Priority Applications (5)

Applications Claiming Priority (2)

Publications (1)

Family

ID=47009585

Family Applications (1)

Country Status (6)

Cited By (1)

Families Citing this family (19)

Citations (4)

Family Cites Families (10)

• 2012
  • 2012-04-11 CN CN201280027434.XA patent/CN103608826B/zh active Active
  • 2012-04-11 JP JP2014505107A patent/JP6049693B2/ja active Active
  • 2012-04-11 SG SG2013075056A patent/SG194442A1/en unknown
  • 2012-04-11 GB GB1319882.5A patent/GB2506028B/en active Active
  • 2012-04-11 US US14/111,149 patent/US9355330B2/en active Active
  • 2012-04-11 WO PCT/SG2012/000127 patent/WO2012141655A1/en not_active Ceased

Patent Citations (4)

Also Published As

Similar Documents

Legal Events