JP2014530390A - Identifying products using multimedia search - Google Patents

Abstract

By extracting at least one decoded frame from the television transmission, a product in the currently watched television program can be identified. The frame can request an image search and can be sent to another mobile device to receive the search results. The product can be identified using the search result. [Selection] Figure 1

Description

  The present invention relates generally to computers, and more specifically to image analysis by computer processing.

  Television images can be distributed by broadcasting television programs using radio frequency transmission of analog or digital signals. Furthermore, television programs can be distributed via cable systems and satellite broadcasting systems. Eventually, television images can be distributed over the Internet using streaming. As used herein, the term “television transmission” includes all of these television video distribution formats. As used herein, “television video” refers to the distribution of program content, with or without commercials, and includes both the distribution of traditional television programs and video games.

  Systems are known that determine which program a user is viewing. For example, the IntoNow service records audio signals from a TV program being viewed on a mobile phone, analyzes those signals, and uses the information to determine which program the viewer is viewing. . One problem with speech analysis is that it suffers from degradation from ambient noise. Of course, ambient noise in the viewing environment is common and therefore speech based systems are significantly constrained.

FIG. 3 illustrates a high level architecture of one embodiment of the present invention. 1 is a block diagram of a set top box according to an embodiment of the present invention. FIG. 5 is a flow diagram for a mobile grabber according to an embodiment of the present invention. 3 is a flow diagram for a multimedia grabber according to an embodiment of the invention. 5 is a flow diagram for a shopping application according to an embodiment of the present invention. It is a flowchart regarding the sequence which maintains and manages the table by one Embodiment.

  According to some embodiments, one or more viewers are actively tuned and from a currently watching television channel, a video frame or video clip, metadata or audio duration limited electronic A multimedia segment such as a static expression can be captured. In one embodiment, the multimedia segment can then be transmitted to the mobile device. The mobile device can then send the information to the search server to identify the product represented in the television program. For example, ultimately, an image search can be used to determine which product is represented. Once the product is identified, a variety of other shopping services can be provided to the viewer. These services can include identifying other vendors of the product, price comparison and retail location services.

  With reference to FIG. 1, a television screen 20 can be coupled to a processor-based device 14 that is further coupled to a television transmission 12. This transmission can be delivered via the Internet or via broadcast radio waves, including analog or digital radio frequency broadcast, cable delivery, satellite broadcast delivery. The processor utilization system 14 can be a stand-alone device separate from the television receiver, or can be integrated within the television receiver. For example, the system can include conventional set-top box components and, in some embodiments, can be responsible for decoding received television transmissions.

  In one embodiment, the processor-based system 14 may receive a video frame or video clip (ie, a video frame or video clip) from a decoded television transmission that is currently tuned by a receiver (which may be part of the device 14 in one embodiment). , A series of frames), a multimedia grabber 16 that captures an electronic representation of metadata or audio. The processor utilization system 14 may also include a wired or wireless interface 18 that allows captured multimedia to be transmitted to the external control device 24. This transmission 22 is widely available in television receivers and set-top boxes, via a wired connection such as a universal serial bus (USB) connection, or a transmission medium using radio frequency signals and transmission using optical signals. This can be done via any available transmission medium, including media.

  In other embodiments, undecoded content can be captured and subsequently decoded at the control device 24 or elsewhere.

  The control device 24 can be a mobile device including a remote control for a mobile phone, laptop computer, tablet computer, mobile internet device, or television receiver, to name a few. Device 24 may be other than mobile, such as a desktop computer or an entertainment system. In one embodiment, device 24 and system 14 may be part of a wireless home network. In general, device 24 has its own separate display so that information can be displayed separately from the television display screen. In embodiments where the device 24 does not include its own display, the display can be displayed over the television display, such as by a picture-in-picture display.

  In one embodiment, the control device 24 can communicate with the cloud 28. If device 24 is, for example, a mobile phone, it can communicate with the cloud by mobile phone signal 26, which is ultimately carried over the Internet. In other cases, the device 24 can communicate with the Internet through a wired connection, such as a network connection. As yet another example, the device 24 may communicate via the same transport medium that transported the television transmission. For example, in the case of a cable system, the device 24 can provide signals to the cable head end or server 11 through the cable system. Of course, in some embodiments, this may consume some of the available transmission bandwidth. Thus, in some embodiments, the device 24 can be other than a mobile device and can also be part of the processor utilization system 14.

  Referring to FIG. 2, one embodiment of a processor utilization system 14 is depicted, but many other architectures can be used. The architecture depicted in FIG. 2 corresponds to the CE4100 platform commercially available from Intel. The architecture includes a central processing unit 24 that is coupled to a system interconnect 25. The system interconnect is coupled to a NAND controller 26, a multi-format hardware decoder 28, a display processor 30, a graphics processor 32 and a video display controller 34. In one embodiment, decoder 28 and processors 30 and 32 may be coupled to controller 22.

  The system interconnect may be coupled to a transport processor 36, a security processor 38, and a dual audio digital signal processor (DSP) 40. Digital signal processor 40 may be responsible for decoding incoming video transmissions. The general purpose input / output (I / O) module 42 may be coupled to a wireless adapter such as, for example, the WiFi adapter 18a. In some embodiments, this adapter allows signals to be transmitted to the wireless control device 24. Also coupled to the system interconnect 25 are audio and video input / output devices 44. This device 44 can provide a decoded video output and, in some embodiments, can be used to output audio or video frames or audio or video clips.

  In some embodiments, the processor utilization system 14 can be programmed to output a multimedia segment when certain criteria are met. One such criterion is user selection by providing input through an input / output device such as a keyboard or touch screen, for example. The video camera can also record user gestures. Those gestures can be analyzed to identify commands that capture multimedia segments. In such a case, a video multimedia signal is output based on the command. Also, for example, the detection of an audible command from the viewer using voice recognition can be used to trigger the capture of the multimedia segment. Another option is that the processor-based system 14 detects various activities in the incoming video transmission to trigger multimedia capture. Examples of activities or events include the detection of commercial starts.

  FIG. 3 shows a sequence for one embodiment of the control device 24. This sequence can be implemented in software, hardware and / or firmware. In embodiments based on software or firmware, the sequence can be realized by computer-executable instructions stored in a non-transitory computer-readable medium such as an optical, magnetic or semiconductor storage device. For example, software or firmware sequences can be stored in the storage device 50 on the control device 24.

  Although embodiments are shown in which the control device 24 is a mobile device, embodiments other than mobile are also contemplated. For example, the control device 24 may be integrated within the system 14.

  First, as shown in the determination unit 52, a check in the determination unit 52 determines whether the grabber 16 has been activated. In some embodiments, the grabber 16 is not necessarily operating so as not to exhaust the computing power of the device 24. For example, the user can activate the application on his / her mobile phone and start the capturing operation. In such a case, the determination unit 52 detects the grabber operation.

  Thereafter, at block 54, a signal may be transmitted from the control device 24 to the processor utilization system 14 to initiate multimedia capture of the electronic representation of the multimedia segment 16. As detected in the decision unit 56, when the control device 24 receives a multimedia segment, in some embodiments, the control device 24 selects the multimedia segment to identify the product being displayed or represented. It can be sent to the analytic cloud 28 (block 58). Of course, in other embodiments, the multimedia segment can be sent to any server over the network. As another example, multimedia segments can be sent to the headend 11 for image, text or audio analysis.

  If an electronic representation of speech is captured, the captured speech representation can be converted to text, for example, at control device 24, system 14, or cloud 28. The text can then be searched to identify the product.

  Similarly, metadata can be analyzed to identify information for use in text searches that identify products. In some embodiments, two or more of audio, metadata, video frames, or video clips can be used as keywords internet or database search input to identify a product. In addition, the user can push information to a friend via a social network in the hope of receiving product information from the friend.

  The analysis engine then performs a multimedia search to identify the represented product. This search can be a simple internet or database search, or it can be a more refined search. For example, the transmission at block 58 may include the current time of video capture and the location of the control device 24. This information can be used to narrow the search using information about which products are displayed at a specific time or location. For example, a website can be provided with a database that correlates television programs available at various locations at various times, and this database can be image searched to find an image that matches the captured frame; The program can be identified. In addition, metadata or advertising content providers can include location or contact information in connection with the content they provide.

  In some embodiments, the user can add annotations to identify target features in the captured segment. In one embodiment, the annotation can be enabled by an application running on the control device 24. Annotations can be used to narrow the search. As another option, gaze detection can be used to identify the target product within a video frame or video clip.

  The product can be identified by using a visual search tool. The image frame or image clip is matched with an existing frame or clip in the search database. In some cases, a series of matches can be identified, and in such cases, those matches can be returned to the control device 24. When the check at the decision unit 60 determines that the search results have been received by the control device 24, the search results can be displayed to the user, as shown at block 62. Thereafter, the control device 24 receives a user selection of one of the search results that matches the information desired by the user, such as a viewed product. Thereafter, as shown in the decision unit 64, when a user selection is received, the selected search results can then be transferred to the cloud, as shown in block 66. This makes it possible to provide other services such as provision of additional information about products to viewers or third parties using product identification on a television.

  Next, referring to FIG. 4, the sequence may be performed within the processor utilization system 14. Again, the sequence can be implemented in firmware, hardware or software. In a software or firmware embodiment, the sequence can be implemented by one or more non-transitory computer readable media. For example, the multimedia grabber sequence can be stored in the storage device 70 on the multimedia grabber device 16.

  Initially, a check at the decision unit 72 determines whether the grabber mechanism has been activated. In some embodiments, video content analysis may be used. For example, a user may request a system screen for a particular product, such as a laptop computer, or an advertisement for a laptop computer, so the system uses video content analysis to analyze the content in progress and The location of the product can be identified and the multimedia segment on which the product is displayed or represented can be captured.

  If a command is received, as determined at decision unit 76, the multimedia is captured and transmitted to control device 24, as indicated at block 78.

  Referring to FIG. 5, the shopping application is shown by a sequence. The sequence can be implemented in software, firmware and / or hardware. In software or firmware based embodiments, the sequence can be implemented by one or more non-transitory computer readable media. For example, computer readable instructions may be stored on storage device 80 associated with server 30 shown in FIG.

  While embodiments using the cloud are illustrated, in other embodiments, the same sequence can be performed by any server coupled via any suitable network, by the control device 24 itself, by the processor utilization system 14, or May be implemented by the head end 11.

  Initially, a check at decision unit 82 determines whether a multimedia segment has been received. If received, if the multimedia is an electronic representation of a video frame or video clip, a visual search is performed, as indicated at block 84. In the case of an audio clip, the audio can be converted into text and searched. If the multimedia segment is metadata, the metadata can be parsed to obtain searchable content. Thereafter, at block 86, the search results are returned to the control device 24, for example. The control device 24 can receive user input or user selection as to which of the search results is most relevant. The system waits for a selection from the user and can perform a task based on the identified product as shown in block 90 when a selection is received, as determined at decision unit 88. For example, a search can be performed to identify other suppliers of the same product, and a vendor comparison can be automatically performed based on, for example, price, location, and purchaseability.

  One way in which such a search can be performed is by matching the current image with images in a database or on the Internet, and then searching for text associated with those images that exist in the Internet or database. There may be. A common term between different images can be analyzed to determine the product name. In this way, the product name can be determined using the image search. Similarly, search the audio segment in the multimedia segment to see if the product name was actually mentioned, and if so, convert the audio to text and then the product in the text Information can be searched.

  In addition, the user can provide input information that provides a clue as to why the user has selected a particular image. This can be accomplished, for example, using a text entry field, annotations on selected messages, or another communication.

  In the determination unit 102, the user can be asked whether he or she wants to purchase the product now. This may mean, for example, purchasing the product indicated in the television program through a television shopping network option or through one of the vendors identified in the search.

  If the user wants to purchase the product now, the system can assist the purchase process. For example, heuristic methods can be used to identify contact information from within web or database information. Using this information, the purchase transaction can be initiated by providing the user's credit card information and address information and filling in an online form. The information can then be conveyed to the vendor and the transaction can be started automatically. Alternatively, contact information in a database of Internet web pages identified in the search can be identified and the information can be provided to the user for the user to select a vendor.

  If the user decides not to purchase now, the user can select a particular vendor that they may want to visit to view the product. In this way, if the user selects a web page for a particular merchant, the merchant location or contact information can be automatically parsed from the web page (block 104). This can be accomplished by recognizing information that is in the format of address information, which may include numbers followed by text, or web page information based on that particular format. Can be identified. Similarly, telephone numbers or fax numbers can be identified in the same way. Once the location or contact information is identified, the location is recorded as shown at block 106.

  The user can specify the proximity factor at this point or during setup. For example, a user may want to be identified when within a given distance of an identified merchant. The check in the determination unit 108 determines whether or not the proximity determination criterion is satisfied. If so, the current location and the recorded location can be compared (block 110), and if those locations match, as determined by the decision unit 112, the user can At 114, it may be notified that the instructed vendor is within a specified distance. In this way, the system can constantly monitor the user's location using a global positioning system sensor in a mobile phone or other mobile device, and simply inform the user when the user is near their merchant. .

  This location monitoring in the background reduces the need for the user to see the product immediately in most cases. Instead, the user can continue with his normal activities and the system will monitor the user's location. Notifications can be given when the user is very close to the identified merchant.

  Similar services can be implemented in other ways. For example, the user can take a picture of a product in the store, give some identification information, or the system can autonomously identify the product, using the same technique, the same The location of other vendors of the product can be identified.

  In addition, location-directed services may be useful even if a product is not identified through a television program or photo. For example, a user may simply see an advertisement that mentions a vendor, or may ask about a store, restaurant, museum, or any other location that the user wants to visit sometime. The user can provide an indication of the location, proximity criteria, and then the system continuously monitors the user's location to detect when the user is near the location for another reason. At that time, it is possible to notify the user that they are close to each other, and when selected, it is possible to give directions to the seller. This saves time and money because there is no need to go to the vendor.

  In some embodiments, multiple users may be watching the same television program. Depending on the household, multiple TVs may be available. Accordingly, many different users may wish to use the services described herein simultaneously. For this reason, the processor utilization system 14 can hold a table for identifying the identifier of the control device 24, the television identifier, and the program information. Thus, in such an embodiment, the processor utilization system 14 simply adapts to various televisions, and all televisions receive signals downstream of the processor utilization system 14 so that a user can move from room to room. It will be possible to continue to receive the services described herein.

  In some embodiments, the table can be stored in the processor utilization system 14, uploaded to the headend 11, or in some cases uploaded to the cloud 28 through the control device 24. .

  Thus, referring to FIG. 6, in some embodiments, the sequence 92 can be used to maintain the table to correlate the control device 24, the television display screen 20, and the selected channel. In that case, the system can be used on at least two or more televisions that are connected by different users through the same television or, for example, all through the same processor utilization system 14 within the home entertainment network. The sequence can be implemented as hardware, software and / or firmware. In software and firmware embodiments, the sequence can be implemented using computer readable instructions stored on one or more non-transitory computer readable media, such as magnetic, semiconductor, or optical storage devices. In one embodiment, the storage device 50 may be used to store those instructions.

  Initially, as shown at block 94, the system 14 receives and stores an identifier for each control device that provides commands to the system. Thereafter, as shown at block 96, the various televisions coupled through the system 14 can be identified and recorded. Finally, a table for correlating the control device and the television receiver is set (block 100). This allows multiple TVs connected to the same control device to be used without interruption, so that the viewer can continue to receive the services described herein even when the viewer moves from room to room. Become. Further, multiple viewers can watch the same television, and each can independently receive the services described herein.

  Reference to “one embodiment” or “one embodiment” throughout this specification refers to at least one implementation in which the particular feature, structure or characteristic described in connection with that embodiment is included in the invention. It is meant to be included in the embodiment. Thus, the appearance of the phrase “in one embodiment” or “in one embodiment” does not necessarily refer to the same embodiment. Furthermore, the particular features, structures or characteristics may be implemented in other suitable forms other than the particular embodiments illustrated, and all such forms are encompassed by the claims herein. be able to.

  Although the present invention has been described with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations from these embodiments. The appended claims are intended to cover all such modifications and variations that fall within the true spirit and scope of this invention.

Although the present invention has been described with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations from these embodiments. The appended claims are intended to cover all such modifications and variations that fall within the true spirit and scope of this invention.
[Item 1]
Detecting the occurrence of an event,
In response to detecting an event, automatically capturing an electronically decoded signal from a television program;
Performing a search using the signal to facilitate identification of a product represented in the television program.
[Item 2]
The method of item 1, comprising capturing a signal comprising a video frame or video clip, audio or an electronic representation of metadata.
[Item 3]
The method of item 1, comprising automatically transferring the signal to a mobile device.
[Item 4]
4. The method of item 3, comprising providing a search result to the mobile device.
[Item 5]
4. The method of item 3, comprising sending the signal to a remote server and performing the search.
[Item 6]
The method of item 1, comprising tracking a plurality of mobile devices, receiving a request from each of the plurality of mobile devices, and providing a response to each of the plurality of mobile devices.
[Item 7]
7. The method of item 6, comprising maintaining a table that correlates mobile devices, televisions, and requests from mobile devices.
[Item 8]
A method according to item 1, comprising automatically providing information about a seller of the product.
[Item 9]
The method of item 1, comprising allowing a user to use one mobile device and access two different televisions at different times.
[Item 10]
At least one non-transitory computer readable medium comprising:
Instructions that allow the computer to detect the occurrence of the event;
Instructions that allow the computer to automatically capture an image in response to detecting an event;
At least one non-transitory computer-readable medium that stores instructions that allow the computer to initiate a search using the image and that facilitates identification of a product represented in the image .
[Item 11]
Item 11. The medium of item 10, further storing instructions for capturing a decoded signal of electronic communication in the form of an electronic representation of a video frame or video clip, audio or metadata from a television program.
[Item 12]
Item 11. The medium of item 10, further storing instructions for transferring the signal to a mobile device.
[Item 13]
Item 13. The medium according to Item 12, further storing instructions for providing a search result to the mobile device.
[Item 14]
Item 13. The medium of item 12, further storing instructions for transmitting the signal to a remote server to perform the search.
[Item 15]
Instructions for tracking a plurality of mobile devices, instructions for receiving a request from each of the plurality of mobile devices, and instructions for giving a response to each mobile device so that two different televisions can be used at different times Item 11. The medium according to Item 10, further storing instructions.
[Item 16]
Item 16. The medium of item 15, further storing instructions for maintaining a table correlating device, television, and mobile device requests.
[Item 17]
Item 10 further stores instructions for capturing a signal that is an electronic representation of a speech signal, instructions for converting the captured signal to text, and instructions for transmitting the text for use as a keyword search input. The medium described in 1.
[Item 18]
Item 11. The medium according to Item 10, further storing an instruction for giving information about a seller of the product.
[Item 19]
A processor that automatically captures an electronic communication signal from a television program in response to an event and transmits the decoded signal for use as an input for keyword search to identify the product represented in the signal When,
A storage device coupled to the processor.
[Item 20]
Item 20. The device of item 19, wherein the device is a mobile device.
[Item 21]
Item 21. The device according to Item 20, wherein the device is a mobile phone.
[Item 22]
Item 21. The device according to Item 20, wherein the device is a remote control.
[Item 23]
Item 20. The device according to Item 19, wherein the device is a television receiver.
[Item 24]
Item 20. The device of item 19, wherein the device sends a signal to a television receiving system and captures a decoded signal of electronic communication in the form of an electronic representation of a video frame or video clip, audio or metadata.
[Item 25]
Item 21. The apparatus of item 20, wherein the apparatus receives the signal from a television system and transmits the signal to a remote device that performs keyword searches in a database or via the Internet.
[Item 26]
At least one non-transitory computer readable medium comprising:
Instructions that allow the computer to receive the specified location;
Instructions that allow the computer to monitor the user's current location;
At least one non-transitory computer-readable medium that stores instructions that allow the computer to notify the user when the user is within a predetermined distance from the designated location.
[Item 27]
27. A medium according to item 26, further storing instructions for searching for a captured electronic representation of the product and instructions for identifying the product using an image search.
[Item 28]
28. A medium according to item 27, further storing an instruction for searching for the captured electronic television signal to identify the product.
[Item 29]
Item 29. The medium according to Item 28, further storing an instruction for deriving a location of a product seller from an Internet search result related to the product.
[Item 30]
27. A medium according to item 26, further storing instructions for analyzing audio from a television program and identifying the program.

Claims (30)

Detecting the occurrence of an event,
In response to detecting an event, automatically capturing an electronically decoded signal from a television program;
Performing a search using the signal to facilitate identification of a product represented in the television program.   The method of claim 1, comprising capturing a signal that includes an electronic representation of a video frame or video clip, audio or metadata.   The method of claim 1, comprising automatically transferring the signal to a mobile device.   4. The method of claim 3, comprising providing search results to the mobile device.   4. The method of claim 3, comprising sending the signal to a remote server to perform the search.   The method of claim 1, comprising tracking a plurality of mobile devices, receiving a request from each of the plurality of mobile devices, and providing a response to each of the plurality of mobile devices.   7. The method of claim 6, comprising maintaining a table that correlates mobile devices, televisions, and requests from mobile devices.   The method of claim 1, comprising automatically providing information about a vendor of the product.   The method of claim 1, comprising allowing a user to use one mobile device to access two different televisions at different times. At least one non-transitory computer readable medium comprising:
Instructions that allow the computer to detect the occurrence of the event;
Instructions that allow the computer to automatically capture an image in response to detecting an event;
At least one non-transitory computer-readable medium that stores instructions that allow the computer to initiate a search using the image and that facilitates identification of a product represented in the image .   The non-transitory computer-readable medium of claim 10, further storing instructions for capturing a decoded signal of electronic communication in the form of an electronic representation of a video frame or video clip, audio or metadata from a television program.   The non-transitory computer readable medium of claim 10, further storing instructions for transferring the signal to a mobile device.   The non-transitory computer readable medium of claim 12, further storing instructions for providing search results to the mobile device.   The non-transitory computer readable medium of claim 12, further storing instructions for sending the signal to a remote server to perform the search.   Instructions for tracking a plurality of mobile devices, instructions for receiving a request from each of the plurality of mobile devices, and instructions for giving a response to each mobile device so that two different televisions can be used at different times The non-transitory computer readable medium of claim 10, further storing instructions.   The non-transitory computer-readable medium of claim 15, further storing instructions for maintaining a table that correlates device, television, and mobile device requests.   An instruction for capturing a signal that is an electronic representation of an audio signal, an instruction for converting the captured signal to text, and an instruction for transmitting the text for use as an input for keyword search are further stored. A non-transitory computer readable medium according to claim 10.   The non-transitory computer readable medium of claim 10, further storing instructions that provide information about a vendor of the product. A processor that automatically captures an electronic communication signal from a television program in response to an event and transmits the decoded signal to identify the product represented in the signal for use as a keyword search input When,
A storage device coupled to the processor.   The apparatus of claim 19, wherein the apparatus is a mobile device.   21. The device of claim 20, wherein the device is a mobile phone.   21. The device of claim 20, wherein the device is a remote control.   The apparatus of claim 19, wherein the apparatus is a television receiver.   21. The apparatus of claim 19, wherein the apparatus sends a signal to a television receiving system and captures an electronic communication decoded signal in the form of an electronic representation of a video frame or video clip, audio or metadata.   21. The apparatus of claim 20, wherein the apparatus receives the signal from a television system and transmits the signal to a remote device that performs keyword searches in a database or over the Internet. At least one non-transitory computer readable medium comprising:
Instructions that allow the computer to receive the specified location;
Instructions that allow the computer to monitor a user's current location;
At least one non-transitory computer-readable medium that stores instructions that allow the computer to notify the user when the user is within a predetermined distance from the designated location.   27. The non-transitory computer readable medium of claim 26, further storing instructions for searching for a captured electronic representation of a product and instructions for identifying the product using image search.   28. The non-transitory computer readable medium of claim 27, further storing instructions for searching for captured electronic television signals to identify the product.   30. The non-transitory computer readable medium of claim 28, further storing instructions for deriving a product vendor location from internet search results associated with the product.   27. The non-transitory computer readable medium of claim 26, further storing instructions for analyzing audio from a television program to identify the program.

Images