WO2010036375A1

WO2010036375A1 - Interactive communication system and method

Info

Publication number: WO2010036375A1
Application number: PCT/US2009/005362
Authority: WO
Inventors: Jonathan A. Kessler; Boris Ralchenko
Original assignee: Kessler Jonathan A; Boris Ralchenko
Priority date: 2008-09-29
Filing date: 2009-09-29
Publication date: 2010-04-01

Abstract

Systems and methods for interactive image based communication are described. To facilitate interactivity with a television broadcast or other streaming image, the present system and method utilizes a coordinate system and method having arbitrary precision (CSAP). In operation, the system uses CSAP to generate a virtual grid associated with a frame of a broadcast. The system assigns an addressable unit to each rectangle of the virtual grid. In a preferred embodiment, each addressable unit is further represented by N sectors of predefined color combinations. The virtual grid further comprises a region of rectangles representing a hot spot of interest. The hot spot includes an assigned action. A user points a mobile device at the hot spot to capture the hot spot information to further activate the assigned action of the hot spot.

Description

TITLE

Interactive Communication System and Method

CROSS-REFERENCE TO RELATED APPLICATIONS (OOOl)The present application is related to and claims the benefit of priority of the following commonly-owned, applications: Application Ser. No. 61/194,595, filed September 29, 2008 , entitled "Interactive Communication System and Method" and Application Ser. No. 61/275,159, filed August 26, 2009, also entitled "Interactive Communication System and Method."

(0002)The present application is also related to the following commonly-owned, provisional applications: Application Ser. No. 12/319,692, filed January 8, 2009, entitled "Image Based Communication System and Method" and Application Ser. No. PCT/US09/00127, filed January 8, 2009, also entitled "Image Based Communication System and Method."

(0003)The disclosure of each of the foregoing applications is hereby incorporated by reference in its entirety, including any appendices or attachments thereof, for all purposes.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT (0004)Not applicable.

REFERENCE OF A "MICROFICHE APPENDIX" (0005) Not applicable.

BACKGROUND OF THE INVENTION (0006)1. Technical Field

(0007)The present invention relates generally to interactive systems and methods.

(0008)2. Description of the Related Art

(0009)Interactive TV (iTV) and applications implementing same have become a hot topic of late.

However, significant difficulties remain, particularly overcoming the gap between the functionality of existing devices and those requiring more advanced functionality. The present invention addresses certain shortcomings of iTV's complexity in that it is relatively easy to implement and deploy and utilizes a common mobile computing device such as the ubiquitous cell phone as an input device.

BRIEF DESCRIPTION OF THE DRAWINGS

(0010) Additional features and advantages of the present invention will become apparent from the following description, taken in combination with the appended drawings, in which:

FIG 1 depicts aspects of an exemplary embodiment of an interactive communication system and method in accordance with the teachings presented herein. FIG 2 depicts an additional exemplary embodiment of the present system and method in accordance with the present invention.

FIG. 3 depicts a further exemplary embodiment of the present system and method in accordance with the present invention.

FIG. 4 depicts another exemplary embodiment of the present system and method in accordance with the present invention.

FIG 5 depicts a second exemplary embodiment of the present system and method in accordance with the present invention.

FIG. 6 depicts a second further exemplary embodiment of the present system and method in accordance with the present invention. FIG. 7 depicts an alternate exemplary embodiment of the present system and method in accordance with the present invention.

FIG. 8 depicts a third exemplary embodiment of the present system and method in accordance with the present invention.

FIG. 9 depicts a third further exemplary embodiment of the present system and method in accordance with the present invention.

FIG. 10 depicts another exemplary embodiment of the present system and method in accordance with the present invention. FIG. 11 depicts a fourth exemplary embodiment of the present system and method in accordance with the present invention.

FlG 12 depicts a fourth further exemplary embodiment of the present system and method in accordance with the present invention. FlG 13 depicts an alternate exemplary embodiment of the present system and method in accordance with the present invention.

FIG 14 depicts a fifth exemplary embodiment of the present system and method in accordance with the present invention.

FIG 15 depicts a fifth further exemplary embodiment of the present system and method in accordance with the present invention.

FIG. 16 depicts another exemplary embodiment of the present system and method in accordance with the present invention.

FIG. 17 depicts a sixth exemplary embodiment of the present system and method in accordance with the present invention. FIG 18 depicts a sixth further exemplary embodiment of the present system and method in accordance with the present invention.

FIG 19 depicts an alternate exemplary embodiment of the present system and method in accordance with the present invention.

FIG. 20 depicts a seventh exemplary embodiment of the present system and method in accordance with the present invention.

FIG. 21 depicts a seventh further exemplary embodiment of the present system and method in accordance with the present invention.

FlG 22 depicts another exemplary embodiment of the present system and method in accordance with the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS OF THE INVENTION

(001 l)The aspects, features and advantages of the present invention will become better understood with regard to the following description with reference to the accompanying drawing(s). What follows are preferred embodiments of the present invention. It should be apparent to those skilled in the art that these embodiments are illustrative only and not limiting, having been presented by way of example only. AU the features disclosed in this description may be replaced by alternative features serving the same purpose, and equivalents or similar purpose, unless expressly stated otherwise. Therefore, numerous other embodiments of the modifications thereof are contemplated as falling within the scope of the present invention as defined herein and equivalents thereto. A preferred application of the present system/method is as an interactive broadcast video application (iTV), details of which follow.

SYSTEM ARCHITECTURE

(0012)FIG. 5 illustrates an exemplary architecture of the present system in accordance with the teachings herein. As shown, the system includes a client side and a server side.

Client Side (0013)The client side comprises at least three clients: a HIT ^M enabled device, a video device, such as a DVR/DVD or set top box (STB) and a television or similar device (TV). The video device/STB is a core component of the client side. The STB relays broadcast to the TV and communicates with the HIT enabled device and the server side. In a preferred embodiment, the STB and HIT™ enabled device communicate via a wireless protocol having sufficient bandwidth, such as BLUETOOTH. Communication between the STB and the server side occurs via a two way communication channel.

(0014)The system assigns each client a unique ID. The server side uses the client ID during processing. For example, the ID of an STB is used to register action per account basis.

Server side

(0015)In an exemplary embodiment of the system, the server side comprises three servers: a Broadcast Engine server, an Ads/Commercials server and a Data server. In an additional embodiment, the system may also comprise a HIT^{I M} server, details of which follow. See also, FIG. 14. One skilled in the art will appreciate that the servers may reside on the same machine or can be distributed across several machines. (0016)The Broadcast Engine server controls the broadcast program per subscriber or group of subscribers. Each subscriber is identified by the ID of the STB. The Broadcast Engine server sends programming to the STB and does not directly receive any information from the STB. During ad air time, the Broadcast Engine server sends commands to the Ads/Commercials server.

(0017)The Ads/Commercials server in turns sends the appropriate commercial to the subscriber or group of subscribers. The Ads/Commercials server does not directly receive any information from the STB. However, the Ads/Commercials server does receive information from the Data server regarding a particular STB or group of STBs.

(0018)The Data server collects the data from a STB and sends commands to the Ads/Commercials server. Information collected at the Data server (e.g., the ID of the STB) is later used by a broadcaster to identify relevant data, such as user preferences and effectiveness of the commercial.

HIT™ ENABLED DEVICE

(0019)The HIT^{1 M} enabled device comprises an image detection module or component, an image recognition module or component, an I/O module or component, such as one or more input buttons, and a communications module or component. In another embodiment, the device further comprises an emitter for providing a visual indication of a point on the screen. The HIT™ enabled device can be a cellular phone, a web camera, remote control or any other device configured with the requisite technology.

(0020)The image detection module acquires an image being displayed on the display medium for further processing. In one embodiment, the detection module is stored locally on the HIT™ enabled device.

(0021)The image recognition module extracts recognizable elements from the acquired image. In one embodiment, the recognition module is stored locally on the HIT™ enabled device. (0022)The I/O buttons are used to activate certain functionality, including switching among the various operational modes (for example, drawing mode versus moving the marker) and triggering certain operations such as calibration and viewing the display menu. Preferably, the buttons are part of the HIT™ enabled device, however a keyboard connected to the host computer can be used to facilitate the operations.

(0023)The communication component can comprise a wired communication channel, a wireless communication channel or a combination thereof. For a cellular phone adapted as the HIT™ enabled device, in addition to the associated cellular network, a preferred communication solution is BLUETOOTH, however other communication protocols are suitable, such as, IrDA, WiFi or ZigBee. The reader is referred to Applicants' co- pending applications cited above, which contain additional aspects of the HIT™ enabled device.

COORDINATE SYSTEM HAVING ARBITRAY PRECISION (CSAP) (0024)To facilitate interactivity with broadcast or streaming images displayed on, for example, a television or computer monitor, the present system utilizes a coordinate system and method having arbitrary precision (CSAP). With CSAP, when a user points the HIT™ enabled device at the screen and activates the requisite image acquisition module by pressing, for example, a button on the device, a marker pattern is briefly displayed on a predefined area of the screen. See FIGS. 2- 4. If the HIT™ enabled device "sees" the marker pattern, the system identifies the location of the marker pattern and registers a "hit" with the system.

(0025)In operation, the system generates a virtual grid that corresponds with the dimensions of the screen. Conceptually, the grid overlays the display screen. See, e.g., FIGS 1 and 12. The system uses certain parameters of the display screen, such as the number of pixels, to generate the dimensions of the virtual grid. The system assigns an addressable unit to each rectangle of the grid. In a preferred embodiment, each addressable unit is further represented by N sectors of predefined color combinations. In the example of FIG 1. each addressable unit comprises four sectors. The color of each sector is selected from a set of four colors, Black, Red, Green and Blue. For example: (0026)Grid rectangle number 1 is represented by a Black, Red, Red, Red (BIkRRR) sector combination.

(0027)Grid rectangle number 6 is represented by a Black, Red, Green, Blue (BIkRGB) sector combination.

(0028)Grid rectangle number 8 is represented by a Black, Red, Blue, Green (BIkRBG) sector combination.

(0029)In one embodiment, the system initiates processing of each grid rectangle with the Black sector of the rectangle. Therefore, in the present example, there are 27 possible color combinations. The system analyzes the grid using a single or one pass approach as follows:

(0030)The system may predefine addressable units 1-9 of the grid in the following manner:

BIkRRR

BIkRRG BIkRRB BIkRGR BIkRGG BIkRGB

BIkRBR BIkRBG BIkRBB

(0031)Starting at the top left corner of the screen, the system identifies grid rectangle number 1. The system next identifies the Black sector as the starting point color. Working clockwise from the Black sector, the system then identifies the remaining color sectors of the rectangle.

(0032)The starting sector determines the starting point in terms of color and geometrical size. More particularly, in the example shown, the screen is divided into 25 rectangles. Therefore, in terms of pixels and for full HDTV resolution of 1920x1600, each rectangle is 480x340 pixels. For a 60 inch size display panel the size of each rectangle is roughly 10x6 inches.

CSAP PSEUDOCODE

(0033)The following pseudocode exemplifies an aspect of an embodiment of the coordinate system component in accordance with the present invention. Specifically, the code generates the color encoding for the virtual grid.

typedef enum { red, green, blue, yellow, magenta, cyan } COLORS;

typedef enum { black, white

} ANCHOR_COLOR;

typedef enum { single, double } ANCHORJTYPE;

typedef {

ANCHOR_COLOR a_color; COLORS color[3]; } RECTANGLE_CODE;

# define MAX-ALLOWED 46656 // six colors on three slots for two passes int ColorsSlots = 3 int AnchorType = 1 COLORS NumberOfColorsNeeded; int NumberOfPassesNeeded;

void Find_number_of_colors_needed(int h_ix, int v_ix) {

Int total_rectangles = h_ix * v_ix;

NumberOfPassesNeeded = 1; if (total_rectangles <= power(3,3)) { II 3 colors in 3 slots NumberOfColorsNeeded = (COLORS) 3; return;

if (total_rectangles <= power(4,3)) {// 4 colors in 3 slots NumberOfColorsNeeded = (COLORS) 4; return; }

if (total_rectangles <= power(5,3)) { II 5 colors in 3 slots NumberOfColorsNeeded - (COLORS) 5; return;

} if (total_rectangles <= power(6,3)) { II 6 colors in 3 slots

NumberOfColorsNeeded = (COLORS) 6; return;

}

NumberOfPassesNeeded = 2; if (total_rectangles <= sqr( power(3,3)) ) { // 3 colors in 3 slots NumberOfColorsNeeded = (COLORS)3; return;

} if (total_rectangles <= sqr( power(4,3)) ) {// 4 colors in 3 slots NumberOfColorsNeeded = (COLORS)4; return;

if (total_rectangles <= sqr( power(5,3)) ) { // 5 colors in 3 slots

NumberOfColorsNeeded = (COLORS)5; return;

} if (total_rectangles <= sqr( power(6,3)) ){ // 6 colors in 3 slots

NumberOfColorsNeeded = (COLORS)6; return; } total_rectangles = sqr( power(6,3)); // Maximum allowed

} void get_next_colors( COLORS& f, COLORS& s, COLORS& t)

{ // third color if( (t+1) > NumberOfColorsNeeded ) { t = (COLORS) 0 ; // red } else { t += 1 ; // next color in the list return ;

}

// second color if ( (s + 1 ) > NumberOfColorsNeeded ) { s = (COLORS) 0 ; // red } else { s += VJI next color in the list return ; }

// first color f += l;

RECTANGLE_CODE rect[MAX_ALLOWED] ;

// this function assigns the proper colors to a rectangle structure // The rectangles are stored in an array indexed from top left per line, void Build_color_combinations(int h_ix, int v_ix) {

// The system determines how many colors are needed to encode

// this particular grid, and how many passes.

Find_number_of_colors_needed(h_ix, v_ix) ;

// The system determined the requisite number of colors and passes int index;

// One pass approach

COLOR first, second, third; first = second = third = red ; for (int i = 0; i < v_ix ; i++) { for (int j = 0; j < h_ix; j++ ) { index = j + h_ix*i ; // this is the index of the rectangle rect[index].a_color = black; // one pass, hence hard coded rect[index].color[0] = third; rect[index].color[l ] = second; rect[index].color[2] = first;

get_next_colors(fϊrst, second, third) ;

} } }

(0034)Referring to the above, the Build_color_combinations function establishes the coordinates for each rectangle of the grid by assigning color combination according to the algorithm. The input parameters are horizontal (h_ix) and vertical (v_ix) sizes of the grid required. The result is stored in the array RECTANGLE_CODE rect[MAX_ALLOWED]. The rectangles are indexed from the top left, line after line.

(0035)First, the system identifies the number of colors required and number of passes. The Anchor or start color (the color that is processed first ) can be combined with another color. Therefore, the system can process color combinations such as Black-Black, Black- White, White-Black and White- White colors, thereby extending the number of possible combination by four. Refer to FIG.15 for additional details

(0036)In the current example shown, the system processes a one pass coordinate system having the color black as the anchor color and three additional color slots. The Function Find_number_of_colors_needed utilizes three color slots, but can extend to calculate the minimum number of color slots required and number of colors needed for that number of color slots.

OTHER APPROACHES:

(0037)In another embodiment, the system may utilize two colors as starting point colors, for example Black and White, thereby doubling the number of addressable units. Further, the system can process an additional color or colors, such as Yellow. With the addition of just one color to the available color choices, the number of addressable units increases to 64 for three recognizable sectors (that is 4 in power of three) or 256 for four sectors beyond starting point. By processing the additional colors, the system can define as precise a coordinate system as is practically required for the application.

(0038)In a further embodiment, the system may utilize a two pass approach. Here, the system captures an image of the addressable unit (colored grid rectangle) during a first pass and processes a first color as the starting point color (for example, Black) to define the X coordinate of the object. During a second pass, the system captures another image of the same addressable unit and processes a second color as the starting point color (for example, White) to identify the Y coordinate of the object.

Although, two images are required to identify the location of the "hit," precision is now finer and can be as fine as a single pixel.

(0039)ln the our example, having a two pass approach creates 27 elements for the X axis coordinate and 27 elements for the Y axis coordinates for a total of 729 grid elements using the same number of colors. A two pass approach is useful, for example, with interactive chess over TV where response speed is not critical but there is a need of fine precision.

VIRTUAL GRID CREATION (0040)The system utilizes a virtual grid creation module to enable a content creator to create and associate a virtual grid with a broadcast/commercial. In operation, a user transmits an image to be encoded with hot spots to the grid creation module. The module receives the image and displays available grids to the user. The user selects a grid from the available grids. The system overlays the image with the selected grid and displays the resultant image to the user for review. If the user is satisfied with the grid selection, the user selects the desired rectangles associated with a hot spot by clicking on each rectangle or by selecting a region of rectangles. If user selects a region, the rectangles associated with the region are considered selected if at least part of a rectangle is within the selected region. The user repeats the above to process next hot spot.

APPLICATION 1 : INTERACTIVE COMMERCIAL PROGRAMMING (004I)In accordance with the teachings herein, the present invention can comprise a system for and method of enabling a user to interact with a commercial being contemporaneously broadcast. Referring to FIGS 6-10, the following section describes an exemplary process of a simple interactive commercial, wherein the commercial does not communicate with any servers during the life of the commercial broadcast on the TV. In this example, a commercial content provider designates the basketball as the hot spot (sensitive area or soft button) and that the hot spot will be active for a short interval of time. If a hit is registered during this time, an extra 25% discount is given to the user after the user supplies a promotion code. The promotional code appears on the screen only if customer hits the hot spot . A 10% code is given to the user if the user misses the hot spot but pulls the trigger during the commercial. This scenario together with the clip is supplied to the broadcaster (e.g., cable company)

(0042)FIG 6 illustrates the message flow/exchange protocol between the client and server (e.g., cable company system) in accordance with one embodiment of the present invention. During the airing of the commercial, the system transmits a Commercial Description Message to the STB. The Commercial Description Message comprises:

NEXTJD (Parameter 1)

ENCODINGJTYPE (Parameters 1-3) NUMBER_OF_HOTSPOTS (Parameter 1)

HOTSPOTJ (Parameters 1 -4)

TIMINGJ ( Parameters 1-2)

RESPONSE_AREA_1( Parameters 1-5)

RESPONSE JvISG J (Parameter 1) RESPONSE JV-REAJ)EFAU LT (Parameters 1-5)

RESPONSE_MSG_DEFAULT( Parameter 1)

RETURN NONE

(0043)NEXTJD: includes one parameter and defines the ID of the commercial. The STB records the commercial ID for internal purposes or in case a return value is required. The commercial ID can be simple value such as a time stamp or hash value or alternatively, a combined from various fields, such as the type of exchange (e.g., commercial or game).

(0044)ENCOD ING-TYPE: includes three parameters and defines the type of coordinate system, either one pass or two pass (first parameter), the number of colors for encoding purposes (second parameter), and the virtual grid size (third parameter). Presently, it is unnecessary to divide the screen onto a maximum number of rectangles allowable by the color combinations.

(0045)NUMBER_OF_HOTSPOTS: includes one parameter and defines the number of hot spots on the screen.

(0046)HOTSPOT_1 : includes four parameters and defines the coordinates of the hot spot in terms of the virtual grid (indexes of the rectangles).

(0047)TIMING_l : includes two parameters and defines the timing of the sensitive area(s) in terms of shift from the beginning of the commercial (parameter 1) and how long it stays sensitive in milliseconds (parameter 2). For example, a hot spot may become active a second after the commercial begins and remain active for the next 5 seconds.

(0048)RESPONSE_ AREAJ : includes 5 parameters and defines in terms of the virtual grid, the area/location of the response message for display.

(0049)RESPONSE_MSG_1 : includes 1 parameter and defines a message for display. For example, the system may display a message to a user in New Jersey that is different from a message shown to a user in San Francisco.

(0050)RESPONSE_AREA_DEFAULT and RESPONSE_MSG_DEFAULT: include 5 and 1 parameters, respectively, and define a default area and response message when a user misses a hot spot ("try again next time" or "10% off for the effort") (005I)RETURN NONE: defines a value for the STB to transmit to the Servers. In the current example, no response is required. In another embodiment, the STB may send certain information, such as the STB ID or time stamp, to the Servers.

(0052)During operation, when the STB receives the signal indicating that a commercial is in the broadcast stream, the STB sends an ENCODINGJTYPE message to the HIT™ enabled device. The HIT™ enabled device, in turn, may indicate to the user that is activated by, for example, flashing an LED. After the commercial airs, the user retrieves any promotional information from the STB using the HIT™ enabled device.

(0053)FIGS 7-10 illustrate exemplary embodiments of the present invention. FIG 7 depicts a frame of a commercial that is ready for broadcast. Here, a content creator identified the basketball 7001 as a hot spot. FIG. 8 depicts a virtual grid created by the system and associated with the frame of FIG. 7. Based on the grid, the encoding type is identified (one or two passes and number of colors sufficient to encode the grid.). FIG 9 illustrates the screen layout after the user presses the designated button to activate the hot spot. Here, the STB covers the target hot spot with color coded rectangles in accordance with CSAP above. Depending on how the system is configured, the user may notice a splash or a pattern on the screen for a brief interval of time, e.g., a fraction of second. FIG. 10 illustrates how the screen may appear to the user after user hits the target.

APPLICATION 2: "CLICK TO BUY" DURING BROADCAST PROGRAMMING (0054)In accordance with the teachings herein, the present invention can additionally comprise a system for and method of enabling a user to interact with a broadcast program to enable said user operating a HIT ^ΓM enabled device (e.g., cellular phone) to purchase a good or service being contemporaneously broadcast. Here, the information is embedded in the broadcast stream and a user makes a transaction during the broadcast.

(0055)In this application, encoding information about hot spots is similar to that of the interactive commercial application above. A user/producer/advertiser identifies the hot spots. The user/producer/advertiser selects a rectangle of appropriate size that represents the hot spot and assigns an action to the hot spot. The action can be as simple as a counter that tracks viewer interest or as elaborate as a commerce transaction to purchase a product or service. The system analyzes possible grids and selects an appropriate grid size, and then updates the rectangle representing the hot spot. Additionally, the system analyzes the properties of the image within the hot spot. One possible property is a color histogram. For every quantum of time (this interval may be selected by user), the system analyzes an area around the initial hot spot in order to track movement. The hot spot position is updated according to the movement. The information about the grid, hot spot location and action associated with the hot spot is transmitted via cable network, preferably on a separate channel, like close captioning.

Establish Communication

(0056)To begin, a user starts the client application on his cellular phone and the client application initiates a communication session with the STB. After the communication session establishes, the STB has all the information required to identify the current cellular phone (e.g., cellular phone number and type of cellular phone).

Show Hot Spots

(0057)To display the hot spots, the user presses a "show the hot spots" designated button on the cellular phone to generate a command. The cell phone transmits the command to the STB. The STB indicates the hot spots by. for example, drawing rectangles around hot spots. The STB may also display additional information, such as the action associated with the hot spot ("purchase the dress", or "select favorite singer").

Acquire the Images

(0058)To begin processing a transaction, the user aims the cellular phone camera at the screen. The system displays a pattern (e.g., a cross hair) appears on top of picture on the cellular phone. If the user's aim is off, the user moves the cell phone to align the aim/cross hair pattern with a hot spot. Once properly positioned at the hot spot, the user presses a button to allow the cellular phone to begin acquiring images of the broadcast and to send a first signal to the the STB. After receiving the first signal from the cell phone, the STB momentarily freezes the current broadcast image and displays the predefined hot spots. During this time, the cellular phone can acquire a number of images for further analysis.

Analyze the images and complete transaction

(0059)The HIT^{1 M} enabled device analyzes the acquired images locally (via the client application running on the device). During the analysis process, the device determines either the coordinates of the predefined hot spot or a predefined value that indicates that no hot spot has been detected. The device transmits the result to the STB. The STB then sends certain information to the Server. The information includes a cellular phone ID, a hot spot ID and an action ID. Thereafter, the Server initiates a communication session with the HIT enabled device and the system prompts the user to input additional data to complete the transaction.

(006O)FIGS. 11-14 are illustrative of the "click to buy" application in accordance with the present invention. Here, the system comprises predefined hot spots for an episode of HBO's Sex and the City™ .

(006I)FIGIl depicts a frame from the show. As shown, the frame includes the character Carrie Bradshaw wearing a purchasable dress and a bag. Also shown are two predefined hot spots 1101 and 1102 associated with the dress and bag. The hot spots do not intersect, are well defined and enable the purchase of the items.

(0062)FIG12 depicts an exemplary virtual grid associated with the frame. During grid creation, the system determines a minimal grid size for the frame having the predefined hot spots. Here, the system determined and generated a 3x3 grid size to allow the two hot spots for the dress and bag to be uniquely defined. As shown, rectangles 2, 3, 5 and 6 define the hot spot for the dress purchase and rectangles 8 and 9 define the hot spot for the bag purchase. If an additional hot spot is required for the purchase of, say, Carrie's shoes, the system will determine and generate a different grid size to account for the additional hot spot.

(0063)FIG13 depicts the image the cellular phone acquires during processing. Here, the virtual grid comprises nine rectangles and only two out of three basic colors are needed. The color black serves as the starting point of analysis. The system analyzes the colors in a clockwise fashion starting from the black sector. The table below illustrates the color encoding of the rectangles in a 3x3 grid.

(0064)In the present example, only two rectangles are shown with patterns. Alternatively, patterns can appear on top of all six selected rectangles.

(0065)FIG14 depicts an exemplary message flow of all operations during a click to buy transaction in accordance with the present invention. At the start of a new broadcast show, the Cable server exchanges information about the show by sending information such as the show's unique ID to the HIT™ server. The HIT™ server contains all information about the hot spots defined for the show. This information is transmitted to the Cable server which in turn transmits the information to each STB. After receiving the information, the STB indicates on the TV that the show is " HIT™ enabled".

(0066)Next, a user having a HIT™ enabled device establishes a connection with the STB. The user can request that the system display available hot spots with pertinent information, such as "purchase this dress starting from X dollars".

(0067)The user initiates the transaction as describe above. After processing, the HIT™ enabled device sends the result to the STB, which in turn sends certain identifying information (e.g., cellular phone ID (number), STB ID, show ID, hot spot ID and time stamp) to the Cable server. The Cable server registers the transaction and sends the information to the HIT™ server.

(0068)The HIT^{1 M} server initiates a session with the user via the cellular network to complete the transaction. For example, the user can select parameters such as color, size, verify delivery address, select method of payment (e.g., add to cellular phone bill or charge credit card registered with the cellular phone for this service), etc. Once this information is received by the HIT™ server, the system will prompt the user to authorize the transaction. Authorization may be done conventionally, for example, by answering one or more security challenge questions.

APPLICATION 3: DRAWING DURING BROADCAST PROGRAMMING (0069)In accordance with the teachings herein and also described in Applicants' co- pending applications cited above, the present invention can also comprise a system for and method of enabling a user to interact with a broadcast program to enable said user operating a HIT™ enabled device (e.g., cellular phone) to annotate the broadcast. For example, the system enables a user watching a football game to annotate a particular player's moves using a HIT ^M enabled device cellular phone (e.g., encircle a player).

(007O)In operation, the user points the cellular phone to the screen and presses a designated button. The cellular phone sends a first signal to the STB. The STB receives the signal, freezes the image on the screen and draws a marker pattern on the screen. The user can now control the movement of the marker.

(0071)The user selects a shape to be drawn (e.g., circle, rectangle or arrow) by pressing a first designated button. The cellular phone transmits certain information to the STB and the STB displays the user's selection.

(0072)The user begins annotation by pressing and holding a second designated button. The system draws the selected shape at the marker's location. The user then manipulates the cellular phone to annotate the frame as desired. The user may draw a number of different shapes during one session. (0073)The user terminates the application by pressing a third designated button. The system removes all annotations and the broadcast resumes.

APPLICATION 4: INTERACTIVITY USING EXISTING MOBILE DEVICE BASE AND A MOBILE BASED CLIENT APPLICATION

(0074)Advertisers exist to grab a viewer's attention. A watch manufacturer will pay significant money to ensure that a main character will wear its watch in a movie. For example, OMEGA and ROLEX watches are featured in James Bond movies and Bruce Willis wears TAG HAYER watches in the Die Hard series. The hot spot functionality of the present invention enables a viewer to purchase the watch while watching the movie.

(0075)Generically, a hot spot is a trigger for interactivity between a mobile device and a broadcast/video being shown on a display medium, such as a television. The hot spot encapsulates items of interest and action with that being displayed. For example, a hot spot can generate: 1) a custom screen to allow a user to make a multiple choice selection; 2) secondary content such as an external video of a person on the broadcast; 3) a download of a small segment of the broadcast; and 4) means to increase loyalty through additional channels - content and interactivity.

(0076)Multiple hot spots can be available during a given broadcast. For example, one hot spot will comprise information about James Bond's OMEGA watch, such as its price and vendor, while another hot spot will comprise information about his tie. During broadcast, the various hot spots will not appear on the TV screen and therefore will not distract the viewer's attention. Alternatively, the present system can display an indication on the screen , such as a logo, representing the availability of hot spots.

(0077)In one embodiment, the hot spot data format comprises starting time, duration, description, type of action associated with the hot spot and additional content. This hot spot format allows for seamless transition to a SetTopBox-based interaction mode. (0078)In this alternative embodiment of the present invention, a user installs a mobile client application on the mobile device. This client application differs from the prior embodiments describe above in that, here the mobile client application contains more information about the associated television program. During a show's broadcast, such as the Home and Garden show, the user starts the client application. The client application connects to a HIT server. The HIT server sends Hot Spot information related to the broadcast show to the user's mobile device. If the user sees an item of interest currently being shown on the TV, the user uses a mobile device to select the appropriate Hot Spot and completes the transaction/action that is associated with the item of interest on the mobile device.

(0079)In one embodiment, the client application is included in a common repository such as the APPSTORE from APPLE. During application and device registration, the user will provide region, TV feed information (cable or satellite provider), credit card information, email address and other pertinent information. Certain information can be automatically detected such as region, which can be determined by using location services on the mobile device.

(0080)Once started, the client application connects to a HIT server. The application may use location data to ensure that the hot spot information is currently relevant for the show. The location data determines which shows are being broadcast in that particular region. The hot spot information is presented in a way that allows user to identify the hot spot and perform a transaction easily. For example, there could be list of hot spots, that can expand to present more information for the user, or alternatively the hot spots can be presented as thumbnails depicting various items. Upon selecting the thumbnail, additional information will appear on the screen.

(008I)A mobile application can be created on a per show or per channel basis (like a Discovery channel application) or even on a per-cable/satellite provider basis where the channel number will be selected with the remote control and on the mobile device at the same time. Alternatively, the system can perform these changes simultaneously on the TV and mobile application. (0082)The HIT server performs a number of operations. The server contains both the database of registered and connected mobile devices and the database of hot spot information per show. The server feeds hot spots to the connected applications. The server facilitates the user's transactions.

(0083)By registering with the HIT server, the system feeds appropriate hot spots at any given time according to the schedule of the show. In addition, registration simplifies the flow of transactions. After the system verifies certain user profile/account information, such as credit card and name and address information, during registration, the user will only have to confirm their password in order to complete the transaction. The hot spot database includes hot spot information per show as well as other pertinent information, such as show schedule per feed (e.g. East/West coast).

(0084)The HIT server pushes hot spots to the mobile device in a time-synchronized manner. After the mobile device is connected to the server, the server identifies the timezone of the user using the mobile device's location service or using the information that user has entered during the registration process. The HIT server then pushes the hot spot information to the mobile device according to the currently available broadcast in the user's area. Optionally, the user can override the location data. For example, a user is watching the West Coast feed via satellite link from the East Coast zone where he is currently located. Here, the HIT server will feed hot spot information to the user according to West Coast 's feed schedule. FIG 19 depicts an exemplary usage flow of the present invention in accordance with the teachings provided herein. FIG. 20 depicts exemplary client-server interaction of the present invention in accordance with the teachings provided herein. FIGS. 21 & 22 illustrate exemplary embodiments of the present system's architecture in accordance with the teachings herein.

ALTERNATIVE HARDWARE SETUPS (0085)FIG 16 describes a system setup where a video device is connected to a cable server via a cable network. This configuration enables the video device to communicate both ways with the cable servers. This is a standard communication model for most cable subscribers where a user can communicate with the servers via STB(order pay-per-view movie, etc). The only hardware update required is the communication path between the Video device and the HIT™ enabled device/cellular phone. The communication path can be a BLUETOOTH dongle. Alternatively, modern video devices may already have IP network connectivity thus providing a software only solution. The communication path between the HIT™ enabled device/cellular phone and video device is one way, i.e. the HIT™ enabled device/cellular phone can issue commands to the video device, however the video device does not communicate with the HIT™ enabled device/cellular phone.

(0086)FIG. 17 describes an additional system setup where the video device is not connected to the cable server. Here, the video device can be a DVR/DVD player, running a pre-recorded show. The video device is connected to the HIT™ server via an IP network. Typically, the video device is connected to a home network that in turn is connected to the INTERNET.

(0087)FIG 18 describes a further system setup where the video device is not connected to any network. Here, the video device can be a DVR/DVD player or a STB. The video device is capable of two-way communication between the HIT™ enabled device/cellular phone and the video device. This setup requires minimal change to existing infrastructure.

CONCLUSION

(0088)The embodiments of the invention described above are intended to be exemplary. Those skilled in the art will therefore appreciate that the forgoing description is illustrative only, and that various alternatives and modifications can be devised without departing from the spirit of the present invention. Accordingly, the present is intended to embrace all such alternatives, modifications and variances which fall within the scope of invention.

(0089)Moreover, the techniques may be implemented in hardware or software, or a combination of the two. Preferably, the techniques are implemented in control programs executing on programmable devices that each include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device and one or more output devices. Program code is applied to data entered using the input device to perform the functions described and to generate output information. The output information is applied to one or more output devices. Each program is preferably implemented in a high level procedural or object oriented programming language to communicate with a computer system, however, the programs can be implemented in assembly or machine language, if desired. Each such computer program is preferably stored on a storage medium or device (e.g., CD-ROM, hard disk or magnetic diskette) that is readable by a general or special purpose programmable computer for configuring and operating the computer when the storage medium or device is read by the computer to perform the procedures described in this document. Features of the system may also be considered to be implemented as a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner.

Claims

What is claimed is:

L A method of interacting with a broadcast program, comprising:

providing a frame of a broadcast program;

generating a virtual grid associated with said frame;

defining a hot spot having an assigned action;

associating said hot spot with a region of said virtual grid;

displaying said frame having said virtual grid;

acquiring a remote image of a portion of said region of said virtual grid representing said hot spot; and;

performing said assigned action of said hot spot.

2. The method as in claim 1 , wherein said virtual grid comprises at least one rectangle.

3. The method as in claim 2, wherein said at least one rectangle is defined as a unique addressable unit.

4. The method as in claim 3, wherein said addressable unit comprises a predefined, unique combination of N sectors of predefined colors.

5. The method as in claim 4, wherein said addressable unit comprises a predefined, unique combination of 4 sectors of predefined color.

6. The method as in claim 5, wherein said predefined colors include Black, Red, Green and Blue.