US20210076106A1 - Systems and methods for generating supplemental content for a program content stream - Google Patents
Systems and methods for generating supplemental content for a program content stream Download PDFInfo
- Publication number
- US20210076106A1 US20210076106A1 US16/566,630 US201916566630A US2021076106A1 US 20210076106 A1 US20210076106 A1 US 20210076106A1 US 201916566630 A US201916566630 A US 201916566630A US 2021076106 A1 US2021076106 A1 US 2021076106A1
- Authority
- US
- United States
- Prior art keywords
- presentation
- program
- content stream
- program content
- content
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000000153 supplemental effect Effects 0.000 title claims abstract description 145
- 238000000034 method Methods 0.000 title claims abstract description 55
- 230000001755 vocal effect Effects 0.000 claims abstract description 54
- 238000003058 natural language processing Methods 0.000 claims abstract description 32
- 238000012545 processing Methods 0.000 claims description 81
- 230000004044 response Effects 0.000 claims description 45
- 239000000284 extract Substances 0.000 claims description 5
- 238000004891 communication Methods 0.000 description 33
- 238000010586 diagram Methods 0.000 description 16
- 241000282320 Panthera leo Species 0.000 description 10
- 230000008569 process Effects 0.000 description 10
- 230000006870 function Effects 0.000 description 8
- 230000002452 interceptive effect Effects 0.000 description 7
- 239000008186 active pharmaceutical agent Substances 0.000 description 4
- 238000012790 confirmation Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000001413 cellular effect Effects 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000002093 peripheral effect Effects 0.000 description 3
- 238000013519 translation Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 230000001537 neural effect Effects 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 239000000344 soap Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000026676 system process Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8545—Content authoring for generating interactive applications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B7/00—Electrically-operated teaching apparatus or devices working with questions and answers
- G09B7/02—Electrically-operated teaching apparatus or devices working with questions and answers of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
- G09B7/04—Electrically-operated teaching apparatus or devices working with questions and answers of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student characterised by modifying the teaching programme in response to a wrong answer, e.g. repeating the question, supplying a further explanation
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B7/00—Electrically-operated teaching apparatus or devices working with questions and answers
- G09B7/06—Electrically-operated teaching apparatus or devices working with questions and answers of the multiple-choice answer-type, i.e. where a given question is provided with a series of answers and a choice has to be made from the answers
- G09B7/08—Electrically-operated teaching apparatus or devices working with questions and answers of the multiple-choice answer-type, i.e. where a given question is provided with a series of answers and a choice has to be made from the answers characterised by modifying the teaching programme in response to a wrong answer, e.g. repeating the question, supplying further information
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/08—Speech classification or search
- G10L15/18—Speech classification or search using natural language modelling
- G10L15/1815—Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/26—Speech to text systems
-
- G10L15/265—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/65—Transmission of management data between client and server
- H04N21/658—Transmission by the client directed to the server
- H04N21/6587—Control parameters, e.g. trick play commands, viewpoint selection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/8126—Monomedia components thereof involving additional data, e.g. news, sports, stocks, weather forecasts
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/84—Generation or processing of descriptive data, e.g. content descriptors
Definitions
- the technical field relates to delivering multimedia content, and particularly to generating supplemental content for a program content stream.
- Supplemental content such as educational questions may be generated based on the content of the program as learned by the system via processing the verbal content of the program content stream (e.g., using natural language processing) as the program is being presented on a television or mobile device.
- the system may pause the original programming (or the playback of recorded programming) during presentation of the questions on the television or mobile device.
- the system then resumes the programming once a correct answer is received or a timer runs out, whichever occurs first.
- any program with verbal content spoke or written
- the generated supplemental content may also be saved as metadata associated with the program for subsequent presentation when the program is played again for the viewer or other viewers, thus further increasing efficiency of generating further supplemental content.
- a method for for generating supplemental content for a program content stream may be summarized as comprising: receiving, by a computerized system for generating supplemental content, the program content stream; processing, by the computerized system for generating supplemental content, verbal content of the program content stream as the program content stream is being received; generating, by the computerized system for generating supplemental content, supplemental content for the program content stream as the program content stream is being received based on the processing of the verbal content of the program content stream; and causing, by the computerized system for generating supplemental content, the supplemental content to be presented as the program content stream is being output for presentation on a presentation device.
- a system for generating supplemental content for a program content stream may be summarized as comprising: at least one computer processor; and at least one memory coupled to the at least one computer processor.
- the at least one memory has computer-executable instructions stored thereon that, when executed by the at least one computer processor, cause the at least one computer processor to: generate questions, while a program content stream is being output for presentation on a presentation device, having answers that are one or more of extracted concepts communicated by verbal content of the program content stream; cause the questions to be presented on the presentation device or a mobile device of a user at different times while the presentation of the program content stream is paused; and store the generated questions, as metadata associated with a program of the program content stream, for presentation as supplemental content of the program during subsequent playback of the program.
- Another method for generating supplemental content for a program content stream may be summarized as comprising: performing natural language processing of content of a program content stream as the as the program content stream is being output for presentation on a presentation device; extracting concepts communicated by verbal content of the program content stream based on the performing of the natural language processing of the verbal content; and causing questions based on the extracted concepts to be presented on the presentation device or a mobile device of a user at different times while the presentation of the program content stream is paused.
- FIG. 1 is an overview block diagram illustrating a technical environment in which embodiments of systems and methods for generating supplemental content for a program content stream may be implemented, according to an example embodiment.
- FIG. 2 is a block diagram illustrating elements of an example receiving device used in systems and methods for generating supplemental content for a program content stream, according to an example embodiment.
- FIGS. 3A and 3B are diagrams of an example presentation device and an example handheld mobile device, respectively, illustrating respective example user interface screens shown in a presentation of an example question as supplemental content for a program content stream, according to an example embodiment.
- FIGS. 4A and 4B are diagrams of an example presentation device and an example handheld mobile device, respectively, illustrating respective example user interface screens shown in an example presentation of a response to an answer received to the example question presented in FIGS. 3A and 3B , according to an example embodiment.
- FIGS. 5A and 5B are diagrams of an example presentation device and an example handheld mobile device, respectively, illustrating respective example user interface screens shown in an example presentation of a response to an incorrect answer received to the example question presented in FIGS. 3A and 3B , according to an example embodiment.
- FIG. 6 is a flow diagram of an example method for generating supplemental content for a program content stream, according to an example embodiment.
- FIG. 7 is a flow diagram of an example method for storing generated questions as metadata associated with a program of the program content stream, according to an example embodiment.
- FIG. 8 is a flow diagram of an example method for extracting educational concepts communicated by verbal content of the program content stream, according to an example embodiment.
- Multimedia content is becoming increasingly available through various different media sources.
- most of the media content is not interactive and lacks educational value, especially to children.
- a technical problem is thus presented in how to make the abundance of popular existing media content that often is received as a program content stream on one's receiving device (such as a cable or satellite set-top box, streaming media device, tablet or smartphone) more educational and interactive in a faster and more efficient manner as such content is being received by the receiving device.
- the present disclosure provides a solution to the technical problem above by, in some embodiments, performing natural language processing (NLP) of the verbal content of the program content stream as the program content stream is being received to generate supplemental content for the program content.
- this supplemental content may be or include educational questions that are based on the content of the program as learned by the system via performance of the NLP of the verbal content of the program content stream.
- the system may then pause the original programming (or the playback of recorded programming) to present the questions and then resume the programming once a correct answer is received or a timer runs out, whichever occurs first.
- any program with verbal content may be processed as it is being received in real time or near real time and morphed into a more educational experience for the viewer.
- the generated supplemental content may also be saved as metadata associated with the program for subsequent presentation when the program is played again for the viewer or other viewers, thus further increasing efficiency of generating further supplemental content.
- FIG. 1 is an overview block diagram illustrating a technical environment 102 in which embodiments of systems and methods for generating supplemental content for a program content stream may be implemented, according to an example embodiment.
- audio, video, and/or data service providers such as television service providers, provide their customers a multitude of video and/or data programming (herein, collectively “programming” or “content”).
- programming is often provided as a program content stream for use by a receiving device 118 communicatively coupled to a presentation device 120 configured to receive the programming.
- the programming may include any type of media content, including, but not limited to: television shows, news, movies, sporting events, advertisements, etc.
- any of this programming may be provided as a type of programming referred to as streaming media content, which is generally digital multimedia data that is substantially constantly received by and presented to an end-user or presented on a device while being delivered by a provider from a stored file source.
- streaming media content which is generally digital multimedia data that is substantially constantly received by and presented to an end-user or presented on a device while being delivered by a provider from a stored file source.
- streaming media content is generally digital multimedia data that is substantially constantly received by and presented to an end-user or presented on a device while being delivered by
- the receiving device 118 is a device such as a set-top box, television, DVR, DVD player, PC, tablet device, game machine, smart phone, mobile device or other computing device or media player configured to receive programming via a connection to a satellite or cable television service provider or Internet service provider outside the customer premises 116 , and to display such programming on a presentation device 120 .
- the receiving device 118 may be configured to receive, process and display on the presentation device 120 programming received directly from the satellite or cable television service provider, such as cable or satellite television broadcasts via various physical and logical channels of communication of communication system 108 .
- the receiving device 118 may be configured to receive, process and display on the presentation device 120 streaming media content received directly from the satellite or cable television service provider and/or other content provider 104 .
- the receiving device 118 interconnects to one or more communications media or sources.
- the various media content may be delivered as data using the Internet Protocol (IP) suite over a packet-switched network such as the Internet or other packet-switched network, which may be included as part of the communication system 108 .
- IP Internet Protocol
- the underlying connection carrying such data may be via a cable head-end, satellite antenna, telephone company switch, cellular telephone system, Ethernet portal, off-air antenna, or the like.
- the receiving device 118 may receive a plurality of programming by way of the communications media or sources, or may only receive programming via a particular channel or source described in greater detail below.
- the receiving device 118 processes and communicates the selected programming to the presentation device 120 .
- the presentation device 120 may also be a receiving device 118 or have a receiving device 118 integrated within it.
- Examples of a receiving device 118 may include, but are not limited to devices such as, or any combination of: a “television converter,” “receiver,” “set-top box,” “television receiving device,” “television receiver,” “television,” “television recording device,” “satellite set-top box,” “satellite receiver,” “cable set-top box,” “cable receiver,” “media player,” “digital video recorder (DVR),” “digital versatile disk (DVD) Player,” “computer,” “mobile device,” “tablet computer,” “smart phone,” “MP3 Player,” “handheld computer,” “gaming console” and/or “television tuner,” etc.
- the receiving device 118 may be any suitable converter device or electronic equipment that is operable to receive programming via a connection to a satellite or cable television service provider or other source outside the customer premises 116 and communicate that programming to another device. Further, the receiving device 118 may itself include user interface devices, such as buttons or switches. In some example embodiments, the receiving device 118 may be configured to receive and decrypt content and/or software or firmware updates according to various digital rights management (DRM) and other access control technologies and architectures as part of or in addition to the process of generating supplemental content for a program content stream, which will be described in further detail below.
- DRM digital rights management
- Examples of a presentation device 120 may include, but are not limited to, one or a combination of the following: a television (“TV”), a personal computer (“PC”), a sound system receiver, a digital video recorder (“DVR”), a compact disk (“CD”) device, DVD Player, game system, tablet device, smartphone, mobile device or other computing device or media player, and the like.
- Presentation devices 120 employ a display, one or more speakers, and/or other output devices to communicate video and/or audio content to a user.
- presentation devices 120 employ a microphone, keyboard, and/or other input devices to communicate commands or other input to the presentation device 120 and/or receiving device 118 .
- one or more presentation devices 120 reside in or near a customer premises 116 and are communicatively coupled, directly or indirectly, to the receiving device 118 . Further, the receiving device 118 and the presentation device 120 may be integrated into a single device. Such a single device may have the above-described functionality of the receiving device 118 and the presentation device 120 , or may even have additional functionality.
- a remote-control device (remote control) 130 is operable to control the receiving device 118 and/or the presentation device 124 .
- the remote control 130 typically communicates with the receiving device 118 using a suitable wireless medium, such as infrared (IR), radio frequency (RF), or the like.
- the remote control 130 may include a microphone that may be voice or audio activated in order to receive commands or other voice input.
- voice input may be communicated to and used by the receiving device 118 , presentation device 120 and/or mobile device 128 and cause such devices to perform other actions.
- voice input may be processed by the receiving device 118 , presentation device 120 and/or mobile device 128 as an answer or other response to supplemental content presented by such systems.
- the remote 130 may be a voice-enabled remote and the same microphone may also be used to receive voice commands from the user and transmit corresponding commands to the receiving device 118 to control the receiving device 118 and/or other peripheral devices.
- handheld mobile device 128 may be communicatively connected to the television receiving device 118 via a physical or a peer-to-peer short range wireless connection.
- mobile device 128 may be a handheld mobile device of a user (e.g., a customer of a satellite, cable or streaming media subscription service associated with customer premises 116 or other user) such as a smartphone, tablet device or other computing or communications device.
- a user e.g., a customer of a satellite, cable or streaming media subscription service associated with customer premises 116 or other user
- smartphone e.g., a smartphone, tablet device or other computing or communications device.
- the mobile device 128 may communicate over communication system 108 with the television receiving device 118 , program distributor 106 , content provider 104 , supplemental content server 122 and/or information provider 138 using the transmission control protocol/Internet protocol (TCP/IP) suite of networking communication protocols or other communication protocols.
- TCP/IP transmission control protocol/Internet protocol
- the mobile device 128 may also or instead communicate directly with the receiving device 128 , remote control 130 and/or presentation device 120 over one or more short range wireless networks or peer-to-peer networks.
- a cable or satellite television service provider may encompass or be in communication with some or all of the content provider 104 , program distributor 106 , supplemental content server 122 , and information provider 138 .
- a content provider 104 provides program content, such as television content, to a distributor, such as the program distributor 106 .
- Example program distributors include, but are not limited to, satellite and cable television service providers.
- Example content providers include television stations which provide local or national television programming and special content providers which provide premium based programming, streaming services, pay-per-view programming and on-demand programming.
- Program content (i.e., a program including or not including advertisements), is communicated to the program distributor 106 from the content provider 104 through suitable communication media, generally illustrated as communication system 108 for convenience.
- Communication system 108 may include many different types of communication media including those utilized by various different physical and logical channels of communication, now known or later developed.
- Non-limiting media and communication channel examples include one or more, or any operable combination of: satellite systems, cable systems, fiber optic systems, microwave systems, asynchronous transfer mode (“ATM”) systems, frame relay systems, digital subscriber line (“DSL”) systems, radio frequency (“RF”) systems, telephone systems, cellular systems, and the Internet.
- the received program content is converted by the program distributor 106 into a suitable signal (a “program signal”) that is ultimately communicated to the receiving device 118 as a program content stream.
- a program signal a suitable signal
- Various embodiments of the receiving device 118 may receive programming from program distributors 106 and/or directly from content providers 104 via locally broadcast radio, RF signals, cable, fiber optic, Internet media, or the like via the communication system 108 .
- Video on Demand (VOD) systems may allow a user of the receiving device 118 to select, watch and/or listen to video and audio content on demand.
- VOD Video on Demand
- IPTV Internet Protocol Television
- IPTV Internet Protocol Television
- such technologies are deployed within the environment 102 such as in subscriber-based telecommunications networks of the communication system 108 with high-speed access channels into the customer premises 116 via the receiving device 118 (e.g., a set-top box or other customer-premises equipment) to bring VOD services to the customer premises 116 .
- the receiving device 118 e.g., a set-top box or other customer-premises equipment
- television VOD systems stream media content via the communications system 108 from files stored at a content storage system of a content delivery network operably coupled to the communication system 108 , under direct or indirect control of the program distributor 106 , to the receiving device 118 .
- Television VOD systems may stream content to a receiving device 118 such as a set-top box, DVD player, game system, smart phone, television (including a smart TV), PC, a sound system receiver, a digital video recorder (“DVR”), a compact disk (“CD”) device, tablet device, mobile device or other computing device or media player, and the like, allowing viewing in real time at the customer premises 116 , or downloading to a receiving device 118 such as a computer, DVR (also called a personal video recorder) or portable media player for viewing at any time.
- a receiving device 118 such as a set-top box, DVD player, game system, smart phone, television (including a smart TV), PC, a sound system receiver, a digital video recorder (“DVR”), a compact disk (“CD”) device, tablet device, mobile device or other computing device or media player, and the like, allowing viewing in real time at the customer premises 116 , or downloading to a receiving device 118 such as a computer, DVR (also called a personal video record
- the receiving device 118 may be a set-top box that is typically provided by the cable provider, satellite provider, or other program distributor 106 to which the customer may subscribe to receive such television services and that also receives programming through other channels using a terrestrial, satellite signal, and/or cable television format.
- the mobile device 128 in communication with the receiving device 118 may be provided by the program distributor, such as a cable provider or satellite provider.
- the program distributor such as a cable provider or satellite provider.
- such mobile devices may be devices other than those provided by the program distributor 106 .
- these may include various user devices such as a tablet device, game machine, smartphone, mobile device or other computing device or media player or communications device not provided by or controlled by the cable provider, satellite provider, or other program distributor to which the customer subscribes for the television services.
- the supplemental content server 122 is operably coupled to the communication system 108 and may be a system of an entity that provides processing and/or analysis of a program content stream sent to receiving device 118 and may also generate supplemental content for the program content stream as the program content stream is being received by the receiving device 118 based on the processing of the verbal content of the program content stream.
- the supplemental content server 122 may then communicate to the receiving device 118 results of such processing and/or analysis of the program content stream.
- the supplemental content server 122 may also or instead transmit generated supplemental content for the program content stream as the program content stream is being delivered to and/or received by the receiving device 118 .
- the environment 102 may include many systems associated with a television service provider, such as the program distributor 106 , that operate in a manner such as the supplemental content server 122 described herein, and the supplemental content server 122 represents one or more of such systems in various embodiments. Such functionality may also or instead be performed by the receiving device 118 , mobile device 128 and/or remote control device 130 .
- information provider 138 may provide various forms of content and/or services to various devices residing in the customer premises 116 .
- information provider 138 may provide information to the receiving device 118 regarding customer or user accounts associated with the program content stream and/or receiving device 118 ; corresponding user preferences and settings; and/or additional processing of the program content stream to enable the receiving device 118 and/or supplemental content server 122 to generate corresponding supplemental content.
- Such services may include artificial intelligence and machine learning functionality to improve the supplemental content in general and for particular programs based on a feedback loop from the receiving device 118 , mobile device 128 and/or receiving devices and mobile devices of other users.
- FIG. 1 illustrates just one example of an environment 102 and the various embodiments discussed herein are not limited to such environments.
- environment 102 and the various devices therein may contain other devices, systems and/or media not specifically described herein.
- the environment 102 may include many different supplemental content servers, content providers, program distributors and information providers that operate in a corresponding manner as the supplemental content server 122 , the content provider 104 , program distributor 106 and information provider 138 as described herein.
- Each of supplemental content server 122 , the content provider 104 , program distributor 106 and information provider 138 may represent one or more of such systems in various embodiments.
- Example embodiments described herein provide applications, tools, data structures and other support to implement generating supplemental content for a program content stream.
- Other embodiments of the described techniques may be used for other purposes, including generating supplemental content for formal educational programs in schools and for remote learning, generating supplemental content for sports programs, movies, news programs and other types of content.
- sports programs often have multiple announcers who typically identify themselves. Therefore, the system may recognize the particular announcer's voice and generate questions or other supplemental content based on individual announcers.
- the system may present a question based on the verbal content in the sports program of the recognized sports announcer, such as the question “Bob Costas thought this was the best play of the game. A,B,C”.
- the system may automatically generate quiz print outs for teachers when not all students have mobile devices. These printouts may be generated from previous natural language processing (NLP) of content stored and sent in the metadata associated with the content.
- NLP natural language processing
- an application running on the mobile device 128 and/or the receiving device 118 may connect to a wireless local printer so students can follow along and answer while watching on the presentation device 120 and/or an individual mobile device 128 .
- the printouts may include multiple choice questions or configured in a way that they are easily readable and scored via the application running on the mobile device 128 by taking a photo of it.
- the application running on the mobile device 128 and/or receiving device 118 may record and display student scores and rankings within a very short time, on the screen of the presentation device 120 or on the screen of the mobile device 128 via the application running on the mobile device 128 .
- generating supplemental content for a program content stream may be performed by various other devices, such as audio and DVD players, digital recorders, computers, peripherals, televisions, mobile devices, active speakers, headphones, telephones, and other electronic devices, etc.
- audio and DVD players digital recorders
- computers peripherals
- televisions mobile devices
- active speakers headphones
- telephones and other electronic devices
- numerous specific details are set forth, such as data formats, program sequences, processes, and the like, in order to provide a thorough understanding of the described techniques.
- the embodiments described also can be practiced without some of the specific details described herein, or with other specific details, such as changes with respect to the ordering of the code flow, different code flows, and the like.
- the scope of the techniques and/or functions described are not limited by the particular order, selection, or decomposition of steps described with reference to any particular module, component, or routine.
- FIG. 2 is a block diagram illustrating elements of an example receiving device 118 used in secure communications between media devices, according to one example embodiment.
- the receiving device 118 is a device such as a set-top box, television, DVR, DVD player, PC, tablet device, game machine, smartphone, mobile device or other computing device or media player configured to receive programming via a connection to a satellite or cable television service provider, or other media content provider, outside the customer premises and to display such programming on a presentation device.
- the receiving device 118 may be configured to receive, process and display on a presentation device 120 streaming media content received directly from the satellite or cable television service provider and/or other programming received directly from the satellite or cable television service provider such as cable or satellite television broadcasts via various other physical and logical channels of communication.
- one or more general purpose or special purpose computing systems/devices may be used to operate the receiving device 118 ; store information regarding the receiving device 118 , store metadata; perform DRM and key management operations, decrypt and encrypt received content; pair with various mobile devices on a home LAN and/or over the Internet; establish connection between the receiving device 118 and various mobile devices; communicate data, including programming, between the receiving device 118 and various mobile devices (including mobile device 128 ) via the home LAN and/or Internet; and communicate with the content provider 104 , program distributor 106 , information provider 138 and/or supplemental content server 122 .
- the receiving device 118 may comprise one or more distinct computing systems/devices and may span distributed locations.
- each block shown may represent one or more such blocks as appropriate to a specific embodiment or may be combined with other blocks.
- the receiving device operation manager 222 may be implemented in software, hardware, firmware, or in some combination to achieve the capabilities described herein.
- receiving device 118 comprises a computer memory (“memory”) 201 , a display 202 (including, but not limited to a light emitting diode (LED) panel, cathode ray tube (CRT) display, liquid crystal display (LCD), touch screen display, etc.), one or more Central Processing Units (“CPU”) 203 , Input/Output (“I/O”) devices 204 (e.g., keyboard, mouse, RF or infrared receiver, universal serial bus (USB) ports, High-Definition Multimedia Interface (HDMI) ports, other communication ports, and the like), other computer-readable media 205 , and network connections 206 .
- the receiving device operation manager 222 is shown residing in memory 201 .
- some portion of the contents and some, or all, of the components of the receiving device operation manager 222 may be stored on and/or transmitted over the other computer-readable media 205 .
- the components of the receiving device 118 and operation manager 222 preferably execute on one or more CPUs 203 and facilitate the receiving, decrypting, decoding, processing, selecting, recording, playback and displaying of programming, as well as the establishing of an Internet Layer end-to-end security connection, such as a secure IP tunnel, over the home LAN and/or Internet between the receiving device 118 and various mobile devices and communication of data, including programming, between the receiving device 118 and various mobile devices via the Internet Layer end-to-end security connection over the home LAN and/or Internet, as described herein.
- an Internet Layer end-to-end security connection such as a secure IP tunnel
- the receiving device operation manager 222 may also facilitate on-demand media services (e.g., VOD services), on-demand program ordering, processing and DRM and key management and storage corresponding to processing received streaming media content and other programming.
- the receiving device operation manager 222 may operate as, be part of, or work in conjunction and/or cooperation with various on-demand service software applications stored in memory 201 and on various mobile devices.
- the receiving device operation manager 222 also facilitates communication with peripheral devices such as a remote control, via the I/O devices 204 , and with the mobile device 128 and remote systems (e.g., the content provider 104 , the supplemental content server 122 , the program distributor 106 , and/or the information provider 138 ) via the network connections 206 .
- Recorded or buffered programming of a program content stream received as streaming media content or other types of programming may reside on the media content storage 216 , either in decrypted or encrypted form as applicable for securely storing, processing and displaying of the received media content according to the applicable DRM associated with the particular programming.
- the media content storage 216 may also store various generated supplemental content, program metadata associated with the recorded or buffered programming and/or supplemental content stored in the media content storage 216 , such as that including, but not limited to, generated questions having answers that are extracted educational concepts communicated by the verbal content of the program content stream; answers to such questions; DRM data; tags; codes, identifiers, format indicators, timestamps, user identifications, authorization codes, digital signatures, etc.
- the verbal content processing engine 224 is configured to receive the program content stream via the network connections 206 and/or other I/O devices 204 and process verbal content of the program content stream as the program content stream is being received.
- the verbal content processing engine 224 may extract audio and/or textual verbal content from the program content stream, convert or transcribe the extracted audio to text and then immediately process such data while the content of the program content stream is being output by the receiving device 118 for presentation on the presentation device 120 .
- the verbal content processing engine 224 may extract textual content from the program content stream via accessing the closed captioning and/or subtitle metadata associated and/or received with the program content stream.
- Such textual data may have been previously inserted in the vertical blanking interval of the television signal comprising the program content stream.
- the verbal content processing engine 224 may perform speech-to-text conversion of the extracted audio of the program content stream as the program content stream is being received by the receiving device 118 .
- the verbal content processing engine 224 may perform natural language processing (NLP) on the extracted textual content or extracted audio that has been converted to text as the program content stream is being output for presentation on a presentation device, such as presentation device 120 .
- NLP is a branch of artificial intelligence that helps computers understand, interpret and manipulate human language.
- NLP may use representation learning and deep neural network-style machine learning methods to recognize concepts, topics, facts, and statements made in the verbal content of the program content stream. Such techniques can perform many such natural language tasks, for example in language modeling, parsing, and many others.
- Some techniques include the use of word embeddings to capture semantic properties of words, and an increase in end-to-end learning of a higher-level task (e.g., question answering) instead of relying on a pipeline of separate intermediate tasks (e.g., part-of-speech tagging and dependency parsing).
- Deep neural network-based approaches may be used instead of or in addition to distinct statistical natural language processing.
- neural machine translation NMT
- deep learning-based approaches to machine translation directly learn sequence-to-sequence transformations, obviating the need for intermediate steps such as word alignment and language modeling that are used in statistical machine translation (SMT).
- the verbal content processing engine 224 may perform NLP on the text associated with the program content stream to extract educational concepts communicated by the text.
- the NLP of the text may recognize that the text, “Lions typically live in the wild for 10-14 years” is a grammatical statement that may serve as a basis for, or may be paired to, a corresponding educational question, “How long do lions live for?”
- the verbal content processing engine 224 may then store such question and answer associations and/or pairings in the media content storage 216 or data repository 220 .
- the verbal content processing engine 224 may communicate such question and answer associations and pairings, as such associations are made, to the supplemental content generation and processing engine 226 for immediate further processing.
- the supplemental content generation and processing engine 226 then generates supplemental content for the program content stream as the program content stream is being received based on the output of the verbal content processing engine 224 that resulted from the processing of the verbal content of the program content stream.
- the supplemental content generation and processing engine 226 may generate questions having answers that are one or more of the educational concepts communicated by the verbal content and that were extracted by the verbal content processing engine 224 .
- the supplemental content generation and processing engine 226 may select or generate the question “How long do lions live for?” based on the NLP of the verbal content of the program content stream.
- the NLP of the verbal content of the program content stream may recognize that the text, “Lions typically live in the wild for 10-14 years” extracted from the verbal content is a grammatical statement that may serve as a basis for the question “How long do lions live for?”
- the supplemental content generation and processing engine 226 may iterate through all such associations made by verbal content processing engine 224 during the NLP of the program content stream, in order, as such associations are being made, such as to present the corresponding questions as the program content stream is being presented on the presentation device 120 . This may be performed in real time or near real time as the program content stream is being output for presentation on the presentation device 120 .
- the supplemental content generation and processing engine 226 may cause the supplemental content to be presented as the program content stream is being output for presentation on the presentation device 120 .
- the supplemental content generation and processing engine 226 may cause the receiving deice 118 to pause the presentation of the program content stream on the presentation device 120 and, in response to the pausing the presentation of the program content stream on the presentation device 120 , cause the corresponding question be presented on the presentation device 120 .
- the program content stream may be buffered in the media content storage 216 or data repository 220 while it is paused such that after the program is un-paused or otherwise resumed, the receiving device 118 can present the program content stream on the presentation device 120 and/or mobile device 128 beginning from the point it was paused.
- the supplemental content generation and processing engine 226 may select particularly relevant or convenient points in the program content stream to pause the presentation in order to present the question. For example, the supplemental content generation and processing engine 226 may select a point in the program immediately before or after a commercial break, a point in the program related to the question and/or a point in the program designated for insertion of supplemental content. In some embodiments, the question may be presented in response to the user pausing the program or otherwise providing input that may indicate the question may be presented.
- the supplemental content generation and processing engine 226 may cause a video portion of the program content stream to be presented that is relevant to a particular question while the question is being presented.
- the supplemental content generation and processing engine 226 may cause this to occur while the presentation of the program content stream is paused in order for the question to be presented. For example, if a question regarding lions is presented, a video clip or image from the program related to lions (or perhaps even including information answering or hinting at the answer to the question) may be presented on the presentation device 120 and/or mobile device 128 while the question is being presented on the presentation device 120 and/or mobile deice 128 .
- Such associations between generated questions and content related or relevant to the questions may be made as a result of the NLP of the verbal content of the program content stream in conjunction with object detection and recognition processing of the program content stream by the supplemental content generation and processing engine 226 in order to recognize relevant objects or video scenes in the program content stream.
- the supplemental content generation and processing engine 226 may generate various types of supplemental content based on the NLP of verbal content and/or object detection and recognition processing of other objects or scenery in the content of the program content stream performed by the supplemental content generation and processing engine 226 .
- supplemental content may include, but is not limited to: questions; fun facts; trivia; additional information about topics, subjects, objects, people or places recognized in the content of the program content stream; purchase opportunities related to topics, subjects, topics, subjects, objects, people or places recognized in the content of the program content stream; games related to topics, subjects, objects, people or places recognized in the content of the program content stream; and advertisements related to topics, subjects, objects, people or places recognized in the content of the program content stream.
- the supplemental content generation and processing engine 226 may cause an interactive element to be presented with the question such that the user may select an answer to the question via the interactive element.
- a multiple choice question may be presented on the presentation device 120 with the answers being selectable by the user with the remote control 30 and/or the mobile device 128 .
- the questions are also or instead presented on the mobile device 128 of the user.
- the program content stream is not paused, such that the user may continue to watch the programming on the presentation device 120 (e.g., on their television) while looking to their mobile device 128 to view and answer the question related to the program content stream.
- the supplemental content generation and processing engine 226 may pause the presentation of the program content stream on the presentation device 120 and/or mobile device 128 at various different times during the presentation of the program content stream.
- the supplemental content generation and processing engine 226 may cause a different question of the generated questions to be presented on the presentation device 120 and/or or on the mobile device 128 , at each of the various different times the presentation of the program content stream is paused on the presentation device 120 and/or mobile device 128 .
- the supplemental content generation and processing engine 226 may customize the questions and other supplemental content according to the preferences and corresponding settings of a specific user.
- the supplemental content generation and processing engine 226 may receive input indicative of a selection of one or more of: an age level of a user that is to view the supplemental content; a number of questions comprising the supplemental content for a particular program of the program content stream; and a frequency of presentation of items comprising the supplemental content.
- the supplemental content generation and processing engine 226 may then cause the supplemental content to be presented as the program content stream is being output for presentation on the presentation device 120 and/or the mobile device 128 based on the input.
- the supplemental content generation and processing engine 226 may select questions to be presented that are categorized at a particular difficulty level associated with the age level input by the user. Also, the supplemental content generation and processing engine 226 may determine to present only 3 questions per television program based on an input of a user indicating a frequency of 3 questions per television program. Additionally, the supplemental content generation and processing engine 226 may determine to present a particular number of questions to present for a particular program based on input from a user indicating how many questions are to be presented based on the name, title or content of the specific program.
- the supplemental content generation and processing engine 226 may also store the generated supplemental content as metadata associated with the program for subsequent presentation when the program is played again for the viewer or other viewers, thus further increasing efficiency of generating further supplemental content.
- the supplemental content generation and processing engine 226 may store, in media content storage 216 and/or data repository 220 , the generated questions as metadata associated with a corresponding episode of a television program of the program content stream. Such generated questions may be stored for presentation as supplemental content of the corresponding program episode during subsequent playback of the program episode on presentation device 120 , mobile device 128 and/or other devices of other users or customers.
- the answer processing engine 228 may receive an input indicative of an answer to the question.
- the input may originate as voice input to the remote control 130 , receiving device 118 and/or the mobile device 128 .
- the remote control 130 , receiving device 118 and/or the mobile device 128 may incorporate and/or use use corresponding voice-command, voice-enabled, speech recognition and/or voice recognition functionality to recognize and process voice commands and/or other speech accordingly.
- the remote control 130 may be a voice-enabled remote control device including a microphone that recognizes speech or relays relevant audio to the receiving device 118 in order for the answer processing engine 228 of the receiving device 118 to recognize the speech as an answer to the question presented.
- the supplemental content generation and processing engine 226 may cause an interactive element to be presented with the question such that the user may select an answer to the question via the interactive element.
- a multiple choice question may be presented on the presentation device 120 and/or the mobile device 128 with the answers being selectable by the user with the remote control 30 , receiving device 118 and/or the mobile device 128 .
- Input indicative of selection of an answer such as a click, press or voice selection, may then be communicated to the answer processing engine 228 .
- the answer processing engine 226 may then determine a response to the answer and communicate the response to the answer.
- the answer processing engine 228 may communicate the response visually on the presentation device 120 and/or mobile device 128 .
- the presentation of the program content stream is paused while the question is being presented and the answer processing engine 228 causes the presentation of the program content stream to resume in response to, or otherwise based on, an answer being received to the question.
- the resumption of the presentation of the program content stream may be in response to an answer being received to the question or the answer processing engine 228 determining that a timer has expired, whichever occurs first.
- the resuming of the presentation of the program content stream may be in response to the answer processing engine 228 determining that the answer to the selected question is correct.
- the answer processing engine 228 may compare the input indicative of the answer to the question to an educational concept associated with the question presented on the presentation device 120 and/or mobile device 128 . This may be performed while the presentation of the program content stream is paused. The answer processing engine 228 may then determine whether the answer to the selected question is correct based on the comparison while the presentation of the program content stream is paused. The answer processing engine 228 may also determine the response to the answer based on the determination of whether the answer to the selected question is correct while the presentation of the program content stream is paused.
- the answer processing engine 228 causes the presentation of the program content stream to resume in response to communicating a response to the answer.
- the answer processing engine 228 may resume the presentation of the program content stream on the presentation device 120 and/or mobile device 128 after each of the various different times the presentation of the program content stream is paused in response to receiving a corresponding correct answer to a question presented on the presentation device 120 and/or on the mobile device 128 at each of the various different times the presentation of the program content stream is paused.
- the answer processing engine 228 may display or otherwise communicate to the user the answer was incorrect, may display or otherwise communicate to the user the correct answer and/or may replay the portion of the content that states, illustrates or otherwise includes the correct answer. For example, this may be accomplished by the answer processing engine 228 finding the portion of content including the correct answer using a timestamp corresponding to the location in the content where the correct answer is included.
- the supplemental content generation and processing engine 226 may record such timestamps that indicate where in the content the corresponding verbal content associated with the correct answer appears and associate the timestamp with the question and corresponding correct answer. The answer processing engine 228 may then access such timestamp to locate the content that includes the correct answer and then play such content in response to receiving an incorrect answer and/or in response to a request from the user to view the content including the correct answer.
- the answer processing engine 228 controls an amount of viewing time allowed on the presentation device 120 and/or one or more mobile devices (such as mobile device 128 ), based on a number of correct answers to the different questions presented on the presentation device 120 and/or on a mobile device of the user received at each of the various different times the presentation of the program content stream is paused on the presentation device.
- the answer processing engine 228 may be in communication with and/or control the mobile device 128 and/or other mobile devices associated with the receiving device 118 .
- the answer processing engine 228 may also have control over or have access to a user account that is associated with the receiving device 118 and/or the mobile device 128 .
- the answer processing engine 228 may also or instead be running on the mobile device 128 .
- a parent may select parental controls available on the receiving device 118 and/or mobile device 128 that limit the viewing time or “screen time” of one or more devices or an account associated with a child to 5 hours per week until 50 supplemental content questions have been correctly answered within that week, which may then be increased to 7 hours for that week.
- the number of viewing hours, number of correctly answered questions and rate of increase or decrease of allowed viewing time may be selectable by the user via a user interface or portal generated or provided by the answer processing engine 228 of the receiving device 118 , the program distributor 106 , the content provider 104 , information provider 138 and/or supplemental content server 122 .
- Other code or programs 230 e.g., further audio/video processing modules, a program guide manager module, a Web server, and the like
- data repositories such as data repository 220 for storing other data (user profiles, preferences and configuration data, etc.)
- data repository 220 for storing other data (user profiles, preferences and configuration data, etc.)
- CPUs 203 may also reside in the memory 201 , and preferably execute on one or more CPUs 203 .
- one or more of the components in FIG. 2 may or may not be present in any specific implementation. For example, some embodiments may not provide other computer readable media 205 or a display 202 .
- the receiving device 118 and operation manager 222 include an application program interface (“API”) that provides programmatic access to one or more functions of the receiving device 118 and operation manager 222 .
- API application program interface
- such an API may provide a programmatic interface to one or more functions of the receiving device operation manager 222 that may be invoked by one of the other programs 230 , a remote control 130 , the mobile device 128 , program distributor 106 , content provider 104 , information provider 138 , supplemental content server 122 or some other module.
- the API may facilitate the development of third-party software, such as various different on-demand service applications, user interfaces, plug-ins, adapters (e.g., for integrating functions of the receiving device operation manager 222 and information provider 138 into mobile device and/or desktop applications), and the like to facilitate generating supplemental content for a program content stream using the receiving device 118 .
- third-party software such as various different on-demand service applications, user interfaces, plug-ins, adapters (e.g., for integrating functions of the receiving device operation manager 222 and information provider 138 into mobile device and/or desktop applications), and the like to facilitate generating supplemental content for a program content stream using the receiving device 118 .
- components/modules of the receiving device 118 and operation manager 222 are implemented using standard programming techniques.
- the receiving device operation manager 222 may be implemented as a “native” executable running on the CPU 203 , along with one or more static or dynamic libraries.
- the receiving device 118 and operation manager 222 may be implemented as instructions processed by a virtual machine that executes as one of the other programs 230 .
- a range of programming languages known in the art may be employed for implementing such example embodiments, including representative implementations of various programming language paradigms, including but not limited to, object-oriented (e.g., Java, C++, C#, Visual Basic.NET, Smalltalk, and the like), functional (e.g., ML, Lisp, Scheme, and the like), procedural (e.g., C, Pascal, Ada, Modula, and the like), scripting (e.g., Perl, Ruby, Python, JavaScript, VBScript, and the like), or declarative (e.g., SQL, Prolog, and the like).
- object-oriented e.g., Java, C++, C#, Visual Basic.NET, Smalltalk, and the like
- functional e.g., ML, Lisp, Scheme, and the like
- procedural e.g., C, Pascal, Ada, Modula, and the like
- scripting e.g., Perl, Ruby, Python, JavaScript, VBScript, and
- instructions stored in a memory configure, when executed, one or more processors of the receiving device 118 to perform the functions of the receiving device operation manager 222 .
- instructions cause the CPU 203 or some other processor, such as an I/O controller/processor, to perform NLP of content of a program content stream as the as the program content stream is being output for presentation on a presentation device; extract educational concepts communicated by verbal content of the program content stream based on the performing of the natural language processing of the verbal content; and cause questions based on the extracted educational concepts to be presented on the presentation device or a mobile device of a user at different times while the presentation of the program content stream is paused.
- the embodiments described above may also use well-known or other synchronous or asynchronous client-server computing techniques.
- the various components may be implemented using more monolithic programming techniques as well, for example, as an executable running on a single CPU computer system, or alternatively decomposed using a variety of structuring techniques known in the art, including but not limited to, multiprogramming, multithreading, client-server, or peer-to-peer, running on one or more computer systems each having one or more CPUs or other processors.
- Some embodiments may execute concurrently and asynchronously, and communicate using message passing techniques.
- Equivalent synchronous embodiments are also supported by a receiving device operation manager 222 implementation.
- other functions could be implemented and/or performed by each component/module, and in different orders, and by different components/modules, yet still achieve the functions of the receiving device 118 and operation manager 222 .
- programming interfaces to the data stored as part of the receiving device 118 and operation manager 222 can be available by standard mechanisms such as through C, C++, C#, and Java APIs; libraries for accessing files, databases, or other data repositories; scripting languages such as XML; or Web servers, FTP servers, NFS file servers, or other types of servers providing access to stored data.
- the media content storage 216 and data repository 220 may be implemented as one or more database systems, file systems, or any other technique for storing such information, or any combination of the above, including implementations using distributed computing techniques.
- some or all of the components of the receiving device 118 and operation manager 222 may be implemented or provided in other manners, such as at least partially in firmware and/or hardware, including, but not limited to one or more application-specific integrated circuits (“ASICs”), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and/or embedded controllers), field-programmable gate arrays (“FPGAs”), complex programmable logic devices (“CPLDs”), and the like.
- ASICs application-specific integrated circuits
- controllers e.g., by executing appropriate instructions, and including microcontrollers and/or embedded controllers
- FPGAs field-programmable gate arrays
- CPLDs complex programmable logic devices
- system components and/or data structures may also be stored as contents (e.g., as executable or other machine-readable software instructions or structured data) on a computer-readable medium (e.g., as a hard disk; a memory; a computer network, cellular wireless network or other data transmission medium; or a portable media article to be read by an appropriate drive or via an appropriate connection, such as a DVD or flash memory device) so as to enable or configure the computer-readable medium and/or one or more associated computing systems or devices to execute or otherwise use, or provide the contents to perform, at least some of the described techniques.
- a computer-readable medium e.g., as a hard disk; a memory; a computer network, cellular wireless network or other data transmission medium; or a portable media article to be read by an appropriate drive or via an appropriate connection, such as a DVD or flash memory device
- FIGS. 3A and 3B are diagrams of an example presentation device 120 and an example handheld mobile device 128 , respectively, illustrating example user interface screens shown in a presentation of an example question as supplemental content for a program content stream, according to an example embodiment.
- the receiving device 118 (shown in FIG. 1 and FIG. 2 connected to the presentation device 120 and handheld mobile device 128 ) is outputting a program content stream to the presentation device 120 including an educational program 306 about the life of lions in the wild in Africa. While outputting a program content stream to the presentation device 120 , the receiving device 118 has extracted the audio of the program 306 and performed NLP on the verbal content of the program 306 . The receiving device 118 has generated a question “How long do lions live for?” 304 based on the NLP of the audio extracted from the program 306 . The receiving device 118 has presented the question “How long do lions live for?” 304 along with multiple choice answers 308 to the question 304 on the presentation device screen 302 as an overlay on top of the program 306 .
- the receiving device 118 has paused the presentation of the program 306 while presenting the question 304 so that the viewer may have time to answer the question 304 without missing any of the program 306 .
- a corresponding question 314 and associated corresponding multiple choice answers 318 are also being presented on a mobile device screen 312 of the mobile device 128 of the user, who may also be viewing the program 306 on the presentation device 120 .
- the question may instead be presented only on the mobile device 128 or only on the presentation device 120 .
- the program 306 may be presented along with the corresponding question 314 on the mobile device 128 and/or along with the question 304 on the presentation device 120 .
- the mobile device 128 may pause the presentation of the program 306 on the mobile device 128 while presenting the corresponding question 314 .
- the program 306 may continue to play on the presentation device 120 while the corresponding question 314 is presented only on the mobile device 128 , such that the user may continue to view the program 306 on the presentation device 120 while looking to their mobile device 128 to view and answer the corresponding question 314 related to the program 306 .
- the user may respond to the question 304 presented on the presentation device 120 and/or the corresponding question 314 presented on the mobile device 128 in a variety of manners.
- the remote control 130 , the receiving device 118 , the presentation device 120 and/or the mobile device 128 are equipped with a microphone and voice-enabled such that the user may respond by speaking the answer to the question 304 and/or corresponding question 314 .
- the presentation of the question 304 on the presentation device 120 and/or the corresponding question 314 on the mobile device 128 may trigger the remote control 130 , the receiving device 118 , the presentation device 120 and/or the mobile device 128 to start listening for an answer.
- the multiple choice answers 308 presented on the presentation device 120 and/or the corresponding multiple choice answers 318 presented on mobile device 128 are selectable interactive user interface elements, such that the user may select the answer by using the remote control 30 , touching the selection if the presentation device screen 302 or mobile device screen 312 is a touch screen, clicking on the selection using an input device such as a mouse, and/or selecting the desired answer using another input device.
- the selection may be communicated to the receiving device 118 and/or presentation device 120 via communication system 108 shown in FIG. 1 .
- FIGS. 4A and 4B are diagrams of an example presentation device 120 and an example handheld mobile device 128 , respectively, illustrating example user interface screens shown in an example presentation of a response to an answer received to the example question presented in FIGS. 3A and 3B , according to an example embodiment.
- a response 404 is presented on the presentation device 120 .
- the user has responded by speaking the choice “A”, which the receiving device 118 (shown in FIG. 1 and FIG. 2 connected to the presentation device 120 and handheld mobile device 128 ) recognized as the correct answer.
- the receiving device 118 has presented the response “That is correct!” 404 on the presentation device screen 302 .
- a confirmation of the answer received 402 may be presented on the on the presentation device screen 302 to verify and reinforce learning.
- the receiving device 118 then resumes presentation of the program 306 on the presentation device screen 302 .
- An indication that the program will resume 406 may also be presented on the presentation device screen 302 .
- the receiving device 118 and/or the mobile device 128 may cause a corresponding response “That is correct!” 414 , a corresponding confirmation of the answer received 412 and a corresponding indication that the program will resume 416 to also or instead be presented on the mobile device screen 312 .
- the program may be resumed on the mobile device screen 312 and/or the presentation device screen 302 based on a preference of the user as indicated by an input or selection of a corresponding setting by the user on the mobile device 128 and/or the receiving device 118 .
- FIGS. 5A and 5B are diagrams of an example presentation device 120 and an example handheld mobile device 128 , respectively, illustrating respective example user interface screens shown in an example presentation of a response to an incorrect answer received to the example question presented in FIGS. 3A and 3B , according to an example embodiment.
- a response 504 is presented on the presentation device 120 .
- the user has responded by speaking the choice “B”, which the receiving device 118 (shown in FIG. 1 and FIG. 2 connected to the presentation device 120 and handheld mobile device 128 ) recognized as an incorrect answer.
- the receiving device 118 has presented the response “Sorry, that is incorrect.” 504 on the presentation device screen 302 .
- a confirmation of the answer received 502 may be presented on the on the presentation device screen 302 for verification.
- the receiving device 118 may then replay the portion of the program 306 that states, illustrates or otherwise includes the correct answer on the presentation device screen 302 .
- An indication that the portion of the program including the correct answer will play 506 may also be presented on the presentation device screen 302 .
- the receiving device 118 and/or the mobile device 128 may cause a corresponding response “Sorry, that is incorrect” 514 , a corresponding confirmation of the answer received 512 and a corresponding indication that the portion of the program including the correct answer will play 516 to also or instead be presented on the mobile device screen 312 .
- the portion of the program including the correct answer may play on the mobile device screen 312 and/or the presentation device screen 302 based on a preference of the user as indicated by an input or selection of a corresponding setting by the user on the mobile device 128 and/or the receiving device 118 .
- FIG. 6 is a flow diagram of an example method 600 for generating supplemental content for a program content stream, according to an example embodiment.
- a computerized system for generating supplemental content for a program content stream receives the program content stream.
- the system processes verbal content of the program content stream as the program content stream is being received.
- the system generates supplemental content for the program content stream as the program content stream is being received based on the processing of the verbal content of the program content stream.
- the system causes the supplemental content to be presented as the program content stream is being output for presentation on a presentation device.
- FIG. 7 is a flow diagram of an example method 700 for storing generated questions as metadata associated with a program of the program content stream, according to an example embodiment.
- a system for generating supplemental content for a program content stream generates questions, while a program content stream is being output for presentation on a presentation device.
- the questions have answers that are one or more extracted educational concepts communicated by verbal content of the program content stream.
- the system causes the questions to be presented on the presentation device or a mobile device of a user at different times while the presentation of the program content stream is paused.
- the system stores the generated questions, as metadata associated with a program of the program content stream, for presentation as supplemental content of the program during subsequent playback of the program.
- FIG. 8 is a flow diagram of an example method 800 for extracting educational concepts communicated by verbal content of the program content stream, according to an example embodiment.
- a system for generating supplemental content for a program content stream performs natural language processing of content of a program content stream as the as the program content stream is being output for presentation on a presentation device.
- the system extracts educational concepts communicated by verbal content of the program content stream based on the performing of the natural language processing of the verbal content.
- the system causes questions based on the extracted educational concepts to be presented on the presentation device or a mobile device of a user at different times while the presentation of the program content stream is paused.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Acoustics & Sound (AREA)
- Human Computer Interaction (AREA)
- Educational Technology (AREA)
- Educational Administration (AREA)
- Business, Economics & Management (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
Description
- The technical field relates to delivering multimedia content, and particularly to generating supplemental content for a program content stream.
- Supplemental content such as educational questions may be generated based on the content of the program as learned by the system via processing the verbal content of the program content stream (e.g., using natural language processing) as the program is being presented on a television or mobile device. The system may pause the original programming (or the playback of recorded programming) during presentation of the questions on the television or mobile device. The system then resumes the programming once a correct answer is received or a timer runs out, whichever occurs first. In this manner, any program with verbal content (spoken or written) may be processed as it is being received in real time or near real time and morphed into a more educational experience for the viewer. This eliminates the need for the program distributor or content creator to manually generate custom content to supplement the program before being broadcast and also enables the supplemental content to be customized according to the preferences and corresponding settings of the individual user. The generated supplemental content may also be saved as metadata associated with the program for subsequent presentation when the program is played again for the viewer or other viewers, thus further increasing efficiency of generating further supplemental content.
- A method for for generating supplemental content for a program content stream, may be summarized as comprising: receiving, by a computerized system for generating supplemental content, the program content stream; processing, by the computerized system for generating supplemental content, verbal content of the program content stream as the program content stream is being received; generating, by the computerized system for generating supplemental content, supplemental content for the program content stream as the program content stream is being received based on the processing of the verbal content of the program content stream; and causing, by the computerized system for generating supplemental content, the supplemental content to be presented as the program content stream is being output for presentation on a presentation device.
- A system for generating supplemental content for a program content stream may be summarized as comprising: at least one computer processor; and at least one memory coupled to the at least one computer processor. The at least one memory has computer-executable instructions stored thereon that, when executed by the at least one computer processor, cause the at least one computer processor to: generate questions, while a program content stream is being output for presentation on a presentation device, having answers that are one or more of extracted concepts communicated by verbal content of the program content stream; cause the questions to be presented on the presentation device or a mobile device of a user at different times while the presentation of the program content stream is paused; and store the generated questions, as metadata associated with a program of the program content stream, for presentation as supplemental content of the program during subsequent playback of the program.
- Another method for generating supplemental content for a program content stream, may be summarized as comprising: performing natural language processing of content of a program content stream as the as the program content stream is being output for presentation on a presentation device; extracting concepts communicated by verbal content of the program content stream based on the performing of the natural language processing of the verbal content; and causing questions based on the extracted concepts to be presented on the presentation device or a mobile device of a user at different times while the presentation of the program content stream is paused.
- The components in the drawings are not necessarily drawn to scale relative to each other. Like reference numerals designate corresponding parts throughout the several views.
-
FIG. 1 is an overview block diagram illustrating a technical environment in which embodiments of systems and methods for generating supplemental content for a program content stream may be implemented, according to an example embodiment. -
FIG. 2 is a block diagram illustrating elements of an example receiving device used in systems and methods for generating supplemental content for a program content stream, according to an example embodiment. -
FIGS. 3A and 3B are diagrams of an example presentation device and an example handheld mobile device, respectively, illustrating respective example user interface screens shown in a presentation of an example question as supplemental content for a program content stream, according to an example embodiment. -
FIGS. 4A and 4B are diagrams of an example presentation device and an example handheld mobile device, respectively, illustrating respective example user interface screens shown in an example presentation of a response to an answer received to the example question presented inFIGS. 3A and 3B , according to an example embodiment. -
FIGS. 5A and 5B are diagrams of an example presentation device and an example handheld mobile device, respectively, illustrating respective example user interface screens shown in an example presentation of a response to an incorrect answer received to the example question presented inFIGS. 3A and 3B , according to an example embodiment. -
FIG. 6 is a flow diagram of an example method for generating supplemental content for a program content stream, according to an example embodiment. -
FIG. 7 is a flow diagram of an example method for storing generated questions as metadata associated with a program of the program content stream, according to an example embodiment. -
FIG. 8 is a flow diagram of an example method for extracting educational concepts communicated by verbal content of the program content stream, according to an example embodiment. - Multimedia content is becoming increasingly available through various different media sources. However, most of the media content is not interactive and lacks educational value, especially to children. A technical problem is thus presented in how to make the abundance of popular existing media content that often is received as a program content stream on one's receiving device (such as a cable or satellite set-top box, streaming media device, tablet or smartphone) more educational and interactive in a faster and more efficient manner as such content is being received by the receiving device.
- The present disclosure provides a solution to the technical problem above by, in some embodiments, performing natural language processing (NLP) of the verbal content of the program content stream as the program content stream is being received to generate supplemental content for the program content. For example, this supplemental content may be or include educational questions that are based on the content of the program as learned by the system via performance of the NLP of the verbal content of the program content stream. The system may then pause the original programming (or the playback of recorded programming) to present the questions and then resume the programming once a correct answer is received or a timer runs out, whichever occurs first. In this manner, any program with verbal content (spoken or written) may be processed as it is being received in real time or near real time and morphed into a more educational experience for the viewer. This eliminates the need for the program distributor or content creator to manually generate custom content to supplement the program before being broadcast and also enables the supplemental content to be customized according to the preferences and corresponding settings of the individual user. The generated supplemental content may also be saved as metadata associated with the program for subsequent presentation when the program is played again for the viewer or other viewers, thus further increasing efficiency of generating further supplemental content.
-
FIG. 1 is an overview block diagram illustrating atechnical environment 102 in which embodiments of systems and methods for generating supplemental content for a program content stream may be implemented, according to an example embodiment. - Before providing additional details regarding the operation and constitution of systems and methods for generating supplemental content for a program content stream, the example
technical environment 102, within which such a system may operate, will briefly be described. - In the
technical environment 102, audio, video, and/or data service providers, such as television service providers, provide their customers a multitude of video and/or data programming (herein, collectively “programming” or “content”). Such programming is often provided as a program content stream for use by areceiving device 118 communicatively coupled to apresentation device 120 configured to receive the programming. The programming may include any type of media content, including, but not limited to: television shows, news, movies, sporting events, advertisements, etc. In various embodiments, any of this programming may be provided as a type of programming referred to as streaming media content, which is generally digital multimedia data that is substantially constantly received by and presented to an end-user or presented on a device while being delivered by a provider from a stored file source. Its verb form, “to stream,” refers to the process of delivering media in this manner. The term refers to how the media is delivered rather than the media itself. - In one embodiment, the
receiving device 118 is a device such as a set-top box, television, DVR, DVD player, PC, tablet device, game machine, smart phone, mobile device or other computing device or media player configured to receive programming via a connection to a satellite or cable television service provider or Internet service provider outside the customer premises 116, and to display such programming on apresentation device 120. For example, thereceiving device 118 may be configured to receive, process and display on thepresentation device 120 programming received directly from the satellite or cable television service provider, such as cable or satellite television broadcasts via various physical and logical channels of communication ofcommunication system 108. Also, thereceiving device 118 may be configured to receive, process and display on thepresentation device 120 streaming media content received directly from the satellite or cable television service provider and/orother content provider 104. - The
receiving device 118 interconnects to one or more communications media or sources. For example, the various media content may be delivered as data using the Internet Protocol (IP) suite over a packet-switched network such as the Internet or other packet-switched network, which may be included as part of thecommunication system 108. The underlying connection carrying such data may be via a cable head-end, satellite antenna, telephone company switch, cellular telephone system, Ethernet portal, off-air antenna, or the like. Thereceiving device 118 may receive a plurality of programming by way of the communications media or sources, or may only receive programming via a particular channel or source described in greater detail below. In some embodiments, based upon selection by a user, thereceiving device 118 processes and communicates the selected programming to thepresentation device 120. Also, in some embodiments, thepresentation device 120 may also be areceiving device 118 or have areceiving device 118 integrated within it. - Examples of a
receiving device 118 may include, but are not limited to devices such as, or any combination of: a “television converter,” “receiver,” “set-top box,” “television receiving device,” “television receiver,” “television,” “television recording device,” “satellite set-top box,” “satellite receiver,” “cable set-top box,” “cable receiver,” “media player,” “digital video recorder (DVR),” “digital versatile disk (DVD) Player,” “computer,” “mobile device,” “tablet computer,” “smart phone,” “MP3 Player,” “handheld computer,” “gaming console” and/or “television tuner,” etc. Accordingly, thereceiving device 118 may be any suitable converter device or electronic equipment that is operable to receive programming via a connection to a satellite or cable television service provider or other source outside the customer premises 116 and communicate that programming to another device. Further, thereceiving device 118 may itself include user interface devices, such as buttons or switches. In some example embodiments, thereceiving device 118 may be configured to receive and decrypt content and/or software or firmware updates according to various digital rights management (DRM) and other access control technologies and architectures as part of or in addition to the process of generating supplemental content for a program content stream, which will be described in further detail below. - Examples of a
presentation device 120 may include, but are not limited to, one or a combination of the following: a television (“TV”), a personal computer (“PC”), a sound system receiver, a digital video recorder (“DVR”), a compact disk (“CD”) device, DVD Player, game system, tablet device, smartphone, mobile device or other computing device or media player, and the like.Presentation devices 120 employ a display, one or more speakers, and/or other output devices to communicate video and/or audio content to a user. In some embodiments,presentation devices 120 employ a microphone, keyboard, and/or other input devices to communicate commands or other input to thepresentation device 120 and/or receivingdevice 118. In many implementations, one ormore presentation devices 120 reside in or near a customer premises 116 and are communicatively coupled, directly or indirectly, to thereceiving device 118. Further, thereceiving device 118 and thepresentation device 120 may be integrated into a single device. Such a single device may have the above-described functionality of the receivingdevice 118 and thepresentation device 120, or may even have additional functionality. - In many embodiments, a remote-control device (remote control) 130 is operable to control the receiving
device 118 and/or the presentation device 124. Theremote control 130 typically communicates with the receivingdevice 118 using a suitable wireless medium, such as infrared (IR), radio frequency (RF), or the like. Theremote control 130 may include a microphone that may be voice or audio activated in order to receive commands or other voice input. In some embodiments, such voice input may be communicated to and used by the receivingdevice 118,presentation device 120 and/ormobile device 128 and cause such devices to perform other actions. For example, such voice input may be processed by the receivingdevice 118,presentation device 120 and/ormobile device 128 as an answer or other response to supplemental content presented by such systems. In some embodiments, the remote 130 may be a voice-enabled remote and the same microphone may also be used to receive voice commands from the user and transmit corresponding commands to the receivingdevice 118 to control the receivingdevice 118 and/or other peripheral devices. - In the example embodiment shown in
FIG. 1 , handheldmobile device 128 may be communicatively connected to thetelevision receiving device 118 via a physical or a peer-to-peer short range wireless connection. For example,mobile device 128 may be a handheld mobile device of a user (e.g., a customer of a satellite, cable or streaming media subscription service associated with customer premises 116 or other user) such as a smartphone, tablet device or other computing or communications device. There may be fewer or additional mobile devices in various embodiments. Themobile device 128 may communicate overcommunication system 108 with thetelevision receiving device 118,program distributor 106,content provider 104,supplemental content server 122 and/orinformation provider 138 using the transmission control protocol/Internet protocol (TCP/IP) suite of networking communication protocols or other communication protocols. Themobile device 128 may also or instead communicate directly with the receivingdevice 128,remote control 130 and/orpresentation device 120 over one or more short range wireless networks or peer-to-peer networks. In the present example, a cable or satellite television service provider may encompass or be in communication with some or all of thecontent provider 104,program distributor 106,supplemental content server 122, andinformation provider 138. - In particular, a
content provider 104 provides program content, such as television content, to a distributor, such as theprogram distributor 106. Example program distributors include, but are not limited to, satellite and cable television service providers. Example content providers include television stations which provide local or national television programming and special content providers which provide premium based programming, streaming services, pay-per-view programming and on-demand programming. - Program content (i.e., a program including or not including advertisements), is communicated to the
program distributor 106 from thecontent provider 104 through suitable communication media, generally illustrated ascommunication system 108 for convenience.Communication system 108 may include many different types of communication media including those utilized by various different physical and logical channels of communication, now known or later developed. Non-limiting media and communication channel examples include one or more, or any operable combination of: satellite systems, cable systems, fiber optic systems, microwave systems, asynchronous transfer mode (“ATM”) systems, frame relay systems, digital subscriber line (“DSL”) systems, radio frequency (“RF”) systems, telephone systems, cellular systems, and the Internet. - In at least one embodiment, the received program content is converted by the
program distributor 106 into a suitable signal (a “program signal”) that is ultimately communicated to the receivingdevice 118 as a program content stream. Various embodiments of the receivingdevice 118 may receive programming fromprogram distributors 106 and/or directly fromcontent providers 104 via locally broadcast radio, RF signals, cable, fiber optic, Internet media, or the like via thecommunication system 108. - For example, Video on Demand (VOD) systems may allow a user of the receiving
device 118 to select, watch and/or listen to video and audio content on demand. For example “Internet Television” and “Internet Protocol Television” (IPTV) are systems through which various media content is delivered using the Internet IP suite over a packet-switched network such as the Internet represented bycommunication system 108 to the receivingdevice 118, instead of being delivered through other channels using terrestrial, satellite signal, and cable television formats of thecommunication system 108. In various example embodiments, such technologies are deployed within theenvironment 102 such as in subscriber-based telecommunications networks of thecommunication system 108 with high-speed access channels into the customer premises 116 via the receiving device 118 (e.g., a set-top box or other customer-premises equipment) to bring VOD services to the customer premises 116. - In various example embodiments, television VOD systems stream media content via the
communications system 108 from files stored at a content storage system of a content delivery network operably coupled to thecommunication system 108, under direct or indirect control of theprogram distributor 106, to the receivingdevice 118. Television VOD systems may stream content to areceiving device 118 such as a set-top box, DVD player, game system, smart phone, television (including a smart TV), PC, a sound system receiver, a digital video recorder (“DVR”), a compact disk (“CD”) device, tablet device, mobile device or other computing device or media player, and the like, allowing viewing in real time at the customer premises 116, or downloading to areceiving device 118 such as a computer, DVR (also called a personal video recorder) or portable media player for viewing at any time. - In some embodiments, the receiving
device 118 may be a set-top box that is typically provided by the cable provider, satellite provider, orother program distributor 106 to which the customer may subscribe to receive such television services and that also receives programming through other channels using a terrestrial, satellite signal, and/or cable television format. Themobile device 128 in communication with the receivingdevice 118 may be provided by the program distributor, such as a cable provider or satellite provider. However, in some instances, such mobile devices may be devices other than those provided by theprogram distributor 106. For example, these may include various user devices such as a tablet device, game machine, smartphone, mobile device or other computing device or media player or communications device not provided by or controlled by the cable provider, satellite provider, or other program distributor to which the customer subscribes for the television services. - The
supplemental content server 122 is operably coupled to thecommunication system 108 and may be a system of an entity that provides processing and/or analysis of a program content stream sent to receivingdevice 118 and may also generate supplemental content for the program content stream as the program content stream is being received by the receivingdevice 118 based on the processing of the verbal content of the program content stream. Thesupplemental content server 122 may then communicate to the receivingdevice 118 results of such processing and/or analysis of the program content stream. Thesupplemental content server 122 may also or instead transmit generated supplemental content for the program content stream as the program content stream is being delivered to and/or received by the receivingdevice 118. Theenvironment 102 may include many systems associated with a television service provider, such as theprogram distributor 106, that operate in a manner such as thesupplemental content server 122 described herein, and thesupplemental content server 122 represents one or more of such systems in various embodiments. Such functionality may also or instead be performed by the receivingdevice 118,mobile device 128 and/orremote control device 130. - In addition,
information provider 138 may provide various forms of content and/or services to various devices residing in the customer premises 116. For example,information provider 138 may provide information to the receivingdevice 118 regarding customer or user accounts associated with the program content stream and/or receivingdevice 118; corresponding user preferences and settings; and/or additional processing of the program content stream to enable the receivingdevice 118 and/orsupplemental content server 122 to generate corresponding supplemental content. Such services may include artificial intelligence and machine learning functionality to improve the supplemental content in general and for particular programs based on a feedback loop from the receivingdevice 118,mobile device 128 and/or receiving devices and mobile devices of other users. - The above description of the
environment 102, the customer premises 116, and the various devices therein, is intended as a broad, non-limiting overview of an example environment in which various embodiments of generating supplemental content for a program content stream may be implemented.FIG. 1 illustrates just one example of anenvironment 102 and the various embodiments discussed herein are not limited to such environments. In particular,environment 102 and the various devices therein, may contain other devices, systems and/or media not specifically described herein. Theenvironment 102 may include many different supplemental content servers, content providers, program distributors and information providers that operate in a corresponding manner as thesupplemental content server 122, thecontent provider 104,program distributor 106 andinformation provider 138 as described herein. Each ofsupplemental content server 122, thecontent provider 104,program distributor 106 andinformation provider 138 may represent one or more of such systems in various embodiments. - Example embodiments described herein provide applications, tools, data structures and other support to implement generating supplemental content for a program content stream. Other embodiments of the described techniques may be used for other purposes, including generating supplemental content for formal educational programs in schools and for remote learning, generating supplemental content for sports programs, movies, news programs and other types of content. For example, sports programs often have multiple announcers who typically identify themselves. Therefore, the system may recognize the particular announcer's voice and generate questions or other supplemental content based on individual announcers. In the example regarding sports announcers, the system may present a question based on the verbal content in the sports program of the recognized sports announcer, such as the question “Bob Costas thought this was the best play of the game. A,B,C”. There may be unique scenarios for other types of content. For example, for news content, there may be a current events quiz presented during or at the end of a news program generated in real-time based on the content of the news program. Also, multiple users may communicate individual answers or other responses to questions or other supplemental content presented on a common presentation device and/or individual mobile devices or computer of each user. In such embodiments, games or competitions (e.g., to answer the most questions correctly or the fastest) managed and/or controlled by the receiving
device 118,supplemental content server 122,content provider 104,program distributor 106 and/orinformation provider 138 may be played between the users, such as between students in a class, or between multiple players remotely overcommunication system 108. - In an example embodiment for an education setting, the system may automatically generate quiz print outs for teachers when not all students have mobile devices. These printouts may be generated from previous natural language processing (NLP) of content stored and sent in the metadata associated with the content. For example, an application running on the
mobile device 128 and/or the receivingdevice 118 may connect to a wireless local printer so students can follow along and answer while watching on thepresentation device 120 and/or an individualmobile device 128. In one embodiment, the printouts may include multiple choice questions or configured in a way that they are easily readable and scored via the application running on themobile device 128 by taking a photo of it. The application running on themobile device 128 and/or receivingdevice 118 may record and display student scores and rankings within a very short time, on the screen of thepresentation device 120 or on the screen of themobile device 128 via the application running on themobile device 128. - Also, generating supplemental content for a program content stream may be performed by various other devices, such as audio and DVD players, digital recorders, computers, peripherals, televisions, mobile devices, active speakers, headphones, telephones, and other electronic devices, etc. In the following description, numerous specific details are set forth, such as data formats, program sequences, processes, and the like, in order to provide a thorough understanding of the described techniques. The embodiments described also can be practiced without some of the specific details described herein, or with other specific details, such as changes with respect to the ordering of the code flow, different code flows, and the like. Thus, the scope of the techniques and/or functions described are not limited by the particular order, selection, or decomposition of steps described with reference to any particular module, component, or routine.
-
FIG. 2 is a block diagram illustrating elements of anexample receiving device 118 used in secure communications between media devices, according to one example embodiment. - In one embodiment, the receiving
device 118 is a device such as a set-top box, television, DVR, DVD player, PC, tablet device, game machine, smartphone, mobile device or other computing device or media player configured to receive programming via a connection to a satellite or cable television service provider, or other media content provider, outside the customer premises and to display such programming on a presentation device. For example, the receivingdevice 118 may be configured to receive, process and display on apresentation device 120 streaming media content received directly from the satellite or cable television service provider and/or other programming received directly from the satellite or cable television service provider such as cable or satellite television broadcasts via various other physical and logical channels of communication. - Note that one or more general purpose or special purpose computing systems/devices may be used to operate the receiving
device 118; store information regarding the receivingdevice 118, store metadata; perform DRM and key management operations, decrypt and encrypt received content; pair with various mobile devices on a home LAN and/or over the Internet; establish connection between the receivingdevice 118 and various mobile devices; communicate data, including programming, between the receivingdevice 118 and various mobile devices (including mobile device 128) via the home LAN and/or Internet; and communicate with thecontent provider 104,program distributor 106,information provider 138 and/orsupplemental content server 122. In addition, in some embodiments, the receivingdevice 118 may comprise one or more distinct computing systems/devices and may span distributed locations. Furthermore, each block shown may represent one or more such blocks as appropriate to a specific embodiment or may be combined with other blocks. Also, the receivingdevice operation manager 222 may be implemented in software, hardware, firmware, or in some combination to achieve the capabilities described herein. - In the embodiment shown, receiving
device 118 comprises a computer memory (“memory”) 201, a display 202 (including, but not limited to a light emitting diode (LED) panel, cathode ray tube (CRT) display, liquid crystal display (LCD), touch screen display, etc.), one or more Central Processing Units (“CPU”) 203, Input/Output (“I/O”) devices 204 (e.g., keyboard, mouse, RF or infrared receiver, universal serial bus (USB) ports, High-Definition Multimedia Interface (HDMI) ports, other communication ports, and the like), other computer-readable media 205, and network connections 206. The receivingdevice operation manager 222 is shown residing inmemory 201. In other embodiments, some portion of the contents and some, or all, of the components of the receivingdevice operation manager 222 may be stored on and/or transmitted over the other computer-readable media 205. The components of the receivingdevice 118 andoperation manager 222 preferably execute on one ormore CPUs 203 and facilitate the receiving, decrypting, decoding, processing, selecting, recording, playback and displaying of programming, as well as the establishing of an Internet Layer end-to-end security connection, such as a secure IP tunnel, over the home LAN and/or Internet between the receivingdevice 118 and various mobile devices and communication of data, including programming, between the receivingdevice 118 and various mobile devices via the Internet Layer end-to-end security connection over the home LAN and/or Internet, as described herein. The receivingdevice operation manager 222 may also facilitate on-demand media services (e.g., VOD services), on-demand program ordering, processing and DRM and key management and storage corresponding to processing received streaming media content and other programming. The receivingdevice operation manager 222 may operate as, be part of, or work in conjunction and/or cooperation with various on-demand service software applications stored inmemory 201 and on various mobile devices. The receivingdevice operation manager 222 also facilitates communication with peripheral devices such as a remote control, via the I/O devices 204, and with themobile device 128 and remote systems (e.g., thecontent provider 104, thesupplemental content server 122, theprogram distributor 106, and/or the information provider 138) via the network connections 206. - Recorded or buffered programming of a program content stream received as streaming media content or other types of programming may reside on the
media content storage 216, either in decrypted or encrypted form as applicable for securely storing, processing and displaying of the received media content according to the applicable DRM associated with the particular programming. Themedia content storage 216 may also store various generated supplemental content, program metadata associated with the recorded or buffered programming and/or supplemental content stored in themedia content storage 216, such as that including, but not limited to, generated questions having answers that are extracted educational concepts communicated by the verbal content of the program content stream; answers to such questions; DRM data; tags; codes, identifiers, format indicators, timestamps, user identifications, authorization codes, digital signatures, etc. - The verbal
content processing engine 224 is configured to receive the program content stream via the network connections 206 and/or other I/O devices 204 and process verbal content of the program content stream as the program content stream is being received. For example, the verbalcontent processing engine 224 may extract audio and/or textual verbal content from the program content stream, convert or transcribe the extracted audio to text and then immediately process such data while the content of the program content stream is being output by the receivingdevice 118 for presentation on thepresentation device 120. For example, in one embodiment, the verbalcontent processing engine 224 may extract textual content from the program content stream via accessing the closed captioning and/or subtitle metadata associated and/or received with the program content stream. Such textual data may have been previously inserted in the vertical blanking interval of the television signal comprising the program content stream. In other embodiments, the verbalcontent processing engine 224 may perform speech-to-text conversion of the extracted audio of the program content stream as the program content stream is being received by the receivingdevice 118. - In one embodiment, the verbal
content processing engine 224 may perform natural language processing (NLP) on the extracted textual content or extracted audio that has been converted to text as the program content stream is being output for presentation on a presentation device, such aspresentation device 120. NLP is a branch of artificial intelligence that helps computers understand, interpret and manipulate human language. NLP may use representation learning and deep neural network-style machine learning methods to recognize concepts, topics, facts, and statements made in the verbal content of the program content stream. Such techniques can perform many such natural language tasks, for example in language modeling, parsing, and many others. Some techniques include the use of word embeddings to capture semantic properties of words, and an increase in end-to-end learning of a higher-level task (e.g., question answering) instead of relying on a pipeline of separate intermediate tasks (e.g., part-of-speech tagging and dependency parsing). Deep neural network-based approaches may be used instead of or in addition to distinct statistical natural language processing. With neural machine translation (NMT), deep learning-based approaches to machine translation directly learn sequence-to-sequence transformations, obviating the need for intermediate steps such as word alignment and language modeling that are used in statistical machine translation (SMT). For example, the verbalcontent processing engine 224 may perform NLP on the text associated with the program content stream to extract educational concepts communicated by the text. For example, the NLP of the text may recognize that the text, “Lions typically live in the wild for 10-14 years” is a grammatical statement that may serve as a basis for, or may be paired to, a corresponding educational question, “How long do lions live for?” The verbalcontent processing engine 224 may then store such question and answer associations and/or pairings in themedia content storage 216 ordata repository 220. The verbalcontent processing engine 224 may communicate such question and answer associations and pairings, as such associations are made, to the supplemental content generation andprocessing engine 226 for immediate further processing. - The supplemental content generation and
processing engine 226 then generates supplemental content for the program content stream as the program content stream is being received based on the output of the verbalcontent processing engine 224 that resulted from the processing of the verbal content of the program content stream. In some embodiments, the supplemental content generation andprocessing engine 226 may generate questions having answers that are one or more of the educational concepts communicated by the verbal content and that were extracted by the verbalcontent processing engine 224. For example, the supplemental content generation andprocessing engine 226 may select or generate the question “How long do lions live for?” based on the NLP of the verbal content of the program content stream. For example, the NLP of the verbal content of the program content stream may recognize that the text, “Lions typically live in the wild for 10-14 years” extracted from the verbal content is a grammatical statement that may serve as a basis for the question “How long do lions live for?” The supplemental content generation andprocessing engine 226 may iterate through all such associations made by verbalcontent processing engine 224 during the NLP of the program content stream, in order, as such associations are being made, such as to present the corresponding questions as the program content stream is being presented on thepresentation device 120. This may be performed in real time or near real time as the program content stream is being output for presentation on thepresentation device 120. - The supplemental content generation and
processing engine 226 may cause the supplemental content to be presented as the program content stream is being output for presentation on thepresentation device 120. In some embodiments, the supplemental content generation andprocessing engine 226 may cause the receivingdeice 118 to pause the presentation of the program content stream on thepresentation device 120 and, in response to the pausing the presentation of the program content stream on thepresentation device 120, cause the corresponding question be presented on thepresentation device 120. The program content stream may be buffered in themedia content storage 216 ordata repository 220 while it is paused such that after the program is un-paused or otherwise resumed, the receivingdevice 118 can present the program content stream on thepresentation device 120 and/ormobile device 128 beginning from the point it was paused. The supplemental content generation andprocessing engine 226 may select particularly relevant or convenient points in the program content stream to pause the presentation in order to present the question. For example, the supplemental content generation andprocessing engine 226 may select a point in the program immediately before or after a commercial break, a point in the program related to the question and/or a point in the program designated for insertion of supplemental content. In some embodiments, the question may be presented in response to the user pausing the program or otherwise providing input that may indicate the question may be presented. - In some embodiments, the supplemental content generation and
processing engine 226 may cause a video portion of the program content stream to be presented that is relevant to a particular question while the question is being presented. The supplemental content generation andprocessing engine 226 may cause this to occur while the presentation of the program content stream is paused in order for the question to be presented. For example, if a question regarding lions is presented, a video clip or image from the program related to lions (or perhaps even including information answering or hinting at the answer to the question) may be presented on thepresentation device 120 and/ormobile device 128 while the question is being presented on thepresentation device 120 and/ormobile deice 128. Such associations between generated questions and content related or relevant to the questions may be made as a result of the NLP of the verbal content of the program content stream in conjunction with object detection and recognition processing of the program content stream by the supplemental content generation andprocessing engine 226 in order to recognize relevant objects or video scenes in the program content stream. - In some embodiments, the supplemental content generation and
processing engine 226 may generate various types of supplemental content based on the NLP of verbal content and/or object detection and recognition processing of other objects or scenery in the content of the program content stream performed by the supplemental content generation andprocessing engine 226. Such supplemental content may include, but is not limited to: questions; fun facts; trivia; additional information about topics, subjects, objects, people or places recognized in the content of the program content stream; purchase opportunities related to topics, subjects, topics, subjects, objects, people or places recognized in the content of the program content stream; games related to topics, subjects, objects, people or places recognized in the content of the program content stream; and advertisements related to topics, subjects, objects, people or places recognized in the content of the program content stream. - The supplemental content generation and
processing engine 226 may cause an interactive element to be presented with the question such that the user may select an answer to the question via the interactive element. For example, a multiple choice question may be presented on thepresentation device 120 with the answers being selectable by the user with the remote control 30 and/or themobile device 128. In some embodiments the questions are also or instead presented on themobile device 128 of the user. In some embodiments where the questions are also or instead presented on themobile device 128 of the user, the program content stream is not paused, such that the user may continue to watch the programming on the presentation device 120 (e.g., on their television) while looking to theirmobile device 128 to view and answer the question related to the program content stream. - As the program content stream is being output by the receiving
device 118 for presentation and being presented on thepresentation device 120 and/ormobile device 128, the supplemental content generation andprocessing engine 226 may pause the presentation of the program content stream on thepresentation device 120 and/ormobile device 128 at various different times during the presentation of the program content stream. The supplemental content generation andprocessing engine 226 may cause a different question of the generated questions to be presented on thepresentation device 120 and/or or on themobile device 128, at each of the various different times the presentation of the program content stream is paused on thepresentation device 120 and/ormobile device 128. - In some embodiments, the supplemental content generation and
processing engine 226 may customize the questions and other supplemental content according to the preferences and corresponding settings of a specific user. The supplemental content generation andprocessing engine 226 may receive input indicative of a selection of one or more of: an age level of a user that is to view the supplemental content; a number of questions comprising the supplemental content for a particular program of the program content stream; and a frequency of presentation of items comprising the supplemental content. The supplemental content generation andprocessing engine 226 may then cause the supplemental content to be presented as the program content stream is being output for presentation on thepresentation device 120 and/or themobile device 128 based on the input. For example, the supplemental content generation andprocessing engine 226 may select questions to be presented that are categorized at a particular difficulty level associated with the age level input by the user. Also, the supplemental content generation andprocessing engine 226 may determine to present only 3 questions per television program based on an input of a user indicating a frequency of 3 questions per television program. Additionally, the supplemental content generation andprocessing engine 226 may determine to present a particular number of questions to present for a particular program based on input from a user indicating how many questions are to be presented based on the name, title or content of the specific program. - The supplemental content generation and
processing engine 226 may also store the generated supplemental content as metadata associated with the program for subsequent presentation when the program is played again for the viewer or other viewers, thus further increasing efficiency of generating further supplemental content. For example, the supplemental content generation andprocessing engine 226 may store, inmedia content storage 216 and/ordata repository 220, the generated questions as metadata associated with a corresponding episode of a television program of the program content stream. Such generated questions may be stored for presentation as supplemental content of the corresponding program episode during subsequent playback of the program episode onpresentation device 120,mobile device 128 and/or other devices of other users or customers. - After the supplemental content generation and
processing engine 226 causes the supplemental content to be presented as the program content stream is being output for presentation on thepresentation device 120 and/or themobile device 128, theanswer processing engine 228 may receive an input indicative of an answer to the question. In some embodiments, the input may originate as voice input to theremote control 130, receivingdevice 118 and/or themobile device 128. In such embodiments, theremote control 130, receivingdevice 118 and/or themobile device 128 may incorporate and/or use use corresponding voice-command, voice-enabled, speech recognition and/or voice recognition functionality to recognize and process voice commands and/or other speech accordingly. For example, theremote control 130 may be a voice-enabled remote control device including a microphone that recognizes speech or relays relevant audio to the receivingdevice 118 in order for theanswer processing engine 228 of the receivingdevice 118 to recognize the speech as an answer to the question presented. - In some embodiments, the supplemental content generation and
processing engine 226 may cause an interactive element to be presented with the question such that the user may select an answer to the question via the interactive element. For example, a multiple choice question may be presented on thepresentation device 120 and/or themobile device 128 with the answers being selectable by the user with the remote control 30, receivingdevice 118 and/or themobile device 128. Input indicative of selection of an answer, such as a click, press or voice selection, may then be communicated to theanswer processing engine 228. Theanswer processing engine 226 may then determine a response to the answer and communicate the response to the answer. For example, theanswer processing engine 228 may communicate the response visually on thepresentation device 120 and/ormobile device 128. In some embodiments, the presentation of the program content stream is paused while the question is being presented and theanswer processing engine 228 causes the presentation of the program content stream to resume in response to, or otherwise based on, an answer being received to the question. In other embodiments, the resumption of the presentation of the program content stream may be in response to an answer being received to the question or theanswer processing engine 228 determining that a timer has expired, whichever occurs first. - In some embodiments, the resuming of the presentation of the program content stream may be in response to the
answer processing engine 228 determining that the answer to the selected question is correct. In some embodiments, theanswer processing engine 228 may compare the input indicative of the answer to the question to an educational concept associated with the question presented on thepresentation device 120 and/ormobile device 128. This may be performed while the presentation of the program content stream is paused. Theanswer processing engine 228 may then determine whether the answer to the selected question is correct based on the comparison while the presentation of the program content stream is paused. Theanswer processing engine 228 may also determine the response to the answer based on the determination of whether the answer to the selected question is correct while the presentation of the program content stream is paused. In some embodiments, theanswer processing engine 228 causes the presentation of the program content stream to resume in response to communicating a response to the answer. Theanswer processing engine 228 may resume the presentation of the program content stream on thepresentation device 120 and/ormobile device 128 after each of the various different times the presentation of the program content stream is paused in response to receiving a corresponding correct answer to a question presented on thepresentation device 120 and/or on themobile device 128 at each of the various different times the presentation of the program content stream is paused. - In some embodiments, in response to receiving an incorrect answer, the
answer processing engine 228 may display or otherwise communicate to the user the answer was incorrect, may display or otherwise communicate to the user the correct answer and/or may replay the portion of the content that states, illustrates or otherwise includes the correct answer. For example, this may be accomplished by theanswer processing engine 228 finding the portion of content including the correct answer using a timestamp corresponding to the location in the content where the correct answer is included. When generating the supplemental content including the question and correct answer, the supplemental content generation andprocessing engine 226 may record such timestamps that indicate where in the content the corresponding verbal content associated with the correct answer appears and associate the timestamp with the question and corresponding correct answer. Theanswer processing engine 228 may then access such timestamp to locate the content that includes the correct answer and then play such content in response to receiving an incorrect answer and/or in response to a request from the user to view the content including the correct answer. - In some embodiments, the
answer processing engine 228 controls an amount of viewing time allowed on thepresentation device 120 and/or one or more mobile devices (such as mobile device 128), based on a number of correct answers to the different questions presented on thepresentation device 120 and/or on a mobile device of the user received at each of the various different times the presentation of the program content stream is paused on the presentation device. In some embodiments, theanswer processing engine 228 may be in communication with and/or control themobile device 128 and/or other mobile devices associated with the receivingdevice 118. Theanswer processing engine 228 may also have control over or have access to a user account that is associated with the receivingdevice 118 and/or themobile device 128. In other embodiments, theanswer processing engine 228 may also or instead be running on themobile device 128. For example, a parent may select parental controls available on the receivingdevice 118 and/ormobile device 128 that limit the viewing time or “screen time” of one or more devices or an account associated with a child to 5 hours per week until 50 supplemental content questions have been correctly answered within that week, which may then be increased to 7 hours for that week. The number of viewing hours, number of correctly answered questions and rate of increase or decrease of allowed viewing time may be selectable by the user via a user interface or portal generated or provided by theanswer processing engine 228 of the receivingdevice 118, theprogram distributor 106, thecontent provider 104,information provider 138 and/orsupplemental content server 122. - Other code or programs 230 (e.g., further audio/video processing modules, a program guide manager module, a Web server, and the like), and potentially other data repositories, such as
data repository 220 for storing other data (user profiles, preferences and configuration data, etc.), may also reside in thememory 201, and preferably execute on one ormore CPUs 203. Of note, one or more of the components inFIG. 2 may or may not be present in any specific implementation. For example, some embodiments may not provide other computerreadable media 205 or a display 202. - In some embodiments, the receiving
device 118 andoperation manager 222 include an application program interface (“API”) that provides programmatic access to one or more functions of the receivingdevice 118 andoperation manager 222. For example, such an API may provide a programmatic interface to one or more functions of the receivingdevice operation manager 222 that may be invoked by one of theother programs 230, aremote control 130, themobile device 128,program distributor 106,content provider 104,information provider 138,supplemental content server 122 or some other module. In this manner, the API may facilitate the development of third-party software, such as various different on-demand service applications, user interfaces, plug-ins, adapters (e.g., for integrating functions of the receivingdevice operation manager 222 andinformation provider 138 into mobile device and/or desktop applications), and the like to facilitate generating supplemental content for a program content stream using thereceiving device 118. - In an example embodiment, components/modules of the receiving
device 118 andoperation manager 222 are implemented using standard programming techniques. For example, the receivingdevice operation manager 222 may be implemented as a “native” executable running on theCPU 203, along with one or more static or dynamic libraries. In other embodiments, the receivingdevice 118 andoperation manager 222 may be implemented as instructions processed by a virtual machine that executes as one of theother programs 230. In general, a range of programming languages known in the art may be employed for implementing such example embodiments, including representative implementations of various programming language paradigms, including but not limited to, object-oriented (e.g., Java, C++, C#, Visual Basic.NET, Smalltalk, and the like), functional (e.g., ML, Lisp, Scheme, and the like), procedural (e.g., C, Pascal, Ada, Modula, and the like), scripting (e.g., Perl, Ruby, Python, JavaScript, VBScript, and the like), or declarative (e.g., SQL, Prolog, and the like). - In a software or firmware implementation, instructions stored in a memory configure, when executed, one or more processors of the receiving
device 118 to perform the functions of the receivingdevice operation manager 222. In one embodiment, instructions cause theCPU 203 or some other processor, such as an I/O controller/processor, to perform NLP of content of a program content stream as the as the program content stream is being output for presentation on a presentation device; extract educational concepts communicated by verbal content of the program content stream based on the performing of the natural language processing of the verbal content; and cause questions based on the extracted educational concepts to be presented on the presentation device or a mobile device of a user at different times while the presentation of the program content stream is paused. - The embodiments described above may also use well-known or other synchronous or asynchronous client-server computing techniques. However, the various components may be implemented using more monolithic programming techniques as well, for example, as an executable running on a single CPU computer system, or alternatively decomposed using a variety of structuring techniques known in the art, including but not limited to, multiprogramming, multithreading, client-server, or peer-to-peer, running on one or more computer systems each having one or more CPUs or other processors. Some embodiments may execute concurrently and asynchronously, and communicate using message passing techniques. Equivalent synchronous embodiments are also supported by a receiving
device operation manager 222 implementation. Also, other functions could be implemented and/or performed by each component/module, and in different orders, and by different components/modules, yet still achieve the functions of the receivingdevice 118 andoperation manager 222. - In addition, programming interfaces to the data stored as part of the receiving
device 118 andoperation manager 222, can be available by standard mechanisms such as through C, C++, C#, and Java APIs; libraries for accessing files, databases, or other data repositories; scripting languages such as XML; or Web servers, FTP servers, NFS file servers, or other types of servers providing access to stored data. Themedia content storage 216 anddata repository 220 may be implemented as one or more database systems, file systems, or any other technique for storing such information, or any combination of the above, including implementations using distributed computing techniques. - Different configurations and locations of programs and data are contemplated for use with techniques described herein. A variety of distributed computing techniques are appropriate for implementing the components of the illustrated embodiments in a distributed manner including but not limited to TCP/IP sockets, RPC, RMI, HTTP, and Web Services (XML-RPC, JAX-RPC, SOAP, and the like). Other variations are possible. Other functionality could also be provided by each component/module, or existing functionality could be distributed amongst the components/modules in different ways, yet still achieve the functions of the receiving
device operation manager 222. - Furthermore, in some embodiments, some or all of the components of the receiving
device 118 andoperation manager 222 may be implemented or provided in other manners, such as at least partially in firmware and/or hardware, including, but not limited to one or more application-specific integrated circuits (“ASICs”), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and/or embedded controllers), field-programmable gate arrays (“FPGAs”), complex programmable logic devices (“CPLDs”), and the like. Some or all of the system components and/or data structures may also be stored as contents (e.g., as executable or other machine-readable software instructions or structured data) on a computer-readable medium (e.g., as a hard disk; a memory; a computer network, cellular wireless network or other data transmission medium; or a portable media article to be read by an appropriate drive or via an appropriate connection, such as a DVD or flash memory device) so as to enable or configure the computer-readable medium and/or one or more associated computing systems or devices to execute or otherwise use, or provide the contents to perform, at least some of the described techniques. -
FIGS. 3A and 3B are diagrams of anexample presentation device 120 and an example handheldmobile device 128, respectively, illustrating example user interface screens shown in a presentation of an example question as supplemental content for a program content stream, according to an example embodiment. - In the example shown, the receiving device 118 (shown in
FIG. 1 andFIG. 2 connected to thepresentation device 120 and handheld mobile device 128) is outputting a program content stream to thepresentation device 120 including aneducational program 306 about the life of lions in the wild in Africa. While outputting a program content stream to thepresentation device 120, the receivingdevice 118 has extracted the audio of theprogram 306 and performed NLP on the verbal content of theprogram 306. The receivingdevice 118 has generated a question “How long do lions live for?” 304 based on the NLP of the audio extracted from theprogram 306. The receivingdevice 118 has presented the question “How long do lions live for?” 304 along with multiple choice answers 308 to thequestion 304 on thepresentation device screen 302 as an overlay on top of theprogram 306. - In the example embodiment shown, the receiving
device 118 has paused the presentation of theprogram 306 while presenting thequestion 304 so that the viewer may have time to answer thequestion 304 without missing any of theprogram 306. As shown inFIG. 3B , acorresponding question 314 and associated corresponding multiple choice answers 318 are also being presented on amobile device screen 312 of themobile device 128 of the user, who may also be viewing theprogram 306 on thepresentation device 120. In some embodiments, the question may instead be presented only on themobile device 128 or only on thepresentation device 120. Also, theprogram 306, a scene from theprogram 306, an image from theprogram 306 or other related content may be presented along with thecorresponding question 314 on themobile device 128 and/or along with thequestion 304 on thepresentation device 120. In embodiments where theprogram 306 is also or instead presented on themobile device 128, themobile device 128 may pause the presentation of theprogram 306 on themobile device 128 while presenting thecorresponding question 314. In some embodiments, theprogram 306 may continue to play on thepresentation device 120 while thecorresponding question 314 is presented only on themobile device 128, such that the user may continue to view theprogram 306 on thepresentation device 120 while looking to theirmobile device 128 to view and answer thecorresponding question 314 related to theprogram 306. - The user may respond to the
question 304 presented on thepresentation device 120 and/or thecorresponding question 314 presented on themobile device 128 in a variety of manners. In some embodiments, theremote control 130, the receivingdevice 118, thepresentation device 120 and/or themobile device 128 are equipped with a microphone and voice-enabled such that the user may respond by speaking the answer to thequestion 304 and/orcorresponding question 314. For example, the presentation of thequestion 304 on thepresentation device 120 and/or thecorresponding question 314 on themobile device 128 may trigger theremote control 130, the receivingdevice 118, thepresentation device 120 and/or themobile device 128 to start listening for an answer. - In some embodiments, the multiple choice answers 308 presented on the
presentation device 120 and/or the corresponding multiple choice answers 318 presented onmobile device 128 are selectable interactive user interface elements, such that the user may select the answer by using the remote control 30, touching the selection if thepresentation device screen 302 ormobile device screen 312 is a touch screen, clicking on the selection using an input device such as a mouse, and/or selecting the desired answer using another input device. In some embodiments, if the answer is selected on themobile device 128, the selection may be communicated to the receivingdevice 118 and/orpresentation device 120 viacommunication system 108 shown inFIG. 1 . -
FIGS. 4A and 4B are diagrams of anexample presentation device 120 and an example handheldmobile device 128, respectively, illustrating example user interface screens shown in an example presentation of a response to an answer received to the example question presented inFIGS. 3A and 3B , according to an example embodiment. - As shown in
FIGS. 4A and 4B , after the user answers thequestion 304 presented on thepresentation device 120 and/or thecorresponding question 314 presented on themobile device 128, aresponse 404 is presented on thepresentation device 120. In particular, in the example shown, the user has responded by speaking the choice “A”, which the receiving device 118 (shown inFIG. 1 andFIG. 2 connected to thepresentation device 120 and handheld mobile device 128) recognized as the correct answer. In response to receiving the correct answer, the receivingdevice 118 has presented the response “That is correct!” 404 on thepresentation device screen 302. Also, a confirmation of the answer received 402 may be presented on the on thepresentation device screen 302 to verify and reinforce learning. In response to receiving the correct answer, the receivingdevice 118 then resumes presentation of theprogram 306 on thepresentation device screen 302. An indication that the program will resume 406 may also be presented on thepresentation device screen 302. - In response to receiving the correct answer, the receiving
device 118 and/or themobile device 128 may cause a corresponding response “That is correct!” 414, a corresponding confirmation of the answer received 412 and a corresponding indication that the program will resume 416 to also or instead be presented on themobile device screen 312. In some embodiments, the program may be resumed on themobile device screen 312 and/or thepresentation device screen 302 based on a preference of the user as indicated by an input or selection of a corresponding setting by the user on themobile device 128 and/or the receivingdevice 118. -
FIGS. 5A and 5B are diagrams of anexample presentation device 120 and an example handheldmobile device 128, respectively, illustrating respective example user interface screens shown in an example presentation of a response to an incorrect answer received to the example question presented inFIGS. 3A and 3B , according to an example embodiment. - As shown in
FIGS. 5A and 5B , after the user answers thequestion 304 presented on thepresentation device 120 and/or thecorresponding question 314 presented on themobile device 128, aresponse 504 is presented on thepresentation device 120. In particular, in the example shown, the user has responded by speaking the choice “B”, which the receiving device 118 (shown inFIG. 1 andFIG. 2 connected to thepresentation device 120 and handheld mobile device 128) recognized as an incorrect answer. In response to receiving the incorrect answer, the receivingdevice 118 has presented the response “Sorry, that is incorrect.” 504 on thepresentation device screen 302. Also, a confirmation of the answer received 502 may be presented on the on thepresentation device screen 302 for verification. In response to receiving the incorrect answer, the receivingdevice 118 may then replay the portion of theprogram 306 that states, illustrates or otherwise includes the correct answer on thepresentation device screen 302. An indication that the portion of the program including the correct answer will play 506 may also be presented on thepresentation device screen 302. - In response to receiving the incorrect answer, the receiving
device 118 and/or themobile device 128 may cause a corresponding response “Sorry, that is incorrect” 514, a corresponding confirmation of the answer received 512 and a corresponding indication that the portion of the program including the correct answer will play 516 to also or instead be presented on themobile device screen 312. In some embodiments, the portion of the program including the correct answer may play on themobile device screen 312 and/or thepresentation device screen 302 based on a preference of the user as indicated by an input or selection of a corresponding setting by the user on themobile device 128 and/or the receivingdevice 118. -
FIG. 6 is a flow diagram of anexample method 600 for generating supplemental content for a program content stream, according to an example embodiment. - At 602, a computerized system for generating supplemental content for a program content stream receives the program content stream.
- At 604, the system processes verbal content of the program content stream as the program content stream is being received.
- At 606, the system generates supplemental content for the program content stream as the program content stream is being received based on the processing of the verbal content of the program content stream.
- At 608, the system causes the supplemental content to be presented as the program content stream is being output for presentation on a presentation device.
-
FIG. 7 is a flow diagram of anexample method 700 for storing generated questions as metadata associated with a program of the program content stream, according to an example embodiment. - At 702, a system for generating supplemental content for a program content stream generates questions, while a program content stream is being output for presentation on a presentation device. The questions have answers that are one or more extracted educational concepts communicated by verbal content of the program content stream.
- At 704, the system causes the questions to be presented on the presentation device or a mobile device of a user at different times while the presentation of the program content stream is paused.
- At 706, the system stores the generated questions, as metadata associated with a program of the program content stream, for presentation as supplemental content of the program during subsequent playback of the program.
-
FIG. 8 is a flow diagram of anexample method 800 for extracting educational concepts communicated by verbal content of the program content stream, according to an example embodiment. - At 802, a system for generating supplemental content for a program content stream performs natural language processing of content of a program content stream as the as the program content stream is being output for presentation on a presentation device.
- At 804, the system extracts educational concepts communicated by verbal content of the program content stream based on the performing of the natural language processing of the verbal content.
- At 806, the system causes questions based on the extracted educational concepts to be presented on the presentation device or a mobile device of a user at different times while the presentation of the program content stream is paused.
- The various embodiments described above can be combined to provide further embodiments. All of the U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications and non-patent publications referred to in this specification and/or listed in the Application Data Sheet are incorporated herein by reference, in their entirety. Aspects of the embodiments can be modified, if necessary to employ concepts of the various patents, applications and publications to provide yet further embodiments.
- These and other changes can be made to the embodiments in light of the above-detailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which such claims are entitled. Accordingly, the claims are not limited by the disclosure.
Claims (23)
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/566,630 US11800202B2 (en) | 2019-09-10 | 2019-09-10 | Systems and methods for generating supplemental content for a program content stream |
PCT/US2020/049985 WO2021050579A1 (en) | 2019-09-10 | 2020-09-09 | Systems and methods for generating supplemental content for a program content stream |
EP20775557.0A EP4029284A1 (en) | 2019-09-10 | 2020-09-09 | Systems and methods for generating supplemental content for a program content stream |
US18/368,486 US20240007722A1 (en) | 2019-09-10 | 2023-09-14 | Systems and methods for generating supplemental content for a program content stream |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/566,630 US11800202B2 (en) | 2019-09-10 | 2019-09-10 | Systems and methods for generating supplemental content for a program content stream |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/368,486 Continuation US20240007722A1 (en) | 2019-09-10 | 2023-09-14 | Systems and methods for generating supplemental content for a program content stream |
Publications (2)
Publication Number | Publication Date |
---|---|
US20210076106A1 true US20210076106A1 (en) | 2021-03-11 |
US11800202B2 US11800202B2 (en) | 2023-10-24 |
Family
ID=72562034
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/566,630 Active US11800202B2 (en) | 2019-09-10 | 2019-09-10 | Systems and methods for generating supplemental content for a program content stream |
US18/368,486 Pending US20240007722A1 (en) | 2019-09-10 | 2023-09-14 | Systems and methods for generating supplemental content for a program content stream |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/368,486 Pending US20240007722A1 (en) | 2019-09-10 | 2023-09-14 | Systems and methods for generating supplemental content for a program content stream |
Country Status (3)
Country | Link |
---|---|
US (2) | US11800202B2 (en) |
EP (1) | EP4029284A1 (en) |
WO (1) | WO2021050579A1 (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11250719B2 (en) * | 2019-11-05 | 2022-02-15 | International Business Machines Corporation | Generating and rating user assessments |
US11455549B2 (en) | 2016-12-08 | 2022-09-27 | Disney Enterprises, Inc. | Modeling characters that interact with users as part of a character-as-a-service implementation |
US11546669B2 (en) * | 2021-03-10 | 2023-01-03 | Sony Interactive Entertainment LLC | Systems and methods for stream viewing with experts |
US11553255B2 (en) | 2021-03-10 | 2023-01-10 | Sony Interactive Entertainment LLC | Systems and methods for real time fact checking during stream viewing |
US12003821B2 (en) * | 2020-04-20 | 2024-06-04 | Disney Enterprises, Inc. | Techniques for enhanced media experience |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060107195A1 (en) * | 2002-10-02 | 2006-05-18 | Arun Ramaswamy | Methods and apparatus to present survey information |
US20070250901A1 (en) * | 2006-03-30 | 2007-10-25 | Mcintire John P | Method and apparatus for annotating media streams |
US20070265910A1 (en) * | 2006-05-10 | 2007-11-15 | Kivin Varghese | Attention marketplace |
US20150100414A1 (en) * | 2013-10-08 | 2015-04-09 | Adobe Systems Incorporated | Method and apparatus for determining brand awareness before dismissing a video advertisement |
US20160173958A1 (en) * | 2014-11-18 | 2016-06-16 | Samsung Electronics Co., Ltd. | Broadcasting receiving apparatus and control method thereof |
US20170072302A1 (en) * | 2015-09-16 | 2017-03-16 | Customplay Llc | Movie Master Game Method |
US20170213469A1 (en) * | 2016-01-25 | 2017-07-27 | Wespeke, Inc. | Digital media content extraction and natural language processing system |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2013519162A (en) | 2010-02-01 | 2013-05-23 | ジャンプタップ,インコーポレイテッド | Integrated advertising system |
US8745683B1 (en) * | 2011-01-03 | 2014-06-03 | Intellectual Ventures Fund 79 Llc | Methods, devices, and mediums associated with supplementary audio information |
KR20140092598A (en) | 2013-01-16 | 2014-07-24 | 오충록 | Education device using RFID |
US20140308646A1 (en) * | 2013-03-13 | 2014-10-16 | Mindmarker BV | Method and System for Creating Interactive Training and Reinforcement Programs |
US10046242B1 (en) | 2014-08-29 | 2018-08-14 | Syrian American Intellectual Property (Saip), Llc | Image processing for improving memorization speed and quality |
US9471203B1 (en) * | 2014-09-02 | 2016-10-18 | Audible, Inc. | Presenting animated visual supplemental content |
US9922095B2 (en) * | 2015-06-02 | 2018-03-20 | Microsoft Technology Licensing, Llc | Automated closed captioning using temporal data |
KR101858499B1 (en) | 2016-12-05 | 2018-05-16 | (주)뤼이드 | Method for displaying study content and application program thereof |
US10187677B1 (en) | 2017-09-20 | 2019-01-22 | Rovi Guides, Inc. | Systems and methods for generating aggregated media assets on related content from different sources |
US10595098B2 (en) | 2018-01-09 | 2020-03-17 | Nbcuniversal Media, Llc | Derivative media content systems and methods |
US20190272770A1 (en) | 2018-03-02 | 2019-09-05 | Pearson Education, Inc. | Systems and methods for automated content evaluation and delivery |
US11176332B2 (en) * | 2019-08-08 | 2021-11-16 | International Business Machines Corporation | Linking contextual information to text in time dependent media |
-
2019
- 2019-09-10 US US16/566,630 patent/US11800202B2/en active Active
-
2020
- 2020-09-09 EP EP20775557.0A patent/EP4029284A1/en active Pending
- 2020-09-09 WO PCT/US2020/049985 patent/WO2021050579A1/en unknown
-
2023
- 2023-09-14 US US18/368,486 patent/US20240007722A1/en active Pending
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060107195A1 (en) * | 2002-10-02 | 2006-05-18 | Arun Ramaswamy | Methods and apparatus to present survey information |
US20070250901A1 (en) * | 2006-03-30 | 2007-10-25 | Mcintire John P | Method and apparatus for annotating media streams |
US20070265910A1 (en) * | 2006-05-10 | 2007-11-15 | Kivin Varghese | Attention marketplace |
US20150100414A1 (en) * | 2013-10-08 | 2015-04-09 | Adobe Systems Incorporated | Method and apparatus for determining brand awareness before dismissing a video advertisement |
US20160173958A1 (en) * | 2014-11-18 | 2016-06-16 | Samsung Electronics Co., Ltd. | Broadcasting receiving apparatus and control method thereof |
US20170072302A1 (en) * | 2015-09-16 | 2017-03-16 | Customplay Llc | Movie Master Game Method |
US20170213469A1 (en) * | 2016-01-25 | 2017-07-27 | Wespeke, Inc. | Digital media content extraction and natural language processing system |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11455549B2 (en) | 2016-12-08 | 2022-09-27 | Disney Enterprises, Inc. | Modeling characters that interact with users as part of a character-as-a-service implementation |
US11250719B2 (en) * | 2019-11-05 | 2022-02-15 | International Business Machines Corporation | Generating and rating user assessments |
US11557219B2 (en) | 2019-11-05 | 2023-01-17 | International Business Machines Corporation | Generating and rating user assessments |
US12003821B2 (en) * | 2020-04-20 | 2024-06-04 | Disney Enterprises, Inc. | Techniques for enhanced media experience |
US11546669B2 (en) * | 2021-03-10 | 2023-01-03 | Sony Interactive Entertainment LLC | Systems and methods for stream viewing with experts |
US11553255B2 (en) | 2021-03-10 | 2023-01-10 | Sony Interactive Entertainment LLC | Systems and methods for real time fact checking during stream viewing |
US11831961B2 (en) | 2021-03-10 | 2023-11-28 | Sony Interactive Entertainment LLC | Systems and methods for real time fact checking during streaming viewing |
Also Published As
Publication number | Publication date |
---|---|
WO2021050579A1 (en) | 2021-03-18 |
US11800202B2 (en) | 2023-10-24 |
US20240007722A1 (en) | 2024-01-04 |
EP4029284A1 (en) | 2022-07-20 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11800202B2 (en) | Systems and methods for generating supplemental content for a program content stream | |
US20240154835A1 (en) | Providing Synchronous Content and Supplemental Experiences | |
US11716514B2 (en) | Methods and systems for recommending content in context of a conversation | |
US20200007946A1 (en) | Selectively delivering a translation for a media asset based on user proficiency level in the foreign language and proficiency level required to comprehend the media asset | |
US20180211556A1 (en) | Systems and methods for adjusting display lengths of subtitles based on a user's reading speed | |
CN110663079A (en) | Method and system for correcting input generated using automatic speech recognition based on speech | |
CN112423081B (en) | Video data processing method, device and equipment and readable storage medium | |
US10182271B1 (en) | Systems and methods for playback of summary media content | |
US10178422B1 (en) | Systems and methods for generating aggregated media assets based on related keywords | |
US9544656B1 (en) | Systems and methods for recognition of sign language for improved viewing experiences | |
US20210084381A1 (en) | Systems and methods for controlling closed captioning | |
US20230353836A1 (en) | Augmentation of audio/video content with enhanced interactive content | |
US9866913B1 (en) | Binary TV | |
US20230079233A1 (en) | Systems and methods for modifying date-related references of a media asset to reflect absolute dates | |
US20170316708A1 (en) | Systems and methods for providing word definitions based on user exposure | |
WO2017181595A1 (en) | Method and device for video display | |
US20180176631A1 (en) | Methods and systems for providing an interactive second screen experience | |
US11343563B2 (en) | Methods and systems for verifying media guidance data | |
EP3628123B1 (en) | Systems and methods for generating aggregated media assets on related content from different sources | |
US20240022791A1 (en) | Systems and methods to adapt a schedule to be played by a media player |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: DISH NETWORK L.L.C., COLORADO Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MARTEN, NEIL;REEL/FRAME:050432/0675 Effective date: 20190910 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
AS | Assignment |
Owner name: U.S. BANK, NATIONAL ASSOCIATION, AS COLLATERAL AGENT, MINNESOTA Free format text: SECURITY INTEREST;ASSIGNORS:DISH BROADCASTING CORPORATION;DISH NETWORK L.L.C.;DISH TECHNOLOGIES L.L.C.;REEL/FRAME:058295/0293 Effective date: 20211126 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |