EP3571837B1 - Content-aware energy savings for video streaming and playback on mobile devices - Google Patents

Content-aware energy savings for video streaming and playback on mobile devices Download PDF

Info

Publication number
EP3571837B1
EP3571837B1 EP18748711.1A EP18748711A EP3571837B1 EP 3571837 B1 EP3571837 B1 EP 3571837B1 EP 18748711 A EP18748711 A EP 18748711A EP 3571837 B1 EP3571837 B1 EP 3571837B1
Authority
EP
European Patent Office
Prior art keywords
content
video content
mobile device
video
unwanted
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP18748711.1A
Other languages
German (de)
French (fr)
Other versions
EP3571837A4 (en
EP3571837A1 (en
Inventor
Zongfang LIN
Jun Wang
Karthik Rao
Chen TIAN
Handong YE
Ziang HU
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of EP3571837A1 publication Critical patent/EP3571837A1/en
Publication of EP3571837A4 publication Critical patent/EP3571837A4/en
Application granted granted Critical
Publication of EP3571837B1 publication Critical patent/EP3571837B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04WWIRELESS COMMUNICATION NETWORKS
    • H04W52/00Power management, e.g. TPC [Transmission Power Control], power saving or power classes
    • H04W52/02Power saving arrangements
    • H04W52/0209Power saving arrangements in terminal devices
    • H04W52/0225Power saving arrangements in terminal devices using monitoring of external events, e.g. the presence of a signal
    • H04W52/0248Power saving arrangements in terminal devices using monitoring of external events, e.g. the presence of a signal dependent on the time of the day, e.g. according to expected transmission activity
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/26Power supply means, e.g. regulation thereof
    • G06F1/32Means for saving power
    • G06F1/3203Power management, i.e. event-based initiation of a power-saving mode
    • G06F1/3206Monitoring of events, devices or parameters that trigger a change in power modality
    • G06F1/3228Monitoring task completion, e.g. by use of idle timers, stop commands or wait commands
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00Handling natural language data
    • G06F40/10Text processing
    • G06F40/12Use of codes for handling textual entities
    • G06F40/14Tree-structured documents
    • G06F40/143Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/30Arrangements for executing machine instructions, e.g. instruction decode
    • G06F9/30098Register arrangements
    • G06F9/3012Organisation of register space, e.g. banked or distributed register file
    • G06F9/30123Organisation of register space, e.g. banked or distributed register file according to context, e.g. thread buffers
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/02Marketing; Price estimation or determination; Fundraising
    • G06Q30/0241Advertisements
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09GARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/36Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
    • G09G5/39Control of the bit-mapped memory
    • G09G5/391Resolution modifying circuits, e.g. variable screen formats
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60Network streaming of media packets
    • H04L65/61Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41Structure of client; Structure of client peripherals
    • H04N21/414Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance
    • H04N21/41407Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/443OS processes, e.g. booting an STB, implementing a Java virtual machine in an STB or power management in an STB
    • H04N21/4436Power management, e.g. shutting down unused components of the receiver
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04WWIRELESS COMMUNICATION NETWORKS
    • H04W52/00Power management, e.g. TPC [Transmission Power Control], power saving or power classes
    • H04W52/02Power saving arrangements
    • H04W52/0209Power saving arrangements in terminal devices
    • H04W52/0251Power saving arrangements in terminal devices using monitoring of local events, e.g. events related to user activity
    • H04W52/0258Power saving arrangements in terminal devices using monitoring of local events, e.g. events related to user activity controlling an operation mode according to history or models of usage information, e.g. activity schedule or time of day
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04WWIRELESS COMMUNICATION NETWORKS
    • H04W52/00Power management, e.g. TPC [Transmission Power Control], power saving or power classes
    • H04W52/02Power saving arrangements
    • H04W52/0209Power saving arrangements in terminal devices
    • H04W52/0261Power saving arrangements in terminal devices managing power supply demand, e.g. depending on battery level
    • H04W52/0267Power saving arrangements in terminal devices managing power supply demand, e.g. depending on battery level by controlling user interface components
    • H04W52/027Power saving arrangements in terminal devices managing power supply demand, e.g. depending on battery level by controlling user interface components by controlling a display operation or backlight unit
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D30/00Reducing energy consumption in communication networks
    • Y02D30/70Reducing energy consumption in communication networks in wireless communication networks

Definitions

  • the present invention relates to energy efficiency, and more particularly to power saving methods associated with streaming or playback of video with embedded advertisements.
  • the battery capacity of mobile devices is extremely important to the consumer's satisfaction with the device.
  • a single/full charge of a battery may need to last 6-8 hours in order for the consumer to use the device while on the go.
  • Each manufacturer of these devices has poured significant resources into implementing techniques to reduce the power consumption of the device in order to extend the life of the battery during operation. Separate power management chips may be used to power gate certain components of the device.
  • software solutions have been implemented to reduce the power consumption of the device, such as by reducing the number of active threads in a low-power mode or reducing the brightness of a display.
  • WO 2009/136236 A1 refers to an electronic device that includes video storage memory, a video signal receiver, and a video storage controller.
  • the video signal receiver is configured to receive a video stream containing advertising content and program/movie content from a broadcast source.
  • the video storage controller is configured to identify locations of advertising content interspersed among program/movie content in the received video stream, and to record addressable chapter marks for the identified locations of the advertising content and a video stream that is based on the received video stream into the memory.
  • the addressable chapter marks may be displayed on a display and used by a viewer to jump playback of the video stream between the chapter marks.
  • a significant portion of the energy consumed by a mobile device is devoted to powering a display device.
  • a backlight of the display may draw a significant amount of power to turn on at the brightest setting.
  • one or more graphics processing cores in a graphics processing unit (GPU) or system-on-chip (SoC) may be dedicated to generating pixel data for presentation on the display device. This activity may rapidly drain the battery, rendering the mobile device inoperable after only a short period.
  • a system and method are provided for reducing the power consumption of a mobile device. This problem is solved by the subject matter of the independent claims. Further implementation forms are provided in the dependent claims.
  • the method includes the steps of detecting video content to be viewed in an application executed by the mobile device; detecting unwanted content associated with the video content; and operating the mobile device in a low-power mode during playback of the video content in the application in response to detecting the unwanted content in the fashion and details as disclosed in the appended set of claims.
  • one or more of the foregoing features of the aforementioned apparatus, system, and/or method may afford a more efficient way to configure exploration operations of large data sets that, in turn, may enable data analysts to work more efficiently and reduce errors in the results obtained by the exploration operations. It should be noted that the aforementioned potential advantages are set forth for illustrative purposes only and should not be construed as limiting in any manner.
  • Solutions for reducing power consumption may focus at least some of their attention on reducing the power consumption of a display device.
  • the main energy consumers associated with a display device are the backlight, an active matrix of liquid crystals, and the processors tasked with generating pixel data for display on the device.
  • some LED backlight televisions have been designed with an array of LEDs positioned behind the array of liquid crystal elements so that the brightness of the backlight can be dimmed in areas of the screen associated with darker colors. Local area dimming can significantly reduce the power consumed by the display device.
  • DFVS dynamic frequency and voltage scaling
  • these techniques typically have an adverse effect on the operation of the device.
  • the frequency of the clock of a processor may need to be reduced in order to reduce the operating voltage of the processor.
  • a user may notice significant delays when performing certain operations under these conditions.
  • reducing the brightness of the display device or the resolution of the images presented on the display device can have an adverse effect on image quality when compared to normal operation.
  • a consumer may not accept this reduction in performance in order to extend the battery life of the device during normal use. However, the consumer may be able to put up with the reduction in performance at certain times, such as when the battery is nearly drained.
  • Advertising revenue is the main source of income for many content providers that distribute content online.
  • Web pages viewed in a web browser may include banner advertisements or sponsored links designed to redirect users to an advertiser's website.
  • Some websites automatically play embedded video content when a web page is loaded into a browser window.
  • Video content may include files having a video format including, but not limited to, MPEG-4 AVC (Moving Picture Experts Group 4, Part 10/H.264), AVI (Audio Video Interleave), and MOV (Apple Quicktime Movie) as well as other suitable video formats are possible.
  • Video content delivered to a viewer may include advertising content that is contained directly in the video content, forcing the viewer to watch the advertising content in order to view the video content.
  • web pages may include other additional content that is secondary to the primary content of the web page.
  • This additional content may include navigation bars, sidebar information, links to other pages on the same website (e.g., links to other articles on a news aggregation website), and so forth.
  • This additional content may not strictly be defined as advertising content such as banner ads linked to a page promoting a certain product or service. Nevertheless, the additional content may be secondary to the primary content included in a webpage (e.g., a news article or video included in the body of the webpage).
  • a viewer is more than likely not as engaged when being forced to watch such advertising content in addition to the primary content that is the focus of the web page. Consequently, the playback of advertising content or the simultaneous display of such additional content creates an ideal opportunity to operate the device in a low-power mode in order to reduce power consumption of the device and extend the battery life.
  • the experience of using the device in a low-power mode is not likely to be as distracting to a consumer if the device is in the low-power mode when the advertising content is being displayed and then returned to a normal mode once the viewer returns to viewing the content of their choice.
  • FIG. 1 illustrates a mobile device 100 , in accordance with one embodiment.
  • the mobile device 100 may be a smart phone or tablet computer or other type of electronic device including a battery as the primary source of power.
  • the mobile device 100 includes a system-on-chip (SoC) 110 coupled to a memory 160 and a display device 180 .
  • SoC 110 is a processor implemented on an integrated circuit.
  • the memory 160 is a volatile storage such as synchronous dynamic random access memory (SDRAM).
  • SDRAM synchronous dynamic random access memory
  • An operating system, applications, and/or data may be stored in the memory 160 during operation.
  • the SoC 110 may be included with one or more discrete memory ICs in a package on package (PoP) configuration.
  • PoP package on package
  • the display device 180 may be a liquid crystal display (LCD), organic light emitting diode (OLED), active matrix OLED (AMOLED), or the like.
  • the display device 180 implements touch functionality such as by including capacitive touch sensing technology integrated into a layer of the display device 180 .
  • the display device 180 may also include a backlight and backlight controller.
  • the backlight may be one or more LEDs configured to generate a diffuse white light that is transmitted through a layer of liquid crystal elements.
  • the backlight controller may be implemented as logic within an IC for controlling the operation of the LEDs.
  • the backlight may be adjusted via the backlight controller to control a brightness of the display device 180 .
  • the SoC 110 includes at least one CPU 120 , at least one GPU 130 , a system bus 102 , a memory management unit (MMU) 140 , and a memory interface 104 .
  • the CPU 120 is a general purpose processor having one or more cores.
  • the CPU 120 is a RISC (reduced instruction set computer) processor core such as an ARM ® Coretex processor.
  • the GPU 130 is a parallel processing unit including of one or more cores configured to process graphics or other data in a parallelizable manner.
  • the GPU 130 may be implemented with SIMD (single instruction, multiple data) architecture.
  • the GPU 130 includes one or more ARM ® Mali GPU cores.
  • the memory 160 is coupled to the memory interface 104 such that instructions and data can be loaded into on-chip memory within the CPU 120 and/or GPU 130 .
  • the MMU 140 manages memory access requests generated by one or more components of the SoC 110 .
  • the MMU 140 may also translate virtual addresses associated with the memory access requests into physical addresses associated with locations in the memory 160 .
  • the mobile device 100 is also coupled to a radio frequency transceiver 150 , a non-volatile storage device 170 , and a power management integrated circuit (PMIC) 190 .
  • the radio frequency transceiver 150 may include at least one antenna and logic for encoding or decoding signals transmitted from or received by the mobile device 100 .
  • the mobile device 100 may include a radio for transmitting cellular signals on one or more cellular frequency bands.
  • the RF transceiver 150 may be configured to transmit signals over Wi-Fi (e.g., within the 2.4 GHz or 5 GHz bands).
  • the non-volatile storage device 170 may be a hard disk drive (HDD), solid state drive (SSD), flash memory such as an EEPROM or Compact Flash (CF) Card, and the like.
  • the non-volatile storage device 170 may store the operating system, applications, and/or data that can be loaded into the memory 160 during normal operation.
  • the PMIC 190 is configured to perform power management operations of the mobile device 100 .
  • the PMIC 190 converts the DC power supply from a battery 195 to a supply voltage provided to the SoC 110 .
  • the PMIC 190 may also be configured to implement dynamic voltage and frequency scaling (DVFS) by adjusting a level of the supply voltage provided to the SoC 110 and/or a frequency of the clock domain utilized by the SoC 110 .
  • the PMIC 190 may also be configured to manage charging of the battery 195 when the mobile device is plugged into an external power source.
  • the PMIC 190 is integrated into the SoC 110 rather than being included as a separate IC within the mobile device 100 .
  • the mobile device 100 shown in Figure 1 is only one exemplary embodiment of a mobile device.
  • the mobile device 100 may include separate and distinct CPU and/or GPU ICs rather than the SoC 110 .
  • the mobile device 100 may include other types of components in addition to or in lieu of the components shown in Figure 1 .
  • the mobile device 100 may include a microphone and/or speakers coupled to I/O ports of the SoC 110 .
  • FIG. 2 illustrates a software environment 200 of the mobile device 100 , in accordance with one embodiment.
  • the mobile device 100 may be configured to load an operating system 210 into the memory 160 .
  • the operating system 210 is executed by the CPU 120 .
  • the operating system 210 manages various hardware interfaces of the mobile device 100 and enables applications to be executed using the CPU 120 and/or GPU 130 .
  • the operating system 210 may include an operating system kernel, which is the set of instructions that define the core of the operating system.
  • the kernel may establish data structures in the memory 160 that enable multiple processes to be managed by the operating system concurrently.
  • the kernel may also implement functions for power management, scheduling, and the operation of various hardware resources.
  • the operating system 210 may also include device drivers that implement various functions associated with components of the mobile device 100 .
  • a device driver may be installed for a GPU 130 that enables the CPU 120 to offload various operations to the GPU 130 .
  • a device driver may also be installed for the display device 180 , RF transceiver 150 , non-volatile storage 170 , network interface controller (NIC), input devices such as a keyboard, mouse, touchpad or touchscreen, and the like.
  • NIC network interface controller
  • the operating system 210 divides a virtual address space into a kernel space and a user space.
  • the kernel space is a range of virtual addresses that are allocated to the operating system kernel and related processes
  • the user space is a range of virtual addresses that are allocated to normal processes.
  • certain programs such as the operating system kernel and certified device drivers, may be executed in the kernel space while other applications are executed in the user space.
  • Processes executing in user space may not be allowed to access memory in the kernel space, which prevents these processes from crashing the system and accessing hardware resources directly.
  • certain processes running in kernel space may implement an application programming interface (API) that can be called from processes running in user space, such that certain low-level functions can be accessed by processes running in user space in a safe manner.
  • API application programming interface
  • An application 240 is stored in the memory 160 and may be executed by the CPU 120 within the software environment 200 .
  • the application 240 is a web browser. Other suitable applications are possible.
  • a plug-in or other software component 230 may be provided that adds certain functionality within the application 240 .
  • the plug-in 230 is a program that enables the mobile device 100 to switch between a normal mode and a low-power mode during the playback of video content.
  • the plug-in 230 may be stored in the non-volatile storage 170 of the mobile device 100 and loaded into the memory 160 when a user opts to run the application 240 .
  • the plug-in 230 is configured to detect when video content is loaded into a window of the application 240 .
  • the plug-in 230 may determine if the video content is associated with any unwanted content.
  • the unwanted content may include advertising content included in the video content as one or more frames of video, advertising content embedded in the window and displayed next to the video content, or any additional content embedded in the window and display next to the video content.
  • the unwanted content may include advertising content included in the video content.
  • a commercial may refer to, e.g., a 30-second clip included within the video content that includes images related to a product or service that may not be associated with the main subject matter of the video.
  • the advertising content may then be defined as the one or more commercials (e.g., sequences of frames) included in the video content.
  • the unwanted content may also include elements not directly contained within the video content.
  • unwanted content may refer to text, sponsored links, images, embedded video, and/or other elements displayed alongside the video content within the window of the application 240 .
  • particular elements of certain websites are flagged as unwanted content and stored in a database.
  • the plug-in 230 may query the database using the uniform resource locator (URL) for the markup language document to determine if any elements of the webpage have been flagged as unwanted content. Consequently, any elements flagged in the database may be identified by the plug-in 230 as unwanted content.
  • URL uniform resource locator
  • the plug-in 230 may cause the mobile device 100 to be operated in a different mode of operation during playback of the video content.
  • the plug-in 230 may utilize an application programming interface (API) 220 that enables the plug-in 230 to change the operating mode of the mobile device 100 between a low-power mode and a normal mode.
  • the API 220 is implemented in a driver for the PMIC 190 .
  • the API 220 may include a set of functions that can be called by the plug-in 230 operating in the user space.
  • the functions may cause the operating system 210 to transmit signals to the PMIC 190 that cause the PMIC 190 to adjust the voltage supplied to the SoC 110 and/or adjust a frequency of the clock domains for the SoC 110 .
  • the driver may be executed in a kernel space such that the API 220 provides a bridge between the plug-in 230 operating in the user space and the PMIC 190 , which is controlled by the driver operating in the kernel space.
  • the API 220 is implemented in a driver for the display device 180 .
  • the API 220 may include a set of functions for adjusting the brightness of a backlight of the display device 180 .
  • the plug-in 230 can control the brightness of the backlight by calling a corresponding function of the API 220 .
  • the API 220 is implemented in a driver for the GPU 130 .
  • the API 220 may enable the plug-in 230 to specify a resolution of images rendered to the display device 180 .
  • the API 220 may include a function that lets the plug-in 230 configure a resolution of a frame buffer that is specified as a render target for pixel data generated from the video content.
  • the video content may be rendered at a particular resolution as specified by the plug-in 230 .
  • the plug-in 230 may adjust the resolution of the video dynamically as each frame of video is rendered by the GPU 130 .
  • more than one API may be implemented in more than one corresponding drivers associated with the operating system 210 in order to enable different functionality to be utilized by the plug-in 230 executing in the user space.
  • the operating system kernel itself may implement an API, through defined system calls, that enable different functionality to be utilized by the plug-in 230 executing in the user space.
  • the plug-in 230 is configured to switch the mobile device 100 into a low-power mode using the API 220 .
  • the low-power mode may include any steps to save power by, for example, a combination of DVFS, reduction in brightness of the display device, and reduction in resolution of rendered images for display.
  • the plug-in 230 may switch the mobile device 100 into the low-power mode during playback of the video content in the application in response to detecting the unwanted content.
  • the application 240 and plug-in 230 may be implemented as a single application.
  • the functionality of the plug-in 230 may be implemented within a media viewer application such as a video player.
  • the plug-in 230 is not a separate extension of a base application such as a webbrowser.
  • Figures 3A through 3C illustrate video content 310 displayed by a mobile device 100 , in accordance with one embodiment.
  • the application 240 may be a web browser configured to read a markup language document (e.g., HTML document, XML document, etc.) that includes at least one element associated with video content (other suitable types of applications are possible).
  • HTML documents may include a video element that specifies a source location of a video file to be displayed within a window of the web browser.
  • the HTML element may also specify various parameters such as whether the video content should begin playing when the document is first loaded into the window or whether a user must first click on the HTML element within the web browser to initiate playback of the video content.
  • the video content may be played back within the extents of the HTML element as rendered in the window of the web browser (i.e., embedded mode) or using the full extents of the display device 180 (i.e., full-screen mode).
  • a mobile device 100 includes a display device 180 that is playing video content 310 in a full-screen mode.
  • the video content 310 may block any other elements of the web page from being displayed as the video content 310 is overlaid on top of the graphical representation of the web browser.
  • the plug-in 230 is configured to detect when video content is played in full-screen mode. In full-screen mode while the video content is playing, the plug-in 230 is configured to detect advertising content included in the video content as one or more frames of video.
  • advertising content may be specified using identifying markers included in metadata associated with the video content. The markers specify explicit start points and end points of the advertising content included in the video content.
  • the markers may comprise timestamps that indicate where in the video the advertising content starts and stops.
  • the markers may be stored as metadata associated with the video file encoding the video content.
  • the markers can be included directly in the video content encoded within the video file, such as by encoding a signal specifying the markers in the vertical blanking interval of a frame of video in the video content.
  • advertising content may be specified utilizing user feedback to identify explicit start points and end points of the advertising content included in the video content.
  • the plug-in 230 may enable users to indicate that certain frames of video within the video content are advertising content.
  • the plug-in 230 may provide graphical user interface (GUI) elements that let a user click a button to indicate the start and stop of advertising content included in the video content.
  • GUI graphical user interface
  • the feedback provided by the user may be encoded and sent to a server along with feedback from a number of other users. The feedback from a large number of users may be analyzed to determine which portions of video content within a particular video file are advertising content.
  • the plug-in 230 may send a request to the server to determine whether the video content includes advertising content.
  • the server may provide the plug-in 230 with the markers that indicate which frames of video within the video content are advertising content.
  • the plug-in 230 may be configured to analyze the video content to identify keywords or images corresponding to advertising content. Commercials are typically separated from other scenes in the video content by black frames for a small number of frames. Therefore, the plug-in 230 can analyze each frame of video in the video content to determine a demarcation between different scenes and then analyze each scene for various keywords or images that indicate that the scene is advertising content.
  • a scene may include at least one frame with a company's name or logo that is commonly used in commercials produced by that company. Image analysis can be used to identify the name or logo within the scene and then the start and stop of the scene, as defined by the black frames, may be specified as the start and stop of the advertising content.
  • a server may analyze the video content in the manner described above and then transmit markers that identify the start and stop of the advertising content to the plug-in 230 on a client device.
  • the plug-in 230 is configured to configure the mobile device 100 to operate in a low-power mode during playback of the video content 310 .
  • the plug-in 230 may utilize DVFS to save power by operating a processor of the mobile device 100 , such as the SoC 110 , at a lower voltage and/or lower frequency while the advertising content is being displayed.
  • the plug-in 230 is configured to cause the mobile device 100 to enter the low-power mode by calling an application programming interface (API) method in response to detecting the display of the advertising content on the mobile device 100 .
  • API application programming interface
  • Detecting the display of the advertising content may refer to detecting a frame that marks the start of advertising content within the video content 310 .
  • the plug-in 230 executing in the user space, calls the API method, which generates a system interrupt.
  • the system interrupt is processed by the operating system 210 and/or a device driver for the PMIC 190 , which generate a signal on an I/O port of the SoC 110 that is transmitted to the PMIC 190 .
  • the signal instructs the PMIC 190 to reduce a voltage supplied to the SoC 110 and/or reduce a frequency of the clock domains utilized by the SoC 110 .
  • the plug-in 230 may cause the mobile device 100 to return to the normal operating mode while playback of the video content is completed.
  • the plug-in 230 may reduce the resolution of the video content by configuring the GPU 130 to render the frames of video marked as advertising content at a lower resolution.
  • the video content rendered by the GPU 130 and displayed on the display device 180 may be rendered at a first resolution such as 1920 ⁇ 1080 pixels.
  • the GPU 130 may be configured to render the video content 310 to a frame buffer configured according to the first resolution.
  • the plug-in 230 is configured to send a command to the GPU 130 that configures a new frame buffer at a lower resolution, such as 640 ⁇ 480 pixels. Each frame of advertising content is then rendered to the new frame buffer at the lower resolution.
  • the GPU 130 may be configured to utilize lower-quality rendering techniques.
  • the GPU 130 may be configured to render the video content 310 into the high resolution frame buffer using MSAA or SSAA techniques in the normal mode, but GPU 130 may be re-configured to render the advertising content into the low-resolution frame buffer using a single sample per pixel in the low-power mode.
  • the effect of reducing the resolution of the video content during display of the advertising content is that fewer pixels and/or samples per pixel will need to be rendered by the GPU 130 , which can drastically reduce the time to render each frame by the GPU 130 .
  • Power saving modes such as clock gating the GPU 130 within the SoC 110 when the GPU 130 is inactive between frames, can save significant amounts of power.
  • the quality of the images displayed for the advertising content may be much less than the quality of the images displayed for the video content, but the viewer will typically not be adverse to the reduction in quality when the reduction extends the life of the battery in the mobile device 100 .
  • the plug-in 230 may reduce a brightness level of a display device 180 included in the mobile device 100 .
  • the plug-in 230 is configured to cause the mobile device 100 to enter the low-power mode by calling an application programming interface (API) method in response to detecting the display of the advertising content on the mobile device.
  • API application programming interface
  • the plug-in 230 executing in the user space, calls the API method, which generates a system interrupt.
  • the system interrupt is processed by the operating system 210 and/or a device driver for the display device 180 , which generate a signal on an I/O port of the SoC 110 that is transmitted to the display device 180 .
  • the signal instructs a backlight controller of the display device 180 to reduce the brightness of the display device 180 .
  • the brightness of the display device 180 may, therefore, be reduced by, e.g., 50% while advertising content is presented on the display device 180 during playback of the video content. Again, this reduction in brightness may be very noticeable to a viewer while advertising content is being played, but the viewer will typically not be adverse to the reduction in brightness when the reduction extends the life of the battery in the mobile device 100 . Reducing the brightness of the backlight may save significant power consumed by the display device 180 while advertising content is being played in a full-screen mode.
  • the plug-in 230 may be configured to combine two or more of the techniques described above when operating the mobile device 100 in the low-power mode.
  • the plug-in 230 may configure the device to use a lower voltage and/or frequency utilizing DVFS as well as reducing the resolution of the advertising content rendered for display and reducing a brightness of the screen during playback of the advertising content. Combinations of these power-saving techniques may be used to further reduce the power consumption of the mobile device 100 in the low-power mode.
  • the plug-in 230 is configured to return the mobile device 100 to normal mode.
  • the plug-in 230 may generate a system call via the API 220 that is processed by the operating system and/or device driver of the PMIC 190 to transmit a signal to the PMIC 190 to increase the voltage and/or frequency of the clock domains of the SoC 110 to the higher voltage and higher frequency.
  • the plug-in 230 may also configure the GPU 130 to render the video content 310 at the higher resolution by reconfiguring the frame buffer, and increase the brightness of the screen by generating a system call via the API 220 that is processed by the operating system and/or device driver of the display device 180 to transmit a signal to the backlight controller of the display device 180 .
  • the video content 310 is not always displayed in full-screen mode. Some video content 310 may be played back in an embedded mode, where the video content 310 is viewed within the extents of an element of a web page.
  • the web page may also include other elements such as text, a menu, hyperlinks, etc. Some of the other elements may also include unwanted content 320 , such as a banner advertisement that is an image that is typically associated with a hyperlink.
  • the plug-in 230 is configured to mask the unwanted content 320 such as by rendering only the video content 310 over a black background, as shown in Figure 3C . Masking the other elements of the web page effectively prevents the unwanted content 320 from being displayed.
  • the plug-in 230 may configure the SoC 110 to throttle one or more threads associated with the unwanted content 320 .
  • some unwanted content 320 may include an animation or video embedded in the web page as a second element separate from the video content 310 . Separate threads may be allocated to render the video content 310 and the unwanted content 320 .
  • the plug-in 230 is configured to deactivate (i.e., throttle) the thread associated with the unwanted content 320 , while the thread associated with the video content 310 remains active.
  • Other threads may be throttled as well, such as threads related to other elements of the web page.
  • the video content 310 playing in the embedded mode also has advertising content included therein.
  • a subset of frames included in the video content 310 are identified as advertising content as well as the unwanted content 320 displayed outside of the element where the video content 310 is rendered.
  • the plug-in 230 may be configured to implement the techniques described above (DVFS, reduced resolution, and reduced brightness) when the advertising content included within the video content 310 is displayed.
  • content associated with video content can be assigned different priority levels. For example, advertising content included within a video can be assigned a first priority level, advertising content embedded in a webpage alongside the video content can be assigned a second priority level, and additional elements of the webpage (e.g., navigation menus, links to additional articles, etc.) can be assigned a third priority level.
  • a user may designate a threshold for priority levels that are identified as unwanted content. For example, one user may only want to designate advertising content included in the video content as unwanted content such that the low-power mode is only entered when the advertising content is displayed during playback of the video content. Another user may designate both advertising content included in the video content as well as other advertising content embedded in the webpage as unwanted content. Yet another user may designate advertising content (both included in the video content and embedded in the webpage alongside the video content) as well as additional elements as unwanted content.
  • FIG 4 illustrates video content 310 that includes advertising content 350 , in accordance with one embodiment.
  • the video content 310 includes a sequence of frames that can be played back by rendering each frame to the display device 180 .
  • the video content 310 starts at a time To 401 where a first frame of video content 310 is encoded in a file format for video (e.g., MPEG-4 AVC, AVI, MOV, etc.).
  • the video content 310 includes a number of frames until a time T 1 402 , where a first frame of advertising content 350 appears in the sequence of frames.
  • the advertising content 350 may be a 30 second commercial for a product or service. As shown in Figure 4 , the commercial is for a company called Burger Joint.
  • the last frame of advertising content 350 in the commercial is located at a time T 2 403 .
  • Additional advertising content 350 included in the video content 310 such as a commercial for Car Co., may be located at frames between time T 3 404 and T 4 405 , and the end of the video content 310 is at time T 5 406 .
  • the plug-in 230 is configured to detect the video content 310 to be viewed within an application of the mobile device 100 .
  • the plug-in 230 is also configured to detect the unwanted content associated with the video content 310 , such as detecting the advertising content 350 included in the video content 310 by reading markers from the metadata associated with video content 310 that indicates that advertising content 350 is included in the video content 310 between times T 1 402 and T 2 403 as well as between times T 3 404 and T 4 405 .
  • the plug-in 230 is configured to operate the mobile device 100 in normal mode from time T 0 401 to time T 1 402 , in low-power mode from time T 1 402 to time T 2 403 , in normal mode from time T 2 403 to time T 3 404 , in low-power mode from time T 3 404 to time T 4 405 , and in normal mode from time T 4 405 to time T 5 406 .
  • FIG. 5 illustrates a system 500 for detecting unwanted content associated with video content, in accordance with one embodiment.
  • J client nodes 502 are coupled to a server node 504 via a network 510 .
  • the network 510 is the Internet and the nodes communicate via the TCP/IP protocol.
  • Each client node 502 may be a mobile device 100 , which includes the plug-in 230 and application 240 .
  • the plug-in 230 of the mobile device 100 may transmit feedback information from the client node 502 to the server node 504 .
  • the feedback information is stored in a database 520 that correlates a file name for the video content with the user-generated markers that indicate the location of advertising content 350 included within the video content 310 .
  • the server node 504 may include software to analyze the feedback from a plurality of client nodes 502 to generate markers for identifying advertising content 350 included in the video content 310 . After feedback from a number of users has been received, the server node 504 may analyze all of the feedback information to identify the advertising content 350 in the video content 310 .
  • each user may indicate where in the video content 310 advertising content 350 starts and stops.
  • An algorithm executed by the server node 504 may count the number of users that identified each frame of video content 310 as advertising content 350 . Any frames with a number of users that identified that frame as advertising content 350 above a threshold number of users are then identified as advertising content 350 .
  • the algorithm may then estimate where advertising content 350 starts and stops within the video content 310 based on which frames have been identified by users as advertising content 350 . Markers that identify those frames as advertising content 350 may then be generated and correlated to that particular video file.
  • the server node 504 includes software for performing image analysis on the frames of video content 310 .
  • the server node 504 may compare objects in each frame with known objects associated with advertising content 350 , such as a number of well-known company trademarks.
  • the server node 504 may also identify keywords from text included in the frames that are associated with advertising content.
  • the analysis may identify certain frames of video as likely advertising content 350 .
  • the server node 504 may then estimate where advertising content 350 starts and stops within the video content 310 based on which frames have been identified via image analysis as advertising content 350 . Markers that identify those frames as advertising content 350 may then be generated and correlated to that particular video file.
  • the server node 504 may be used to identify unwanted content that is not included within the video content 310 .
  • users may mark specific elements of a markup language document as unwanted content and submit their feedback to the server node 504 .
  • the server node 504 may collect user feedback and add the feedback to a database. Once a threshold number of users have identified a particular element in a markup language document as unwanted content, then the server node 504 may mark that element as unwanted content in a database. The server node 504 may then identify that element as unwanted content in response to a request from a client node 502 .
  • the plug-in 230 may then request information from the server node 504 in order to identify the location of advertising content 350 included in the video content 310 and/or unwanted content associated with the video content 310 .
  • FIG. 6 is a flowchart of a method 600 for reducing the power consumption of a mobile device, in accordance with one embodiment.
  • video content 310 to be viewed in an application 240 executed by a mobile device 100 is detected.
  • the application is a web browser, and the video content is detected while parsing markup language documents.
  • An element of the markup language document may be identified during parsing the markup language document to identify a multimedia element that includes a pointer to a video file that indicates that video content is going to be embedded within a browser window that is displayed to a user of the mobile device 100 .
  • the application may be a media player application, and the video content is detected when a user requests the application open a particular file having a video format.
  • a plug-in 230 for the application determines whether the video content 310 includes advertising content 350 by requesting such information from a server node 504 .
  • the plug-in 230 reads metadata associated with the video content 310 in order to determine that the video content 310 includes advertising content 350 .
  • the plug-in 230 identifies whether the video content 310 is played in a full-screen mode or an embedded mode. If the video content is played in an embedded mode, then unwanted content 320 may be identified by querying a server node 504 . In yet another embodiment, unwanted content may be identified as any content rendered outside the extents of the video content 310 .
  • the mobile device 100 is configured to operate in a low-power mode during playback of the video content in the application in response to detecting the unwanted content.
  • the plug-in 230 may mask any elements displayed alongside the video content 310 and throttle any threads associated with the unwanted content during playback of the video content 310 .
  • the plug-in 230 may cause the mobile device 100 to operate in the low-power mode when advertising content 350 included in the video content 310 is displayed during playback of the video content 310 .
  • FIG. 7 illustrates an exemplary system 700 in which the various architecture and/or functionality of the various previous embodiments may be implemented.
  • a system 700 is provided including at least one central processor 701 that is connected to a communication bus 702 .
  • the communication bus 702 may be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol(s).
  • the system 700 also includes a main memory 704 . Control logic (software) and data are stored in the main memory 704 which may take the form of random access memory (RAM).
  • RAM random access memory
  • the system 700 also includes input devices 712 , a graphics processor 706 , and a display 708 , i.e. a conventional CRT (cathode ray tube), LCD (liquid crystal display), LED (light emitting diode), plasma display or the like.
  • User input may be received from the input devices 712 , e.g., keyboard, mouse, touchpad, microphone, and the like.
  • the graphics processor 706 may include a plurality of shader modules, a rasterization module, etc. Each of the foregoing modules may even be situated on a single semiconductor platform to form a graphics processing unit (GPU).
  • GPU graphics processing unit
  • a single semiconductor platform may refer to a sole unitary semiconductor-based integrated circuit or chip. It should be noted that the term single semiconductor platform may also refer to multi-chip modules with increased connectivity which simulate on-chip operation, and make substantial improvements over utilizing a conventional central processing unit (CPU) and bus implementation. Of course, the various modules may also be situated separately or in various combinations of semiconductor platforms per the desires of the user.
  • CPU central processing unit
  • the system 70 0 may also include a secondary storage 710 .
  • the secondary storage 710 includes, for example, a hard disk drive and/or a removable storage drive, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, digital versatile disk (DVD) drive, recording device, universal serial bus (USB) flash memory.
  • the removable storage drive reads from and/or writes to a removable storage unit in a well-known manner.
  • Computer programs, or computer control logic algorithms may be stored in the main memory 704 and/or the secondary storage 710 . Such computer programs, when executed, enable the system 700 to perform various functions.
  • the memory 704 , the storage 710 , and/or any other storage are possible examples of computer-readable media.
  • the architecture and/or functionality of the various previous figures may be implemented in the context of the central processor 701 , the graphics processor 706, an integrated circuit (not shown) that is capable of at least a portion of the capabilities of both the central processor 701 and the graphics processor 706 , a chipset (i.e., a group of integrated circuits designed to work and sold as a unit for performing related functions, etc.), and/or any other integrated circuit for that matter.
  • the central processor 701 the graphics processor 706, an integrated circuit (not shown) that is capable of at least a portion of the capabilities of both the central processor 701 and the graphics processor 706 , a chipset (i.e., a group of integrated circuits designed to work and sold as a unit for performing related functions, etc.), and/or any other integrated circuit for that matter.
  • the architecture and/or functionality of the various previous figures may be implemented in the context of a general computer system, a circuit board system, a game console system dedicated for entertainment purposes, an application-specific system, and/or any other desired system.
  • the system 700 may take the form of a desktop computer, laptop computer, server, workstation, game consoles, embedded system, and/or any other type of logic.
  • the system 700 may take the form of various other devices including, but not limited to a personal digital assistant (PDA) device, a mobile phone device, a television, etc.
  • PDA personal digital assistant
  • system 700 may be coupled to a network (e.g., a telecommunications network, local area network (LAN), wireless network, wide area network (WAN) such as the Internet, peer-to-peer network, cable network, or the like) for communication purposes.
  • a network e.g., a telecommunications network, local area network (LAN), wireless network, wide area network (WAN) such as the Internet, peer-to-peer network, cable network, or the like
  • LAN local area network
  • WAN wide area network
  • peer-to-peer network such as the Internet
  • cable network or the like
  • the system 700 includes a video content detection module detecting video content to be viewed in an application executed by the mobile device, an unwanted content detection module detecting unwanted content associated with the video content, and a low-power module operating the mobile device in a low-power mode during playback of the video content in the application in response to detecting the unwanted content.
  • the system 700 may include other or additional modules for performing any one of or combination of steps described in the embodiments. Further, any of the additional or alternative embodiments or aspects of the method, as shown in any of the figures or recited in any of the claims, are also contemplated to include similar modules.
  • a "computer-readable medium” includes one or more of any suitable media for storing the executable instructions of a computer program such that the instruction execution machine, system, apparatus, or device may read (or fetch) the instructions from the computer readable medium and execute the instructions for carrying out the described methods.
  • Suitable storage formats include one or more of an electronic, magnetic, optical, and electromagnetic format.
  • a non-exhaustive list of conventional exemplary computer readable medium includes: a portable computer diskette; a RAM; a ROM; an erasable programmable read only memory (EPROM or flash memory); optical storage devices, including a portable compact disc (CD), a portable digital video disc (DVD), a high definition DVD (HD-DVDTM), a BLU-RAY disc; and the like.
  • one or more of these system components may be realized, in whole or in part, by at least some of the components illustrated in the arrangements illustrated in the described Figures.
  • the other components may be implemented in software that when included in an execution environment constitutes a machine, hardware, or a combination of software and hardware.
  • At least one component defined by the claims is implemented at least partially as an electronic hardware component, such as an instruction execution machine (e.g., a processor-based or processor-containing machine) and/or as specialized circuits or circuitry (e.g., discreet logic gates interconnected to perform a specialized function).
  • an instruction execution machine e.g., a processor-based or processor-containing machine
  • specialized circuits or circuitry e.g., discreet logic gates interconnected to perform a specialized function.
  • Other components may be implemented in software, hardware, or a combination of software and hardware. Moreover, some or all of these other components may be combined, some may be omitted altogether, and additional components may be added while still achieving the functionality described herein.
  • the subject matter described herein may be embodied in many different variations, and all such variations are contemplated to be within the scope of what is claimed.
  • the computer-readable non-transitory media includes all types of computer-readable media, including magnetic storage media, optical storage media, and solid state storage media and specifically excludes signals.
  • the software can be installed in and sold with the mobile device. Alternatively, the software can be obtained and loaded into the mobile device, including obtaining the software via a disc medium or from any manner of network or distribution system, including, for example, from a server owned by the software creator or from a server not owned but used by the software creator.
  • the software can be stored on a server for distribution over the Internet, for example.

Description

    Field of the Invention
  • The present invention relates to energy efficiency, and more particularly to power saving methods associated with streaming or playback of video with embedded advertisements.
  • BACKGROUND
  • The battery capacity of mobile devices such as tablet computers and smart phones is extremely important to the consumer's satisfaction with the device. A single/full charge of a battery may need to last 6-8 hours in order for the consumer to use the device while on the go. Each manufacturer of these devices has poured significant resources into implementing techniques to reduce the power consumption of the device in order to extend the life of the battery during operation. Separate power management chips may be used to power gate certain components of the device. In addition, software solutions have been implemented to reduce the power consumption of the device, such as by reducing the number of active threads in a low-power mode or reducing the brightness of a display.
  • WO 2009/136236 A1 refers to an electronic device that includes video storage memory, a video signal receiver, and a video storage controller. The video signal receiver is configured to receive a video stream containing advertising content and program/movie content from a broadcast source. The video storage controller is configured to identify locations of advertising content interspersed among program/movie content in the received video stream, and to record addressable chapter marks for the identified locations of the advertising content and a video stream that is based on the received video stream into the memory. The addressable chapter marks may be displayed on a display and used by a viewer to jump playback of the video stream between the chapter marks.
  • Typically, a significant portion of the energy consumed by a mobile device is devoted to powering a display device. For example, a backlight of the display may draw a significant amount of power to turn on at the brightest setting. In addition, one or more graphics processing cores in a graphics processing unit (GPU) or system-on-chip (SoC) may be dedicated to generating pixel data for presentation on the display device. This activity may rapidly drain the battery, rendering the mobile device inoperable after only a short period. Thus, there is a need for addressing this issue and/or other issues associated with the prior art.
  • SUMMARY
  • A system and method are provided for reducing the power consumption of a mobile device. This problem is solved by the subject matter of the independent claims. Further implementation forms are provided in the dependent claims. The method includes the steps of detecting video content to be viewed in an application executed by the mobile device; detecting unwanted content associated with the video content; and operating the mobile device in a low-power mode during playback of the video content in the application in response to detecting the unwanted content in the fashion and details as disclosed in the appended set of claims.
  • To this end, in some optional embodiments, one or more of the foregoing features of the aforementioned apparatus, system, and/or method may afford a more efficient way to configure exploration operations of large data sets that, in turn, may enable data analysts to work more efficiently and reduce errors in the results obtained by the exploration operations. It should be noted that the aforementioned potential advantages are set forth for illustrative purposes only and should not be construed as limiting in any manner.
  • BRIEF DESCRIPTION OF THE DRAWINGS
    • Figure 1 illustrates a mobile device, in accordance with one embodiment;
    • Figure 2 illustrates a software environment of the mobile device, in accordance with one embodiment;
    • Figures 3A through 3C illustrate video content displayed by a mobile device, in accordance with one embodiment;
    • Figure 4 illustrates video content that includes advertising content, in accordance with one embodiment;
    • Figure 5 illustrates a system for detecting unwanted content associated with video content, in accordance with one embodiment;
    • Figure 6 is a flowchart of a method for reducing the power consumption of a mobile device, in accordance with one embodiment; and
    • Figure 7 illustrates an exemplary system in which the various architecture and/or functionality of the various previous embodiments may be implemented.
    DETAILED DESCRIPTION
  • Much effort has been expended designing techniques to reduce power consumption of electronic devices. Mobile devices particularly benefit from these efforts as reducing power consumption extends battery life, thereby enabling these devices to be used for longer amounts of time before requiring re-charging of the battery. Solutions for reducing power consumption may focus at least some of their attention on reducing the power consumption of a display device. The main energy consumers associated with a display device are the backlight, an active matrix of liquid crystals, and the processors tasked with generating pixel data for display on the device. For example, some LED backlight televisions have been designed with an array of LEDs positioned behind the array of liquid crystal elements so that the brightness of the backlight can be dimmed in areas of the screen associated with darker colors. Local area dimming can significantly reduce the power consumed by the display device.
  • Other techniques can be employed to reduce the power consumed by a display device as well. Some examples of these techniques are dynamic frequency and voltage scaling (DFVS) for the processor tasked with generating pixel data for display, reducing resolution of the rendered pixel data, throttling processing capacity dedicated to rendering operations, and so forth. However, these techniques typically have an adverse effect on the operation of the device. For example, the frequency of the clock of a processor may need to be reduced in order to reduce the operating voltage of the processor. A user may notice significant delays when performing certain operations under these conditions. As another example, reducing the brightness of the display device or the resolution of the images presented on the display device can have an adverse effect on image quality when compared to normal operation. A consumer may not accept this reduction in performance in order to extend the battery life of the device during normal use. However, the consumer may be able to put up with the reduction in performance at certain times, such as when the battery is nearly drained.
  • One of these times may be when the consumer is being forced to watch advertising content. Advertising revenue is the main source of income for many content providers that distribute content online. Web pages viewed in a web browser may include banner advertisements or sponsored links designed to redirect users to an advertiser's website. Some websites automatically play embedded video content when a web page is loaded into a browser window. Video content may include files having a video format including, but not limited to, MPEG-4 AVC (Moving Picture Experts Group 4, Part 10/H.264), AVI (Audio Video Interleave), and MOV (Apple Quicktime Movie) as well as other suitable video formats are possible. Video content delivered to a viewer may include advertising content that is contained directly in the video content, forcing the viewer to watch the advertising content in order to view the video content.
  • In addition to advertising content, web pages may include other additional content that is secondary to the primary content of the web page. This additional content may include navigation bars, sidebar information, links to other pages on the same website (e.g., links to other articles on a news aggregation website), and so forth. This additional content may not strictly be defined as advertising content such as banner ads linked to a page promoting a certain product or service. Nevertheless, the additional content may be secondary to the primary content included in a webpage (e.g., a news article or video included in the body of the webpage).
  • A viewer is more than likely not as engaged when being forced to watch such advertising content in addition to the primary content that is the focus of the web page. Consequently, the playback of advertising content or the simultaneous display of such additional content creates an ideal opportunity to operate the device in a low-power mode in order to reduce power consumption of the device and extend the battery life. The experience of using the device in a low-power mode is not likely to be as distracting to a consumer if the device is in the low-power mode when the advertising content is being displayed and then returned to a normal mode once the viewer returns to viewing the content of their choice.
  • Figure 1 illustrates a mobile device 100, in accordance with one embodiment. The mobile device 100 may be a smart phone or tablet computer or other type of electronic device including a battery as the primary source of power. As shown in Figure 1 , the mobile device 100 includes a system-on-chip (SoC) 110 coupled to a memory 160 and a display device 180. The SoC 110 is a processor implemented on an integrated circuit. The memory 160 is a volatile storage such as synchronous dynamic random access memory (SDRAM). An operating system, applications, and/or data may be stored in the memory 160 during operation. In one embodiment, the SoC 110 may be included with one or more discrete memory ICs in a package on package (PoP) configuration.
  • The display device 180 may be a liquid crystal display (LCD), organic light emitting diode (OLED), active matrix OLED (AMOLED), or the like. In one embodiment, the display device 180 implements touch functionality such as by including capacitive touch sensing technology integrated into a layer of the display device 180. In some embodiments, such as LCD displays, the display device 180 may also include a backlight and backlight controller. The backlight may be one or more LEDs configured to generate a diffuse white light that is transmitted through a layer of liquid crystal elements. The backlight controller may be implemented as logic within an IC for controlling the operation of the LEDs. The backlight may be adjusted via the backlight controller to control a brightness of the display device 180.
  • In one embodiment, the SoC 110 includes at least one CPU 120, at least one GPU 130, a system bus 102, a memory management unit (MMU) 140, and a memory interface 104. The CPU 120 is a general purpose processor having one or more cores. In one embodiment, the CPU 120 is a RISC (reduced instruction set computer) processor core such as an ARM® Coretex processor. The GPU 130 is a parallel processing unit including of one or more cores configured to process graphics or other data in a parallelizable manner. The GPU 130 may be implemented with SIMD (single instruction, multiple data) architecture. In one embodiment, the GPU 130 includes one or more ARM® Mali GPU cores.
  • The memory 160 is coupled to the memory interface 104 such that instructions and data can be loaded into on-chip memory within the CPU 120 and/or GPU 130. The MMU 140 manages memory access requests generated by one or more components of the SoC 110. The MMU 140 may also translate virtual addresses associated with the memory access requests into physical addresses associated with locations in the memory 160.
  • The mobile device 100 is also coupled to a radio frequency transceiver 150, a non-volatile storage device 170, and a power management integrated circuit (PMIC) 190. The radio frequency transceiver 150 may include at least one antenna and logic for encoding or decoding signals transmitted from or received by the mobile device 100. For example, the mobile device 100 may include a radio for transmitting cellular signals on one or more cellular frequency bands. Alternatively, the RF transceiver 150 may be configured to transmit signals over Wi-Fi (e.g., within the 2.4 GHz or 5 GHz bands). The non-volatile storage device 170 may be a hard disk drive (HDD), solid state drive (SSD), flash memory such as an EEPROM or Compact Flash (CF) Card, and the like. The non-volatile storage device 170 may store the operating system, applications, and/or data that can be loaded into the memory 160 during normal operation.
  • The PMIC 190 is configured to perform power management operations of the mobile device 100. In one embodiment, the PMIC 190 converts the DC power supply from a battery 195 to a supply voltage provided to the SoC 110. The PMIC 190 may also be configured to implement dynamic voltage and frequency scaling (DVFS) by adjusting a level of the supply voltage provided to the SoC 110 and/or a frequency of the clock domain utilized by the SoC 110. The PMIC 190 may also be configured to manage charging of the battery 195 when the mobile device is plugged into an external power source. In some embodiments, the PMIC 190 is integrated into the SoC 110 rather than being included as a separate IC within the mobile device 100.
  • It will be appreciated that the mobile device 100 shown in Figure 1 is only one exemplary embodiment of a mobile device. In different embodiments, the mobile device 100 may include separate and distinct CPU and/or GPU ICs rather than the SoC 110. In other embodiments, the mobile device 100 may include other types of components in addition to or in lieu of the components shown in Figure 1 . For example, the mobile device 100 may include a microphone and/or speakers coupled to I/O ports of the SoC 110.
  • Figure 2 illustrates a software environment 200 of the mobile device 100, in accordance with one embodiment. The mobile device 100 may be configured to load an operating system 210 into the memory 160. The operating system 210 is executed by the CPU 120. The operating system 210 manages various hardware interfaces of the mobile device 100 and enables applications to be executed using the CPU 120 and/or GPU 130. The operating system 210 may include an operating system kernel, which is the set of instructions that define the core of the operating system. The kernel may establish data structures in the memory 160 that enable multiple processes to be managed by the operating system concurrently. The kernel may also implement functions for power management, scheduling, and the operation of various hardware resources. The operating system 210 may also include device drivers that implement various functions associated with components of the mobile device 100. For example, a device driver may be installed for a GPU 130 that enables the CPU 120 to offload various operations to the GPU 130. A device driver may also be installed for the display device 180, RF transceiver 150, non-volatile storage 170, network interface controller (NIC), input devices such as a keyboard, mouse, touchpad or touchscreen, and the like.
  • In one embodiment, the operating system 210 divides a virtual address space into a kernel space and a user space. The kernel space is a range of virtual addresses that are allocated to the operating system kernel and related processes, and the user space is a range of virtual addresses that are allocated to normal processes. In other words, certain programs, such as the operating system kernel and certified device drivers, may be executed in the kernel space while other applications are executed in the user space. Processes executing in user space may not be allowed to access memory in the kernel space, which prevents these processes from crashing the system and accessing hardware resources directly. However, certain processes running in kernel space may implement an application programming interface (API) that can be called from processes running in user space, such that certain low-level functions can be accessed by processes running in user space in a safe manner.
  • An application 240 is stored in the memory 160 and may be executed by the CPU 120 within the software environment 200. In one embodiment, the application 240 is a web browser. Other suitable applications are possible. A plug-in or other software component 230 may be provided that adds certain functionality within the application 240. In one embodiment, the plug-in 230 is a program that enables the mobile device 100 to switch between a normal mode and a low-power mode during the playback of video content. The plug-in 230 may be stored in the non-volatile storage 170 of the mobile device 100 and loaded into the memory 160 when a user opts to run the application 240. The plug-in 230 is configured to detect when video content is loaded into a window of the application 240. When the video content is detected, the plug-in 230 may determine if the video content is associated with any unwanted content. The unwanted content may include advertising content included in the video content as one or more frames of video, advertising content embedded in the window and displayed next to the video content, or any additional content embedded in the window and display next to the video content.
  • For example, the unwanted content may include advertising content included in the video content. A commercial may refer to, e.g., a 30-second clip included within the video content that includes images related to a product or service that may not be associated with the main subject matter of the video. The advertising content may then be defined as the one or more commercials (e.g., sequences of frames) included in the video content. The unwanted content may also include elements not directly contained within the video content. For example, unwanted content may refer to text, sponsored links, images, embedded video, and/or other elements displayed alongside the video content within the window of the application 240.
  • In one embodiment, particular elements of certain websites are flagged as unwanted content and stored in a database. When a markup language document is loaded by the plug-in 230, the plug-in 230 may query the database using the uniform resource locator (URL) for the markup language document to determine if any elements of the webpage have been flagged as unwanted content. Consequently, any elements flagged in the database may be identified by the plug-in 230 as unwanted content.
  • If the plug-in 230 determines that the video content is associated with unwanted content, then the plug-in 230 may cause the mobile device 100 to be operated in a different mode of operation during playback of the video content. In one embodiment, the plug-in 230 may utilize an application programming interface (API) 220 that enables the plug-in 230 to change the operating mode of the mobile device 100 between a low-power mode and a normal mode. In one embodiment, the API 220 is implemented in a driver for the PMIC 190. The API 220 may include a set of functions that can be called by the plug-in 230 operating in the user space. The functions may cause the operating system 210 to transmit signals to the PMIC 190 that cause the PMIC 190 to adjust the voltage supplied to the SoC 110 and/or adjust a frequency of the clock domains for the SoC 110. The driver may be executed in a kernel space such that the API 220 provides a bridge between the plug-in 230 operating in the user space and the PMIC 190, which is controlled by the driver operating in the kernel space. In another embodiment, the API 220 is implemented in a driver for the display device 180. The API 220 may include a set of functions for adjusting the brightness of a backlight of the display device 180. The plug-in 230 can control the brightness of the backlight by calling a corresponding function of the API 220. In yet another embodiment, the API 220 is implemented in a driver for the GPU 130. The API 220 may enable the plug-in 230 to specify a resolution of images rendered to the display device 180. For example, the API 220 may include a function that lets the plug-in 230 configure a resolution of a frame buffer that is specified as a render target for pixel data generated from the video content. Thus, the video content may be rendered at a particular resolution as specified by the plug-in 230. The plug-in 230 may adjust the resolution of the video dynamically as each frame of video is rendered by the GPU 130. It will be appreciated that, in some embodiments, more than one API may be implemented in more than one corresponding drivers associated with the operating system 210 in order to enable different functionality to be utilized by the plug-in 230 executing in the user space. In other embodiments, the operating system kernel itself may implement an API, through defined system calls, that enable different functionality to be utilized by the plug-in 230 executing in the user space.
  • In one embodiment, the plug-in 230 is configured to switch the mobile device 100 into a low-power mode using the API 220. The low-power mode may include any steps to save power by, for example, a combination of DVFS, reduction in brightness of the display device, and reduction in resolution of rendered images for display. The plug-in 230 may switch the mobile device 100 into the low-power mode during playback of the video content in the application in response to detecting the unwanted content.
  • It will be appreciated that, in some embodiments, the application 240 and plug-in 230 may be implemented as a single application. For example, the functionality of the plug-in 230 may be implemented within a media viewer application such as a video player. In such embodiments, the plug-in 230 is not a separate extension of a base application such as a webbrowser.
  • Figures 3A through 3C illustrate video content 310 displayed by a mobile device 100, in accordance with one embodiment. In one embodiment, the application 240 may be a web browser configured to read a markup language document (e.g., HTML document, XML document, etc.) that includes at least one element associated with video content (other suitable types of applications are possible). For example, HTML documents may include a video element that specifies a source location of a video file to be displayed within a window of the web browser. The HTML element may also specify various parameters such as whether the video content should begin playing when the document is first loaded into the window or whether a user must first click on the HTML element within the web browser to initiate playback of the video content. Furthermore, the video content may be played back within the extents of the HTML element as rendered in the window of the web browser (i.e., embedded mode) or using the full extents of the display device 180 (i.e., full-screen mode).
  • As shown in Figure 3A , a mobile device 100 includes a display device 180 that is playing video content 310 in a full-screen mode. The video content 310 may block any other elements of the web page from being displayed as the video content 310 is overlaid on top of the graphical representation of the web browser. The plug-in 230 is configured to detect when video content is played in full-screen mode. In full-screen mode while the video content is playing, the plug-in 230 is configured to detect advertising content included in the video content as one or more frames of video. In one embodiment, advertising content may be specified using identifying markers included in metadata associated with the video content. The markers specify explicit start points and end points of the advertising content included in the video content. The markers may comprise timestamps that indicate where in the video the advertising content starts and stops. The markers may be stored as metadata associated with the video file encoding the video content. Alternatively, the markers can be included directly in the video content encoded within the video file, such as by encoding a signal specifying the markers in the vertical blanking interval of a frame of video in the video content.
  • In another embodiment, advertising content may be specified utilizing user feedback to identify explicit start points and end points of the advertising content included in the video content. The plug-in 230 may enable users to indicate that certain frames of video within the video content are advertising content. For example, the plug-in 230 may provide graphical user interface (GUI) elements that let a user click a button to indicate the start and stop of advertising content included in the video content. The feedback provided by the user may be encoded and sent to a server along with feedback from a number of other users. The feedback from a large number of users may be analyzed to determine which portions of video content within a particular video file are advertising content. When the user loads a web page including the video content into the web browser, the plug-in 230 may send a request to the server to determine whether the video content includes advertising content. The server may provide the plug-in 230 with the markers that indicate which frames of video within the video content are advertising content.
  • In yet another embodiment, the plug-in 230 may be configured to analyze the video content to identify keywords or images corresponding to advertising content. Commercials are typically separated from other scenes in the video content by black frames for a small number of frames. Therefore, the plug-in 230 can analyze each frame of video in the video content to determine a demarcation between different scenes and then analyze each scene for various keywords or images that indicate that the scene is advertising content. For example, a scene may include at least one frame with a company's name or logo that is commonly used in commercials produced by that company. Image analysis can be used to identify the name or logo within the scene and then the start and stop of the scene, as defined by the black frames, may be specified as the start and stop of the advertising content. Alternatively, a server may analyze the video content in the manner described above and then transmit markers that identify the start and stop of the advertising content to the plug-in 230 on a client device.
  • It will be appreciated that the techniques for detecting advertising content included within video content described above are merely exemplary embodiments for detecting such advertising content. Other well-known techniques for detecting advertising content included within video content may be utilized by a plug-in 230 and/or service accessible via a server in lieu of the techniques described above and are within the scope of the present disclosure.
  • The plug-in 230 is configured to configure the mobile device 100 to operate in a low-power mode during playback of the video content 310. In the low-power mode, the plug-in 230 may utilize DVFS to save power by operating a processor of the mobile device 100, such as the SoC 110, at a lower voltage and/or lower frequency while the advertising content is being displayed. In one embodiment, the plug-in 230 is configured to cause the mobile device 100 to enter the low-power mode by calling an application programming interface (API) method in response to detecting the display of the advertising content on the mobile device 100. Detecting the display of the advertising content may refer to detecting a frame that marks the start of advertising content within the video content 310. The plug-in 230, executing in the user space, calls the API method, which generates a system interrupt. The system interrupt is processed by the operating system 210 and/or a device driver for the PMIC 190, which generate a signal on an I/O port of the SoC 110 that is transmitted to the PMIC 190. The signal instructs the PMIC 190 to reduce a voltage supplied to the SoC 110 and/or reduce a frequency of the clock domains utilized by the SoC 110. Once the advertising content has been displayed, the plug-in 230 may cause the mobile device 100 to return to the normal operating mode while playback of the video content is completed.
  • In another embodiment, in the low-power mode, the plug-in 230 may reduce the resolution of the video content by configuring the GPU 130 to render the frames of video marked as advertising content at a lower resolution. For example, the video content rendered by the GPU 130 and displayed on the display device 180 may be rendered at a first resolution such as 1920 × 1080 pixels. The GPU 130 may be configured to render the video content 310 to a frame buffer configured according to the first resolution. The plug-in 230 is configured to send a command to the GPU 130 that configures a new frame buffer at a lower resolution, such as 640 × 480 pixels. Each frame of advertising content is then rendered to the new frame buffer at the lower resolution. In addition, the GPU 130 may be configured to utilize lower-quality rendering techniques. For example, the GPU 130 may be configured to render the video content 310 into the high resolution frame buffer using MSAA or SSAA techniques in the normal mode, but GPU 130 may be re-configured to render the advertising content into the low-resolution frame buffer using a single sample per pixel in the low-power mode. The effect of reducing the resolution of the video content during display of the advertising content is that fewer pixels and/or samples per pixel will need to be rendered by the GPU 130, which can drastically reduce the time to render each frame by the GPU 130. Power saving modes, such as clock gating the GPU 130 within the SoC 110 when the GPU 130 is inactive between frames, can save significant amounts of power. The quality of the images displayed for the advertising content may be much less than the quality of the images displayed for the video content, but the viewer will typically not be adverse to the reduction in quality when the reduction extends the life of the battery in the mobile device 100.
  • In yet another embodiment, in the low-power mode, the plug-in 230 may reduce a brightness level of a display device 180 included in the mobile device 100. In one embodiment, the plug-in 230 is configured to cause the mobile device 100 to enter the low-power mode by calling an application programming interface (API) method in response to detecting the display of the advertising content on the mobile device. The plug-in 230, executing in the user space, calls the API method, which generates a system interrupt. The system interrupt is processed by the operating system 210 and/or a device driver for the display device 180, which generate a signal on an I/O port of the SoC 110 that is transmitted to the display device 180. The signal instructs a backlight controller of the display device 180 to reduce the brightness of the display device 180. The brightness of the display device 180 may, therefore, be reduced by, e.g., 50% while advertising content is presented on the display device 180 during playback of the video content. Again, this reduction in brightness may be very noticeable to a viewer while advertising content is being played, but the viewer will typically not be adverse to the reduction in brightness when the reduction extends the life of the battery in the mobile device 100. Reducing the brightness of the backlight may save significant power consumed by the display device 180 while advertising content is being played in a full-screen mode.
  • In some embodiments, the plug-in 230 may be configured to combine two or more of the techniques described above when operating the mobile device 100 in the low-power mode. For example, the plug-in 230 may configure the device to use a lower voltage and/or frequency utilizing DVFS as well as reducing the resolution of the advertising content rendered for display and reducing a brightness of the screen during playback of the advertising content. Combinations of these power-saving techniques may be used to further reduce the power consumption of the mobile device 100 in the low-power mode.
  • Once the advertising content has completed playback, then the plug-in 230 is configured to return the mobile device 100 to normal mode. The plug-in 230 may generate a system call via the API 220 that is processed by the operating system and/or device driver of the PMIC 190 to transmit a signal to the PMIC 190 to increase the voltage and/or frequency of the clock domains of the SoC 110 to the higher voltage and higher frequency. The plug-in 230 may also configure the GPU 130 to render the video content 310 at the higher resolution by reconfiguring the frame buffer, and increase the brightness of the screen by generating a system call via the API 220 that is processed by the operating system and/or device driver of the display device 180 to transmit a signal to the backlight controller of the display device 180.
  • The video content 310 is not always displayed in full-screen mode. Some video content 310 may be played back in an embedded mode, where the video content 310 is viewed within the extents of an element of a web page. The web page may also include other elements such as text, a menu, hyperlinks, etc. Some of the other elements may also include unwanted content 320, such as a banner advertisement that is an image that is typically associated with a hyperlink. When the video content 310 is played back in the embedded mode, the plug-in 230 is configured to mask the unwanted content 320 such as by rendering only the video content 310 over a black background, as shown in Figure 3C . Masking the other elements of the web page effectively prevents the unwanted content 320 from being displayed. In addition, the plug-in 230 may configure the SoC 110 to throttle one or more threads associated with the unwanted content 320. For example, some unwanted content 320 may include an animation or video embedded in the web page as a second element separate from the video content 310. Separate threads may be allocated to render the video content 310 and the unwanted content 320. When the user begins playback of the video content 310, the plug-in 230 is configured to deactivate (i.e., throttle) the thread associated with the unwanted content 320, while the thread associated with the video content 310 remains active. Other threads may be throttled as well, such as threads related to other elements of the web page.
  • In some embodiments, the video content 310 playing in the embedded mode also has advertising content included therein. In other words, a subset of frames included in the video content 310 are identified as advertising content as well as the unwanted content 320 displayed outside of the element where the video content 310 is rendered. In addition to implementing the techniques of masking and thread throttling described above to suppress rendering of unwanted content 320 that is separate from the video content 310 in a low-power mode, the plug-in 230 may be configured to implement the techniques described above (DVFS, reduced resolution, and reduced brightness) when the advertising content included within the video content 310 is displayed.
  • In one embodiment, content associated with video content can be assigned different priority levels. For example, advertising content included within a video can be assigned a first priority level, advertising content embedded in a webpage alongside the video content can be assigned a second priority level, and additional elements of the webpage (e.g., navigation menus, links to additional articles, etc.) can be assigned a third priority level. A user may designate a threshold for priority levels that are identified as unwanted content. For example, one user may only want to designate advertising content included in the video content as unwanted content such that the low-power mode is only entered when the advertising content is displayed during playback of the video content. Another user may designate both advertising content included in the video content as well as other advertising content embedded in the webpage as unwanted content. Yet another user may designate advertising content (both included in the video content and embedded in the webpage alongside the video content) as well as additional elements as unwanted content.
  • Figure 4 illustrates video content 310 that includes advertising content 350, in accordance with one embodiment. As shown in Figure 4 , the video content 310 includes a sequence of frames that can be played back by rendering each frame to the display device 180. The video content 310 starts at a time To 401 where a first frame of video content 310 is encoded in a file format for video (e.g., MPEG-4 AVC, AVI, MOV, etc.). The video content 310 includes a number of frames until a time T 1 402, where a first frame of advertising content 350 appears in the sequence of frames. The advertising content 350 may be a 30 second commercial for a product or service. As shown in Figure 4, the commercial is for a company called Burger Joint. The last frame of advertising content 350 in the commercial is located at a time T 2 403. Additional advertising content 350 included in the video content 310, such as a commercial for Car Co., may be located at frames between time T 3 404 and T 4 405, and the end of the video content 310 is at time T 5 406.
  • The plug-in 230 is configured to detect the video content 310 to be viewed within an application of the mobile device 100. The plug-in 230 is also configured to detect the unwanted content associated with the video content 310, such as detecting the advertising content 350 included in the video content 310 by reading markers from the metadata associated with video content 310 that indicates that advertising content 350 is included in the video content 310 between times T 1 402 and T 2 403 as well as between times T 3 404 and T 4 405. Then, during playback of the video content 310, the plug-in 230 is configured to operate the mobile device 100 in normal mode from time T 0 401 to time T 1 402, in low-power mode from time T 1 402 to time T 2 403, in normal mode from time T 2 403 to time T 3 404, in low-power mode from time T 3 404 to time T 4 405, and in normal mode from time T 4 405 to time T 5 406.
  • Figure 5 illustrates a system 500 for detecting unwanted content associated with video content, in accordance with one embodiment. As shown in Figure 5 , J client nodes 502 are coupled to a server node 504 via a network 510. In one embodiment, the network 510 is the Internet and the nodes communicate via the TCP/IP protocol. Each client node 502 may be a mobile device 100, which includes the plug-in 230 and application 240.
  • In one embodiment, the plug-in 230 of the mobile device 100 may transmit feedback information from the client node 502 to the server node 504. The feedback information is stored in a database 520 that correlates a file name for the video content with the user-generated markers that indicate the location of advertising content 350 included within the video content 310. The server node 504 may include software to analyze the feedback from a plurality of client nodes 502 to generate markers for identifying advertising content 350 included in the video content 310. After feedback from a number of users has been received, the server node 504 may analyze all of the feedback information to identify the advertising content 350 in the video content 310.
  • For example, each user may indicate where in the video content 310 advertising content 350 starts and stops. An algorithm executed by the server node 504 may count the number of users that identified each frame of video content 310 as advertising content 350. Any frames with a number of users that identified that frame as advertising content 350 above a threshold number of users are then identified as advertising content 350. The algorithm may then estimate where advertising content 350 starts and stops within the video content 310 based on which frames have been identified by users as advertising content 350. Markers that identify those frames as advertising content 350 may then be generated and correlated to that particular video file.
  • In another embodiment, the server node 504 includes software for performing image analysis on the frames of video content 310. The server node 504 may compare objects in each frame with known objects associated with advertising content 350, such as a number of well-known company trademarks. The server node 504 may also identify keywords from text included in the frames that are associated with advertising content. The analysis may identify certain frames of video as likely advertising content 350. The server node 504 may then estimate where advertising content 350 starts and stops within the video content 310 based on which frames have been identified via image analysis as advertising content 350. Markers that identify those frames as advertising content 350 may then be generated and correlated to that particular video file.
  • In addition to identifying advertising content 350 included within video content 310, the server node 504 may be used to identify unwanted content that is not included within the video content 310. In one embodiment, users may mark specific elements of a markup language document as unwanted content and submit their feedback to the server node 504. The server node 504 may collect user feedback and add the feedback to a database. Once a threshold number of users have identified a particular element in a markup language document as unwanted content, then the server node 504 may mark that element as unwanted content in a database. The server node 504 may then identify that element as unwanted content in response to a request from a client node 502.
  • During playback of the video content, the plug-in 230 may then request information from the server node 504 in order to identify the location of advertising content 350 included in the video content 310 and/or unwanted content associated with the video content 310.
  • Figure 6 is a flowchart of a method 600 for reducing the power consumption of a mobile device, in accordance with one embodiment. At step 602, video content 310 to be viewed in an application 240 executed by a mobile device 100 is detected. In one embodiment, the application is a web browser, and the video content is detected while parsing markup language documents. An element of the markup language document may be identified during parsing the markup language document to identify a multimedia element that includes a pointer to a video file that indicates that video content is going to be embedded within a browser window that is displayed to a user of the mobile device 100. In another embodiment, the application may be a media player application, and the video content is detected when a user requests the application open a particular file having a video format.
  • At step 604, unwanted content associated with the video content 310 is detected. In one embodiment, a plug-in 230 for the application determines whether the video content 310 includes advertising content 350 by requesting such information from a server node 504. In another embodiment, the plug-in 230 reads metadata associated with the video content 310 in order to determine that the video content 310 includes advertising content 350. In yet another embodiment, the plug-in 230 identifies whether the video content 310 is played in a full-screen mode or an embedded mode. If the video content is played in an embedded mode, then unwanted content 320 may be identified by querying a server node 504. In yet another embodiment, unwanted content may be identified as any content rendered outside the extents of the video content 310.
  • At step 606, the mobile device 100 is configured to operate in a low-power mode during playback of the video content in the application in response to detecting the unwanted content. In one embodiment, the plug-in 230 may mask any elements displayed alongside the video content 310 and throttle any threads associated with the unwanted content during playback of the video content 310. In another embodiment, the plug-in 230 may cause the mobile device 100 to operate in the low-power mode when advertising content 350 included in the video content 310 is displayed during playback of the video content 310.
  • Figure 7 illustrates an exemplary system 700 in which the various architecture and/or functionality of the various previous embodiments may be implemented. As shown, a system 700 is provided including at least one central processor 701 that is connected to a communication bus 702. The communication bus 702 may be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol(s). The system 700 also includes a main memory 704. Control logic (software) and data are stored in the main memory 704 which may take the form of random access memory (RAM).
  • The system 700 also includes input devices 712, a graphics processor 706, and a display 708, i.e. a conventional CRT (cathode ray tube), LCD (liquid crystal display), LED (light emitting diode), plasma display or the like. User input may be received from the input devices 712, e.g., keyboard, mouse, touchpad, microphone, and the like. In one embodiment, the graphics processor 706 may include a plurality of shader modules, a rasterization module, etc. Each of the foregoing modules may even be situated on a single semiconductor platform to form a graphics processing unit (GPU).
  • In the present description, a single semiconductor platform may refer to a sole unitary semiconductor-based integrated circuit or chip. It should be noted that the term single semiconductor platform may also refer to multi-chip modules with increased connectivity which simulate on-chip operation, and make substantial improvements over utilizing a conventional central processing unit (CPU) and bus implementation. Of course, the various modules may also be situated separately or in various combinations of semiconductor platforms per the desires of the user.
  • The system 700 may also include a secondary storage 710. The secondary storage 710 includes, for example, a hard disk drive and/or a removable storage drive, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, digital versatile disk (DVD) drive, recording device, universal serial bus (USB) flash memory. The removable storage drive reads from and/or writes to a removable storage unit in a well-known manner.
  • Computer programs, or computer control logic algorithms, may be stored in the main memory 704 and/or the secondary storage 710. Such computer programs, when executed, enable the system 700 to perform various functions. The memory 704, the storage 710, and/or any other storage are possible examples of computer-readable media.
  • In one embodiment, the architecture and/or functionality of the various previous figures may be implemented in the context of the central processor 701, the graphics processor 706, an integrated circuit (not shown) that is capable of at least a portion of the capabilities of both the central processor 701 and the graphics processor 706, a chipset (i.e., a group of integrated circuits designed to work and sold as a unit for performing related functions, etc.), and/or any other integrated circuit for that matter.
  • Still yet, the architecture and/or functionality of the various previous figures may be implemented in the context of a general computer system, a circuit board system, a game console system dedicated for entertainment purposes, an application-specific system, and/or any other desired system. For example, the system 700 may take the form of a desktop computer, laptop computer, server, workstation, game consoles, embedded system, and/or any other type of logic. Still yet, the system 700 may take the form of various other devices including, but not limited to a personal digital assistant (PDA) device, a mobile phone device, a television, etc.
  • Further, while not shown, the system 700 may be coupled to a network (e.g., a telecommunications network, local area network (LAN), wireless network, wide area network (WAN) such as the Internet, peer-to-peer network, cable network, or the like) for communication purposes.
  • In an example embodiment, the system 700 includes a video content detection module detecting video content to be viewed in an application executed by the mobile device, an unwanted content detection module detecting unwanted content associated with the video content, and a low-power module operating the mobile device in a low-power mode during playback of the video content in the application in response to detecting the unwanted content. In some embodiments, the system 700 may include other or additional modules for performing any one of or combination of steps described in the embodiments. Further, any of the additional or alternative embodiments or aspects of the method, as shown in any of the figures or recited in any of the claims, are also contemplated to include similar modules.
  • It is noted that the techniques described herein, in an aspect, are embodied in executable instructions stored in a computer readable medium for use by or in connection with an instruction execution machine, apparatus, or device, such as a computer-based or processor-containing machine, apparatus, or device. It will be appreciated by those skilled in the art that for some embodiments, other types of computer readable media are included which may store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memory (RAM), read-only memory (ROM), and the like.
  • As used here, a "computer-readable medium" includes one or more of any suitable media for storing the executable instructions of a computer program such that the instruction execution machine, system, apparatus, or device may read (or fetch) the instructions from the computer readable medium and execute the instructions for carrying out the described methods. Suitable storage formats include one or more of an electronic, magnetic, optical, and electromagnetic format. A non-exhaustive list of conventional exemplary computer readable medium includes: a portable computer diskette; a RAM; a ROM; an erasable programmable read only memory (EPROM or flash memory); optical storage devices, including a portable compact disc (CD), a portable digital video disc (DVD), a high definition DVD (HD-DVD™), a BLU-RAY disc; and the like.
  • It should be understood that the arrangement of components illustrated in the Figures described are exemplary and that other arrangements are possible. It should also be understood that the various system components (and means) defined by the claims, described below, and illustrated in the various block diagrams represent logical components in some systems configured according to the subject matter disclosed herein.
  • For example, one or more of these system components (and means) may be realized, in whole or in part, by at least some of the components illustrated in the arrangements illustrated in the described Figures. In addition, while at least one of these components are implemented at least partially as an electronic hardware component, and therefore constitutes a machine, the other components may be implemented in software that when included in an execution environment constitutes a machine, hardware, or a combination of software and hardware.
  • More particularly, at least one component defined by the claims is implemented at least partially as an electronic hardware component, such as an instruction execution machine (e.g., a processor-based or processor-containing machine) and/or as specialized circuits or circuitry (e.g., discreet logic gates interconnected to perform a specialized function). Other components may be implemented in software, hardware, or a combination of software and hardware. Moreover, some or all of these other components may be combined, some may be omitted altogether, and additional components may be added while still achieving the functionality described herein. Thus, the subject matter described herein may be embodied in many different variations, and all such variations are contemplated to be within the scope of what is claimed.
  • In the description above, the subject matter is described with reference to acts and symbolic representations of operations that are performed by one or more devices, unless indicated otherwise. As such, it will be understood that such acts and operations, which are at times referred to as being computer-executed, include the manipulation by the processor of data in a structured form. This manipulation transforms the data or maintains it at locations in the memory system of the computer, which reconfigures or otherwise alters the operation of the device in a manner well understood by those skilled in the art. The data is maintained at physical locations of the memory as data structures that have particular properties defined by the format of the data. However, while the subject matter is being described in the foregoing context, it is not meant to be limiting as those of skill in the art will appreciate that various acts and operations described hereinafter may also be implemented in hardware.
  • The computer-readable non-transitory media includes all types of computer-readable media, including magnetic storage media, optical storage media, and solid state storage media and specifically excludes signals. It should be understood that the software can be installed in and sold with the mobile device. Alternatively, the software can be obtained and loaded into the mobile device, including obtaining the software via a disc medium or from any manner of network or distribution system, including, for example, from a server owned by the software creator or from a server not owned but used by the software creator. The software can be stored on a server for distribution over the Internet, for example.
  • To facilitate an understanding of the subject matter described herein, many aspects are described in terms of sequences of actions. At least one of these aspects defined by the claims is performed by an electronic hardware component. For example, it will be recognized that the various actions may be performed by specialized circuits or circuitry, by program instructions being executed by one or more processors, or by a combination of both. The description herein of any sequence of actions is not intended to imply that the specific order described for performing that sequence must be followed. All methods described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context.
  • The use of the terms "a" and "an" and "the" and similar referents in the context of describing the subject matter (particularly in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein. The use of any and all examples, or exemplary language (e.g., "such as") provided herein, is intended merely to better illustrate the subject matter and does not pose a limitation on the scope of the subject matter unless otherwise claimed. The use of the term "based on" and other like phrases indicating a condition for bringing about a result, both in the claims and in the written description, is not intended to foreclose any other conditions that bring about that result. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the invention as claimed.

Claims (11)

  1. A computer-implemented method for reducing power consumption of a mobile device, the method comprising:
    • detecting video content to be viewed in an application executed by the mobile device (step 602);
    • detecting unwanted content associated with the video content (step 604),
    wherein the detecting the unwanted content comprises at least one of:
    ∘ identifying markers included in metadata associated with the video content, wherein the markers specify explicit start points and end points of advertising content included in the video content;
    ∘ utilizing user feedback to identify explicit start points and end points of advertising content included in the video content; or
    ∘ analyzing the video content to identify keywords or images corresponding to advertising content; and
    • operating the mobile device in a low-power mode during playback of the video content in the application in response to detecting the unwanted content,
    wherein the operating the mobile device in the low-power mode comprises at least one of using Dynamic Voltage and Frequency Scaling, DVFS, to adjust power consumption of a processor (701) included in the mobile device, reducing the resolution of a display device (708) included in the mobile device, or reducing a brightness level of the display device (step 606).
  2. The method of claim 1, wherein the unwanted content includes advertising content included in the video content as one or more frames of video.
  3. The method of any of claims 1 to 2, wherein the unwanted content is embedded within a markup language document displayed in a web browser, and wherein the video content is displayed next to the unwanted content.
  4. The method of any of claims 1 to 3, wherein the video content is displayed in an embedded mode, and wherein operating the mobile device in the low-power mode comprises at least one of masking the unwanted content or throttling one or more threads associated with the unwanted content.
  5. The method of any of claims 1 to 4, wherein the application includes a plug-in configured to call an application programming interface , API method in response to detecting the unwanted content to cause the mobile device to enter the low-power mode.
  6. The method of any of claims 1 to 5, wherein detecting video content comprises parsing a markup language document to identify a multimedia element that includes a pointer to a video file.
  7. A mobile device, comprising:
    • a non-transitory memory storage (704) comprising instructions; and one or more processors (701) in communication with the memory (704), wherein the one or more processors (701) execute the instructions to:
    • detect video content to be viewed in an application;
    • detect unwanted content associated with the video content,
    wherein the detecting the unwanted content comprises at least one of:
    ∘ identifying markers included in metadata associated with the video content, wherein the markers specify explicit start points and end points of advertising content included in the video content;
    ∘ utilizing user feedback to identify explicit start points and end points of advertising content included in the video content; or
    ∘ analyzing the video content to identify keywords or images corresponding to advertising content; and
    • operate the mobile device in a low-power mode during playback of the video content in the application in response to detecting the unwanted content,
    wherein the operating the mobile device in the low-power mode comprises at least one of using Dynamic Voltage and Frequency Scaling, DVFS, to adjust power consumption of a processor (701) included in the mobile device, reducing the resolution of a display device (708) included in the mobile device, or reducing a brightness level of the display device (step 606).
  8. The mobile device of claim 7, wherein the unwanted content includes advertising content included in the video content as one or more frames of video.
  9. The mobile device of any of claims 7 to 8, wherein the unwanted content is embedded within a markup language document displayed in a web browser, and wherein the video content is displayed next to the unwanted content.
  10. The mobile device of any of claims 7 to 9, wherein the video content is displayed in an embedded mode, and wherein operating the mobile device in the low-power mode comprises at least one of masking the unwanted content or throttling one or more threads associated with the unwanted content.
  11. The mobile device of any of claims 7 to 10, wherein the application includes a plug-in configured to call an application programming interface , API method in response to detecting the unwanted content to cause the mobile device to enter the low-power mode.
EP18748711.1A 2017-02-02 2018-01-25 Content-aware energy savings for video streaming and playback on mobile devices Active EP3571837B1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US15/423,490 US10117185B1 (en) 2017-02-02 2017-02-02 Content-aware energy savings for video streaming and playback on mobile devices
PCT/CN2018/074192 WO2018141228A1 (en) 2017-02-02 2018-01-25 Content-aware energy savings for video streaming and playback on mobile devices

Publications (3)

Publication Number Publication Date
EP3571837A1 EP3571837A1 (en) 2019-11-27
EP3571837A4 EP3571837A4 (en) 2019-12-11
EP3571837B1 true EP3571837B1 (en) 2021-10-27

Family

ID=63040241

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18748711.1A Active EP3571837B1 (en) 2017-02-02 2018-01-25 Content-aware energy savings for video streaming and playback on mobile devices

Country Status (4)

Country Link
US (1) US10117185B1 (en)
EP (1) EP3571837B1 (en)
CN (1) CN110268718B (en)
WO (1) WO2018141228A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023214642A1 (en) * 2022-05-02 2023-11-09 Samsung Electronics Co., Ltd. Method and system for detecting private video

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101777660B1 (en) * 2016-10-25 2017-09-12 주식회사 티에스피글로벌 The flash storage device and operation control method thereof
DE112018002109T5 (en) 2017-04-21 2020-01-09 Zenimax Media, Inc. SYSTEMS AND METHODS FOR CODING GUIDED ADAPTIVE QUALITY RENDERING
US11271071B2 (en) * 2019-11-15 2022-03-08 Nuvia, Inc. Integrated system with power management integrated circuit having on-chip thin film inductors
CN111443888B (en) * 2020-03-27 2024-03-22 Tcl华星光电技术有限公司 Display control method, display control device, electronic equipment and storage medium

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100960342B1 (en) 2002-05-09 2010-05-28 톰슨 라이센싱 Content identification in a digital video recorder
KR20050056639A (en) * 2003-12-10 2005-06-16 주식회사 대우일렉트로닉스 Device for controlling screen power in the power-saving television
US20070242926A1 (en) 2006-04-13 2007-10-18 Huang Chen-Hsiu Video Program Processing Method and System
US8521892B2 (en) * 2008-02-29 2013-08-27 Red Hat, Inc. Method and apparatus for controlling web page advertisement through incentives and restrictions
US8249423B2 (en) * 2008-05-08 2012-08-21 Sony Ericsson Mobile Communications Ab Electronic devices and methods that insert addressable chapter marks relative to advertising content in video streams
US20100035656A1 (en) * 2008-08-11 2010-02-11 Yang Pan Delivering Advertisement Messages to a User by the Use of Idle Screens of Mobile Devices with Integrated Sensors
US20100036734A1 (en) * 2008-08-11 2010-02-11 Yang Pan Delivering Advertisement Messages to a User by the Use of Idle Screens of Electronic Devices
CN101996313A (en) 2009-08-18 2011-03-30 宏碁股份有限公司 System and method for detecting advertisement
CN101980531A (en) 2010-10-25 2011-02-23 中兴通讯股份有限公司 Method for detecting video of mobile terminal and mobile terminal
CN103747288A (en) 2013-11-27 2014-04-23 乐视致新电子科技(天津)有限公司 Automatic mode switching television apparatus and control method thereof
US20170221109A1 (en) * 2016-01-29 2017-08-03 Qualcomm Incorporated Ads management in a browser application

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023214642A1 (en) * 2022-05-02 2023-11-09 Samsung Electronics Co., Ltd. Method and system for detecting private video

Also Published As

Publication number Publication date
CN110268718A (en) 2019-09-20
US10117185B1 (en) 2018-10-30
WO2018141228A1 (en) 2018-08-09
EP3571837A4 (en) 2019-12-11
EP3571837A1 (en) 2019-11-27
CN110268718B (en) 2021-01-05

Similar Documents

Publication Publication Date Title
EP3571837B1 (en) Content-aware energy savings for video streaming and playback on mobile devices
EP3566154B1 (en) Content-aware energy savings for web browsing utilizing selective loading priority
CN107748686B (en) Application program starting optimization method and device, storage medium and intelligent terminal
US10740945B2 (en) Animation control methods and systems
US11269953B2 (en) Server-based conversion of autoplay content to click-to-play content
EP2521971B1 (en) Policy-based switching between graphics-processing units
US8621253B1 (en) Processor boost based on user interface demand
KR101973045B1 (en) Management of the channel bar
US20120144288A1 (en) Web page content display priority and bandwidth management
JP5335830B2 (en) Display information conversion system and apparatus for power saving and context display
RU2615224C2 (en) Methods and systems of dynamic control of display device brightness basing on content
US20180129534A1 (en) Thread Importance Based Processor Core Partitioning
US9591366B2 (en) Multimedia providing system and multimedia providing method
JP6212447B2 (en) Movie reproduction control apparatus, movie reproduction control method and program
CN112825539B (en) Hardware management method and associated machine readable medium and electronic device
TWI774053B (en) Hardware management methods and associated machine-readable medium, electronic devices
US8610828B2 (en) Moving picture reproduction and display frequency change, an apparatus, method, and program
US20230082747A1 (en) System and method for improving graphical user interface rendering

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20190819

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

RIN1 Information on inventor provided before grant (corrected)

Inventor name: WANG, JUN

Inventor name: HU, ZIANG

Inventor name: YE, HANDONG

Inventor name: RAO, KARTHIK

Inventor name: LIN, ZONGFANG

Inventor name: TIAN, CHEN

A4 Supplementary search report drawn up and despatched

Effective date: 20191111

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 21/41 20110101ALI20191105BHEP

Ipc: H04N 21/44 20110101ALI20191105BHEP

Ipc: H04N 5/445 20110101AFI20191105BHEP

RIN1 Information on inventor provided before grant (corrected)

Inventor name: WANG, JUN

Inventor name: TIAN, CHEN

Inventor name: HU, ZIANG

Inventor name: YE, HANDONG

Inventor name: RAO, KARTHIK

Inventor name: LIN, ZONGFANG

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20210607

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1442941

Country of ref document: AT

Kind code of ref document: T

Effective date: 20211115

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602018025730

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20211027

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1442941

Country of ref document: AT

Kind code of ref document: T

Effective date: 20211027

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220127

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220227

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220228

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220127

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220128

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602018025730

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20220131

26N No opposition filed

Effective date: 20220728

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220125

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220131

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220131

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220131

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220125

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211027

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20221130

Year of fee payment: 6

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20231207

Year of fee payment: 7

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20231212

Year of fee payment: 7