WO2020128625A1 - Method for operating an electronic device when playing an audiovisual content - Google Patents

Method for operating an electronic device when playing an audiovisual content Download PDF

Info

Publication number
WO2020128625A1
WO2020128625A1 PCT/IB2019/001369 IB2019001369W WO2020128625A1 WO 2020128625 A1 WO2020128625 A1 WO 2020128625A1 IB 2019001369 W IB2019001369 W IB 2019001369W WO 2020128625 A1 WO2020128625 A1 WO 2020128625A1
Authority
WO
WIPO (PCT)
Prior art keywords
video
output module
audiovisual content
user
quality level
Prior art date
Application number
PCT/IB2019/001369
Other languages
French (fr)
Inventor
Shi JIAO
Zhihong Guo
Original Assignee
Orange
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Orange filed Critical Orange
Publication of WO2020128625A1 publication Critical patent/WO2020128625A1/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
    • H04N21/44213—Monitoring of end-user related data
    • H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
    • H04N21/462—Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
    • H04N21/4621—Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/637—Control signals issued by the client directed to the server or network components
    • H04N21/6377—Control signals issued by the client directed to the server or network components directed to server
    • H04N21/6379—Control signals issued by the client directed to the server or network components directed to server directed to encoder, e.g. for requesting a lower encoding rate

Definitions

  • the field of this invention is that of mobile phones playing audiovisual components. More particularly, the invention relates to a method for operating a device when playing an audiovisual content.
  • Recent mobile phones offer numerous features, including GPS, internet, music, video playing, games and countless apps. Lots of users regularly play various videos on their smartphone.
  • the problem is that video playback costs sensibly more power and more data traffic than audio. Indeed, the video stream is heavier than the audio stream. When the user does not look at the screen, the data/power used for playing the video stream is wasted.
  • the present invention provides a method for operating an electronic device comprising a processing unit, a video output module and an audio output module, the method being characterized in that it comprises:
  • the electronic device comprises a camera, further comprising acquiring (a1 ) images using the camera, the determining step (a) being performed based on the acquired images;
  • the determining step (a) further comprises analyzing (a2) the acquired images so as to detect a face of the user, the determining step (a) being performed based on the result of the detection of a face of the user;
  • the determining step (a) further comprises analyzing (a2) the acquired images so as to detect an eye of the user, the determining step (a) being performed based on the result of the detection of an eye of the user;
  • the determining step (a) is performed based on the result of the detection of an open eye of the user in said at least one acquired image, the user being determined as paying attention to the video output module (13) if an open eye is detected in said at least one acquired image;
  • the video component of the audiovisual content with the first quality level and the audio component of the audiovisual content are transmitted from a remote server to the electronic device, and wherein interrupting (b) the output of the video component of the audiovisual content with the first quality level comprising interrupting the transmission of the video component of the audiovisual content with the first quality level; • the method further comprises outputting (b1 ) the video component of the audiovisual content with a second quality level at the video output module, said second quality level being lower than the first quality level;
  • • outputting (b1 ) the video component of the audiovisual content with a second quality level at the video output module comprises triggering the transmission, from the remote server, of the video component of the audiovisual content with the second quality level;
  • the video component of the audiovisual content with the second quality level has a lower resolution and/or a reduced number of frames by seconds than the video component of the audiovisual content with the first quality level;
  • the method further comprises, in response to determining that the user has not been paying attention to the video output module for a duration exceeding an inattention threshold, interrupting (b2) the output of the video component of the audiovisual content with the second quality level;
  • the method further comprises determining, by the processing unit, if the user is paying again attention to the video output module of the audiovisual content; and, in response to determining that the user is paying again attention to the video output module, outputting (c) the video component of the audiovisual content with the first quality level.
  • the invention provides an electronic device comprising:
  • a video output module for outputting a video component of an audiovisual content with a first quality level
  • an audio output module for outputting an audio component of the audiovisual content at the audio output module
  • processing unit is configured to determine (a) whether a user of the electronic device is paying attention to the video output module and in response to determining that the user is not paying attention to the video output module, interrupt (b) the output of the video component of the audiovisual content with the first quality level.
  • the video output module is a screen
  • the electronic device comprises further, next to the screen, a front camera for acquiring images to be used by the processing unit to determine whether the user is paying attention to the video output module;
  • the electronic device comprises further at least one of a proximity sensor, an accelerometer, an infrared detector and a motion sensor, the output of which is used by the processing unit to determine whether the user is paying attention to the video output module.
  • the invention provides a computer program product, comprising code instructions for executing a method according to the first aspect for operating a device comprising at least one processing unit, a video output module and an audio output module; and a computer-readable medium, on which is stored a computer program product comprising code instructions for executing a method according to the first aspect for operating a device comprising at least one processing unit, a video output module and an audio output module.
  • FIG. 1 illustrates an example of architecture in which is performed the method according to the invention.
  • FIG. 2 is a diagram representing steps of an embodiment of a method according to the invention.
  • the present invention relates to a method for operating an electronic device 1 as represented by figure 1 , in particular for playing an audiovisual content.
  • the“video” it will be meant only the visual content, in other words the“moving picture” without the sound.
  • audio it will be meant only the sound content, without the video.
  • by“audiovisual content” it will be meant a content possessing both an audio component and a video component, i.e. an AV media.
  • An audiovisual content is often inappropriately referred to as a“video”, despite having an audio component.
  • the electronic device 1 comprises one or more processing unit(s) 1 1 , a video output module 13 (for instance a built-in screen, acting as the default video output unit of the device 1 ), and an audio output module 14 (for instance a built-in speaker, acting as the default audio output unit of the device 1 ). It is to be noted that the electronic device 1 could alternately comprise as audio and/or video output module 13,
  • video and/or audio connector toward an external device such as a TV, a projector, etc.
  • an external device such as a TV, a projector, etc.
  • video and audio such as a HDMI port
  • video-only such as a jack for earphones
  • audio and/or video connector could be virtual, for instance a Bluetooth connection to a VR headset.
  • the electronic device 1 further comprises a camera
  • the electronic device 1 typically further comprises a communication unit for connecting (in particular wirelessly) the electronic device 1 to a network 20 (for example WiFi, Bluetooth, and preferably a mobile network, in particular a GSM/UMTS/LTE network), a memory 12 (for example flash memory), in particular for storing applications, which can be of various types.
  • a network 20 for example WiFi, Bluetooth, and preferably a mobile network, in particular a GSM/UMTS/LTE network
  • a memory 12 for example flash memory
  • applications which can be of various types.
  • the processing unit 1 1 is able to run these applications, and one or more of them could be players, i.e. applications able to retrieve an audiovisual content from a remote server 2 of the network, and play it.
  • the electronic device 1 may further comprise others units such as a battery, a user interface (for instance, if it comprises a screen as video output module 13, this screen could be touch sensitive), a location unit for providing location data representative of the position of the electronic device 1 (using for example GPS, network triangulation), etc.
  • the electronic device 1 is typically a smartphone, a tablet computer, a laptop, etc. In the following description, the example of a smartphone will be used.
  • the present method is performed mostly by the processing unit 1 1 of the electronic device 1 , and is advantageously implemented by the operating system of this electronic device 1 , or a dedicated application such as an audio/video player.
  • the present method starts with a step (aO) of playing an audiovisual content, which is typically obtained from a remote server 2, i.e. downloaded via the network 20.
  • the audiovisual content is preferably streamed from such a remote server 2.
  • the present invention may also apply to downloaded audiovisual content which is not streamed, i.e. audiovisual content which is played after having been completely downloaded by the electronic device 1 .
  • such audiovisual content has a video component and an audio component, so that playing the audiovisual content includes outputting the video component of the audiovisual content at the video output module 13 and outputting an audio component of the audiovisual content at the audio output module 14.
  • versions of the video component might be available, corresponding to different qualities. For example, there might be a plurality of resolutions (generally six resolutions: 144p, 240p, 360p, 480p, 720p or 1080p), and/or several numbers of frames by seconds (generally 30 fps or 60 fps).
  • resolutions generally six resolutions: 144p, 240p, 360p, 480p, 720p or 1080p
  • numbers of frames by seconds generally 30 fps or 60 fps.
  • versions of the audio component might be available, even if it is rarer than for the video component (indeed the video component is far heavier than the audio component).
  • the user selects the quality he wishes, or it is automatically selected thanks to the Adaptive bitrate (ABR) streaming technique.
  • ABR Adaptive bitrate
  • Such an adaptive bitrate streaming works by detecting a user's bandwidth and CPU capacity in real time and adjusting the quality of the component accordingly: the quality selected shall be the highest possible without degrading the user experience (i.e. with very little buffering, fast start time, etc.).
  • the Real Time Streaming Protocol is preferably used: when the user wishes to play the audiovisual content, a RSTP request is sent from the electronic device 1 to the remote server 2, and the content with the “optimal” video and audio components is received in return.
  • a first quality level of video and audio components as the quality level of the components as normally outputted at the electronic device 1 during step (aO), even if such quality could vary over time thanks to the ABR because of some events: for example, should a network congestion happens, the current quality will be temporary reduced (with respect to the quality when starting playing) to avoid uncomfortable rebuffering, but the provided components will always be the“optimal” ones given the available resources.
  • the present method comprises a determining step (a) of determining whether a user is paying attention to the video output module 13.
  • This determining step (a) can be continuously performed after having initiated (step aO) the playing of the audiovisual content by the electronic device 1 . More precisely, playing an audiovisual content (step (a0)) triggers the determining step (a). To rephrase again, the step (a) is advantageously performed only during the playing of audiovisual contents by the electronic device 1 , so as to save battery of this electronic device.
  • the principle behind the present invention is that, if the user is not paying attention to the video output module 13, the video component could be“lightened”, or even cancelled, without altering the user experience: (s)he will not even realize it, while the network bandwidth usage and/or electronic device power consumption are reduced.
  • the present method proposes to assess the attention of the user to the video output module 13, i.e. to detect whether or not the user is looking at the video, and only listening to the audio component.
  • a check is performed at a given frequency (for instance every minutes), and the user is determined as not paying attention if the user has not looked at the video output module 13 for a duration exceeding a first inattention threshold (for example 5 minutes). Indeed, the user might be distracted during a short moment without this meaning that (s)he is only listening to the audio component.
  • a first inattention threshold for example 5 minutes
  • Determining step (a) is typically a step of“observing” the user.
  • the present method comprises (a1 ) acquiring images using the camera 15 (in particular at a given frequency such as 1 image each minute), the recorded images being used by the processing unit 1 1 to determine whether the user is paying attention to the video output module 13.
  • the acquired images are analyzed by the processing unit 1 1 (step a2) so as to detect a face of the user, the determining step (a) being performed based on the result of the detection of a face of the user. Indeed, from the position of the face, it can be determined what the user is watching, and thus whether (s)he is staring at the video output module 13 or not.
  • Such a so-called“face detection” embodiment based on analyzing an acquired image in order to detect the face of the user, for determining whether the user pays attention to the video output module or not, is advantageous when compared to another embodiment where the electrical device 1 would simply detect if a user is present in a specific area in front of the video output module (for instance using an infrared sensor): in the last case, the electronic device 1 is not able to infer for sure that the user is indeed paying attention to the video output module, as the user could be looking away from the electronic device 1 while still being detected in front of the electronic device 1 . With such a detection of the face of the user from acquired image(s), the determination is quite easy to implement while still more accurate than detecting the presence of a user in front of the electronic device 1 .
  • the acquired images are analyzed by the processing unit 1 1 (step a2) so as to detect an eye of the user, the determining step (a) being performed based on the result of the detection of an eye of the user. Indeed, from the detection of an eye facing the video output module 13, it can be determined that the user is watching, in particular when this eye is detected as being open, and thus whether (s)he is staring at the video output module 13 or not.
  • Such an embodiment based on analyzing an acquired image in order to detect an (open) eye of the user, for determining whether the user pays attention to the video output module or not, is advantageous when compared to the previous embodiments based on presence detection or face detection: in these last cases, the electronic device 1 is not able to discriminate for sure between an awaken user watching the video and a sleeping user merely facing electronic device 1 . With the detection of an eye of the user facing the video output module 13, and in particular the detection of an open eye facing this video output module 13, such a determination can be made even more accurate.
  • the video output module 13 is a screen
  • the camera 15 is a front camera next to the screen (generally just above the screen).
  • what the camera 15 sees is what sees the screen: if an eye of the user is visible, it means that the user can pay attention to the screen.
  • the face if the face is not visible it means that either the screen is concealed (and not visible) or that the user is staring at something else, so that the user cannot be paying attention to the video output module 13.
  • the user is paying intention to the video output module simply when its face or its eye is detectable on the acquired images, which is easy to implement.
  • the determination of whether the user of the electronic device 1 is paying attention to the video output module 13 is not limited to the analysis of acquired images, and other techniques could be used, as for instance :
  • a proximity sensor of the electronic device 1 if the proximity sensor detects a nearby surface it means that the screen is not visible - for instance the device 1 is put down on a table -and thus that the user is not paying attention),
  • an infrared detector which can detect if there are people in front of the electronic device 1 or not
  • - a motion sensor which can detect if there are people in front of the electronic device 1 by motion detection.
  • the output of the video component of the audiovisual content with the first quality level is interrupted. It is to be understood that the output of the audio component with the first quality level is not interrupted, so that the playing of the audiovisual content continues.
  • the processing unit 1 1 may instruct this media player application to stop outputting the video, or to replace the video component with a still image (for instance a black screen).
  • this application directly stops the video output.
  • a software application such as the operating system of the electronic device 1 , may replace the video output of the video component with the locked screen, i.e. may trigger the locking of the screen, while the audio output continues.
  • the method further comprises outputting (b1 ) the video component of the audiovisual content with a second quality level at the video output module 13, said second quality level being lower than the first quality level.
  • the video component of the audiovisual content with a second quality level at the video output module 13, said second quality level being lower than the first quality level.
  • the processing unit 1 1 may instruct this media player application to replace the initial audiovisual component comprising a video component with a first quality level with a corresponding audiovisual content comprising a video component with a second quality level, for instance by instructing this media player to read a file storing such a video component with a second quality level instead of reading a file storing the video component with the first quality level.
  • Such files with different video quality levels may be stored locally, in the electronic device 1 , after having been downloaded from a remote server 2, or may be stored remotely, in the remote server 2, in which case they may be streamed from this remote server 2.
  • this software application may directly replace the video output with the first quality level by the video output with a second quality level, for instead by reading a file storing the video component with the second quality level.
  • the method does not further comprise outputting any video component of the audiovisual content (i.e. whatever the quality) at the video output module 13.
  • the audiovisual content is further played without outputting any video component after interrupting (b) the output of the video component of the audiovisual content with the first quality level. This time, this is a“total degradation” of the video component (i.e. no more video component is outputted), which allows to save even more in terms of power consumption of the electronic device 1
  • the audio component is not degraded.
  • the audio component keeps being outputted with the same quality level as it was before that the video component was degraded.
  • a“total degradation” of the video component it is to be understood that the playing of the audiovisual content becomes“audio-only”.
  • the processing unit 1 1 may instruct this media player application to replace the initial audiovisual content with a video component with a corresponding audiovisual content having only an audio component, for instance by instructing this media player to read a file storing such an audiovisual content with only audio component instead of reading a file storing the full audiovisual content, i.e. with a video component.
  • corresponding files, with or without video component may be stored locally, in the electronic device 1 , after having been downloaded from a remote server 2, or may be stored remotely, in the remote server 2, in which case they may be streamed from this remote server 2.
  • step (b) preferably comprises (in response to determining that the user is not paying attention to the video output module 13) interrupting the transmission of the video component of the audiovisual content with the first quality level, so as to also reduce bandwidth consumption in the network.
  • This interruption can be achieved typically by sending, from the electronic device 1 to the remote server 2, a RTSP request to stop streaming the audiovisual content.
  • Other protocols can naturally be used for such requests to stop streaming, for instance the HTTP Live Streaming (HLS) protocol.
  • the invention is not limited to interrupting both, leading to reduction of the power consumption of the electronic device 1 but not reduction of the bandwidth consumption on the network side, with a very simple implementation on the electronic device.
  • the interruption of the video component transmission could be either partial of total (i.e. the video component is no more transmitted at all).
  • the method preferably comprises triggering the transmission, from the remote server 2, of the video component of the audiovisual content with the second quality level.
  • the second quality level could correspond to a video resolution of 480p.
  • a new RSTP request could be sent to the remote server 2, in order to trigger the transmission of the video component with the second quality level.
  • Other protocols can naturally be used for such requests to stop streaming, for instance the HTTP Live Streaming (HLS) protocol.
  • the application preferably records the progress of the user watching the video with the first quality level, so as to switch to the second quality level from current progress when receiving the video component with the second quality level in response to the request: the playing is not interrupted and the users does not even notice that the video component has been switched or even interrupted.
  • the output of the video component is not interrupted (it is kept outputted with the first quality level), i.e. the playing of the audiovisual content continues as such, with the same quality level as initially provided.
  • the determining step (a) could be performed during the whole audiovisual content playing, even after that the output of the video component has been interrupted.
  • the video component of the audiovisual content may be outputted again with the first quality level.
  • the video component is restored to its initial quality, i.e. the video component is no more degraded.
  • the determination that the user is paying again attention to the video output module 13 is preferably instantaneous: as soon as it is detected that the user is looking at the screen, (s)he is determined as paying again attention (there is no need to wait for the duration of an inattention threshold, contrary to the determination of a user not paying attention to the video output module) and the output of the video component with the first quality level may be resumed immediately.
  • the electronic device 1 also triggers the resuming of the transmission, from the remote server 2, of the video component with the first quality level.
  • the application preferably records the progress of the user watching the video with the second quality level, so as to switch to the first quality level from current progress when receiving again the video component with the first quality level in response to a request to resume to the first quality level: the playing is not interrupted and the users does not even notice that the video component has been switched or even interrupted.
  • the method of the present invention can go on as long as the audiovisual content is played (i.e. as long as the user does not voluntarily stop its playing) : after having determined that the user is paying again attention to the video output module, if the user is again determined as not paying attention to the video output module 13, step (b) may be performed again (by interrupting the output of the video component with the first quality level), and so on, for the whole duration of the audiovisual content playing.
  • the method when the video component of the audiovisual content is outputted with the second quality level, and when the user is still determined as not paying attention to the video output module 13 for a duration exceeding a second inattention threshold (longer that the first inattention threshold), the method further comprises a step (b2) of interrupting the output of the video component of the audiovisual content with the second quality level.
  • step (b2) if the video component of the audiovisual content with the second quality level is transmitted from a remote server 2 to the electronic device 1 , step (b2) preferably comprises the interruption of this transmission.
  • the method could further comprises outputting the video component of the audiovisual content with a third quality level at the video output module 13, then possibly when the user is still determined as not paying attention to the video output module 13 for a duration exceeding a third inattention threshold (longer that the second inattention threshold) interrupting the output of the video component of the audiovisual content with the third quality level, then outputting the video component of the audiovisual content with a fourth quality level at the video output module 13, etc.
  • a third inattention threshold longer that the second inattention threshold
  • a “last” quality level could be defined as the absence of further output of any video component (the so-called “total degradation” mentioned before).
  • the quality is reduced, i.e. changed from the n-th level to the n+1-th level.
  • the quality keeps draping, in particular at a given rate, for example 1 level per more minute (i.e. the n+ 1-th inattention threshold is one minute longer than the n-th inattention threshold).
  • a given rate for example 1 level per more minute (i.e. the n+ 1-th inattention threshold is one minute longer than the n-th inattention threshold).
  • step (c) when step (c) is performed (i.e. in response to determining that the user is paying again attention to the video output module 13), the quality level is directly set back to the first level (no degradation at all).
  • the electronic device 1 is a mobile terminal having a screen lock functionality, where the screen has been locked following the determination that the user was not paying attention to the video output module
  • this can be achieved by the software application (typically the operating system of the mobile terminal) which triggers the unlocking of the screen, thereby leading to outputting again the video component with the first quality level on the video output module 13.
  • the present invention concerns an electronic device 1 comprising at least one processing unit 1 1 , a video output module 13 (in particular a screen) and an audio output module 14 (in particular a speaker or an audio jack for connecting an external speaker or headset), wherein the processing unit 11 is adapted for carrying out the method for operating as previously described.
  • the video and audio output modules 13, 14 are initially respectively for outputting a video and an audio component of an audiovisual content with a first quality level.
  • This electronic device 1 may also comprise a memory 12, a camera 15.
  • the electronic device 1 is typically a smartphone.
  • the electronic device can also comprise a proximity sensor, an accelerometer, an infrared detector and/or a motion sensor (not illustrated), the output of which can be used by the processing unit 1 1 to determine if a user pays attention to the smartphone.
  • the aforementioned processing unit 1 1 is in particular configured to:
  • the present method determines (a) whether a user of the electronic device 1 is paying attention to the video output module 13 (in particular by analyzing images acquired by the camera 15 so as to detect a face or an eye of the user and/or the output of any combination of a proximity sensor, an accelerometer, infrared detector and/or motion sensor); - in response to determining that the user is not paying attention to the video output module 13, interrupt (b) the output of the video component of the audiovisual content with the first quality level (preferably by outputting the video component with a second quality level or by not outputting the video component); - and possibly, in response to determining that the user is paying again attention to the video output module 13, re-outputting the video component of the audiovisual content with the first quality level (i.e. reverting to the video component as initially outputted at the video output module 13).
  • the present method is preferably implemented by an application such as an audio/video player.
  • the video output module 13 is a screen
  • the electronic device 1 comprises further, next to the screen, a front camera 15 for acquiring images to be used by the processing unit 13 to determine whether the user is paying attention to the video output module 13.
  • the invention further proposes a computer program product, comprising code instructions for executing (in particular with a processing unit 11 of the device 1 ) the previously described method, as well as a computer-readable medium (in particular a memory 12 of the device 1 ), on which is stored a computer program product comprising code instructions for executing said method.
  • this computer program product may correspond to the operating system, and especially the media controller.

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Social Psychology (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

The present invention relates to a method for operating an electronic device (1) comprising a processing unit (11), a video output module (13) and an audio output module (14), the method being characterized in that it comprises: playing (a0) an audiovisual content on the electronic device (1), by outputting a video component of the audiovisual content with a first quality level at the video output module (13) and outputting an audio component of the audiovisual content at the audio output module (14); determining (a), by the processing unit (11), whether a user of the electronic device (1) is paying attention to the video output module (13); and in response to determining that the user is not paying attention to the video output module (13), interrupting (b) the output of the video component of the audiovisual content with the first quality level.

Description

Method for operating an electronic device when playing an audiovisual content
FIELD OF THE INVENTION
The field of this invention is that of mobile phones playing audiovisual components. More particularly, the invention relates to a method for operating a device when playing an audiovisual content.
BACKGROUND OF THE INVENTION
Recent mobile phones (smartphones) offer numerous features, including GPS, internet, music, video playing, games and countless apps. Lots of users regularly play various videos on their smartphone.
Nevertheless, it is quite common to not“watch” the video and to only listen to it, for instance in the case of music videos, presentations, talkshows, podcasts, discussions, news, etc., wherein the audio component is the most interesting one.
And even for“normal” videos that are expected to be watched, sometimes the user is doing other things (for example cooking or doing homework), and does not pay attention to the image all the time (he may look occasionally at the screen, for example when hearing interesting things).
The problem is that video playback costs sensibly more power and more data traffic than audio. Indeed, the video stream is heavier than the audio stream. When the user does not look at the screen, the data/power used for playing the video stream is wasted.
There exists ways (for example browser extensions) for downloading and playing only the audio stream of video (the video stream is blocked), for instance if the user already knows that he will only listen to the audio.
Such solution is efficient but is not very convenient as it is“all-or-nothing”: should the user changes his mind and wishes to watch a part of the video, he would have to manually reload the video this time with both streams, and restart its playing from the beginning.
There is consequently a need for a simple and user-friendly solution which could further help reducing the power and bandwidth consumption when playing videos. SUMMARY OF THE INVENTION
For these purposes, the present invention provides a method for operating an electronic device comprising a processing unit, a video output module and an audio output module, the method being characterized in that it comprises:
playing (aO) an audiovisual content on the electronic device, by outputting a video component of the audiovisual content with a first quality level at the video output module and outputting an audio component of the audiovisual content at the audio output module;
determining (a), by the processing unit, whether a user of the electronic device is paying attention to the video output module; and
in response to determining that the user is not paying attention to the video output module, interrupting (b) the output of the video component of the audiovisual content with the first quality level.
Preferred but non limiting features of the present invention are as follow:
• the electronic device comprises a camera, further comprising acquiring (a1 ) images using the camera, the determining step (a) being performed based on the acquired images;
• the determining step (a) further comprises analyzing (a2) the acquired images so as to detect a face of the user, the determining step (a) being performed based on the result of the detection of a face of the user;
• the determining step (a) further comprises analyzing (a2) the acquired images so as to detect an eye of the user, the determining step (a) being performed based on the result of the detection of an eye of the user;
• the determining step (a) is performed based on the result of the detection of an open eye of the user in said at least one acquired image, the user being determined as paying attention to the video output module (13) if an open eye is detected in said at least one acquired image;
• the video component of the audiovisual content with the first quality level and the audio component of the audiovisual content are transmitted from a remote server to the electronic device, and wherein interrupting (b) the output of the video component of the audiovisual content with the first quality level comprising interrupting the transmission of the video component of the audiovisual content with the first quality level; • the method further comprises outputting (b1 ) the video component of the audiovisual content with a second quality level at the video output module, said second quality level being lower than the first quality level;
• outputting (b1 ) the video component of the audiovisual content with a second quality level at the video output module comprises triggering the transmission, from the remote server, of the video component of the audiovisual content with the second quality level;
• the video component of the audiovisual content with the second quality level has a lower resolution and/or a reduced number of frames by seconds than the video component of the audiovisual content with the first quality level;
• the method further comprises, in response to determining that the user has not been paying attention to the video output module for a duration exceeding an inattention threshold, interrupting (b2) the output of the video component of the audiovisual content with the second quality level;
• the audiovisual content is further played without outputting any video component after interrupting (b) the output of the video component of the audiovisual content with the first quality level;
• the method further comprises determining, by the processing unit, if the user is paying again attention to the video output module of the audiovisual content; and, in response to determining that the user is paying again attention to the video output module, outputting (c) the video component of the audiovisual content with the first quality level.
In a second aspect, the invention provides an electronic device comprising:
a processing unit,
a video output module for outputting a video component of an audiovisual content with a first quality level; and
an audio output module for outputting an audio component of the audiovisual content at the audio output module,
wherein said processing unit is configured to determine (a) whether a user of the electronic device is paying attention to the video output module and in response to determining that the user is not paying attention to the video output module, interrupt (b) the output of the video component of the audiovisual content with the first quality level. According to preferred but non limiting features:
• the video output module is a screen, and the electronic device comprises further, next to the screen, a front camera for acquiring images to be used by the processing unit to determine whether the user is paying attention to the video output module;
• the electronic device comprises further at least one of a proximity sensor, an accelerometer, an infrared detector and a motion sensor, the output of which is used by the processing unit to determine whether the user is paying attention to the video output module.
According to a third and a fourth aspects, the invention provides a computer program product, comprising code instructions for executing a method according to the first aspect for operating a device comprising at least one processing unit, a video output module and an audio output module; and a computer-readable medium, on which is stored a computer program product comprising code instructions for executing a method according to the first aspect for operating a device comprising at least one processing unit, a video output module and an audio output module.
BRIEF DESCRIPTION OF THE DRAWINGS
The above and other objects, features and advantages of this invention will be apparent in the following detailed description of an illustrative embodiment thereof, which is to be read in connection with the accompanying drawings wherein:
- figure 1 illustrates an example of architecture in which is performed the method according to the invention; and
- figure 2 is a diagram representing steps of an embodiment of a method according to the invention.
DETAILED DESCRIPTION PREFERRED EMBODIMENTS
Architecture
The present invention relates to a method for operating an electronic device 1 as represented by figure 1 , in particular for playing an audiovisual content.
For the sake of clarity, in the following description, a distinction is made between the“video” and the“audio”. By video, it will be meant only the visual content, in other words the“moving picture” without the sound. By audio, it will be meant only the sound content, without the video. Thus, by“audiovisual content”, it will be meant a content possessing both an audio component and a video component, i.e. an AV media. An audiovisual content is often inappropriately referred to as a“video”, despite having an audio component.
As explained, the electronic device 1 comprises one or more processing unit(s) 1 1 , a video output module 13 (for instance a built-in screen, acting as the default video output unit of the device 1 ), and an audio output module 14 (for instance a built-in speaker, acting as the default audio output unit of the device 1 ). It is to be noted that the electronic device 1 could alternately comprise as audio and/or video output module 13,
14 at least one video and/or audio connector toward an external device such as a TV, a projector, etc. Indeed, such connector could be video and audio (such a HDMI port), video-only, or audio-only (such as a jack for earphones). Furthermore, such audio and/or video connector could be virtual, for instance a Bluetooth connection to a VR headset.
It will be understood that the present invention can be implemented on any electronic device 1 able to play in any way an audiovisual content with audio and video components.
In a preferred embodiment, the electronic device 1 further comprises a camera
15 (in particular a front camera). The electronic device 1 typically further comprises a communication unit for connecting (in particular wirelessly) the electronic device 1 to a network 20 (for example WiFi, Bluetooth, and preferably a mobile network, in particular a GSM/UMTS/LTE network), a memory 12 (for example flash memory), in particular for storing applications, which can be of various types. In a known fashion, the processing unit 1 1 is able to run these applications, and one or more of them could be players, i.e. applications able to retrieve an audiovisual content from a remote server 2 of the network, and play it.
The electronic device 1 may further comprise others units such as a battery, a user interface (for instance, if it comprises a screen as video output module 13, this screen could be touch sensitive), a location unit for providing location data representative of the position of the electronic device 1 (using for example GPS, network triangulation), etc. The electronic device 1 is typically a smartphone, a tablet computer, a laptop, etc. In the following description, the example of a smartphone will be used.
Playing an audiovisual content
The present method is performed mostly by the processing unit 1 1 of the electronic device 1 , and is advantageously implemented by the operating system of this electronic device 1 , or a dedicated application such as an audio/video player.
The present method starts with a step (aO) of playing an audiovisual content, which is typically obtained from a remote server 2, i.e. downloaded via the network 20. In other words, the audiovisual content is preferably streamed from such a remote server 2. However, the present invention may also apply to downloaded audiovisual content which is not streamed, i.e. audiovisual content which is played after having been completely downloaded by the electronic device 1 .
As explained, such audiovisual content has a video component and an audio component, so that playing the audiovisual content includes outputting the video component of the audiovisual content at the video output module 13 and outputting an audio component of the audiovisual content at the audio output module 14.
It is to be understood that several versions of the video component might be available, corresponding to different qualities. For example, there might be a plurality of resolutions (generally six resolutions: 144p, 240p, 360p, 480p, 720p or 1080p), and/or several numbers of frames by seconds (generally 30 fps or 60 fps). Similarly, several versions of the audio component might be available, even if it is rarer than for the video component (indeed the video component is far heavier than the audio component).
Typically, either the user selects the quality he wishes, or it is automatically selected thanks to the Adaptive bitrate (ABR) streaming technique. Such an adaptive bitrate streaming works by detecting a user's bandwidth and CPU capacity in real time and adjusting the quality of the component accordingly: the quality selected shall be the highest possible without degrading the user experience (i.e. with very little buffering, fast start time, etc.). To this end, the Real Time Streaming Protocol (RTSP) is preferably used: when the user wishes to play the audiovisual content, a RSTP request is sent from the electronic device 1 to the remote server 2, and the content with the “optimal” video and audio components is received in return.
In the following description, we will refer to a first quality level of video and audio components as the quality level of the components as normally outputted at the electronic device 1 during step (aO), even if such quality could vary over time thanks to the ABR because of some events: for example, should a network congestion happens, the current quality will be temporary reduced (with respect to the quality when starting playing) to avoid uncomfortable rebuffering, but the provided components will always be the“optimal” ones given the available resources.
As represented by the figure 2, the present method comprises a determining step (a) of determining whether a user is paying attention to the video output module 13.
This determining step (a) can be continuously performed after having initiated (step aO) the playing of the audiovisual content by the electronic device 1 . More precisely, playing an audiovisual content (step (a0)) triggers the determining step (a). To rephrase again, the step (a) is advantageously performed only during the playing of audiovisual contents by the electronic device 1 , so as to save battery of this electronic device.
The principle behind the present invention is that, if the user is not paying attention to the video output module 13, the video component could be“lightened”, or even cancelled, without altering the user experience: (s)he will not even realize it, while the network bandwidth usage and/or electronic device power consumption are reduced.
Image analysis
The present method proposes to assess the attention of the user to the video output module 13, i.e. to detect whether or not the user is looking at the video, and only listening to the audio component.
Advantageously, to avoid false negatives, a check is performed at a given frequency (for instance every minutes), and the user is determined as not paying attention if the user has not looked at the video output module 13 for a duration exceeding a first inattention threshold (for example 5 minutes). Indeed, the user might be distracted during a short moment without this meaning that (s)he is only listening to the audio component.
Determining step (a) is typically a step of“observing” the user. To this end, when the electronic device 1 comprises a camera 15, the present method comprises (a1 ) acquiring images using the camera 15 (in particular at a given frequency such as 1 image each minute), the recorded images being used by the processing unit 1 1 to determine whether the user is paying attention to the video output module 13. In a particular embodiment, the acquired images are analyzed by the processing unit 1 1 (step a2) so as to detect a face of the user, the determining step (a) being performed based on the result of the detection of a face of the user. Indeed, from the position of the face, it can be determined what the user is watching, and thus whether (s)he is staring at the video output module 13 or not.
Such a so-called“face detection” embodiment based on analyzing an acquired image in order to detect the face of the user, for determining whether the user pays attention to the video output module or not, is advantageous when compared to another embodiment where the electrical device 1 would simply detect if a user is present in a specific area in front of the video output module (for instance using an infrared sensor): in the last case, the electronic device 1 is not able to infer for sure that the user is indeed paying attention to the video output module, as the user could be looking away from the electronic device 1 while still being detected in front of the electronic device 1 . With such a detection of the face of the user from acquired image(s), the determination is quite easy to implement while still more accurate than detecting the presence of a user in front of the electronic device 1 .
In another more refined embodiment, the acquired images are analyzed by the processing unit 1 1 (step a2) so as to detect an eye of the user, the determining step (a) being performed based on the result of the detection of an eye of the user. Indeed, from the detection of an eye facing the video output module 13, it can be determined that the user is watching, in particular when this eye is detected as being open, and thus whether (s)he is staring at the video output module 13 or not.
Such an embodiment based on analyzing an acquired image in order to detect an (open) eye of the user, for determining whether the user pays attention to the video output module or not, is advantageous when compared to the previous embodiments based on presence detection or face detection: in these last cases, the electronic device 1 is not able to discriminate for sure between an awaken user watching the video and a sleeping user merely facing electronic device 1 . With the detection of an eye of the user facing the video output module 13, and in particular the detection of an open eye facing this video output module 13, such a determination can be made even more accurate.
The above-mentioned face and/or eye detection in an image can be implemented using well-known techniques for the skilled person in the field of image processing, such as the face and eye detection stages discussed for instance in the article “Real-time Eye Gaze Direction Classification Using Convolutional Neural Network”, Anjith George et al., 2016 International Conference on Signal Processing and Communications (SPCOM), IEEE, 12-15 June 2016, Bangalore, India.
In a preferred embodiment, the video output module 13 is a screen, and the camera 15 is a front camera next to the screen (generally just above the screen). In such an embodiment, what the camera 15 sees is what sees the screen: if an eye of the user is visible, it means that the user can pay attention to the screen. To the contrary, if the face is not visible it means that either the screen is concealed (and not visible) or that the user is staring at something else, so that the user cannot be paying attention to the video output module 13.
In other words, in such an embodiment, it can be considered that the user is paying intention to the video output module simply when its face or its eye is detectable on the acquired images, which is easy to implement.
More generally speaking, lots of techniques for image analysis able to detect a face or an eye and possibly track their direction are known to the skilled person and could be used here, in order to determine if the user is watching the video output module 13. A pre-trained Al model can be used effectively.
It will be understood that the determination of whether the user of the electronic device 1 is paying attention to the video output module 13 is not limited to the analysis of acquired images, and other techniques could be used, as for instance :
- a proximity sensor of the electronic device 1 (if the proximity sensor detects a nearby surface it means that the screen is not visible - for instance the device 1 is put down on a table -and thus that the user is not paying attention),
- an accelerometer (if small movements are detected it means that the user is holding the electronic device 1 in his hands and is probably paying attention to the video output module 13),
- an infrared detector, which can detect if there are people in front of the electronic device 1 or not,
- a motion sensor, which can detect if there are people in front of the electronic device 1 by motion detection.
Interrupting the output of the video component with the first quality
When it has been determined that the user is not paying attention to the video output module 13, in a further step (b), the output of the video component of the audiovisual content with the first quality level is interrupted. It is to be understood that the output of the audio component with the first quality level is not interrupted, so that the playing of the audiovisual content continues.
By interrupting the output of the video component of the audiovisual content with the first quality level, it is simply meant that there is no more output at this first quality level, but there might still be an output of the video component of the audiovisual content at another quality level. In other words, the video component is“degraded”.
As an example of such an interruption of the video output, if the audiovisual content is played through a media player application, the processing unit 1 1 may instruct this media player application to stop outputting the video, or to replace the video component with a still image (for instance a black screen). Alternatively, when the present method is implemented by a video player application running on the processing unit 1 1 , this application directly stops the video output.
In another embodiment wherein the electronic device 1 is a mobile terminal having lock screen functionality, a software application, such as the operating system of the electronic device 1 , may replace the video output of the video component with the locked screen, i.e. may trigger the locking of the screen, while the audio output continues.
According to a first embodiment, the method further comprises outputting (b1 ) the video component of the audiovisual content with a second quality level at the video output module 13, said second quality level being lower than the first quality level. Here, there is a partial degradation of the video component, as its quality is simply reduced, which reduces the power consumption of the electronic device 1 .
As an example of such a video output using a second quality level, if the audiovisual content is played through a media player application, the processing unit 1 1 (or the operating system of the electronic device 1 ) may instruct this media player application to replace the initial audiovisual component comprising a video component with a first quality level with a corresponding audiovisual content comprising a video component with a second quality level, for instance by instructing this media player to read a file storing such a video component with a second quality level instead of reading a file storing the video component with the first quality level. Such files with different video quality levels may be stored locally, in the electronic device 1 , after having been downloaded from a remote server 2, or may be stored remotely, in the remote server 2, in which case they may be streamed from this remote server 2. Alternatively, when the present method is implemented by a software application running on the processing unit 11 (e.g. the operating system of the electronic device 1 ), this software application may directly replace the video output with the first quality level by the video output with a second quality level, for instead by reading a file storing the video component with the second quality level.
According to a second embodiment, the method does not further comprise outputting any video component of the audiovisual content (i.e. whatever the quality) at the video output module 13. In other words, the audiovisual content is further played without outputting any video component after interrupting (b) the output of the video component of the audiovisual content with the first quality level. This time, this is a“total degradation” of the video component (i.e. no more video component is outputted), which allows to save even more in terms of power consumption of the electronic device 1
By contrast, the audio component is not degraded. The audio component keeps being outputted with the same quality level as it was before that the video component was degraded. In the case of a“total degradation” of the video component, it is to be understood that the playing of the audiovisual content becomes“audio-only”.
As an example of such an audio output kept outputted, if the audiovisual content is played through a media player application, the processing unit 1 1 may instruct this media player application to replace the initial audiovisual content with a video component with a corresponding audiovisual content having only an audio component, for instance by instructing this media player to read a file storing such an audiovisual content with only audio component instead of reading a file storing the full audiovisual content, i.e. with a video component. Here also, such corresponding files, with or without video component, may be stored locally, in the electronic device 1 , after having been downloaded from a remote server 2, or may be stored remotely, in the remote server 2, in which case they may be streamed from this remote server 2.
Lowering the quality level of the video component, or even completely stopping its output, reduces the power consumption of the electronic device 1 , as a screen is for example one of the more consuming units of the electronic device 1 .
In the case of a content obtained from a remote server 2, typically by a streaming technique, step (b) preferably comprises (in response to determining that the user is not paying attention to the video output module 13) interrupting the transmission of the video component of the audiovisual content with the first quality level, so as to also reduce bandwidth consumption in the network. This interruption can be achieved typically by sending, from the electronic device 1 to the remote server 2, a RTSP request to stop streaming the audiovisual content. Other protocols can naturally be used for such requests to stop streaming, for instance the HTTP Live Streaming (HLS) protocol.
It is to be noted that it is possible that only the output of the video component with the first quality level may be interrupted on the electronic device 1 , but not its transmission from the remote server 2 (i.e. the invention is not limited to interrupting both), leading to reduction of the power consumption of the electronic device 1 but not reduction of the bandwidth consumption on the network side, with a very simple implementation on the electronic device.
Similarly, the interruption of the video component transmission could be either partial of total (i.e. the video component is no more transmitted at all). In the case where there is a step of outputting (b1 ) the video component of the audiovisual content with a second quality level at the video output module 13, the method preferably comprises triggering the transmission, from the remote server 2, of the video component of the audiovisual content with the second quality level.
For instance, if the first quality level corresponds to a video resolution of 720p, the second quality level could correspond to a video resolution of 480p.
To this end, besides the aforementioned RTSP request to stop the transmission of the video component with the first quality level, a new RSTP request could be sent to the remote server 2, in order to trigger the transmission of the video component with the second quality level. Other protocols can naturally be used for such requests to stop streaming, for instance the HTTP Live Streaming (HLS) protocol.
The application preferably records the progress of the user watching the video with the first quality level, so as to switch to the second quality level from current progress when receiving the video component with the second quality level in response to the request: the playing is not interrupted and the users does not even notice that the video component has been switched or even interrupted.
In the opposite case, if it has not been detected (for a duration exceeding said first inattention threshold) that the user is not paying attention to the video output module 13 (i.e. it has been determined that the user is actually paying attention to the video output module 13, in particular at least once during the duration of said first inattention threshold), the output of the video component is not interrupted (it is kept outputted with the first quality level), i.e. the playing of the audiovisual content continues as such, with the same quality level as initially provided. As explained, the determining step (a) could be performed during the whole audiovisual content playing, even after that the output of the video component has been interrupted. Indeed, in a further step (c), in response to determining that the user is paying again attention to the video output module 13, the video component of the audiovisual content may be outputted again with the first quality level. In other words, the video component is restored to its initial quality, i.e. the video component is no more degraded.
It is to be understood that the determination that the user is paying again attention to the video output module 13 is preferably instantaneous: as soon as it is detected that the user is looking at the screen, (s)he is determined as paying again attention (there is no need to wait for the duration of an inattention threshold, contrary to the determination of a user not paying attention to the video output module) and the output of the video component with the first quality level may be resumed immediately.
In that case, if the transmission from the remote server 2 of the video component with the first quality level was also interrupted, the electronic device 1 also triggers the resuming of the transmission, from the remote server 2, of the video component with the first quality level. Here also, the application preferably records the progress of the user watching the video with the second quality level, so as to switch to the first quality level from current progress when receiving again the video component with the first quality level in response to a request to resume to the first quality level: the playing is not interrupted and the users does not even notice that the video component has been switched or even interrupted.
The method of the present invention can go on as long as the audiovisual content is played (i.e. as long as the user does not voluntarily stop its playing) : after having determined that the user is paying again attention to the video output module, if the user is again determined as not paying attention to the video output module 13, step (b) may be performed again (by interrupting the output of the video component with the first quality level), and so on, for the whole duration of the audiovisual content playing.
In another embodiment, when the video component of the audiovisual content is outputted with the second quality level, and when the user is still determined as not paying attention to the video output module 13 for a duration exceeding a second inattention threshold (longer that the first inattention threshold), the method further comprises a step (b2) of interrupting the output of the video component of the audiovisual content with the second quality level. In other words, there might be a further degradation of the video component. Again, if the video component of the audiovisual content with the second quality level is transmitted from a remote server 2 to the electronic device 1 , step (b2) preferably comprises the interruption of this transmission.
In a particularly preferred embodiment, there could be more than two quality levels, and for instance the method could further comprises outputting the video component of the audiovisual content with a third quality level at the video output module 13, then possibly when the user is still determined as not paying attention to the video output module 13 for a duration exceeding a third inattention threshold (longer that the second inattention threshold) interrupting the output of the video component of the audiovisual content with the third quality level, then outputting the video component of the audiovisual content with a fourth quality level at the video output module 13, etc.
For the sake of convenience, a “last” quality level could be defined as the absence of further output of any video component (the so-called “total degradation” mentioned before).
For instance, in the case of 4 quality levels:
- first level · no degradation, video component as initially outputted (for example 1080p);
- second level
Figure imgf000016_0001
video component with slightly reduced quality (for example 480p);
- third level
Figure imgf000016_0002
video component with strongly reduced quality (for example 240p);
- fourth level
Figure imgf000016_0003
total degradation, no video component outputted.
At each time an inattention threshold is reached, the quality is reduced, i.e. changed from the n-th level to the n+1-th level.
Then, if the user keeps not paying attention to the video output module 13, the quality keeps draping, in particular at a given rate, for example 1 level per more minute (i.e. the n+ 1-th inattention threshold is one minute longer than the n-th inattention threshold). Indeed, if despite the degradation of the video component the user is still not looking at the screen, it means that the degradation does not bother him and this confirms that (s)he is not paying attention. The degradation can therefore go further so as to even more reduce the power and/or bandwidth consumption. In the present example, after 7 minutes of not paying attention, the fourth level is reached so that there is no more outputted video component, while the audio component is continuously outputted.
Whatever the quality level reached, when step (c) is performed (i.e. in response to determining that the user is paying again attention to the video output module 13), the quality level is directly set back to the first level (no degradation at all).
In the aforementioned embodiment wherein the electronic device 1 is a mobile terminal having a screen lock functionality, where the screen has been locked following the determination that the user was not paying attention to the video output module, this can be achieved by the software application (typically the operating system of the mobile terminal) which triggers the unlocking of the screen, thereby leading to outputting again the video component with the first quality level on the video output module 13.
Device, computer product program, and computer-readable medium
The present invention concerns an electronic device 1 comprising at least one processing unit 1 1 , a video output module 13 (in particular a screen) and an audio output module 14 (in particular a speaker or an audio jack for connecting an external speaker or headset), wherein the processing unit 11 is adapted for carrying out the method for operating as previously described.
The video and audio output modules 13, 14 are initially respectively for outputting a video and an audio component of an audiovisual content with a first quality level.
This electronic device 1 may also comprise a memory 12, a camera 15. The electronic device 1 is typically a smartphone. In various embodiments, instead or in addition to the camera 15, the electronic device can also comprise a proximity sensor, an accelerometer, an infrared detector and/or a motion sensor (not illustrated), the output of which can be used by the processing unit 1 1 to determine if a user pays attention to the smartphone.
The aforementioned processing unit 1 1 is in particular configured to:
- determine (a) whether a user of the electronic device 1 is paying attention to the video output module 13 (in particular by analyzing images acquired by the camera 15 so as to detect a face or an eye of the user and/or the output of any combination of a proximity sensor, an accelerometer, infrared detector and/or motion sensor); - in response to determining that the user is not paying attention to the video output module 13, interrupt (b) the output of the video component of the audiovisual content with the first quality level (preferably by outputting the video component with a second quality level or by not outputting the video component); - and possibly, in response to determining that the user is paying again attention to the video output module 13, re-outputting the video component of the audiovisual content with the first quality level (i.e. reverting to the video component as initially outputted at the video output module 13). As already explained, the present method is preferably implemented by an application such as an audio/video player.
Advantageously, the video output module 13 is a screen, and the electronic device 1 comprises further, next to the screen, a front camera 15 for acquiring images to be used by the processing unit 13 to determine whether the user is paying attention to the video output module 13.
The invention further proposes a computer program product, comprising code instructions for executing (in particular with a processing unit 11 of the device 1 ) the previously described method, as well as a computer-readable medium (in particular a memory 12 of the device 1 ), on which is stored a computer program product comprising code instructions for executing said method. In particular, this computer program product may correspond to the operating system, and especially the media controller.

Claims

1. A method for operating an electronic device (1 ) comprising a processing unit (1 1 ), a video output module (13) and an audio output module (14), the method being characterized in that it comprises:
playing (aO) an audiovisual content on the electronic device (1 ), by outputting a video component of the audiovisual content with a first quality level at the video output module (13) and outputting an audio component of the audiovisual content at the audio output module (14);
determining (a), by the processing unit (1 1 ), whether a user of the electronic device (1 ) is paying attention to the video output module (13); and
in response to determining that the user is not paying attention to the video output module (13), interrupting (b) the output of the video component of the audiovisual content with the first quality level.
2. A method according to claim 1 , wherein the electronic device (1 ) comprises a camera (15), further comprising acquiring (a1 ) at least one image using the camera (15), the determining step (a) being performed based on said at least one acquired image.
3. A method according to claim 2, wherein step (a) further comprises analyzing (a2) said at least one acquired image so as to detect a face of the user in said at least one acquired image, the determining step (a) being performed based on the result of the detection of a face of the user.
4. A method according to claim 2 or 3, wherein step (a) further comprises analyzing (a2) said at least one acquired image so as to detect an eye of the user in said at least one acquired image, the determining step (a) being performed based on the result of the detection of an eye of the user.
5. A method according to claim 4, wherein the determining step (a) is performed based on the result of the detection of an open eye of the user in said at least one acquired image, the user being determined as paying attention to the video output module (13) if an open eye is detected in said at least one acquired image.
6. A method according to any one of claims 1 to 5, wherein the video component of the audiovisual content with the first quality level and the audio component of the audiovisual content are transmitted from a remote server (2) to the electronic device (1 ), and wherein interrupting (b) the output of the video component of the audiovisual content with the first quality level comprising interrupting the transmission of the video component of the audiovisual content with the first quality level.
7. A method according to any one of claims 1 to 6, further comprising outputting (b1 ) the video component of the audiovisual content with a second quality level at the video output module (13), said second quality level being lower than the first quality level.
8. A method according to claims 6 and 7 in combination, wherein outputting (b1 ) the video component of the audiovisual content with a second quality level at the video output module (13) comprises triggering the transmission, from the remote server (2), of the video component of the audiovisual content with the second quality level.
9. A method according to any one of claims 7 and 8, wherein the video component of the audiovisual content with the second quality level has a lower resolution and/or a reduced number of frames by seconds than the video component of the audiovisual content with the first quality level.
10. A method according to any one of claims 7 to 9, wherein the method further comprises, in response to determining that the user has not been paying attention to the video output module (13) for a duration exceeding an inattention threshold, interrupting (b2) the output of the video component of the audiovisual content with the second quality level.
11. A method according to any one of claims 1 to 6, wherein the audiovisual content is further played without outputting any video component after interrupting (b) the output of the video component of the audiovisual content with the first quality level.
12. An electronic device (1 ) comprising:
a processing unit (1 1 ),
a video output module (13) for outputting a video component of an audiovisual content with a first quality level; and
an audio output module (14) for outputting an audio component of the audiovisual content;
wherein said processing unit (1 1 ) is configured to determine (a) whether a user of the electronic device (1 ) is paying attention to the video output module (13), and, in response to determining that the user is not paying attention to the video output module (13), interrupt (b) the output of the video component of the audiovisual content with the first quality level.
13. The electronic device according to claim 12, wherein the video output module (13) is a screen, and the electronic device (1 ) comprises further, next to the screen, a front camera (15) for acquiring images to be used by the processing unit (11 ) to determine whether the user is paying attention to the video output module (13).
14. The electronic device according to any one of claims 12 or 13, comprising further at least one of a proximity sensor, an accelerometer, an infrared detector and a motion sensor, the output of which is used by the processing unit (1 1 ) to determine whether the user is paying attention to the video output module (13).
15. A computer program product, comprising code instructions for executing a method according to any one of claims 1 to 1 1 for operating a device (1 ) comprising at least one processing unit (11 ), a video output module (13) and an audio output module (14).
PCT/IB2019/001369 2018-12-21 2019-12-09 Method for operating an electronic device when playing an audiovisual content WO2020128625A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CNPCT/CN2018/122585 2018-12-21
PCT/CN2018/122585 WO2020124535A1 (en) 2018-12-21 2018-12-21 Method for operating an electronic device when playing an audiovisual content

Publications (1)

Publication Number Publication Date
WO2020128625A1 true WO2020128625A1 (en) 2020-06-25

Family

ID=69593734

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/CN2018/122585 WO2020124535A1 (en) 2018-12-21 2018-12-21 Method for operating an electronic device when playing an audiovisual content
PCT/IB2019/001369 WO2020128625A1 (en) 2018-12-21 2019-12-09 Method for operating an electronic device when playing an audiovisual content

Family Applications Before (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/122585 WO2020124535A1 (en) 2018-12-21 2018-12-21 Method for operating an electronic device when playing an audiovisual content

Country Status (1)

Country Link
WO (2) WO2020124535A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230127471A1 (en) * 2021-10-25 2023-04-27 Sony Interactive Entertainment Inc. Streaming system and method

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030052911A1 (en) * 2001-09-20 2003-03-20 Koninklijke Philips Electronics N.V. User attention-based adaptation of quality level to improve the management of real-time multi-media content delivery and distribution
EP1843592A1 (en) * 2006-04-05 2007-10-10 BRITISH TELECOMMUNICATIONS public limited company Controlling media content quality
US20150341411A1 (en) * 2013-01-10 2015-11-26 Telefonaktiebolaget L M Ericsson (Publ) Apparatus and Method for Controlling Adaptive Streaming of Media

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2008010118A1 (en) * 2006-07-12 2008-01-24 Koninklijke Philips Electronics N.V. A device and a method for playing audio-video content
CN102800341A (en) * 2012-07-02 2012-11-28 宇龙计算机通信科技(深圳)有限公司 Terminal and multimedia playing method thereof
CN103391466A (en) * 2013-06-26 2013-11-13 深圳市睿立南方科技有限公司 Set top box of television and video output method thereof
CN104869453A (en) * 2015-05-04 2015-08-26 小米科技有限责任公司 Video playing equipment control method and device
CN106454456A (en) * 2016-10-31 2017-02-22 维沃移动通信有限公司 Video playing control method and mobile terminal

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030052911A1 (en) * 2001-09-20 2003-03-20 Koninklijke Philips Electronics N.V. User attention-based adaptation of quality level to improve the management of real-time multi-media content delivery and distribution
EP1843592A1 (en) * 2006-04-05 2007-10-10 BRITISH TELECOMMUNICATIONS public limited company Controlling media content quality
US20150341411A1 (en) * 2013-01-10 2015-11-26 Telefonaktiebolaget L M Ericsson (Publ) Apparatus and Method for Controlling Adaptive Streaming of Media

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
ANJITH GEORGE ET AL.: "2016 International Conference on Signal Processing and Communications (SPCOM", 12 June 2016, IEEE, article "Real-time Eye Gaze Direction Classification Using Convolutional Neural Network"

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230127471A1 (en) * 2021-10-25 2023-04-27 Sony Interactive Entertainment Inc. Streaming system and method

Also Published As

Publication number Publication date
WO2020124535A1 (en) 2020-06-25

Similar Documents

Publication Publication Date Title
CN108495142B (en) Video coding method and device
US11036458B2 (en) User interface for screencast applications
WO2017219347A1 (en) Live broadcast display method, device and system
CN110213616B (en) Video providing method, video obtaining method, video providing device, video obtaining device and video providing equipment
US10771736B2 (en) Compositing and transmitting contextual information during an audio or video call
US9955058B2 (en) Multi-codec camera system and image acquisition program
CN113141514B (en) Media stream transmission method, system, device, equipment and storage medium
US11336950B2 (en) Methods, systems, and media for transferring playback of media content
US10530825B2 (en) Catching up to the live playhead in live streaming
JP5528496B2 (en) Information processing apparatus and control method thereof
US20100156760A1 (en) Motion controlled multimedia content viewing method and system
US9756373B2 (en) Content streaming and broadcasting
US9131106B2 (en) Obscuring a camera lens to terminate video output
US10560753B2 (en) Method and system for image alteration
US10764649B2 (en) Falling back from three-dimensional video
US10091265B2 (en) Catching up to the live playhead in live streaming
EP3466081A1 (en) Catching up to the live playhead in live streaming
WO2021143388A1 (en) Bitrate switching method and device
FR3000632A1 (en) METHOD FOR DISPLAYING DATA IN A NETWORK AND ASSOCIATED MOBILE DEVICE
WO2020128625A1 (en) Method for operating an electronic device when playing an audiovisual content
JP4991671B2 (en) Information processing apparatus and control method thereof
JP6800930B2 (en) Communication equipment, its control method, and its program
US9277261B2 (en) Information processing apparatus and control method thereof
CN114501122B (en) Screen projection method and device for audio and video data, storage medium and screen projector
JP2011259232A (en) Display device, control method therefor, program, and recording medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19850855

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19850855

Country of ref document: EP

Kind code of ref document: A1