US20210089256A1

US20210089256A1 - Method and system for browser-based screen sharing

Info

Publication number: US20210089256A1
Application number: US17/110,828
Authority: US
Inventors: Adam Michael Lieb; James L. Benton; Zack Thomsen-Gray; Brad Tofel
Original assignee: Clearslide Inc
Current assignee: Clearslide Inc
Priority date: 2009-11-24
Filing date: 2020-12-03
Publication date: 2021-03-25
Also published as: US20170300286A1; US10860279B2

Abstract

A method for browser-based screen sharing comprises: loading, by a presenter device, a screen sharing script directly executable by a web browser of the presenter device; capturing, by the screen sharing script, a plurality of screen images; adjusting the plurality of screen images based on a network bandwidth between the presenter device and a screen sharing server in order to maintain a specified frame rate of the plurality of screen images; and transmitting, to the screen sharing server, the adjusted plurality of screen images.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No. 15/632,126, filed on Jun. 23, 2017. U.S. patent application Ser. No. 15/632,126 is a continuation-in-part (CIP) of U.S. patent application Ser. No. 12/953,054, entitled “METHOD AND SYSTEM FOR BROWSER-BASED SCREEN SHARING,” filed on Nov. 23, 2010, issued on Aug. 15, 2017 as U.S. Pat. No. 9,733,886, and is a continuation application of U.S. patent application No. 14/448,620 entitled “SCREEN SHARING USING SCRIPTING COMPUTER LANGUAGE CODE DIRECTLY EXECUTABLE BY WEB BROWSER,” filed on Jul. 31, 2014. U.S. patent application Ser. No. 12/953,054 claims the benefit of U.S. Provisional Application No. 61/264,185, entitled “METHOD FOR BROWSER-BASED SCREEN SHARING,” filed on Nov. 24, 2009. U.S. patent application Ser. No. 14/448,620 claims the benefit of U.S. Provisional Application No. 61/860,647 entitled “SCREEN SHARING USING SOFTWARE CODE WHOLLY IMPLEMENTED WITHIN WEB PAGES ACCESSIBLE VIA WEB BROWSER,” filed on Jul. 31, 2013. The above-referenced applications are incorporated by reference herein in their respective entireties.

TECHNICAL FIELD

The present disclosure relates to sharing computer screens to remote computers, and more specifically, to sharing the presenter's screen to one or more viewers over a communications network (e.g., the Internet) using scripting computer language codes that are directly executable by a web browser.

BACKGROUND

Often it is useful for a presenter working on a computer to broadcast a portion of the images shown on the computer screen over a network to remote viewers, such as to demonstrate the capabilities of a software product or website. Several commercial solutions, such as WebEx™ and GoToMyPC™, offer screen sharing related products. Although useful, in these screen sharing products the presenters must download and install software (such as executables or plug-ins) to the presentation computer, while the viewer must complete a time-consuming setup process, which can include software downloads and email-based invitation setup process, to connect the viewer to the presenter. These limitations prevent the usage in certain situations, such as a sales call, and they limit the devices upon which these products can run.

BRIEF DESCRIPTION OF DRAWINGS

The present embodiments are illustrated by way of example and are not intended to be limited by the figures of the accompanying drawings. In the drawings:

FIG. 1A depicts a diagram of an environment within which the present embodiments may be implemented;

FIG. 1B depicts a diagram of an exemplary system of implementing the techniques introduced here;

FIG. 2 depicts a flowchart illustrating an example of actions performed among the presenter device, presentation server(s), and one or more viewer devices for carrying out screen sharing using scripting computer language codes that are directly executable by a web browser;

FIG. 3 depicts a flowchart illustrating an example process performed (e.g., by the presenter device as being instructed by the scripting language codes, or by the screen sharing server) for adaptively adjusting the captured screen image;

FIG. 4 depicts a flowchart illustrating an example process performed by the presenter device (e.g., as being instructed by the scripting language codes) for capturing and transmitting screen images;

FIGS. 5A- 5E show examples of various screen displays that can be generated by the screen sharing server(s) and loaded onto a presenter's device to enable the screen sharing; and

FIG. 6 depicts a diagrammatic representation of a machine in the example form of a computer system within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed.

The same reference numbers and any acronyms identify elements or acts with the same or similar structure or functionality throughout the drawings and specification for ease of understanding and convenience.

DETAILED DESCRIPTION

Techniques are disclosed for facilitating browser-based screen sharing using scripting computer language codes that are directly executable by a web browser. An example method comprises loading a presentation webpage in a presenter's web browser. The presentation webpage includes scripting language codes that are configured to cause the presenter's web browser to capture a screen image without requiring the presenter's web browser to load an applet, which would require to be executed by a process separate from the presenter's web browser on the presenter device. The method further includes receiving data indicative of the captured screen image from the presenter device, wherein the data is generated by the scripting language codes. The method further includes processing the received data to form a processed screen image that is in an image format natively displayable to a viewer's web browser. In addition, the method includes transmitting a viewer webpage including the processed screen image to the viewer's web browser.
Among other benefits, some embodiments provided herein enable browser-based screen sharing capabilities for a presenter to one or more viewers without requiring a software application to be installed on the presenter's computer, nor a plugin (e.g., a Java plugin) to be installed with the presenter's web browser for executing an applet (e.g., a Java applet).
Various examples of the present disclosure will now be described. The following description provides specific details for a thorough understanding and enabling description of these examples. One skilled in the relevant art will understand, however, that the embodiments disclosed herein may be practiced without many of these details. Likewise, one skilled in the relevant art will also understand that the present embodiments may include many other obvious features not described in detail herein. Additionally, some well-known methods, procedures, structures or functions may not be shown or described in detail below, so as to avoid unnecessarily obscuring the relevant description.
The techniques disclosed below is to be interpreted in its broadest reasonable manner, even though it is being used in conjunction with a detailed description of certain specific examples of the present disclosure. Indeed, certain terms may even be emphasized below; however, any terminology intended to be interpreted in any restricted manner will be overtly and specifically defined as such in this Detailed Description section.

SYSTEM OVERVIEW

FIG. 1A depicts a diagram of an environment including a system 100 within which the present embodiments may be implemented. The system 100 includes a presenter operating a presenter device 160, one or more viewers operating viewer devices 180A-180N, a screen sharing server 140 (which is collectively represented by a screen sharing cluster 140A-140N, and optionally in some embodiments, a presentation control server 120), and a network 110.
The viewer devices 180 and the presenter device 160 can be any system and/or device, and/or any combination of devices/systems that is able to establish a connection, including wired, wireless, cellular connections with another device, a server and/or other systems such as the screen sharing server 140. The viewer devices 180 and the presenter device 160 typically include a display and/or other output functionalities to present information and data exchanged between or among the devices 180, 160 and/or the screen sharing server 140. In one embodiment, there is only a single screen sharing server 140. In one or more embodiments, there are multiple screen sharing servers 140 operating collectively or independently, such as the screen sharing server cluster 140A-140N illustrated in FIG. 1A.
The viewer devices 180 and the presenter device 160 may include computing devices such as mobile (or portable) devices or non-portable devices. Non-portable devices can include a desktop computer, a computer server or cluster. Portable devices can including a laptop computer, a mobile phone, a smart phone, a personal digital assistant (PDA), a handheld tablet computer, or a combination thereof. Typical input mechanism on client devices 102A-N can include a touch screen display (including a single-touch (e.g., resistive) type or a multi-touch (e.g., capacitive) type), gesture control sensors (e.g., for 2D or 3D gesture sensing), a physical keypad, a mouse, motion detectors (e.g., including 1-axis, 2-axis, 3-axis accelerometer, etc.), light sensors, temperature sensor, proximity sensor, device orientation detector (e.g., electronic compass, gyroscope, or GPS), and so forth. Note that those skilled in the art will be familiar with the computer systems suitable for use in implanting servers and user devices.
In one example, the viewer devices 180, screen sharing server 140, and presenter's device 160 are coupled via the network 110. In some other examples, the viewer devices 180, the presenter device 160, and screen sharing server 140 may be directly connected to one another.
In those embodiments with the presentation control server 120, the presentation control server 120 sets up an environment for the screen sharing. For example, the presentation control server 120 can give a presenter the tools to navigate through the slides of the presentation, and then, when the presenter reaches a screen sharing slide, display the appropriate web pages to the presenter and viewers. The presentation control server 120 can also act as a load balancer. In this case, the presentation control server can consider the geographic and/or network locations of the presenters, viewers, and sharing servers and the current load of each sharing server, and then pick the best sharing server (e.g., among the screen sharing server cluster 140A-140N) to use which minimizes transmission distance while also distributing load. In one example, the presentation control server 120 act as a central sever that determines to which of the screen sharing servers 140A-140N the server 120 should direct a given screen sharing session. Afterwards, the designated shared server (e.g., SS server 140A) will act as the intermediate component between the presenter device 160 and the viewer devices 180 for performing screen sharing.
As mentioned before, the presentation control server 120 is an optional component. Screen sharing can be implemented without a separate control server, e.g., if screen sharing is not operating in the context of a larger presentation. That is, the presentation control server 120 can be useful for setting up the context in which the screen sharing operates but is not necessary. Alternatively, the functionality of the presentation control server 120 can be implemented on the presenter sharing server 140, the presenter device 160, or distributed across several different systems.
The system 100 supports screen sharing of a presenter's screen on the presenter device 160 via the network 110 and the screen sharing server 140 to one or more viewers operating viewer devices 180A-180N, regardless of whether one or more typically required plugins (e.g., a Java plugin) are installed with viewers' browsers 181. Such technique is described in more detail in U.S. patent application Ser. No. 12/756,110, filed Apr. 7, 2010, entitled “MIXED CONTENT TYPE PRESENTATION SYSTEM,” which is incorporated by reference herein in its entirety.
The presenter device 160 and the viewer devices 180 should each be capable of running a web browser 161, 181. The viewer device web browser 181 is used by the viewer operating the viewer device 180 to access a uniform resource locator (URL) to gain access to a viewer's page, which includes webpage scripts 182 (e.g., Java scripts) that can download a series of images (e.g., from the screen sharing servers 140) of a shared screen of the presenter's device 160 for the viewers to view. The webpage scripts 182 (e.g., Javascripts) can also enable the viewers to perform a variety of functions (e.g., interacting with the presenter). For example, some embodiments of the webpage script 182 embedded in the viewer's page can detect input or control events made by the viewer's input mechanism, such as mouse movements, clicks, mouse wheel rotations, or keyboard strokes, and sends the control events to the server 140.
However, it is recognized here that, in current screen sharing products, the presenter may still need to run an applet 162 on the presentation device 160. In a particular example, a screen sharing webpage containing a sharing applet or an embedded applet 162 (e.g., a Java applet) is to be loaded in the web browser 161 running on the presenter device 160. The applet 162 contains the software code that can capture screenshots of the presenter's screen and upload the screenshots to the screen sharing servers 140, so that screens can be shared with viewers who access the provided viewer URL. Running such applet (e.g., a Java applet) typically requires permission and installation of a plug-in (e.g., a Java plug-in) software with the presenter's browser 161 on the presenter's device 160, thereby create technical difficulty, time delay, and frustration. Also, operating the plug-in software creates a process (or thread) (e.g., a Java Virtual Machine (JVM)) that is separate from the presenter's web browser 161 on the presenter device 160, and this can increase the resource consumption on (thereby slowing down) the presenter device 160.
Accordingly, one or more embodiments of the present disclosure provide screen sharing techniques using only scripting computer language codes that are directly executable by the presenter's web browser 161. The present disclosure enables the presenter to use a stock web browser, without installing any plug-in software, to perform screen sharing functionalities with the viewer devices 180 via the screen sharing server 140.
FIG. 1B depicts a diagram of an exemplary system 105 implementing the techniques introduced here. The system 105 includes the same devices as the system 100, including the presenter device 160, the viewer devices 180A-180N, the screen sharing server cluster 140A-140N, the presentation control server 120, and the network 110, only that at least some of them are configured differently. Specifically, in the system 105, the presenter's browser 164 need not have the webpage applet 162 (e.g., a Java applet) installed, but instead, the presenter's browser 164 can utilize one or more techniques introduced here to perform screen sharing functionalities. In some embodiments, the presenter's web browser 164 includes capability to perform plug-in-free real-time communication (RTC) with another browser. Examples of such capability include WebRTC, CU-RTC-WEB, and so forth. Overall, the technique introduced here can utilize the presenter's web browser 164's built-in capability so as to reduce or completely remove the need of the webpage applet 162.
For purposes of discussion herein, a webpage script (e.g., Javascript) means a piece of software code that is written in scripting computer programming language, which can be directly executed (e.g., without being compiled by a compiler first) by a web browser. A scripting language is most commonly used and implemented as client-side scripts to facilitate interaction with the user, control the browser, communicate asynchronously, alter the document content that is displayed, and so forth. A person skilled in the art will know that it typically only takes an interpreter (e.g., a software application such as a web browser) to directly executes, i.e. performs, instructions written in a programming or scripting language, without previously compiling them into a machine language program.
Specifically, to facilitate screenshot capturing, the presenter's webpage that is loaded in the presenter's web browser 164 need only to include webpage script 165 (e.g., a Javascript) in lieu of the webpage applet 162. In some embodiments, the presenter's webpage only contains webpage script 165 and does not include any webpage applet 162. In variations, the presenter's webpage can contain both webpage script 165 and webpage applet 162, but only the webpage script 165 performs the screenshot capturing. The webpage script 165 leverages the built-in functionalities that are embedded within the web browser 164. In one or more embodiments, the presenter's webpage that is loaded onto the presenter's web browser can automatically determine the browser's capability (e.g., such as whether the browser supports plug-in-free RTC with another browser) in order to determine whether to require the web applet 162 or simply include the webpage script 165 for screen sharing. Such automatic determination can be achieved by, for example, inspecting the headers of the page requests that are sent to the presentation control server 120 to identify the web browser's type and ability.
More specifically, it is observed that, when the web browser 164 is equipped with built-in (therefore, plug-in-free) RTC capability with another browser, this capability can be adapted by the present embodiments to realize screen sharing using only the webpage script 165 without requiring a webpage applet. Conventionally, the RTC capability (e.g., WebRTC) of the web browser 164 is designed to enable browser-to-browser applications for voice calling, video chat, and peer-to-peer (P2P) file sharing without plugins. In other words, native web browser RTC capability can enable direct video streaming from one browser to another via the network 110 (e.g., the Internet). However, RTC at its full capability may require a network bandwidth that is too large, and a computing machine too powerful, thereby making pure RTC less ideal for screen sharing in presentation scenarios. Additionally, relying solely on web browsers' RTC capability to implement browser-to-browser screen sharing would typically require both the presenter device 160 and the viewer devices 180 to have equivalent/compatible RTC functions, making it cumbersome for purposes of screen sharing during presentations (e.g., of a sales pitch). Furthermore, the RTC functions may be blocked by network administrators or other firewall settings, which may be typical for big companies or other commercial settings. These drawbacks have traditionally limited the RTC's applicability on screen sharing applications.
Briefly described, in accordance with some aspects of the technique introduced here, the webpage script 165 can be loaded onto the presenter's browser 164 so that the webpage script 165 can capture a screen image from a display of the presenter device 160 without requiring the web browser 164 to load an applet, which would typically require execution by a process or a thread separate from the web browser 164 on the presenter device 160. More precisely, embodiments of the webpage script 165 adapts the RTC functionality of the web browser 164 so that, instead of being used for video streaming with another browser (thus incurring all the aforementioned issues), the webpage script 165 instructs the web browser 164 to capture static screenshots of the presenter device 160. In some embodiments, the webpage script 165 does so via an application programming interface (API) of the presenter's web browser 164. Then, the webpage script 165 processes the captures screenshots, and send the screenshots to the screen sharing servers 140. The screen sharing servers 140 can thereafter transmit the screenshots to one or more of the view devices 180.
For purposes of facilitating a deeper understanding of the present embodiments, various functions and configurations of system 105 are now described in conjunction with the flow charts of FIGS. 2-4 and the screenshots of FIGS. 5A-5E.

METHODOLOGY

FIG. 2 depicts a flowchart 200 illustrating an example of actions performed among the presenter device 160, presentation server(s) 140 (and optionally, control server 120), and one or more viewer devices 180 for carrying out screen sharing using the webpage script 165 that are directly executable by the presenter's web browser 164. The exemplary actions depicted in flowchart 200, upon being implemented by instructions, may be downloaded onto and performed by a presenter's device to share the presenter's screen with the viewers.
Generally, a presenter starts by providing to viewers a viewer URL that uniquely identifies the presenter. When a viewer goes to the viewer URL using a web browser, thereby loading a presentation viewing webpage, the viewer automatically sees a presentation slide or other content, such as images, selected by the presenter.
When the presenter wishes to share his screen with the viewers, a particular presentation slide that may be selected by the presenter is a “live demo” slide, which loads a live sharing webpage containing the screen sharing script 165 (e.g., a Javascript and/or a series of AJAX code) in a web browser 164 running on the presenter device 160. Additionally or alternatively, the presenter can select to start a live screen sharing by directly loading onto the web browser 164 the live screen sharing webpage containing the sharing script 165. An example of such live screen sharing webpage is illustrated in the screenshot of FIG. 5A. Because the script 165 is composed of scripting language codes that are directly executable by the web browser 164, the presenter does not need to download any software or plug-ins. As an option, the webpage script 165 may ask the presenter to confirm or to give permission to share the screen. Then, the webpage script 165 shares the presenter's screen with viewers who access the provided viewer URL without the viewer having to download any software or plug-ins.
More specifically, in some embodiments, when the presenter initiates a screen sharing session (e.g., by clicking on the screen sharing button 510 in FIG. 5A) the webpage script 165 start the flowchart 200 by sending (202) a request for a screen sharing server to the control server 120. Similarly, when the viewer loads the viewer's link given by the presenter onto the viewer's web browser 181, the webpage script 182 sends (203) a request to view the presentation to the control server 120. As mentioned, the control server 120 can assign (204) one or more screen sharing servers 140 among the screen sharing servers 140A-140N (e.g., of FIG. 2) to perform the screen sharing session based on their current workload, network bandwidth and availability, and/or the requestor's geographic location. Under some circumstances (e.g., when one screen sharing server is close to one viewer device while another screen sharing server is close to another viewer device), more than one screen sharing servers may be utilized for optimized screen sharing experience. Thereafter, the presenter device 160 receives (206) its screen sharing server assignment from the presentation control server 120. The viewer devices 180 also receive (205) their respective screen sharing server assignment from the presentation control server 120. During this screen sharing session establishing phase, a session key which identifies and differentiates the presenter device 160 from the viewer devices 180 may be transmitted by the control server 120 to the presenter device 160. As noted, the assignment of the screen sharing server(s) may or may not be the same, depending on the scenario. Further, in some embodiments, the presentation control server 120 may be absent, and one or more of the screen sharing servers 140A-140N may take the role of the control server 120.
Using the received screen sharing server assignment, the webpage script 165 then establishes (208) communications with the screen sharing server 140. Optionally, either the webpage script 165 or the screen sharing server 140 can detect (210) whether the web browser 164 includes capability to perform RTC with another browser. For example, in some embodiments, the web browser 164 may include built-in WebRTC capability, such as a Google Chrome™ browser. If the web browser 164 does not include built-in capability to perform plug-in-free real-time communication with another browser, the webpage script 165 can prompt (212) the presenter to load an webpage applet (e.g., webpage applet 162) or to install a plug-in to support RTC capability, such as the dialog window 560 shown in the screenshot of FIG. 5E. In variations, the webpage script 165 can allow the presenter himself to manually select (or switch) between using the script 165 and the applet 162 to perform the screen sharing, such example shown as a manually selectable link 515 in the screenshot of FIG. 5A.
If the web browser 164 does include the capability to perform real-time communication with another browser, the webpage script 165 then instructs (216) the web browser 164 to capture screen images from a display of the presenter device 160 without requiring the web browser 164 to load an applet (e.g., applet 162). In some examples, the webpage script 165 scripting language codes are configured to cause the presenter's web browser 164 to capture the screen image by issuing instructions through an application programming interface (API) of the presenter's web browser 164.
By adapting the RTC capability to its own use, the webpage script 165 can replace the functionality of the webpage applet 162, taking screenshots (or pieces of images collectively representing the designated screen sharing area), processing the screenshots, and sending them through the screen sharing server 140 to the viewer device 180. More specifically, after acquiring or parsing the pixel information on the screen (of the presenter device 160), the webpage script 165 is configured to generate and/or adjust data indicative of the captured screen image. Specifically, the webpage script 165 can perform one or more different actions to generate/adjust such data. Example actions include encoding the raw screen image (as captured by the web browser 164) into one image format, converting the image format from one to another, shrinking the resolution of the image, and/or reducing the color depth of the image. As examples, these actions can be performed in order to reduce the data size, and can be performed based on detected network bandwidth. More example processes on how to generate/adjust data (e.g., with dynamic adjustments to the captured screen) are described in FIG. 3. Then, the webpage script 165 transmits (216) the adjusted data (that remain indicative of the captured screen image) from the presenter device 160 to the screen sharing server 140 via the network 110.
As an option, when the screen sharing server 140 receives (218) the adjusted data from the presenter device 160, the screen sharing server 140 can further perform similar processes of adaptively adjusting the captured screen image using the techniques described in FIG. 3.
Next, the screen sharing server 140 processes (220) the adjusted data to form processed screen images. As mentioned above, this technique of forming processed screen images is discussed in more detail in U.S. patent application Ser. No. 12/756,110, entitled “MIXED CONTENT TYPE PRESENTATION SYSTEM.” The processed screen images are to be in an image format that is natively displayable to the viewer's web browser 181. Examples of such format may include, for example, JPG, .BMP, .GIF, and .PNG files. Note that, among other suitable reasons, because the processed screen images are in image formats natively displayable to viewers' browsers 181, the viewer webpage (which includes shared images or presentations from the presenter device 160) can be correctly displayed in the viewer's web browser 181 regardless of the browser 18Ts capability to perform plug-in-free real-time communication (RTC) with another browser. Among other benefits, this alleviates or even eliminates the aforementioned problems of using pure RTC for video conferencing, which would require both the presenter's and the viewer's browsers to have RTC capability.
After forming the processed screen images, the screen sharing server 140 transmits (222) the processed screen images to the viewer's device 180. The webpage script 182 on the viewer device 180 receives (224) the processed screen images, and displays (226) the images to the viewer.
Overall, the webpage script 165 utilize RTC capability to generate screenshot images and thereafter, through the system 105, share a screen with any browser. In this way, system 105 enables screen sharing without requiring special software, plug-ins, or special network configuration on either the viewer or the presenter. Because only basic images in formats that are natively displayable to the viewer's web browser 181 is being shared, the system 105 reduces or even removes the traditional problems around browser compatibility (e.g., requiring a particular plug-in or software on the viewer's browser, such as Java, Flash or even WebRTC), firewalls, network configuration, and viewer's computer user privileges (e.g., to install software or change browser configuration).

ADAPTIVE IMAGE ADJUSTMENT

FIG. 3 depicts a flowchart illustrating an example process 300 performed (e.g., by the presenter device 160 as being instructed by the webpage script 165, or by the screen sharing server 140) for adaptively adjusting the captured screen image.
It is further recognized in the present disclosure that, in addition to the obstacles of traditional screen sharing technique-installing software (e.g., Java), configuring firewalls and connection ports, and resolving different browsers' capabilities-another important limitation that may affect the user experience is network bandwidth. Because almost every network environment is different and its condition fluctuates, in order to maintain both the quality of the screen sharing experience as well as its efficacy, the webpage script 165 and/or the screen sharing server 140 can perform adaptive image adjustment based on the detected network bandwidth (and/or other network condition).
Accordingly, as an option or a variation, after the webpage script 165 captures screen images (via instructing the web browser 164), the webpage script 165 is further configured to detect (302) a network bandwidth between the presenter device 160 and the screen sharing server 140. Then, the webpage script 165 adaptively (or dynamically) adjust (304) the captured screen image so as to reduce an overall size of the data indicative of the captured screen image based on the network bandwidth between the presenter device and the server. For purposes of discussion herein, the terms “adaptively” and “dynamically” are used interchangeably; both terms generally mean that the function or action described is performed in response to a detected difference (e.g., a sudden drop of network speed) or that different functions are performed based on different detected levels (e.g., a 50 Mbps bandwidth versus a 500 Kbps bandwidth).
More specifically, one embodiment of the webpage script 165 can adjust the data (e.g., of the parsed pixels) by reducing (304A) an image resolution of the captured screen image. For example, by the webpage script 165 reducing an image resolution from 1000×1000 to 500×500, the webpage script 165 can reduce the raw data size to ¼. As a variation, the webpage script 165 can adjust the data by reducing (304B) a color depth (e.g., from 24-bit to 16-bit) of the captured screen image. In some embodiments, based on the results from the webpage script 165 detecting the bandwidth/condition from the presenter's side, the webpage script 165 can adaptively scale the image in both pixel dimensions and the color depth based on the available bandwidth.
In some examples, the webpage script 165 can also perform the adaptive screen image adjustments to maintain a certain frame rate. For example, in one embodiment, the webpage script 165 starts to scale down the captured screen images when the detected bandwidth is not enough to support a 2.5 frame-per-second frame rate. Note that the network bandwidth is merely one example factor on which the adaptive image adjustment may be based; other suitable factors may include, for example, an operating or a processing speed of the presenter device 160. Also, for some embodiments, the webpage script 165 can change (304C) the captured screen images to another image format, e.g., from .BMP to .PNG. In variations, the webpage script 165 and/or the screen sharing server 140 can perform one or more of these adjustments introduced here without being based on any condition or factor.
As an additional or alternative embodiment, the screen sharing server 140 may perform process 300 to improve the system 105's user experience for the viewer devices 180.
Similar to how the web script 165 performs adaptive image adjustment based on the detected network bandwidth between the presenter device 160 and the screen sharing server 140, the screen sharing server 140 can also perform adaptive image adjustment based on the detected network bandwidth between the screen sharing server 140 and one or more of the viewer devices 180. Specifically, the screen sharing server 140 can detect (302) a network bandwidth between a viewer device 180 and the server 140. Thereafter, the screen sharing server 140 can adaptively adjust (304) the processed screen image so as to reduce an overall size of the processed screen image based on the network bandwidth between the viewer device 180 and the server 140. In some embodiments, the adjusting of the processed screen image performed by the server 140 can be reducing (304A) an image resolution of the processed screen image. Additionally or alternatively, the adjusting of the processed screen image can be reducing (304B) a color depth of the processed screen image, or can be changing (304C) the format of the captured screen image to another format.

SCREEN CAPTURE REFRESH AND DIFFERENTIAL UPDATE

FIG. 4 depicts a flowchart illustrating an example process 400 performed by the presenter device (e.g., as being instructed by the webpage script 165) for capturing and transmitting screen images. The process 400 provides additional examples and details for implementing Step 214 of FIG. 2.
First, the screen sharing script 165 takes (402) a screenshot at presenter device 160 and records pointer location. As will be appreciated by a person skilled in the art, the present teaching does not involve continuous screen sharing or presentation video streaming, but rather periodic and/or triggered sharing of captured screen images or representations. Then, the script 165 determines (404) the “display area” for the presentation. In one or more embodiments, each time the screen sharing script 165 records the screen, the script first determines the “display area,” i.e., the specific portion of the screen that includes the content the presenter wants to share. Often the presenter would rather show only the active window, rather than the entire screen, because this requires less scrolling on the viewer's part, especially if the viewer's window is smaller than the presenter's, and because the presenter may get pop-ups and instant messages not intended for sharing. Some embodiments provide a mechanism for determining the active window, such as a screen sharing controller 520 or a screen sharing window selection interface 530, both illustrated in FIGS. 5B. In certain embodiments, this mechanism (e.g., the controller 520 or the selection interface 530) may be selectively triggered by the presenter.
The selection interface 530, which may be an introductory screen first shown to the presenter when he or she elects to share the screen, can enable the presenter to identify which exact window that the presenter would like to share. On the other hand, the controller 520 can allow the presenter to select and customize the active sharing area that the presenter so desires. The controller 520 may include a highlighted or bolded area to identify the currently active screen sharing area, such as the active area 540 illustrated in the screenshot of FIG. 5C. Further, the controller 520 can allow the presenter to customize the active area by using a mouse cursor, for example, to drag and drop on the controller interface so as to specify a new active area 550 for the screen image to be captured, such as shown in the screenshot of FIG. 5D.
One aspect of the suitable mechanism for determining the active window includes placing an identifying image on the webpage. The webpages (whether the introductory screen sharing script webpage 530 or newly launched webpages the presenter has identified for showing) can include small unique markings. These markings can include an image strategically located in the active window. For example, the image can be placed in the corners of the display area and include a unique combination of colors (like an image-based password). Alternatively, or in addition, a one-pixel wide or multiple-pixel wide strip of a specific color can connect the markings and frame the display area. Then, the screen sharing script 165 can determine where these markings and strips are located to determine the active window. If the markings cannot be found, or the window containing the markings is significantly obscured, the screen sharing script 165 can default to identifying the presenter's entire screen as the display area, or in some implementations, allowing the presenter to manually define the active window.
Using the aforementioned techniques (e.g., via the RTC API of the presenter's web browser 164), the screen sharing script 165 can capture representations of the display area at set intervals and/or at other suitable triggering events. Each time the screen sharing script 165 captures or records the screen, the screen sharing script 165 determines (406) what to send to the server 140.
More specifically, the screen sharing script 165 can perform (408) a full refresh of the entire display area. Alternatively, the screen sharing script 165 can perform (410) a differential update by determining which areas of the display area have changed and send just the changed areas (“differential update”). In some embodiments, differential updates are sent when appropriate because full refresh takes more time and bandwidth.
In some embodiments, the screen sharing script 165 sends a full refresh only in specific circumstances such as:

- (a) The presentation has just begun and the screen sharing script 165 is sending a first presentation.
- (b) The presentation display area has changed.
- (c) The screen sharing server 140 requested that the screen sharing script 165 send a full refresh, for example, when a new viewer comes online and requests a full refresh of the screen sharing server 140. Note that, in this way, the screen sharing server 140 does not need to maintain a “full, current image” of the presenter's screen.
- (d) Most (e.g., above a specific percentage) of the active window or captured representations has changed so that there is not a large difference between an incremental update and a full update.
- (e) The screen sharing script 165 has sent a maximum number of consecutive differential updates. In some embodiments, differential updates are restricted to a maximum number because certain layering methods that the viewer device 180 uses to update the screen may result in memory problems, and full updates can generally clean up minor display inconsistencies if any occurs for any reason (e.g. momentary network issues).

In the case of the differential update performed, the screen sharing script 165 compares each pixel of the current display area to the last display area, and determines the changes.
Thereafter, for each image or differential update, the screen sharing script 165 adjust the data indicative of the captured image (whether full or differential) in manners described above. Optionally, the script 165 can encode the images using one or more encoding methods so as to store the images into known formats, such as GIF, PNG, or JPEG. The screen sharing script 165 may optionally split the image or representation into multiple smaller images or representations that can be sent separately, as this may shorten the time it takes for the viewer to see at least a partial screen update. The screen sharing script 165 transmits (414) the data to the server 140. In some embodiments, the script 165 also transmits, along the data, other relevant information, including the presentation token, key, current mouse position relative to the current display area. The data can also include update images, which may include, for example, the x, y, width, height, sequence identifier and encoded image data, and/or a flag indicating a full-refresh versus differential update images. The screen sharing script 165 may choose to use a standard web encoding method, e.g. HTTP Multi-part POST, should it detect any firewall issues associated with non-standard ports or transmission methods. The screen sharing script 165 then receives a response from the screen sharing server 140, which may include a request for a refresh.
The screen sharing script 165 then waits (416) for a suitable trigger, e.g., wait period since last capture expired, to repeat the cycle at Step 406. In addition, if the screen sharing script 165 has just received a refresh request, it may start the cycle immediately, to minimize the viewer's initial wait time. In some embodiments, the script 165 can determine a frequency for refreshing the screen image based on an operating speed of a presenter device 160.
FIG. 6 depicts a diagrammatic representation of a machine in the example form of a computer system 600 within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed. The computer system 600 can represent any of the devices described above, such as the presenter device 160, the control server 120, one or more of the presentation servers in the screen sharing server cluster 140A-140N, or the viewer devices 180A-180N. As noted above, any of these systems may include two or more subsystems or devices, which may be coupled to each other via a network or multiple networks.
In the illustrated embodiment, the computer system 600 includes one or more processors, memory, a communication device, and one or more input/output (I/O) devices, all coupled to each other through an interconnect. The interconnect may be or include one or more conductive traces, buses, point-to-point connections, controllers, adapters and/or other conventional connection devices. The processor(s) may be or include, for example, one or more general-purpose programmable microprocessors, microcontrollers, application specific integrated circuits (ASICs), programmable gate arrays, or the like, or a combination of such devices. The processor(s) control the overall operation of the processing device. The memory may be or include one or more physical storage devices, which may be in the form of random access memory (RAM), read-only memory (ROM) (which may be erasable and programmable), flash memory, miniature hard disk drive, or other suitable type of storage device, or a combination of such devices. The memory may store data and/or instructions that configure the processor(s) to execute operations in accordance with the techniques described above. The communication device may be or include, for example, an Ethernet adapter, cable modem, Wi-Fi adapter, cellular transceiver, Bluetooth transceiver, or the like, or a combination thereof. Depending on the specific nature and purpose of the processing device, the I/O devices can include devices such as a display (which may be a touch screen display), audio speaker, keyboard, mouse or other pointing device, microphone, camera, etc.
Unless contrary to physical possibility, it is envisioned that (i) the methods/steps described above may be performed in any sequence and/or in any combination, and that (ii) the components of respective embodiments may be combined in any manner.
The techniques introduced above can be implemented by programmable circuitry programmed/configured by software and/or firmware, or entirely by special-purpose circuitry, or by a combination of such forms. Such special-purpose circuitry (if any) can be in the form of, for example, one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), etc.
Software or firmware to implement the techniques introduced here may be stored on a machine-readable storage medium and may be executed by one or more general-purpose or special-purpose programmable microprocessors. A “machine-readable medium”, as the term is used herein, includes any mechanism that can store information in a form accessible by a machine (a machine may be, for example, a computer, network device, cellular phone, personal digital assistant (PDA), manufacturing tool, any device with one or more processors, etc.). For example, a machine-accessible medium includes recordable/non-recordable media (e.g., read-only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; etc.), etc.

CONCLUSION

In the foregoing specification, the examples have been described with reference to specific exemplary implementations thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader scope of the disclosure. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense. Aspects of the disclosure can be combined, separated, and/or modified, to the extent that is necessary, to employ the systems, functions, and concepts of the various references described above to provide yet further embodiments of the disclosure.
Note that, while the methods introduced here include a number of operations that are discussed and/or depicted as performed in a specific order, these methods may include more or fewer operations, which can be performed in serial or in parallel. An order of two or more operations may be changed, performance of two or more operations may overlap, and two or more operations may be combined into a single operation.
Unless the context clearly requires otherwise, throughout the description and the claims, the words “comprise,” “comprising,” and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to.” As used herein, the terms “connected,” “coupled,” or any variant thereof, means any connection or coupling, either direct or indirect, between two or more elements; the coupling of connection between the elements can be physical, logical, or a combination thereof. Additionally, the words “herein,” “above,” “below,” and words of similar import, when used in this application, shall refer to this application as a whole and not to any particular portions of this application. Where the context permits, words in the above Detailed Description using the singular or plural number may also include the plural or singular number respectively. The word “or,” in reference to a list of two or more items, covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list.
As used herein, a “module,” “a manager,” an “interface,” a “platform,” or an “engine” includes a general purpose, dedicated or shared processor and, typically, firmware or software modules that are executed by the processor. Depending upon implementation-specific or other considerations, the module, manager, interface, platform, or engine can be centralized or its functionality distributed. The module, manager, interface, platform, or engine can include general or special purpose hardware, firmware, or software embodied in a computer-readable (storage) medium for execution by the processor.

Claims

What is claimed is:

1. A method comprising:

loading, by a presenter device, a screen sharing script directly executable by a web browser of the presenter device;

capturing, by the screen sharing script, a plurality of screen images;

adjusting the plurality of screen images based on a network bandwidth between the presenter device and a screen sharing server in order to maintain a specified frame rate of the plurality of screen images; and

transmitting, to the screen sharing server, the adjusted plurality of screen images.

2. The method of claim 1, wherein the screen sharing script is configured to issue instructions through an application programming interface of the web browser to capture the plurality of screen images.

3. The method of claim 1, further comprising:

adaptively adjusting the plurality of screen images to reduce an overall size of the plurality of screen images based on a network bandwidth between the presenter device and the screen sharing server.

4. The method of claim 1, further comprising:

adaptively adjusting the plurality of screen images to reduce an overall size of the plurality of screen images based on a network bandwidth between a viewer device and the screen sharing server.

5. The method of claim 1, wherein adjusting the plurality of screen images further comprises reducing an image resolution of a screen image of the plurality of screen images.

6. The method of claim 1, wherein adjusting the plurality of screen images further comprises reducing a color depth of a screen image of the plurality of screen images.

7. The method of claim 1, further comprising:

determining a frequency for refreshing the screen image based on an operating speed of the presenter device; and

capture a screen image at the determined frequency.

8. The method of claim 1, further comprising:

adjusting a screen area to be captured.

9. The method of claim 1, wherein transmitting, to the screen sharing server, the adjusted plurality of screen images further comprises:

determining a difference between a first screen image of the plurality of screen images and a second screen image of the plurality of screen images; and

transmitting to the screen sharing server one of: the second screen image or data indicative of the difference.

10. The method of claim 1, further comprising:

displaying one or more markings within the presentation webpage, wherein an active sharing area of the screen image is determined by the screen sharing script based on locations of the one or more markings within the presentation webpage.

11. The method of claim 1, further comprising:

transmitting, to the screen sharing server, data reflecting a position of a user-controlled pointer within an active sharing area of the screen.

12. A computer system implementing a presenter device, the computer system comprising:

a memory; and

a processor communicatively coupled to the memory, wherein the processor is configured to:

load a screen sharing script directly executable by a web browser of the presenter device;

capture, by the screen sharing script, a plurality of screen images;

adjust the plurality of screen images based on a network bandwidth between the presenter device and a screen sharing server in order to maintain a specified frame rate of the plurality of screen images; and

transmit, to the screen sharing server, the adjusted plurality of screen images.

13. The system of claim 12, wherein the screen sharing script is configured to issue instructions through an application programming interface of the web browser to capture the plurality of screen images.

14. The system of claim 12, wherein the processor is further configured to:

adaptively adjust the plurality of screen images to reduce an overall size of the plurality of screen images based on at least one of: a network bandwidth between the presenter device and the screen sharing server or a network bandwidth between a viewer device and the screen sharing server.

15. The system of claim 12, wherein the processor is further configured to:

16. The system of claim 12, wherein the processor is further configured to:

17. A non-transitory computer-readable storage medium comprising executable instructions that, when executed by a computer system implementing a presenter device, cause the computer system to:

capture, by the screen sharing script, a plurality of screen images;

18. The non-transitory computer-readable storage medium of claim 17, further comprising executable instructions that, when executed by the computer system, cause the computer system to:

19. The non-transitory computer-readable storage medium of claim 17, further comprising executable instructions that, when executed by the computer system, cause the computer system to:

display one or more markings within the presentation webpage, wherein an active sharing area of the screen image is determined by the screen sharing script based on locations of the one or more markings within the presentation webpage.

20. The non-transitory computer-readable storage medium of claim 17, further comprising executable instructions that, when executed by the computer system, cause the computer system to:

transmit, to the screen sharing server, data reflecting a position of a user-controlled pointer within an active sharing area of the screen.