KR20210018353A - A method, apparatus, and computer readable medium for sharing a desktop via a web socket connection within a networked collaboration workspace. - Google Patents

Abstract

A system, method, and computer-readable medium for sharing a desktop via a web socket connection within a networked collaboration workspace, accessible to multiple participants on multiple computing devices via a web socket connection and hosted on a server Transmitting a representation of the collaboration workspace, a request to share at least one portion of the local desktop of the local computing device in the collaboration workspace, and within the representation of the collaboration workspace Receiving a selection of one region, generating a streaming object configured to output a video stream of the at least one portion of the local desktop of the local computing device, and one or more commands through the web socket connection ) (The one or more commands include information corresponding to the streaming object and the selected area, and are configured to cause the server to insert the streaming object into the collaboration workspace in the selected area) to the server Includes steps.

Description

A method, apparatus, and computer readable medium for sharing a desktop via a web socket connection within a networked collaboration workspace.

The present invention relates to a method, apparatus, and computer-readable medium for propagating cropped images over a web socket connection in a networked collaborative workspace.

Operating systems and applications running within the operating system utilize external hardware devices to enable the user to provide input to the program and output to the user. Common examples of external hardware devices are keyboards, computer mice, microphones, and external speakers. These external hardware devices interface with an operating system through the use of a driver, which is a specialized software program configured to interface between the operating system and a hardware command used by a specific hardware device.

Applications will sometimes be designed to interface with certain hardware devices. For example, a speech-to-text word processing application can be designed to interface with an audio headset that includes a microphone. In this case, the application must be specifically configured to receive a voice command, to perform speech recognition, to convert the recognized word into text content, and to output text content to a document. This functionality will typically be implemented in an application's Application Programming Interface (API), which is a defined set of communication methods between various software components. In the example of a speech recognition application, the API includes an interface between the application program and software on the driver responsible for interfacing with the hardware device (microphone) itself.

One problem with existing software that utilizes specialized hardware devices is that the application or operating system software itself must be tailored, and in particular, must be designed to utilize the hardware device. This fit means that the hardware device cannot exceed the criteria defined for it by the application and cannot be utilized for a context other than the specific application for which it was designed to be used. For example, a user of a speech-to-text processing application can manipulate other application programs or other components within the operating system, unless the other application program or operating system is specifically designed to utilize voice commands received through a microphone. none.

1 shows an example of an existing architecture of a system that utilizes a combined hardware device for user input. The operating system 100A of FIG. 1 includes running applications 101A and 102A, each application having its own API 101B and API 102B, respectively. Operating system 100A also has its own API 100B, as well as specialized drivers 100C, 101C, and 102C configured to interface with hardware devices 100D, 101D, and 102D.

As shown in Fig. 1, the application API 101B is configured to interface with a driver 101C that itself interfaces with the hardware device 101D. Similarly, application API 102B is configured to interface with driver 102C, which itself interfaces with hardware device 102D. At the operating system level, the operating system API 100B is configured to interface with the driver 100C, which itself interfaces with the hardware device 100D.

The architecture of the system shown in Figure 1 limits a user's ability to utilize hardware devices other than certain application or operating system contexts. For example, the user cannot utilize hardware device 101D to provide input to application 102A, but utilize hardware device 102D to provide input to application 101A or to operating system 100A. Can't.

Thus, there is a need for improvements in hardware-software interfaces that allow the use of hardware devices in multiple software contexts.

It is an object of the present invention to provide a method, apparatus, and computer-readable medium for solving problems associated with previous hardware-software interfaces used in hardware devices. Specifically, another object of the present invention is to develop and provide a general-purpose hardware-software interface that enables a user to utilize a hardware device communicatively coupled in various software contexts.

According to an aspect of the present invention, in a method of propagating a cropped image through a web socket connection in a networked collaboration workspace, the method comprises: On the user interface, a representation of a collaboration workspace (the collaboration workspace contains one or more images) accessible to multiple participants on multiple computing devices via a web socket connection and hosted on a server. Transmitting; Detecting a user input (the user input corresponds to a plurality of coordinates) for selecting an image area of one image in the one or more images by a cropping tool executed on the local computing device; By the cropping tool running on the local computing device, a transparent layer running on the local computing device (the transparent layer includes at least one of an operating system or one or more applications configured to run on the operating system) Transmitting the plurality of coordinates to (including an application programming interface (API)) configured to interface; Capturing the image region based at least in part on the plurality of coordinates, by the transparent layer running on the local computing device; Detecting a second user input for dragging the selected image area to a location in the collaboration workspace by the transparent layer executed on the local computing device; And a plurality of commands to the server hosting the collaborative workspace by the transparent layer executed on the local computing device (the plurality of commands are configured to remove the image from the collaborative workspace, In addition, the image area is configured to be inserted into the collaborative workspace based at least in part on the location).

1 shows an example of an existing architecture of a system that utilizes a combined hardware device for user input.
Fig. 2 shows the architecture of a system utilizing a universal hardware-software interface, according to an exemplary embodiment.
3 shows a flow diagram for an implementation of a general purpose hardware-software interface, according to an exemplary embodiment.
4 is a user based at least in part on information captured by one or more hardware devices communicatively coupled to the system when information captured by one or more hardware devices includes one or more images, according to an exemplary embodiment. A flow chart for determining the input is shown.
Fig. 5A shows an example of object recognition, according to an exemplary embodiment.
Fig. 5B shows an example of determining input position coordinates according to an exemplary embodiment.
6 shows a flow chart for determining a user input based at least in part on information captured by one or more hardware devices communicatively coupled to the system when the captured information is sound information, according to an exemplary embodiment. .
Fig. 7 shows a Tool Interface, which may be part of a transparent layer, according to an exemplary embodiment.
8 shows an example of a stylus that may be part of a system, according to an exemplary embodiment.
Fig. 9 shows a flow chart for identifying a context corresponding to a user input, according to an exemplary embodiment.
Fig. 10 shows an example of determining a context using input coordinates, according to an exemplary embodiment.
Fig. 11 is a flow chart for converting a user input into a transparent layer command according to an exemplary embodiment.
Fig. 12A shows an example of receiving input coordinates when a selection mode is toggled, according to an exemplary embodiment.
Fig. 12B shows an example of receiving input coordinates when a pointing mode is toggled, according to an exemplary embodiment.
Fig. 12C illustrates an example of receiving input coordinates when a drawing mode is toggled, according to an exemplary embodiment.
Fig. 13 shows an example of a transparent layer command determined based on one or more words identified in input speech data, according to an exemplary embodiment.
Fig. 14 shows another example of a transparent layer command determined based on one or more words identified in input speech data, according to an exemplary embodiment.
Fig. 15 is a flow chart for executing one or more transparent layer commands on a transparent layer, according to an exemplary embodiment.
Fig. 16 shows an exemplary interface for adding a new command corresponding to a user input, according to an exemplary embodiment.
Fig. 17 shows various components and options of a drawing interface and a draw mode, according to an exemplary embodiment.
Fig. 18 shows a calibration and settings interface for a video camera hardware device that is used to recognize an object and allows a user to provide input using touches and gestures, according to an exemplary embodiment.
Fig. 19 shows a general settings interface that allows a user to customize various aspects of the interface, toggle input modes, and make other changes, according to an exemplary embodiment.
Fig. 20 shows a flow chart for sharing a desktop through a web socket connection within a networked collaboration workspace, according to an exemplary embodiment.
Fig. 21A shows a network architecture used to host and transport collaborative workspaces, according to an exemplary embodiment.
21B shows a process for propagating edits to a collaborative workspace within a network, according to an exemplary embodiment.
22 illustrates multiple representations of a collaborative workspace, according to an exemplary embodiment.
Fig. 23A shows an example of a user interface (desktop) of a local computing device prior to receiving a request and selection of one area, according to an exemplary embodiment.
Fig. 23B is a diagram showing an example of a user interface (desktop) of a local computing device after receiving a request but before selecting a region, according to an exemplary embodiment.
24A-24C show an example of a source selection process, according to an exemplary embodiment.
Fig. 25 shows a flowchart for creating a streaming object configured to output a video stream of at least one portion of a local desktop of a local computing device, according to an exemplary embodiment.
Fig. 26 shows a process of sending a command and propagating a streaming object from a local computing device, according to an exemplary embodiment.
Fig. 27 shows an example of an interface of a local computing device after a server embeds a streaming object in a collaboration workspace, according to an exemplary embodiment.
Fig. 28 shows a flow chart for controlling a desktop or a portion of a desktop from a local computing device via an embedded streaming object, according to an exemplary embodiment.
29A-29C illustrate an example of controlling a desktop or a portion of a desktop from a local computing device through an embedded streaming object, according to an exemplary embodiment.
Fig. 30 shows a flow chart for controlling a desktop or a portion of a desktop from a remote computing device via an embedded streaming object, according to an exemplary embodiment.
31A-31C illustrate an example of controlling a desktop or a portion of a desktop from a remote computing device through an embedded streaming object, according to an exemplary embodiment.
32 illustrates an example computing environment configured to perform the disclosed method.

Although a method, an apparatus, and a computer-readable medium are described herein as examples and embodiments, those skilled in the art will describe a method, apparatus, and computer-readable medium for implementation of a general-purpose hardware-software interface. Recognize that you are not limited. It is to be understood that the drawings and description are not intended to be limited to the specific form disclosed. Rather, its intent is to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the appended claims. Any subject matter used herein is for structural purposes only and is not intended to limit the scope of the description or claims. As used herein, the word "can" is used in the meaning of permissive (ie, means to have potential) rather than in the sense of force (ie, means must). . Similarly, the words "Include, Includes" and "Including" are meant to include, but are not limited to.

Applicants have discovered a method, apparatus, and computer-readable medium that solve problems associated with previous hardware-software interfaces used in hardware devices. Specifically, Applicants have developed a general-purpose hardware-software interface that allows a user to utilize a communicatively coupled hardware device in a variety of software contexts. The disclosed implementation eliminates the need for an application or operating system to be custom designed to interface with a specific hardware device through the use of specialized virtual drivers and corresponding transparent layers, as described in more detail below.

Fig. 2 shows the architecture of a system utilizing a universal hardware-software interface, according to an exemplary embodiment. As shown in FIG. 2, the operating system 200A includes a transparent layer 203 that communicates with a virtual driver 204. As described in more detail below, the transparent layer 203 is an API configured to interface between a virtual driver and an operating system and/or application(s) running on the operating system. In this example, the transparent layer 203 is between the virtual driver 204 and the API 201B of the application 201A, the API 202B of the application 202A, and the operating system API 200B of the operating system 200A. Interfacing at

The transparent layer 203 may be part of a software process running on an operating system, and its own user interface (UI) elements, including a transparent UI superimposed on the basic user interface and/or a visible UI element that a user can interact with. Can have

Virtual driver 204 is configured to emulate drivers 205A and 205B, which interface with hardware devices 206A and 206B, respectively. The virtual driver is a user input that instructs the virtual driver to emulate which virtual driver, for example, in the form of voice commands, selections made on the user interface, and/or gestures made by the user in front of the combined web camera. Can be received. For example, each of the connected hardware devices may operate in a "listening" mode, and each of the emulated drivers in the virtual driver 204 receives an initialization signal that serves as a signal to the virtual driver to switch to a specific emulation mode. It can be configured to detect. For example, stating that the user is "Start Voice Commands" can activate the driver corresponding to the microphone to receive new voice commands. Similarly, providing a predetermined gesture by a user may activate a driver corresponding to a web camera to receive a gesture input or a touch input.

The virtual driver may also be configured to interface with a native driver, such as a native driver 205C, which itself communicates with the hardware device 206C. In one example, hardware device 206C may be a standard input device, such as a keyboard or mouse, that is natively supported by the operating system.

The system shown in Figure 2 is a general-purpose hardware-software interface that allows a user to utilize any combined hardware device within a variety of contexts, such as a particular application or operating system, without the need for an application or operating system to interface with a hardware device. Allow implementation.

For example, hardware device 206A may capture the information, which is then received by virtual driver 204 that emulates driver 205A. Virtual driver 204 may determine user input based on the captured information. For example, if the information is a series of images that the user moves his hand, the virtual driver may determine that the user has performed a gesture.

Based on the identified context (eg, a specific application or operating system), the user can be converted to a transparent layer command and sent to the transparent layer 203 for execution. Transparent layer commands may include native commands within the identified context. For example, if the identified context is application 201A, the native command will be in a format compatible with application API 201B of application 201A. Execution of the transparent layer command may then be configured to cause execution of one or more native commands within the identified context. This is achieved by the transparent layer 203 interfacing with the operating system 200A as well as the respective APIs of the application running on the operating system API 200B. For example, when the native command is an operating system command such as a command for executing a new program, the transparent layer 203 may provide the native command to the operating system API 200B for execution.

As shown in Fig. 2, there is a two-way communication between all the components shown. This means, for example, that execution of a transparent layer command in the transparent layer 203 may result in the transfer of information to the virtual driver 204 and onto one of the connected hardware devices. For example, a voice command is recognized as input, converted to a transparent layer command containing a native command, executed by the transparent layer (causing the execution of the native command in the identified context), and then from the transparent layer to the speaker. A signal can be transmitted (via a virtual driver), allowing the sound output "Command Received" to be transmitted.

Of course, the architecture shown in FIG. 2 is for illustrative purposes only, and it is understood that the number of applications to be executed, the number and type of connected hardware devices, the number of drivers, and the emulated drivers may vary.

3 shows a flow diagram for an implementation of a general purpose hardware-software interface, according to an exemplary embodiment.

In step 301, user input is determined based at least in part on information captured by one or more hardware devices communicatively coupled to the system. A system, as used herein, may refer to one or more computing devices that perform steps of a method, devices that include one or more processors and one or more memories to execute steps of a method, or any other computing system. have.

User input can be determined by a virtual driver running on the system. As discussed above, a virtual driver may be operating in an emulation mode where it is emulating another hardware driver, thereby receiving information captured from a hardware device, or configured to interface with a specific hardware device. It is possible to selectively receive captured information from one or more other hardware drivers.

Motion capture device including camera, video camera, microphone, headset with two-way communication, mouse, touch pad, trackball, controller, game pad, joystick, touch screen, accelerometer and/or tilt sensor, remote, stylus ( Stylus), or any combination of these devices may be utilized. Of course, such a list of hardware devices is provided only as an example, and any hardware device that can be utilized to detect voice, image, video, or touch information may be utilized.

The communication coupling between the hardware device and the system can take many forms. For example, the hardware device may communicate with the system through a wireless network, Bluetooth protocol, radio frequency, infrared signal, and/or by a physical connection such as a Universal Serial Bus (USB) connection. Communication may also include both wireless and wired communication. For example, a hardware device may include two components, one of which wirelessly transmits a signal to a second component which itself is connected to the system via a wired connection (e.g., USB) (e.g., Bluetooth Through). Various communication techniques may be utilized depending on the system described herein, and these examples are not intended to be limiting.

The information captured by the one or more hardware devices may be any type of information, such as image information including one or more images, frames of video, sound information, and/or touch information. The captured information can be in any suitable format, such as a wav file or mp3 file for sound information, a jpeg file for an image, numerical coordinates for touch information, and the like.

The techniques described herein allow any display device to function as a virtually "touch" screen device in any context, even if the display device does not include any hardware for detecting touch signals or touch-based gestures. . This is described in more detail below and can be achieved through analysis of the images captured by the camera or video camera.

4 is a flow chart for determining user input based at least in part on information captured by one or more hardware devices communicatively coupled to a system when information captured by one or more hardware devices includes one or more images. Shows.

In step 401, one or more images are received. These images, as discussed above, may be captured by a hardware device such as a camera or video camera, and may be received by a virtual driver.

In step 402, objects in one or more images are recognized. The object may be, for example, a hand, a finger, or other body part of the user. The object may also be a special purpose device such as a stylus or pen, or a special purpose hardware device such as a motion tracking stylus/remote communicatively coupled to the system and including an accelerometer and/or tilt sensor. Object recognition may be performed by a virtual driver based on early training, for example through a calibration routine run using an object.

Fig. 5A shows an example of object recognition, according to an exemplary embodiment. As shown in FIG. 5A, image 501 includes a user's hand that was recognized as an object 502. The recognition algorithm can, of course, be configured to recognize different objects such as fingers.

Returning to Fig. 4, in step 403, one or more orientations and one or more positions of the recognized object are determined. This can be accomplished in a variety of ways. If the object is not a hardware device and instead is a body part such as a hand or finger, the object is mapped in a three-dimensional coordinate system using the camera's revealed position as a reference point, and the three-dimensional coordinates of the object and the X, Y, and Z axes It can be made to determine various angles for. If the object is a hardware device and includes motion tracking hardware such as an accelerometer and/or tilt sensor, the image information can be used in conjunction with the information represented by the accelerometer and/or tilt sensor to determine the position and orientation of the object. have.

In step 404, the user input is determined based at least in part on one or more orientations and one or more locations of the recognized object. This may include determining location coordinates on the transparent user interface (UI) of the transparent layer based at least in part on the one or more orientations and the one or more locations. The transparent UI is part of the transparent layer and is superimposed on the underlying UI corresponding to the operating system and/or any application running on the operating system.

5B shows an example of this step when the object is the user's finger. 5B, the display device 503 includes a basic UI 506 and a transparent UI 507 overlaid on the basic UI 506. For clarity, the transparent UI 507 is shown as having dot shading, but in practice it is understood that the transparent UI is a transparent layer that is not visible to the user. In addition, the transparent UI 507 is more than the basic UI 506. Although shown as being slightly smaller, in practice, it is understood that the transparent UI will cover the same screen area as the base UI.

As shown in FIG. 5B, the position and orientation information of the object (user's finger) is used to project a line onto the plane of the display device 503 and determine the intersection point 505. Image information captured by the camera 504 and the revealed position of the display device 503 under the camera can be used to aid in this projection. As shown in FIG. 5B, the user input is determined to be the input coordinates at the intersection point 505.

As discussed further below, the actual transparent layer command generated based on this input may be based on user settings and/or identified context. For example, the command may be a touch command indicating that the object at the coordinates of point 505 should be selected and/or opened. The command may also be a pointing command indicating that the pointer (eg, a mouse pointer) should be moved to the coordinates of the point 505. Additionally, the command may be an editing command that modifies the graphic output at that location (eg, to annotate an interface or draw a certain element).

Although FIG. 5B shows the recognized object 502 as being at a partial distance from the display device 503, a touch input may be detected regardless of the distance. For example, if the user has physically touched the display device 503, the technique described above will still determine the input coordinates. In that case, the Projection Line between the object 502 and the point of intersection will only be shorter.

Of course, touch input is not the only type of user input that can be determined from the captured image. Determining a user input based at least in part on one or more orientations and one or more positions of the recognized object may include determining a gesture input. Specifically, the position and orientation of the recognized object across multiple images can be analyzed to determine a corresponding gesture, such as a swipe gesture, a pinch gesture, and/or any revealed or matched gesture. . The user can calibrate the virtual driver to recognize specific contexts and custom gestures that map to commands within those contexts. For example, a user may create a custom gesture that maps to the operating system context and results in execution of a native operating system command that executes a particular application.

As discussed above, the information captured by the one or more hardware devices in step 301 of FIG. 3 may also include sound information captured by the microphone. 6 shows a flowchart for determining a user input based at least in part on information captured by one or more hardware devices communicatively coupled to the system when the captured information is sound information. As discussed below, speech recognition is performed on the sound information to identify one or more words corresponding to user input.

In step 601, sound data is received. Sound data, as discussed above, may be captured by a hardware device such as a microphone, and may be received by a virtual driver. In step 602, the received sound data may be compared to a sound dictionary. The sound dictionary may include a sound signature of one or more recognized words, such as command words or command modifiers. In step 603, one or more words in the sound data are identified as user input based on the comparison. The identified one or more words can then be converted into a transparent layer command and passed to the transparent layer.

As discussed above, the driver emulated by the virtual driver, the expected type of user input, and the command generated based on the user input may all be determined based at least in part on one or more settings or previous user input.

7 shows a tool interface 701 that may also be part of a transparent layer. Unlike the transparent UI, the tool interface 701 is visible to the user and can be used to select a native command generated based on user input or perform additional functions, among different options for changing the emulation mode of the virtual driver. have.

Button 701A is a drawing tool used by the user to graphically modify the user interface when the user input is input coordinates (e.g., coordinates based on the user touching the screening with his hand or stylus/remote) Allows you to select the type of (Drawing Tool). The various drawing tools can include different brushes, colors, pens, highlighters, and the like. These tools can result in graphic changes that change style, thickness, color, and more.

Button 701B allows a user to switch between selection, pointing, or drawing modes when input coordinates are received as user input. In the selection mode, the input coordinates can be treated as a "touch", resulting in selection or opening of an object at the input coordinates. In the pointing mode, the coordinates are treated as a pointer (eg, mouse pointer) location, which, in fact, allows the user to emulate a mouse. In the drawing mode, the coordinates can be treated as a location to change the graphic output of the user interface to present the appearance of the drawing or recording on the user interface. The nature of the change may depend on the drawing tool selected, as discussed with reference to button 701A. The button 701B may also change to the virtual driver to anticipate image input and/or motion input (if a motion tracking device is used) and, accordingly, emulate the appropriate driver.

The button 701C can be changed to a virtual driver to predict a voice command. This, as described with respect to FIG. 6, emulates a driver corresponding to a microphone to which a virtual driver is coupled, so that a voice input can be received and the voice input can be parsed.

The button 701D opens a launcher application, which may be part of a transparent layer and may be used to launch an application within the operating system or execute a specific command within the application. The launcher also customizes options within the transparent layer, such as custom voice commands, custom gestures, custom native commands for the application associated with user input and/or hardware device and user input (e.g., voice calibration, motion capture device calibration, and /Or object recognition calibration) can be used to calibrate.

Button 701E can be used to capture a screenshot of the user interface and export the screenshot as an image. This can be used with the drawing mode of the button 701B and the drawing tool 701A. After the user marks up a specific user interface, the marked-up version may be exported as an image.

The button 701F also allows graphic editing and can be used to change the color of the drawing or the aspect of the drawing the user is creating on the user interface. Similar to the Draw mode of button 701B, this button changes the nature of the graphic change in the input coordinates.

Button 701G cancels drawing on the user interface. Selecting this button can remove all graphic markings on the user interface and reset the underlying UI to the state it was in before the user created the drawing.

Button 701H can be used to launch a whiteboard application that allows a user to create or record a drawing using a drawing mode on a virtual whiteboard.

The button 701I may be used to add a text note to an object such as an object shown in an operating system UI or an application UI. Text notes can be interpreted from spoken signals or typed by the user using a keyboard.

Button 701J may be used to open or close tool interface 701. When closed, the tool interface can be minimized or removed entirely from the underlying user interface.

As discussed above, a stylus or remote hardware device may be used in the present system in conjunction with a camera or other hardware device such as a video camera. 8 shows an example of a stylus 801 that may be used in the system. The stylus 801 may communicate with the hardware receiver 802, for example via Bluetooth. The hardware receiver can be connected to the computer system, e.g. via USB 802B, and the signal transmitted from the stylus to the computer system via the hardware receiver can be used to control and interact with the menu 803, which The menu is similar to the tool interface shown in FIG. 7.

As shown in FIG. 8, the stylus 801 may include a physical button 801A. These physical buttons 801 can be used to power the stylus, navigate menus 803, and make selections. Additionally, stylus 801 may include a unique tip 801B that is captured in an image by a camera and recognized by a virtual driver. This may allow the stylus 801 to be used for drawing and editing when in the drawing mode. The stylus 801 may also include motion tracking hardware such as an accelerometer and/or tilt sensor to aid in position detection when the stylus is used to provide input coordinates or gestures. Additionally, the hardware receiver 802 may include a calibration button 802A, which, when pressed, may initiate a calibration utility in the user interface. This allows the stylus to be calibrated.

Referring again to Fig. 3, in step 302, a context corresponding to a user input is identified. The identified context includes either the operating system or an application running on the operating system.

Fig. 9 shows a flow chart for identifying a context corresponding to a user input, according to an exemplary embodiment. As shown in FIG. 9, operating system data 901, application data 902, and user input data 903 can all be used to determine context 904.

The operating system data 901 may include, for example, information about an active window in the operating system. For example, if the active window is a calculator window, the context may be determined to be a calculator application. Similarly, if the active window is a Microsoft Word window, the context can be determined to be a Microsoft Word application. Meanwhile, when the active window is a file folder, it may be determined that the active context is an operating system. Operating system data may also include additional information such as the currently running application, the last launched application, and any other operating system information that may be used to determine the context.

The application data 902 may include, for example, information about one or more running applications and/or information for mapping a specific application to a predetermined type of user input. For example, the first application may be mapped to a voice input so that the context is automatically determined to be the first application each time a voice command is received. In another example, a particular gesture is associated with a second application, and when the gesture is received as input, the second application is launched or closed, or some action within the second application is performed.

User input 903 can also be used to determine the context in a variety of ways. As discussed above, certain types of user input may be mapped to certain applications. In the above example, the voice input is associated with the context of the first application. Additionally, the nature of the user input can also be used to determine the context. Gestures or motions can be mapped to an application or to an operating system. Certain words in spoken commands can also be mapped to an application or to an operating system. Input coordinates can also be used to determine the context. For example, a window in the user interface at the location of the input coordinates may be determined, and the application corresponding to that window may be determined as the context.

10 shows an example of determining a context using input coordinates. As shown in FIG. 10, the display device 1001 is displaying a user interface 1002. Further, a camera 1004 is shown, and a transparent layer 1003 is overlaid on the basic user interface 1003. The user utilizes the stylus 1000 to point to the location 1005 in the user interface 1002. Since location 1005 lies within the application window corresponding to application 1, application 1 may be determined to be the context for user input as opposed to application 2, application 3, or the operating system.

Referring again to FIG. 3, in step 303, user input is converted into one or more transparent layer commands based at least in part on the identified context. As discussed above, the transparent layer includes an application program interface (API) configured to interface between the virtual driver and the operating system and/or applications running on the operating system.

11 shows a flowchart for converting a user input into a transparent layer command. As shown in step 1104 of FIG. 11, the transparent layer command may be determined based at least in part on the identified context 1102 and user input 1103. The transparent layer command may include one or more native commands configured to execute in one or more corresponding contexts. The transparent layer command may also include a response output to be sent to the virtual driver and onto the hardware device(s).

The identified context 1102 can be used to determine which transparent layer command should be mapped to user input. For example, if the identified context is "operating system", then the user interface (by minimizing one open window and maximizing the next one) is transparent, resulting in the user interface scrolling through the currently open window within the operating system. It can be mapped to a layer command. Alternatively, if the identified context is a "web browser application", the same swipe gesture may be mapped to a transparent layer command that results in the web page being scrolled.

User input 1103 also determines transparent layer commands, since user inputs are specifically mapped to certain native commands in one or more contexts and these native commands are part of the transparent layer commands. For example, the voice command "Open email" can be mapped to a specific operating system native command to launch the email application Outlook. When a voice input comprising the recognized word "open email" is received, this results in a transparent layer command comprising a native command to launch Outlook is determined.

As shown in FIG. 11, the transparent layer command may also be determined based on one or more user settings 1101 and API library 1104. The API library 1104 can be used to look up the identified context and native commands corresponding to specific user input. In the example of a swipe gesture and web browser application context, the API library corresponding to the web browser application may be queried that the appropriate API will cause the web page to scroll. Alternatively, the API library 1104 can be omitted, and native commands directed to specific user input and identified contexts can be mapped.

In a situation where the user input is determined to be the input coordinates, the transparent layer command is determined based at least in part on the input location coordinates and the identified context. In this case, the transparent layer command may include at least one native command in the identified context, and in this case, the at least one native command is configured to perform a certain operation at a corresponding position coordinate in the basic UI.

When there is more than one possible action mapped to a particular context and user input, the setting 1101 can be used to determine the corresponding transparent layer command. For example, button 701B of FIG. 7 allows the user to select between selection, pointing, or drawing modes when input coordinates are received as user input. This setting can be used to determine the transparent layer command, and furthermore, to determine which native command is performed and which operation is performed. In this case, possible native commands are a selection command configured to select an object associated with a corresponding position coordinate in the base UI, a pointer command configured to move the pointer to a corresponding position coordinate in the base UI, and a corresponding position in the base UI. It may include a graphic command configured to change the display output in coordinates.

12A shows an example of receiving the input coordinates when the selection mode is toggled. As shown in FIG. 12A, the user instructed the stylus 1200 in the operating system UI 1202 (with the superimposed transparent UI 1203) on the display device 1201. Similar to the previous example, the camera 1204 may be used to determine position and orientation information for the stylus 1200 and input coordinates. Since the selection mode is toggled and the stylus 1200 is directed to the folder 1205 in the operating system UI 12102, the determined transparent layer command is the native operating system to select the object associated with the input coordinates, in this case the folder 1205. May contain commands. In another example, if the window was positioned at the input coordinates, this would result in selection of the entire window.

12B shows an example of receiving input coordinates when the pointing mode is toggled. In this case, the determined transparent layer command may include a native operating system command to move the mouse pointer 1206 to the position of the input coordinate.

12C shows an example of receiving input coordinates when the drawing mode is toggled and the user swipes the stylus 1200 over a plurality of input coordinates. In this case, the determined transparent layer command includes a native operating system command to change the display output at each location of the input coordinates, resulting in the user drawing a line 1207 on the user interface 1202. . The modified graphic output generated in the drawing mode may be stored as a part of the transparent layer 1203, for example, as metadata related to the path of the input coordinates. The user can then select the option to export the modified display output as an image.

In a situation in which user input is identified as a gesture, converting the user input to one or more transparent layer commands based at least in part on the identified context determines the transparent layer command based at least in part on the identified gesture and the identified context. May include. The transparent layer command may include at least one native command within the identified context, wherein the at least one native command is configured to perform an action associated with the identified gesture within the identified context. An example of this is discussed above for web browser application context and swipe gestures resulting in the native command being configured to perform a scroll operation in the web browser.

In situations in which user input is identified as one or more words (e.g., by using speech recognition), converting the user input to one or more transparent layer commands based at least in part on what is identified may be performed on the identified one or more words and the identified context. And determining the transparent layer command based at least in part on. The transparent layer command may include at least one native command within the identified context, wherein the at least one native command is configured to perform an action associated with the identified one or more words within the identified context.

13 shows an example of a transparent layer command 1300 determined based on one or more words identified in input speech data. The identified word 1301 includes one of the phrases “whiteboard” or “blank page”. The transparent layer command 1300 also includes a description 1302 of the command, and a response instruction 1303, which is an output instruction sent by the transparent layer to the virtual driver and to the hardware output device upon execution of the transparent layer command. Additionally, the transparent layer command 1300 includes an actual native command 1304 used to invoke the whiteboard function.

Fig. 14 shows another example of a transparent layer command 1400 determined based on one or more words identified in input voice data, according to an exemplary embodiment. In this example, one or more words are "open email". As shown in Fig. 14, the transparent layer command 1400 includes a native command "outlook.exe" which is an instruction for executing a specific executable file for executing an Outlook application. The transparent layer command 1400 also includes a voice response "Email Opened" to be output in response to receiving the voice command.

Referring again to FIG. 3, in step 304, one or more transparent layer commands are executed on the transparent layer. Execution of one or more transparent layer commands is configured to cause execution of one or more native commands within the identified context.

Fig. 15 is a flow chart for executing one or more transparent layer commands on a transparent layer, according to an exemplary embodiment. In step 1501, at least one native command in the transparent layer command is identified. The native command can be designated as a native command in the structure of the transparent layer command, for example, to allow identification.

In step 1502, at least one native command is executed in the identified context. This step may include passing at least one native command to the identified context via the identified API for the identified context, and executing the native command within the identified context. For example, if the identified context is an operating system, the native command may be passed to the operating system for execution through the operating system API. Additionally, if the identified context is an application, the native command may be passed to the application for execution through the application API.

Optionally, in step 1503, a response may be sent to the hardware device(s). As discussed above, this response can be routed from the transparent layer to the virtual driver and onto the hardware device.

16-19 illustrate additional features of the system disclosed herein. Fig. 16 shows an exemplary interface for adding a new command corresponding to a user input, according to an exemplary embodiment. The dashboard within interface 1600 includes an icon of an application 1601 that has already been added and can be launched using predetermined user input and hardware devices (eg, voice commands). The dashboard may also show other commands that are application specific and are mapped to certain user inputs. Selection of the add button 1602 opens the add command menu 1603. These menus allow the user to choose between the following options: Item Type: Fixed item to add on the lower bar menu/generic item to add to the drag menu; Icon: Select an image icon; Background: Select the background icon color; Color: Select the icon color; Name: Set a new item name; Voice command: Set a voice activation command to open a new application; Feedback Response: Set the application voice response feedback; Command: Select and start an application type or a custom command type (eg, start an application command, perform an operation within an application command, close an application command, etc.); Process start: When launching a new process or application, the name of the process or application; And parameters: any parameters to be passed to the new process or application.

Fig. 17 shows a drawing interface 1700 and various components and options of a drawing mode, according to an exemplary embodiment. FIG. 18 shows a calibration and setup interface 1800 for a video camera hardware device that is used to recognize an object and allows a user to provide input using touch and gestures. 19 shows a general settings interface 1900 that allows a user to customize various aspects of the interface, toggle input modes, and make other changes. As shown in interface 100, a user can also access a settings page to calibrate and adjust settings for a hardware stylus (referred to as "Magic Stylus").

The systems disclosed herein can be implemented on multiple networked computing devices and can be used to assist in conducting networked collaboration sessions. For example, the whiteboard function described above may be a shared whiteboard between multiple users on multiple computing devices.

However, one of the problems with existing whiteboards or other shared collaboration spaces is that there is no easy way to interact with a remote computing device or share a desktop screen without interrupting or interrupting a collaboration session. . For example, if a participant in a collaboration workspace wants to share a presentation with another participant, all participants must minimize or close the collaboration session and launch a screen sharing application to join the screen sharing meeting. The collaboration space is designed to facilitate the workflow and shared brainstorming sessions, but this is often interrupted for the above reasons during shared collaboration sessions.

In addition to the above-described methods and systems for implementing a universal hardware-software interface, the Applicant has additionally discovered a method, apparatus and computer-readable medium capable of sharing a desktop via a web socket connection within a networked collaborative workspace. I did.

Fig. 20 shows a flow chart for sharing a desktop through a web socket connection within a networked collaborative workspace, according to an exemplary embodiment. All of the steps shown in FIG. 20 may be performed on a local computing device, such as a client device connected to a server, and do not require a plurality of computing devices. The disclosed process can also be implemented by multiple devices connected to the server.

In step 2001, a representation of the collaborative workspace hosted on the server is transmitted on the user interface of the local computing device. The collaboration workspace is accessible to a plurality of participants on a plurality of computing devices through a web socket connection, the plurality of participants including a local participant at a local computing device and one or more remote participants at a remote computing device. A remote computing device and a remote participant, as used herein, refers to a computing device and participant other than the local computing device and the local participant. The remote computing device is separated from the local device by a network such as a wide area network (WAN).

21A shows a network architecture used to host and transport collaborative workspaces, according to an exemplary embodiment. As shown in Fig. 21A, the server 2100 is connected to the computing devices 2101A to 2101F. The server 2100 and the computing devices 2101A to 2101F may be connected through a network connection such as a web socket connection that allows two-way communication between the computing devices 2101A to 2101F (client) and the server 2100. 21A, the computing device may be any type of computing device, such as a notebook, desktop, smart phone, or other mobile device.

The collaborative workspace may be, for example, a digital whiteboard configured to propagate any edits from any of the plurality of participants to other participants via a web socket connection. 21B shows a process for propagating edits to a collaborative workspace within a network, according to an exemplary embodiment. 21B, when a user at the computing device 2101B makes an edit or change to a collaborative workspace, this edit or change 2102B is transmitted to the server 2100, where the workspace Used to update the hosted version of. Thereafter, edits or changes are propagated by server 2100 as updates 2102A, 2102C, 2102D, 2102E, 2102F to other connected computing devices 2101A, 2101C, 2101D, 2101E, 2101F.

Each representation of the collaboration workspace may be a version of the collaboration workspace tailored to local participants. For example, as described above, each representation of the collaboration workspace may include one or more remote participant objects corresponding to one or more remote computing devices connected to the server.

22 is a diagram illustrating a plurality of representations of a collaborative workspace according to an embodiment. As shown in FIG. 22, the server 2200 hosts a collaborative workspace 2201. The version of the collaborative workspace hosted on the server is propagated to connected devices, as described above. In addition, FIG. 22 shows a representation of a collaboration workspace for three connected users (User 1, User 2, and User 3). Each representation can optionally be tailored to a local participant (to a local computing device at each location).

Returning to Fig. 20, in step 2002, a request to share at least one portion of the local desktop of the local computing device in the collaboration workspace and one area in the representation of the collaboration workspace by the local computing device. A selection of is received.

23A and 23B are for receiving a request to share at least one portion of a local desktop of a local computing device in a collaboration workspace and a selection of one area in a representation of the collaboration workspace, according to an exemplary embodiment. An example of steps is shown.

Fig. 23A shows an example of a user interface (desktop) of a local computing device prior to receiving a request and selection of one area, according to an exemplary embodiment. As shown in FIG. 23A, the user interface 2301 includes a collaboration application 2302 that locally displays a representation of a collaboration workspace 2303 hosted on a server, as well as a separate presentation application 2308 ( Examples: Powerpoint ^TM ) and separate document editing applications (eg Word ^TM ). All user applications running on the local computing device are displayed as tabs on the taskbar 2306 of the operating system (“OS”) in addition to the operating system menu button that brings up a menu associated with the operating system.

Collaboration application 2302 may include a representation of the collaborative workspace 2303, including all edits and contributions by local participants and any other participants, as well as toolbar 2304. The toolbar 2304 may include various editing tools, settings, commands, and options for interacting with or configuring a representation of a collaborative workspace. For example, the toolbar 2304 may include an editing tool to draw on the representation of the collaborative workspace 2303, and the editing is propagated to the server and other connected computing devices via a web socket connection.

Additionally, the toolbar 2304, when the toolbar 2304 is selected, causes the local computing device to receive a request to share at least one portion of the local desktop of the local computing device in the collaborative workspace, a screen sharing button ( 2305). Accordingly, the user can initiate screen sharing within the collaborative workspace by selecting the screen sharing button 2305.

Fig. 23B is a diagram showing an example of a user interface (desktop) of a local computing device after receiving a request but before selecting a region, according to an exemplary embodiment. As shown in FIG. 23B, selection of the screen sharing button 2305 may cause the area window 2309 to appear in the representation of the collaboration workspace 2303. The area window 2309 determines a result output area for screen sharing of the local desktop (or a part of the local desktop), and may be moved or aligned in terms of size, shape, direction, position, etc. by a user. After selecting the location/size/shape of the window 2309, the user may confirm the selection through a predetermined input (eg, pressing a pointing device, reselecting the button 2305, or other input). Thereafter, the selected area including related parameters (size, shape, direction, etc.) in the collaboration workspace may be received by the local computing device. Optionally, the area may be set to a predetermined default value including a default size, position, and direction, and may be additionally configured by the user when the user wants to leave the area.

Of course, the process shown in FIGS. 23A and 23B is to receive a request to share at least one portion of the local desktop of a local computing device within the collaboration workspace and a selection of one area within the representation of the collaboration workspace. , It is just one example. These steps can be implemented in a variety of ways. For example, instead of the screen share button 2305 being selected, it may be dragged into the collaboration workspace 2303. The screen sharing request may be initiated by a user using a keyboard command or a predetermined input command (the command may be recognized as a screen sharing request by a collaborative application), such as a selection in a menu or sub-menu. The request to initiate screen sharing within the collaborative workspace may be initiated even after a separate screen sharing session has already started. For example, the user may drag a taskbar tab, icon, or screen sharing window to a location within the collaboration workspace, so that the computing device receives both the selection and request of one area within the collaboration workspace.

Receiving a request for sharing a selection of at least one portion of a local desktop of a local computing device and a selection of an area within a representation of a collaborative workspace may include a source for screen sharing, for example, the entire desktop It may include a sub-step of allowing a source to be selected, such as whether to share an output, whether to share one or more windows within a desktop, or an output associated with one or more applications running on a local computing device. These sub-steps may include sending a source selection interface within the user interface, the source selection interface receiving a selection of at least one portion of the local desktop, and receiving a selection of at least one portion of the local desktop within the source selection interface. It is configured to receive a selection.

24A-24C show an example of a source selection process, according to an exemplary embodiment. 24A shows a user interface (desktop) 2406 of the local computing device before the user selects any screen share command or button. Numerals 2401-2408 represent the same elements as numbers 2301-2308 in FIG. 23A described above.

24B shows the user interface 2406 after the user selects the screen share button 2405. As shown in FIG. 24B, the source selection interface 2409 allows a user to select whether to share the entire desktop or a portion of the desktop, and which part of the desktop, within the collaboration workspace 2403, It may be transmitted within the collaboration application 2404. The source selection interface can list all currently active applications running on the local computing device, as well as any window (e.g., a window corresponding to an operating system or a window created by an application), allowing the user to It allows the user to choose whether to share one or more interfaces corresponding to one or more applications running on the local computing device, or one or more windows within the local desktop. For example, when a user selects an application to be shared, all interfaces (eg, windows, prompts, displays, etc.) associated with the application may be shared. If the user chooses to share a single window, only that window is shared. In addition, if the user chooses to share the entire desktop, the content of the entire desktop can be shared with other participants.

24C shows the interface 2401 after the user selects "a document editing app" in the selection interface 2409. This selection will designate the document editing app as the source of the screen sharing stream, allowing other participants in the collaborative workspace to see the interface corresponding to the document editing application running on the local computing device. The selection may be stored in memory and/or communicated to an application or program used to create a streaming object that captures the relevant portion of the desktop, as discussed further below.

24A-24C, the source selection step described above may be performed before or after, as part of the selection of one region discussed with respect to FIGS. 23A-23C. For example, the system may display the source selection interface after the user selects one area of the screen sharing window. Alternatively, the source selection interface can be displayed prior to selecting one region. Also, the source selection process may be performed at a later stage of the entire process, for example when a streaming object is created.

The source selection process may be omitted (set the entire desktop sharing as default) and/or may be performed in another way. For example, instead of displaying the source selection interface, a prompt may be displayed, instructing the user to select all active windows to be shared or to input a command to share the entire desktop. Various variations are possible and these examples are not intended to be limiting.

The input described in connection with step 2002, FIGS. 23A and 23B, and FIGS. 24A-24C may be received through any type of pointing device such as a mouse, touch screen, or stylus. Input can be detected using the techniques described above, including virtual drivers and/or transparent layers. For example, the input may be a user's pointing gesture. Further, the above-described operations, such as a drag-and-drop operation, selection, deselection, or other input or input sequence, may be input using the above-described techniques including virtual drivers and/or transparent layers.

Turning to Fig. 20, in step 2003, a streaming object configured to output a video stream of at least one portion of a local desktop of a local computing device is created. The streaming object may be a media stream, such as a video stream configured to capture a stream of at least one portion of the local desktop.

As described above, a representation of a collaboration workspace hosted on a server may be transmitted to a local computing device by a local collaboration application running on the local computing device. Such a collaborative application can be, for example, a web application, and can communicate and interface with a screen capture program on a local computing device. The screen capture program is a program configured to create a stream of at least a portion of the desktop. Collaborative applications can interface with screen capture programs through an application programming interface (API). Collaborative applications can also interface with screen capture programs through a transparent layer, which interfacing itself with multiple applications running on local computing devices. The functionality of the screen capture program used to create the media stream can be integrated into the collaborative application, allowing the collaborative application to instantiate the streaming object simply by calling the relevant routine or process.

Fig. 25 shows a flowchart for creating a streaming object configured to output a video stream of at least one portion of a local desktop of a local computing device, according to an exemplary embodiment.

In step 2501, the local collaboration application transmits a request for a source identifier to the screen capture program running on the local computing device through an API between the local collaboration application and the screen capture program. As mentioned above, this API can itself be a transparent layer. These requests may include additional attributes, such as the selected source of the screen sharing stream (eg, a specific application or window). Also, the source selection process can be performed after the request has been submitted or omitted on behalf of the default source (eg, the entire desktop). The source identifier is the handle or address of the media stream to be created, and allows the application to access the output of the media stream and the screen sharing results.

In step 2502, the screen capture program initiates a stream of at least one portion of the local desktop of the local computing device, the stream having a corresponding source identifier. If source parameters are provided to the screen capture program, the screen capture program can initiate the stream using only the identified components (eg, specific application or window). Otherwise, the screen capture program may initiate a stream of the entire local desktop to the user with the default option or the current source selection option as described above. The disclosed stream is a series of screen captures that periodically (eg 30 times per second) capture snapshots of at least one portion of the desktop. The stream can be accessed using a source identifier, which is a handle that allows a program to access the stream, as described above.

In step 2503, the screen capture program sends the source identifier to the local collaboration application. In step 2504, the local collaboration application creates a streaming object based at least in part on the source identifier. In addition to the source identifier, the local collaboration application may create a streaming object by selectively utilizing previously provided information such as an area designated by the user. A streaming object is a media stream with a defined format and a corresponding output interface. Optionally, the defined format can be based on user input, such as a selected area. The streaming object is a media stream object that is configured and compatible to be embedded within a collaborative workspace, similar to the video stream of a participant camera.

A screen capture program is a program that is configured to generate a stream of a local desktop or a portion of a local desktop, or is a program that creates a component integrated into a local collaboration application and configured to generate a stream of a local desktop or a portion of the local desktop. For example, the screen capture program may be a web browser or a browser engine component, which is the base or endpoint of Web Real-Time Communication (WebRTC) streaming. The next section provides an example of the steps for creating a streaming object when the screen capture program is Chrome.

Chrome's screen capture functionality can be accessed through the MediaDevices.getUserMedia() function interface. The gUM function is called once to retrieve the user audio/video stream, and can be called a second time to get the screen stream.

In Chrome, permission to use the screen capture function can be granted by using the Chrome extension in a web application (eg, one possible implementation of a collaborative application). The extension returns the sourceID using the chrome.desktopCapture.chooseDesktopMedia() function. Thereafter, the corresponding stream can be searched using sourceID as an argument of the gUM function.

The extension for screen sharing may include a content script executed in the context of a collaborative application and a background script executed in a separate extension context. Content scripts can communicate with collaborative applications by sending messages to windows or manipulating the document object model (DOM), but background scripts cannot. Background scripts can access all Chrome extension APIs, but content scripts cannot. Content scripts and background scripts can communicate with each other through the chrome.runtime.connect() function. Given such an architecture, the process of creating a streaming object configured to output a video stream of at least one portion of a local desktop of a local computing device can be performed in the following steps:

(1) the collaboration application sends a request for the screen sharing source identifier to the content script;

(2) the content script forwards the request to the background script;

(3) The background script calls the chrome.desktopCapture.chooseDesktopMedia() function and returns the source identifier to the content script;

(4) The content script returns this to the collaborative application, and finally calls the getUserMedia() function, using the source identifier as one of the constraints/arguments.

For Chrome's gUM function, the constraint for the video stream is {chromeMediaSource:'desktop'; maxWidth: 1920; maxHeight: 1080; maxFrameRate: 10; minAspectRatio: 1.77; chromeMediaSourceId: sourceId} or {maxWidth: 1920; maxHeight: 1080; maxFrameRate: 10; minAspectRatio: 1.77; Can include chromeMediaSourceId: sourceId}.

The screen sharing gUM call returns a mediaStream that can be shared over a peer connection as a WebRTC mediaStream.

Of course, the above-described implementation using the Chrome browser as a screen capture program is provided as an example only, and the step of creating a streaming object is to use another program or browser that supports a screen capture function, such as Firefox, or a separate standalone screen capture program. Can be done using.

Returning to FIG. 20, in step 2004, the local computing device sends one or more commands to the server through a web socket connection. The one or more commands may include a streaming object and information corresponding to the selected area, and are configured to cause the server to insert the streaming object into the collaborative workspace based at least in part on the selected area.

For example, if the user previously selected the circular area in the lower right corner of the collaborative workspace as the selected area for screen sharing, a streaming object will be inserted into the collaborative workspace by the server, and the media stream will be inserted into the collaborative workspace. When embedded, it can be displayed in a circular format in the lower-right corner of the collaboration workspace. The size and direction of the circle can be based on the same properties of the selected area. Of course, like any other object in the collaborative workspace, the streaming object interacts with the representation of the collaborative workspace to be embedded within the collaborative workspace and then manipulated or moved by the participant.

The format of the streaming object in the collaboration workspace may be determined based on the previously selected area, including attributes of the selected area such as shape, size, and location. These attributes may be transmitted along with the streaming object itself in one or more commands sent to the server. Thereafter, the server may determine the insertion point and format for embedding the streaming object in the collaboration workspace based on these attributes.

Alternatively, the streaming object may be a media stream object having spatial properties predefined based on the user's selection of a previous area. In this case, when the streaming object is created in the local computing device, display properties of the streaming object may be integrated into the streaming object. Thereafter, the streaming object (including the embedded spatial attribute) may be transmitted to the server, which embeds the streaming object into the collaboration workspace in an appropriate format at an appropriate location based on the embedded spatial attribute.

Instead of including the streaming object itself, the one or more commands may optionally include an address of the streaming object or other identifier available in the server to search for the streaming object or instantiate its own instance of the streaming object.

When a streaming object is inserted into the collaboration workspace by the server, the representation of the streaming object is propagated to a plurality of computing devices through a web socket connection. Therefore, each connected computing device has a representation of a streaming object in each representation of the collaboration workspace.

The inserted streaming object receives a video stream of at least one portion of the local desktop of the local computing device, and transmits the video stream of at least one portion of the local desktop of the local computing device to a plurality of computer devices through a web socket connection. Is configured to

As described above, this process is transmitted to the server from the local computing device (identified as the source of the media stream by the stream identifier) that instantiated the streaming object, and then connected to the server in the representation of the collaborative workspace. It includes stream information transmitted to each of the computing devices. Thus, the streaming object itself can be embedded within the server's collaborative workspace, and the resulting stream can be propagated to connected clients.

Fig. 26 shows a process of sending a command and propagating a streaming object from a local computing device, according to an exemplary embodiment. As illustrated in FIG. 26, the local computing device 2601 transmits a command (including either a streaming object or a reference/pointer to a streaming object) to the server 2600. Thereafter, the server 2600 inserts the streaming object into the collaboration workspace so that the streaming object embedded in the collaboration workspace is propagated to all connected devices including the local computing device 2601 and remote computing devices 2602, 2603. do.

Fig. 27 shows an example of an interface of a local computing device after a server embeds a streaming object in a collaboration workspace, according to an exemplary embodiment. Numbers 2701-2708 correspond to the same elements described in connection with numbers 2301-2308 in FIG. 23A. 27 further shows an embedded streaming object 2709 that displays a media stream of the user's desktop. In this case, it is assumed that the selected source is the entire desktop. Each remote participant connected to the server will have the same streaming object embedded within the representation of the collaboration workspace. As shown in Fig. 27, the stream of embedded results is a "picture-in-picture" that allows both the local participant and the remote participant to view the contents of the shared screen within the context of the collaboration workspace. "Provides an effect. Thus, participants can share related programs and information without interrupting the collaboration session.

In addition to the techniques described above, Applicants have further discovered a new technique that allows both local and remote participants to control the desktop or portions of the desktop displayed within the embedded streaming object. This new technology utilizes a transparent layer to allow users (both local and remote) to effectively browse the desktop or parts of the desktop that appear within the embedded streaming object.

Fig. 28 shows a flow chart for controlling a desktop or a portion of a desktop from a local computing device via an embedded streaming object, according to an exemplary embodiment.

In step 2801, the inserted streaming object is transmitted into a representation of the collaborative workspace on the user interface of the local computing device. The inserted streaming object is associated with the network address of the source of the video stream. This association can be provided by the server in the form of tags or metadata associated with the streaming object. In addition, the association may be part of the streaming object, for example, based on the source identifier described above. For example, when a streaming object is generated, a device generating the streaming object may include a tag indicating the IP address of the device.

In step 2802, the transparent layer executed on the local computing device detects a user input associated with the inserted streaming object (the user input corresponds to one location in the local desktop). As described above, the transparent layer includes an API, configured to interface with one or more of the operating system or one or more applications configured to run on the operating system. The transparent layer may detect a user input associated with the inserted streaming object based on the location of the input (determined by coordinates) and the location of the streaming object. For example, if a mouse click and a portion of the streaming object overlap, this input may be detected as a user input associated with the inserted streaming object.

The user input may be additionally mapped to a specific location in the local desktop based on the location of the input in the inserted streaming object. Once again, a map may be stored indicating the area or coordinates within the inserted streaming object associated with different parts of the local desktop. The location can be mapped to each part of the local desktop. For example, the sub-region of the inserted streaming object may be associated with a specific application occupying a corresponding region in the local desktop, or may be associated with a corresponding coordinate in the local desktop.

The mapping procedure may utilize a scaling mechanism or process that detects the relative position of the input within the inserted streaming object and maps the relative position to an absolute position within the desktop (or part of the desktop) streamed by the streaming object.

Additionally, as described above, the input may be from a pointing device such as a mouse, or through other input means such as an input mechanism that relies on a virtual driver and a transparent layer.

In step 2804, the transparent layer executed on the local computing device determines that the network address associated with the inserted streaming object corresponds to the network address of the local computing device. This may be determined, for example, by comparing the IP address associated with the streaming object with the IP address of the device providing the input to determine whether there is a match.

In step 2805, based on the determination that the network address associated with the inserted streaming object corresponds to the network address of the computing device that provides the input, the transparent layer includes one or more second commands (the one or more second commands are local desktop Configured to perform user input at the location within) to the operating system or one or more of the one or more applications configured to run on the operating system.

As described above, the transparent layer may interface with an operating system or an application running on the operating system. Thus, any input in the inserted streaming object can be mapped to a corresponding location within the local desktop, and the command is sent to the appropriate application or operating system (depending on the context involved, as described above), and the corresponding Input can be performed at the location.

29A-29C illustrate an example of controlling a desktop or a portion of a desktop from a local computing device through an embedded streaming object, according to an exemplary embodiment.

As shown in FIG. 29A, a local user interface (desktop) 2901 includes a collaboration application 2902 that displays a representation of the collaboration workspace. The representation includes an embedded/embedded streaming object 2903 that streams the local desktop itself. The local user interface 2901 also includes a taskbar 2906, which includes an operating system menu button 2905. As shown in the figure, the mouse pointer is over the button 2904 in the inserted streaming object 2902, corresponding to the operating system menu button 2905 in the local desktop.

29B shows the result of clicking the location of the button 2904 in the streaming object 2903 by the user. As a result of the input detected by the transparent layer, the location of the input in the streaming object 2903 is mapped to a corresponding location in the desktop 2901. Since the corresponding location is the operating system menu button 2905, this input causes the transparent layer to send a command to the operating system to activate the operating system menu button 2905. This change of desktop 2901 is captured by the streaming object itself, and it can also be seen that the button 2904 in the inserted streaming object is activated.

29C shows the interface 2901 and the inserted streaming object 2903 after the input has been delivered to the local desktop. As shown in FIG. 29C, an operating system menu is opened and a list of selectable indicators 2907 is included. As a result, this change is captured by the inserted streaming object 2903, which itself shows the opening of the corresponding button 2904, which contains a list of selectable indicators 2908.

As described above, the transparent layer can be effectively used to control the local desktop through the embedded streaming object. This provides a remote control interface to users participating in the collaboration session, allowing them to stay in the collaboration session while effectively navigating the desktops or applications shared with other participants.

The system can also be utilized to allow remote participants to control the desktop or part of the desktop being shared. This feature is extremely useful as it allows remote participants to access other shared desktops and applications through streaming objects embedded within the collaborative workspace.

Fig. 30 shows a flow chart for controlling a desktop or a portion of a desktop from a remote computing device via an embedded streaming object, according to an exemplary embodiment.

In step 3001, the inserted streaming object is transmitted into a representation of the collaborative workspace on the user interface of the remote computing device. The inserted streaming object is associated with the network address of the source of the video stream. Such a connection may be provided by the server in the form of a tag or metadata associated with the streaming object. In addition, the connection may be a part of the streaming object, for example, may be based on the above-described source identifier. For example, when a streaming object is generated, a device generating the streaming object may include a tag indicating the IP address of the device.

In step 3002, the transparent layer executed on the remote computing device detects a user input associated with the inserted streaming object (the user input corresponds to one location in the local desktop). As described above, the transparent layer includes an API, configured to interface with one or more of the operating system or one or more applications configured to run on the operating system. The transparent layer may detect a user input associated with the inserted streaming object based on the location of the input (determined by coordinates) and the location of the streaming object. For example, if a mouse click and a portion of the streaming object overlap, this input may be detected as a user input associated with the inserted streaming object.

The user input may be additionally mapped to a specific location in the local desktop according to the location of the input in the inserted streaming object. Once again, a map may be stored indicating the area or coordinates within the inserted streaming object associated with different parts of the local desktop. The location can be mapped to each part of the local desktop. For example, the sub-region of the inserted streaming object may be associated with a specific application occupying a corresponding region in the local desktop, or may be associated with a corresponding coordinate in the local desktop.

The mapping procedure may utilize a scaling mechanism or process that detects the relative position of the input within the inserted streaming object and maps the relative position to an absolute position within the desktop (or part of the desktop) streamed by the streaming object.

Additionally, as described above, the input may be from a pointing device such as a mouse, or through other input means such as an input mechanism that relies on a virtual driver and a transparent layer.

In step 3004, the transparent layer executed on the remote computing device determines that the network address associated with the inserted streaming object does not correspond to the network address of the remote computing device. This can be determined, for example, by comparing the IP address associated with the streaming object with the IP address of the device (remote computer device) providing the input to determine if there is a match.

In step 3005, based on the determination that the network address associated with the inserted streaming object does not correspond to the network address of the computing device providing the input, the transparent layer provides one or more second commands (the one or more The second command is configured to cause a local transparent layer running on the local computing device to perform user input at the location within the local desktop) to the local computing device.

One or more second commands may be routed from a remote computing device to a local computing device through a server through a web socket connection. In particular, the one or more second commands may be routed to the local computing device by the server after the destination address is transmitted to the server, which is the IP address of the local computing device.

The one or more second commands may be configured such that the local transparent layer of the local computing device transmits a local command to one or more of a local operating system or one or more local applications configured to be executed in the local operating system, and the one The above local command is configured to perform user input at the location in the local desktop.

As described above, the transparent layer may interface with an operating system or an application running on the operating system. Thus, any input within the inserted streaming object can be mapped to a corresponding location within the local desktop, and commands can be mapped from the local transparent layer to the appropriate application or operating system on the local computing device (depending on the context involved, as described above). Sent, allowing input to be performed at a corresponding location within the local desktop

31A-31C illustrate an example of controlling a desktop or a portion of a desktop from a remote computing device through an embedded streaming object, according to an exemplary embodiment.

As shown in FIG. 31A, the remote user interface (desktop) 3101 includes a collaboration application 3102 that displays a representation of the collaboration workspace. The representation includes an embedded/embedded streaming object 3103 that streams a local desktop ("local" as used herein refers to a device that instantiates a streaming object and shares its desktop or part of the desktop). In addition, the remote user interface 3101 includes a window and a task bar corresponding to a web browser application running on the remote desktop. As shown in the figure, the mouse pointer is over the button 3104 in the inserted streaming object 3103, corresponding to the operating system menu button in the local desktop being streamed.

31B shows the result of clicking the location of the button 3104 in the streaming object 3103 by the user. As a result of the input detected by the remote transparent layer, the location of the input in the streaming object 3103 is mapped to the corresponding location in the local desktop being streamed. Thereafter, the remote transparent layer transmits a command to the local transparent layer on the local computing device to generate an input at a corresponding location in the local desktop. Since the corresponding location is the operating system menu button on the local desktop, this input causes the remote transparent layer to send commands to the local transparent layer, and sends the command itself to the local operating system to activate the operating system menu button on the local desktop. do. This change in the local desktop is captured by the streaming object 3103 showing the button 3104 in which the inserted streaming object is being activated. It should be noted that the inserted streaming object is not affected by this input (except for updates to the streaming object 3103) because it streams other desktops associated with the local computing device, not the remote desktop 3101.

31C shows the interface 3101 and the inserted streaming object 3103 after the input has been passed to the local desktop. At the time shown in Fig. 31C, a local operating system menu of the streamed local desktop is opened, and a list of selectable indicators is included. This change is consequently captured by the inserted streaming object 3103, which itself shows the opening of the corresponding button 3104 containing a list of selectable indicators.

As described above, the transparent layer can be used to control the remote desktop via an embedded streaming object. Through this, a user participating in the collaboration session can effectively use the remote control interface, which allows the user to stay in the collaboration session and at the same time browse the desktops or applications of other participants in the collaboration workspace. For example, if two participants are presenting a presentation to another group of participants, the first presenting participant will share the presentation application on the desktop, and the first set of slides shared with the streaming object within the collaboration workspace. I can. Thereafter, the first presenter can "transfer" control of the presentation application to the second presenter, and the second presenter can remotely control the presentation application from the first presenter's desktop.

Optionally, the remote control function includes authorization, authentication, or allowing each participant to set whether another participant can remotely control the shared desktop via a streaming object, and which participants can remotely control the shared desktop. Other access control mechanisms may be included. For example, each user could store preferences indicating whether to allow other participants to control his or her local desktop or portions of the local desktop. These preferences are stored on each computing device (the transparent layer can be accessed, and can be used to allow or block remote control input by the transparent layer), or stored on the server to allow remote control input between the server and the computing device. Can be used to allow or block Such an access control mechanism can be used to determine whether a remote participant can provide input to another participant's desktop via an embedded streaming object, regardless of how it is stored.

One or more of the techniques described above may be implemented in or include one or more computer systems. 32 shows an example of a special computing environment 3200. Computing environment 3200 is not intended to imply any limitations on the scope of functionality or use of the described embodiment(s).

Referring to FIG. 32, the computing environment 3000 includes at least one processing unit 3210 and a memory 3220. The processing unit 3210 executes computer-executable instructions, and may be a real or virtual processor. In a multiple processing system, multiple processing units execute computer-executable instructions to increase processing power. The memory 3220 may be volatile memory (eg, a register, cache, RAM), non-volatile memory (eg, ROM, EEPROM, flash memory, etc.), or some combination of the two. Memory 3220 may store software 3280 implementing the described techniques.

The computing environment may have additional features. For example, computing environment 3200 includes storage 3240, one or more input devices 3250, one or more output devices 3260, and one or more communication connections 3290. An interconnection mechanism 3270, such as a bus, controller, or network interconnect, interconnects the components of the computing environment 3200. Typically, operating system software or firmware (not shown) provides an operating environment for other software running in computing environment 3200 and coordinates the activation of components of computing environment 3200.

Storage 3240 may be removable or non-removable, may be a magnetic disk, magnetic tape or cassette, CD-ROM, CD-RW, DVD, or used to store information and can be accessed within the computing environment 3200. It can be any other medium that is there. Storage 3240 may store instructions for software 3280.

The input device(s) 3250 is input to a touch input device such as a keyboard, mouse, pen, trackball, touch screen, or game controller, a voice input device, a scanning device, a digital camera, a remote control unit, or a computing environment 3200 It may be another device that provides. The output device(s) 3260 may be a display, television, monitor, printer, speaker, or other device that provides output from the computing environment 3200.

The communication connection(s) 3290 enables communication over a communication medium to another computing entity. Communication media carry information such as computer executable instructions, audio or video information, or other data in a modulated data signal. A modulated data signal is a signal having one or more of its characteristics set or changed in a manner to encode information in the signal. By way of example and not limitation, communication media includes wired or wireless techniques implemented with electricity, light, RF, infrared, acoustic, or other carriers.

Implementations may be described in the context of a computer-readable medium. Computer-readable media is any available media that can be accessed within a computing environment. By way of example and not limitation, within computing environment 3200, computer-readable media includes memory 3220, storage 3240, communication media, and combinations of any of the above.

Of course, FIG. 32 illustrates computing environment 3200, display device 3260, and input device 3250 as separate devices for ease of identification only. Computing environment 3200, display device 3260, and input device 3250 may be separate devices (e.g., a personal computer wired to a monitor and mouse), integrated into a single device (e.g., A mobile device having a touch display, such as a smartphone or tablet), or any combination of devices (e.g., a computing device operably coupled to a touch screen display device, a single display device, and a plurality of devices attached to the input device). Computing devices, etc.). The computing environment 3200 may be a set-top box, a personal computer, or one or more servers, such as a farm of networked servers, a clustered server environment, or a cloud network of computing devices.

While the principles of the present invention have been described and illustrated with reference to the described embodiments, it will be appreciated that the described embodiments may be modified in arrangements and details without departing from such principles. The elements of the described embodiment shown in software may be implemented in hardware, and vice versa.

In view of the many possible embodiments to which the principles of the invention may be applied, all such embodiments that fall within the scope and spirit of the following claims and their equivalents are claimed as the invention.

Claims (20)

A method of sharing a desktop through a web socket connection within a networked collaboration workspace, the method comprising:
Transmitting, on a user interface of a local computing device, a representation of the collaborative workspace, hosted on a server and accessible to a plurality of participants on the plurality of computing devices via a web socket connection;
Receiving, by the local computing device, a request to share at least one portion of a local desktop of the local computing device in the collaborative workspace and a selection of one area in the representation of the collaborative workspace;
Creating, by the local computing device, a streaming object configured to output a video stream of the at least one portion of the local desktop of the local computing device; And
By the local computing device, one or more commands through the web socket connection-the one or more commands include information corresponding to the streaming object and the selected region, and cause the server to select the streaming object Sending an area to the server, configured to be inserted into the collaboration workspace;
Containing, method. The method of claim 1,
Receiving a request to share the at least one portion of the local desktop of the local computing device and a selection of one area within the representation of the collaboration workspace, comprising:
Transmitting a source selection interface within the user interface, the source selection interface configured to receive a selection of the at least one portion of the local desktop; And
Receiving a selection of the at least one portion of the local desktop;
Containing, method. The method of claim 1,
Wherein the at least one portion comprises one of a window in the local desktop, an interface corresponding to an application running on the local computing device, or the local desktop. The method of claim 1,
The representation of the collaboration workspace hosted on the server is transmitted on the local computing device by a local collaboration application running on the local computing device,
Creating the streaming object configured to output the video stream of the at least one portion of the local desktop of the local computing device;
Transmitting, by the local collaboration application, a request for a source identifier to the screen capture program executed on the local computing device through an application programming interface (API) between the local collaboration application and a screen capture program;
Initiating, by the screen capture program, a stream of the at least one portion of the local desktop of the local computing device, the stream having a corresponding source identifier;
Transmitting the source identifier to the local collaboration application by the screen capture program; And
Creating, by the local collaboration application, the streaming object based at least in part on the source identifier;
Containing, method. The method of claim 1,
The inserted streaming object is configured to receive the video stream of the at least one portion of the local desktop of the local computing device, and through the web socket connection, the at least one of the local desktop of the local computing device And transmitting the video stream of the portion of the to the plurality of computing devices. The method of claim 1,
Transmitting, on the user interface of the local computing device, the inserted streaming object into the representation of the collaborative workspace, the inserted streaming object being associated with the network address of the source of the video stream;
By a transparent layer running on the local computing device, the transparent layer including an application programming interface (API), configured to interface with one or more of an operating system or one or more applications configured to be executed on the operating system. Detecting a user input associated with the inserted streaming object, the user input corresponding to a location in the local desktop;
Determining, by the transparent layer executed on the local computing device, that the network address associated with the inserted streaming object corresponds to a network address of the local computing device; And
By the transparent layer running on the local computing device, one or more second commands, the one or more second commands configured to perform the user input at the location within the local desktop, the operating system or the operating system Transmitting to one or more of the one or more applications configured to run on the application;
The method further comprising. The method of claim 1,
On a remote user interface of a remote computing device in the plurality of computing devices, the inserted streaming object into the remote representation of the collaborative workspace, the inserted streaming object is associated with the network address of the source of the video stream. Transmitting;
By a remote transparent layer running on the remote computing device, the transparent layer comprising an application programming interface (API), configured to interface with one or more of an operating system or one or more applications configured to run on the operating system, Detecting a remote user input associated with the inserted streaming object, the remote user input corresponding to a location within the local desktop;
Determining, by the remote transparent layer running on the remote computing device, that the network address associated with the inserted streaming object does not correspond to a network address of the remote computing device; And
By the remote transparent layer running on the remote computing device, through the web socket connection, one or more second commands-the one or more second commands cause a local transparent layer executed on the local computing device to be in the local desktop. Transmitting, to the local computing device, configured to perform the user input at the location;
The method further comprising. A local computing device for sharing a desktop through a web socket connection within a networked collaboration workspace, the local computing device comprising:
One or more processors; And
One or more memories-the one or more memories are operatively interlocked with at least one of the one or more processors -,
When an instruction stored in the one or more memories is executed by at least one of the one or more processors, at least one of the one or more processors causes:
A process of transmitting, on a user interface of the local computing device, a representation of a collaborative workspace accessible to a plurality of participants on a plurality of computing devices through the web socket connection and hosted on a server;
A process of receiving a request to share at least one portion of a local desktop of the local computing device in the collaborative workspace and a selection of an area within the representation of the collaborative workspace;
Creating a streaming object configured to output a video stream of the at least one portion of the local desktop of the local computing device; And
One or more commands through the web socket connection-The one or more commands include information corresponding to the streaming object and the selected area, and cause the server to transfer the streaming object from the selected area to the collaboration workspace. Configured to insert-to the server;
To perform, a local computing device. The method of claim 8,
When executed by at least one of the one or more processors, a request for at least one of the one or more processors to share the at least one portion of the local desktop of the local computing device and within the representation of the collaboration workspace The instruction to receive a selection of one region causes at least one of the one or more processors to:
A process of transmitting a source selection interface within the user interface, the source selection interface configured to receive a selection of the at least one portion of the local desktop; And
Receiving a selection of the at least one portion of the local desktop;
The local computing device to further perform. The method of claim 8,
The at least one portion includes one of a window in the local desktop, an interface corresponding to an application running on the local computing device, or the local desktop. The method of claim 8,
The representation of the collaboration workspace hosted on the server is transmitted on the local computing device by a local collaboration application running on the local computing device,
The at least one of the one or more processors to generate the streaming object configured to output the video stream of the at least one portion of the local desktop of the local computing device, when executed by at least one of the one or more processors The instruction causes at least one of the one or more processors;
A process of transmitting, by the local collaboration application, a request for a source identifier to the screen capture program running on the local computing device through an application programming interface (API) between the local collaboration application and a screen capture program;
Initiating, by the screen capture program, a stream of the at least one portion of the local desktop of the local computing device, the stream having a corresponding source identifier;
Sending the source identifier to the local collaboration application by the screen capture program; And
Creating, by the local collaboration application, the streaming object based at least in part on the source identifier;
The local computing device to further perform. The method of claim 8,
The inserted streaming object is configured to receive the video stream of the at least one portion of the local desktop of the local computing device, and through the web socket connection, the at least one of the local desktop of the local computing device A local computing device configured to transmit a portion of the video stream to the plurality of computing devices. The method of claim 8,
When an instruction stored in at least one of the one or more memories is executed by at least one of the one or more processors, at least one of the one or more processors causes:
A process of transmitting, on the user interface of the local computing device, the inserted streaming object into the representation of the collaborative workspace, the inserted streaming object being associated with the network address of the source of the video stream;
By a transparent layer running on the local computing device, the transparent layer including an application programming interface (API), configured to interface with one or more of an operating system or one or more applications configured to be executed on the operating system. A process of detecting a user input associated with the inserted streaming object, the user input corresponding to a location within the local desktop;
A process of determining, by the transparent layer running on the local computing device, that the network address associated with the inserted streaming object corresponds to a network address of the local computing device; And
By the transparent layer running on the local computing device, one or more second commands, the one or more second commands configured to perform the user input at the location within the local desktop, the operating system or the operating system A process of transmitting to one or more of the one or more applications configured to run on the application;
The local computing device to further perform. At least one non-transitory computer-readable medium storing computer-readable instructions, wherein the at least one non-transitory computer-readable medium, when executed by a local computing device, causes the local computing device to :
A process of transmitting, on a user interface of a local computing device, a representation of a collaborative workspace, hosted on a server and accessible to a plurality of participants on the plurality of computing devices via a web socket connection;
A process of receiving a request to share at least one portion of a local desktop of the local computing device in the collaborative workspace and a selection of an area within the representation of the collaborative workspace;
Creating a streaming object configured to output a video stream of the at least one portion of the local desktop of the local computing device; And
One or more commands through the web socket connection-The one or more commands include information corresponding to the streaming object and the selected area, and cause the server to transfer the streaming object from the selected area to the collaboration workspace. Configured to insert-to the server;
At least one non-transitory computer-readable medium to perform. The method of claim 14,
When executed by the local computing device, a request for the local computing device to share the at least one portion of the local desktop of the local computing device and selection of an area within the representation of the collaboration workspace The instruction to receive: causes the local computing device to:
A process of transmitting a source selection interface within the user interface, the source selection interface configured to receive a selection of the at least one portion of the local desktop; And
Receiving a selection of the at least one portion of the local desktop;
At least one non-transitory computer-readable medium to further perform. The method of claim 14,
The at least one portion includes one of a window in the local desktop, an interface corresponding to an application running on the local computing device, or the local desktop. The method of claim 14,
The representation of the collaboration workspace hosted on the server is transmitted on the local computing device by a local collaboration application running on the local computing device,
The instruction, when executed by the local computing device, causes the local computing device to create the streaming object configured to output the video stream of the at least one portion of the local desktop of the local computing device. Computing devices:
A process of transmitting, by the local collaboration application, a request for a source identifier to the screen capture program running on the local computing device through an application programming interface (API) between the local collaboration application and a screen capture program;
Initiating, by the screen capture program, a stream of the at least one portion of the local desktop of the local computing device, the stream having a corresponding source identifier;
Sending the source identifier to the local collaboration application by the screen capture program; And
Creating, by the local collaboration application, the streaming object based at least in part on the source identifier;
At least one non-transitory computer-readable medium to further perform. The method of claim 14,
The inserted streaming object is configured to receive the video stream of the at least one portion of the local desktop of the local computing device, and through the web socket connection, the at least one of the local desktop of the local computing device At least one non-transitory computer-readable medium configured to transmit a portion of the video stream to the plurality of computing devices. The method of claim 14,
Computer-readable instructions stored in the at least one non-transitory computer-readable medium, when executed by the local computing device, cause the local computing device to:
A process of transmitting, on the user interface of the local computing device, the inserted streaming object into the representation of the collaborative workspace, the inserted streaming object being associated with the network address of the source of the video stream;
By a transparent layer running on the local computing device, the transparent layer including an application programming interface (API), configured to interface with one or more of an operating system or one or more applications configured to be executed on the operating system. A process of detecting a user input associated with the inserted streaming object, the user input corresponding to a location within the local desktop;
A process of determining, by the transparent layer running on the local computing device, that the network address associated with the inserted streaming object corresponds to a network address of the local computing device; And
By the transparent layer running on the local computing device, one or more second commands, the one or more second commands configured to perform the user input at the location within the local desktop, the operating system or the operating system A process of transmitting to one or more of the one or more applications configured to run on the application;
At least one non-transitory computer-readable medium to further perform. The method of claim 14,
Computer-readable instructions stored in the at least one non-transitory computer-readable medium, when executed by a remote computing device in the plurality of computing devices, cause the remote computing device to:
Transmitting, on a remote user interface of the remote computing device, the inserted streaming object into the remote representation of the collaborative workspace, the inserted streaming object being associated with the network address of the source of the video stream;
By a remote transparent layer running on the remote computing device, the transparent layer comprising an application programming interface (API), configured to interface with one or more of an operating system or one or more applications configured to run on the operating system, Detecting a remote user input associated with the inserted streaming object, the remote user input corresponding to a location within the local desktop;
Determining, by the remote transparent layer running on the remote computing device, that the network address associated with the inserted streaming object does not correspond to the network address of the remote computing device; And
By the remote transparent layer running on the remote computing device, through the web socket connection, one or more second commands-the one or more second commands cause a local transparent layer executed on the local computing device to be in the local desktop. Configured to perform the user input at the location-to the local computing device;
At least one non-transitory computer-readable medium to further perform.

Images