JP2021517302A - Methods, devices, and computer-readable media for sending files over websocket connections in a networked collaborative workspace. - Google Patents

Abstract

Sending a representation of a collaborative workspace that is hosted on a server and accessible to participants via a web socket connection, where the representation of the collaborative workspace is a remote computing connected to the server. Sending, including the remote participant object corresponding to the device, and creating one or more live folders corresponding to the remote participant object, each live folder becoming a remote participant object. Detecting user input to generate, which maps to the network address of the corresponding remote computing device, and to drag the icon corresponding to the file proximal to the remote participant object, and remote join Store at least one file in the live folder that corresponds to the person object, thereby sending at least one file to the network address of the remote computing device over a web socket connection. Systems, methods and computer-readable media for the transmission of files over web socket connections in networked collaborative workspaces, including.

Description

(Background technology)
The operating system, and applications running within the operating system, frequently utilize external hardware devices to allow the user to give input to the program and to give output to the user. Common examples of external hardware devices include keyboards, computer mice, microphones, and external speakers. These external hardware devices interface with the operating system through the use of drivers, and the drivers are configured to interface between the hardware commands used by a particular hardware device and the operating system. Dedicated software program.

Applications may be designed to interface with several hardware devices. For example, voice-text word processing applications can be designed to interface with audio headsets, including microphones. In this case, the application must be configured, in particular, to receive voice commands, perform voice recognition, convert the recognized words into text content, and output the text content to a document. This functionality will generally be implemented in the application programming interface (API) of an application, which is a defined set of methods of communication between various software components. In an example of a speech recognition application, the API can include an interface between the application program and the software on the driver, which is responsible for interfacing with the hardware device (microphone) itself.

One problem with existing software that utilizes dedicated hardware devices is that the application or operating system software itself must be customized and specially designed to utilize the hardware device. .. This customization does not allow the hardware device to exceed the scope defined for it by the application and cannot be utilized for contexts other than the specific application in which it was designed to be used. Means. For example, a user of a voice text word processing application may receive other application programs or other components in the operating system that the other application program or operating system receives via a microphone. It cannot be operated using voice commands unless it is specifically designed to use voice commands.

FIG. 1 shows an example of an existing architecture of a system that utilizes combined hardware devices for user input. The operating system 100A of FIG. 1 includes execution applications 101A and 102A, each of which has its own APIs 101B and 102B, respectively. The operating system 100A also has its own API 100B and dedicated drivers 100C, 101C, and 102C configured to interface with the hardware devices 100D, 101D, and 102D.

As shown in FIG. 1, the application API 101B is configured to interface with the driver 101C, which itself interfaces with the hardware device 101D. Similarly, the application API 102B is configured to interface with the driver 102C, which itself interfaces with the hardware device 102D. At the operating system level, the operating system API 100B is configured to interface with the driver 100C, which itself interfaces with the hardware device 100D.

The system architecture shown in FIG. 1 limits the ability of users to utilize hardware devices outside of some application or operating system context. For example, the user may not be able to use the hardware device 101D to give input to application 102A and may use the hardware device 102D to give input to application 101A or operating system 100A. Can not.

Therefore, improvements in the hardware software interface are needed to allow the use of hardware devices in multiple software contexts.

FIG. 5 illustrates an example of an existing architecture of a system that utilizes combined hardware devices for user input. It is a figure which shows the architecture of the system which utilizes a universal hardware software interface by an exemplary example. It is a flowchart for the implementation of the universal hardware software interface by an exemplary embodiment. One or more hardware devices when the information captured by one or more hardware devices communicatively coupled to the system, according to an exemplary embodiment, contains one or more images. It is a flowchart for determining a user input based at least partially based on the information captured by. It is a figure which shows an example of object recognition by an exemplary example. It is a figure which shows an example of determining the input location coordinates by an exemplary embodiment. User input based at least in part on the captured information when the information captured by one or more hardware devices communicatively coupled to the system is sound information, according to an exemplary embodiment. It is a flowchart for determining. It is a figure which shows the tool interface which can be a part of a transparent layer by an exemplary embodiment. It is a figure which shows an example of a stylus which can be a part of a system by an exemplary embodiment. FIG. 5 is a flowchart for identifying a context corresponding to user input according to an exemplary embodiment. FIG. 5 is a diagram illustrating an example of using input coordinates to determine a context, according to an exemplary embodiment. It is a flowchart for converting a user input into a transparent layer command by an exemplary embodiment. FIG. 5 is a diagram illustrating an example of receiving input coordinates when the selection mode is toggled, according to an exemplary embodiment. FIG. 5 is a diagram illustrating an example of receiving input coordinates when the pointing mode is toggled, according to an exemplary embodiment. It is a figure which shows an example of receiving an input coordinate when a drawing mode is toggled by an exemplary embodiment. FIG. 5 is a diagram illustrating an example of a transparent layer command determined based on one or more words identified in input voice data according to an exemplary embodiment. It is a figure which shows another example of the transparent layer command which is determined based on one or more words identified in the input voice data by an exemplary example. It is a flowchart for executing one or more transparent layer commands on a transparent layer by an exemplary embodiment. It is a figure which shows the exemplary interface for adding a new command corresponding to a user input by an exemplary embodiment. FIG. 5 shows various components and options of a drawing interface and drawing mode according to exemplary embodiments. An exemplary example of a calibration and configuration interface for a video camera hardware device that is used to recognize an object and allows the user to give input using touch and gestures. It is a figure which shows. FIG. 6 illustrates a general configuration interface that allows a user to customize various aspects of an interface, toggle input modes, and make other exchanges, according to an exemplary embodiment. A flowchart for sending a file over a websocket connection in a networked collaboration workspace, according to an exemplary embodiment. FIG. 5 illustrates the network architecture used to host and transmit a collaborative workspace, according to an exemplary embodiment. FIG. 5 illustrates a process for communicating edits to a collaborative workspace within a network, according to an exemplary embodiment. It is a figure which shows a plurality of representations of a collaborative workspace by an exemplary example. It is a flowchart for generating one or more live folders corresponding to one or more remote participant objects by an exemplary embodiment. It is a figure which shows an example of the query process by an exemplary example. It is a figure which shows an example of generating one or more local folders corresponding to one or more remote participants by an exemplary example. FIG. 5 is a diagram illustrating an example of mapping one or more local folders to one or more IP addresses according to an exemplary embodiment. It is a figure which shows an example of the drag-and-drop process and detection by an exemplary example. It is a figure which shows another example of a drag-and-drop process and detection by an exemplary example. Store at least one file in the live folder corresponding to the remote participant object according to the exemplary embodiment and transfer at least one file to the network address of the remote computing device over a websocket connection. It is a flowchart for transmission. FIG. 5 illustrates an exemplary computing environment configured to perform the disclosed method.

Methods, devices, and computer-readable media are described herein as examples and examples, but those skilled in the art will appreciate methods, devices, and computer-readable media for implementing universal hardware software interfaces. Recognize that it is not limited to the embodiments or drawings described. It should be understood that the drawings and description are not limited to the particular format disclosed. Rather, the intent is to cover the intent and all modifications, equivalents and alternatives that fall within the scope of the appended claims. Any headings used herein are for organization purposes only and are not intended to limit the scope of the specification or the claims. As used herein, the word "can" has an acceptable meaning (ie, potential) rather than a compulsory meaning (ie, meaning must). Meaning). Similarly, the words "include," "include," and "includes" mean, but are not limited to.

Applicants have discovered methods, devices, and computer-readable media used for hardware devices to solve problems associated with previous hardware software interfaces. In particular, Applicants have developed a universal hardware software interface that allows users to utilize communicably coupled hardware devices in a variety of software contexts. The disclosed implementations are custom designed to allow an application or operating system to interface with a particular hardware device through the use of dedicated virtual drivers and corresponding transparent layers, as described in more detail below. Get rid of the need.

FIG. 2 shows the architecture of a system utilizing a universal hardware software interface according to an exemplary embodiment. As shown in FIG. 2, the operating system 200A includes a transparent layer 203 that communicates with the virtual driver 204. As described in more detail below, the transparent layer 203 is configured to interface between the virtual driver and the operating system and / or the application (s) running on the operating system. It is an API. In this example, the transparent layer 203 interfaces between the virtual driver 204 and the API 201B of the application 201A, the API 202B of the application 202A, and the operating system API 200B of the operating system 200A.

The transparent layer 203 can be part of a software process running on an operating system and is a transparent user interface (UI) and / or user superimposed on an underlying user interface. Can have UI elements of the transparent layer 203 itself, including visible UI elements to which can interact.

The virtual driver 204 is configured to emulate drivers 205A and 205B, which interface with hardware devices 206A and 206B, respectively. The virtual driver tells the virtual driver which virtual driver to emulate, such as voice commands, selections made on the user interface, and / or gestures made by the user in front of the combined web camera. Can receive user input in the form of. For example, each of the connected hardware devices can operate in "listening" mode, and each of the emulated drivers in virtual driver 204 signals to the virtual driver to switch to a particular emulation mode. It can be configured to detect an initialization signal that acts as. For example, a user saying "start voice command" can activate the driver for the microphone to receive a new voice command. Similarly, giving a gesture by the user can activate the driver corresponding to the webcam for receiving the gesture input or touch input.

The virtual driver may also be configured to interface with a native driver, such as the native driver 205C, which itself communicates with the hardware device 206C. In one embodiment, the hardware device 206C can be a standard input device, such as a keyboard or mouse, which is natively supported by the operating system.

In the system shown in FIG. 2, a user interfaces with any combined hardware device in various contexts, such as a particular application or operating system, and the application or operating system interfaces with the hardware device. Allows the implementation of available universal hardware software interfaces without having to be customized to do so.

For example, hardware device 206A can capture information, which is then received by virtual driver 204, which emulates driver 205A. The virtual driver 204 can determine the user input based on the captured information. For example, if the information is a series of images in which the user moves the user's hand, the virtual driver can determine that the user has performed a gesture.

Based on the identified context (such as a particular application or operating system), user input can be translated into transparent layer commands for execution and sent to transparent layer 203. Transparent layer commands can include native commands in the identified context. For example, if the identified context is application 201A, the native command will be in a format that matches application API201B of application 201A. The execution of a transparent layer command can then be configured to trigger the execution of one or more native commands in the identified context. This is achieved by the transparent layer 203 interfacing with each of the operating system 200A and the API of the application running on the operating system API 200B. For example, if the native command is an operating system command, such as a command to launch a new program, the transparent layer 203 may give the operating system API 200B that native command for execution.

As shown in FIG. 2, there is bidirectional communication between all of the components shown. This means that, for example, execution of a transparent layer command on transparent layer 203 may result in sending information to virtual driver 204 and up one of the connected hardware devices. means. For example, a voice command is recognized as input, converted to a transparent layer command containing a native command, executed by the transparent layer (causing the execution of the native command in the identified context), and then the sound output "command is A signal can be sent from the transparent layer (via the virtual driver) to the speaker to send "received".

Of course, the architecture shown in Figure 2 is for illustration purposes only, and the number of applications running, the number and type of hardware devices connected, and the number of drivers and emulated drivers will vary. Please understand that there are things to do.

FIG. 3 is a flowchart for implementing a universal hardware software interface according to an exemplary embodiment.

In step 301, user input is determined based at least in part on the information captured by one or more hardware devices communicatively coupled to the system. A system as used herein is a computing device that performs a method step, a device that comprises one or more processors that perform a method step and one or more memories. Or it may refer to any other computing system.

User input can be determined by a virtual driver running on the system. As previously described, the virtual driver may be operating in embroidery mode, in which the virtual driver emulates another hardware driver, thereby hardware. Receive captured information from a hardware device or, optionally, capture information from one or more other hardware drivers configured to interface with a particular hardware device. Can be received.

Motion capture devices including cameras, video cameras, microphones, headsets with two-way communication, mice, touchpads, trackpads, controllers, gamepads, joysticks, touch screens, accelerometers and / or tilt sensors. Various hardware devices may be utilized, such as a remote controller, stylus, or any combination of these devices. Of course, this list of hardware devices is given merely as an example, and any hardware device that can be used to detect audio, image, video, or touch information can be utilized.

Communication coupling between a hardware device and a system can take various forms. For example, a hardware device can communicate with the system via a wireless network, Bluetooth protocol, radio frequency, infrared signal, and / or by a physical connection such as a Universal Serial Bus (USB) connection. .. Communication can also include both wireless and wired communication. For example, a hardware device can contain two components, one of which signals the second component wirelessly (such as via Bluetooth), and the second component itself , Connect to the system via a wired connection (such as USB). Various communication techniques may be utilized in accordance with the systems described herein, and these examples are not limited.

The information captured by one or more hardware devices can be any type of information, such as image information containing one or more images, video frames, sound information, and / or touch information. The captured information is for sound information. wav or. For mp3 files, images. It can be in any suitable format, such as a jpg file, numerical coordinates for touch information, and so on.

The techniques described herein are "touch" in any context, even if any display device does not include hardware for detecting touch signals or touch-based gestures. It can be made possible to effectively function as a screen device. This is described in more detail below and can be achieved through analysis of images captured by cameras or video cameras.

FIG. 4 is captured by one or more hardware devices when the information captured by one or more hardware devices communicatively coupled to the system contains one or more images. A flowchart for determining a user input based on at least a part of the information obtained is shown.

In step 401, one or more images are received. These images can be captured by a hardware device, such as a camera or video camera, and received by a virtual driver, as previously described.

In step 402, objects in one or more images are recognized. The object can be, for example, a user's hand, finger, or other body part. The object can also be a dedicated device such as a stylus or pen, or a dedicated hardware device such as a motion tracking stylus / remote controller that is communicably coupled to the system and includes an accelerometer and / or tilt sensor. .. Object recognition can be performed by virtual drivers and can be based on previous training, such as through calibration routines that operate with objects.

FIG. 5A shows an example of object recognition according to an exemplary embodiment. As shown in FIG. 5A, image 501 includes the user's hand recognized as object 502. The recognition algorithm can, of course, be configured to recognize different objects, such as fingers.

Returning to FIG. 4, in step 403, one or more orientations and one or more positions of the recognized object are determined. This can be achieved in various ways. If the object is not a hardware device, but instead a body part, such as a hand or finger, the object determines the 3D coordinates of the object and various angles with respect to the X, Y, and Z axes. Therefore, it can be mapped in a three-dimensional coordinate system that uses a known location of the camera as a reference point. If the object is a hardware device and includes motion tracking hardware such as an accelerometer and / or tilt sensor, the image information is provided by the accelerometer and / or tilt sensor to determine the position and orientation of the object. Can be used with the information provided.

In step 404, user input is determined based at least in part on one or more orientations and one or more positions of the recognized object. This can include determining location coordinates on the transparent user interface (UI) of the transparent layer based at least in part on one or more orientations and one or more positions. The transparent UI is part of the transparent layer and is overlaid on top of the operating system and / or the underlying UI corresponding to any application running on the operating system.

FIG. 5B shows an example of this step when the object is a user's finger. As shown in FIG. 5B, the display device 503 includes a lower UI 506 and a transparent UI 507 overlaid on the lower UI 506. For clarity, the transparent UI 507 is shown with dot shading, but it should be understood that the transparent UI is actually a transparent layer that is not visible to the user. Further, although the transparent UI 507 is shown as being slightly smaller than the lower UI 506, it should be understood that in practice the transparent UI will cover the same screen area as the lower UI.

As shown in FIG. 5B, the position and orientation information of the object (user's finger) is used to project a line onto the plane of the display device 503 and determine the intersection 505. The image information captured by the camera 504 and the known position of the display device 503 under the camera can be used to aid in this projection. As shown in FIG. 5B, the user input is determined to be the input coordinates at intersection 505.

As further described below, the actual transparent layer commands generated based on this input may be based on user settings and / or identified contexts. For example, the command can be a touch command indicating that an object at coordinates at point 505 should be selected and / or opened. The command can also be a pointing command indicating that the pointer (such as a mouse pointer) should be moved to the coordinates of point 505. In addition, the command can be an edit command that modifies the graphical output at the location (for example, to annotate the interface or draw an element).

FIG. 5B shows the recognized object 502 as being at some distance from the display device 503, but touch input can be detected regardless of the distance. For example, if the user physically touches the display device 503, the technique described above will still determine the input coordinates. In that case, the projection line between the object 502 and the intersection would only be shorter.

Of course, touch input is not the only type of user input that can be determined from the captured image. The step of determining a user input based at least in part on one or more orientations and one or more positions of a recognized object can include determining a gesture input. In particular, the position and orientation of the recognized object across multiple images can be analyzed to determine the corresponding gesture, such as a swipe gesture, a pinch gesture, and / or any known or customized gesture. The user can calibrate the virtual driver to recognize custom gestures that map to specific contexts and commands within those contexts. For example, a user can create a custom gesture that maps to an operating system context and results in the execution of a native operating system command that launches a particular application.

As previously described, the information captured by one or more hardware devices in step 301 of FIG. 3 may also include sound information captured by a microphone. FIG. 6 determines user input based at least in part on the captured information when the information captured by one or more hardware devices communicatively coupled to the system is sound information. The flowchart for this is shown. Speech recognition is performed on the sound information to identify one or more words corresponding to user input, as described below.

In step 601 the sound data is received. Sound data can be captured by a hardware device, such as a microphone, and received by a virtual driver, as described above. In step 602, the received sound data can be compared with a sound dictionary. The sound dictionary can include sound signatures of one or more recognized words, such as command words or command modifiers. In step 603, one or more words in the sound data are identified as user input based on comparison. The identified word may then be converted to a transparent layer command and passed to the transparent layer.

As explained earlier, the driver emulated by the virtual driver, the expected type of user input, and the commands generated based on the user input are all one or more settings or the previous user input. Can be determined on the basis of at least partly.

FIG. 7 shows a tool interface 701 that can also be part of a transparent layer. Unlike the transparent UI, the tool interface 701 is visible to the user and can be selected between different options to change the virtual driver emulation mode, native commands generated based on user input, or additional Can be used to perform a function.

Button 701A is for the user to graphically modify the user interface when the user input is input coordinates (such as coordinates based on the user touching the screen with the user's hand or stylus / remote controller). Allows you to select the type of drawing tool used for. Various drawing tools can include different brushes, colors, pens, highlighters, and so on. These tools can make graphical changes in varying styles, thicknesses, colors, etc.

Button 701B allows the user to switch between selection mode, pointing mode, or drawing mode when the input coordinates are received as user input. In selection mode, the input coordinates are treated as a "touch", which can result in selecting or opening an object in the input coordinates. In pointing mode, the coordinates can be treated as pointer positions (such as mouse pointers), effectively allowing the user to emulate the mouse. In drawing mode, the coordinates can be treated as a location for modifying the graphical output of the user interface in order to present the appearance of drawing or writing on the user interface. The nature of the changes may depend on the drawing tool selected, as described for Button 701A. Button 701B can also alert the virtual driver to anticipate image and / or motion input (if motion tracking devices are used) and emulate the appropriate driver accordingly.

Button 701C alerts the virtual driver to anticipate voice commands. This can cause the virtual driver to emulate the driver corresponding to the combined microphone for receiving the voice input and parsing the voice input, as described with respect to FIG.

Button 701D opens a launcher application that can be part of a transparent layer and can be used to launch an application within the operating system or to launch a specific command within the application. The launcher is for customizing options in the transparent layer, such as custom voice commands, custom gestures, custom native commands for applications associated with user input, and / or (voice calibration, motion capture device calibration). , And / or object recognition calibration, etc.) Can also be used to calibrate hardware devices and user inputs.

Button 701E can be used to capture a screenshot of the user interface and to export the screenshot as an image. It can be used with the drawing mode of button 701B and the drawing tool of 701A. After the user marks up a particular user interface, the marked up version can be exported as an image.

Button 701F also allows graphical editing and can be used to exchange drawing colors or drawing modes that the user is creating on the user interface. Similar to the drawing mode of button 701B, this button changes the nature of the graphical changes in the input coordinates.

Button 701G erases the drawing on the user interface. This button selection removes all graphical markings on the user interface and resets the underlying UI to what it was before the user created the drawing.

Button 701H can be used to launch a whiteboard application that allows the user to create a draw or write on a virtual whiteboard using the draw mode.

Button 701I can be used to add text notes to an object, such as the object shown in the operating system UI or application UI. Text notes can be interpreted from audio signals or typed by the user using the keyboard.

Button 701J can be used to open or close the tool interface 701. When closed, the tool interface can be minimized or completely removed from the subordinate user interface.

As previously described, a stylus or remote hardware device can be used with the system along with other hardware devices such as cameras or video cameras. FIG. 8 shows an example of a stylus 801 that can be used with this system. The stylus 801 can communicate with the hardware receiver 802 via Bluetooth or the like. The hardware receiver can be connected to the computer system via USB802B or the like, and the signal from the stylus passed to the computer system via the hardware receiver is the tool shown in FIG. It can be used to control and interact with menu 803, which is similar to the interface.

As shown in FIG. 8, the stylus 801 can include a physical button 801A. These physical buttons 801 can be used to power on the stylus, navigate the menu 803, and make selections. In addition, the stylus 801 can include a characteristic tip 801B that is captured in the image taken by the camera and recognized by the virtual driver. This can allow the stylus 801 to be used for drawing and editing when in drawing mode. The stylus 801 can also include motion tracking hardware, such as an accelerometer and / or tilt sensor to assist in position detection when the stylus is used to give input coordinates or gestures. In addition, the hardware receiver 802 can include a calibration button 802A, which, when pressed, can activate the calibration utility in the user interface. This allows the stylus to be calibrated.

Returning to FIG. 3, in step 302, the context corresponding to the user input is identified. The identified context comprises the operating system or one of the applications running on the operating system.

FIG. 9 shows a flowchart for identifying a context corresponding to user input according to an exemplary embodiment. As shown in FIG. 9, operating system data 901, application data 902, and user input data 903 can all be used to determine context 904.

Operating system data 901 can include, for example, information about the active window in the operating system. For example, if the active window is a calculator window, the context can be determined to be a calculator application. Similarly, if the active window is a Microsoft Word window, the context can be determined to be a Microsoft Word application. On the other hand, if the active window is a file folder, the active context can be determined to be the operating system. Operating system data should also contain additional information, such as which application is currently running, the last application launched, and any other operating system information that may be used to determine the context. Can be done.

Application data 902 can include, for example, information about one or more applications running and / or information that maps a particular application to some type of user input. For example, whenever a voice command is received, the first application may be mapped to a voice input so that the context is automatically determined to be the first application. In another example, when a particular gesture is received as input, the gesture is launched or closed, or an action within the second application is performed. Can be associated with a second application.

User input 903 can also be used to determine context in various ways. As described above, some types of user input can be mapped to some applications. In the above example, the voice input is associated with the context of the first application. In addition, user-input attributes can also be used to determine the context. Gestures or movements can be mapped to applications or operating systems. Certain words in voice commands can also be mapped to applications or operating systems. Input coordinates can also be used to determine the context. For example, a window in the user interface at the location of the input coordinates can be determined, and the application corresponding to that window can be determined as the context.

FIG. 10 shows an example of using input coordinates to determine the context. As shown in FIG. 10, the display device 1001 displays the user interface 1002. Also shown are the camera 1004 and the transparent layer 1003 overlaid on the lower user interface 1003. The user utilizes the stylus 1000 to point to location 1005 in user interface 1002. Since location 1005 is in the application window corresponding to application 1, application 1 is then determined to be the context for user input, as opposed to application 2, application 3, or the operating system. obtain.

Returning to FIG. 3, in step 303, the user input is transformed into one or more transparent layer commands based at least in part based on the identified context. As previously described, the transparent layer comprises an application programming interface (API) configured to interface between the virtual driver and the operating system and / or the application running on the operating system. ..

FIG. 11 shows a flowchart for converting user input into a transparent layer command. As shown in step 1104 of FIG. 11, the transparent layer command may be determined at least in part based on the identified context 1102 and user input 1103. A transparent layer command can include one or more native commands configured to execute in one or more corresponding contexts. The transparent layer command can also include a response output that should be sent to the virtual driver and on the hardware device (s).

The identified context 1102 can be used to determine which transparent layer command should be mapped to user input. For example, if the identified context is "operating system", swipe gesture input will cause the user interface to operate (by minimizing one open window and maximizing the next). It can be mapped to a transparent layer command that results in scrolling the currently open window in the system. Alternatively, if the identified context is a "web browser application", the same swipe gesture input can be mapped to a transparent layer command that causes the web page to scroll.

User input 1103 also determines the transparent layer command because the user input is clearly mapped to several native commands in one or more contexts, and these native commands are part of the transparent layer command. To do. For example, the voice command "Open Email" can be mapped to a specific operating system native command to launch the email application Outlook. When a voice input containing the recognized word "Open Email" is received, this results in a transparent layer command being determined, including a native command for invoking Outlook.

As shown in FIG. 11, the transparent layer command may also be determined based on one or more user settings 1101 and API library 1104. The API library 1104 can be used to look up native commands that correspond to an identified context and a particular user input. In the swipe gesture and web browser application context examples, the API library corresponding to the web browser application may be queried for the appropriate API call to cause the web page to scroll. Alternatively, API library 1104 can be omitted and native commands can be mapped towards specific user input and identified contexts.

In situations where the user input is determined to be input coordinates, the transparent layer command is determined at least in part based on the input location coordinates and the identified context. In this case, the transparent layer command can contain at least one native command in the identified context, and at least one native command is configured to perform an action at the corresponding location coordinate in the underlying UI. ..

Setting 1101 can be used to determine the corresponding transparent layer command when there are two or more possible actions that are mapped to a particular context and user input. For example, button 701B of FIG. 7 allows the user to select between a selection mode, a pointing mode, or a drawing mode when the input coordinates are received as user input. This setting can be used to determine the transparent layer command and, by extension, which native command will be executed and which action will be performed. In this case, the possible native commands are selection commands configured to select objects related to the corresponding location coordinates in the lower UI, pointers configured to move the pointer to the corresponding location coordinates in the lower UI. It can include commands and graphical commands configured to change the display output at the corresponding location coordinates in the underlying UI.

FIG. 12A shows an example of receiving input coordinates when the selection mode is toggled. As shown in FIG. 12A, the user points to the stylus 1200 in operating system UI 1202 (with superposed transparent UI 1203) on display device 1201. Similar to the previous example, the camera 1204 can be used to determine the position and orientation information and input coordinates for the stylus 1200. Since the selection mode is toggled and the stylus 1200 is pointed to in folder 1205 in operating system UI 1202, the determined lucidum command is to select the object associated with the input coordinates (in this case, folder 1205). Can contain native operating system commands. In another example, if the window is located in input coordinates, this will result in a selection of the entire window.

FIG. 12B shows an example of receiving input coordinates when the pointing mode is toggled. In this case, the determined transparent layer command can include a native operating system command to move the mouse pointer 1206 to the location of the input coordinates.

FIG. 12C shows an example of receiving input coordinates when the drawing mode is toggled and the user sweeps the stylus 1200 over a plurality of input coordinates. In this case, the determined transparent layer command can include a native operating system command to change the display output at each location of the input coordinates, and the user's drawing line 1207 on the user interface 1202. Occurs. The modified graphical output provided in the drawing mode can be stored as part of the transparent layer 1203, for example, as metadata related to the path of the input coordinates. The user can then select an option to export the modified display output as an image.

In situations where user input is identified as a gesture, translating user input into one or more transparent layer commands based at least in part on the identified context is at least to the identified gesture and the identified context. It can include determining the transparent layer command on a partial basis. A transparent layer command can include at least one native command in the identified context, and at least one native command is configured to perform an action related to the identified gesture in the identified context. To. An example of this has been described above with respect to swipe gestures and web browser application contexts that result in native commands configured to perform scrolling actions in a web browser.

In situations where user input is identified as one or more words (such as by using speech recognition), convert user input to one or more transparent layer commands based at least in part on what is identified. What you do can include determining the Stratum lucidum command based at least in part on the identified word and the identified context. A transparent layer command can include at least one native command in the identified context, and at least one native command performs an action related to one or more identified words in the identified context. It is configured to do.

FIG. 13 shows an example of the transparent layer command 1300 determined based on one or more words identified in the input voice data. The identified word 1301 includes one of the phrases "whiteboard" or "blank page". The transparent layer command 1300 also includes a description of the command 1302 and a response instruction 1303 which is an output instruction sent by the transparent layer to the virtual driver and to the hardware output device when the transparent layer command is executed. In addition, the transparent layer command 1300 includes the actual native command 1304 used to call the whiteboard function.

FIG. 14 shows another example of the transparent layer command 1400 determined based on one or more words identified in the input voice data according to an exemplary embodiment. In this example, one or more words is "open email". As shown in FIG. 14, the transparent layer command 1400 includes a native command "outlook.exe", which is an instruction to operate a specific executable file that launches an outlook application. The transparent layer command 1400 also includes a voice response "opened an email" that is output in response to receiving a voice command.

Returning to FIG. 3, in step 304, one or more transparent layer commands are executed on the transparent layer. Execution of one or more transparent layer commands is configured to trigger execution of one or more native commands in the identified context.

FIG. 15 shows a flowchart for executing one or more transparent layer commands on a transparent layer according to an exemplary embodiment. In step 1501, at least one native command in the transparent layer command is identified. Native commands can be specified, for example, as native commands within the structure of transparent layer commands, allowing identification.

In step 1502, at least one native command is executed in the identified context. This step may include passing at least one native command to the identified context via the identified API for that context and executing the native command within the identified context. it can. For example, if the identified context is the operating system, native commands may be passed to the operating system for execution via the operating system API. In addition, if the identified context is an application, native commands may be passed to the application for execution via the application API.

Optionally, at step 1503, the response may be sent to the hardware device (s). As explained earlier, this response can be routed from the transparent layer to the virtual driver and onto the hardware device.

16-19 show additional features of the system disclosed herein. FIG. 16 shows an exemplary interface for adding new commands corresponding to user input, according to an exemplary embodiment. A dashboard at interface 1600 includes an icon for application 1601 that has already been added and can be launched using predetermined user input and hardware devices (eg, voice commands). The dashboard is application-specific and can also show other commands that are mapped to some user input. Selecting the add button 1602 opens the add command menu 1603. This menu allows the user to choose between the following options: Item Type: Fixed Items to Add on Bottom Bar Menu / Regular Items to Add in Drag Menu, Icon: Select image icon, Background: Select background icon color, Color: Select icon color, Name: Set new item name, Voice command: Set voice activation command to open new application, Feedback Response: Application Voice Response Set feedback, Command: Select the application type or custom command type to launch (for example, launch an application command, perform an action within an application command, application (Close command, etc.), Start process: When starting a new process or application, the name of the process or application, and parameters: Any parameters to pass to the new process or application.

FIG. 17 shows various components and options of the drawing interface 1700 and drawing mode according to an exemplary embodiment. FIG. 18 shows a calibration and configuration interface 1800 for a video camera hardware device that is used to recognize an object and allows the user to give input using touch and gesture. FIG. 19 shows a general configuration interface 1900 that allows the user to customize various aspects of the interface, toggle input modes, and make other exchanges. As shown in Interface 1900, the user can also access the settings page for calibrating and adjusting the settings for the hardware stylus (called the "magic stylus").

The systems disclosed herein can be implemented on multiple networked computing devices and used to assist in conducting networked collaborative sessions. For example, the whiteboard functionality described earlier can be a shared whiteboard among multiple users on multiple computing devices.

However, one of the problems with existing whiteboards or other shared communal spaces is that there is no easy way to share files with connected participants. Some applications allow the entire group to review files such as documents in a communal space, but send files to other users to share files with other users. It is necessary to open a new application (such as an email client or file sharing application) for this. During shared joint sessions, this frequently interrupts workflows and shared brainstorming sessions designed to facilitate the shared space.

In addition to the methods and systems previously described for the implementation of the universal hardware software interface, Applicants are required to submit files in real time via a web socket connection during a joint session between networked computers. We have further discovered methods, devices and computer-readable media that enable transmission.

FIG. 20 shows a flow chart for sending a file over a websocket connection in a networked collaborative workspace, according to an exemplary embodiment. All of the steps shown in FIG. 20 can be performed on a local computing device, such as a client device connected to a server, and does not require multiple computing devices.

In step 2001, a representation of a collaborative workspace hosted on a server and accessible to multiple participants via a websocket connection is transmitted over the user interface of the local computing device. A representation of a collaborative workspace can include one or more remote participant objects that correspond to one or more remote computing devices connected to a server. Remote computing devices and remote participants as used herein refer to local participants and computing devices and participants other than local computing devices. Remote computing devices are separated from local devices by networks such as wide area networks (WANs).

FIG. 21A shows a network architecture used to host and transmit a collaborative workspace, according to an exemplary embodiment. As shown in FIG. 21A, the server 2100 is connected to computing devices 2101A-2101F. The server 2100 and the computing devices 2101A to 2101F are connected via a network connection such as a web socket connection that enables bidirectional communication between the computing devices 2101A to 2101F (client) and the server 2100. Can be done. As shown in FIG. 21A, the computing device can be any type of computing device, such as a laptop, desktop, smartphone, or other mobile device.

A collaborative workspace can be, for example, a digital whiteboard configured to convey edits from any participant among multiple participants to other participants via a websocket connection. FIG. 21B shows a process for communicating edits to a collaborative workspace within a network, according to an exemplary embodiment. As shown in FIG. 21B, when a user in computing device 2101B makes edits or changes to a collaborative workspace, the edits or changes 2102B are sent to server 2100 to provide a hosted version of the workspace. Used to update. The edit or modification is then transmitted by the server 2100 as updates 2102A, 2102C, 2102D, 2102E, and 2102F to the other connected computing devices 2101A, 2101C, 2101D, 2101E, and 2101F.

Each representation of a collaborative workspace can be a customized collaborative workspace version for local participants. For example, as described above, each representation of a collaborative workspace may include one or more remote participant objects that correspond to one or more remote computing devices connected to a server. it can.

FIG. 22 shows a plurality of representations of a collaborative workspace according to an exemplary embodiment. As shown in FIG. 22, server 2200 hosts the collaborative workspace 2201. The version of the collaborative workspace hosted on the server is communicated to the connected device as described earlier. FIG. 22 also shows a representation of a collaborative workspace for three connected users, user 1, user 2, and user 3. As shown, each representation is customized for local participants (for local computing devices). For example, representation 2201A for user 1 includes remote participant objects corresponding to user 2 and user 3. Similarly, representation 2201B for user 2 includes remote participant objects corresponding to user 1 and user 3, and representation 2201C for user 3 includes remote participant objects corresponding to user 1 and user 2. ..

The remote participant object represents a remote participant and can take many forms. For example, a remote participant object can be an embedded video stream of a remote participant connected via a video conference or webcam. The remote participant object can also be an icon representing the remote participant, a remote participant's avatar, or any other visual or audio indicator of a particular remote participant. The remote participant object can be a custom object that can be dragged, moved, and / or resized within the representation of the workspace.

Returning to FIG. 20, in step 2002, one or more live folders corresponding to one or more remote participant objects are generated on the local computing device, and each live folder is remote. Maps to the network address of the remote computing device that corresponds to the participant object.

FIG. 23 shows a flow chart for generating one or more live folders corresponding to one or more remote participant objects according to an exemplary embodiment.

In step 201, the local computing device is a server for one or more Internet Protocol (IP) addresses of one or more remote computing devices that correspond to one or more remote participant objects. Contact.

FIG. 24 shows an example of this query process. As shown in FIG. 24, the computing devices 2401A, 2402, and 2403 are all connected to the server 2400 via a bidirectional network connection, such as a websocket connection. The server hosts a collaborative workspace (not shown). FIG. 24 also shows representation 2401B of a collaborative workspace that is visible on the user interface of computing device 2401A. Representation 2401B includes remote participant objects corresponding to users 2 and 3, which are users of computing devices 2402 and 2403, respectively. As shown in FIG. 24, a query for the IP address (or other type of network address) of a remote computing device that corresponds to a remote participant object is made to the computing device over a network connection. It is transmitted from 2401A to server 2400. In response, the server sends the IP addresses (or other types of network addresses) of the other connected computing devices 2402 and 2403 to the computing device 2401A (requesting computing device). The server can also send information, such as user identification information or other information, that allows the computing device 2401A to identify which IP address corresponds to which remote participant object.

Returning to FIG. 23, in step 2302, the local computing device creates one or more local folders corresponding to one or more remote participant objects. One or more local folders may be created and stored in the memory of the local computing device. For example, a temporary cache can be created on a local computing device when a collaborative workspace session is started. This temporary cache stores information about the session, such as the conference identifier and other session details. One or more local folders may be created and stored in a temporary cache.

FIG. 25 shows an example of creating one or more local folders corresponding to one or more remote participants according to an exemplary embodiment. The local computing device 2500 includes a display (not shown) displaying the user interface 2501, which includes a representation 2502 of the collaborative workspace. Collaborative workspace representation 2502 includes two remote participant objects corresponding to user 2 and user 3. As shown in FIG. 25, local folders F2 and F3 are created in local storage 2503 on computing device 2500 and linked with remote participant objects for users 2 and 3. .. Folders can be linked to remote participant objects in various ways. For example, a custom data structure can be created that associates each remote participant object with a local folder. The local folder can be incorporated into the user interface 2501 as an invisible element and placed at the same location on the corresponding remote participant object. The remote participant object can have a public API that allows it to be linked to the corresponding local folder. Many variants are possible and these examples are not limited.

Returning to FIG. 23, in step 2303, one or more live folders are generated by mapping one or more local folders to one or more IP addresses.

FIG. 26 shows an example of mapping one or more local folders to one or more IP addresses according to an exemplary embodiment. The local computing device 2600 includes a display (not shown) displaying the user interface 2601, including a collaborative workspace 2602. The collaborative workspace representation 2602 includes two remote participant objects corresponding to user 2 and user 3. Local folders F2 and F3 in local storage 2603 on computing device 2600 are linked with remote participant objects for users 2 and 3 in the collaborative workspace representation 2602.

As shown in FIG. 26, the local folders F2 and F3 are mapped to the network addresses of the remote computing devices 2604 and 2605, respectively. The computing device 2604 corresponds to the remote participant user 2 and the computing device 2605 corresponds to the remote participant user 3. Mapping local folders to network addresses of remote computing devices can be achieved in a variety of ways. Each local folder can have the network address of the corresponding remote computing device as its address. In this case, the local folder is an instantiation of a remote folder on local storage. In addition, custom data structures or scripts may be configured to transfer the contents of the local folder to the destination network address. The script can interface with a network connection (such as a web socket) to achieve the transfer. Many variants are possible and these examples are not limited.

Returning to FIG. 20, in step 2003, the user input for dragging at least one icon corresponding to at least one file is local to the proximal of the remote participant object in one or more remote participant objects. -Detected by the computing device, the remote participant object corresponds to the remote computing device in one or more remote computing devices. This is commonly referred to as a drag-and-drop action and can be entered using a variety of input devices. For example, the user can use the mouse to drag and drop. The user can also drag and drop using hand gestures or a stylus as previously described. The techniques described above, with virtual drivers and / or transparency, can be used to detect drag-and-drop motion.

The local computing device may be configured to store one or more spatial locations of one or more remote participant objects in the user interface. Detection of whether a particular icon has been dragged and dropped proximal to a particular remote participant object drags the icon from the remote participant object's spatial position to a destination spatial position within a threshold distance. It can be done by detecting the user input to do so. The threshold distance can be set by the user or can be some default value. For example, the threshold distance can be less than 10 pixels or less than 0 pixels (in which case the dragged icon must intersect or overlap the remote participant object in the user interface. There will be).

FIG. 27 shows an example of a drag-and-drop process and detection according to an exemplary embodiment. User interface 2701 includes a representation of a collaborative workspace 2702 that includes remote participant objects corresponding to user 2 (U2) and user 3 (U3). Interface 2701 also includes, for example, a file window or file interface 2703, which can be an open folder in a file browser. The file interface 2703 can be the desktop of a local computing device, a file sharing application, a web browser, or any other application. As shown in FIG. 27, the user dragged the file "D3" onto the remote participant object for user 3 (U3).

Of course, files can also be dragged from within the collaborative workspace. FIG. 28 shows another example of the drag-and-drop process and detection according to an exemplary embodiment. Interface 2801 corresponds to the representation of a collaborative workspace. Interface portion 2802 can be a section of representation of a collaborative workspace listing various files. Further, in this case, the remote participant object 2803 is an embedded video stream, which can be received over a network (eg, websocket) connection. As shown in FIG. 28, the local participant (user) dragged the adobe file 2804 onto the remote participant object 2803. Of course, the file can be any type of file, such as an audio file, a video file, an audiovisual file, a text document, a spreadsheet, a slide presentation, and so on.

Returning to FIG. 20, in step 2004, at least one file is stored by the local computing device in the live folder corresponding to the remote participant object, thereby storing at least one file in the web socket. Send to the network address of the remote computing device over the connection (or other network connection).

FIG. 29 stores at least one file in a live folder corresponding to a remote participant object according to an exemplary example, and at least one file of a remote computing device over a websocket connection. A flowchart for sending to a network address is shown.

In step 2901, at least one file corresponding to at least one dragged icon is stored in the live folder. At step 2902, a copy of at least one file is sent over the websocket connection (or other network connection) to the mapped network address. WebSocket connections can be configured to route data stored in live folders through a server to a mapped network address. This routing is shown in Figure 21B, except that the file is sent to a particular remote computing device and not all connected remote computing devices. It can be performed in the same way as the updates that have been made. For example, the file may be sent to the server via websocket, and the destination address may be used by the server to further route the file through the websocket to the appropriate destination remote computing device.

The process described is intended because the websocket connection has already been established as part of a collaborative workspace (when the remote participant object is an embedded video stream) and / or as part of a video conferencing. Utilize a web socket to perform the routing of files to the received receiver.

At step 2903, at least one file may be deleted from the live folder when the transmission is complete. For example, if a live folder is stored in a temporary cache on the local computing device, then when a copy of the files stored in the live folder is sent to the remote computing device (such as by streaming). , Files stored in the local live folder can be deleted.

One or more of the techniques described above may be implemented in one or more computer systems or may involve one or more computer systems. FIG. 30 shows an example of the dedicated computing environment 3000. Computing Environment 3000 does not imply any limitation on the use or scope of functionality of the described (s) examples.

Referring to FIG. 30, the computing environment 3000 includes at least one processing unit 3010 and memory 3020. The processing unit 3010 executes computer-executable instructions and can be a real processor or a virtual processor. In a multiprocessing system, a plurality of processing units execute computer-executable instructions in order to increase the processing capacity. The memory 3020 can be a volatile memory (eg, register, cache, RAM), a non-volatile memory (eg, ROM, EEPROM, flash memory, etc.), or any combination thereof. Memory 3020 can store software 3080 that implements the techniques described.

The computing environment can have additional features. For example, the computing environment 3000 includes storage 3040, one or more input devices 3050, one or more output devices 3060, and one or more communication connections 3090. An interconnect mechanism 3070, such as a bus, controller, or network, interconnects the components of the computing environment 3000. In general, operating system software or firmware (not shown) provides an operating environment for other software running in computing environment 3000, coordinating the activities of the components of computing environment 3000.

Storage 3040 can be removable or non-removable and can be used to store magnetic disks, magnetic tapes or cassettes, CD-ROMs, CD-RWs, DVDs, or information and can be accessed within the computing environment 3000. Includes any other medium. Storage 3040 can store instructions for software 3080.

The (s) input device 3050 may be a touch input device such as a keyboard, mouse, pen, trackball, touch screen, or game controller, voice input device, scanning device, digital camera, remote control, or It can be another device that provides input to the computing environment 3000. The output device (s) 3060 can be a display, television, monitor, printer, speaker, or another device that provides output from the computing environment 3000.

The communication connection (s) 3090 allows communication to another computing entity via a communication medium. The communication medium transmits information in the modulated data signal, such as computer executable instructions, audio or video information, or other data. A modulated data signal is a signal in which one or more of the characteristics of the signal are set or exchanged in such a way that the information in the signal is encoded. As an example, but not a limitation, communication media include wired or wireless techniques implemented using electrical, optical, RF, infrared, acoustic, or other carriers.

Implementations can be described in the context of a computer-readable medium. A computer-readable medium is any available medium that can be accessed within a computing environment. As an example, but not limited to, within a computing environment 3000, a computer-readable medium includes memory 3020, storage 3040, a communication medium, and any combination of the above.

Of course, FIG. 30 shows the computing environment 3000, the display device 3060, and the input device 3050 as separate devices for easy identification. The computing environment 3000, display device 3060, and input device 3050 can be separate devices (eg, personal computers connected to monitors and mice by wires), and can be a single device (eg, smartphone or tablet, etc.). , A mobile device with a touch display), or any combination of devices (eg, a computing device operably coupled to a touch screen display device, a single display device and an input device. Can be integrated into multiple computing devices, etc.). A computing environment 3000 is a set-top box, personal computer, or one or more servers, such as a farm of networked servers, a clustered server environment, or a cloud network of computing devices. obtain.

Although the principles of the invention have been described and illustrated with respect to the embodiments described, it will be appreciated that the embodiments described may be modified in configuration and detail without departing from such principles. The elements of the illustrated examples shown in the software can be implemented in hardware and vice versa.

In view of the many possible embodiments to which the principles of the invention may be applied, the invention claims all embodiments that may fall within the scope and intent of the following claims and their equivalents.

Claims (20)

A method for sending files over a websocket connection in a networked collaborative workspace, said method.
The step of transmitting a representation of a collaborative workspace hosted on a server and accessible to multiple participants over a web socket connection on the user interface of a local computing device, said co-operation. A step of transmitting, wherein the representation of the workspace comprises one or more remote participant objects corresponding to one or more remote computing devices connected to the server.
A step of generating one or more live folders corresponding to the one or more remote participant objects on the local computing device, where each live folder is the remote participant object. Maps to the network address of the corresponding remote computing device, and the steps to generate,
The local computing device detects user input for dragging at least one icon corresponding to at least one file proximal to the remote participant object in the one or more remote participant objects. A step of detecting that the remote participant object corresponds to a remote computing device in the one or more remote computing devices.
The local computing device stores the at least one file in a live folder corresponding to the remote participant object, thereby causing the at least one file to be stored over the web socket connection. A method comprising sending to said network address of a remote computing device. The one or more remote participant objects corresponding to the one or more remote computing devices
The method of claim 1, comprising one or more of an embedded video stream or a remote participant icon. The collaborative workspace comprises a digital whiteboard configured to convey edits from any participant among the plurality of participants to other participants via the web socket connection. The method of claim 1, wherein said representation of the digital whiteboard comprises a customized representation of the digital whiteboard for local participants. The step of creating one or more live folders corresponding to the one or more remote participant objects is
A step of querying the server for one or more Internet Protocol (IP) addresses of the one or more remote computing devices corresponding to the one or more remote participant objects.
The step of creating one or more local folders corresponding to the one or more remote participant objects,
The method of claim 1, comprising the step of generating the one or more live folders by mapping the one or more local folders to the one or more IP addresses. The local computing device is configured to store one or more spatial locations of the one or more remote participant objects in the user interface and said one or more remote participants. The step of detecting user input for dragging at least one icon corresponding to at least one file is proximal to the remote participant object in the person object.
The method of claim 1, comprising detecting a user input for dragging the at least one icon from a spatial position of the remote participant object to a destination spatial position within a threshold distance. Store the at least one file in the live folder corresponding to the remote participant object, thereby storing the at least one file in the network of the remote computing device via the web socket connection.・ The step to send to the address is
The step of storing the at least one file in the live folder, and
A step of transmitting a copy of the at least one file to the mapped network address through the web socket connection, wherein the web socket connection goes through the server to the mapped network address. Steps to send, configured to route data,
The method of claim 1, comprising removing the at least one file from the live folder upon completion of the transmission. The method of claim 1, wherein the live folder is stored in a temporary cache on the local computing device. A local computing device for sending files over a websocket connection in a networked collaborative workspace, said local computing device.
With one or more processors
At least one of the one or more processors when operably coupled to at least one of the one or more processors and executed by at least one of the one or more processors. For one,
Sending a representation of a collaborative workspace hosted on a server and accessible to multiple participants over a web socket connection on the user interface of the local computing device. Sending that the representation of the collaborative workspace contains one or more remote participant objects corresponding to one or more remote computing devices connected to the server.
Creating one or more live folders corresponding to the one or more remote participant objects, each live folder of a remote computing device corresponding to the remote participant object. Mapped to a network address, generated and
The detection of user input for dragging at least one icon corresponding to at least one file proximal to the remote participant object in the one or more remote participant objects. Detecting that an object corresponds to a remote computing device in the one or more remote computing devices.
Store the at least one file in the live folder corresponding to the remote participant object, thereby storing the at least one file in the network of the remote computing device via the web socket connection. A local computing device with one or more memories that store instructions to send and do to an address. The one or more remote participant objects corresponding to the one or more remote computing devices
The local computing device of claim 8, comprising one or more of an embedded video stream or a remote participant icon. The collaborative workspace comprises a digital whiteboard configured to convey edits from any participant among the plurality of participants to other participants via the web socket connection. 8. The local computing device of claim 8, wherein said representation of the digital whiteboard is customized for local participants. When executed by at least one of the one or more processors, at least one of the one or more processors corresponds to one or more of the remote participant objects. The instruction that causes the generation of a plurality of live folders causes at least one of the one or more processors to generate the plurality of live folders.
Querying the server for one or more Internet Protocol (IP) addresses of the one or more remote computing devices corresponding to the one or more remote participant objects.
Creating one or more local folders corresponding to the one or more remote participant objects.
The local according to claim 8, further performing the generation of the one or more live folders by mapping the one or more local folders to the one or more IP addresses. -Computing device. The local computing device is configured to store one or more spatial locations of the one or more remote participant objects in the user interface and of the one or more processors. When executed by at least one of them, at least one of the one or more processors has at least one file proximal to the remote participant object in the one or more remote participant objects. The instruction that causes the user input to detect the user input for dragging at least one icon corresponding to the above is given to at least one of the one or more processors.
8. The eighth aspect of claim 8, wherein the user input for dragging the at least one icon from the spatial position of the remote participant object to a destination spatial position within a threshold distance is further detected. Local computing device. When executed by at least one of the one or more processors, at least one of the one or more processors has the at least one in the live folder corresponding to the remote participant object. The instruction is to store the file and thereby cause the at least one file to be sent to the network address of the remote computing device over the web socket connection. Or at least one of multiple processors,
To store the at least one file in the live folder,
Sending a copy of the at least one file to the mapped network address through the web socket connection, the web socket connection to the mapped network address through the server. Sending and sending, configured to route data
The local computing device of claim 8, which further deletes the at least one file from the live folder upon completion of the transmission. The local computing device of claim 8, wherein the live folder is stored in a temporary cache on the local computing device. When executed by a local computing device, the local computing device,
Sending a representation of a collaborative workspace hosted on a server and accessible to multiple participants over a web socket connection on the user interface of the local computing device. Sending that the representation of the collaborative workspace contains one or more remote participant objects corresponding to one or more remote computing devices connected to the server.
Creating one or more live folders corresponding to the one or more remote participant objects, each live folder of a remote computing device corresponding to the remote participant object. Mapped to a network address, generated and
To detect user input for dragging at least one icon corresponding to at least one file proximal to the remote participant object in the one or more remote participant objects, said remote participant. Detecting that an object corresponds to a remote computing device in the one or more remote computing devices.
Store the at least one file in a live folder corresponding to the remote participant object, thereby bringing the at least one file to the network of the remote computing device via the web socket connection. • At least one non-transitory computer-readable medium that stores computer-readable instructions that cause it to be sent to an address. The one or more remote participant objects corresponding to the one or more remote computing devices
The at least one non-transitory computer-readable medium of claim 15, comprising one or more of an embedded video stream or remote participant icon. The collaborative workspace comprises a digital whiteboard configured to convey edits from any participant among the plurality of participants to other participants via the web socket connection. The at least one non-transitory computer-readable medium of claim 15, wherein said representation of the digital whiteboard is customized for local participants. When executed by the local computing device, cause the local computing device to generate one or more live folders corresponding to the one or more remote participant objects. The instruction sends to the local computing device.
Querying the server for one or more Internet Protocol (IP) addresses of the one or more remote computing devices corresponding to the one or more remote participant objects.
Creating one or more local folders corresponding to the one or more remote participant objects.
15. At least according to claim 15, which further creates the one or more live folders by mapping the one or more local folders to the one or more IP addresses. One non-temporary computer-readable medium. The local computing device is configured to store one or more spatial locations of the one or more remote participant objects in the user interface and is configured by the local computing device. When executed, to drag at least one icon corresponding to at least one file to the local computing device proximal to the remote participant object in the one or more remote participant objects. The instruction that causes the user input to be detected causes the local computing device to receive the instruction.
15. The 15th aspect, wherein the user input for dragging the at least one icon from the spatial position of the remote participant object to a destination spatial position within a threshold distance is further detected. At least one non-temporary computer-readable medium. When executed by the local computing device, the local computing device stores the at least one file in a live folder corresponding to the remote participant object, thereby at least one. The instruction to cause the remote computing device to send the file over the web socket connection to the network address of the remote computing device causes the local computing device to send the file.
To store the at least one file in the live folder,
Sending a copy of the at least one file to the mapped network address through the web socket connection, the web socket connection to the mapped network address through the server. Sending and sending, configured to route data
The at least one non-transitory computer-readable medium according to claim 15, which further deletes the at least one file from the live folder upon completion of the transmission.

Images