JP2021523484A - Methods, devices, and computer-readable media for delivering cropped images through websocket connections in a networked collaborative workspace. - Google Patents

Abstract

Hosted on a server to send a representation of a collaborative workspace, including images, to select image parts of an image, to detect user input corresponding to multiple coordinates, and to interface with an operating system or application. Sending coordinates to a transparent layer with an application programming interface configured to capture image parts based on the coordinates, at least partially, and dragging the selected image part to a location. Multiple commands configured to detect a second user input to do so and to remove the image from the collaborative workspace and insert the image portion into the collaborative workspace, at least partially based on location. Systems, methods, and computer-readable media for transmitting cropped images through web socket connections in a networked collaborative workspace, including sending to a server.

Description

The operating system and applications running within the operating system frequently utilize external hardware devices to allow the user to give input to the program and to give output to the user. Common examples of external hardware devices include keyboards, computer mice, microphones, and external speakers. These external hardware devices interface with the operating system through the use of drivers. A driver is a dedicated software program configured to interface between a hardware command used by a particular hardware device and an operating system.

Applications may be designed to interface with certain hardware devices. For example, voice text word processing applications can be designed to interface with audio headsets, including microphones. In this case, the application must be configured, in particular, to receive voice commands, perform voice recognition, convert the recognized words into text content, and output the text content to a document. This functionality will generally be implemented in the application programming interface (API) of an application, which is a defined set of methods of communication between various software components. In an example of a speech recognition application, the API can include an interface between the application program and the software on the driver, which serves to interface with the hardware device (microphone) itself.

One problem with existing software that utilizes dedicated hardware devices is that the application or operating system software itself must be customized and specially designed to utilize the hardware device. .. This customization does not allow the hardware device to exceed the scope defined for it by the application and cannot be utilized for contexts other than the specific application in which it was designed to be used. Means. For example, a user of a voice text word processing application may have other application programs or other components in the operating system, voice commands received by those other application programs or operating system through a microphone. It cannot be operated using voice commands unless it is specifically designed to take advantage of.

FIG. 1 shows an example of an existing architecture of a system that utilizes combined hardware devices for user input. The operating system 100A of FIG. 1 includes running applications 101A and 102A, each of which has its own APIs 101B and 102B, respectively. The operating system 100A also has its own API 100B and dedicated drivers 100C, 101C, and 102C configured to interface with the hardware devices 100D, 101D, and 102D.

As shown in FIG. 1, the application API 101B is configured to interface with the driver 101C, which itself interfaces with the hardware device 101D. Similarly, the application API 102B is configured to interface with the driver 102C, which itself interfaces with the hardware device 102D. At the operating system level, the operating system API 100B is configured to interface with the driver 100C, which itself interfaces with the hardware device 100D.

The system architecture shown in FIG. 1 limits the ability of a user to use a hardware device outside an application or operating system context. For example, the user may not be able to use the hardware device 101D to give input to application 102A and may use the hardware device 102D to give input to application 101A or operating system 100A. Can not.

Therefore, there is a need to improve the hardware software interface to allow the use of hardware devices in multiple software contexts.

FIG. 5 illustrates an example of an existing architecture of a system that utilizes combined hardware devices for user input. It is a figure which shows the architecture of the system which utilizes a universal hardware software interface by an exemplary example. It is a flowchart for the implementation of the universal hardware software interface by an exemplary embodiment. One or more hardware devices when the information captured by one or more hardware devices communicatively coupled to the system, according to an exemplary embodiment, contains one or more images. It is a flowchart for determining a user input based at least partially based on the information captured by. It is a figure which shows an example of object recognition by an exemplary example. It is a figure which shows an example of determining the input location coordinates by an exemplary embodiment. User input based at least in part on the captured information when the information captured by one or more hardware devices communicatively coupled to the system is sound information, according to an exemplary embodiment. It is a flowchart for determining. It is a figure which shows the tool interface which can be a part of a transparent layer by an exemplary embodiment. It is a figure which shows an example of a stylus which can be a part of a system by an exemplary embodiment. FIG. 5 is a flowchart for identifying a context corresponding to user input according to an exemplary embodiment. FIG. 5 is a diagram illustrating an example of using input coordinates to determine context, according to an exemplary embodiment. It is a flowchart for converting a user input into a transparent layer command by an exemplary embodiment. FIG. 5 is a diagram illustrating an example of receiving input coordinates when the selection mode is toggled, according to an exemplary embodiment. FIG. 5 is a diagram illustrating an example of receiving input coordinates when the pointing mode is toggled, according to an exemplary embodiment. FIG. 5 is a diagram illustrating an example of receiving input coordinates when the drawing mode is toggled, according to an exemplary embodiment. FIG. 5 is a diagram illustrating an example of a transparent layer command determined based on one or more words identified in input voice data according to an exemplary embodiment. It is a figure which shows another example of the transparent layer command which is determined based on one or more words identified in the input voice data by an exemplary example. It is a flowchart for executing one or more transparent layer commands on a transparent layer by an exemplary embodiment. It is a figure which shows the exemplary interface for adding a new command corresponding to a user input by an exemplary embodiment. It is a figure which shows the various components and options of a drawing interface and a drawing mode by an exemplary embodiment. An exemplary example of a calibration and configuration interface for a video camera hardware device that is used to recognize an object and allows the user to give input using touch and gestures. It is a figure which shows. FIG. 6 illustrates a general configuration interface that allows the user to customize various aspects of the interface, toggle input modes, and make other changes, according to an exemplary embodiment. It is a flowchart for transmitting a cropped image through a web socket connection in a networked collaborative workspace according to an exemplary embodiment. FIG. 5 illustrates the network architecture used to host and transmit a collaborative workspace, according to an exemplary embodiment. FIG. 5 illustrates a process for communicating edits to a collaborative workspace within a network, according to an exemplary embodiment. It is a figure which shows a plurality of representations of a collaborative workspace by an exemplary example. It is a figure which shows an example of the cropping input detection by an exemplary example. It is a figure which shows an example of the cropping input detection by an exemplary example. It is a figure which shows an example of the cropping input detection by an exemplary example. It is a figure which shows an example of the process flow for transmitting a coordinate to a transparent layer and capturing an image part by an exemplary example. FIG. 5 illustrates an example of a user dragging an image portion to a location within a collaborative workspace, according to an exemplary embodiment. It is a flowchart for sending a plurality of commands to a collaborative workspace hosted on a server according to an exemplary embodiment. It is a flowchart for sending a plurality of commands to a collaborative workspace hosted on a server for two different contexts according to an exemplary embodiment. It is a flowchart for sending a plurality of commands to a collaborative workspace hosted on a server for two different contexts according to an exemplary embodiment. It is a figure which shows an example of the result of a user dragging an image part into an editing interface of a collaborative workspace and dragging an image part into an editing interface according to an exemplary example. It is a figure which shows an example of the result of a user dragging an image part into an editing interface of a collaborative workspace and dragging an image part into an editing interface according to an exemplary example. It is a figure which shows an example of the result of a user dragging an image part into an editing interface of a collaborative workspace and dragging an image part into an editing interface according to an exemplary example. FIG. 5 shows an example of the result of a user dragging an image portion onto the toolbar interface of a collaborative workspace and dragging the image portion onto the toolbar interface according to an exemplary embodiment. FIG. 5 shows an example of the result of a user dragging an image portion onto the toolbar interface of a collaborative workspace and dragging the image portion onto the toolbar interface according to an exemplary embodiment. FIG. 5 shows an example of the result of a user dragging an image portion onto the toolbar interface of a collaborative workspace and dragging the image portion onto the toolbar interface according to an exemplary embodiment. FIG. 5 shows an example of the result of a user dragging an image portion onto the toolbar interface of a collaborative workspace and dragging the image portion onto the toolbar interface according to an exemplary embodiment. FIG. 5 shows an example of the result of a user dragging an image portion onto the toolbar interface of a collaborative workspace and dragging the image portion onto the toolbar interface according to an exemplary embodiment. FIG. 5 illustrates an exemplary computing environment configured to perform the disclosed method.

Methods, devices, and computer-readable media are described herein as examples and examples, but those skilled in the art will appreciate methods, devices, and computer-readable media for implementing universal hardware software interfaces. Recognize that it is not limited to the embodiments or drawings described. It should be understood that the drawings and description are not limited to the particular format disclosed. Rather, the intent is to cover the intent and all modifications, equivalents and alternatives that fall within the scope of the appended claims. Any headings used herein are for organization purposes only and are not intended to limit the scope of the specification or claims. As used herein, the word "can" has an acceptable meaning (ie, potential) rather than a compulsory meaning (ie, meaning must). Meaning). Similarly, the words "include," "include," and "includes" mean, but are not limited to.

Applicants have discovered methods, devices, and computer-readable media to solve problems associated with previous hardware software interfaces used for hardware devices. In particular, Applicants have developed a universal hardware software interface that allows users to utilize communicably coupled hardware devices in a variety of software contexts. The disclosed implementations are custom designed to allow an application or operating system to interface with a particular hardware device by using dedicated virtual drivers and corresponding transparent layers, as described in more detail below. Get rid of the need.

FIG. 2 shows the architecture of a system utilizing a universal hardware software interface according to an exemplary embodiment. As shown in FIG. 2, the operating system 200A includes a transparent layer 203 that communicates with the virtual driver 204. As described in more detail below, the transparent layer 203 is configured to interface between the virtual driver and the operating system and / or the application (s) running on the operating system. It is an API. In this example, the transparent layer 203 interfaces between the virtual driver 204 and the API 201B of the application 201A, the API 202B of the application 202A, and the operating system API 200B of the operating system 200A.

The transparent layer 203 can be part of a software process running on an operating system, allowing the transparent user interface (UI) and / or user to interact on top of the underlying user interface. It is possible to have a UI element of the transparent layer 203 itself, including a visible UI element.

The virtual driver 204 is configured to emulate drivers 205A and 205B, which interface with hardware devices 206A and 206B, respectively. The virtual driver tells the virtual driver which virtual driver to emulate, eg, voice commands, selections made on the user interface, and / or gestures made by the user in front of the combined web camera. Can receive user input in the form of. For example, each of the connected hardware devices can operate in "listening" mode, and each of the emulated drivers in virtual driver 204 signals to the virtual driver to switch to a particular emulation mode. It can be configured to detect an initialization signal that acts as. For example, a user saying "start voice command" can activate the driver corresponding to the microphone to receive a new voice command. Similarly, giving a gesture by the user can activate the driver corresponding to the webcam for receiving the gesture input or touch input.

The virtual driver may also be configured to interface with a native driver, such as the native driver 205C, which itself communicates with the hardware device 206C. In one embodiment, the hardware device 206C can be a standard input device such as a keyboard or mouse, which is natively supported by the operating system.

The system shown in FIG. 2 allows a user to interface any combined hardware device in various contexts, such as a particular application or operating system, so that the application or operating system interfaces with the hardware device. Allows implementation of universal hardware software interfaces that can be used without the need for customization.

For example, hardware device 206A can capture information, which is then received by virtual driver 204, which emulates driver 205A. The virtual driver 204 can determine the user input based on the captured information. For example, if the information is a series of images in which the user moves his hand, the virtual driver can determine that the user has performed a gesture.

Based on the identified context (such as a particular application or operating system), user input can be translated into transparent layer commands for execution and sent to transparent layer 203. Transparency commands can include native commands in the identified context. For example, if the identified context is application 201A, the native command will be in a format that matches application API201B of application 201A. The execution of a transparent layer command can then be configured to trigger the execution of one or more native commands in the identified context. This is achieved by the transparent layer 203 interfacing with each of the APIs of the application running on the operating system 200A and the operating system API 200B. For example, if the native command is an operating system command, such as a command to launch a new program, the transparent layer 203 can give the native command to the operating system API 200B for execution.

As shown in FIG. 2, there is bidirectional communication between all of the components shown. This means that, for example, the execution of a transparent layer command on transparent layer 203 may result in the transmission of information to the virtual driver 204 and the transmission of information up one of the connected hardware devices. means. For example, a voice command is recognized as input, converted to a transparent layer command containing a native command, executed by the transparent layer (causing the execution of the native command in the identified context), and then the sound output "command is A signal can be sent from the transparent layer (via the virtual driver) to the speaker to send "received".

Of course, the architecture shown in Figure 2 is for illustration purposes only, and the number of applications running, the number and type of hardware devices connected, and the number of drivers and emulated drivers can vary. Please understand that there is.

FIG. 3 is a flowchart for implementing a universal hardware software interface according to an exemplary embodiment.

In step 301, user input is determined based at least in part on the information captured by one or more hardware devices communicatively coupled to the system. A system as used herein is a computing device that performs a method step, a device that comprises one or more processors that perform a method step and one or more memories. Or it may refer to any other computing system.

User input can be determined by a virtual driver running on the system. As mentioned earlier, a virtual driver may be operating in embroidery mode, in which the virtual driver emulates another hardware driver, thereby hardware device. To receive captured information from, or optionally from one or more other hardware drivers configured to interface with a particular hardware device. Can be done.

Motion capture devices including cameras, video cameras, microphones, headsets with two-way communication, mice, touchpads, trackpads, controllers, gamepads, joysticks, touch screens, accelerometers and / or tilt sensors. Various hardware devices may be utilized, such as a remote controller, stylus, or any combination of these devices. Of course, this list of hardware devices is given merely as an example, and any hardware device that can be used to detect audio, image, video, or touch information can be utilized.

Communication coupling between a hardware device and a system can take various forms. For example, hardware devices can communicate with the system via wireless networks, Bluetooth protocols, wireless frequencies, infrared signals, and / or by physical connections such as universal serial bus (USB) connections. Communication can also include both wireless and wired communication. For example, a hardware device can contain two components, one of which wirelessly signals a second component (such as through Bluetooth), and the second component itself (such as USB). ) Connect to the system via a wired connection. Various communication techniques may be utilized in accordance with the systems described herein, and examples thereof are not limited.

The information captured by one or more hardware devices can be any type of information such as image information containing one or more images, video frames, sound information, and / or touch information. The captured information is for sound information. wav or. For mp3 files, images. It can be in any suitable format, such as a jpg file, numerical coordinates for touch information, and so on.

The techniques described herein are virtually as "touch" screen devices in any context, even if any display device does not include the hardware to detect touch signals or touch-based gestures. It can be made possible to work. This is described in more detail below and can be achieved through analysis of images captured by cameras or video cameras.

FIG. 4 is captured by one or more hardware devices when the information captured by one or more hardware devices communicatively coupled to the system contains one or more images. It is a flowchart for deciding a user input based at least partly based on the information.

In step 401, one or more images are received. These images can be captured by a hardware device such as a camera or video camera and received by a virtual driver, as described above.

In step 402, objects in one or more images are recognized. The object can be, for example, a user's hand, finger, or other body part. The object can also be a dedicated device such as a stylus or pen, or a dedicated hardware device such as a motion tracking stylus / remote controller that is communicably coupled to the system and includes an accelerometer and / or tilt sensor. .. Object recognition can be performed by virtual drivers and can be based on previous training, such as through calibration routines that operate with objects.

FIG. 5A shows an example of object recognition according to an exemplary embodiment. As shown in FIG. 5A, image 501 includes a user's hand recognized as object 502. The recognition algorithm can, of course, be configured to recognize different objects such as fingers.

Returning to FIG. 4, in step 403, one or more orientations and one or more positions of the recognized object are determined. This can be achieved in a variety of ways. If the object is not a hardware device, but instead a body part such as a hand or finger, the object determines the 3D coordinates of the object and various angles with respect to the X, Y, and Z axes. Therefore, it can be mapped in a three-dimensional coordinate system that uses a known location of the camera as a reference point. If the object is a hardware device and includes motion tracking hardware such as an accelerometer and / or tilt sensor, the image information is provided by the accelerometer and / or tilt sensor to determine the position and orientation of the object. Can be used with the information provided.

In step 404, the user input is determined at least in part based on one or more orientations and one or more positions of the recognized object. This can include determining location coordinates on the transparent user interface (UI) of the transparent layer based at least in part on one or more orientations and one or more positions. The transparent UI is part of the transparent layer and is overlaid on top of the underlying UI that corresponds to any application running on the operating system and / or operating system.

FIG. 5B shows an example of this step when the object is a user's finger. As shown in FIG. 5B, the display device 503 includes a lower UI 506 and a transparent UI 507 overlaid on the lower UI 506. For clarity, the transparent UI 507 is shown with dot shading, but it should be understood that the transparent UI is actually a transparent layer that is not visible to the user. Further, although the transparent UI 507 is shown as being slightly smaller than the lower UI 506, it should be understood that in practice the transparent UI will cover the same screen area as the lower UI.

As shown in FIG. 5B, the position and orientation information of the object (user's finger) is used to project a line onto the plane of the display device 503 and determine the intersection 505. The image information captured by the camera 504 and the known position of the display device 503 under the camera can be used to aid in this projection. As shown in FIG. 5B, the user input is determined to be the input coordinates at the intersection 505.

As further described below, the actual transparent layer commands generated based on this input may be based on user settings and / or identified contexts. For example, the command can be a touch command indicating that the object at the coordinates of point 505 should be selected and / or opened. The command can also be a pointing command indicating that the pointer (such as a mouse pointer) should be moved to the coordinates of point 505. In addition, the command can be an edit command that modifies the graphical output at the location (for example, to annotate the interface or draw an element).

FIG. 5B shows the recognized object 502 as being at some distance from the display device 503, but touch input can be detected regardless of the distance. For example, if the user physically touches the display device 503, the technique described above will still determine the input coordinates. In that case, the projection line between the object 502 and the intersection would only be shorter.

Of course, touch input is not the only type of user input that can be determined from the captured image. The step of determining a user input based at least in part on one or more orientations and one or more positions of a recognized object can include determining a gesture input. In particular, the position and orientation of the recognized object across multiple images can be analyzed to determine the corresponding gesture, such as a swipe gesture, a pinch gesture, and / or any known or customized gesture. The user can calibrate the virtual driver to recognize custom gestures that map to specific contexts and commands within those contexts. For example, a user can create a custom gesture that maps to an operating system context and results in the execution of a native operating system command that launches a particular application.

As mentioned above, the information captured by one or more hardware devices in step 301 of FIG. 3 may also include sound information captured by a microphone. FIG. 6 determines user input based at least in part on the captured information when the information captured by one or more hardware devices communicatively coupled to the system is sound information. It is a flowchart for. As described below, speech recognition is performed on the sound information to identify one or more words corresponding to user input.

In step 601 the sound data is received. Sound data can be captured by a hardware device such as a microphone and received by a virtual driver, as described above. In step 602, the received sound data can be compared with a sound dictionary. The sound dictionary can include sound signatures of one or more recognized words, such as command words or command modifiers. In step 603, one or more words in the sound data are identified as user input based on comparison. The identified word may then be converted to a transparent layer command and passed to the transparent layer.

As mentioned above, the driver emulated by the virtual driver, the expected type of user input, and the commands generated based on the user input are all at least part of one or more settings or previous user input. Can be determined on the basis of.

FIG. 7 shows a tool interface 701 that can also be part of a transparent layer. Unlike the transparent UI, the tool interface 701 is visible to the user and can be selected between different options to change the virtual driver emulation mode, native commands generated based on user input, or additional Can be used to perform a function.

Button 701A is used by the user to graphically modify the user interface when the user input is input coordinates (such as coordinates based on the user touching the screen with a hand or stylus / remote controller). Allows you to select the type of drawing tool that will be used. Various drawing tools can include different brushes, colors, pens, highlighters, and so on. These tools can make graphical changes in varying styles, thicknesses, colors, etc.

Button 701B allows the user to switch between selection mode, pointing mode, or drawing mode when the input coordinates are received as user input. In selection mode, the input coordinates are treated as a "touch", which can result in selecting or opening an object in the input coordinates. In pointing mode, the coordinates can be treated as pointer positions (such as mouse pointers), effectively allowing the user to emulate the mouse. In drawing mode, the coordinates can be treated as a location for modifying the graphical output of the user interface to present the appearance of drawing or writing on the user interface. The nature of the changes may depend on the drawing tool selected, as described for Button 701A. Button 701B can also alert the virtual driver to anticipate image and / or motion input (if motion tracking devices are used) and emulate the appropriate driver accordingly.

Button 701C alerts the virtual driver to anticipate voice commands. This can cause the virtual driver to emulate the driver corresponding to the combined microphone for receiving the voice input and parsing the voice input, as described with respect to FIG.

Button 701D opens a launcher application that can be part of a transparent layer and can be used to launch an application within the operating system or to launch a specific command within the application. Launchers are used to customize options in the transparency layer, such as custom voice commands, custom gestures, and custom native commands for applications associated with user input, and / or (voice calibration, motion capture devices). It can also be used to calibrate hardware devices and user inputs (such as calibration and / or object recognition calibration).

Button 701E can be used to capture a screenshot of the user interface and to export the screenshot as an image. It can be used with the drawing mode of button 701B and the drawing tool of 701A. After the user marks up a particular user interface, the marked up version can be exported as an image.

Button 701F also allows graphical editing and can be used to change the color or mode of drawing that the user is creating on the user interface. Similar to the drawing mode of button 701B, this button changes the nature of the graphical changes in the input coordinates.

Button 701G erases the drawing on the user interface. This button selection removes all graphical markings on the user interface and resets the underlying UI to what it was before the user created the drawing.

Button 701H can be used to launch a whiteboard application that allows the user to create a draw or write on a virtual whiteboard using the draw mode.

Button 701I can be used to add text notes to objects such as those shown in the operating system UI or application UI. Text notes can be interpreted from audio signals or typed by the user using the keyboard.

Button 701J can be used to open or close the tool interface 701. When closed, the tool interface can be minimized or completely removed from the subordinate user interface.

As mentioned above, a stylus or remote hardware device can be used with the system, along with other hardware devices such as cameras or video cameras. FIG. 8 shows an example of a stylus 801 that can be used with this system. The stylus 801 can communicate with the hardware receiver 802 through Bluetooth or the like. The hardware receiver can be connected to the computer system via USB802B or the like, and the signal from the stylus passed to the computer system via the hardware receiver is the tool shown in FIG. It can be used to control and interact with menu 803, which is similar to the interface.

As shown in FIG. 8, the stylus 801 can include a physical button 801A. These physical buttons 801 can be used to power on the stylus, navigate menu 803, and make selections. In addition, the stylus 801 can include a characteristic tip 801B that is captured in the image taken by the camera and recognized by the virtual driver. This can allow the stylus 801 to be used for drawing and editing when in drawing mode. The stylus 801 can also include motion tracking hardware such as an accelerometer and / or a tilt sensor to assist in position detection when used to provide input coordinates or gestures. In addition, the hardware receiver 802 can include a calibration button 802A, which, when pressed, can activate the calibration utility in the user interface. This allows the stylus to be calibrated.

Returning to FIG. 3, in step 302, the context corresponding to the user input is identified. The identified context comprises the operating system or one of the applications running on the operating system.

FIG. 9 is a flowchart for identifying a context corresponding to user input according to an exemplary embodiment. As shown in FIG. 9, operating system data 901, application data 902, and user input data 903 can all be used to determine context 904.

Operating system data 901 can include, for example, information about the active window in the operating system. For example, if the active window is a calculator window, the context can be determined to be a calculator application. Similarly, if the active window is a Microsoft Word window, the context can be determined to be a Microsoft Word application. On the other hand, if the active window is a file folder, the active context can be determined to be the operating system. Operating system data should also include additional information such as which application is currently running, the last launched application, and any other operating system information that can be used to determine the context. Can be done.

Application data 902 can include, for example, information about one or more applications running and / or information that maps a particular application to some type of user input. For example, a first application may be mapped to a voice input such that the context is automatically determined to be the first application whenever a voice command is received. In another example, when a particular gesture is received as input, the gesture causes the second application to be launched or closed, or to take some action within the second application. Can be associated with a second application.

User input 903 can also be used to determine context in various ways. As mentioned above, some type of user input can be mapped to an application. In the above example, the voice input is associated with the context of the first application. In addition, user-input attributes can also be used to determine the context. Gestures or movements can be mapped to applications or operating systems. Certain words in voice commands can also be mapped to applications or operating systems. Input coordinates can also be used to determine the context. For example, a window in the user interface at the location of the input coordinates can be determined, and the application corresponding to that window can be determined as the context.

FIG. 10 shows an example of using input coordinates to determine the context. As shown in FIG. 10, the display device 1001 displays the user interface 1002. Also shown are the camera 1004 and the transparent layer 1003 overlaid on the lower user interface 1003. The user utilizes the stylus 1000 to point to location 1005 in user interface 1002. Since location 1005 is in the application window corresponding to application 1, application 1 can be determined to be the context for user input, as opposed to application 2, application 3, or the operating system.

Returning to FIG. 3, in step 303, the user input is transformed into one or more transparent layer commands based at least in part based on the identified context. As mentioned above, the transparent layer comprises an application programming interface (API) configured to interface between the virtual driver and the operating system and / or the application running on the operating system.

FIG. 11 is a flowchart for converting user input into a transparent layer command. As shown in step 1104 of FIG. 11, the transparent layer command can be determined at least in part based on the identified context 1102 and user input 1103. A transparent layer command can include one or more native commands configured to execute in one or more corresponding contexts. The transparent layer command can also include a response output that should be sent to the virtual driver and on the hardware device (s).

The identified context 1102 can be used to determine which transparent layer command should be mapped to user input. For example, if the identified context is "operating system", swipe gesture input will cause the user interface to minimize one open window and maximize the next). It can be mapped to a transparent layer command that results in scrolling the currently open window in the operating system. Alternatively, if the identified context is a "web browser application", the same swipe gesture input can be mapped to a transparent layer command that causes the web page to scroll.

User input 1103 also determines transparent layer commands because user input is clearly mapped to native commands in one or more contexts, and these native commands are part of the transparent layer commands. For example, the voice command "Open Email" can be mapped to a specific operating system native command to launch the email application Outlook. When a voice input containing the recognized word "Open Email" is received, this results in a transparent layer command being determined, including a native command to launch Outlook.

As shown in FIG. 11, the transparent layer command may also be determined based on one or more user settings 1101 and API library 1104. The API library 1104 can be used to look up native commands that correspond to an identified context and a particular user input. In the swipe gesture and web browser application context examples, the API library corresponding to the web browser application can be queried for the appropriate API call to cause web page scrolling. Alternatively, API library 1104 can be omitted and native commands can be mapped towards specific user inputs and identified contexts.

In situations where user input is determined to be input coordinates, the transparent layer command is determined at least in part based on the input location coordinates and the identified context. In this case, the transparent layer command can contain at least one native command in the identified context, and at least one native command is configured to perform an action at the corresponding location coordinate in the underlying UI. NS.

Setting 1101 can be used to determine the corresponding transparent layer command when there are two or more possible actions that are mapped to a particular context and user input. For example, button 701B of FIG. 7 allows the user to select between a selection mode, a pointing mode, or a drawing mode when the input coordinates are received as user input. This setting can be used to determine the transparent layer command and, by extension, which native command will be executed and which action will be performed. In this case, the possible native command is a selection command configured to select the object associated with the corresponding location coordinate in the lower UI, configured to move the pointer to the corresponding location coordinate in the lower UI. It can include pointer commands that have been made, and graphical commands that are configured to change the display output at the corresponding location coordinates in the underlying UI.

FIG. 12A shows an example of receiving input coordinates when the selection mode is toggled. As shown in FIG. 12A, the user points to the stylus 1200 in operating system UI 1202 (with superposed transparent UI 1203) on display device 1201. Similar to the previous example, the camera 1204 can be used to determine the position and orientation information as well as the input coordinates for the stylus 1200. The determined transparent layer command selects the object associated with the input coordinates (in this case, folder 1205), as the selection mode is toggled and the stylus 1200 is pointed to in folder 1205 in operating system UI 1202. Can contain native operating system commands for. In another example, if the window is located in input coordinates, this will result in a selection of the entire window.

FIG. 12B shows an example of receiving input coordinates when the pointing mode is toggled. In this case, the determined transparent layer command can include a native operating system command to move the mouse pointer 1206 to the location of the input coordinates.

FIG. 12C shows an example of receiving input coordinates when the drawing mode is toggled and the user sweeps the stylus 1200 over a plurality of input coordinates. In this case, the determined transparent layer command can include a native operating system command to change the display output at each location of the input coordinates, and the user's drawing line 1207 on the user interface 1202. Occurs. The modified graphical output provided in the drawing mode can be stored as part of the transparent layer 1203, for example, as metadata related to the path of the input coordinates. The user can then select an option to export the modified display output as an image.

In situations where user input is identified as a gesture, translating user input into one or more transparent layer commands based at least in part on the identified context is at least to the identified gesture and the identified context. It can include determining the transparent layer command on a partial basis. A transparent layer command can contain at least one native command in the identified context, and at least one native command is configured to perform the action associated with the identified gesture in the identified context. Will be done. An example of this has been described above with respect to swipe gestures and web browser application contexts that result in native commands configured to perform scrolling actions in a web browser.

In situations where user input is identified as one or more words (such as by using speech recognition), convert user input to one or more transparent layer commands based at least in part on what is identified. What you do can include determining the Stratum lucidum command based at least in part on the identified word and the identified context. A transparent layer command can include at least one native command in the identified context, and at least one native command can perform an action associated with one or more identified words in the identified context. Configured to carry out.

FIG. 13 shows an example of the transparent layer command 1300 determined based on one or more words identified in the input voice data. The identified word 1301 includes one of the phrases "whiteboard" or "blank page". The transparent layer command 1300 also includes a description of the command 1302 and a response command 1303 which is an output command sent by the transparent layer to the virtual driver and to the hardware output device when the transparent layer command is executed. In addition, the transparent layer command 1300 includes the actual native command 1304 used to call the whiteboard function.

FIG. 14 shows another example of the transparent layer command 1400 determined based on one or more words identified in the input voice data according to an exemplary embodiment. In this example, one or more words are "open email". As shown in FIG. 14, the transparent layer command 1400 includes a native command "outlook.exe", which is an instruction to operate a specific executable file that starts an outlook application. The transparent layer command 1400 also includes a voice response "opened an email" that is output in response to receiving a voice command.

Returning to FIG. 3, in step 304, one or more transparent layer commands are executed on the transparent layer. Execution of one or more transparent layer commands is configured to trigger execution of one or more native commands in the identified context.

FIG. 15 is a flowchart for executing one or more transparent layer commands on the transparent layer according to an exemplary embodiment. In step 1501, at least one native command in the transparent layer command is identified. Native commands can be specified, for example, as native commands within the structure of transparent layer commands, allowing identification.

In step 1502, at least one native command is executed in the identified context. This step can include passing at least one native command to the identified context via the API identified for that context and executing the native command within the identified context. .. For example, if the identified context is the operating system, native commands may be passed to the operating system for execution via the operating system API. In addition, if the identified context is an application, native commands may be passed to the application for execution via the application API.

Optionally, at step 1503, the response may be sent to the hardware device (s). As mentioned above, this response can be routed from the transparent layer to the virtual driver and onto the hardware device.

16-19 show additional features of the system disclosed herein. FIG. 16 shows an exemplary interface for adding new commands corresponding to user input, according to an exemplary embodiment. A dashboard at interface 1600 includes an icon for application 1601 that has already been added and can be launched using predetermined user input and hardware devices (eg, voice commands). The dashboard is application-specific and can also show other commands that are mapped to one user input. Selecting the add button 1602 opens the add command menu 1603. This menu allows the user to choose between the following options: Item Type: Fixed Items to Add on Bottom Bar Menu / Regular Items to Add in Drag Menu, Icon: Select image icon, Background: Select background icon color, Color: Select icon color, Name: Set new item name, Voice command: Set voice activation command to open new application, Feedback Response: Application Voice Response Set feedback, Command: Select the application type or custom command type to launch (eg, launch an application command, perform an action within an application command, application (Close command, etc.), Start process: When starting a new process or application, the name of the process or application, and parameters: Any parameters that should be passed to the new process or application.

FIG. 17 shows various components and options of drawing interface 1700 and drawing modes according to exemplary embodiments. FIG. 18 shows a calibration and configuration interface 1800 for a video camera hardware device that is used to recognize an object and allows the user to give input using touch and gesture. FIG. 19 shows a general configuration interface 1900 that allows the user to customize various aspects of the interface, toggle input modes, and make other changes. As shown in Interface 1900, the user can also access the settings page for calibrating and adjusting the settings for the hardware stylus (called the "magic stylus").

The systems disclosed herein can be implemented on multiple networked computing devices and used to assist in conducting networked collaborative sessions. For example, the whiteboard functionality described above can be a shared whiteboard among multiple users on multiple computing devices.

Whiteboards and communal spaces often contain images that are the subject of discussion or the subject of a particular conference. You will need to edit the images in the collaborative workspace and interact with them many times. For example, a graphic design team may want to edit a particular design and suggest a review to colleagues. One of the problems with existing whiteboards or other shared communal spaces is co-working on the fly, in real time, in a way that changes are automatically communicated to other participants in the co-workspace. There is currently no way to clip, edit, or otherwise review images in the workspace. For example, the process of modifying and clipping an image is difficult because the user must incorporate the image into another application for editing and modification, and then re-import the image into a collaborative program. This process causes a loss of productivity in the meeting where the editing is performed, such as it takes time for a particular user to export the image to an editing program, make the necessary edits, and then re-import the image. Moreover, many users are unfamiliar or unskilled in performing these types of operations. What's more, the process of exporting and re-importing images is the resources needed to export and import images, the resources needed to store the exported images, and the resources needed to launch and use editing software. , As well as wasting computing resources such as the resources needed to store edited images. This puts a strain on local computing devices, in addition to the computing resources that can be used for the collaborative session itself.

In addition to the methods and systems described above for implementing universal hardware software interfaces, applicants can be allowed to convey cropped images through websocket connections in a networked collaborative workspace, Further discoveries of equipment and computer readable media.

FIG. 20 is a flow chart for transmitting a cropped image through a websocket connection in a networked collaborative workspace, according to an exemplary embodiment. All of the steps shown in FIG. 20 can be performed on a local computing device, such as a client device connected to a server, and does not require multiple computing devices.

In step 2001, a representation of the collaborative workspace hosted on the server is transmitted over the user interface of the local computing device. The collaborative workspace is accessible to multiple participants on multiple computing devices through WebSocket connections. Representations of a collaborative workspace can include one or more images, such as images discussed or utilized in a collaborative session.

FIG. 21A shows a network architecture used to host and transmit a collaborative workspace, according to an exemplary embodiment. As shown in FIG. 21A, the server 2100 is connected to computing devices 2101A-2101F. The server 2100 and the computing devices 2101A-2101F are connected via a network connection such as a web socket connection that enables bidirectional communication between the computing devices 2101A-2101F (client) and the server 2100. Can be done. As shown in FIG. 21A, the computing device can be any type of computing device, such as a laptop, desktop, smartphone, or other mobile device.

A collaborative workspace can be, for example, a digital whiteboard configured to convey edits from any participant in multiple participants to other participants through a websocket connection. FIG. 21B shows a process for communicating edits to a collaborative workspace within a network, according to an exemplary embodiment. As shown in FIG. 21B, when a user in computing device 2101B makes edits or changes to a collaborative workspace, the edits or changes 2102B are sent to server 2100 to update the hosted version of the workspace. Used for. The edit or modification is then transmitted by the server 2100 to the other connected computing devices 2101A, 2101C, 2101D, 2101E, and 2101F as updates 2102A, 2102C, 2102D, 2102E, and 2102F.

Each representation of a collaborative workspace can be a customized collaborative workspace version for local participants. For example, as mentioned above, each representation of a collaborative workspace can include one or more remote participant objects that correspond to one or more remote computing devices connected to a server.

FIG. 22 shows a plurality of representations of a collaborative workspace according to an exemplary embodiment. As shown in FIG. 22, server 2200 hosts a collaborative workspace 2201 that includes image 2202. The version of the collaborative workspace hosted on the server is communicated to the connected device as described above. FIG. 22 also shows a representation of a collaborative workspace for three connected users, user 1, user 2, and user 3. As shown, each representation includes all objects and edits in the collaborative workspace. For example, representations 2201A, 2201B, and 2201C all include representations of image 2202.

Returning to FIG. 20, in step 2002, the cropping tool running on the local computing device enters the user input to select the image portion of the image in one or more images shown in the collaborative workspace. To detect. This user input corresponds to a plurality of coordinates.

23A-23C show an example of this detection process according to an exemplary embodiment. As shown in FIG. 23A, user interface 2301 includes a representation of collaborative workspace 2302 including image 2306. The user interface 2301 may correspond to the user interface of a local computing device connected to the server hosting the collaborative workspace.

User interface 2301 also includes cropping tool interface 2303. As shown, the cropping tool interface 2303 can display indicators or icons corresponding to a plurality of different cropping tools that can be selected. Selectable cropping tool interfaces are, for example, square, circular, rectangular quoting box, oval quoting box, manual freehand cropping tool (indicated by scissors icon), or manual border cropping tool. It can include cropping tools that correspond to specific cropping shapes, such as (indicated by pencil and ruler icons). Of course, a variety of different cropping shapes are available, and these examples are not limited.

The cropping tool interface 2303 is shown as an interface separate from the representation of the collaborative workspace interface 2302, whereas the cropping tool interface is where each user's representation of the collaborative workspace provides the interface of the cropping tool. It should be understood that it can be integrated into a collaborative workspace representation to include. Similarly, the cropping tool is a separate program configured to interface with programs that support collaborative workspaces (and additional programs such as Internet browsers, word processing programs, and graphical editing programs). Although obtained, the cropping tool can also be implemented as a module or component of a collaborative workspace program.

FIG. 23B shows the user's selection of a particular cropping shape within the cropping tool interface 2303. As shown, the user has selected the circular shape 2304 as the cropping shape. The selection can be made using a pointing device such as a mouse, as indicated by the mouse pointer in the figure. The selection can also be made using a hand gesture or stylus, as described above. Selections can be detected using the techniques described above with virtual drivers and / or transparent layers.

FIG. 23C shows the user interface 2301 after selecting the circular cropping shape 2304 and after the user has selected the area to be cropped. The user may select an area for cropping, for example, by dragging and dropping the cropping shape to a specific location within interface 2301. The cropping shape selection can also bring up the contours of the cropping area of the selected shape, which the user can move or drag and drop to a specific location within interface 2301. The user can use the mouse to drag and drop. The user can also use the hand gesture or stylus to drag and drop, as described above. Drag-and-drop motion can be detected using the techniques described above with virtual drivers and / or transparent ones.

As shown in FIG. 23C, the user used the circular cropping shape of the cropping tool to select circular area 2305 in image 2306. As mentioned above, the selection can be made using user input from the pointing device. Once the selection is made, the cropping tool detects multiple coordinates corresponding to the input and cropping selection. These coordinates can be defined using the horizontal axis (X) and vertical axis (Y) of user interface 2301. In addition, the coordinates will correspond to the selected cropping area. In the example shown in FIG. 23C, the plurality of coordinates correspond to a circle. The coordinates themselves can be obtained by cropping tools from the operating system of the local computing device. For example, a cropping tool application (or, if the cropping tool is part of a collaborative workspace application, the collaborative workspace application) wants to request the coordinates corresponding to each point in the dashed circle 2305. , You can contact the operating system via API. Alternatively, the cropping tool can include functionality for determining multiple coordinates based on the location of the pointer when the selection was made. For example, if the cropping shape is a circle and the center of the circle corresponds to the location of the pointing device when the selection was made, the cropping application will use the circle (for example, by asking the operating system for the location of the pointer). The screen coordinates of the center of the can be determined, and those center coordinates can be used to calculate multiple coordinates corresponding to the cropping shape. In this case, the calculation can use the radius of the circle to project lines of equal length 360 degrees outward from the center point and record each endpoint of the resulting line as multiple coordinates. ..

Returning to FIG. 20, in step 2003, the cropping tool running on the local computing device transmits a plurality of coordinates corresponding to the boundaries of the cropped image portion to the aforementioned transparent layer. As mentioned above, the transparent layer is with one or more of one or more applications that run on a local computing device and are configured to run on an operating system or operating system. It has an application programming interface (API) configured to interface. In this case, the transparent layer is configured to interface with the cropping tool application. However, if the cropping tool is part of a collaborative workspace application, a local instance of the collaborative workspace application running on a local computing device can also send multiple coordinates to the transparent layer. can.

In step 2004, the transparent layer running on the local computing device captures an image portion based on at least a portion of the received coordinates. Step 2004 is an image defined by multiple coordinates for one or more of the operating system or applications running on the local computing device by a transparent layer running on the local computing device. Sending a request for a screen capture of a part, receiving a screen capture of an image part defined by multiple coordinates by a transparent layer running on a local computing device, and local computing. A transparent layer running on the wing device can include storing the image portion in memory such as the random access memory of the local computing device. Step 2004 can, of course, be performed in other ways. For example, step 2004 is a (“screen print” command) for one or more of the operating systems or applications running on the local computing device by a transparent layer running on the local computing device. Sending a request for a screen capture of the entire screen (by sending to the operating system, etc.), using features that are part of the transparency layer, based on at least some of the multiple coordinates of the screen It can be performed by extracting the image portion from the capture and then storing the image portion in a memory such as a random access memory of the local computing device.

FIG. 24 shows an example of the process flow of steps 2003 and 2004 of FIG. As shown, the user has captured the image portion defined by the boundary 2402 in the image 2406, which is displayed in the collaborative workspace 2401 in the user interface 2400. For simplicity, the cropping tool interface is not shown in this figure, but it should be understood that the cropping tool interface is also displayed in user interface 2400. The cropping tool application 2403 detects the coordinates of the cropping boundary (corresponding to the circle 2402) and sends those coordinates to the transparent layer 2404. The transparent layer 2404 then uses the plurality of coordinates to capture the cropped image portion and stores the resulting image portion in storage 2405 (eg, which may be random access memory).

Returning to FIG. 20, in step 2005, the transparent layer running on the local computing device detects a second user input for dragging the selected image portion to a location in the collaborative workspace. This second user input can be a drag-and-drop movement, an input using a pointing device, or an input using the hand gesture or stylus described above. Drag-and-drop motion can also be detected using the techniques described above with virtual drivers and / or transparent ones.

Locations within the collaborative workspace can include spaces within the "whiteboard" or other shared displays and editing areas, as described below, with any interface or toolbar that is part of the collaborative workspace. Can also be included.

Step 2005 may include detecting a second user input for dragging the selected image portion to a new location in the collaborative workspace by a cropping tool running on the local computing device. can. The new location can be determined, for example, by querying the operating system when a second user input is detected. Step 2005 also sends a new location to the transparent layer running on the local computing device by a cropping tool running on the local computing device (thus making the transparent layer a second input and). It can include changing to the location of the second input).

Alternatively, detection can be performed directly by the transparent layer based on the variables or values sent by the cropping tool when the coordinates are sent to the transparent layer. For example, if the cropping tool sends the coordinates to the transparent layer, the transparent layer will have both the location of the next input and the location of the next input (in this case, the "drop" input made by the user after dragging the image portion). You can set a flag that tells the transparent layer to detect. The transparent layer can determine its location, for example, by querying the operating system when a second input is detected.

FIG. 25 shows an example of a user dragging an image portion to a location within a collaborative workspace. As shown in interface 2501, the user has used the circular cropping tool of the cropping tool interface 2503 to drag image portion 2305 of image 2302 to a new location within the co-workspace 2502. Interface 2501 is an interface after the user drags the image portion, but before the user selects the final position of the image portion (ie, by completing the "drop" portion of the "drag and drop"). 2501 is shown. Therefore, FIG. 25 shows the interface 2501 before the reception of the second input but after the first input. If the user releases the image portion by completing the drop input, the current location corresponding to the image portion 2305 will be detected by the transparent layer (either directly or via a cropping tool). This location can be represented by a point that corresponds to the location of the pointer at the moment the image was released (ie, dropped).

Returning to FIG. 20, in step 2006, the transparent layer running on the local computing device sends multiple commands to the collaborative workspace hosted on the server. Multiple commands remove the image (ie, the rest of the original image) from the collaborative workspace and further collaborate on the image portion based at least in part based on the location of the second input detected in step 2005. It is configured to be inserted into the workspace. Since the collaborative workspace communicates with local computing devices through websocket connections, the transparent layer can send multiple commands through websocket connections.

FIG. 26 is a flowchart for sending a plurality of commands to a collaborative workspace hosted on a server according to an exemplary embodiment. The step shown in FIG. 26 is performed by the transparent layer after detecting the second input.

In step 2601, a first command is sent from the transparent layer to the server hosting the co-workspace, and the first command is configured to remove the image from the co-workspace. The specific instructions used to remove the image may depend on the configuration and parameters of the collaborative workspace. The first command can, for example, refer to an object in the collaborative workspace that corresponds to the image and instruct the collaborative workspace to delete that object. If the collaborative workspace does not remember the image as an object, the transparent layer performs image boundary detection (based on the first input) from the screen capture to determine the coordinates that define the image boundaries. You can decide and send commands to the collaborative workspace to replace the area defined by the boundary (as identified by the coordinates) with whitespace. Many variants are possible and these examples are not limited.

In step 2602, the transparent layer determines the context associated with the location of the second input. As mentioned above, the collaborative workspace can include various toolbars and / or interface sections. Thus, the context is, for example, the editing interface of the workspace used to enter information that is displayed to all users, or accessible to all users (and optionally to each user). Can be a locally customized toolbar interface.

In step 2603, the second command is sent from the transparent layer to the server hosting the collaborative workspace, and the second command is based on the context associated with the location of the second input. The second command may depend on the context, such as whether the location is within the editing area of the collaborative workspace, or whether the location is the location associated with the collaborative workspace toolbar.

27A, 27B are flowcharts for sending multiple commands to a collaborative workspace hosted on a server for two different contexts, according to an exemplary embodiment.

The flowchart of FIG. 27A corresponds to the context of the editing interface when the location of the second input corresponds to the location associated with the editing interface (such as the whiteboard interface 2502 shown in FIG. 25). In step 2701, the first command is sent from the transparent layer to the server hosting the co-workspace, and the first command is configured to remove the image from the co-workspace. In step 2702, a second command is sent from the transparent layer to the server hosting the co-workspace, and the second command is configured to insert an image portion into the co-workspace at the location. When the image portion is stored in memory such as random access memory, the transparent layer can retrieve the image portion from memory and send the image portion to the server for insertion at the location. Alternatively, the transparent layer can send the memory location of the image portion to the server as part of a second command, which uses the memory location to send the image from the local computing device. You can get the part.

The flowchart in FIG. 27B corresponds to the context of the toolbar interface when the location of the second input corresponds to the location associated with the editing interface. In step 2703, a first command is sent from the transparent layer to the server hosting the co-workspace, and the first command is configured to remove the image from the co-workspace. In step 2704, a second command is sent from the transparent layer to the server hosting the collaborative workspace, and the second command is configured to store the image portion in the toolbar associated with the toolbar interface. When the toolbar is customized for each user connected to the collaborative workspace, the local version of the toolbar on the local computing device can remember the image portion. In this case, the server can instruct the local computing device to store the image portion locally in local memory. Alternatively, if the toolbar is not customized for each user and is uniform across all users in the collaborative workspace, the server can store the image portion in the server's memory and within the collaborative workspace. All users of the image can access the image portion via a web socket connection using the toolbar interface.

28A-28C show an example of the result of a user dragging an image portion to the editing interface of a collaborative workspace and dragging the image portion to the editing interface, according to an exemplary embodiment.

As shown in FIG. 28A, the user has selected circular cropping 2803 for image 2802 in the editing interface 2801 of the collaborative workspace. Cropping tools are not shown in this figure, but it should be understood that the user interface will include cropping tools. Further, in this example, the collaborative workspace is shown to include only the editing interface 2801, but as mentioned above, the collaborative workspace can include additional interfaces and toolbars.

FIG. 28B shows an editing interface 2801 after the user drags the image portion 2803 to a location within the editing interface 2801 of the collaborative workspace, but before the user releases (drops the image portion). The dashed arrow between the remaining image 2802 and the trash can icon 2804 indicates that when the user completes the drag and drop (ie, when a second input is detected), the rest, as described above. Image 2802 shows that it will be removed from the collaborative workspace. The trash can icon 2804 is presented in the figure to show the result of drag and drop of image portion 2803 by the user rather than as part of the user interface or collaborative workspace.

FIG. 28C shows the collaborative workspace editing interface 2801 after the user has completed the drag-and-drop movement (ie, after the second input has been detected). As shown, the image portion 2803 has been inserted into the collaborative workspace editing interface 2801 at the drop location, and the rest of the original image (2802 in FIG. 28B) has been removed from the collaborative workspace editing interface 2801.

It is important to note that changes to the collaborative workspace editing interface shown in Figure 28C are propagated across all instances of the collaborative workspace for all users connected to the collaborative workspace. In other words, changes to the collaborative workspace occur not only on the local device or the local instance of the collaborative workspace application, but also on the instance of the collaborative workspace on the remote device being used by the remote user. As mentioned earlier, this is because the transparent layer sends a command to remove the original image and insert the image part at the location through a websocket connection to the server hosting the collaborative workspace, and then the server , Because these changes are communicated to all connected devices through websocket connections.

29A-29E are diagrams showing an example of the result of a user dragging an image portion onto the toolbar interface of a collaborative workspace and dragging the image portion onto the toolbar interface, according to an exemplary embodiment.

As shown in FIG. 29A, the user has selected circular cropping 2903 of image 2902 in the editing interface 2901 of the collaborative workspace. Cropping tools are not shown in this figure, but it should be understood that the user interface will include cropping tools. In addition to the editing interface 2901, the collaborative workspace includes a toolbar interface 2904. Toolbar interface 2904 can include various tools accessible to the user to edit or mark up the content within editing interface 2904. As described below, the toolbar interface can also include the functionality of storing images for reuse.

FIG. 29B shows a collaborative workspace after the user drags the image portion 2903 to the location associated with the toolbar interface 2904 of the collaborative workspace, but before the user releases (drops the image portion). The dashed arrow between the remaining image 2902 and the trash can icon 2906 indicates that when the user completes the drag and drop (ie, when a second input is detected), the rest, as described above. Image 2902 shows that it will be removed from the collaborative workspace. The trash can icon 2904 is presented in the figure to show the result of drag and drop of image portion 2903 by the user rather than as part of the user interface or collaborative workspace.

FIG. 29B further shows a new icon 2905 that may appear within the toolbar interface 2904 when the user drags the image portion 2903 through the toolbar interface 2904. Icon 2905 (indicated as "S") can indicate to the user that the image portion is dropped onto the toolbar interface and the image is saved on the toolbar for subsequent reuse by the user. ..

FIG. 29C shows a collaborative workspace after the user has completed the drag-and-drop movement (ie, after the second input has been detected). As shown, neither the image portion (2903 in FIG. 29B) nor the remaining image (2902 in FIG. 29B) appears in the communal workspace. This was associated with toolbar interface 2904, where the transparent layer sent a first command to the co-application on the server to remove the original image from the co-workspace to the server hosting the co-workspace. This is the result of sending a second command to the collaborative application hosting the collaborative workspace to instruct the collaborative application to store the image portion in the toolbar. Toolbar interface 2904 indicates this by the presence of icon 2905, which alerts the user that the stored image is stored on the toolbar and is accessible through toolbar interface 2904.

Changes to the collaborative workspace shown in FIG. 29C can be communicated across all instances of the collaborative workspace for all users connected to the collaborative workspace. In other words, changes to the collaborative workspace can occur on all instances of the collaborative workspace for all users. As mentioned earlier, this is because the transparent layer sends a command to remove the original image and insert the image part at the location through a websocket connection to the server hosting the collaborative workspace, and then the server , Because these changes are communicated to all connected devices through websocket connections. In this case, the stored image is then stored or hosted on a server, and by all users connected to the collaborative workspace, each of the toolbar interfaces (image parts can be retrieved through a web socket connection). Can be accessed through the instance. Alternatively, the server instructs the local computing device to store the image portion locally in local memory so that only the local computing device can access the image portion. Can be done.

FIG. 29D shows the use of the toolbar interface 2904 to obtain a previously stored image portion. As shown in FIG. 29D, the stored image interface 2907 is presented to the user by the user selecting the icon 2905 in the toolbar interface. The storage image interface 2907 can display a representation of all previously stored images or image portions. For example, representation 2908 corresponds to image portion 2903 of FIG. 29B.

FIG. 29E shows the use of a toolbar interface for inserting previously saved images. As shown in FIG. 29E, the image portion 2903 is reinserted into the editing interface 2901 of the collaborative workspace by the user dragging the representation 2908 from the storage image interface onto the editing interface 2901.

One or more of the techniques described above may be implemented in one or more computer systems or may involve one or more computer systems. FIG. 30 shows an example of the dedicated computing environment 3000. Computing Environment 3000 does not imply any limitation on the use or scope of functionality of the described (s) examples.

Referring to FIG. 30, the computing environment 3000 includes at least one processing unit 3010 and memory 3020. The processing unit 3010 executes computer-executable instructions and can be a real processor or a virtual processor. In a multiplex processing system, a plurality of processing units execute computer-executable instructions in order to increase the processing capacity. The memory 3020 can be a volatile memory (eg, register, cache, RAM), a non-volatile memory (eg, ROM, EEPROM, flash memory, etc.), or any combination thereof. Memory 3020 can store software 3080 that implements the techniques described.

The computing environment can have additional features. For example, the computing environment 3000 includes storage 3040, one or more input devices 3050, one or more output devices 3060, and one or more communication connections 3090. An interconnect mechanism 3070, such as a bus, controller, or network, interconnects the components of the computing environment 3200. In general, operating system software or firmware (not shown) provides an operating environment for other software running in computing environment 3000, coordinating the activities of the components of computing environment 3000.

Storage 3040 can be removable or non-removable and can be used to store magnetic disks, magnetic tapes or cassettes, CD-ROMs, CD-RWs, DVDs, or information and can be accessed within the computing environment 3000. Includes any other medium. Storage 3040 can store instructions for software 3080.

The (one or more) input device 3050 is a touch input device such as a keyboard, mouse, pen, trackball, touch screen, or game controller, voice input device, scanning device, digital camera, remote control device, etc. Alternatively, it may be another device that provides input to the computing environment 3000. The output device (s) 3060 can be a display, television, monitor, printer, speaker, or another device that provides output from the computing environment 3000.

The communication connection (s) 3090 allows communication to another computing entity through the communication medium. The communication medium transmits information such as computer executable instructions, audio or video information, or other data in the modulated data signal. A modulated data signal is a signal in which one or more of the characteristics of the signal are set or modified in such a way that the information in the signal is encoded. By way of example, but not by limitation, communication media include wired or wireless techniques implemented using electrical, optical, RF, infrared, acoustic, or other carriers.

Implementations can be described in the context of computer readable media. A computer-readable medium is any available medium that can be accessed within a computing environment. As an example, but not limited to, within a computing environment 3000, a computer-readable medium includes memory 3020, storage 3040, a communication medium, and any combination of the above.

Of course, FIG. 30 shows the computing environment 3000, the display device 3060, and the input device 3050 as separate devices for easy identification. The computing environment 3000, display device 3060, and input device 3050 can be separate devices (eg, personal computers connected to monitors and mice by wires) and single devices (eg, smartphones or tablets, etc.). Attached to any combination of devices (eg, a mobile device with a touch display), or a computing device operably coupled to a touchscreen display device, a single display device and an input device. Can be integrated into multiple computing devices, etc.). A computing environment 3000 is a set-top box, personal computer, or one or more servers, such as a farm of networked servers, a clustered server environment, or a cloud network of computing devices. obtain.

Although the principles of the invention have been described and illustrated with respect to the embodiments described, it will be appreciated that the embodiments described may be modified in configuration and detail without departing from such principles. The elements of the illustrated examples shown in software can be implemented in hardware and vice versa.

In view of the many possible embodiments to which the principles of the invention may apply, the invention claims all embodiments that may fall within the scope and intent of the following claims and their equivalents.

Claims (21)

A way to convey cropped images through websocket connections in a networked collaborative workspace.
On the user interface of a local computing device, the step of sending a representation of a collaborative workspace hosted on a server and accessible to multiple participants on multiple computing devices through a websocket connection. A step and a step in which the collaborative workspace contains one or more images.
A step of detecting a user input for selecting an image portion of an image in the one or more images by a cropping tool running on the local computing device, wherein the user input is at multiple coordinates. Corresponding steps and
A step of transmitting the plurality of coordinates to a transparent layer running on the local computing device by the cropping tool running on the local computing device, wherein the transparent layer is operating. A step and a step comprising an application programming interface (API) configured to interface with one or more of one or more applications configured to run on the system or said operating system.
A step of capturing the image portion by the transparent layer running on the local computing device, at least partially based on the plurality of coordinates.
A step of detecting a second user input for dragging the selected image portion to a location in the collaborative workspace by the transparent layer running on the local computing device.
A step of sending a plurality of commands to the server hosting the collaborative workspace by the transparent layer running on the local computing device, wherein the plurality of commands transmit the image to the collaborative workspace. A method comprising a step, which is configured to be removed from, and further to insert the image portion into the collaborative workspace, at least in part based on the location. The step of capturing the image portion by the transparent layer running on the local computing device, at least partially based on the plurality of coordinates.
Defined by the plurality of coordinates with respect to one or more of the operating system or the application running on the local computing device by the transparent layer running on the local computing device. The step of sending a request for a screen capture of the image portion,
The step of receiving the screen capture of the image portion defined by the plurality of coordinates by the transparent layer running on the local computing device.
The method of claim 1, comprising the step of storing the image portion in the random access memory of the local computing device by the transparent layer running on the local computing device. The method of claim 1, wherein the plurality of commands are transmitted through the web socket connection. The step of detecting a second user input for dragging the selected image portion to a new position in the collaborative workspace by the transparent layer running on the local computing device.
The step of detecting the second user input for dragging the selected image portion to a new position in the collaborative workspace by the cropping tool running on the local computing device.
The method of claim 1, comprising the step of transmitting the new location to the transparent layer running on the local computing device by the cropping tool running on the local computing device. .. The location corresponds to the location associated with the editing interface.
The step of sending a plurality of commands to the server hosting the collaborative workspace by the transparent layer running on the local computing device.
A step of transmitting a first command from the transparent layer to the server hosting the collaborative workspace, wherein the first command is configured to remove the image from the collaborative workspace. When,
A step of transmitting a second command from the transparent layer to the server hosting the collaborative workspace, wherein the second command is configured to insert the image portion into the editing workspace at the location. The method of claim 1, comprising: The method of claim 5, wherein the image portion is stored in random access memory by the transparent layer and the second command comprises a storage location for the image portion in the random access memory. The location corresponds to the location associated with the toolbar interface.
The step of sending a plurality of commands to the server hosting the collaborative workspace by the transparent layer running on the local computing device.
A step of transmitting a first command from the transparent layer to the server hosting the collaborative workspace, wherein the first command is configured to remove the image from the collaborative workspace. When,
A step of transmitting a second command from the transparent layer to the server hosting the collaborative workspace, such that the second command causes the toolbar associated with the toolbar interface to store the image portion. The method of claim 1, wherein the method comprises: A local computing device for delivering cropped images over websocket connections in a networked collaborative workspace.
With one or more processors
At least one of the one or more processors when operably coupled to at least one of the one or more processors and executed by at least one of the one or more processors. For one,
On the user interface of the local computing device, it sends a representation of a collaborative workspace hosted on a server and accessible to multiple participants on multiple computing devices through a websocket connection. That is, to transmit, where the collaborative workspace contains one or more images.
A cropping tool running on the local computing device detects a user input that selects an image portion of an image in the one or more images, the user input being in multiple coordinates. Corresponding, detecting and
By transmitting the plurality of coordinates to the transparent layer running on the local computing device by the cropping tool running on the local computing device, the transparent layer is operating. Sending, including an application programming interface (API) configured to interface with one or more of one or more applications configured to run on the system or said operating system. When,
Capturing the image portion, at least partially based on the plurality of coordinates, by the transparent layer running on the local computing device.
The transparent layer running on the local computing device detects a second user input for dragging the selected image portion to a location in the collaborative workspace.
The transparent layer running on the local computing device sends a plurality of commands to the server hosting the collaborative workspace, wherein the plurality of commands transfer the image to the collaborative workspace. One or more memorized commands to be removed from, and to transmit, configured to insert the image portion into the collaborative workspace, at least in part based on the location. A local computing device with memory. When executed by at least one of the one or more processors, the transparent layer running on the local computing device to at least one of the one or more processors. The instruction that causes the image portion to be captured based on at least a portion of the plurality of coordinates causes at least one of the one or more processors to capture the image portion.
Defined by the plurality of coordinates with respect to one or more of the operating system or the application running on the local computing device by the transparent layer running on the local computing device. Sending a request for a screen capture of the image portion and
Receiving the screen capture of the image portion defined by the plurality of coordinates by the transparent layer running on the local computing device.
The local according to claim 8, wherein the transparent layer running on the local computing device further causes the random access memory of the local computing device to store the image portion. -Computing device. The local computing device according to claim 8, wherein the plurality of commands are transmitted through the web socket connection. When executed by at least one of the one or more processors, the transparent layer running on the local computing device to at least one of the one or more processors. The instruction that causes the detection of a second user input for dragging the selected image portion to a new position in the collaborative workspace is given to at least one of the one or more processors. ,
The cropping tool running on the local computing device detects the second user input for dragging the selected image portion to a new position in the collaborative workspace.
8. The cropping tool running on the local computing device further causes the new location to be transmitted to the transparent layer running on the local computing device. Local computing device. The location corresponds to the location associated with the editing interface.
When executed by at least one of the one or more processors, the transparent layer running on the local computing device to at least one of the one or more processors. The instruction, which causes the server hosting the collaborative workspace to send a plurality of commands, causes at least one of the one or a plurality of processors to send a plurality of commands.
Sending a first command from the transparent layer to the server hosting the collaborative workspace, wherein the first command is configured to remove the image from the collaborative workspace. To do and
The transparent layer is to send a second command from the transparent layer to the server hosting the collaborative workspace, the second command being configured to insert the image portion into the editing workspace at the location. The local computing device according to claim 8, which further causes the transmission to be performed. 12. The local according to claim 12, wherein the image portion is stored in a random access memory by the transparent layer and the second command includes a storage location of the image portion in the random access memory. Computing device. The location corresponds to the location associated with the toolbar interface.
When executed by at least one of the one or more processors, the transparent layer running on the local computing device to at least one of the one or more processors. The instruction, which causes the server hosting the collaborative workspace to send a plurality of commands, causes at least one of the one or a plurality of processors to send a plurality of commands.
Sending a first command from the transparent layer to the server hosting the collaborative workspace, wherein the first command is configured to remove the image from the collaborative workspace. To do and
Sending a second command from the transparent layer to the server hosting the collaborative workspace so that the second command causes the toolbar associated with the toolbar interface to store the image portion. The local computing device according to claim 8, wherein the local computing device is configured to further perform transmission and transmission. When executed by a local computing device, the local computing device,
On the user interface of the local computing device, it sends a representation of a collaborative workspace hosted on a server and accessible to multiple participants on multiple computing devices through a websocket connection. That is, to transmit, where the collaborative workspace contains one or more images.
A cropping tool running on the local computing device detects a user input that selects an image portion of an image in the one or more images, the user input being in multiple coordinates. Corresponding, detecting and
By transmitting the plurality of coordinates to the transparent layer running on the local computing device by the cropping tool running on the local computing device, the transparent layer is operating. Sending, including an application programming interface (API) configured to interface with one or more of one or more applications configured to run on the system or said operating system. When,
Capturing the image portion, at least partially based on the plurality of coordinates, by the transparent layer running on the local computing device.
The transparent layer running on the local computing device detects a second user input for dragging the selected image portion to a location in the collaborative workspace.
The transparent layer running on the local computing device sends a plurality of commands to the server hosting the collaborative workspace, wherein the plurality of commands transfer the image to the collaborative workspace. Stores at least one computer-readable command that is configured to insert the image portion into the collaborative workspace, and to perform transmission, based on the location, at least in part, from the location. Non-temporary computer readable medium. When executed by the local computing device, the transparent layer running on the local computing device to the local computing device, at least partially based on the coordinates. The instruction that causes the image portion to be captured gives the local computing device.
Defined by the plurality of coordinates with respect to one or more of the operating system or the application running on the local computing device by the transparent layer running on the local computing device. Sending a request for a screen capture of the image portion and
Receiving the screen capture of the image portion defined by the plurality of coordinates by the transparent layer running on the local computing device.
At least according to claim 15, the transparent layer running on the local computing device further causes the random access memory of the local computing device to store the image portion. One non-temporary computer readable medium. The at least one non-transitory computer-readable medium according to claim 15, wherein the plurality of commands are transmitted through the web socket connection. When executed by the local computing device, the selected image portion is brought to the local computing device by the transparent layer running on the local computing device in the collaborative workspace. The instruction that causes the local computing device to detect a second user input for dragging to a new position in
The cropping tool running on the local computing device detects the second user input for dragging the selected image portion to a new position in the collaborative workspace.
15. The cropping tool, which runs on the local computing device, further causes the new location to be transmitted to the transparent layer, which runs on the local computing device. At least one non-temporary computer-readable medium. The location corresponds to the location associated with the editing interface.
Multiple to the server hosting the collaborative workspace by the transparent layer running on the local computing device to the local computing device when executed by the local computing device. An instruction that causes the local computing device to send a command
Sending a first command from the transparent layer to the server hosting the collaborative workspace, wherein the first command is configured to remove the image from the collaborative workspace. To do and
The transparent layer is to send a second command from the transparent layer to the server hosting the collaborative workspace, the second command being configured to insert the image portion into the editing workspace at the location. At least one non-transitory computer-readable medium according to claim 15, which further causes the transmission to be performed. At least one of claim 19, wherein the image portion is stored in random access memory by the transparent layer and the second command includes a storage location for the image portion in the random access memory. Two non-temporary computer-readable media. The location corresponds to the location associated with the toolbar interface.
Multiple to the server hosting the collaborative workspace by the transparent layer running on the local computing device to the local computing device when executed by the local computing device. The command, which causes the command to be sent, causes the local computing device to send the command.
Sending a first command from the transparent layer to the server hosting the collaborative workspace, wherein the first command is configured to remove the image from the collaborative workspace. To do and
Sending a second command from the transparent layer to the server hosting the collaborative workspace, such that the second command causes the toolbar associated with the toolbar interface to store the image portion. The at least one non-transitory computer-readable medium according to claim 15, which further comprises transmitting and performing.

Images