RU2733488C1 - Method and system for displaying a cursor when broadcast a video stream - Google Patents

Abstract

FIELD: computer equipment.

SUBSTANCE: method of displaying a cursor when broadcast a video stream comprises steps of: initiating transmission of a video stream from a server part to a user device; capturing a cursor image in a first OS; transmitting a cursor image and a video stream to a second OS on a user device, wherein the cursor image is transmitted separately from the video stream; obtained video stream and cursor image are displayed in second OS on user device; recording user input generating cursor position change and transmitting coordinates of changed cursor position to server part; generating a cursor image in the first OS in accordance with the changed coordinates and sending the cursor image to the second OS; receiving, on user device, corresponding frame of video stream and image of cursor with changed position; displaying a cursor image in a second OS in a broadcast video stream.

EFFECT: reduced delay and high accuracy of displaying cursor position during video stream broadcast due to reduced time between change of input device position and updating cursor position on user display device.

6 cl, 4 dwg

Description

FIELD OF TECHNOLOGY

[0001] The present technical solution relates to the field of computer technology, in particular, to a method and system for displaying a cursor when broadcasting a video stream over a data network.

LEVEL OF TECHNOLOGY

[0002] Streaming services are now increasingly evolving in the gaming industry and offer end users the ability to perform gameplay via cloud-based video streaming to client devices, while directly performing hardware processing on the server. The main technical problem of this technology is a high requirement for network bandwidth and network connection stability, which is especially important for the implementation of the gameplay in dynamic games with maintaining a high frame rate, and ensuring smooth control with minimal delays on the part of the user.

[0003] A method of synchronizing a pointer on a user's local machine when making remote presentations via a data network is known from the prior art (US 9292108 B2, 03.22.2016). In the known solution, the cursor image is captured on the server and sent to the client along with the background image, which results in a network delay between mouse movement and cursor display. The total delay consists of two components: the delay between sending the mouse coordinates from the client and drawing the cursor on the server; delay between rendering the cursor on the server and displaying the image with the new cursor position on the client.

[0004] A significant drawback is a large delay when applying this solution for broadcasting a video stream, in which there is a fast frame change and dynamic movement of the pointer, which leads to a loss of smoothness of movement of the cursor and inaccurate reflection of its position during broadcast and image acquisition on the client device.

SUMMARY OF THE INVENTION

[0005] To solve the existing technical problem in terms of reducing the delay in cursor movement when broadcasting a video stream online, the claimed invention is proposed that implements a new principle of tracking and drawing a cursor.

[0006] The technical result is to reduce the delay and improve the display accuracy of the cursor position when broadcasting a video stream, by reducing the time between changing the position of the input device and updating the cursor position on the user's display device.

[0007] The claimed result is achieved due to a computer-implemented method for displaying a cursor when broadcasting a video stream, performed using a processor and containing the steps at which:

- initiate the broadcast of the video stream from the server part to the user device, while the server part generates the video stream in the first operating system (OS);

- fix the cursor image in the first OS;

- transmit the cursor image and the video stream to the second OS to the user's device, and the cursor image is transmitted separately from the video stream;

- display in the second OS on the user's device the received video stream and the cursor image, the cursor image being superimposed on top of the video stream;

- fix the user input generating a change in the cursor position in the second OS when broadcasting a video stream frame on the user's device, and transmit the coordinates of the changed cursor position to the server part;

- accept on the server side the coordinates of the changed position of the cursor in the video stream on the user's device and generate an event of changing the position of the cursor for the first OS;

- form a cursor image in the first OS in accordance with the coordinates of the changed cursor position and transmit the cursor image to the second OS to the user's device;

- receive on the user's device the corresponding frame of the video stream from the server part and the image of the cursor with a changed position in accordance with user input;

- display the cursor image in the second OS in the broadcast video stream.

[0008] In one of the particular embodiments of the method, the cursor image in the second OS corresponds to the movement of the input device connected to the user device.

[0009] In another particular embodiment of the method, the input device is selected from the group: mouse manipulator, joystick, stylus, touch display.

[0010] In another particular embodiment of the method, the user's device is selected from the group: personal computer, laptop, smartphone, tablet, game console.

[0011] In another particular embodiment of the method, on the user's device, the change in the cursor position is fixed for the second OS, regardless of the received video stream.

[0012] The claimed invention is also implemented through a system for displaying a cursor when broadcasting a video stream, comprising a server and a user device connected by a data transmission network, in which:

the server is configured

initiating the broadcast of the video stream to the user's device, while the server part generates the video stream in the first operating system (OS);

fixing the cursor image in the first OS;

transmitting the cursor image and the video stream to a second OS on the user's device, the cursor image being transmitted separately from the video stream;

obtaining coordinates of the changed position of the cursor in the video stream in the second OS on the user's device;

generating an event of changing the position of the cursor for the first OS based on the obtained coordinates;

forming an image of the cursor in the first OS in accordance with the mentioned event of changing its position;

transferring the cursor image to the second OS to the user's device;

the user's device is configured

receiving a video stream from the first OS;

displaying the cursor in the second OS over the received video stream;

fixing user input generating a change in the cursor position in the second OS when broadcasting a video stream frame on the user's device;

transferring the coordinates of the changed position of the cursor to the server;

obtaining an image of the cursor from the second OS with a changed position in accordance with user input captured for the first OS;

superimposing the obtained image with the changed position of the cursor on the corresponding frame of the video stream;

displaying the cursor image in the second OS in the broadcast video stream.

BRIEF DESCRIPTION OF DRAWINGS

[0013] FIG. 1 illustrates a general diagram of the interaction of computing devices.

[0014] FIG. 2 illustrates flowcharts of the claimed method.

[0015] FIG. 3 illustrates an example of forming a cursor in a sequence of frames.

[0016] FIG. 4 illustrates an example of a computing device suitable for implementing the claimed technical solution.

CARRYING OUT THE INVENTION

[0017] As shown in FIG. 1, the claimed solution is a computing system (100) including a server (110) and a user device (120), interconnected by a data transmission network (150), for example, using the Internet.

[0018] The server (110) and the user device (120) implement the predetermined functionality using the corresponding OS (111, 121) and software applications installed on the devices (110, 120). The server (110), using the OS (111), generates, upon request of the user device (110), a video stream for its broadcast.

[0019] The technology for transmitting the broadcast video stream is based on its initial formation on the server (110) in its OS (111) and its subsequent capture and frame-by-frame broadcast in the OS (121) of the user device (120). The game stream, in particular, can be a computer game, which is controlled on the user's device (120) using appropriate data input means, for example, a mouse, joystick, touch screen, etc.

[0020] To implement the claimed solution, it is important to separate the broadcast video stream and the cursor image when it moves on the user's side, since the main change in the game process flow is processed on the server (110) in the OS (111), and therefore it is necessary to form a separate image of the pointer to based on the coordinates of the change in the trajectory of its movement on the user's device (120) in the OS (121).

[0021] A wide class of devices can be used as the user device (120), for example, a personal computer, laptop, smartphone, tablet, game console, and the like. The organization of a data transmission network (150) is also a standardized and widely known solution in the prior art, using wired and / or wireless communication technologies, for example, LAN, WAN, PAN, WLAN, GSM / 3G / LTE / 5G, etc.

[0022] FIG. 2 shows a block diagram of the claimed method (200) for forming a cursor. At the first stage (201), the user, using the client application in the OS (121) of the device (120), to start receiving the video stream, initiates a request for its broadcast from the server (110). The request can be made using a software application (client application) in the OS (121).

[0023] When the broadcast is initiated, the video stream is initially displayed in the OS (111) of the server (110), after which it is captured and transmitted via the network (150) to the OS (121) to the user device (120).

[0024] At step (202) in the OS (111) on the server (110), the function of tracking the change in the position of the cursor in the OS (111) is activated based on the manipulations performed by the user (120), which will be discussed in more detail below.

[0025] Initially, the cursor position is captured in the OS (111) at step (203), after which the cursor image and the video stream are transmitted to the user device (120) separately (step 204). The resulting video stream is displayed in the OS (121) of the user device (120) (step 205).

[0026] A cursor image is generated and displayed in the OS (121) on the device (120) over the video stream (step 206). When changing the position of the cursor in the OS (121) by moving the input device of the user device (120), the OS (121) fixes the change in the pixel coordinates X / Y of the cursor at step (207). The formation of an image of the cursor in the OS (121) on the user's device (120) is carried out by the software independently of the incoming video stream.

[0027] At step (208), after changing the position of the cursor in the OS (121), data is transmitted containing the coordinates of the cursor to which it was moved to the server (110). On the server (110), the coordinates of the cursor are received and fixed in the OS (111) for its subsequent rendering in the corresponding display position.

[0028] Following the formation of the cursor image in the OS (111) on the server (110) in accordance with the coordinates of the changed cursor position received from the client OS (121), the cursor image is transmitted to the user's device (120) to be drawn over the broadcast video stream in OS (121).

[0029] The process of tracking the change in the position of the cursor is iteratively repeated when receiving a video stream in the online mode and performing manipulations using the input means on the user's device (120), leading to the movement of the cursor.

[0030] FIG. 3 shows an example of forming a cursor image in a set of frames of a video stream. At the start of the video stream transmission, the initially transmitted frame (301) from the server (110) is transmitted to the user device (120) without the cursor image (31). Initially, the cursor image (31) is fixed at a predetermined location of the frame (301) upon subsequent transmission to the client application. In this example, the frame displays the graphical user interface of the computer game, containing the controlled character (310) and the position of the cursor image (31).

[0031] When changing the position of the cursor in one of the following frames (302) from position (31) in (32) OS (121) of the client application, the coordinates (X / Y) are fixed on the pixel grid throughout the entire trajectory of its movement and their dynamic sending to the server (software). FIG. 3 shows the step of transferring the coordinates of the end position (32) of the cursor as an example, but this is not a limiting example of the implementation of the claimed solution. The obtained coordinates of the position (32) are transmitted to the server (110) for their fixation in the OS (111), in which the frames of the video stream are captured and the subsequent formation of the cursor image itself (33) in the position corresponding to the coordinates of the changed position (32) of the OS (121) ... The cursor image (33) is formed for the next frame (303) and is transmitted separately from the video stream to the user device (120), where it is subsequently superimposed on the next frame (304).

[0032] FIG. 4 illustrates a general example of a computing computing device (400) that may be used to implement computing devices included in the present invention. The device (400) can be used as a basis for a user device (120) and a server (110). The components of the device (400), as a rule, are interfaced through a common data bus.

[0033] In general, the device (400) contains components such as: one or more processors (401), at least one random access memory (402), persistent data storage (403), input / output interfaces (404), I / O means (405), networking means (406).

[0034] The processor (401) of the device performs the basic computational operations necessary for the operation of the device (400) or the functionality of one or more of its components. The processor (401) executes the necessary machine-readable instructions contained in the main memory (402).

[0035] Memory (402), as a rule, is made in the form of RAM and contains the necessary program logic to provide the required functionality. The data storage medium (403) can be performed in the form of HDD, SSD disks, raid array, network storage, flash memory, optical information storage devices (CD, DVD, MD, Blue-Ray disks), etc. The tool (403) allows for long-term storage of various types of information, for example, the history of processing requests (logs), user IDs, sound files, etc.

[0036] Interfaces (404) are standard means for connecting and operating various types of devices (400), for example, USB, RS232, RJ45, LPT, COM, HDMI, PS / 2, Lightning, Fire Wire, etc. interfaces (404) depends on the specific implementation of the device (400), which can be a personal computer, mainframe, server cluster, thin client, smartphone, laptop, etc.

[0037] As I / O data means (405), a keyboard, joystick, display (touchscreen display), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc. can be used.

[0038] Networking means (406) are selected from a device that provides network reception and transmission of data, for example, Ethernet card, WLAN / Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. .P. The tool (406) provides the organization of data exchange via a wired or wireless data transmission channel, for example, WAN, PAN, LAN, Intranet, Internet, WLAN, WMAN or GSM.

[0039] In the present application materials, the preferred disclosure of the implementation of the claimed technical solution was presented, which should not be used as limiting other, particular embodiments of its implementation, which do not go beyond the claimed scope of legal protection and are obvious to specialists in the relevant field of technology.

Claims (31)

1. A computer-implemented method of displaying a cursor when broadcasting a video stream, performed using a processor and containing stages at which: initiate the broadcast of the video stream from the server side to the user device, while the server side generates the video stream in the first operating system (OS); fix the cursor image in the first OS; transmitting the cursor image and the video stream to the second OS to the user device, the cursor image being transmitted separately from the video stream; displaying the received video stream and the cursor image in the second OS on the user's device, the cursor image being superimposed over the video stream; fixing user input generating a change in the position of the cursor in the second OS when broadcasting a frame of the video stream on the user's device and transmitting the coordinates of the changed position of the cursor to the server part; take on the server side coordinates of the changed position of the cursor in the video stream on the user's device and generate an event of changing the position of the cursor for the first OS; forming a cursor image in the first OS in accordance with the coordinates of the changed cursor position and transmitting the cursor image to the second OS to the user device; receive on the user's device the corresponding frame of the video stream from the server part and the image of the cursor with a changed position in accordance with the user input; display the cursor image in the second OS in the broadcast video stream. 2. The method according to claim 1, characterized in that the cursor image in the second OS corresponds to the movement of the input device connected to the user device. 3. The method according to claim 2, characterized in that the input device is selected from the group: manipulator, mouse, joystick, stylus, touch display. 4. The method according to claim 1, characterized in that the user's device is selected from the group: personal computer, laptop, smartphone, tablet, game console. 5. The method according to claim 2, characterized in that, on the user's device, the change in the cursor position is fixed for the second OS, regardless of the received video stream. 6. A system for displaying a cursor when broadcasting a video stream, containing a server and a user device connected by a data transmission network, in which: the server is configured initiating the broadcast of the video stream to the user's device, while the server part generates the video stream in the first operating system (OS); fixing the cursor image in the first OS; transmitting the cursor image and the video stream to a second OS on the user's device, the cursor image being transmitted separately from the video stream; obtaining coordinates of the changed position of the cursor in the video stream in the second OS on the user's device; generating an event of changing the position of the cursor for the first OS based on the obtained coordinates; forming an image of the cursor in the first OS in accordance with the mentioned event of changing its position; transferring the cursor image to the second OS to the user's device; the user's device is configured receiving a video stream from the first OS; displaying the cursor in the second OS over the received video stream; fixing user input generating a change in the cursor position in the second OS when broadcasting a video stream frame on the user's device; transferring coordinates of the changed cursor position to the server; obtaining an image of the cursor from the second OS with a changed position in accordance with user input captured for the first OS; superimposing the obtained image with the changed position of the cursor on the corresponding frame of the video stream; displaying the cursor image in the second OS in the broadcast video stream.

Images