CN113849073A - Remote control-oriented mouse and returned picture synchronization method and system - Google Patents
Remote control-oriented mouse and returned picture synchronization method and system Download PDFInfo
- Publication number
- CN113849073A CN113849073A CN202110980595.1A CN202110980595A CN113849073A CN 113849073 A CN113849073 A CN 113849073A CN 202110980595 A CN202110980595 A CN 202110980595A CN 113849073 A CN113849073 A CN 113849073A
- Authority
- CN
- China
- Prior art keywords
- picture
- mouse cursor
- original
- mouse
- host
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000012545 processing Methods 0.000 claims abstract description 19
- 230000033001 locomotion Effects 0.000 claims abstract description 18
- 238000004590 computer program Methods 0.000 claims description 11
- 230000005012 migration Effects 0.000 claims description 4
- 238000013508 migration Methods 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 230000006835 compression Effects 0.000 description 3
- 238000007906 compression Methods 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000006073 displacement reaction Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/0354—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
- G06F3/03543—Mice or pucks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/04812—Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1407—General aspects irrespective of display type, e.g. determination of decimal point position, display with fixed or driving decimal point, suppression of non-significant zeros
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- User Interface Of Digital Computer (AREA)
- Controls And Circuits For Display Device (AREA)
Abstract
The invention discloses a synchronization method and a system for a remote control-oriented mouse and a return picture, which are used for decoding the return picture returned by a controlled terminal to obtain an original image of a host picture and acquiring the original coordinate of a mouse cursor from the original image of the host picture; intercepting and analyzing a control command of the mouse at a control end to obtain a target coordinate corresponding to a mouse cursor, and transferring a pixel point corresponding to the mouse cursor in a host frame from an original coordinate to the target coordinate; and (3) carrying out interpolation processing on pixel points of original coordinates of a mouse cursor in the host picture, displaying and outputting the processed host picture as a reference picture during control, and realizing the synchronization of the mouse and a returned picture. By adopting the method, the experience of eye-hand fit can be improved when a manager refers to the redrawn return picture for remote control, the movement of a mouse cursor on the real-time picture of the host computer at the controlled end cannot be influenced, and the original working condition of the host computer at the controlled end is not changed.
Description
Technical Field
The invention relates to the technical field of remote control and video transmission, in particular to a method and a system for synchronizing a remote control-oriented mouse and a returned picture.
Background
With the development of audio and video encoding and decoding technology and human-computer interaction technology, remote control technology is widely applied in the fields of distributed seat management, cooperative office, desktop sharing and the like. The current remote control technology based on IP mainly realizes that a manager can remotely control a remote computer at different geographical positions through keyboard and mouse signals and video signals of a host by respectively carrying out encoding transmission and remote decoding on the keyboard and mouse signals and the video signals of the host, thereby reducing the configuration of personnel on duty at station positions in a system and improving the real-time response of system scheduling.
Since the administrator needs to use the return picture of the remote host as a reference picture when performing remote control through the keyboard and mouse, the synchronization between the keyboard and mouse and the return picture is very important. Especially, the movement of the mouse can generate continuous change of the returned picture, whether the mouse cursor can reach the target point in time or not, and the action amplitude of the manager needs to be large, so that the experience of remote control is determined to a great extent.
Further, the real-time performance of the mouse cursor of the returned picture mainly depends on two aspects, namely the picture real-time performance of the mouse cursor of the remote controlled host and the returned picture real-time performance of the coded, transmitted and decoded controlled host in the local.
The data codec flow of the keyboard and mouse and video pictures is shown in fig. 1. The encoding and transmission of the keyboard and mouse signals in the local area network are generally about 5-20 ms, and the delay is basically ignored for an operator. The delay of high-definition video pictures is about 100ms when the pictures are coded, transmitted through a network and decoded at 60fps, and the delay of 4K pictures can even reach 200 ms. This causes the situation that the mouse cursor on the controlled host computer screen is actually in place and the mouse cursor seen on the return screen lags behind. When the delay difference between the return image and the real-time image of the controlled host is too large, the mouse cursor on the return image is easy to move to the target position without any desire when the user operates the remote host through the local mouse, and the phenomena of delay, delay, difficulty in alignment and the like exist. Therefore, how to solve the problem of synchronization between the return picture delay and the actual moving speed of the mouse cursor is a key problem for improving the remote control experience.
At present, no relevant patent specially aiming at improving the real-time performance of the mouse and the returned picture is found. From the technical point of view, to achieve the purpose, a method of reducing the delay of the returned picture is generally adopted to improve the synchronism of the returned picture and the mouse. At present, there are two main technical approaches for reducing the picture transmission delay:
1. the time delay is reduced from the aspect of video coding technology, such as the shallow compression technology which is mainstream at present, and the optical fiber video transmission technology. For example, "a display capable of remotely controlling a host", application No.: 201621043186.X, which uses coaxial fiber to modulate KVM interface signal for remote transmission, while the return picture of the controlled host has higher real-time, it does not perform video compression, and does not support distributed architecture based on IP exchange, belonging to traditional video signal extension processing method, and it can not realize simultaneous transmission of audio and video data and IP data; the flow rate of shallow compression is basically about 300M, and the system architecture and the application flexibility are greatly limited.
2. And the time delay of the returned picture is reduced by adopting a mode of improving the network communication efficiency of the video stream. In the patent ' a method and system for realizing zero copy transmission of streaming media data ' (application number: 2008101429.2) ' a streaming media decoding method, system and mobile terminal ' (application number: 201310122798.2) ', the field of a video message is identified, protocol uninstalled and other optimization processing is mainly carried out by optimizing a network protocol stack or sharing a memory and other modes, the method mainly reduces the times of stack entering and exiting and copying of the network message from the software perspective to improve the real-time performance of video transmission, but the video coding and decoding delay cannot be optimized, the network performance and the hardware platform have larger correlation, and the improvement capability is limited.
Disclosure of Invention
The invention provides a method and a system for synchronizing a remote control-oriented mouse and a returned picture, which aim to overcome the technical defects.
In order to achieve the above technical object, a first aspect of the present invention provides a method for synchronizing a remote-control-oriented mouse and a returned picture, including the steps of:
decoding the returned picture returned by the controlled terminal to obtain an original image of the host picture, and acquiring the original coordinate of the mouse cursor from the original image of the host picture;
intercepting and analyzing a control command of the mouse at a control end to obtain a target coordinate corresponding to a mouse cursor, and transferring a pixel point corresponding to the mouse cursor in a host frame from an original coordinate to the target coordinate;
and carrying out interpolation processing on pixel points of original coordinates of a mouse cursor in the host picture, and displaying and outputting the processed host picture as a reference picture during control.
The second aspect of the present invention provides a synchronization system for a remote-controlled mouse and a returned picture, which includes the following functional modules:
the original coordinate acquisition module is used for decoding the returned picture returned by the controlled terminal to obtain an original image of the host picture and acquiring the original coordinate of the mouse cursor from the original image of the host picture;
the target coordinate migration module is used for intercepting and analyzing the control command of the mouse at the control end to obtain a target coordinate corresponding to the mouse cursor and migrating a pixel point corresponding to the mouse cursor in the host frame from an original coordinate to the target coordinate;
and the pixel processing output module is used for carrying out interpolation processing on pixel points of original coordinates of a mouse cursor in the host picture, and displaying and outputting the processed host picture as a reference picture during control.
A third aspect of the present invention provides a server, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the method for synchronizing a remote-control mouse and a return screen when executing the computer program.
A fourth aspect of the present invention provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the steps of the method for synchronizing the remote control-oriented mouse and the returned picture.
Compared with the prior art, the synchronization method and the synchronization system for the remote control-oriented mouse and the return picture, provided by the invention, have the advantages that the received return picture from the controlled terminal is decoded and subjected to image recognition processing at the control terminal, the original coordinates of the mouse cursor in the return picture are obtained, and the corresponding pixel points are modified by combining the mouse displacement command intercepted by the control terminal to redraw the return picture, so that the synchronization of the mouse and the return picture is realized.
By adopting the method, the experience of eye-hand fit can be improved when a manager refers to the redrawn return picture for remote control, the movement of a mouse cursor on the real-time picture of the host computer at the controlled end cannot be influenced, and the original working condition of the host computer at the controlled end is not changed.
Drawings
FIG. 1 is a schematic representation of a data codec flow of a keyboard and mouse and video pictures;
FIG. 2 is a block flow diagram of a method for synchronizing a remote-control mouse and a return frame according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a remote control application according to an embodiment of the present invention;
FIG. 4 is a block diagram of a sub-flow of step S1 in FIG. 1;
FIG. 5 is another block diagram of a sub-flow of step S1 in FIG. 1;
FIG. 6 is a block diagram of a sub-flow of step S2 in FIG. 1;
FIG. 7 is a diagram illustrating a redrawing process of a returned picture according to an embodiment of the present invention;
fig. 8 is a block diagram of a synchronization system for a remote mouse and a return screen according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Based on the above, an embodiment of the present invention provides a method for synchronizing a remote-control-oriented mouse and a returned image, as shown in fig. 2, the method includes the following steps:
and S1, decoding the returned picture returned by the controlled terminal to obtain the original image of the host picture, and acquiring the original coordinate of the mouse cursor from the original image of the host picture.
As shown in fig. 3, in the embodiment of the present invention, a video encoder and a video decoder at a console end both use a Hi3531DV200 haisi embedded image processing chip to implement an H264 decoding function on a host picture, and a built-in NNIE engine supports a hardware unit for accelerating a deep learning convolutional neural network and supports most of the existing algorithms such as image identification, image segmentation, and image positioning; encoding and decoding of USB keyboard and mouse signals are achieved by adopting a Qinchang CH9350 chip, and encoded keyboard and mouse control instructions are sent to Hi3531DV200 through a UART; the video stream and the keyboard and mouse control command realize remote data interaction through an Ethernet interface of Hi3531DV 200.
When the image feature of the mouse cursor is included in the preset mouse cursor feature, as shown in fig. 4, the step S1 includes the following steps:
s11, decoding the returned picture returned by the controlled terminal to obtain the original image of the host picture;
s12, segmenting the current latest returned picture frame to obtain the original contour of the mouse cursor by the preset mouse cursor characteristics;
s13, extracting an image section area related to the mouse cursor attribute in the current return picture frame, and recording the original pixel point coordinates MS (X, Y) and YUV components Sigma YUv (X, Y) of the mouse cursor.
That is, when the Hi3531DV200 receives the network video stream of the return picture from the remote host, a single frame image is formed after the VDEC unit built in the Hi3531DV200 is adopted for decoding, and the YUV component values of the pixels of the image are correspondingly generated.
Then, according to the preset mouse cursor characteristics, the current latest returned picture frame is segmented by utilizing an NNIE engine to obtain the original outline of the mouse cursor; and extracting an image section area related to the attribute of the mouse cursor from the current return picture frame, and recording the original pixel point coordinates MS (X, Y) and YUV components Sigma YUv (X, Y) of the mouse cursor.
However, when the image feature of the mouse cursor is not included in the preset mouse cursor feature, as shown in fig. 5, the step S1 includes the following steps:
s11, decoding the returned picture returned by the controlled terminal to obtain the original image of the host picture;
s12, comparing the difference value of the three frames of continuous images which are received and decoded latest to obtain the motion trail of the moving target in the returned picture;
s13, combining the motion track of the moving target with the preset mouse cursor characteristics, and obtaining the original contour of the mouse cursor by dividing from the current latest returned picture frame;
s14, extracting an image section area related to the mouse cursor attribute in the current return picture frame, and recording the original pixel point coordinates MS (X, Y) and YUV components Sigma YUv (X, Y) of the mouse cursor.
That is, when the Hi3531DV200 receives the network video stream of the return picture from the remote host, a single frame image is formed after the VDEC unit built in the Hi3531DV200 is adopted for decoding, and the YUV component values of the pixels of the image are correspondingly generated.
Then, by comparing the difference value of the three frames of continuous images which are received and decoded recently, the motion trail of the moving target in the returned picture can be obtained; combining the motion track of the moving target with the preset mouse cursor characteristics, and segmenting the current latest returned picture frame by using an NNIE engine to obtain the original contour of the mouse cursor; and extracting an image section area related to the attribute of the mouse cursor from the current return picture frame, and recording the original pixel point coordinates MS (X, Y) and YUV components Sigma YUv (X, Y) of the mouse cursor.
And S2, intercepting and analyzing the control command of the mouse at the control end to obtain the target coordinate corresponding to the mouse cursor, and transferring the pixel point corresponding to the mouse cursor in the host picture from the original coordinate to the target coordinate.
As shown in fig. 6, the step S2 includes the following steps:
s21, intercepting and analyzing the control command of the mouse at the control end, and extracting to obtain the movement increment delta M (X, Y) of the mouse cursor;
s22, adding the original pixel point coordinates MS (X, Y) of the mouse cursor with the movement increment delta M (X, Y) to obtain the target pixel point coordinates corresponding to the mouse cursor;
and S23, filling the YUV value of the original pixel point of the mouse cursor into the YUV value of the target pixel point one by one.
When the Hi3531DV200 receives mouse instruction data through the UART, the movement increment delta M (X, Y) of the mouse cursor is extracted, YUV components sigma Yuv (X, Y) are filled one by one pixel according to coordinates obtained by adding the movement increment delta M (X, Y) to an original pixel point MS (X, Y) of the mouse cursor, and the operation of moving the mouse cursor image slice to a target coordinate is realized.
And S3, performing interpolation processing on pixel points of the original coordinates of the mouse cursor in the host picture, and displaying and outputting the processed host picture as a reference picture during control.
In order to avoid the problem that two cursors appear on the returned picture at the same time, interpolation processing is required to be carried out on the original pixel points of the mouse cursor after the movement is carried out. Specifically, based on the original contour of the mouse cursor, the pixel points in the area of the original contour of the mouse cursor are covered with the adjacent pixel points at the periphery of the contour. Part of the pixel points of the interpolated area may be slightly blurred, but only affect the current frame, and the normal display may be resumed in the picture of the next frame because the position of the mouse has moved to the current position in the image of the next frame.
Finally, the redrawn return picture is output through the HDMI interface by calling the interface function Hi _ VO _ Sendframe () of the Hi3531DV 200. The display picture can be used as a reference picture when the control end performs mouse control, so that the hand-eye fit of management personnel can be improved, and the mouse cursor movement of the controlled end host can be ensured to be real-time, thereby improving the experience of remote control.
The invention relates to a synchronization method for a remote control-oriented mouse and a return picture, which decodes and identifies the received return picture from a controlled terminal at a control terminal to obtain the original coordinates of a mouse cursor in the return picture, and modifies a corresponding pixel point to redraw the return picture by combining a mouse displacement command intercepted by the control terminal, thereby realizing the synchronization of the mouse and the return picture. The redrawing process of the returned picture according to the embodiment of the present invention is shown in fig. 7.
By adopting the method, the experience of eye-hand fit can be improved when a manager refers to the redrawn return picture for remote control, the movement of a mouse cursor on the real-time picture of the host computer at the controlled end cannot be influenced, and the original working condition of the host computer at the controlled end is not changed.
As shown in fig. 8, an embodiment of the present invention further provides a synchronization system for a remote-controlled mouse and a returned picture, which includes the following functional modules:
an original coordinate obtaining module 10, configured to decode a returned picture returned by the controlled end to obtain an original image of a host picture, and obtain an original coordinate of a mouse cursor from the original image of the host picture;
the target coordinate migration module 20 is configured to intercept and analyze a control command of the mouse at the control end to obtain a target coordinate corresponding to the mouse cursor, and migrate a pixel point corresponding to the mouse cursor in the host frame from the original coordinate to the target coordinate;
and the pixel processing output module 30 is configured to perform interpolation processing on pixel points of original coordinates of a mouse cursor in a host image, and display and output the processed host image as a reference image during operation and control.
The execution mode of the synchronization system for the remote control-oriented mouse and the returned picture is basically the same as the execution mode of the synchronization system for the remote control-oriented mouse and the returned picture, and therefore detailed description is omitted.
The server in this embodiment is a device for providing computing services, and generally refers to a computer with high computing power, which is provided to a plurality of consumers via a network. The server of this embodiment includes: a memory including an executable program stored thereon, a processor, and a system bus, it will be understood by those skilled in the art that the terminal device structure of the present embodiment does not constitute a limitation of the terminal device, and may include more or less components than those shown, or some components in combination, or a different arrangement of components.
The memory may be used to store software programs and modules, and the processor may execute various functional applications of the terminal and data processing by operating the software programs and modules stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the terminal, etc. Further, the memory may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
The executable program of the synchronization method for the remote control-oriented mouse and the return picture is contained in a memory, the executable program can be divided into one or more modules/units, the one or more modules/units are stored in the memory and are executed by a processor to complete the acquisition and implementation process of the information, and the one or more modules/units can be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used for describing the execution process of the computer program in the server. For example, the computer program may be divided into an original coordinate acquisition module 10, a destination coordinate migration module 20, and a pixel processing output module 30.
The processor is a control center of the server, connects various parts of the whole terminal equipment by various interfaces and lines, and executes various functions of the terminal and processes data by running or executing software programs and/or modules stored in the memory and calling data stored in the memory, thereby performing overall monitoring of the terminal. Alternatively, the processor may include one or more processing units; preferably, the processor may integrate an application processor, which mainly handles operating systems, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor.
The system bus is used to connect functional units in the computer, and can transmit data information, address information and control information, and the types of the functional units can be PCI bus, ISA bus, VESA bus, etc. The system bus is responsible for data and instruction interaction between the processor and the memory. Of course, the system bus may also access other devices such as network interfaces, display devices, etc.
The server at least includes a CPU, a chipset, a memory, a disk system, and the like, and other components are not described herein again.
In the embodiment of the present invention, the executable program executed by the processor included in the terminal specifically includes: a synchronization method for a remote control-oriented mouse and a returned picture comprises the following steps:
decoding the returned picture returned by the controlled terminal to obtain an original image of the host picture, and acquiring the original coordinate of the mouse cursor from the original image of the host picture;
intercepting and analyzing a control command of the mouse at a control end to obtain a target coordinate corresponding to a mouse cursor, and transferring a pixel point corresponding to the mouse cursor in a host frame from an original coordinate to the target coordinate;
and carrying out interpolation processing on pixel points of original coordinates of a mouse cursor in the host picture, and displaying and outputting the processed host picture as a reference picture during control.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (8)
1. A synchronization method for a remote control-oriented mouse and a returned picture is characterized by comprising the following steps:
decoding the returned picture returned by the controlled terminal to obtain an original image of the host picture, and acquiring the original coordinate of the mouse cursor from the original image of the host picture;
intercepting and analyzing a control command of the mouse at a control end to obtain a target coordinate corresponding to a mouse cursor, and transferring a pixel point corresponding to the mouse cursor in a host frame from an original coordinate to the target coordinate;
and carrying out interpolation processing on pixel points of original coordinates of a mouse cursor in the host picture, and displaying and outputting the processed host picture as a reference picture during control.
2. The method according to claim 1, wherein the returned frame returned by the controlled end is decoded to obtain an original image of the host frame, and the original coordinates of the mouse cursor are obtained from the original image of the host frame; the method specifically comprises the following steps:
decoding the returned picture returned by the controlled terminal to obtain an original image of the host picture;
the original contour of the mouse cursor is obtained by dividing the current latest returned picture frame according to the preset mouse cursor characteristics;
and extracting an image section area related to the attribute of the mouse cursor from the current return picture frame, and recording the original pixel point coordinates MS (X, Y) and YUV components Sigma YUv (X, Y) of the mouse cursor.
3. The method according to claim 1, wherein the returned frame returned by the controlled end is decoded to obtain an original image of the host frame, and the original coordinates of the mouse cursor are obtained from the original image of the host frame; the method specifically comprises the following steps:
decoding the returned picture returned by the controlled terminal to obtain an original image of the host picture;
comparing the difference value of the three frames of continuous images which are received and decoded newly to obtain the motion track of the moving target in the returned picture;
combining the motion track of the moving target with the preset mouse cursor characteristics, and segmenting from the current latest returned picture frame to obtain the original contour of the mouse cursor;
and extracting an image section area related to the attribute of the mouse cursor from the current return picture frame, and recording the original pixel point coordinates MS (X, Y) and YUV components Sigma YUv (X, Y) of the mouse cursor.
4. The method according to claim 2 or 3, wherein the operation command of the mouse is intercepted and analyzed at the operation terminal to obtain the destination coordinate corresponding to the mouse cursor, and the pixel point corresponding to the mouse cursor in the host screen is migrated from the original coordinate to the destination coordinate; the method specifically comprises the following steps:
intercepting and analyzing a control command of the mouse at a control end, and extracting to obtain a movement increment delta M (X, Y) of a mouse cursor;
adding the original pixel point coordinates MS (X, Y) of the mouse cursor to the movement increment delta M (X, Y) to obtain the target pixel point coordinates corresponding to the mouse cursor;
and filling the YUV value of the original pixel point of the mouse cursor to the YUV value of the target pixel point one by one.
5. The method according to claim 2 or 3, wherein the interpolation of the pixel points of the original coordinates of the mouse cursor in the host frame specifically includes the following steps:
and covering and processing pixel points in the original contour area of the mouse cursor by using adjacent pixel points at the periphery of the contour based on the original contour of the mouse cursor.
6. A synchronization system for a remote control-oriented mouse and a returned picture is characterized by comprising the following functional modules:
the original coordinate acquisition module is used for decoding the returned picture returned by the controlled terminal to obtain an original image of the host picture and acquiring the original coordinate of the mouse cursor from the original image of the host picture;
the target coordinate migration module is used for intercepting and analyzing the control command of the mouse at the control end to obtain a target coordinate corresponding to the mouse cursor and migrating a pixel point corresponding to the mouse cursor in the host frame from an original coordinate to the target coordinate;
and the pixel processing output module is used for carrying out interpolation processing on pixel points of original coordinates of a mouse cursor in the host picture, and displaying and outputting the processed host picture as a reference picture during control.
7. A server comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for synchronizing a remote control-oriented mouse with a backtransmission screen according to any one of claims 1 to 5 when executing the computer program.
8. A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, carries out the steps of the method for synchronizing a remote-control-oriented mouse with a backtransmission screen according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110980595.1A CN113849073B (en) | 2021-08-25 | 2021-08-25 | Remote control-oriented mouse and return picture synchronization method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110980595.1A CN113849073B (en) | 2021-08-25 | 2021-08-25 | Remote control-oriented mouse and return picture synchronization method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113849073A true CN113849073A (en) | 2021-12-28 |
| CN113849073B CN113849073B (en) | 2024-09-17 |
Family
ID=78976247
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110980595.1A Active CN113849073B (en) | 2021-08-25 | 2021-08-25 | Remote control-oriented mouse and return picture synchronization method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113849073B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115037727A (en) * | 2022-08-10 | 2022-09-09 | 深圳海卫通网络科技有限公司 | Broadband blocking processing method and device for shipborne satellite communication video transmission |
| CN115543714A (en) * | 2022-12-02 | 2022-12-30 | 北京志翔科技股份有限公司 | Remote desktop delay detection method and device, electronic equipment and system |
| CN118097311A (en) * | 2024-04-24 | 2024-05-28 | 广东保伦电子股份有限公司 | Cursor detection method and system based on deep learning |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3997256B1 (en) * | 2007-02-28 | 2007-10-24 | Ｓｋｙ株式会社 | Playback device and program for playback device |
| CN101097507A (en) * | 2006-06-28 | 2008-01-02 | 广达电脑股份有限公司 | Image and cursor image transmitting display system and method thereof |
| CN102591493A (en) * | 2011-12-30 | 2012-07-18 | 浙江海康科技有限公司 | Mouse cursor synchronization method for internet protocol K virtual machine (IPKVM) system |
| CN103092432A (en) * | 2011-11-08 | 2013-05-08 | 深圳市中科睿成智能科技有限公司 | Trigger control method and system of man-machine interaction operating instruction and laser emission device |
| CN105338318A (en) * | 2015-11-09 | 2016-02-17 | 浙江宇视科技有限公司 | Monitoring system and remote control method thereof |
| CN109634444A (en) * | 2018-10-30 | 2019-04-16 | 山东超越数控电子股份有限公司 | A kind of KVM remote mouse synchronisation control means and system |
| CN111831135A (en) * | 2019-04-23 | 2020-10-27 | 凌华科技股份有限公司 | Mouse cursor image detection comparison and feedback state judgment method |
-
2021
- 2021-08-25 CN CN202110980595.1A patent/CN113849073B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101097507A (en) * | 2006-06-28 | 2008-01-02 | 广达电脑股份有限公司 | Image and cursor image transmitting display system and method thereof |
| JP3997256B1 (en) * | 2007-02-28 | 2007-10-24 | Ｓｋｙ株式会社 | Playback device and program for playback device |
| CN103092432A (en) * | 2011-11-08 | 2013-05-08 | 深圳市中科睿成智能科技有限公司 | Trigger control method and system of man-machine interaction operating instruction and laser emission device |
| CN102591493A (en) * | 2011-12-30 | 2012-07-18 | 浙江海康科技有限公司 | Mouse cursor synchronization method for internet protocol K virtual machine (IPKVM) system |
| CN105338318A (en) * | 2015-11-09 | 2016-02-17 | 浙江宇视科技有限公司 | Monitoring system and remote control method thereof |
| CN109634444A (en) * | 2018-10-30 | 2019-04-16 | 山东超越数控电子股份有限公司 | A kind of KVM remote mouse synchronisation control means and system |
| CN111831135A (en) * | 2019-04-23 | 2020-10-27 | 凌华科技股份有限公司 | Mouse cursor image detection comparison and feedback state judgment method |
Non-Patent Citations (1)
| Title |
|---|
| 孙振赢;王毅刚;叶乐晓;: "投影显示中光笔交互技术的研究", 机电工程, no. 06, 20 June 2009 (2009-06-20) * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115037727A (en) * | 2022-08-10 | 2022-09-09 | 深圳海卫通网络科技有限公司 | Broadband blocking processing method and device for shipborne satellite communication video transmission |
| CN115543714A (en) * | 2022-12-02 | 2022-12-30 | 北京志翔科技股份有限公司 | Remote desktop delay detection method and device, electronic equipment and system |
| CN118097311A (en) * | 2024-04-24 | 2024-05-28 | 广东保伦电子股份有限公司 | Cursor detection method and system based on deep learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113849073B (en) | 2024-09-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113849073B (en) | Remote control-oriented mouse and return picture synchronization method and system | |
| US9247029B1 (en) | Efficient processing of streams of images within a moving window session | |
| US20220255979A1 (en) | Video stream management for remote graphical user interfaces | |
| WO2022100522A1 (en) | Video encoding method, video decoding method, apparatus, electronic device, storage medium, and computer program product | |
| CN102883135B (en) | Screen sharing and control method | |
| EP2344957B1 (en) | Remote computing platforms providing high-fidelity display and interactivity for clients | |
| US20110276653A1 (en) | Information processing device, image transmission program, image display program, and image display method | |
| US7275212B2 (en) | Synchronized graphics and region data for graphics remoting systems | |
| US7447997B2 (en) | Reducing information transfer in screen capture series | |
| EP3836555A1 (en) | Video processing method and apparatus, electronic device, and computer-readable medium | |
| CN111818120A (en) | End cloud user interaction method and system, corresponding equipment and storage medium | |
| US20120236199A1 (en) | Information processing apparatus, image transmitting program, image transmitting method and image display method | |
| CN109298806B (en) | Remote quick interface interaction method and device based on object recognition | |
| CN113296865B (en) | Cloud desktop mouse position sensing and mouse operation event management method | |
| US11792245B2 (en) | Network resource oriented data communication | |
| US9226003B2 (en) | Method for transmitting video signals from an application on a server over an IP network to a client device | |
| CN114222185B (en) | Video playing method, terminal equipment and storage medium | |
| CN117609646A (en) | Scene rendering method and device, electronic equipment and storage medium | |
| CN115243102A (en) | Video playing method, device, equipment and storage medium based on Web technology | |
| CN115174917A (en) | H264-based video display method and device | |
| JP5171655B2 (en) | Image transmitting apparatus, method, and storage medium | |
| CN112218003B (en) | Desktop image acquisition method and device and electronic equipment | |
| CN113794887A (en) | Method and related equipment for video coding in game engine | |
| CN107318020B (en) | Data processing method and system for remote display | |
| CN103618968A (en) | Network television playing method and system under cloud environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |