WO2013155735A1

WO2013155735A1 - Off-screen touch control interaction system having projection point coordinate indication of detected proximity

Info

Publication number: WO2013155735A1
Application number: PCT/CN2012/074957
Authority: WO
Inventors: 李波
Original assignee: Li Bo
Priority date: 2012-04-16
Filing date: 2012-05-01
Publication date: 2013-10-24
Also published as: CN102622182A

Abstract

Provided in the present invention is a system for human-computer interaction wherein a touch control device is separate from a graphical interface display screen, said system having projection point coordinate indication of detected proximity. The touch control device detects projection point coordinates when an object approaches a touch panel of said device, and detects touch point coordinates when an object comes into contact with said touch panel; the device also transmits said two types of coordinate data, by means of wired or wireless means, to a graphical interface control module. The graphical interface control module draws a graphical interface, and on the basis of the projection point coordinate data received, draws indication points at corresponding locations on the graphical interface, thereby, by direct visual means, indicating and continuing to track the projection point coordinates of an object close to the touch panel. When an object makes contact with the touch panel, the graphical interface control module, on the basis of the coordinates of and the movement of the touch points, executes corresponding touch graphical interface interactive operations; said control module also simultaneously draws, on the graphical interface, the indication points of the touch points.

Description

Off-screen touch interaction system with proximity sensing projection point coordinate indication

Technical field

The present invention relates to a touch-sensitive graphical interface human-computer interaction system, in particular, the touch device is separated from the display device, the touch device has a coordinate mapping of a close object projection point, and the graphical interface can display a position of a close object projection point. A touch-sensitive graphical interface human-computer interaction system for pointing points.

Background technique

At present, the graphical interface human-computer interaction system mainly has the following two categories.

One type is a graphical interface based on the mouse standard input input that has been widely used for many years, and is represented by the graphical interface of the Window operating system of Microsoft Corporation of the United States. The main feature of this graphical interface interaction is that an indicator point is always displayed on the interface, such as a mouse arrow or a text input cursor. The user operates a two-dimensional plane coordinate input device, such as a mouse, a trackball,

Book

Touchpad (T _0UC hPad), pointing stick (TrackPoint), etc., the device generates two-dimensional coordinate data, the graphical interface locates the mouse or the cursor according to the coordinate data, and moves the mouse or the cursor on the graphical interface according to the change of the coordinate data. , thereby locating various elements on the graphical interface, such as buttons, menus, icons, texts, etc., and then performing a click operation on the positioned interface elements by clicking on the mechanical buttons or the touchpad, or Drag and drop before clicking and releasing the button.

The other type is the touch-sensitive graphical interface that has become popular in recent years. It is represented by the graphical interface adopted by Apple Computer's mobile phones and tablets. The main feature of this graphical interface interaction mode is that the display screen is also a touch panel, whether the touch panel and the display screen are integrated into a single device, or the touch panel is attached as an additional device on the display screen. There is no perceptible difference for the user, and the two are integrated. This touch interface does not need to display a pointing point because the object such as a finger or a stylus itself indicates the location of the touch. Therefore, the user can perform operations such as clicking and dragging on the elements in the graphic interface by using an object such as a finger or a stylus, and can support multiple touch points and perform more complicated functions. Gesture based operation. This kind of interaction brings a very good user experience, and the application on portable devices has been rapidly developed in recent years.

The main drawbacks of the above-mentioned first type of interactive system are: 1. The intuitiveness and ease of use of the graphical interface are relatively poor. For example, selecting an interface element requires two actions: first, moving the indication point to the interface element through the two-dimensional coordinate input device. Above, then click on the mechanical button or trackpad to perform the click operation. 2, such a system usually only supports one indication point at the same time. Even if there are multiple input devices connected to the system, only one indication point can be moved. For example, the notebook computer has both a touchpad and a mouse, but the mouse on the graphical interface Or the cursor is always only one. Therefore, more gesture operations cannot be realized, such as two fingers that are easy to implement on the touch-sensitive graphical interface to enlarge the image separately, and to close the image. 3. The input device of such an interactive system usually requires a mechanical button or a keyboard to fully implement all interactions.

The second type of interactive system is superior to the first type in terms of its intuitiveness and ease of use, but its biggest drawback or limitation is that the touch panel must be integrated with the display screen in order to achieve intuitive operation and interaction. Ease of use. This requires either the touch panel to be fully integrated into the display (in-cel l) or transparent over the display. The problems caused by this are: 1. The technical requirements and production cost of the touch panel are relatively high, and the yield rate in the production process is relatively low. 2, for non-portable devices, such as flat-panel TVs, projectors, etc., the display screen is usually large in size, so the user needs to maintain a certain distance from the display screen in order to obtain a more comfortable field of view and clarity, thus The user cannot directly touch the display screen. If the user must be close to the display screen to perform the touch operation, and then return to the original position after completing the operation, it is undoubtedly inconvenient to use, and the user experience is even worse than the way of using the mouse. This is the biggest limitation of this type of interactive system.

At present, there are some touch-type remote controllers, which are equipped with a touch panel, which can transmit touch information to the receiving end, thereby controlling the remote graphical interactive interface, but the existing products are not well solved. One major contradiction: Because the touch panel is separated from the display screen, the user cannot intuitively position the elements on the graphical interface through the touch panel, and thus the interactive operation cannot be realized intuitively and quickly. Known solutions include: 1. In essence, the two-dimensional coordinate input device of the first type of interactive system is modified into a wireless transmission, or a long-distance wired transmission, and the user controls the display through the touch panel on the remote controller. The mouse or cursor on the screen moves to the specified interface element and then clicks through the click action. This scheme inherits all the defects of the first type of interactive system described above. 2, the remote control has a display screen and a touch panel, the remote controller establishes a two-way data connection with the controlled end, and the controlled end can transmit part or all of the graphical interface displayed on the remote display to the remote controller, in the remote control The display is synchronized on the display, and the user can intuitively perform touch interaction on the remote control. The drawback of this solution is that the user's line of sight must be switched between the remote display and the remote display, and the user cannot simultaneously view the content on the remote display while performing the interactive operation, and the ease of use is greatly reduced; And based on the transmission rate limitation, the content displayed on the remote display is sometimes not completely and real-time displayed on the remote control display; in addition, the cost of the remote controller and the receiving end of the solution is high.

Summary of the invention

The invention provides a touch-sensitive graphical interface human-computer interaction system, comprising: a touch device separated from the graphic interface display device, wherein the touch panel has a proximity sensing function, which can sense the object when contacting the touch panel. The coordinate data of the touch point can also sense the coordinate data of the projection point of the object close to the touch panel on the touch panel, and transmit the two coordinate data to the graphic interface control module; the graphic interface control module displays The device is displayed in the form of a pointing point. The graphical interface control module can be integrated with the display device in the same device, such as a flat panel television, or as a separate device and output a picture to the display device, such as a set top box. When the user operates the touch device, the line of sight does not have to leave the display screen. When an object such as a finger or a stylus is close to the touch panel and sensed, the position of the corresponding indication point can be visually seen on the display screen. And the trajectory of its movement. Therefore, the user can perform an intuitive touch operation on the remote display as if the portable device touch interface is operated.

The touch device according to the present invention includes: a touch panel, a control module, and a data transmission module. The touch panel senses the position of the touch point when the object is in contact and outputs the coordinate data, and can sense the position of the object relative to the projection point of the touch panel when the object approaches and output the coordinate data. The touch panel can simultaneously support sensing of one or more touch points and projection points. The control module reads the coordinate data of the touch point and the projection point output by the touch panel, and converts it into a format that can be sent by the data transmission module. The data transmission module receives data from the control module and transmits the data to the receiving end.

The graphic interface control module according to the present invention comprises: a data transmission module, a data processing module, a touch graphic interface control module, a pointing point control module, and a screen merge display module. The data transmission module is responsible for communicating with the touch device A wireless or wired data connection receives data transmitted by the touch device and forwards it to the data processing module. The data processing module parses the received data to obtain the coordinates of the touch point and the coordinates of the projection point output by the touch panel, and distributes the coordinates to the pointing point control module and the touch graphic interface control module. The touch-type graphical interface control module is responsible for drawing a graphical interface, and performing corresponding touch interface interaction operations according to the received coordinate data of one or more touch points, that is, implementing the functions of the second type of interactive system as described above. The indication point control module draws one or more indication points that are obviously visible according to the received one or more touch point coordinate data and the projection point coordinate data at a position corresponding to the graphic interface, the touch point coordinate data and the projection The change of the point coordinate data directly represents the movement of the indication point on the graphical interface. The screen merge display module combines the graphical interface and all the indication points into one screen for display on the display device, and ensures that all the indication points are located on the graphical interface in the Z-axis direction without being occupied by any interface elements. Cover.

The indication point according to the present invention includes: having the same number of indication points on the display screen according to the number of projected close-point object projection points; having the number of touch points of the touched object on the display screen The same number of indicator points. The indicator point is clearly visible anywhere on any graphical interface. The indication point can be distinguished by different display modes to indicate whether it is a touch point or a projection point. Different display modes can be distinguished by indicating the shape, size, texture, color or other manner of the point, and whether it is necessary to distinguish between two types. The point of instruction depends on the needs of the user and can be configured by the user. When the proximity object moves within the sensible range but has not touched the touch panel, the indication point follows the movement of the projection point of the proximity object to move correspondingly on the graphical interface. When the proximity object continues to approach and then touches the touch panel, the indication point indicates the location of the touch point. When the object remains in contact with the touch panel and moves, the pointing point moves correspondingly as the touch point moves. When the object leaves the touch panel but is still within the proximity sensing range, the pointing point in turn indicates the position of the projection point. The pointing point disappears when the object continues to move away from the touch panel and beyond the range of proximity sensing.

The correspondence between the touch panel of the touch device and the graphical interface drawn by the graphical interface control module has a shape, a size, and a coordinate, and includes: the effective sensing area of the touch panel and the graphical interface The touch areas are similar in shape, and do not have to be identical; the two sizes do not have to be the same; the resolutions of the two are not necessarily the same; when the shapes or sizes or resolutions of the two are not identical, the coordinate data output by the touch panel is fixed. After the ratio conversion, the touchable area of the graphical interface can be completely covered.

Through the touch-type human-computer interaction system provided by the present invention, the user can stay on the display screen of the display device while operating, observe the graphical interface, and place an object such as a finger or a stylus on the touch panel of the touch device. When the upper distance is within a certain distance, the indication point corresponding to the projection point of the object on the touch panel can be visually seen on the display screen. When the user moves a finger or a stylus to sense the object without leaving the sensible range, the pointing point can be visually seen to move accordingly. When the user continues to approach the touchable panel such as a finger or a stylus and then touches the touch panel, the pointing point reflects the position touched by the user on the graphical interface. As shown in Figure 1. The user can intuitively operate the touch-sensitive graphical interface on the remote display without any obstacles.

The invention has the advantages of overcoming the defects and limitations while completely retaining the intuitiveness and ease of use of the foregoing second type of interactive system, and the user can still obtain an intuitive and excellent touch when the display screen is far away. Controlled interactive experience. The invention can be applied to a flat panel television, a set top box, a video conference system, and other occasions where a touch type human-machine interaction in which a touch device and a display device are separated is required. DRAWINGS

FIG. 1 is a schematic diagram showing that a touch device senses finger contact and proximity, and a corresponding indication point is synchronously displayed on a display screen of the display device.

2 is a scenario in which a user does not approach or touch a finger to a touch panel according to an embodiment of the present invention.

FIG. 3 is a view of a user when a user approaches a finger to a touch panel according to an embodiment of the present invention.

FIG. 4 is a view showing a scenario in which a user moves a finger within a sensible range above the touch panel to locate an interface icon that is desired to be touched according to an embodiment of the present invention.

FIG. 5 is a view showing a scenario in which a user continues to approach a finger until it touches the touch panel according to an embodiment of the present invention.

FIG. 6 is a view of a user when a user slides a finger on a touch panel according to an embodiment of the present invention.

FIG. 7 is a view showing a scenario in which a user lifts a finger away from a touch panel according to an embodiment of the present invention.

FIG. 8 is a scenario of a user moving a finger out of a touch panel beyond a sensing range according to an embodiment of the present invention.

FIG. 9 is a structural diagram of a system according to an embodiment of the present invention.

FIG. 10 is a schematic diagram of calculating a conversion ratio between coordinate data of a touch panel and coordinate data of a touch interface of a graphical interface. FIG. 11 is a schematic diagram of combining the indication point screen and the touch type graphic interface.

detailed description

A specific embodiment of the invention is set forth herein to illustrate and explain the invention in detail, but not to limit the invention.

In the present embodiment, a flat panel television is taken as an example, which has a remote controller with a touch panel, and displays a touch-sensitive graphical user interface on the television screen. When the user operates the remote control, the line of sight can stay on the TV screen without having to switch to the remote control. Here, the operation of the user moving an icon on the graphical interface from the upper right corner of the screen to the lower left corner of the screen is taken as an example, and the operation steps are as follows.

Step one, when the user does not directly touch any finger to the touch panel of the remote controller, and does not bring any finger close to the touch panel of the remote controller and enters the proximity sensing range, no indication point is displayed on the television screen. as shown in picture 2.

Step 2: When the user's finger approaches the touch panel of the remote controller and is sensed, an indication point indicating the position of the finger appears on the television screen. In this embodiment, an open circle is used as an indication point of the projection point, as shown in the figure. 3 is shown.

Step 3: When the user moves the finger over the touch panel but has not touched the touch panel, the pointing point on the TV screen moves accordingly. The user can thus move the finger while finding the position of the icon that he wants to touch based on the pointing point on the TV screen, as shown in FIG.

Step 4: After the user moves the finger to make the indication point reach the predetermined position, the finger continues to approach the touch panel until the touch panel is touched. In this embodiment, the indication point becomes a solid dot to indicate the touch point. At the same time, a touch operation instruction is obtained by the interface element positioned by the indication point, for example, an icon at the position is selected. As shown in Figure 5.

Step 5: When the user keeps the finger in contact with the touch panel and moves on the touch panel, the pointing point also follows the finger touch point to perform corresponding movement, and the touch interface can perform a corresponding touch operation according to the movement, for example , selected icon Moves along the position of the indicated point. As shown in Figure 6.

Step 6: When the user moves the icon to the predetermined position and lifts the finger away from the touch panel but is still within the range of the proximity sensing, the corresponding indication point is still visible on the TV screen, but becomes The circle back to the hollow indicates that the projected point is not the touch point, as shown in FIG.

Step 7: When the user continues to lift the finger and exceeds the proximity sensing range of the touch panel, the corresponding indication point disappears on the TV screen. As shown in Figure 8.

It should be understood that the above operational steps are merely illustrative of a typical mobile icon interaction, and the above operations are not limited to limit the embodiments of the present invention.

Fig. 9 is a structural diagram of the system of the embodiment. Specifically, only the components and modules related to the present invention are shown in the drawing, and the television signal receiving processing module, the video signal receiving processing module, and the like which are commonly used in flat panel televisions are not shown.

The touch panel 10 in the figure is a rectangular capacitive touch panel (Touch Pad), which can sense a plurality of fingers or a stylus hovering within a certain range above the same, and output the finger or the stylus The two-dimensional coordinates of the point are projected on the touch panel. The touch panel can also sense the position of the finger or stylus that is in direct contact with it, and output two-dimensional coordinates. At present, the more mature capacitive touch panels are mainly divided into self-capacitance and mutual capacitance, or a combination of the two. By carefully adjusting the arrangement of the capacitive sensing contacts and the signal-to-noise ratio (SNR), sensing of the proximity and touch objects can be achieved and accurate coordinate data can be output. Touch panel manufacturers have introduced such products.

The control module 11 is a main control chip of the remote controller 1 and adopts a mainstream low-power type MCU, which initializes the touch panel 10 to make it work to sense the proximity and touch of the object, and can read and touch at any time. The touch point coordinate data 110 and the projection point coordinate data 120 outputted by the control panel 10 are then packed into a format suitable for transmission and then forwarded to the data transmission module 12.

The data transmission module 12 on the remote controller 1 establishes a data connection with the data transmission module 25 of the flat panel television 2. Currently available wireless transmission methods include infrared, Bluetooth, WiFi, Zigbee, etc., and the embodiment adopts Zigbee wireless communication technology, and the power consumption thereof Low, suitable for short-distance transmission, most existing Zigbee chips have a data transmission rate of 250Kbps or higher, which is fully capable of the amount of data of two-dimensional coordinates that need to be transmitted in this embodiment. Taking the sampling rate of 200Hz for each touch point or projection point as an example, the X-axis and Y-axis coordinates of each coordinate point occupy 2 bytes respectively, and the data amount of each touch point or projection point is only 200*4*8=6400bps. Even if data of 10 touch points or projection points are transmitted at the same time, the total data amount is only 60 Kbps.

When the remote controller 1 and the flat panel TV 2 first establish a connection through the data transmission modules of the two sides, the control module 11 of the remote controller 1 transmits the resolution of the touch panel 10 to the data processing module 24 of the flat panel television 2 through the data transmission channel, and the data The processing module 24 calculates the ratios RatioX and RatioY of the linear conversion between the X-axis and the Y-axis coordinate data according to the resolution of the touch panel 10 and the resolution of the effective touch area of the touch-type graphical interface control module 22, As shown in Figure 10. The data processing module 24 is provided with floating point computing capability to improve the accuracy of coordinate conversion. Specifically, in this embodiment, the resolution of the effective touch area of the graphical interface is equal to the resolution of the display screen 20, that is, all areas displayed on the display screen 20 are touch-operable areas, and in other specific In the application, the two do not have to be exactly equal, and the effective touch area may be smaller than the resolution of the display screen, that is, only the display area displayed on the display screen can be touch-operated. The effective touch area may also be larger than the resolution of the display screen, and the present invention does not limit this, but since the excess portion does not appear on the display screen Shown on it, there is not much significance in practical applications.

The data transmission module 25 of the flat panel television 2 forwards the received data to the data processing module 24, and the data processing module 24 unpacks the data to obtain each touch point coordinate data 110 and projection point coordinate data 120, and according to the foregoing The calculated coordinate data linear conversion ratios RatioX and RatioY, the converted touch point coordinate data 210 and the projection point coordinate data 220 are calculated and transmitted to the indication point control module 23, and at the same time, the converted touch position coordinates are The data 210 is transmitted to the touch graphic interface control module 22.

The pointing point control module 23 outputs a screen having the same resolution as the effective touch area of the graphical interface, that is, equal to the resolution of the display screen 20, the background of the screen is transparent, and is transmitted according to the data processing module 24. The converted touch point coordinate data 210 and the projected point coordinate data 220 are drawn on the screen, and the indication points indicating the touch points and the indication points indicating the projection points are distinguished in different display forms. In this embodiment, the indication point of the projection point is indicated by an open circle, and the indication point of the touch point is indicated by a solid dot.

The touch-sensitive graphical interface control module 22 outputs a screen on which the graphical interface is drawn, and performs corresponding touch interaction operations according to the converted touch point coordinate data 210 sent by the data processing module 24, and updates accordingly. Draw a graphical interface. For example, when the touch point is on a button, the action defined by the button is performed; when a series of consecutive touch point coordinate changes are detected, that is, the finger slides on the touch panel, the corresponding operation of the sliding gesture is performed. , and many more. The touch-sensitive graphical interface control module 22 can be developed by itself or directly using existing technologies, such as the Android tablet touch interface developed by Google.

The screen merge display module 21 merges the indication point screen outputted by the indication point control module 23 and the graphical interface output by the touch-sensitive graphic interface control module 22, and the merged rule is: the indication point screen is located at the graphical interface in the Z-axis direction. Up; The point display screen is merged onto the graphical interface in alpha blending. This ensures that all the pointing points are clearly visible after being merged with the graphical interface, and the content on the graphical interface is not completely covered by the indicated points, as shown in Figure 11. The screen merge display module 21 is usually implemented by a dedicated graphics processing unit (GPU), which does not occupy a large amount of processing time of the main control chip (CPU), and ensures a smooth display effect. In the current consumer electronics field, especially in medium and high-end devices with touch-sensitive graphical interfaces, most of the mainstream embedded single-chips used have integrated GPUs.

The display screen 20 receives the screen output from the screen merge display module 21 and displays it to the user.

As described above, according to the technical solution provided by the embodiment of the present invention, when the user interacts with the touch-sensitive graphical interaction interface on the flat-panel TV 2 through the remote controller 1, the finger or the stylus can be intuitively perceived by using the indication point. The location, and whether it is in contact with the touchpad. Its user experience is more intuitive and convenient than the direct touch on the display with a finger or stylus. The shortcomings that will block some screens can be said to further enhance the user experience. On a flat-panel TV based on this technology solution, users can smoothly operate the graphical interface, browse web pages, pictures, and operate rich applications, as well as games.

The above description is only a preferred embodiment of the present invention on a flat panel television product, and is not intended to limit the present invention. The present invention is susceptible to various modifications and changes to those skilled in the art. Any modifications, substitutions, improvements, etc. made within the spirit and scope of the present invention are intended to be included within the scope of the present invention.

Claims

Claim

A touch-sensitive graphical interface human-computer interaction system, comprising a touch device, a graphic interface control module, and a display device, wherein: the touch device is separated from the display device, and the touch panel on the touch device can sense When the object touches the touch panel, the coordinates of the touch point and the coordinates of the projection point on the touch panel when the object approaches, the data connection is established between the touch device and the graphic interface control module, and the coordinates of the touch point and the projection point are set. Transferred to the graphical interface control module, the graphical interface control module draws corresponding indication points on the graphical interface according to the coordinates of the received touch points and projection points, and all the indicated indication points are not covered by other interface elements. When the touch point and the projection point move, the corresponding indication point moves on the graphical interface. If the touch point and the projection point disappear, the corresponding indication point disappears on the graphical interface, and the graphical interface control module draws The graphical interface and indicator points are output to the display device.

2. The touch-sensitive graphical interface human-computer interaction system according to claim 1, wherein:

The touch device, the graphic interface control module, and the display device are separate devices.

3. The touch-sensitive graphical interface human-computer interaction system according to claim 1, wherein:

The touch device is a separate device, and the graphical interface control module and display device are integrated into another separate device.

4. The touch-sensitive graphical interface human-computer interaction system according to claim 1, wherein:

The touch device and the graphical interface control module are integrated into one device, and the display device is another separate device.

5. The touch-sensitive graphical interface human-computer interaction system according to claim 2 or claim 3, wherein:

A data connection is established between the touch device and the graphical interface control module through the wireless communication module and data is transmitted.

6. The touch-sensitive graphical interface human-computer interaction system according to claim 2 or claim 3, wherein:

A data connection is established between the touch device and the graphical interface control module through a wired communication module and data is transmitted.

A touch device comprising a touch panel and a data transmission module, wherein: the touch panel has a proximity sensing function, which can sense the coordinates of the touch point when the object touches the touch panel, or The coordinates of the projection point on the touch panel when the object approaches the touch panel are sensed, and the coordinate data of the touch point and the projection point are transmitted through the data transmission module.

The touch device according to claim 7, wherein: the touch panel can simultaneously sense coordinates of one or more touch points and projection points, and send all touch points and projection points through the data transmission module. Coordinate data.

9. The touch device according to claim 7, wherein the data transmission module adopts a wireless communication mode.

10. The touch device according to claim 7, wherein the data transmission module adopts a wired communication method.

11. A graphical interface control module, which outputs a screen of a graphical interface, receives data from the touch device, performs a corresponding touch operation, and updates a screen of the graphical interface, wherein: according to the touch point received from the touch device The coordinate data of the projection point is positioned to the corresponding position on the graphical interface, and the indication point is drawn at the position, and the indicated indication point is not covered by any interface element on the graphic interface; when the coordinates of the touch point and the projection point are When the data changes, the indicated point drawn at the original position is erased, and the indication point is drawn at the changed position, so that the movement of the indication point on the graphical interface reflects the movement of the touch point and the projection point; when the touch point or When the projection point disappears, the indication point corresponding to the touch point or projection point is erased.

12. The graphic interface control module according to claim 11, wherein: distinguishing whether the coordinate data received from the touch device indicates a touch point or a projection point, and drawing different indication points for the touch point and the projection point. .

13. The graphical interface control module according to claim 11, wherein: the configurable option can specify, by using a configuration option, whether an indication point of the touch point needs to be drawn, and whether an indication point of the projection point needs to be drawn.

The graphic interface control module according to claim 11, comprising:

a data transmission module, configured to establish a data connection with the touch device according to claim 7, receive data from the touch device, and forward the data to the data processing module;

The data processing module parses the touch point coordinate data and the projection point coordinate data from the data forwarded by the data transmission module, and forwards the data to the indication point control module and the touch graphic interface control module;

The indication point control module draws all the indication points according to the received coordinate data of the touch point and the coordinate data of the projection point, and outputs the indication points to the screen merge display module;

The touch-type graphic interface control module draws a picture of the graphic interface, performs a corresponding touch operation according to the received touch point position data and the projection point position data, and simultaneously updates the graphic interface picture, and the graphic interface The screen is output to the screen merge display module;

The screen merge display module combines the received screen of the graphical interface and the pointing point screen into one screen, and then outputs the image to the display device.