CN119440441A

CN119440441A - Screen projection display control system, method and device

Info

Publication number: CN119440441A
Application number: CN202310955859.7A
Authority: CN
Inventors: 汪然; 张超
Original assignee: BOE Technology Group Co Ltd; Beijing BOE Technology Development Co Ltd
Current assignee: BOE Technology Group Co Ltd; Beijing BOE Technology Development Co Ltd
Priority date: 2023-07-31
Filing date: 2023-07-31
Publication date: 2025-02-14

Abstract

The embodiment of the invention provides a screen-throwing display control system, a screen-throwing display control method and screen-throwing display control equipment, wherein the system comprises screen-throwing equipment and at least one piece of equipment to be thrown, the screen-throwing equipment is in communication connection with the equipment to be thrown, the screen-throwing equipment is used for sending screen-throwing pictures, the equipment to be thrown is used for receiving the screen-throwing pictures and displaying the screen-throwing pictures, the screen-throwing equipment is also used for acquiring the gaze point information of a user, the screen-throwing equipment or the equipment to be thrown is also used for determining first content which needs to be highlighted in the screen-throwing pictures according to the gaze point information, and the equipment to be thrown is used for highlighting the first content. Highlighting of the first content is achieved in the device to be cast based on the gaze point information, and display effect and user experience are improved.

Description

System, method and equipment for controlling screen display

Technical Field

The embodiment of the invention relates to the technical field of display, in particular to a system, a method and equipment for controlling screen display.

Background

With the continuous development of technology, the use frequency of users for screen projection applications is also higher and higher. Meanwhile, with the rapid development of large-screen display devices, large-size conference whiteboard devices, business display devices, projectors, digital advertising signs and the like are also more applied to conference, education and exhibition scenes, so that contents on small-screen display devices such as notebook computers, mobile phones and tablets often need to be displayed and explained by throwing the contents on the large-screen display devices onto the large-screen display devices, or the contents on the large-screen display devices need to be thrown onto a plurality of small-screen display devices to be controlled.

When the content of the large-screen display device is projected to a plurality of mobile phones or flat panels, more content needs to be displayed in a smaller screen, so that a user cannot see the content of the projected screen clearly, and viewing and controlling experience are affected.

In summary, no matter the content on the large-screen display device is projected onto a plurality of small-screen display devices or the content on the small-screen display device is projected onto the large-screen display device, the conventional method cannot obtain the ideal display effect, and the user experience is poor.

Disclosure of Invention

The embodiment of the invention provides a screen-throwing display control system, a screen-throwing display control method and screen-throwing display control equipment, which are used for solving the problems of poor display effect and poor user experience of the existing screen-throwing method.

In a first aspect, an embodiment of the present invention provides a screen display control system, including a screen display device and at least one device to be projected, where the screen display device is communicatively connected with the device to be projected;

the screen throwing device is used for sending screen throwing pictures;

the device to be thrown is used for receiving and displaying the screen throwing picture;

the screen throwing equipment is also used for acquiring the gaze point information of the user;

The screen throwing device or the device to be thrown is also used for determining first content which needs to be highlighted in the screen throwing picture according to the gaze point information;

the thrown device is used for highlighting the first content.

In one embodiment, the projected device is configured to highlight the first content when the screen physical size of the projected device is greater than the screen physical size of the projected device, including the projected device being configured to magnify the first content in the screen of the projected device.

In one embodiment, when the screen physical size of the screen throwing device is smaller than the screen physical size of the thrown device, the thrown device is used for highlighting the first content, and the thrown device is used for highlighting the first content in the screen of the thrown device through at least one mode selected by highlighting, a cursor and a frame.

In a second aspect, an embodiment of the present invention provides a method for controlling a projection display, which is applied to a device to be projected, including:

Acquiring screen casting data and gaze point information, wherein the gaze point information is used for representing the position information of a user sight focus in a screen of the screen casting device;

And highlighting the screen throwing data according to the gaze point information.

In one embodiment, before acquiring the screen casting data and the gaze point information, the method further comprises acquiring a resolution of the screen casting device;

highlighting the screen casting data according to the gaze point information, including:

Determining first content to be highlighted from the screen casting data according to the gaze point information, the resolution of the screen casting device and the resolution of the device to be cast;

The first content is highlighted.

In one embodiment, highlighting the first content when the screen physical size of the projection device is greater than the screen physical size of the projected device comprises:

the first content is displayed in an enlarged manner in a screen of the device to be thrown.

In one embodiment, highlighting the first content when the screen physical size of the projection device is smaller than the screen physical size of the projected device comprises:

the first content is highlighted in the screen of the projected device by at least one of highlighting, cursor and box selection.

In one embodiment, the method further comprises:

responding to touch operation of a user on a screen of the device to be thrown, and acquiring touch point coordinates;

And updating the touch point coordinates to target coordinates.

In one embodiment, the method further comprises:

generating a click event according to the touch operation;

and sending the generated click event to the screen throwing equipment so that the screen throwing equipment controls the screen throwing equipment according to the click event.

In one embodiment, obtaining a resolution of a projection device comprises:

Receiving a response frame of the screen throwing equipment for searching the broadcast frame sent by the thrown equipment;

And obtaining the resolution of the screen throwing device from the response frame.

In a third aspect, an embodiment of the present invention provides a method for controlling a screen display, which is applied to a screen display device, including:

acquiring the gaze point information of a user;

Acquiring first content to be highlighted in a screen projection picture according to the gaze point information of a user;

And performing highlighting processing on the first content, and sending the first content after the highlighting processing to the thrown device.

In a fourth aspect, an embodiment of the present invention provides an electronic device, including:

A processor and a memory;

The memory stores computer-executable instructions;

the processor executing computer-executable instructions stored in the memory causes the processor to perform the method of controlling a projection display as described in any of the second or third aspects.

In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, where computer-executable instructions are stored, where the computer-executable instructions are used to implement the method for controlling a screen display according to any one of the second aspect or the third aspect when the computer-executable instructions are executed by a processor.

The system comprises a screen throwing device and at least one device to be thrown, wherein the screen throwing device is in communication connection with the device to be thrown, the screen throwing device is used for sending a screen throwing picture, the device to be thrown is used for receiving the screen throwing picture and displaying the screen throwing picture, the screen throwing device is also used for acquiring the gaze point information of a user, the screen throwing device or the device to be thrown is also used for determining first content which needs to be highlighted in the screen throwing picture according to the gaze point information, and the device to be thrown is used for highlighting the first content. Highlighting of the first content is achieved in the device to be cast based on the gaze point information, and display effect and user experience are improved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.

FIG. 1 is a flowchart of a method for controlling a projection display according to an embodiment of the present invention;

FIG. 2 is a flowchart of a method for controlling a projection display according to another embodiment of the present invention;

FIG. 3 is a schematic diagram of determining display scale according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a projection display according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of a projection display according to another embodiment of the present invention;

FIG. 6 is a flowchart of a method for controlling a projection display according to another embodiment of the present invention;

FIG. 7 is a flowchart of a method for controlling a projection display according to another embodiment of the present invention;

FIG. 8 is a schematic diagram of a projection display control system according to an embodiment of the present invention;

Fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Specific embodiments of the present invention have been shown by way of the above drawings and will be described in more detail below. The drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but rather to illustrate the inventive concepts to those skilled in the art by reference to the specific embodiments.

Detailed Description

The application will be described in further detail below with reference to the drawings by means of specific embodiments. Wherein like elements in different embodiments are numbered alike in association. In the following embodiments, numerous specific details are set forth in order to provide a better understanding of the present application. However, one skilled in the art will readily recognize that some of the features may be omitted, or replaced by other elements, materials, or methods in different situations. In some instances, related operations of the present application have not been shown or described in the specification in order to avoid obscuring the core portions of the present application, and may be unnecessary to persons skilled in the art from a detailed description of the related operations, which may be presented in the description and general knowledge of one skilled in the art.

Furthermore, the described features, operations, or characteristics of the description may be combined in any suitable manner in various embodiments. Also, various steps or acts in the method descriptions may be interchanged or modified in a manner apparent to those of ordinary skill in the art. Thus, the various orders in the description and drawings are for clarity of description of only certain embodiments, and are not meant to be required orders unless otherwise indicated.

The numbering of the components itself, e.g. "first", "second", etc., is used herein merely to distinguish between the described objects and does not have any sequential or technical meaning. The term "coupled" as used herein includes both direct and indirect coupling (coupling), unless otherwise indicated.

In the embodiment of the application, when the interface of the device A is projected onto the device X, the device A is the projected device, and the device X is the projected device. The screen projection display control method provided by the application is described below from the angles of the projected equipment and the screen projection equipment respectively.

Fig. 1 is a flowchart of a method for controlling a projection display according to an embodiment of the present invention. As shown in fig. 1, the method for controlling the screen display provided in this embodiment is applied to a device to be thrown, and may specifically include:

s101, screen casting data and gaze point information are acquired, wherein the gaze point information is used for representing position information of a user' S sight focus in a screen of the screen casting device.

After the screen is started, the interface of the screen throwing device is projected onto the device to be thrown, and the screen throwing data in the embodiment can be understood as data related to the screen throwing picture. The gaze point information in this embodiment is used to represent the position information of the user's gaze focus in the screen of the screen capturing device, and may be, for example, the position coordinates of the user's gaze focus in the screen capturing device.

In the screen projection process, the interpreter looks at the screen projection equipment, different gaze points are formed at different positions of the browsing screen through eyeballs, and in the embodiment, gaze point coordinates can be obtained through locating eyeballs based on an eyeball tracking technology and can represent the position of a user gaze focus in the screen of the screen projection equipment. Specifically, a person always rotates an eyeball while seeing an object, so that the eyeball position of a person exactly provides a good indication, namely, the eyeball position can reflect which part of a scene the person observes. Typically, human eye movement occurs nearly instantaneously and rapidly. And then there is a relatively stationary staring process of 200 ms-600 ms to see the object. The eye is now relatively completely stationary with minor jitter, but during this gaze the jitter is typically less than 1 degree. This also happens only when a person observes a moving object if the eye has a smooth rotation. Other eye movements, such as rotation, dispersion, etc., appear relatively unimportant in the human-machine conversation and can be ignored. Overall, a person sitting in front of a computer operates by using the smooth gaze direction of the eye as input. The eye positioning device and the sight tracking device can be fixedly arranged in front of the computer screen, and when the face of a person is irradiated by an infrared auxiliary light source (LED array), a reflection image is formed on the cornea surface of the eye, and the image is called Purkinje point. As the human eye looks at different positions on the computer screen, the eye can rotate correspondingly. The eye positioning device records an initial position of the eyeball. Assuming that the experimenter's head is stationary and the position of the infrared light emitting diode is fixed, the eyeball is an approximate sphere, so the absolute position of the purkinje point can be considered unchanged when the eyeball rotates; the positions of the iris and the pupil are correspondingly changed, so that the relative position relation between the purkinje point formed by the infrared auxiliary light source on the iris and the pupil and the iris is also changed, the infrared camera can determine the positions of the purkinje point and the pupil in real time, and the determination of the relative position relation can be realized through image processing. The direction of the line of sight input is then derived from the relative positional relationship between them to determine the gaze point on the screen and to determine the gaze point coordinates. And then the screen throwing device sends the screen throwing data and the fixation point coordinates to the thrown device in real time.

S102, highlighting the screen throwing data according to the gaze point information.

After the device to be cast obtains the screen casting data and the gaze point information, the screen casting data can be highlighted based on the gaze point information. Specifically, the data to be highlighted may be obtained from the projection screen data according to the gaze point information, for example, the gaze area of the user may be determined according to the gaze point information, and the data in the gaze area is the data to be highlighted. Wherein the gaze area may be determined from gaze point coordinates. Specifically, the region where the human eye can look is a circular region, and a circular region can be determined by determining the coordinates of the point of gaze of the human eye, so that the region of gaze can be determined according to the circular region. Thus, the viewing area may comprise an circumscribed rectangular area of the circular area on the display panel, or other areas, such as triangular areas, etc.

According to the screen projection display control method, the screen projection data and the gaze point information are acquired, the gaze point information is used for representing the position information of the user's gaze focus in the screen of the screen projection device, the screen projection data are highlighted according to the gaze point information, highlighting of the screen projection data is achieved in the projected device based on the gaze point information, and the improvement of display effect and user experience is facilitated.

In general, the projected device and the screen-throwing device have different resolutions, which causes the same position to have different coordinate information in the projected device and the screen-throwing device, so that the conversion of the position coordinates needs to be performed according to the resolutions of the projected device and the screen-throwing device, so that the content watched by the user can be accurately highlighted. That is, in addition to the screen casting data and the gaze point information, the resolution of the screen casting device needs to be acquired, and further, the first content needing to be highlighted can be determined from the screen casting data according to the gaze point information, the resolution of the screen casting device and the resolution of the device to be cast, and the first content is highlighted. Referring to fig. 2 specifically, fig. 2 is a flowchart of a method for controlling a screen display according to still another embodiment of the present invention. As shown in fig. 2, the method for controlling the screen display provided in this embodiment is applied to a device to be thrown, and may specifically include:

S201, acquiring the resolution of the screen throwing equipment.

The screen throwing device and the thrown device can communicate in a wireless or wired mode, for example, the screen throwing device and the thrown device can communicate through Wifi, bluetooth, a mobile network and the like. Before the screen-throwing control is started, related information interaction is carried out between the two devices in the negotiation stage, other devices in the communication range are found by searching the broadcast frames, and related information is acquired from response frames returned by the other devices. In this embodiment, the device to be thrown may send a seeking broadcast frame through a user datagram protocol (User Datagram Protocol, UDP), where the content of the seeking broadcast frame may include information of an own internet protocol (Internet Protocol, IP) Address and a media access Control Address (MEDIA ACCESS Control Address, MAC), and after the device to be thrown listens to obtain the seeking broadcast frame sent by the device to be thrown, send a response frame including the own IP, MAC and resolution information through UDP.

In an alternative implementation manner, obtaining the resolution of the screen throwing device can specifically include receiving a response frame of the screen throwing device to the seeking broadcast frame sent by the device to be thrown, and obtaining the resolution of the screen throwing device from the response frame.

S202, screen casting data and gaze point information are acquired, and the gaze point information is used for representing position information of a user' S sight focus in a screen of the screen casting device.

Reference is made to step S101 in the above embodiment, and the description thereof is omitted here.

S203, determining the first content which needs to be highlighted from the projection data according to the gaze point information, the resolution of the projection device and the resolution of the projected device.

After the resolution of the screen throwing device and the resolution of the device to be thrown are obtained, the display proportion can be determined according to the resolution of the device to be thrown and the resolution of the device to be thrown, and then coordinate conversion can be carried out between the device to be thrown and the device to be thrown according to the display proportion, for example, the position coordinates in the device to be thrown can be converted into the corresponding position coordinates in the device to be thrown.

After obtaining the resolution of the screen-throwing device, the device to be thrown can determine the display proportion according to the self resolution and the obtained resolution of the screen-throwing device. Specifically, taking fig. 3 as an example, in the process of establishing a communication connection, a device to be cast (receiving end device) X firstly transmits its own information through broadcasting, and after receiving the information of the device X, a screen casting device (transmitting end device) a transmits its own resolution (Wa, ha) to the device X in a response frame manner. After receiving the resolution of the device a, the device X calculates the resolution with the self resolution (Wx, hx) to obtain a display proportion (wd=wx/Wa, hd=hx/Ha), and the display proportion can be used for subsequent coordinate conversion.

The display scale may be used for coordinate conversion, and coordinates in the projection device may be converted into coordinates in the projected device according to the display scale. Taking fig. 3 as an example, assume that the device a obtains the gaze point coordinate (Xa, ya) through the eye tracking module and sends the gaze point coordinate to the device X, and the device X performs coordinate conversion calculation according to the display proportion to obtain (xb=xa×wd, yb=ya×hd), so as to obtain the target coordinate according with the resolution of the device X, where the target coordinate represents the corresponding position of the gaze point coordinate in the device to be cast.

The position information representing the focus of the user's line of sight in the screen of the projection device may be converted into position information that needs to be highlighted in the projected device according to the determined display scale. And further determining the first content needing to be highlighted from the projection screen data according to the position information needing to be highlighted in the projected equipment.

S204, highlighting the first content.

In an alternative embodiment, the first content may be highlighted according to the needs of the user. For example, highlighting may be provided in the interface of the projected device for selection by the user, including but not limited to zooming in, highlighting, framing hints, changing font colors, etc.

In another alternative embodiment, the specific implementation of highlighting may also be determined according to the size relationship between the physical dimensions of the screen of the device to be projected and the screen of the device to be projected. Specifically, when the screen physical size of the screen-throwing device is larger than the screen physical size of the device to be thrown, for example, the screen is thrown from a notebook computer to a mobile phone, the screen is thrown from a smart television to a tablet device, the screen is thrown from a commercial large-size screen to the notebook computer, and the like, the first content is highlighted, and the first content is enlarged and displayed in the screen of the device to be thrown. The display effect can be shown by referring to fig. 4, only the content of the gazing area is displayed in the projected device of the small screen through enlarged display, so that the situation that more content is displayed in the smaller screen, and the projected content cannot be seen clearly is avoided. In this embodiment, the screen projection data may be displayed in an enlarged manner according to a preset magnification factor, or the magnification factor may be determined according to a ratio between the resolution of the screen projection device and the user's gaze area, so that the gaze area after the enlarged display is spread over the entire screen of the device to be projected.

When the screen physical size of the screen throwing device is smaller than the screen physical size of the device to be thrown, for example, the screen is thrown from a mobile phone to a notebook computer, the screen is thrown from a tablet device to a smart television, the screen is thrown from the notebook computer to a commercial display large-size screen, and the first content is highlighted, wherein the first content is highlighted in the screen of the device to be thrown in at least one mode selected by highlighting, a cursor and a frame. The display effect can be shown by referring to fig. 5, in fig. 5, the target area with the target coordinate as the center is highlighted in a frame selection manner, so that a listener can quickly locate the key area of the explanation picture, and the user experience is improved.

According to the screen display control method, on the basis of the embodiment, the resolution of the screen display device, the screen display data and the gaze point information are obtained, and further the first content to be highlighted is determined from the screen display data according to the gaze point information, the resolution of the screen display device and the resolution of the device to be highlighted, so that the content watched by the user can be accurately highlighted. And the specific implementation mode for highlighting the first content is provided for the two cases that the screen physical size of the screen throwing equipment is larger than that of the equipment to be thrown and the screen physical size of the screen throwing equipment is smaller than that of the equipment to be thrown, so that the user experience is remarkably improved.

It will be appreciated that the user's operation of the device to be projected is higher in priority than the control of the device to be projected by the point of view coordinates of the device to be projected, so when there is a finger clicking on the screen of the device to be projected, the anchor position of the display area will be recalculated according to the finger clicking position coordinates, and the coordinates of the far-end (the end of the device to be projected) point of view fed back by the communication module are ignored, that is, the finger clicking position coordinates are taken as target coordinates to determine the first content to be highlighted. That is, the method for controlling the screen display according to the present embodiment may further include, based on any of the foregoing embodiments, acquiring the coordinates of the touch point in response to a touch operation performed by the user on the screen of the device to be projected, and updating the coordinates of the touch point to the target coordinates. And determining the first content which needs to be highlighted from the screen projection data according to the target coordinates.

In some application scenarios, the device end of the screen throwing and the device end to be thrown need to perform some interactions. For example, a presenter initiates a voting activity and throws a screen from his notebook computer to a smart phone of a listener in a meeting place, the listener performs a vote by clicking the smart phone, and at this time, the screen throwing device (i.e. the notebook computer) needs to perform a corresponding operation according to the clicking operation of the listener. The screen display control method provided by the embodiment can further comprise the steps of generating a click event according to touch operation, and sending the generated click event to the screen display device so that the screen display device can control the screen display device according to the click event.

The above embodiments are combined to provide another embodiment of the method for controlling a projection display. Referring to fig. 6, fig. 6 is a flowchart of a method for controlling a screen display according to another embodiment of the present invention. As shown in fig. 6, the method for controlling the screen display provided in this embodiment is applied to a device to be thrown, and may specifically include:

s601, acquiring the resolution and the device type of the screen-throwing device.

The screen throwing device and the thrown device can communicate in a wireless or wired mode, for example, the screen throwing device and the thrown device can communicate through Wifi, bluetooth, a mobile network and the like. Before the screen-throwing control is started, related information interaction is carried out between the two devices in the negotiation stage, other devices in the communication range are found by searching the broadcast frames, and related information is acquired from response frames returned by the other devices. In this embodiment, the device to be thrown may send a seeking broadcast frame through a user datagram protocol (User Datagram Protocol, UDP), where the content of the seeking broadcast frame may include information of an own internet protocol (Internet Protocol, IP) Address and a media access Control Address (MEDIA ACCESS Control Address, MAC), and after the device to be thrown listens to obtain the seeking broadcast frame sent by the device to be thrown, send a response frame including own IP, MAC, resolution information and device type information through UDP.

In an alternative implementation, acquiring the resolution and the device type of the screen-throwing device can specifically comprise receiving a response frame of the screen-throwing device to the seeking broadcast frame sent by the screen-throwing device, and acquiring the resolution and the device type of the screen-throwing device from the response frame. It should be noted that, the device type in this embodiment may be, for example, a type identifier for indicating that the device is a mobile phone, a tablet, a notebook, a conference integrated machine, a smart television, a large-size screen with a commercial display, a projector, a digital advertising sign, and the like, and further may further include a physical size of the screen of the device.

S602, determining the display proportion according to the resolution of the device to be thrown and the resolution of the screen throwing device.

After obtaining the resolution of the screen-throwing device, the device to be thrown can determine the display proportion according to the self resolution and the obtained resolution of the screen-throwing device. Taking fig. 3 as an example, in the process of establishing a communication connection, a device to be thrown (receiving end device) X firstly transmits its own information through broadcasting, and after receiving the information of the device X, a screen throwing device (transmitting end device) a transmits its own resolution (Wa, ha) to the device X in a response frame manner. After receiving the resolution of the device a, the device X calculates the resolution with the self resolution (Wx, hx) to obtain a display proportion (wd=wx/Wa, hd=hx/Ha), and the display proportion can be used for subsequent coordinate conversion.

S603, determining a screen throwing type according to the equipment type of the equipment to be thrown and the equipment type of the screen throwing equipment, wherein the screen throwing type is used for representing the size relation between the screen throwing equipment and the screen physical size of the equipment to be thrown.

After receiving the equipment type of the screen throwing equipment, the equipment to be thrown compares the equipment type with the equipment type of the equipment to obtain the screen throwing type. The screen projection types in this embodiment may include a first screen projection type and a second screen projection type, where the first screen projection type is used to indicate that the screen physical size of the screen projection device is larger than the screen physical size of the device to be projected, and the second screen projection type is used to indicate that the screen physical size of the screen projection device is smaller than the screen physical size of the device to be projected.

For example, if the equipment type of the equipment to be thrown indicates that the equipment to be thrown is a mobile phone, and if the equipment type of the equipment to be thrown indicates that the equipment to be thrown is a conference integrated machine, the equipment type of the equipment to be thrown is a first equipment type, and if the equipment type of the equipment to be thrown indicates that the equipment to be thrown is an intelligent television, the equipment type of the equipment to be thrown indicates that the equipment to be thrown is a flat plate, the equipment type to be thrown is a second equipment type.

S604, acquiring screen casting data and gaze point coordinates, wherein the gaze point coordinates represent coordinates of a user sight focus in a screen of the screen casting device.

After the screen is started, the screen is projected on the screen projecting device through the interface of the screen projecting device, at the moment, the interpreter looks at the screen projecting device, different gazing points are formed at different positions of the browsing picture through eyeballs, and in the embodiment, the gazing point coordinates can be obtained by positioning eyeballs based on the eyeball tracking technology and can represent the position of the user sight focus in the screen of the screen projecting device. Specifically, a person always rotates an eyeball while seeing an object, so that the eyeball position of a person exactly provides a good indication, namely, the eyeball position can reflect which part of a scene the person observes. Typically, human eye movement occurs nearly instantaneously and rapidly. And then there is a relatively stationary staring process of 200 ms-600 ms to see the object. The eye is now relatively completely stationary with minor jitter, but during this gaze the jitter is typically less than 1 degree. This also happens only when a person observes a moving object if the eye has a smooth rotation. Other eye movements, such as rotation, dispersion, etc., appear relatively unimportant in the human-machine conversation and can be ignored. Overall, a person sitting in front of a computer operates by using the smooth gaze direction of the eye as input. The eye positioning device and the sight tracking device can be fixedly arranged in front of the computer screen, and when the face of a person is irradiated by an infrared auxiliary light source (LED array), a reflection image is formed on the cornea surface of the eye, and the image is called Purkinje point. As the human eye looks at different positions on the computer screen, the eye can rotate correspondingly. The eye positioning device records an initial position of the eyeball. Assuming that the experimenter's head is stationary and the position of the infrared light emitting diode is fixed, the eyeball is an approximate sphere, so the absolute position of the purkinje point can be considered unchanged when the eyeball rotates; the positions of the iris and the pupil are correspondingly changed, so that the relative position relation between the purkinje point formed by the infrared auxiliary light source on the iris and the pupil and the iris is also changed, the infrared camera can determine the positions of the purkinje point and the pupil in real time, and the determination of the relative position relation can be realized through image processing. The direction of the line of sight input is then derived from the relative positional relationship between them to determine the gaze point on the screen and to determine the gaze point coordinates. And then the screen throwing device sends the screen throwing data and the fixation point coordinates to the thrown device in real time.

And S605, determining target coordinates according to the display scale and the fixation point coordinates.

S606, highlighting the screen throwing data in the screen of the thrown device according to the screen throwing type and the target coordinates.

After the screen-throwing type and the target coordinates are obtained, aiming at different screen-throwing types, the target coordinates are taken as the center, and the gazing area is highlighted in the screen of the thrown equipment. Wherein the gaze area may be determined from gaze point coordinates. Specifically, the region where the human eye can look is a circular region, and a circular region can be determined by determining the coordinates of the point of gaze of the human eye, so that the region of gaze can be determined according to the circular region. Thus, the viewing area may comprise an circumscribed rectangular area of the circular area on the display panel, or other areas, such as triangular areas, etc. And carrying out coordinate conversion by adopting the display proportion, so that a corresponding region of the gazing region in the device to be cast can be obtained, and the region is highlighted. For example, the gazing area is enlarged and displayed when the screen is projected from the large screen device to the small screen device, the gazing area is highlighted when the screen is projected from the small screen device to the large screen device, and the like. Therefore, the display effect is improved, and the user experience is further improved.

The screen projection display control method comprises the steps of obtaining resolution and equipment type of screen projection equipment, determining display proportion according to the resolution and the resolution of the screen projection equipment, determining the screen projection type according to the equipment type and the equipment type of the screen projection equipment, obtaining screen projection data and gaze point coordinates, wherein the gaze point coordinates represent coordinates of a user gaze focus in a screen of the screen projection equipment, determining target coordinates according to the display proportion and the gaze point coordinates, highlighting the screen projection data in the screen of the screen projection equipment according to the screen projection type and the target coordinates, and highlighting the screen projection data in the screen of the screen projection equipment based on the gaze point coordinates, so that the screen projection control method is beneficial to improving display effect and user experience.

On the basis of the above embodiment, the screen projection display control method provided by the application is further described below for different screen projection types. In an alternative implementation manner, when the screen projection type indicates that the screen physical size of the screen projection device is larger than the screen physical size of the projected device, for example, the screen projection is performed from a notebook computer to a mobile phone, the screen projection is performed from a smart television to a tablet device, the screen projection is performed from a commercial large-size screen to a notebook computer, and the like, the screen projection data are highlighted in the screen of the projected device according to the screen projection type and the target coordinates, and specifically, the method can include determining the target coordinates as a center point of display, and amplifying and displaying the screen projection data in the screen of the projected device. The display effect can be shown by referring to fig. 4, only the content of the gazing area is displayed in the projected device of the small screen through enlarged display, so that the situation that more content is displayed in the smaller screen, and the projected content cannot be seen clearly is avoided. In this embodiment, the screen projection data may be displayed in an enlarged manner according to a preset magnification factor, or the magnification factor may be determined according to a ratio between the resolution of the screen projection device and the user's gaze area, so that the gaze area after the enlarged display is spread over the entire screen of the device to be projected.

The following describes a specific implementation procedure by taking an Android system as an example. Through SurfaceComposerClient components of the Android system of the device A, a display device client is created for connecting to SurfaceFlinger, so that a Surface object is obtained, and the Surface object can be used as data rendered by a consumer in a current display interface. The data is subjected to video coding through a MediaCodec module, so that a video stream recorded on a screen can be obtained, and then the video stream is transmitted through a communication module, so that a plurality of devices X can decode and display corresponding system interfaces. One current approach to scaled display of video regions is based on TextureView move and scale operations at the player end. TextureView, incorporated in android4.0 (API level 14), can project content streams directly into View and can be used to implement LIVE PREVIEW, etc. Unlike SurfaceView, it does not create a window alone in the WMS, but rather is a common View in VIEW HIERACHY, and thus can be moved, rotated, scaled, animated, etc. as is the case with other common views. Based on TextureView, the content stream can be displayed, and the characteristics of movement and scaling can be performed, so that the dynamic adjustment of the screen display of the screen throwing area on the mobile terminal equipment can be realized. And starting TextureView at the screen-throwing receiving end, and sending the data decoded by MediaCodec to a Surface of TextureView, so that the display interface content of the screen-throwing sending end can be rendered and displayed.

In the process of establishing a link through the communication module, the receiving end device X firstly transmits its own information through broadcasting, and the transmitting end device a transmits its own resolution (Wa, ha) to the device X in a response frame manner after receiving the information of the device X. Device X, upon receiving the resolution of device a, will calculate with its own resolution (Wx, hx) to derive a ratio (wd=wx/Wa, hd=hx/Ha) for subsequent coordinate conversion use. When the device a generates the gaze point coordinate (Xa, ya) through the eye tracking module, the gaze point coordinate is sent to the device X to calculate (xb=xa×wd, yb=ya×hd) so as to obtain the gaze point coordinate value conforming to the resolution of the device X. When the device X displays the screen of the device a, the default anchor point is the center point of View, and at this time, the anchor point can be set to (Xb, yb) by the methods setPivotX and setPivotY of View, and then scaled by the methods setScaleX and SETSCALEY, where the scaling values Dx and Dy are scaling ratios based on the original size of the View area, and are set to 2 by default, so as to realize the function of amplifying and displaying the local area of the gaze point.

In another alternative embodiment, when the screen projection type indicates that the screen physical size of the screen projection device is smaller than the screen physical size of the projected device, for example, the screen projection is performed from a mobile phone to a notebook computer, the screen projection is performed from a tablet device to a smart television, the screen projection is performed from the notebook computer to a large-size screen with a commercial display, and the projected data is highlighted in the screen of the projected device according to the screen projection type and the target coordinates, specifically, the method can include highlighting a target area centering on the target coordinates in the projected data in at least one mode selected by highlighting, a cursor and a frame in the screen of the projected device. The display effect can be shown by referring to fig. 5, in fig. 5, the target area with the target coordinate as the center is highlighted in a frame selection manner, so that a listener can quickly locate the key area of the explanation picture, and the user experience is improved.

In order to prevent the situation that the gaze point is slightly deviated and the picture is moved to cause discomfort of a viewer, the method for controlling the screen projection display provided by the embodiment can further comprise the steps of determining new target coordinates according to the display proportion and the new gaze point coordinates when the new gaze point coordinates are received, calculating the distance between the new target coordinates and the current target coordinates, and updating the current target coordinates by using the new target coordinates when the distance is larger than a preset threshold value. At the device to be thrown end, the target coordinates are updated only when the distance between the new target coordinates and the current target coordinates is larger than the preset threshold value by setting the threshold for the distance between the new target coordinates and the current target coordinates, so that frequent movement of pictures can be effectively avoided, and the watching experience of users is improved. Of course, in order to further reduce the communication overhead between the screen throwing device and the device to be thrown, a threshold may be set for the distance between the new gaze point coordinate and the current gaze point coordinate at the screen throwing device end, and the new gaze point coordinate may be sent to the screen throwing device end only when the distance is greater than the preset threshold.

It can be understood that the user's operation of the device to be thrown is higher in priority than the control of the device to be thrown by the point of regard coordinates of the screen throwing device, so when a finger clicks the screen of the device to be thrown, the anchor point position of the display area is recalculated according to the position coordinates of the finger clicking, and the coordinates of the far-end (the end of the screen throwing device) point of regard fed back by the communication module are ignored. At this time, the display area adjustment mode is identical to the above scheme, but the anchor point coordinate acquisition mode is different. That is, the method for controlling the screen display according to the present embodiment may further include, based on any of the foregoing embodiments, acquiring the coordinates of the touch point in response to a touch operation performed by the user on the screen of the device to be projected, and updating the coordinates of the touch point to the target coordinates.

In some application scenarios, the device end of the screen throwing and the device end to be thrown need to perform some interactions. For example, a presenter initiates a voting activity and throws a screen from his notebook computer to a smart phone of a listener in a meeting place, the listener performs a vote by clicking the smart phone, and at this time, the screen throwing device (i.e. the notebook computer) needs to perform a corresponding operation according to the clicking operation of the listener. The screen display control method provided by the embodiment can further comprise the steps of generating a click event according to touch operation, and sending the generated click event to the screen display device so that the screen display device can control the screen display device according to the click event. Taking an Android system as an example, the method can be based on a touch event frame and a main type MotionEvent of the Android system, and the method can be used for collecting self clicking events of the equipment to be thrown X and the like and sending the self clicking events to the screen throwing equipment A, so that clicking operation is converted into clicking operation of the equipment A, and remote control is carried out.

The above embodiment illustrates the screen display control method provided by the present application from the perspective of the device to be projected, and the screen display control method provided by the present application will be further illustrated from the perspective of the device to be projected.

Fig. 7 is a flowchart of a screen display control method according to another embodiment of the present invention. As shown in fig. 7, the method for controlling a screen display provided in this embodiment is applied to a screen display device, and may specifically include:

s701, obtaining the gaze point information of the user.

The gaze point information of the user represents the position information of the user's gaze point in the screen of the screen-casting device, for example, the position coordinates of the user's gaze point in the screen of the screen-casting device may be obtained. In the screen projection process, the user looks at the screen projection equipment, different gaze points are formed at different positions of the eye browsing picture, and the gaze point information of the user can be obtained by positioning the eyeballs based on the eye tracking technology.

S702, acquiring first content which needs to be highlighted in a screen projection picture according to the gaze point information of the user.

The gazing area of the user can be determined according to the gazing point information of the user, and the content in the gazing area in the screen throwing picture is determined to be the first content which needs to be highlighted. For example, the content in the area of the preset size in the screen to be projected may be determined as the first content that needs to be highlighted with the position indicated by the gaze point information as the center. The area of the preset size may be a circular area, a rectangular area, a triangular area, etc.

S703, performing highlighting processing on the first content, and sending the first content after highlighting processing to the thrown device.

The first content is highlighted at the screen-throwing device, including but not limited to, enlarging, highlighting, selecting, etc. the first content. The highlighted first content is sent to the thrown device so that the thrown device highlights the first content after the processing. It can be understood that the projection device performs the highlighting processing on the first content, and the projected device does not need to perform the highlighting processing any more, but only needs to display the first content after the highlighting processing. At the moment, the fixation point information is not required to be sent to the device to be thrown any more, and the data transmission quantity is reduced.

The method for controlling the projection display comprises the steps of obtaining gaze point information of a user, obtaining first content which needs to be highlighted in a projection screen according to the gaze point information of the user, highlighting the first content, and sending the first content after highlighting to a device to be projected. The method and the device have the advantages that the first content can be highlighted in the device to be cast, the display effect and the user experience are improved, the processing process is concentrated at the end of the device to be cast, the processing load of the device to be cast can be reduced, and the data transmission quantity is reduced to a certain extent (the gaze point information is not required to be sent to the device to be cast).

Fig. 8 is a schematic structural diagram of a projection display control system according to an embodiment of the present invention. As shown in fig. 8, the screen display control system provided in this embodiment includes a screen display device 801 and at least one device 802 to be displayed, where the screen display device 801 is communicatively connected to the device 802 to be displayed, the screen display device 801 is configured to send a screen display, the device 802 to be displayed is configured to receive the screen display and display the screen display, the screen display device 801 is further configured to obtain gaze point information of a user, the screen display device 801 or the device 802 is further configured to determine, according to the gaze point information, first content that needs to be highlighted in the screen display, and the device 802 to be highlighted is configured to highlight the first content. Highlighting of the first content is achieved in the device to be cast based on the gaze point information, and display effect and user experience are improved.

It should be noted that there may be a plurality of devices to be thrown in the screen display control system. For example, when a presenter initiates a screen-casting from his notebook computer to a smart phone of a listener in a meeting place, the notebook computer is a screen-casting device in the system, and all smart phones receiving screen-casting data in the meeting place are heavy screen-casting devices of the system. The screen projection device 801 in the screen projection display control system may also implement screen projection by using the screen projection display control method applied to the screen projection device provided in any of the above embodiments. The device 802 to be thrown in the system for controlling the screen display may also adopt the method for controlling the screen display applied to the device to be thrown provided in any of the embodiments described above to realize screen throwing.

In an alternative embodiment, when the screen physical size of the screen projection device 801 is larger than the screen physical size of the projected device 802, the projected device 802 is used to enlarge the first content in the screen of the projected device 802. So that the content of interest to the user can be clearly displayed in the smaller screen of the thrown device 802.

In an alternative embodiment, when the screen physical size of the device 801 is smaller than the screen physical size of the device 802 to be projected, the device 802 to be projected is configured to highlight the first content in the screen of the device 802 by at least one of highlighting, cursor and frame selection. The key area of the explanation picture can be quickly positioned in the thrown device 802 of the large screen by the user, so that the user experience is improved.

An embodiment of the present invention further provides an electronic device, and referring to fig. 9, the embodiment of the present invention is illustrated by taking fig. 9 as an example only, and the present invention is not limited thereto. Fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. As shown in fig. 9, the electronic device 90 provided in this embodiment may include a memory 901, a processor 902, and a bus 903. Wherein the bus 903 is used to implement connections between the various components.

The memory 901 stores a computer program, which when executed by the processor 902, may implement the technical solutions of any of the above-described method embodiments.

Wherein the memory 901 and the processor 902 are electrically connected, either directly or indirectly, to enable transmission or interaction of data. For example, the elements may be electrically coupled to each other via one or more communication buses or signal lines, such as via bus 903. The memory 901 stores a computer program for implementing a screen display control method, including at least one software function module which may be stored in the memory 901 in the form of software or firmware, and the processor 902 executes various function applications and data processing by running the software program and the module stored in the memory 901.

The Memory 901 may be, but is not limited to, random access Memory (Random Access Memory, abbreviated as RAM), read Only Memory (ROM), programmable Read Only Memory (Programmable Read-Only Memory, abbreviated as PROM), erasable Read Only Memory (Erasable Programmable Read-Only Memory, abbreviated as EPROM), electrically erasable Read Only Memory (Electric Erasable Programmable Read-Only Memory, abbreviated as EEPROM), etc. The memory 901 is used for storing a program, and the processor 902 executes the program after receiving an execution instruction. Further, the software programs and modules within the memory 901 may also include an operating system, which may include various software components and/or drivers for managing system tasks (e.g., memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components.

The processor 902 may be an integrated circuit chip having signal processing capabilities. The processor 902 may be a general-purpose processor, including a central processing unit (Central Processing Unit, abbreviated as CPU), a network processor (Network Processor, abbreviated as NP), and the like. The disclosed methods, steps, and logic blocks in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. It will be appreciated that the configuration of fig. 9 is merely illustrative and may include more or fewer components than shown in fig. 9 or have a different configuration than shown in fig. 9. The components shown in fig. 9 may be implemented in hardware and/or software. It should be noted that, the electronic device provided in this embodiment may be any electronic device capable of displaying a screen, including, but not limited to, a mobile phone, a tablet, a computer, a television, a large-size screen with a commercial display, or the like, and may be a screen-throwing device, a device to be thrown, or may be a screen-throwing device in some periods, or a device to be thrown in other periods. For example, when content is projected onto a television from a mobile phone, the mobile phone is a projection device, and when the content is projected onto the mobile phone from a computer, the mobile phone becomes a projected device.

The embodiment of the invention also provides a computer readable storage medium, on which a computer program is stored, the computer program being executed by a processor to implement the technical solution of any of the method embodiments described above.

The various embodiments in this disclosure are described in a progressive manner, and identical and similar parts of the various embodiments are all referred to each other, and each embodiment is mainly described as different from other embodiments.

The scope of the present disclosure is not limited to the above-described embodiments, and it is apparent that various modifications and variations can be made to the present disclosure by those skilled in the art without departing from the scope and spirit of the disclosure. Such modifications and variations are intended to be included herein within the scope of the following claims and their equivalents.

Claims

1. The screen projection display control system is characterized by comprising screen projection equipment and at least one piece of equipment to be projected, wherein the screen projection equipment is in communication connection with the equipment to be projected;

The screen throwing equipment is used for sending screen throwing pictures;

the device to be thrown is used for receiving and displaying a screen throwing picture;

The screen throwing equipment or the equipment to be thrown is also used for determining first content which needs to be highlighted in a screen throwing picture according to the gaze point information;

the posted device is to highlight the first content.

2. The system of claim 1, wherein the projected device is configured to highlight the first content when the screen physical size of the projected device is greater than the screen physical size of the projected device, comprising the projected device being configured to zoom in on the first content in the screen of the projected device.

3. The system of claim 1, wherein the projected device being configured to highlight the first content when the screen physical size of the projected device is less than the screen physical size of the projected device comprises the projected device being configured to highlight the first content in the screen of the projected device by at least one of highlighting, cursors, and boxes.

4. The screen projection display control method is applied to the equipment to be projected and is characterized by comprising the following steps:

Acquiring screen casting data and gaze point information, wherein the gaze point information is used for representing the position information of a user's sight focus in a screen of screen casting equipment;

5. The method of claim 4, wherein prior to the acquiring the gaze data and gaze point information, the method further comprises acquiring a resolution of the gaze device;

the highlighting the screen projection data according to the gaze point information comprises:

Determining first content to be highlighted from the screen casting data according to the gaze point information, the resolution of the screen casting device and the resolution of the cast device;

highlighting the first content.

6. The method of claim 5, wherein the highlighting the first content when the screen physical size of the screen-casting device is greater than the screen physical size of the projected device comprises:

and amplifying and displaying the first content in a screen of the thrown device.

7. The method of claim 5, wherein the highlighting the first content when the screen physical size of the screen-casting device is smaller than the screen physical size of the projected device comprises:

Highlighting the first content in a screen of the thrown device by at least one of highlighting, cursor and frame selection.

8. The method according to claim 4, wherein the method further comprises:

responding to touch operation of a user on the screen of the thrown device, and acquiring touch point coordinates;

And updating the touch point coordinates to target coordinates.

9. The method of claim 8, wherein the method further comprises:

Generating a click event according to the touch operation;

And sending the generated click event to a screen throwing device so that the screen throwing device controls the screen throwing device according to the click event.

10. The method of any of claims 5-9, wherein the obtaining the resolution of the projection device comprises:

and obtaining the resolution of the screen throwing equipment from the response frame.

11. The screen projection display control method is applied to screen projection equipment and is characterized by comprising the following steps of:

acquiring the gaze point information of a user;

12. An electronic device is characterized by comprising a processor and a memory;

the memory stores computer-executable instructions;

the processor executing computer-executable instructions stored in the memory, causing the processor to perform the method of controlling a projection display as claimed in any one of claims 4 to 11.

13. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are for implementing a method of controlling a drop display as claimed in any one of claims 4 to 11.