CN114244884A

CN114244884A - Eyeball tracking-based video coding method applied to cloud game

Info

Publication number: CN114244884A
Application number: CN202111576190.8A
Authority: CN
Inventors: 雷小刚; 李瑞亮; 郭建君
Original assignee: Beijing Weiling Times Technology Co Ltd
Current assignee: Beijing Weiling Times Technology Co Ltd
Priority date: 2021-12-21
Filing date: 2021-12-21
Publication date: 2022-03-25
Anticipated expiration: 2041-12-21
Also published as: CN114244884B

Abstract

Embodiments of the present disclosure provide an eye tracking-based video encoding method, apparatus, device, and computer-readable storage medium applied to a cloud game. The method comprises the steps of collecting human eye images of a user watching a game video; analyzing the human eye image to determine the human eye visual direction; calculating the mass center of the pupil outline based on the human eye visual direction, and determining the image position focused on the game picture; taking the centroid as an image center, and performing area division on a game picture based on a preset radius range; and according to different regions, matching different coding strategies to complete coding. In this way, the picture quality of the user during game is effectively improved, and the user experience is enhanced.

Description

Eyeball tracking-based video coding method applied to cloud game

Technical Field

Embodiments of the present disclosure relate generally to the field of image input, and more particularly, to an eye tracking-based video encoding method, apparatus, device, and computer-readable storage medium applied to a cloud game.

Background

In the cloud game experience, the quality of a picture is reduced due to encoding of a transmitted video picture video encoder, and no strategy is provided for improving the image quality of a visual focus of a user while the image is reduced.

Meanwhile, the existing eyeball tracking system identifies the position of the pupil by acquiring a frame of image, but in the process of actually acquiring the eye image, when the eyelid of a user is drooped, or the user looks down, or when the user wears glasses, the pupil can be shielded in the eye image acquired at some angles, so that the center position of the pupil cannot be identified, and the efficiency and the accuracy of the eyeball tracking system are influenced. Further, the focus of the user on the screen is affected, and the position of the image focused on the game picture by the user cannot be accurately positioned.

Disclosure of Invention

According to an embodiment of the present disclosure, there is provided an eye tracking-based video encoding scheme applied to a cloud game.

In a first aspect of the present disclosure, an eye tracking-based video encoding method applied to a cloud game is provided. The method comprises the following steps:

collecting human eye images of a user watching a game video;

analyzing the human eye image to determine the human eye visual direction;

calculating the mass center of the pupil outline based on the human eye visual direction, and determining the image position focused on the game picture; taking the centroid as an image center, and performing area division on a game picture based on a preset radius range; and according to different regions, matching different coding strategies to complete coding.

Further, the analyzing the human eye image and the determining the human eye vision direction includes:

analyzing the human eye image based on an eye tracking sensor;

if the pupil area in the human eye image is blocked, correcting the human eye image by a preset method to obtain a second human eye image; and analyzing the second human eye image to determine the human eye visual direction.

Further, the modifying the eye image by a preset method to obtain a second eye image includes:

matching the human eye image with a preset human eye image library to obtain a plurality of matching values; the human eye image library comprises a plurality of human eye image templates;

and sorting the matching values, selecting a template with the highest matching value to correct the human eye image, and obtaining a second human eye image.

Further, the selecting the template with the highest matching value to correct the eye image to obtain a second eye image includes:

determining an occluded area and a template pupil area corresponding to the occluded area based on the template with the highest matching value and the human eye image;

and correcting the eye image of the person through the template pupil area corresponding to the shielding area to obtain a second eye image.

Further, the calculating the centroid of the pupil profile comprises:

the centroid of the pupil profile is calculated by Open CV.

Further, the matching different coding strategies according to the different regions includes:

if the game picture area is within the preset radius range, the image quality of the corresponding area is improved by adopting a high-coding strategy;

and if the game picture area is outside the preset radius range, reducing the image quality of the corresponding area by adopting a low-coding strategy.

In a second aspect of the present disclosure, an eye tracking-based video encoding apparatus applied to a cloud game is provided. The device includes:

the acquisition module is used for acquiring human eye images of a user watching the game video;

the determining module is used for analyzing the human eye image and determining the human eye visual direction;

the coding module is used for calculating the mass center of the pupil outline based on the human eye visual direction and determining the image position focused on the game picture; taking the centroid as an image center, and performing area division on a game picture based on a preset radius range; and according to different regions, matching different coding strategies to complete coding.

analyzing the human eye image based on an eye tracking sensor;

In a third aspect of the disclosure, an electronic device is provided. The electronic device includes: a memory having a computer program stored thereon and a processor implementing the method as described above when executing the program.

In a fourth aspect of the present disclosure, a computer readable storage medium is provided, having stored thereon a computer program, which when executed by a processor, implements a method as in accordance with the first aspect of the present disclosure.

According to the video coding method based on eyeball tracking applied to the cloud game, the human eye images of the game video watched by the user are collected; analyzing the human eye image to determine the human eye visual direction; calculating the mass center of the pupil outline based on the human eye visual direction, and determining the image position focused on the game picture; taking the centroid as an image center, and performing area division on a game picture based on a preset radius range; according to different regions, different coding strategies are matched to complete coding, so that the picture quality of a user during game is effectively improved, and the user experience is enhanced.

It should be understood that the statements herein reciting aspects are not intended to limit the critical or essential features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, like or similar reference characters designate like or similar elements, and wherein:

FIG. 1 illustrates a schematic diagram of an exemplary operating environment in which embodiments of the present disclosure can be implemented;

fig. 2 illustrates a flowchart of an eye tracking based video encoding method applied to a cloud game according to an embodiment of the present disclosure;

FIG. 3 illustrates an optimization effect graph according to an embodiment of the disclosure;

fig. 4 illustrates a block diagram of an eye-tracking based video encoding apparatus applied to a cloud game according to an embodiment of the present disclosure;

FIG. 5 illustrates a block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure.

Detailed Description

To make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are some, but not all embodiments of the present disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.

In addition, the term "and/or" herein is only one kind of association relationship describing an associated object, and means that there may be three kinds of relationships, for example, a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.

FIG. 1 illustrates a schematic diagram of an exemplary operating environment 100 in which embodiments of the present disclosure can be implemented. Included in the operating environment 100 are storage and processing circuitry 110, and a sensor 170 coupled to the storage and processing circuitry 110.

The electronic device 100 may include control circuitry, which may include storage and processing circuitry 110. The storage and processing circuitry 110 may include memory, such as hard drive memory, non-volatile memory (e.g., flash memory or other electronically programmable read-only memory used to form a solid state drive, etc.), volatile memory (e.g., static or dynamic random access memory, etc.), and so on, and embodiments of the present application are not limited thereto. Processing circuitry in storage and processing circuitry 110 may be used to control the operation of electronic device 100. The processing circuitry may be implemented based on one or more microprocessors, microcontrollers, digital signal processors, baseband processors, power management units, audio codec chips, application specific integrated circuits, display driver integrated circuits, and the like.

The storage and processing circuitry 110 may be used to run software in the electronic device 100, such as an Internet browsing application, a Voice Over Internet Protocol (VOIP) telephone call application, an email application, a media playing application, operating system functions, and so forth. Such software may be used to perform control operations such as, for example, camera-based image capture, ambient light measurement based on an ambient light sensor, proximity sensor measurement based on a proximity sensor, information display functionality based on status indicators such as status indicator lights of light emitting diodes, touch event detection based on a touch sensor, functionality associated with displaying information on multiple (e.g., layered) display screens, operations associated with performing wireless communication functionality, operations associated with collecting and generating audio signals, control operations associated with collecting and processing button press event data, and other functions in the electronic device 100, to name a few.

The electronic device 100 may include input-output circuitry 150. The input-output circuit 150 may be used to enable the electronic device 100 to input and output data, i.e., to allow the electronic device 100 to receive data from an external device and also to allow the electronic device 100 to output data from the electronic device 100 to the external device. The input-output circuit 150 may further include a sensor 170. Sensor 170 may include the ultrasonic fingerprint identification module, may also include ambient light sensor, proximity sensor based on light and electric capacity, touch sensor (for example, based on light touch sensor and/or capacitanc touch sensor, wherein, touch sensor may be a part of touch display screen, also can regard as a touch sensor structure independent utility), acceleration sensor, and other sensors etc., the ultrasonic fingerprint identification module can be integrated in the screen below, or, the ultrasonic fingerprint identification module can set up in electronic equipment's side or back, do not do the restriction here, this ultrasonic fingerprint identification module can be used to gather the fingerprint image.

The sensor 170 may include an Infrared (IR) camera or an RGB camera, and when the IR camera takes a picture, the pupil reflects infrared light, so the IR camera takes a pupil image more accurately than the RGB camera; the RGB camera needs to carry out more follow-up pupil detection, and calculation accuracy and accuracy are higher than the IR camera, and the commonality is better than the IR camera, but the calculated amount is big.

Input-output circuit 150 may also include one or more display screens, such as display screen 130. The display 130 may include one or a combination of liquid crystal display, organic light emitting diode display, electronic ink display, plasma display, display using other display technologies. The display screen 130 may include an array of touch sensors (i.e., the display screen 130 may be a touch display screen). The touch sensor may be a capacitive touch sensor formed by a transparent touch sensor electrode (e.g., an Indium Tin Oxide (ITO) electrode) array, or may be a touch sensor formed using other touch technologies, such as acoustic wave touch, pressure sensitive touch, resistive touch, optical touch, and the like, and the embodiments of the present application are not limited thereto.

The electronic device 100 may also include an audio component 140. The audio component 140 may be used to provide audio input and output functionality for the electronic device 100. The audio components 140 in the electronic device 100 may include a speaker, a microphone, a buzzer, a tone generator, and other components for generating and detecting sound.

The communication circuit 120 may be used to provide the electronic device 100 with the capability to communicate with external devices. The communication circuit 120 may include analog and digital input-output interface circuits, and wireless communication circuits based on radio frequency signals and/or optical signals. The wireless communication circuitry in communication circuitry 120 may include radio-frequency transceiver circuitry, power amplifier circuitry, low noise amplifiers, switches, filters, and antennas. For example, the wireless Communication circuitry in Communication circuitry 120 may include circuitry to support Near Field Communication (NFC) by transmitting and receiving Near Field coupled electromagnetic signals. For example, the communication circuit 120 may include a near field communication antenna and a near field communication transceiver. The communications circuitry 120 may also include a cellular telephone transceiver and antenna, a wireless local area network transceiver circuitry and antenna, and so forth.

The electronic device 100 may further include a battery, power management circuitry, and other input-output units 160. The input-output unit 160 may include buttons, joysticks, click wheels, scroll wheels, touch pads, keypads, keyboards, cameras, light emitting diodes and other status indicators, and the like.

A user may input commands through input-output circuitry 150 to control the operation of electronic device 100, and may use output data of input-output circuitry 150 to enable receipt of status information and other outputs from electronic device 100.

Fig. 2 shows a flowchart of an eye tracking-based video encoding method applied to a cloud game according to an embodiment of the present disclosure, including:

s210, collecting human eye images of a user watching the game video.

In some embodiments, when the user conducts the cloud game, the configured application program is opened, and the eye image of the user watching the game video, namely the eye image of the player during the game, is collected through the camera on the electronic device.

S220, analyzing the human eye image and determining the human eye visual direction.

In some embodiments, the human eye image is analyzed based on an eye tracking sensor (configured on an electronic device);

if the pupil area in the human eye image is blocked, matching the human eye image with a preset human eye image library to obtain a plurality of matching values; the human eye image library comprises a plurality of human eye image templates;

sorting the matching values, and selecting a template with the highest matching value to correct the human eye image; that is to say that the first and second electrodes,

correcting the eye image through the template pupil area corresponding to the shielding area to obtain a second eye image; namely, pixel points in the pupil area of the template are supplemented into the human eye image;

further, the second eye image is analyzed through a pupil identification technology, and the visual direction of the human eyes is determined.

S230, calculating the mass center of the pupil outline based on the human eye visual direction, and determining the image position focused on the game picture; taking the centroid as an image center, and performing area division on a game picture based on a preset radius range; and according to different regions, matching different coding strategies to complete coding.

In some embodiments, the centroid of the pupil profile may be obtained using a cross-platform computer vision and machine learning software library (Open CV) running on operating systems such as Linux, Windows, Android, and Mac OS.

Specifically, find image Contours in Open CV using find contacts; obtaining multi-order moments of the polygon according to the outline information of the polygon; all moments within 3 orders of the polygon and the grating shape are obtained by using classes, wherein variables within the classes are 0 order moment (m00), 1 order moment (m10, m01), 2 order moment (m20, m11, m02) and 3 order moment (m30, m21, m12, m03), and the centroid of the polygon is x-m 10/m00, and y-m 01/m 00.

In some embodiments, the position of the image focused on the game picture (the intersection of the plane in which the game picture is located and the direction of the visual straight line) is determined according to the centroid; taking the centroid as an image center, and performing area division on the game picture based on a preset radius range, namely dividing the game picture into an image area (a core area) smaller than or equal to the preset radius range and an image area (a far area) larger than the preset radius range; the radius range may be preset according to information such as a type of the cloud game, a game scene, and/or a user age.

For the far-away area (image far away from the focus), appropriate blurring, contrast reduction and other processing (based on hardware conditions of the device) are performed to improve the compression ratio of the far-away area in video coding, that is, more space is reserved for the area in focus (core area) under the condition of the same compression ratio of the whole image. In the video coding process of the image, the more the details of the image are reserved, the larger the space occupied by coding is, the image quality is reduced, and the data size after coding can be effectively reduced. Correspondingly, a high-coding strategy is adopted for the core area image based on the hardware condition of the electronic device, so that the corresponding image display effect is improved, and reference is made to fig. 3.

According to the embodiment of the disclosure, the following technical effects are achieved:

under the same bandwidth, the visual image quality of the user in the cloud game is improved.

It is noted that while for simplicity of explanation, the foregoing method embodiments have been described as a series of acts or combination of acts, it will be appreciated by those skilled in the art that the present disclosure is not limited by the order of acts, as some steps may, in accordance with the present disclosure, occur in other orders and concurrently. Further, those skilled in the art should also appreciate that the embodiments described in the specification are exemplary embodiments and that acts and modules referred to are not necessarily required by the disclosure.

The above is a description of embodiments of the method, and the embodiments of the apparatus are further described below.

Fig. 4 illustrates a block diagram of an eye-tracking based video encoding apparatus 400 applied to a cloud game according to an embodiment of the present disclosure. As shown in fig. 4, the apparatus 400 includes:

the acquisition module 410 is used for acquiring human eye images of a user watching a game video;

a determining module 420, configured to analyze the human eye image and determine a human eye visual direction;

the encoding module 430 is configured to calculate a centroid of a pupil contour based on the human eye visual direction, and determine an image position focused on a game picture; taking the centroid as an image center, and performing area division on a game picture based on a preset radius range; according to different regions, different coding strategies are matched to complete coding

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the described module may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.

FIG. 5 shows a schematic block diagram of an electronic device 500 that may be used to implement embodiments of the present disclosure. As shown, device 500 includes a Central Processing Unit (CPU)501 that may perform various appropriate actions and processes in accordance with computer program instructions stored in a Read Only Memory (ROM)502 or loaded from a storage unit 508 into a Random Access Memory (RAM) 503. In the RAM503, various programs and data required for the operation of the device 500 can also be stored. The CPU 501, ROM 502, and RAM503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.

A number of components in the device 500 are connected to the I/O interface 505, including: an input unit 506 such as a keyboard, a mouse, or the like; an output unit 507 such as various types of displays, speakers, and the like; a storage unit 508, such as a magnetic disk, optical disk, or the like; and a communication unit 509 such as a network card, modem, wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.

The processing unit 501 performs the various methods and processes described above. For example, in some embodiments, the methods may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 500 via the ROM 502 and/or the communication unit 509. When the computer program is loaded into RAM503 and executed by CPU 501, one or more steps of the method described above may be performed. Alternatively, in other embodiments, CPU 501 may be configured to perform the method by any other suitable means (e.g., by way of firmware).

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a load programmable logic device (CPLD), and the like.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

Further, while operations are depicted in a particular order, this should be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims, and the scope of the invention is not limited thereto, as modifications and substitutions may be readily made by those skilled in the art without departing from the spirit and scope of the invention as disclosed herein.

Claims

1. A video coding method based on eyeball tracking applied to a cloud game is characterized by comprising the following steps:

collecting human eye images of a user watching a game video;

analyzing the human eye image to determine the human eye visual direction;

2. The method of claim 1, wherein analyzing the image of the human eye to determine the visual orientation of the human eye comprises:

analyzing the human eye image based on an eye tracking sensor;

3. The method according to claim 2, wherein the modifying the eye image by a preset method to obtain a second eye image comprises:

4. The method of claim 3, wherein the selecting the template with the highest matching value to modify the eye image to obtain the second eye image comprises:

5. The method of claim 4, wherein the calculating the centroid of the pupil profile comprises:

the centroid of the pupil profile is calculated by Open CV.

6. The method of claim 5, wherein matching different coding strategies according to different regions comprises:

7. An eye tracking-based video encoding apparatus applied to a cloud game, comprising:

8. The apparatus of claim 7, wherein analyzing the image of the human eye to determine the visual direction of the human eye comprises:

analyzing the human eye image based on an eye tracking sensor;

9. An electronic device comprising a memory and a processor, the memory having stored thereon a computer program, wherein the processor, when executing the program, implements the method of any of claims 1-6.

10. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, carries out the method of any one of claims 1 to 6.