WO2020015468A1

WO2020015468A1 - Image transmission method and apparatus, terminal device, and storage medium

Info

Publication number: WO2020015468A1
Application number: PCT/CN2019/089353
Authority: WO
Inventors: 付阳; 王云飞; 黄通兵
Original assignee: 北京七鑫易维信息技术有限公司
Priority date: 2018-07-16
Filing date: 2019-05-30
Publication date: 2020-01-23
Also published as: CN108919958A; CN108919958B

Abstract

Disclosed in the present invention are an image transmission method and apparatus, a terminal device, and a storage medium. Said method comprises: acquiring an eye image when a user gazes at a target image; determining a gaze point of the user on the basis of the line of sight corresponding to the eye image; acquiring a pupil radius when the user gazes at the gaze point; on the basis of the user's gaze point, the degree of attention corresponding to the pupil radius, and the pixel values of pixel points in the target image, performing area division on the target image, and assigning priorities for the divided areas, so as to obtain data to be transmitted; and transmitting said data.

Description

Image transmission method, device, terminal equipment and storage medium

Technical field

Embodiments of the present invention relate to the field of transmission technologies, and in particular, to an image transmission method, device, terminal device, and storage medium.

Background technique

The display device is a device capable of outputting an image or a touch signal (for example, a braille display designed for the blind). After receiving the image data of an external signal source (such as a computer), the display device displays the corresponding image content in real time. In the process of a computer transmitting an image to a display device, the original image is usually directly transmitted. Therefore, the amount of image data to be transmitted in the process of transmitting images is large, and the speed of transmitting images is slow.

At present, the image transmission method proposed to solve the problem of a large number of image transmission processes generally compresses the entire image data, which results in poor definition of the image content displayed by the display device and affects the user experience of viewing the image.

Summary of the invention

An image transmission method, device, terminal device and storage medium provided by the present invention can effectively improve image compression efficiency.

In a first aspect, an embodiment of the present invention provides an image transmission method, including:

Obtaining an eye image when a user looks at a target image;

Determining a gaze point of the user based on a line of sight corresponding to the eye image;

Obtaining a pupil radius at which the user looks at the fixation point;

Divide the target image into regions based on the user's fixation point, the degree of attention corresponding to the pupil radius, and the pixel value of each pixel in the target image, and assign priorities to the divided regions to obtain transfer data;

Transmitting the data to be transmitted.

In a second aspect, an embodiment of the present invention further provides an image transmission device, including:

An eye image acquisition module configured to acquire an eye image when a user looks at a target image;

A fixation point determining module, configured to determine a fixation point of a user based on a line of sight corresponding to the eye image;

A pupil radius determination module, configured to obtain a pupil radius at which the user looks at the fixation point;

The data to be transmitted determination module is configured to divide the target image into regions based on the user's fixation point, the degree of attention corresponding to the pupil radius, and the pixel value of each pixel in the target image, The priority of the assigned area to get the data to be transmitted;

A transmission module configured to transmit the data to be transmitted.

According to a third aspect, an embodiment of the present invention further provides a terminal device, including:

One or more processors;

A storage device configured to store one or more programs;

The one or more programs are executed by the one or more processors, so that the one or more processors implement the image transmission method provided by the embodiment of the present invention.

According to a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the image transmission method provided by the embodiment of the present invention is implemented.

Embodiments of the present invention provide an image transmission method, device, terminal device, and storage medium. First, an eye image of a user when gazing at a target image is acquired; secondly, a gaze point of the user is determined based on a line of sight corresponding to the eye image; Then, the pupil radius at which the user fixes the fixation point is obtained; and then the target image is area-based based on the user's fixation point, the degree of attention corresponding to the pupil radius, and the pixel value of each pixel in the target image. Dividing and assigning priorities to each divided area to obtain data to be transmitted; and finally transmitting the data to be transmitted. With the above technical solution, before the terminal device transmits the target image, the fixation point determined by the line of sight corresponding to the eye image when the user fixes the target image, the degree of attention corresponding to the pupil radius when the user fixes the fixation point, and the target image The pixel value of each pixel points divides the target image, and then transmits it with different priorities (different priorities may correspond to different code rates or different rendering accuracy). Compared with directly transmitting the target image, it effectively reduces the amount of data to be transmitted, thereby reducing the time required to transmit the target image, and improving the efficiency of image transmission; compared with compressing the target image as a whole, it effectively improves It improves the sharpness of the image of the user's area of interest and improves the compression effect of the target image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1a is a schematic flowchart of an image transmission method according to Embodiment 1 of the present invention; FIG.

FIG. 1b is a schematic diagram of an eye image provided by Embodiment 1 of the present invention; FIG.

2a is a schematic flowchart of an image transmission method according to a second embodiment of the present invention;

FIG. 2b is a schematic diagram of an application scenario of an image transmission method provided in Embodiment 2 of the present invention; FIG.

FIG. 2c is a schematic diagram after assigning priorities to the divided target images according to the second embodiment of the present invention; FIG.

FIG. 3 is a schematic structural diagram of an image transmission device according to a third embodiment of the present invention; FIG.

FIG. 4 is a schematic structural diagram of a terminal device according to a fourth embodiment of the present invention.

detailed description

The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. It should also be noted that, for the convenience of description, only some parts related to the present invention are shown in the drawings instead of all the structures.

Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although flowcharts describe operations (or steps) as sequential processing, many of these operations can be performed in parallel, concurrently, or simultaneously. In addition, the order of operations can be rearranged. The process may be terminated when its operation is completed, but may also have additional steps not included in the drawings. The processing may correspond to methods, functions, procedures, subroutines, subroutines, and so on.

Example one

FIG. 1a is a schematic flowchart of an image transmission method according to Embodiment 1 of the present invention. The method is applicable to a case where a target image is transmitted between different image transmission devices (such as a terminal device and a display device or between different terminal devices). This method may be performed by an image transmission apparatus provided in an embodiment of the present invention, where the apparatus may be implemented by software and / or hardware, and is generally integrated on a terminal device. In this embodiment, the terminal device includes, but is not limited to, a computer, a mobile phone, or a handheld computer.

As shown in FIG. 1a, an image transmission method provided in Embodiment 1 of the present invention includes the following steps:

S101. Obtain an eye image when a user looks at a target image.

In this embodiment, the image transmission method may be applied to a terminal device. The target image can be understood as an image to be processed in the terminal device.

It can be understood that in this step, the user can directly stare at the target image in the terminal device, and then process the target image based on the eye image when the user looks at the target image, and send the processed data to the display device or other terminal device. Generally, in this step, an eye image when a user looks at a target image can be obtained through an image acquisition device provided on a terminal device.

In addition, when acquiring the eye image when the user looks at the target image, the user can also wear an AR device, and an image acquisition device is provided on the AR device to collect the eye image when the user looks at the target image. This step can be achieved by The communication connection established by the AR device acquires the eye image when the user looks at the target image; if the user wears a VR device, the terminal device can be built into the VR device, the target image is displayed by the terminal device, and the user's gaze target image is obtained by the image acquisition device Eye image of the time, wherein the image acquisition device can be set on the VR device. The terminal device may acquire an eye image when the user looks at the target image through a communication connection established with the VR device.

Specifically, in this step, an image of the eye of the user when the user looks at the target image may be obtained through an image acquisition device on the terminal device. The image acquisition device may be a common camera or an infrared camera. When the image acquisition device in this embodiment is an infrared camera, the infrared camera needs to be configured with an infrared lamp to be set to manufacture a light spot in an eye diagram, so that the user's sight can be determined in combination with the characteristics of the pupil of the user.

S102. Determine a gaze point of the user based on a line of sight corresponding to the eye image.

In this embodiment, the fixation point can be understood as the position where the user fixates in the target image. After obtaining the eye image when the user looks at the target image, in this step, the eye image can be identified through the eye recognition algorithm to determine the line of sight corresponding to the eye image, and then the user's line of sight can be determined based on the determined line of sight. The coordinates in the image to get the user's fixation point; or the fixation point of the user can be determined through a pre-built fixation point model.

Specifically, when an ordinary camera is used in this embodiment to obtain an eye image when the user looks at the target image, this step may determine the direction of the user's line of sight based on the pupil characteristics in the eye image. When an infrared camera is used in this embodiment to obtain an eye image when a user looks at a target image, in this step, pupil characteristics in the eye image and spot information presented by the infrared lamp in the eye image may be acquired, and then according to the acquired The pupil feature and spot information are determined by the corneal reflection method to determine the direction of the user's line of sight in the eye image.

When using an infrared camera to determine the direction of the user's line of sight, the main hardware requirements include, but are not limited to: a light source: generally an infrared light source, because infrared light will not affect the vision of the eye; and multiple infrared light sources can be arranged in a predetermined manner , Such as a character shape, a shape, etc .; image acquisition device: such as infrared camera equipment, infrared image sensor, camera or video camera.

The process of determining the user's line of sight includes:

First obtain an eye image: the light source shines on the user's eye, and the user's eye is photographed by the image acquisition device, and the corresponding reflection point on the cornea, that is, the light spot (also known as Purkin's spot) is captured, thereby obtaining the Eye image. Then perform gaze / fixation point estimation: as the eyeball rotates, the relative positional relationship between the pupil center and the light spot changes, and several eye images with light spots collected correspondingly reflect this position change relationship; according to the description The position change relationship is estimated by the line of sight / fixation point.

In general, eye tracking can also be called gaze tracking, which is a technique for estimating the eye's sight and / or gaze point by measuring the movement of the user's eyes. Optical recording method is currently widely used: using an image acquisition device (such as a camera or video camera) to record the subject's eye movements, that is, to obtain an eye image that reflects the eye movement, and to extract an eye from the acquired eye image Features are used to build a gaze / gaze point estimation model. The eye features may include: pupil position, pupil shape, iris position, iris shape, eyelid position, eye corner position, and / or light spot (also referred to as Purchin spot) position.

It can be understood that when obtaining the direction of the user's line of sight, a contact / non-contact sensor (such as an electrode or a capacitance sensor) can also be used to estimate the eye movement.

FIG. 1b shows a schematic diagram of an eye image provided in Embodiment 1 of the present invention. FIG. 1b shows one eye, which is not a limitation on the number of eyes. The eye image obtained in this embodiment may also include two eyes. Eyes only to improve the accuracy of the target image processing. Specifically, in this embodiment, an eye feature (feature information) when the user looks at the target image may be determined based on the obtained eye image, so as to determine a gaze point corresponding to the user's line of sight direction. As shown in FIG. 1b, in this embodiment, the direction of the user's line of sight can be obtained by analyzing the pupil 11 or the iris 12 in the eye image. In addition, when an infrared camera is used to collect the eye image, the spot information contained in the eye image can also be acquired to assist in obtaining the direction of the user's line of sight.

S103. Obtain a pupil radius at which the user looks at the gaze point.

After the user's fixation point is determined, in this step, the pupil radius at which the user fixes the fixation point can be obtained.

Generally, both the positive and negative movements of the eye can be obtained by analyzing the pupil radius, so that the pupil radius can be analyzed to determine the positive and negative movements of the eye, so that the user's degree of attention to the current object of interest can be obtained. When the user is interested in the content being watched, it will have a positive feeling, the corresponding pupil will be dilated, and the pupil radius will become larger; when the user is not interested in the content being watched, it will have a negative feeling, and the corresponding pupil will be Shrinks and pupil radius decreases. Therefore, in this embodiment, the corresponding degree of attention can be determined by the pupil radius when the user pays attention to the target image, so as to obtain different processing methods for the target image.

It can be understood that, in this step, an eye image obtained for judging a user's fixation point may be analyzed to determine a pupil radius at which the user fixes the fixation point; or when the user fixes the fixation point, the pupil radius of the fixation point is obtained. Specifically, in this step, edge information of a pupil in an eye image may be extracted, and then a corresponding pupil radius may be determined based on the edge information of the pupil.

S104. Divide the target image into regions based on the user's gaze point, the degree of attention corresponding to the pupil radius, and the pixel value of each pixel in the target image, and assign priorities to the divided regions. Get the data to be transmitted.

In this embodiment, the degree of attention can be understood as the degree of interest of the user in the viewing object determined based on the characteristics of the user's eyes (such as the pupil radius). The image to be transmitted can be understood as data to be transmitted based on the divided target image.

After determining the pupil radius of the gaze point at which the user looks at the target image, this step may determine the corresponding degree of attention based on the pupil radius and a preset pupil data table. Then combine the user's gaze point with the pixel value of each pixel of the target image to divide the target image into regions.

It should be noted that in this embodiment, the degree of attention may also be determined in combination with the remaining eye features in the eye image. For example, to obtain the pupil features (such as pupil radius and pupil position) and spot features of the gaze point of the user, determine the fixation information based on the pupil position and spot features (fixation information includes the duration of fixation, the number of fixations, and / or the first fixation time), and The information and pupil radius determine the degree of attention.

It should be noted that the preset pupil data table may be a general-purpose data table obtained through training; it may also be a dedicated data table obtained by training for different users (the data tables corresponding to different users are different). If it is a special data table, after obtaining the eye image of the user when gazing at the target image, the iris features in the eye image can also be extracted to identify the user, thereby determining the user's gaze point The corresponding degree of attention can be determined after the pupil radius of.

In addition, when the target image is divided into regions based on the attention point, the degree of attention corresponding to the pupil radius, and the pixel value of each pixel of the target image, and priority is assigned to the divided regions, this step may first be based on each pixel of the user's target image The pixel values of the points perform edge segmentation on the target image, and then select each closed area and the area containing the gaze point. Finally, the divided areas are assigned priorities based on the user's attention to the gaze point. This can effectively re-segment the target image based on the user's gaze point and attention and assign priorities.

When the gaze point of the user's gaze is not the content the user wants to pay attention to (at this time, the user's attention to the gaze point is low), the priority of the area containing the point of interest can be lowered, and the priority of the remaining areas can be increased. Among them, the attention degree may have a one-to-one correspondence relationship with the priority of the divided regions. Based on the determined attention degree, different priorities can be assigned to the divided regions. Different attention levels correspond to different priorities.

It should be noted that when assigning priorities to the divided regions in this step, different priorities may correspond to different processing methods based on different application scenarios. For example, higher priority areas are assigned higher code rates, lower priority areas are assigned lower code rates; higher priority areas are assigned higher resolutions, and lower priority areas are assigned lower resolutions. Areas with higher rates or priorities use larger compression ratios, and areas with lower priorities use smaller compression ratios. It should be noted that the processing methods of regions with different priorities are not limited here. Different priorities can be assigned to different areas for different degrees of attention, and different processing methods can be used for different priorities.

In addition, if the determined degree of attention is lower than a certain threshold, this embodiment may also continue to obtain the fixation point and the corresponding pupil radius of the user gazing at the target image until the acquired degree of attention is greater than a certain threshold. Then divide the target image and assign priorities to the divided regions. This can effectively ensure that the user's attention is more interesting to the user

After dividing the target image and assigning priorities to the divided regions, this step can perform different processing on each region to which the priority is assigned to obtain data to be transmitted, and the to-be-transmitted data can store data in each region. The location information of the pixel point and the corresponding pixel value are not limited in the storage form, and may be stored in accordance with the location information or in areas.

S105. Transmit the data to be transmitted.

In this embodiment, after the data to be transmitted is obtained, this step may transmit the data to be transmitted to the display device, so that the display device displays an image corresponding to the data to be transmitted. The display device can receive the data to be transmitted one by one, and then display the corresponding content on the display device.

An image transmission method provided in Embodiment 1 of the present invention is to first obtain an eye image when a user fixes a target image; second, to determine the user's fixation point based on the line of sight corresponding to the eye image; and then obtain the user fixation location The pupil radius of the fixation point is described; the target image is then divided into regions based on the user's fixation point, the degree of attention corresponding to the pupil radius, and the pixel value of each pixel in the target image, and after each division, Assign priority to the area to obtain data to be transmitted; and finally transmit the data to be transmitted. By using the above method, before the terminal device transmits the target image, the fixation point determined by the line of sight corresponding to the eye image when the user fixes the target image, the degree of attention corresponding to the pupil radius when the user fixes the fixation point, and each of the target image The pixel value of the pixel points divides the target image, and then processes and transmits it with different priorities. Compared with directly transmitting the target image, it effectively reduces the amount of data to be transmitted, thereby reducing the time required to transmit the target image, and increasing the efficiency of image transmission; compared to compressing the target image as a whole, it is effective The sharpness of the image of the user's area of interest is improved, and the compression effect of the target image is improved.

Example two

FIG. 2a is a schematic flowchart of an image transmission method according to the second embodiment of the present invention. This second embodiment is optimized based on the foregoing embodiments. In this embodiment, the fixation point of the user is determined based on the line of sight corresponding to the eye image, which is further embodied as: extracting feature information in the eye image, where the feature information includes pupil features; The feature information determines a line of sight corresponding to the eye image; and determines a gaze point of the user in the target image according to the determined line of sight.

Optionally, in this embodiment, a pupil radius at which the user fixes the fixation point is further obtained, and is further optimized as: identifying an eye image of the user fixation at the fixation point, and determining the user's fixation on the fixation point. Pupil radius.

On the basis of the above optimization, the target image will be divided into regions based on the user's fixation point, the degree of attention corresponding to the pupil radius, and the pixel value of each pixel in the target image. Priority is assigned to the area to obtain data to be transmitted, and the optimization is specifically: comparing the pupil radius with a preset pupil data table to determine the corresponding degree of attention; using an edge operator to perform edge detection on the target image to determine Extract each closed area in the target image; select a first area containing the user's gaze point from the determined closed areas, and use the area in the target image other than the first area as the second Area; according to the attention degree and a preset attention degree comparison table, assigning corresponding priorities to the first area and the second area to obtain data to be transmitted.

Optionally, this embodiment further transmits the data to be transmitted, which is specifically optimized to transmit position information and corresponding pixel values of each pixel in the data to be transmitted in a preset order.

Optionally, before acquiring the eye image when the user looks at the target image, this embodiment further includes: when the startup instruction is detected, displaying the target image. For details that are not yet detailed in this embodiment, please refer to the first embodiment.

As shown in FIG. 2a, an image transmission method provided in Embodiment 2 of the present invention includes the following steps:

S201. When a start instruction is detected, a target image is displayed.

In this embodiment, the startup instruction may be understood as an instruction to start the terminal device to perform image processing.

Generally, when the target image is processed by the image transmission method improved in this embodiment, this step may first monitor whether there is a startup instruction. If the startup instruction is monitored, the target image can be displayed on the terminal device, so that the target image can be processed according to the user's eye image. Thereby, the corresponding data to be transmitted is obtained.

It can be understood that when the terminal device and the receiver of the data to be transmitted (such as a display device) transmit video, the startup instruction in this step may only be used to start displaying the first frame image in the video, and subsequent image frames in the video may be No need to listen to the start instruction to process the data to be transmitted. Among them, in actual applications, the startup instruction may be set according to actual conditions, may be generated by setting key control on the terminal device, or may be generated by collecting specific actions of the user's eye image, which is not limited here.

S202. Obtain an eye image when the user looks at the target image.

After receiving the start instruction to display the target image, this step can analyze the eye image when the user looks at the target image to obtain the corresponding data to be transmitted. When analyzing the next frame of image, there is no need to monitor the start instruction.

S203. Extract feature information in the eye image, where the feature information includes pupil characteristics.

After obtaining the eye image of the user looking at the target image, in this step, the eye image can be identified, and feature information in the eye image is extracted to determine the line of sight of the user looking at the target image based on the determined feature information. The feature information may include pupil features, such as pupil edge features, pupil radius, and / or pupil center position.

When an infrared camera is used in this embodiment to obtain an eye image when a user looks at a target image, the feature information may further include light spot information.

S204. Determine a line of sight corresponding to the eye image according to the feature information.

Specifically, when an infrared camera is used in this embodiment to obtain an eye image when a user fixes a target image, this step may determine a corresponding line of sight by using a corneal reflection method based on the pupil feature and the spot information in the feature information. When an ordinary camera is used in this embodiment to obtain an eye image of a user when gazing at a target image, in this step, the user's line of sight can be determined through feature information and a pre-built comparison table of features and line of sight. In addition, in this step, when determining the line of sight, a contact / non-contact sensor (such as an electrode or a capacitance sensor) can be assisted.

S205. Determine a gaze point of the user in the target image according to the determined line of sight.

After determining the line of sight corresponding to the eye image when the user looks at the target image, this step can determine the gaze point coordinates of the line of sight in the target image based on the line of sight, and then determine the user's gaze point in the target image based on the gaze point coordinates. . The line of sight can be understood as a three-dimensional vector, and the fixation point can be understood as the two-dimensional coordinates of the above-mentioned three-dimensional vector projected on a certain plane.

S206. Identify an eye image of the user gazing at the gaze point, and determine a pupil radius of the user gazing at the gaze point.

In this embodiment, after the eye image and fixation point when the user fixes the target image are determined, this step may further identify the eye image of the fixation point when the user fixes the fixation point, and determine the pupil radius of the fixation point when the user fixes the fixation point. Decide how to process the target image.

Specifically, in this step, the eye image may be processed to obtain a grayscale gradient value of the eye image in a specified direction, and then the position where the grayscale gradient value reaches the maximum value is determined as the pupil edge position. After determining the pupil edge position, it is fitted, and then the radius of the fitted figure is determined to obtain the corresponding pupil radius.

S207: Compare the pupil radius with a preset pupil data table to determine a corresponding degree of attention.

In this embodiment, the preset pupil data table can be understood as a comparison table of pupil radius and attention degree obtained in advance. After determining the pupil radius of the user's gaze point, in this step, a preset pupil data table can be searched to obtain the degree of attention corresponding to the pupil radius.

S208. Use an edge operator to perform edge detection on the target image to determine each closed area in the target image.

In this embodiment, the edge operator can be understood as an operator that performs edge detection on the target image based on the pixel value of each pixel point of the target image. For example, a Laplacian-of-Gaussian (LoG) operator, a Roberts operator, or a Prewitt operator.

In this step, when edge detection is performed on the target image by using an edge operator, each edge information in the target image can be determined. After determining the edge information, in this step, a region with continuous edges can be selected to form a closed region, and the area except the closed regions in the target image can be used to form a closed region.

S209. Select a first region including the user's gaze point from the determined closed regions, and use a region other than the first region in the target image as the second region.

After determining each closed region included in the target image, in this step, a region containing the fixation point can be selected as the first region from each closed region based on the coordinates of the fixation point, and then the area other than the first region in the target image is used as the first region. The second region, thereby dividing the target image into a region containing a fixation point and a region not containing a fixation point.

S210. Assign corresponding priorities to the first area and the second area according to the attention degree and a preset attention degree comparison table to obtain data to be transmitted.

In the embodiment, the attention degree comparison table may be understood as a preset correspondence relationship between the attention degree and the priority, and the set relationship may be obtained through training.

After determining the first area, the second area, and the attention degree, this step may look up a preset attention degree comparison table to determine the corresponding priorities, and then assign corresponding priorities to the first area and the second area to obtain transfer data.

S211. Transmit the position information of each pixel in the data to be transmitted and the corresponding pixel value in the preset order.

In this embodiment, the preset order may be understood as a preset transmission order of data to be transmitted. The position information can be understood as coordinate information.

After obtaining the data to be transmitted, this step may transmit the position information of each pixel in the data to be transmitted and the corresponding pixel value in a preset order, so that the display device performs an image based on the received position information and the corresponding pixel value. display. It can be understood that the setting order can be set according to the actual application, can be determined based on the position of each pixel, or can be determined after the divided area.

FIG. 2b is a schematic diagram of an application scenario of an image transmission method provided in Embodiment 2 of the present invention. As shown in FIG. 2b, the user 211 wears the AR device 212 to watch videos or images in the terminal device 213. The terminal device 213 may be a computer, which is not limited herein. The terminal device 213 may also be a device such as a mobile phone or a palmtop computer.

The process of image transmission based on this application scenario may be: after the terminal device 213 obtains the startup instruction, the target image may be displayed on the display screen of the terminal device 213, and then the terminal device 213 may obtain the communication connection established with the AR device 212 The eye image collected by the image acquisition device of the AR device 212 when the user 21 fixes the target image, and then extracts feature information in the eye image to determine the sight of the user 211, and determines the user 211 at the target based on the determined sight. The gaze point of the image. Then determine the pupil radius at which the user 211 looks at the fixation point, look up the degree of attention corresponding to the pupil radius in the preset pupil data table, and use the degree of attention to assign priorities to the first region and the second region after the target image is divided. Get the data to be transmitted, and finally transmit the position information and corresponding pixel values of each pixel in the data to be transmitted in a set order. When the target image is divided, an edge operator may be used to perform edge detection on the target image to determine each closed area, and then select a first area and a second area including the gaze point from each closed area.

It can be understood that when the user 211 does not wear the AR device 212, the terminal device 213 can acquire the eye image when the user 211 fixes the target image through the image acquisition device provided on the terminal device 213. If the user 211 is wearing a VR device, the terminal device 213 may be built into the VR device, and obtain the eye image of the user 211 gazing at the target image collected by the image acquisition device in the VR device through a communication connection established with the VR device.

FIG. 2c is a schematic diagram after assigning priorities to the divided target images according to the second embodiment of the present invention. As shown in FIG. 2c, assuming that a target phone 2 displays a mobile phone 221 and a host 222, and the user 211 gaze point is located on the mobile phone 221, the closed area formed by the edge of the mobile phone 221 in the target image 2 (For example, the area formed by the outer contour of the mobile phone 221) may be a first area, and the area other than the first area in the target image 2 may be a second area. The priority of the first region and the second region can be determined by analyzing the degree of attention corresponding to the pupil radius of the user 211's gaze point. If the user 211 pays more attention to the mobile phone 221, a higher priority can be set for the first region, and accordingly, the rendering accuracy or bit rate of the first region can be improved, and the rendering accuracy or bit rate of the second region can be reduced, thereby The mobile phone 221 in the first area can be displayed clearly, and the host 222 in the second area can be displayed blurry.

An image transmission method provided in Embodiment 2 of the present invention embodies a gaze point determination operation, a pupil radius determination operation, a data to be transmitted determination operation, and a data to be transmitted transmission operation. In addition, the display target image operation has been optimized. With this method, the target image can be displayed after the startup instruction is monitored, so as to determine the corresponding line of sight based on the feature information in the eye image when the user looks at the target image, and then determine the user's gaze point and look at the The degree of attention corresponding to the pupil radius when gazing at the point. Each closed area in the target image is determined by the edge operator, and then the first area containing the fixation point and the second area except the first area in the target image are selected, and the first area and The second area is assigned a priority to obtain the data to be transmitted. Finally, the position information of each pixel in the data to be transmitted and the corresponding pixel value are transmitted in a preset order. On the basis of reducing the number of transmissions, the target image is compressed. The efficiency enables the places with high user attention to be displayed clearly and the places with low attention to be displayed ambiguously, and can effectively assign appropriate priorities to the first and second regions according to the user's degree of attention at the gaze point.

Example three

FIG. 3 is a schematic structural diagram of an image transmission apparatus according to Embodiment 3 of the present invention. The apparatus is applicable to a case where a target image is transmitted between different image transmission devices (such as a terminal device and a display device). The device may be implemented by software and / or hardware, and is generally integrated on a terminal device.

As shown in FIG. 3, the image transmission device includes an eye image acquisition module 31, a fixation point determination module 32, a pupil radius determination module 33, a data to be transmitted determination module 34, and a transmission module 35.

The eye image acquisition module 31 is configured to acquire an eye image when a user looks at a target image;

A fixation point determining module 32 configured to determine a fixation point of a user based on a line of sight corresponding to the eye image;

The pupil radius determination module 33 is configured to obtain a pupil radius at which the user looks at the gaze point;

The to-be-transmitted data determining module 34 is configured to divide the target image into regions based on the user's fixation point, the degree of attention corresponding to the pupil radius, and the pixel value of each pixel in the target image, and divide the target image into segments. Subsequent areas are assigned priorities to get data to be transmitted;

The transmission module 35 is configured to transmit the data to be transmitted.

In this embodiment, the image transmission device first obtains the eye image when the user fixes the target image through the eye image acquisition module 31; secondly, the gaze point determination module 32 determines the user's gaze point based on the line of sight corresponding to the eye image ; Then obtain the pupil radius of the gaze point of the user through the pupil radius determination module 33; and then, through the pending data determination module 34, based on the user's fixation point, the degree of attention corresponding to the pupil radius, and the target image The pixel value of each pixel in the target area is used to divide the target image, and priority is assigned to each divided area to obtain data to be transmitted; finally, the data to be transmitted is transmitted through the transmission module 35.

An image transmission device provided in this embodiment can, before a terminal device transmits a target image, a fixation point determined by a line of sight corresponding to an eye image when a user fixes on the target image, and a pupil radius corresponding to the fixation point when the user fixes the fixation point. The degree of attention and the pixel value of each pixel in the target image divide the target image, and then process and transmit it with different priorities. Compared with directly transmitting the target image, it effectively reduces the amount of data to be transmitted, thereby reducing the time required to transmit the target image, and increasing the efficiency of image transmission; compared to compressing the target image as a whole, it is effective The sharpness of the image of the user's area of interest is improved, and the compression effect of the target image is improved.

Optionally, the fixation point determining module 32 is specifically configured to: extract feature information in the eye image, the feature information including pupil characteristics; determine a line of sight corresponding to the eye image according to the feature information; and The out of sight determines the gaze point of the user in the target image.

Based on the above optimization, the pupil radius determination module 33 is specifically configured to identify an eye image of the user looking at the fixation point, and determine a pupil radius of the user looking at the fixation point.

Based on the above technical solution, the data to be transmitted determination module 34 is specifically configured to: compare the pupil radius with a preset pupil data table to determine a corresponding degree of attention; use an edge operator to perform edge detection on the target image, Determine each closed area in the target image; select a first area containing the user's gaze point from the determined closed areas, and use the area other than the first area in the target image as a first Two regions; assigning corresponding priorities to the first region and the second region according to the attention degree and a preset attention degree comparison table to obtain data to be transmitted.

Optionally, the transmission module 35 is specifically configured to transmit position information and corresponding pixel values of each pixel in the data to be transmitted in a preset order.

Optionally, the image transmission device further includes a target image display module 36 configured to display a target image when a startup instruction is detected.

The above-mentioned image transmission device can execute the image transmission method provided by any embodiment of the present invention, and has corresponding function modules and beneficial effects of executing the method.

Embodiment 4

FIG. 4 is a schematic structural diagram of a terminal device according to a fourth embodiment of the present invention. As shown in FIG. 4, the terminal device provided in the fourth embodiment of the present invention includes: one or more processors 41 and a storage device 42; the processor 41 in the terminal device may be one or more, and FIG. The storage device 42 is taken as an example; the storage device 42 is configured to store one or more programs; the one or more programs are executed by the one or more processors 41, so that the one or more processors 41 are implemented as the present invention The image transmission method according to any one of the embodiments.

The terminal device may further include an input device 43 and an output device 44.

The processor 41, the storage device 42, the input device 43, and the output device 44 in the terminal device may be connected through a bus or other manners. In FIG. 4, the connection through the bus is taken as an example.

As a computer-readable storage medium, the storage device 42 in the terminal device may be configured to store one or more programs. The programs may be software programs, computer-executable programs, and modules, as in the first or second embodiment of the present invention. Program instructions / modules corresponding to the provided image transmission method (for example, the modules in the image transmission device shown in FIG. 3 include: eye image acquisition module 31, fixation point determination module 32, pupil radius determination module 33, to be transmitted The data determination module 34 and the transmission module 35 further include a target image display module 36). The processor 41 runs software programs, instructions, and modules stored in the storage device 42 to execute various functional applications and data processing of the terminal device, that is, to implement the image transmission method in the foregoing method embodiment.

The storage device 42 may include a storage program area and a storage data area, wherein the storage program area may store an operating system and application programs required for at least one function; the storage data area may store data created according to the use of the device, and the like. In addition, the storage device 42 may include a high-speed random access memory, and may further include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some examples, the storage device 42 may further include memories remotely provided with respect to the processor 41, and these remote memories may be connected to the device through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The input device 43 may be configured to receive inputted numeric or character information, and generate key signal input related to user settings and function control of the terminal device or acquire an eye image when the user looks at the target image. The input device 43 may include, but is not limited to, an image acquisition device (such as an infrared camera configured with an infrared lamp), input devices such as keys and / or a microphone. The output device 44 may include, but is not limited to, a display screen.

In addition, when the one or more programs included in the terminal device are executed by the one or more processors 41, the program performs the following operations: acquiring an eye image when the user looks at the target image; and based on the corresponding eye image, The line of sight determines the user's gaze point; obtains the pupil radius of the user's gaze point; based on the user's gaze point, the degree of attention corresponding to the pupil radius, and the pixel value of each pixel in the target image Divide the target image into regions, and assign priorities to the divided regions to obtain data to be transmitted; and transmit the data to be transmitted.

In addition, an embodiment of the present invention further provides a computer storage medium on which a computer program is stored. The computer storage medium may include a readable storage medium and / or a writable storage medium. When the program is executed by a processor, the method is used to execute an image transmission method. The method includes: acquiring an eye image when a user fixes a target image; determining a user's gaze point based on a line of sight corresponding to the eye image; The pupil radius of the user gazing at the fixation point; dividing the target image into regions based on the user's fixation point, the degree of attention corresponding to the pupil radius, and the pixel value of each pixel in the target image, and Priority is assigned to each divided area to obtain data to be transmitted; and the data to be transmitted is transmitted.

Optionally, when the program is executed by the processor, the program may also be used to execute a technical solution of an image transmission method provided by any embodiment of the present invention. Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by software and necessary general hardware, and of course, can also be implemented by hardware, but in many cases the former is a better implementation . Based on such an understanding, the technical solution of the present invention that is essentially or contributes to the existing technology can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a computer floppy disk , Read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including several instructions to make a computer device (can be a personal computer , Server, or network device, etc.) perform the methods described in the embodiments of the present invention.

Note that the above are only the preferred embodiments of the present invention and the applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and those skilled in the art can make various obvious changes, readjustments and substitutions without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, more equivalent embodiments may be included, and the present invention The scope is determined by the scope of the appended claims.

Industrial applicability

The solution provided by the embodiment of the present invention may be applied to an image transmission process. The solution first obtains an eye image when a user fixes a target image; secondly, determines a user's fixation point based on a line of sight corresponding to the eye image; Obtaining a pupil radius at which the user fixates on the fixation point; and thereafter dividing the target image into regions based on the fixation point of the user, a degree of attention corresponding to the pupil radius, and a pixel value of each pixel in the target image And assign priority to each divided area to obtain data to be transmitted; and finally transmit the data to be transmitted. With the above technical solution, before the terminal device transmits the target image, the fixation point determined by the line of sight corresponding to the eye image when the user fixes the target image, the degree of attention corresponding to the pupil radius when the user fixes the fixation point, and the target image The pixel value of each pixel points divides the target image, and then transmits it with different priorities (different priorities may correspond to different code rates or different rendering accuracy). Compared with directly transmitting the target image, it effectively reduces the amount of data to be transmitted, thereby reducing the time required to transmit the target image, and improving the efficiency of image transmission; compared with compressing the target image as a whole, it effectively improves It improves the sharpness of the image of the user's area of interest and improves the compression effect of the target image.

Claims

An image transmission method includes:

Obtaining an eye image when a user looks at a target image;

Determining a gaze point of the user based on a line of sight corresponding to the eye image;

Obtaining a pupil radius at which the user looks at the fixation point;

Divide the target image into regions based on the user's fixation point, the degree of attention corresponding to the pupil radius, and the pixel value of each pixel in the target image, and assign priorities to the divided regions to obtain transfer data;

Transmitting the data to be transmitted.
The method according to claim 1, wherein the determining a gaze point of the user based on a line of sight corresponding to the eye image comprises:

Extract feature information in the eye image, where the feature information includes pupil features;

Determining a line of sight corresponding to the eye image according to the feature information;

Determining a gaze point of the user in the target image according to the determined line of sight.
The method according to claim 1, wherein the obtaining a pupil radius of the user gazing at the gaze point comprises:

An eye image of the user gazing at the gaze point is identified, and a pupil radius of the user gazing at the gaze point is determined.
The method according to claim 1, wherein the target image is divided into regions based on the user's fixation point, the degree of attention corresponding to the pupil radius, and the pixel value of each pixel in the target image, Priority is assigned to each divided area to obtain data to be transmitted, including:

Comparing the pupil radius with a preset pupil data table to determine a corresponding degree of attention;

Performing edge detection on the target image using an edge operator to determine each closed area in the target image;

Selecting a first region containing the user's gaze point from the determined closed regions, and using a region other than the first region in the target image as a second region;

According to the attention degree and a preset attention degree comparison table, corresponding priorities are assigned to the first area and the second area to obtain data to be transmitted.
The method according to claim 4, wherein said transmitting said data to be transmitted comprises:

The position information of each pixel in the data to be transmitted and the corresponding pixel value are transmitted in a preset order.
The method according to claim 1, before the acquiring an eye image when a user fixes a target image, further comprising:

When a start instruction is detected, a target image is displayed.
An image transmission device includes:

An eye image acquisition module configured to acquire an eye image when a user looks at a target image;

A fixation point determining module, configured to determine a fixation point of a user based on a line of sight corresponding to the eye image;

A pupil radius determination module, configured to obtain a pupil radius at which the user looks at the fixation point;

The data to be transmitted determination module is configured to divide the target image into regions based on the user's fixation point, the degree of attention corresponding to the pupil radius, and the pixel value of each pixel in the target image, The priority of the assigned area to get the data to be transmitted;

A transmission module configured to transmit the data to be transmitted.
The apparatus according to claim 7, further comprising:

The target image display module is configured to display a target image when a start instruction is detected.
A terminal device includes:

One or more processors;

A storage device configured to store one or more programs;

The one or more programs are executed by the one or more processors, so that the one or more processors implement the image transmission method according to any one of claims 1 to 6.
A computer storage medium having stored thereon a computer program that, when executed by a processor, implements the image transmission method according to any one of claims 1 to 6.