CN111782050B - Image processing method and apparatus, storage medium, and electronic device - Google Patents
Image processing method and apparatus, storage medium, and electronic device Download PDFInfo
- Publication number
- CN111782050B CN111782050B CN202010634979.3A CN202010634979A CN111782050B CN 111782050 B CN111782050 B CN 111782050B CN 202010634979 A CN202010634979 A CN 202010634979A CN 111782050 B CN111782050 B CN 111782050B
- Authority
- CN
- China
- Prior art keywords
- sight line
- gaze
- page
- eye image
- page operation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
Abstract
The present disclosure relates to an image processing method and apparatus, a storage medium, and an electronic device, the method including: acquiring an eye image of a user; inputting the eye image into a sight line detection model, and acquiring a sight line direction; and generating a page operation instruction based on the sight line direction, wherein the page operation instruction is used for instructing the electronic equipment to execute a preset page operation corresponding to the sight line direction.
Description
Technical Field
The present disclosure relates to the field of image processing, and in particular, to an image processing method and apparatus, a storage medium, and an electronic device.
Background
The existing control modes of equipment operation are quite diverse, and a user can realize page operation on the equipment through various modes, such as finger click, mouse click, sound control, gesture control and the like, but the existing control modes still have the problem of insufficient convenience.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
In a first aspect, the present disclosure provides a method of image processing, the method comprising: acquiring an eye image of a user; inputting the eye image into a sight line detection model, and acquiring a sight line direction; and generating a page operation instruction based on the sight line direction, wherein the page operation instruction is used for instructing the electronic equipment to execute a preset page operation corresponding to the sight line direction.
In a second aspect, the present disclosure provides an image processing apparatus, including an obtaining module configured to obtain an eye image of a user; the input module is used for inputting the eye image into a sight line detection model and acquiring a sight line direction; and the generating module is used for generating a page operation instruction based on the sight direction, and the page operation instruction is used for indicating the electronic equipment to execute a preset page operation corresponding to the sight direction.
In a third aspect, a computer-readable medium is provided, on which a computer program is stored which, when being executed by a processing means, carries out the steps of the method of the first aspect.
In a fourth aspect, an electronic device is provided, comprising: storage means having a computer program stored thereon, and processing means for executing the computer program in the storage means to carry out the steps of the method of the first aspect.
Based on the technical scheme, at least the following technical effects can be realized:
the sight direction can be determined through the eye image of the user, and a page operation instruction is generated based on the sight direction to instruct the electronic equipment to execute the preset page operation corresponding to the sight direction, so that the convenience of page control of the electronic equipment can be improved, and the use experience of the user is improved.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows.
Drawings
The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale.
In the drawings:
FIG. 1 is a flow chart illustrating a method of image processing in accordance with an exemplary disclosed embodiment.
Fig. 2 is a schematic diagram of determining an iris image and an eyeball image based on an eye image.
Fig. 3 is a schematic diagram of a viewing direction corresponding to a predetermined page operation.
FIG. 4 is a flow chart illustrating a method of image processing in accordance with an exemplary disclosed embodiment.
Fig. 5 is a block diagram of an image processing apparatus shown in an exemplary disclosed embodiment.
FIG. 6 is a block diagram of an electronic device shown in an exemplary disclosed embodiment.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.
It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
FIG. 1 is an illustration of an image processing method, as illustrated in FIG. 1, including the steps of:
and S11, acquiring an eye image of the user.
The eye image may be a sub-image including an eye portion captured from an image acquired by an image capturing apparatus, where the image capturing apparatus may refer to an image capturing device of an electronic apparatus to be controlled, for example, the image capturing apparatus is a front camera of a mobile phone to be controlled; the image pickup apparatus may also be other image pickup apparatuses that do not belong to the electronic apparatus to be controlled, for example, the image pickup apparatus may be smart glasses, and the electronic apparatus to be controlled may be a mobile phone. The present disclosure does not limit the manner in which an eye image is acquired.
And S12, inputting the eye image into a sight line detection model, and acquiring a sight line direction.
Wherein the eye gaze detection model may determine an iris image and an eye image based on the eye image, and determine a gaze direction based on the iris image and the eye image
Wherein g is a sight line vector representing a sight line direction, theta is a sight line pitch angle of the sight line direction,
is the line of sight yaw angle in the direction of the line of sight. By learning the iris image and the eyeball image of the sample image and learning the gaze pitch angle and the gaze yaw angle corresponding to the iris image and the eyeball image of the sample, the gaze detection model may generate the gaze direction corresponding to the eye image based on the input eye image. Fig. 2 is a schematic diagram of determining an iris image and an eyeball image based on an eye image, and the input eye image (simplified), the iris image, the eyeball image, the superposition visualization of the iris image and the eyeball image are sequentially displayed from left to right.
The iris image and the eyeball image can well help to calculate the actual sight direction of human eyes. The visual line direction of the human eye represents the 3d line direction connecting the eyeball direction and the center of the iris. The iris image and the eyeball image obtained by using the network prediction respectively include the iris position and the eyeball position, and the 3d direction is projected on the 2d plane. When the network is trained, true value supervision and multilayer supervision are provided when an iris image, an eyeball image and the sight line of human eyes are calculated, and the accuracy of a network prediction result can be better guaranteed. In the first part of the network of the model, three hourglass networks are used for cascading to obtain an iris image and an eyeball image. In the second part of the network of models, regression of the gaze specific angle values, the final gaze direction can be obtained based on DenseNet using 5 dense modules (each module comprising 5 layers) and one linear regression layer.
And S13, generating a page operation instruction based on the sight direction, wherein the page operation instruction is used for instructing the electronic equipment to execute a preset page operation corresponding to the sight direction.
Wherein the predetermined page operation comprises at least one of the following operations: turning pages towards the sight line direction, sliding towards the sight line direction, amplifying the sub-interface of the sight line direction, and moving the target object in the page to the sight line direction.
For example, if the area projected on the page in the gaze direction is a right area, operations such as sliding to the right, turning to the right, enlarging a right sub-interface, and moving an object to the right may be performed. For the embodiment of magnifying the sub-interface and moving the target object to the viewing direction, the target area projected on the page by the viewing direction may be magnified or the target object may be moved to the target area.
The direction to which the predetermined page operation is directed may also be determined according to the direction of the line of sight, for example, the direction to which the predetermined page operation is directed may be determined by determining within which angle range the pitch angle and the yaw angle of the direction of the line of sight are located, through the preset angle ranges of the yaw angle and the pitch angle, and the correspondence between the directions to which the predetermined page operation is directed.
It should be noted that the present disclosure may be applied to a server, where the server acquires an eye image from an image capturing device, and after generating a page operation instruction, sends the page operation instruction to at least one preset terminal electronic device, and the terminal electronic device executes the page operation instruction. The disclosure may also be applied to a terminal, which executes a page operation instruction by itself after the page operation instruction is generated, or may transmit the page operation instruction to at least one other terminal, and execute the page operation instruction by the other terminal.
For example, an eye image of a user can be acquired through glasses with a preset camera, the eye image is sent to a mobile phone of the user, the mobile phone processes the eye image to generate a page operation instruction, and the page operation instruction is sent to the projector, so that the projector can execute a preset page operation corresponding to the page operation instruction, and operations such as page turning, sliding, amplifying and moving of an object are achieved.
When the user looks to the right, the user's eye image is acquired, so that the user's sight line direction can be determined to point to the right, and instructions such as turning the page to the right, sliding the page to the right, enlarging the sub-interface on the right side, or moving the target object to the right side of the page can be generated.
When the method is applied to the electronic equipment for executing the page operation instruction, the page special effect matched with the page operation instruction can be generated based on the page operation instruction. For example, when the operation instruction is to page to the right, a page turning special effect, a transition special effect and the like on the right side can be generated, so as to improve the visual effect when the page operation instruction is executed. When the page operation instruction is a moving object, a special effect related to the target object may be generated, for example, when the target object is a vehicle icon, a dust special effect may be generated after the vehicle icon is moved to the sight line direction, and when the target object is a star icon, a starlight special effect may be generated after the star icon is moved to the sight line direction, so as to increase the interaction interest.
Fig. 3 is a schematic diagram illustrating a view direction and a predetermined page operation, and as shown in fig. 3, sliding, page turning, zooming, and moving objects can be performed through the view direction, wherein an area corresponding to a dotted line frame is a target area pointed by the view direction, and a solid circular pattern represents the target object.
In a possible implementation manner, before the page operation instruction is generated, it may be further determined that the gesture of the user is consistent with a preset gesture, so as to determine that the line-of-sight mode is opened by the user. In the gaze mode, the user may direct the device to perform different predetermined page operations through changes in the eye gaze direction. The starting mode of the sight mode can also be that the voice content of the user is determined to be consistent with the preset voice content, an instruction for representing the starting of the sight mode of the electronic equipment to be controlled is obtained, and the like. It should be noted that the view mode may be a mode in which the page operation of the electronic device may be controlled by the view direction, or a mode in which the page operation of the application program may be controlled by the view direction, and the opening control of the view mode may be determined by a subunit of the electronic device or a plug-in of the application program.
As an example of a scene, when a user uses a tablet computer to watch a menu in the kitchen, and when a page needs to be turned, the traditional way of swiping with a finger is very inconvenient to use, and the user may make a preset gesture to open a view mode of the tablet computer and instruct the tablet computer to turn the page through the view.
In one possible implementation, the line-of-sight direction of the user at the previous time may be determined, and in a case where a variation amount of the line-of-sight direction from the line-of-sight direction at the previous time is greater than a variation amount threshold, a page operation instruction may be generated based on the line-of-sight direction.
The eye image of the user can be sampled at preset time intervals and the sight direction can be judged, and the sight direction at the previous moment can be the sight direction corresponding to the eye image acquired at the previous sampling moment before the current sampling moment.
The view direction may include a view pitch angle and a view yaw angle, and the variation may be one or more of a variation of the view pitch angle, a variation of the view yaw angle, and a vector variation value of a combination of the two.
Based on the technical scheme, at least the following technical effects can be realized:
the sight direction can be determined through the eye image of the user, and a page operation instruction is generated based on the sight direction to instruct the electronic equipment to execute the preset page operation corresponding to the sight direction, so that the convenience of page control of the electronic equipment can be improved, and the use experience of the user is improved.
FIG. 4 is a flow chart illustrating a method of image processing, as shown in FIG. 4, including the steps of:
and S41, acquiring a first eye image at a first moment.
And S42, acquiring a second eye image at a second time.
Here, the eye image of the user may be sampled at preset time intervals and the determination of the gaze direction may be performed, and the first time and the second time in S41 and S42 may be different sampling times.
The first time and the second time may also be preset times, for example, the first time may be a preset time when the application is opened, and the second time may be a preset time three seconds after the application is opened.
The eye image may be a sub-image including an eye portion captured from an image acquired by an image pickup apparatus, which may be an image pickup device of an electronic apparatus to be controlled, for example, a front camera of a mobile phone to be controlled; the image pickup apparatus may also be other image pickup apparatuses that do not belong to the electronic apparatus to be controlled, for example, the image pickup apparatus may be smart glasses, and the electronic apparatus to be controlled may be a mobile phone. The present disclosure does not limit the manner in which an eye image is acquired.
S43, inputting the first eye image into the sight line detection model, and acquiring a first sight line direction.
And S44, inputting the second eye image into the sight line detection model to obtain a second sight line direction.
S43 and S44 may be executed simultaneously, or S43 may be executed before S44 or S44 may be executed before S43, and the execution order of S43 and S44 is not limited in this disclosure. Moreover, S43 and S42 may also be executed simultaneously or sequentially, and the execution order of S43 and S42 is not limited in this disclosure. It is to be noted that since the second eye image in S44 is obtained through S42, when S43 is performed simultaneously with S42 or S43 is performed before S42, S43 should be performed before S44.
The gaze detection model may determine an iris image and an eye image based on the eye image and determine a gaze direction based on the iris image and the eye image
Wherein g is a sight line vector representing a sight line direction, theta is a sight line pitch angle of the sight line direction,
is the line of sight yaw angle in the direction of the line of sight. By learning the iris image and the eyeball image of the sample image and learning the gaze pitch angle and the gaze yaw angle corresponding to the iris image and the eyeball image of the sample, the gaze detection model can generate the gaze direction corresponding to the eye image based on the input eye image. Fig. 2 is a schematic diagram for determining an iris image and an eyeball image based on an eye image, and the input eye image, the iris image, the eyeball image, the superposition of the iris image and the eyeball image are visualized from left to right.
The iris image and the eyeball image can well help to calculate the actual sight direction of human eyes. The visual line direction of the human eye represents the 3d line direction connecting the eyeball direction and the center of the iris. The iris image and the eyeball image obtained by using the network prediction respectively include the iris position and the eyeball position, and the 3d direction is projected on the 2d plane. When the network is trained, true value supervision and multilayer supervision are provided when an iris image, an eyeball image and a human eye sight line are calculated, and accuracy of a network prediction result can be better guaranteed. In the first part of the network of the model, three hourglass networks are used for cascading to obtain an iris image and an eyeball image. In the second part of the network of models, regression of the gaze specific angle values, the final gaze direction can be obtained based on DenseNet using 5 dense modules (each module comprising 5 layers) and one linear regression layer.
And S45, determining a sight line change direction based on the first sight line direction and the second sight line direction.
The sight line direction is a vector value, and the change value of the sight line direction can also be a vector value of the same unit, and the sight line change direction can be
Wherein g 'is a sight line change vector representing the sight line change direction, theta' is a change value of the sight line pitch angle,
is the variation value of the line of sight yaw angle.
And S46, generating a page operation instruction based on the sight line change direction.
The page operation instruction is used for instructing the electronic equipment to execute a preset page operation corresponding to the sight line change direction, wherein the preset page operation comprises at least one of the following operations: turning pages to the sight line change direction, sliding to the sight line change direction, amplifying the sub-interface of the sight line change direction, and moving the target object in the page along the sight line change direction.
It should be noted that the present disclosure may be applied to a server, where the server acquires an eye image from an image capturing device, and after generating a page operation instruction, sends the page operation instruction to at least one preset terminal electronic device, and the terminal electronic device executes the page operation instruction. The disclosure may also be applied to a terminal, which executes a page operation instruction by itself after the page operation instruction is generated, or may transmit the page operation instruction to at least one other terminal, and execute the page operation instruction by the other terminal.
Specifically, S13 in the previous embodiment may be simply deformed according to the specific manner of generating the page operation instruction according to the direction of the line of sight change, and the direction of the line of sight change is used to replace the direction of the line of sight, which is not described herein again.
In a possible implementation, the predetermined page is operated to move a target object in the page along the gaze direction, and the object in the page located in the first gaze direction may be determined to be the target object based on the first gaze direction, and the target object may be moved to the second gaze direction. Wherein, the moving amount and the moving angle of the target object correspond to the direction of the change of the sight line.
For example, an object in a page area corresponding to a first direction of the user's line of sight may be used as a target object, and the movement distance and the movement angle of the target object may be calculated from the direction of the line of sight change and moved to a page area corresponding to a second direction of the line of sight.
When the method is applied to the electronic equipment for executing the page operation instruction, the page special effect matched with the page operation instruction can be generated based on the page operation instruction. For example, when the operation instruction is to page to the right, a page turning special effect, a transition special effect and the like on the right side can be generated, so as to improve the visual effect when the page operation instruction is executed. When the page operation instruction is a moving object, a special effect related to the target object can be generated, for example, when the target object is a vehicle icon, a dust special effect can be generated after the vehicle icon is moved to the sight line direction, and when the target object is a star icon, a starlight special effect can be generated after the star icon is moved to the sight line direction, so that the interaction interestingness is increased.
In a possible implementation manner, before the page operation instruction is generated, it may be further determined that the gesture of the user is consistent with a preset gesture, so as to determine that the user opens the view mode. In the gaze mode, the user may direct the device to perform different page operations by a change in the eye gaze direction. The starting mode of the sight mode can also be that the voice content of the user is determined to be consistent with the preset voice content, an instruction for representing the starting of the sight mode of the electronic equipment to be controlled is obtained, and the like. It should be noted that the gaze mode may be a mode in which the page operation representing the electronic device may be controlled by the gaze direction, or a mode in which the page operation representing the application may be controlled by the gaze direction, and the opening controls of the gaze mode may be determined by subunits of the electronic device, or may be determined by a plug-in of the application.
In a possible implementation manner, in a case that a modulus of the direction of change of the line of sight is greater than a preset change amount threshold, a page operation instruction is generated based on the direction of the line of sight. The gaze direction is a vector value, and a modulus of the vector value can be used to represent the magnitude of the gaze direction, i.e. to implement the strength of the change. By the mode, page changes caused by normal eye sight changes without reaching the indication intensity can be avoided, and therefore the use experience of a user can be improved.
For example, when the user slightly rotates the eyes during reading, but the variation of the line of sight does not reach the preset variation threshold, the page operation instruction may not be generated, so as to avoid the influence of too frequent page operations on the reading experience of the user due to too sensitive line of sight variation identification.
Based on the technical scheme, at least the following technical effects can be realized:
the sight line change direction can be determined through the eye images of the user at the first moment and the second moment, and a page operation instruction is generated based on the sight line change direction to instruct the electronic equipment to execute the preset page operation corresponding to the sight line change direction, so that the convenience of electronic equipment page control can be improved, and the use experience of the user is improved.
FIG. 5 is a block diagram illustrating an image processing apparatus according to an exemplary disclosed embodiment. As shown in fig. 5, the image processing apparatus 500 includes an acquisition module 510, an input module 520, and a generation module 530.
The obtaining module 510 is configured to obtain an eye image of a user.
An input module 520, configured to input the eye image into a sight line detection model, and acquire a sight line direction.
A generating module 530, configured to generate a page operation instruction based on the gaze direction, where the page operation instruction is used to instruct an electronic device to perform a predetermined page operation corresponding to the gaze direction, where the predetermined page operation includes at least one of the following operations: turning pages towards the sight line direction, sliding towards the sight line direction, amplifying the sub-interface of the sight line direction, and moving the target object in the page to the sight line direction.
Optionally, the obtaining module is further configured to obtain an eye image of the user, where the eye image includes a first eye image obtained at a first time and a second eye image obtained at a second time; the input module is further configured to input the first eye image into the gaze detection model to obtain a first gaze direction, input the second eye image into the gaze detection model to obtain a second gaze direction, and determine a gaze variation direction based on the first gaze direction and the second gaze direction; the generating module is further configured to generate a page operation instruction based on the gaze direction, where the page operation instruction is used to instruct an electronic device to execute a predetermined page operation corresponding to the gaze direction, where the predetermined page operation includes at least one of the following operations: turning pages to the sight line change direction, sliding to the sight line change direction, amplifying the sub-interface of the sight line change direction, and moving the target object in the page along the sight line change direction.
Optionally, the predetermined page is operable to move a target object in the page along the gaze direction, and the apparatus further includes a moving module configured to determine, based on the first gaze direction, that an object in the page located in the first gaze direction is the target object; moving the target object to the second gaze direction.
Optionally, the apparatus further comprises a determining module, configured to determine a gaze direction of the user at a previous time; the generating module is configured to generate a page operation instruction based on the gaze direction when a variation of the gaze direction from the gaze direction at the previous time is greater than a variation threshold.
Optionally, the apparatus further includes a gesture module configured to determine that the gesture of the user is consistent with a preset gesture.
Optionally, the apparatus is applied to the electronic device, and the apparatus further includes a special effect module, configured to generate a page special effect matched with the page operation instruction based on the page operation instruction.
Optionally, the gaze detection model is configured to determine an iris image and an eyeball image based on the eye image; determining a gaze direction based on the iris image and the eye image, the gaze direction including a gaze pitch angle and a gaze yaw angle.
The specific implementation of the modules in the above device has been described in detail in the embodiments of the method, and is not described herein again.
Based on the technical scheme, at least the following technical effects can be realized:
the sight direction can be determined through the eye image of the user, and a page operation instruction is generated based on the sight direction to instruct the electronic equipment to execute the preset page operation corresponding to the sight direction, so that the convenience of page control of the electronic equipment can be improved, and the use experience of the user is improved.
Referring now to fig. 6, a schematic diagram of an electronic device (e.g., a terminal or server in the above-described embodiments) 600 suitable for implementing embodiments of the present disclosure is shown. The terminal device in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 6, the electronic device 600 may include a processing means (e.g., central processing unit, graphics processor, etc.) 601 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the electronic apparatus 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
Generally, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While fig. 6 illustrates an electronic device 600 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means 609, or may be installed from the storage means 608, or may be installed from the ROM 602. The computer program, when executed by the processing device 601, performs the above-described functions defined in the methods of the embodiments of the present disclosure.
It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring at least two internet protocol addresses; sending a node evaluation request comprising the at least two internet protocol addresses to node evaluation equipment, wherein the node evaluation equipment selects the internet protocol addresses from the at least two internet protocol addresses and returns the internet protocol addresses; receiving an internet protocol address returned by the node evaluation equipment; wherein the obtained internet protocol address indicates an edge node in the content distribution network.
Alternatively, the computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: receiving a node evaluation request comprising at least two internet protocol addresses; selecting an internet protocol address from the at least two internet protocol addresses; returning the selected internet protocol address; wherein the received internet protocol address indicates an edge node in the content distribution network.
Computer program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present disclosure may be implemented by software or hardware. The name of the module does not in some cases form a limitation of the module itself, for example, the first obtaining module may also be described as a "module for obtaining at least two internet protocol addresses".
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems on a chip (SOCs), complex Programmable Logic Devices (CPLDs), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
Example 1 provides, in accordance with one or more embodiments of the present disclosure, an image processing method, the method including: acquiring an eye image of a user; inputting the eye image into a sight line detection model, and acquiring a sight line direction; and generating a page operation instruction based on the sight line direction, wherein the page operation instruction is used for instructing the electronic equipment to execute a preset page operation corresponding to the sight line direction.
Example 2 provides the method of example 1, the predetermined page operation comprising at least one of: turning pages towards the sight line direction, sliding towards the sight line direction, amplifying the sub-interface of the sight line direction, and moving the target object in the page to the sight line direction.
Example 3 provides the method of example 1, the eye images including a first eye image acquired at a first time and a second eye image acquired at a second time; the inputting the eye image into a sight line detection model and acquiring a sight line direction includes: inputting the first eye image into the sight line detection model to obtain a first sight line direction; inputting the second eye image into the sight line detection model to obtain a second sight line direction; determining a gaze direction of change based on the first gaze direction and the second gaze direction; generating a page operation instruction based on the sight line direction, wherein the page operation instruction comprises: generating a page operation instruction based on the gaze direction, wherein the page operation instruction is used for instructing the electronic equipment to execute a predetermined page operation corresponding to the gaze direction, and the predetermined page operation comprises at least one of the following operations: turning pages to the sight line change direction, sliding to the sight line change direction, amplifying the sub-interface of the sight line change direction, and moving the target object in the page along the sight line change direction.
Example 4 provides the method of example 3, the predetermined page operative to move a target object in the page in the gaze change direction, the method further comprising: determining an object in the page located in the first sight line direction as the target object based on the first sight line direction; moving the target object to the second gaze direction.
Example 5 provides the method of example 1, further comprising, in accordance with one or more embodiments of the present disclosure: determining the sight line direction of the user at the previous moment; generating a page operation instruction based on the sight line direction, wherein the page operation instruction comprises: and generating a page operation instruction based on the sight line direction under the condition that the variation of the sight line direction relative to the sight line direction at the previous moment is larger than a variation threshold.
Example 6 provides the method of example 1, further including, prior to generating the page operation instructions based on the gaze direction: and determining that the gesture of the user is consistent with a preset gesture.
Example 7 provides the methods of examples 1-6, applied to the electronic device, further including, in accordance with one or more embodiments of the present disclosure: and generating a page special effect matched with the page operation instruction based on the page operation instruction.
Example 8 provides the methods of examples 1-6, the gaze detection model to: determining an iris image and an eyeball image based on the eye image; determining a gaze direction based on the iris image and the eye image, the gaze direction including a gaze pitch angle and a gaze yaw angle.
Example 9 provides an image processing apparatus according to one or more embodiments of the present disclosure, including an acquisition module configured to acquire an eye image of a user; the input module is used for inputting the eye image into a sight detection model and acquiring a sight direction; and the generating module is used for generating a page operation instruction based on the sight line direction, and the page operation instruction is used for indicating the electronic equipment to execute a preset page operation corresponding to the sight line direction.
Example 10 provides the apparatus of example 9, the predetermined page operations comprising at least one of: turning pages towards the sight line direction, sliding towards the sight line direction, amplifying the sub-interface of the sight line direction, and moving the target object in the page to the sight line direction.
Example 11 provides the apparatus of example 9, the acquisition module further to acquire an eye image of the user, the eye image including a first eye image acquired at a first time and a second eye image acquired at a second time; the input module is further configured to input the first eye image into the gaze detection model to obtain a first gaze direction, input the second eye image into the gaze detection model to obtain a second gaze direction, and determine a gaze variation direction based on the first gaze direction and the second gaze direction; the generating module is further configured to generate a page operation instruction based on the gaze direction, where the page operation instruction is used to instruct an electronic device to execute a predetermined page operation corresponding to the gaze direction, where the predetermined page operation includes at least one of the following operations: turning pages to the sight line change direction, sliding to the sight line change direction, amplifying the sub-interface of the sight line change direction, and moving the target object in the page along the sight line change direction.
Example 12 provides the apparatus of example 11, the predetermined page operative to move a target object in a page in the gaze change direction, the apparatus further comprising a movement module to determine, based on the first gaze direction, that an object in the page that is located in the first gaze direction is the target object; moving the target object to the second gaze direction.
Example 13 provides the apparatus of example 9, further including a determination module to determine a gaze direction of the user at a previous time, in accordance with one or more embodiments of the present disclosure; the generating module is configured to generate a page operation instruction based on the gaze direction when a variation of the gaze direction from the gaze direction at the previous time is greater than a variation threshold.
Example 14 provides the apparatus of example 9, further comprising a gesture module to determine that the gesture of the user is consistent with a preset gesture, in accordance with one or more embodiments of the present disclosure.
Example 15 provides the apparatus of examples 9 to 14, which is applied to the electronic device, and further includes a special effect module, configured to generate, based on the page operation instruction, a page special effect matching the page operation instruction.
Example 16 provides the apparatus of examples 9-14, the gaze detection model to determine an iris image and an eyeball image based on the eye image, in accordance with one or more embodiments of the present disclosure; determining a gaze direction based on the iris image and the eye image, the gaze direction including a gaze pitch angle and a gaze yaw angle.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.
Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Claims (7)
1. An image processing method, characterized in that the method comprises:
acquiring a first eye image of a user at a first time and a second eye image of the user at a second time;
inputting the first eye image into a sight detection model, and acquiring a first sight direction of the user at the first moment;
inputting the second eye image into the sight line detection model, and acquiring a second sight line direction of the user at the second moment;
determining a gaze direction of change based on the first gaze direction and the second gaze direction;
under the condition that the modulus of the sight line change direction is larger than a preset variation threshold, generating a page operation instruction based on the sight line change direction, wherein the page operation instruction is used for instructing the electronic equipment to execute a preset page operation corresponding to the sight line change direction;
the predetermined page operations include moving a target object in a page in the gaze change direction, the method further comprising:
determining an object located in the first sight line direction in the page as the target object based on the first sight line direction of the user at the first moment;
moving the target object to the second gaze direction of the user at the second time;
wherein the gaze detection model is to:
determining an iris image and an eyeball image based on the input eye image;
determining a gaze direction based on the iris image and the eye image, the gaze direction including a gaze pitch angle and a gaze yaw angle.
2. The method of claim 1, wherein the predetermined page operations further comprise at least one of: and turning pages to the sight line change direction, sliding to the sight line change direction, and amplifying the sub-interface of the sight line change direction.
3. The method of claim 1, wherein before generating page operation instructions based on the gaze direction, the method further comprises:
and determining that the gesture of the user is consistent with a preset gesture.
4. The method according to any one of claims 1-3, applied to the electronic device, further comprising:
and generating a page special effect matched with the page operation instruction based on the page operation instruction.
5. An image processing apparatus, characterized in that the apparatus comprises:
the device comprises an acquisition module, a display module and a control module, wherein the acquisition module is used for acquiring a first eye image of a user at a first moment and a second eye image of the user at a second moment;
the input module is used for inputting the first eye image into a sight line detection model, acquiring a first sight line direction of a user at the first moment, inputting the second eye image into the sight line detection model, acquiring a second sight line direction of the user at the second moment, and determining a sight line change direction based on the first sight line direction and the second sight line direction;
the generating module is used for generating a page operation instruction based on the sight line change direction under the condition that the modulus of the sight line change direction is larger than a preset change threshold, wherein the page operation instruction is used for indicating the electronic equipment to execute a preset page operation corresponding to the sight line change direction;
the predetermined page operation comprises moving a target object in a page along the gaze direction, the apparatus further comprises a moving module for determining an object in the page located in the first gaze direction as the target object based on the first gaze direction, and moving the target object to the second gaze direction;
wherein the gaze detection model is to:
determining an iris image and an eyeball image based on the input eye image;
determining a gaze direction based on the iris image and the eye image, the gaze direction including a gaze pitch angle and a gaze yaw angle.
6. A non-transitory computer readable storage medium, on which a computer program is stored, characterized in that the program, when executed by a processing device, implements the steps of the method of any one of claims 1-4.
7. An electronic device, comprising:
a storage device having a computer program stored thereon;
processing means for executing the computer program in the storage means to carry out the steps of the method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010634979.3A CN111782050B (en) | 2020-07-03 | 2020-07-03 | Image processing method and apparatus, storage medium, and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010634979.3A CN111782050B (en) | 2020-07-03 | 2020-07-03 | Image processing method and apparatus, storage medium, and electronic device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111782050A CN111782050A (en) | 2020-10-16 |
| CN111782050B true CN111782050B (en) | 2022-12-06 |
Family
ID=72758428
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010634979.3A Active CN111782050B (en) | 2020-07-03 | 2020-07-03 | Image processing method and apparatus, storage medium, and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111782050B (en) |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4687097B2 (en) * | 2004-12-16 | 2011-05-25 | 株式会社国際電気通信基礎技術研究所 | Gaze direction estimation apparatus, gaze direction estimation method, and program for causing computer to execute gaze direction estimation method |
| CN104571508A (en) * | 2014-12-29 | 2015-04-29 | 北京元心科技有限公司 | Method for operating data displayed by mobile terminal |
| CN106708251A (en) * | 2015-08-12 | 2017-05-24 | 天津电眼科技有限公司 | Eyeball tracking technology-based intelligent glasses control method |
| CN109828807A (en) * | 2018-12-24 | 2019-05-31 | 天津字节跳动科技有限公司 | Method, apparatus, electronic equipment and the storage medium of the small routine gesture switching page |
| CN110231871A (en) * | 2019-06-14 | 2019-09-13 | 腾讯科技（深圳）有限公司 | Page reading method, device, storage medium and electronic equipment |
| CN110287925A (en) * | 2019-06-29 | 2019-09-27 | Oppo广东移动通信有限公司 | Read flipping-over control method and Related product |
-
2020
- 2020-07-03 CN CN202010634979.3A patent/CN111782050B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111782050A (en) | 2020-10-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210326587A1 (en) | Human face and hand association detecting method and a device, and storage medium | |
| WO2022166872A1 (en) | Special-effect display method and apparatus, and device and medium | |
| JP2022542668A (en) | Target object matching method and device, electronic device and storage medium | |
| CN111246095B (en) | Method, device and equipment for controlling lens movement and storage medium | |
| US11741583B2 (en) | Face image processing method and apparatus, electronic device, and storage medium | |
| CN112488783B (en) | Image acquisition method and device and electronic equipment | |
| WO2019205605A1 (en) | Facial feature point location method and device | |
| CN111459364B (en) | Icon updating method and device and electronic equipment | |
| CN111432245B (en) | Multimedia information playing control method, device, equipment and storage medium | |
| CN110956128A (en) | Method, apparatus, electronic device, and medium for generating lane line image | |
| CN113989470A (en) | Picture display method and device, storage medium and electronic equipment | |
| WO2023151558A1 (en) | Method and apparatus for displaying images, and electronic device | |
| CN111782050B (en) | Image processing method and apparatus, storage medium, and electronic device | |
| US11810336B2 (en) | Object display method and apparatus, electronic device, and computer readable storage medium | |
| CN111586295B (en) | Image generation method and device and electronic equipment | |
| CN114897688A (en) | Video processing method, video processing device, computer equipment and medium | |
| CN114419298A (en) | Virtual object generation method, device, equipment and storage medium | |
| CN116527993A (en) | Video processing method, apparatus, electronic device, storage medium and program product | |
| CN114332224A (en) | Method, device and equipment for generating 3D target detection sample and storage medium | |
| CN113703704A (en) | Interface display method, head-mounted display device and computer readable medium | |
| CN111710046A (en) | Interaction method and device and electronic equipment | |
| CN111757005A (en) | Shooting control method and device, computer readable medium and electronic equipment | |
| CN112231023A (en) | Information display method, device, equipment and storage medium | |
| CN109842738A (en) | Method and apparatus for shooting image | |
| CN111860209B (en) | Hand recognition method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |