US12271518B2 - Gaze point calculation apparatus and driving method therefor, and electronic device - Google Patents

Gaze point calculation apparatus and driving method therefor, and electronic device Download PDF

Info

Publication number
US12271518B2
US12271518B2 US17/787,012 US202117787012A US12271518B2 US 12271518 B2 US12271518 B2 US 12271518B2 US 202117787012 A US202117787012 A US 202117787012A US 12271518 B2 US12271518 B2 US 12271518B2
Authority
US
United States
Prior art keywords
coordinates
arithmetic unit
unit assembly
gaze point
cache register
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US17/787,012
Other versions
US20230015967A1 (en
Inventor
Gaoming SUN
Guofeng Hu
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
BOE Technology Group Co Ltd
Beijing BOE Optoelectronics Technology Co Ltd
Original Assignee
BOE Technology Group Co Ltd
Beijing BOE Optoelectronics Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by BOE Technology Group Co Ltd, Beijing BOE Optoelectronics Technology Co Ltd filed Critical BOE Technology Group Co Ltd
Assigned to BOE TECHNOLOGY GROUP CO., LTD., BEIJING BOE OPTOELECTRONICS TECHNOLOGY CO., LTD. reassignment BOE TECHNOLOGY GROUP CO., LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: HU, Guofeng, SUN, Gaoming
Publication of US20230015967A1 publication Critical patent/US20230015967A1/en
Application granted granted Critical
Publication of US12271518B2 publication Critical patent/US12271518B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
    • G06F12/0806—Multiuser, multiprocessor or multiprocessing cache systems
    • G06F12/0811—Multiuser, multiprocessor or multiprocessing cache systems with multilevel cache hierarchies
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
    • G06F7/38—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation
    • G06F7/48—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices
    • G06F7/52—Multiplying; Dividing
    • G06F7/523—Multiplying only
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/60—Memory management
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/94—Hardware or software architectures specially adapted for image or video understanding
    • G06V10/955—Hardware or software architectures specially adapted for image or video understanding using specific electronic processors
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/193—Preprocessing; Feature extraction

Definitions

  • the present disclosure relates to the field of chip technologies, and in particular, to a gaze point calculation apparatus and a driving method therefor, and an electronic device.
  • augmented reality (AR) display devices and virtual reality (VR) display devices need to rely on central processing units (CPUs) of computers to realize gaze point recognition based on eye tracking, which may cause problems of huge related devices, inconvenience for carrying and being not conducive to human-computer interaction.
  • CPUs central processing units
  • a gaze point calculation apparatus includes a first cache register, a multiplexer, an arithmetic unit assembly and a state machine.
  • the multiplexer is coupled to the first cache register.
  • the arithmetic unit assembly is coupled to the multiplexer.
  • the state machine is coupled to the multiplexer and the arithmetic unit assembly.
  • the first cache register is configured to receive and store first coordinates and a plurality of calibration parameters required when second coordinates are obtained through calculation according to the first coordinates.
  • the first coordinates are position coordinates of a pupil
  • the second coordinates are position coordinates of a gaze point.
  • the arithmetic unit assembly includes a multiplier and an accumulator.
  • the multiplier is coupled to the multiplexer, and the multiplier is configured to calculate x 1 2 , a 1 x 1 , a 4 (x 1 2 ), x 1 y 1 , y 1 2 , a 2 y 1 , a 3 (x 1 y 1 ), a 5 (y 1 2 ), b 1 x 1 , b 2 y 1 , b 3 (x 1 y 1 ), b 4 (x 1 2 ) and b 5 (y 1 2 ).
  • the accumulator is coupled to the multiplier, the multiplexer and the state machine.
  • the accumulator is configured to calculate a 0 +a p x 1 +a 2 y 1 +a 3 xy 1 +a 4 x 1 2 +a 5 y 1 2 to output X G , and calculate b 0 +b 1 x 1 +b 2 y 1 +b 3 x 1 y 1 +b 4 x 1 2 +b 5 y 1 2 to output Y G .
  • the second multiplier is coupled to the multiplexer, and the second multiplier is configured to calculate b 1 x 1 , b 4 (x 1 2 ), b 2 y 1 , b 3 (x 1 y 1 ), and b 5 (y 1 2 ).
  • the second accumulator is coupled to the second multiplier, the multiplexer and the state machine.
  • the second accumulator is configured to calculate b 0 +b 1 x 1 +b 2 y 1 +b 3 x 1 y 1 +b 4 x 1 2 +b 5 y 1 2 to output Y G .
  • At least one of the first multiplier and the second multiplier is further configured to calculate x 1 2 , x 1 y 1 and y 1 2 .
  • the first multiplier is further configured to calculate a part of x 1 2 , x 1 y 1 and y 1 2
  • the second multiplier is further configured to calculate a remaining part of x 1 2 , x 1 y 1 and y 1 2 .
  • the arithmetic unit assembly is further configured to: convert the received at least one value from an original code into a complement code; calculate the at least one value converted into the complement code to obtain an intermediate calculation result; and convert the intermediate calculation result from another complement code into another original code to obtain an operation result.
  • the arithmetic unit assembly is further coupled to the first cache register, and the arithmetic unit assembly is further configured to transmit, under the control of the state machine, at least a part of intermediate values that need to be stored in intermediate values generated by the arithmetic unit assembly to the first cache register for storage.
  • the arithmetic unit assembly further includes at least one second cache register, and the arithmetic unit assembly is further configured to transmit, under the control of the state machine, at least a part of intermediate values that need to be stored in the intermediate values generated by the arithmetic unit assembly to the at least one second cache register for storage.
  • the arithmetic unit assembly is further coupled to the first cache register, and the arithmetic unit assembly is further configured to transmit, under the control of the state machine, some of at least a part of intermediate values that need to be stored in intermediate values generated by the arithmetic unit assembly to the first cache register for storage; and the arithmetic unit assembly further includes at least one second cache register, and the arithmetic unit assembly is further configured to transmit, under the control of the state machine, another of the part of intermediate values that need to be stored in the intermediate values generated by the arithmetic unit assembly to the at least one second cache register for storage.
  • the gaze point calculation apparatus further includes a normalization processing module coupled to the first cache register.
  • the normalization processing module is configured to: receive initial coordinates of the pupil in an acquired image of a human eye; and perform normalization processing on the initial coordinates to obtain the first coordinates, and transmit the first coordinates after the normalization processing to the first cache register.
  • the driving method includes: controlling, by the state machine, the multiplexer to select each time at least one value from the first cache register and transmit the at least one value to the arithmetic unit assembly, the at least one value being selected from first coordinates and a plurality of calibration parameters required when second coordinates are obtained through calculation according to the first coordinates, both the first coordinates and the plurality of calibration parameters being stored in the first cache register; performing, by the arithmetic unit assembly, a preset operation on the at least one value received each time until the second coordinates are obtained; and outputting, by the arithmetic unit assembly, the second coordinates under control of the state machine.
  • FIG. 1 D is a block diagram showing a structure of yet another gaze point calculation apparatus, in accordance with some embodiments.
  • FIG. 1 E is a block diagram showing a structure of yet another gaze point calculation apparatus, in accordance with some embodiments.
  • FIG. 2 is a diagram showing a positional relationship between a pupil and a gaze point, in accordance with some embodiments
  • FIG. 3 is a block diagram showing a structure of yet another gaze point calculation apparatus, in accordance with some embodiments.
  • FIG. 4 is a block diagram showing a structure of yet another gaze point calculation apparatus, in accordance with some embodiments.
  • FIG. 5 is a diagram showing an order of operations of a multiplier, in accordance with some embodiments.
  • FIG. 6 is a block diagram showing a structure of yet another gaze point calculation apparatus, in accordance with some embodiments.
  • FIG. 7 is a flow diagram of a method for driving a gaze point calculation apparatus, in accordance with some embodiments.
  • FIG. 8 is a block diagram showing a structure of an electronic device, in accordance with some embodiments.
  • FIG. 9 is a block diagram showing a structure of a processing module, in accordance with some embodiments.
  • the term “comprise” and other forms thereof such as the third-person singular form “comprises” and the present participle form “comprising” are construed in an open and inclusive sense, i.e., “including, but not limited to”.
  • the terms such as “one embodiment”, “some embodiments”, “exemplary embodiments”, “example”, “specific example” or “some examples” are intended to indicate that specific features, structures, materials or characteristics related to the embodiment(s) or example(s) are included in at least one embodiment or example of the present disclosure. Schematic representation of the above term does not necessarily refer to the same embodiment(s) or example(s).
  • the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any suitable manner.
  • first and second are only used for descriptive purposes, and are not to be construed as indicating or implying the relative importance or implicitly indicating the number of indicated technical features. Thus, features defined as “first” or “second” may explicitly or implicitly include one or more of the features.
  • the term “a plurality of” or “the plurality of” means two or more unless otherwise specified.
  • the expressions “coupled” and “connected” and derivatives thereof may be used.
  • the term “connected” may be used in the description of some embodiments to indicate that two or more components are in direct physical or electrical contact with each other.
  • the term “coupled” may be used in the description of some embodiments to indicate that two or more components are in direct physical or electrical contact.
  • the term “coupled” or “communicatively coupled” may also mean that two or more components are not in direct contact with each other, but still cooperate or interact with each other.
  • the embodiments disclosed herein are not necessarily limited to the contents herein.
  • phrases “at least one of A, B and C” has a same meaning as the phrase “at least one of A, B or C”, and they both include the following combinations of A, B and C: only A, only B, only C, a combination of A and B, a combination of A and C, a combination of B and C, and a combination of A, B and C.
  • a and/or B includes the following three combinations: only A, only B, and a combination of A and B.
  • the gaze point calculation apparatus 100 includes a first cache register 1 , a multiplexer 2 , an arithmetic unit assembly 3 and a state machine 4 .
  • the multiplexer 2 is coupled to the first cache register 1
  • the arithmetic unit assembly 3 is coupled to the multiplexer 2
  • the state machine 4 is coupled to the multiplexer 2 and the arithmetic unit assembly 3 .
  • the state machine 4 may be a finite-state machine.
  • the state machine 4 is configured to control the multiplexer 2 to select each time at least one value from the first cache register 1 and transmit the at least one value to the arithmetic unit assembly 3 .
  • the arithmetic unit assembly 3 is further coupled to the first cache register 1 .
  • the arithmetic unit assembly 3 is further configured to transmit, under the control of the state machine 4 , at least a part of intermediate values that need to be stored in intermediate values generated by the arithmetic unit assembly 3 to the first cache register 1 for storage.
  • the at least one value may include at least one of the intermediate values, the calibration parameters or the first coordinates.
  • the arithmetic unit assembly 3 further includes at least one second cache register 11 .
  • the arithmetic unit assembly 3 is further configured to transmit, under the control of the state machine 4 , at least a part of intermediate values that need to be stored in the intermediate values generated by the arithmetic unit assembly 3 to the at least one second cache register 11 for storage. It can be understood that, intermediate values stored in different second cache registers 11 may be different.
  • the arithmetic unit assembly 3 may include one group of arithmetic units or at least two groups of arithmetic units connected in parallel, and each group of arithmetic units may include at least two arithmetic units connected in series.
  • a second cache register 11 may be integrated in an arithmetic unit.
  • the arithmetic unit assembly 3 is couple to the first cache register 1 , and the arithmetic unit assembly 3 further includes at least one second cache register 11 .
  • the intermediate values that need to be stored generated by the arithmetic unit assembly 3 may be divided into a plurality of parts, and are stored in different cache registers (i.e., the first cache register 1 and the at least one second cache register 11 ).
  • the gaze point calculation apparatus 100 includes a plurality of hardware (e.g., the first cache register 1 , the multiplexer 2 , the arithmetic unit assembly 3 and the state machine 4 ). That is, through a hardware-based implementation solution, the gaze point calculation apparatus 100 achieves processing of the first coordinates (i.e., the position coordinates of the pupil) to output the second coordinates (i.e., the position coordinates of the gaze point).
  • the first coordinates i.e., the position coordinates of the pupil
  • the second coordinates i.e., the position coordinates of the gaze point
  • the gaze point calculation apparatus 100 may be a field programmable gate array (FPGA) chip. In some other examples, the gaze point calculation apparatus 100 may be an application specific integrated circuit (ASIC) chip.
  • FPGA field programmable gate array
  • ASIC application specific integrated circuit
  • the gaze point calculation apparatus 100 may be well integrated into an all-in-one device (e.g., an augmented reality (AR) display device or a virtual reality (VR) display device) to achieve the calculation of the gaze point.
  • an all-in-one device e.g., an augmented reality (AR) display device or a virtual reality (VR) display device
  • AR augmented reality
  • VR virtual reality
  • using the gaze point calculation apparatus 100 also has advantages of fast operation speed and low overall power consumption.
  • a process of achieving the calculation of the gaze point by using the gaze point calculation apparatus 100 will be described below through some embodiments.
  • (x 1 , y 1 ) is the first coordinates (i.e., the position coordinates of the pupil); a 0 to a 5 and b 0 to b 5 are the calibration parameters, and a 0 to a 5 and b 0 to b 5 are vectors; and (X G , Y G ) is the second coordinates (i.e., the position coordinates of the gaze point, e.g., the coordinates of the gaze point in the display area B shown in FIG. 2 ). Therefore, the gaze point calculation apparatus 100 may process the first coordinates (x 1 , y 1 ) according to the above formulas, and then output the second coordinates (X G , Y G ). That is, the gaze point calculation apparatus 100 may obtain the position coordinates of the gaze point through calculation according to the position coordinates of the pupil.
  • (x 1 , y 1 ) here may be initial coordinates (x′, y′) of a center of the pupil in an acquired image A of a human eye shown in FIG. 2 , or initial coordinates after normalization processing.
  • the gaze point calculation apparatus 100 may further include a normalization processing module 5 , which is coupled to the first cache register 1 .
  • the normalization processing module 5 is configured to receive the initial coordinates (x′, y′) of the pupil in the acquired image of the human eye, perform the normalization processing on the initial coordinates (x′, y′) to obtain the first coordinates (x 1 , y 1 ), and transmit the first coordinates (x 1 , y 1 ) to the first cache register 1 .
  • the lateral resolution of the acquired image of the human eye may be a resolution in an x-axis direction of the acquired image A of the human eye shown in FIG. 2
  • the vertical resolution of the acquired image of the human eye may be a resolution in a y-axis direction of the acquired image A of the human eye shown in FIG. 2
  • calculation results x 1 and y 1 are the first coordinates.
  • the arithmetic unit assembly 3 is configured to: convert the received at least one value from an original code into a complement code; calculate the at least one value converted into the complement code to obtain an intermediate calculation result; and convert the intermediate calculation result from a complement code into an original code to obtain an operation result.
  • operations of signed (i.e., positive or negative) values may be accurately achieved.
  • at least one calibration parameter is a negative value
  • operations such as accumulation, multiplication and root may still be accurately achieved.
  • the arithmetic unit assembly 3 may be of various structures, which, for example, include but are not limited to the possible implementations described below.
  • the signed fixed-point multiplier (or the signed fixed-point accumulator) may first convert the received value from the original code into the complement code, and then obtain the intermediate calculation result through calculation, and finally convert the intermediate calculation result from the complement code into the original code to obtain the operation result.
  • the operations of the signed (i.e., positive or negative) values may be accurately achieved.
  • the operations such as accumulation, multiplication and root may still be accurately achieved.
  • a hardware circuit structure of the signed fixed-point multiplier (or the signed fixed-point accumulator) is simpler and easier to achieve. Therefore, using the signed fixed-point multiplier (or the signed fixed-point accumulator) may facilitate to simplify a circuit structure of the arithmetic unit assembly 3 , and further facilitate to simplify a structure of the gaze point calculation apparatus 100 .
  • the multiplier 31 is coupled to the multiplexer 2 , and the multiplier 31 is configured to calculate x 1 2 , a 1 x 1 , a 4 (x 1 2 ), x 1 y 1 , y 1 2 , a 2 y 1 , a 3 (x 1 y 1 ), a 5 (y 1 2 ), b 1 x 1 , b 2 y 1 , b 3 (x 1 y 1 ), b 4 (x 1 2 ) and b 5 (y 1 2 ).
  • the multiplier 31 may sequentially perform thirteen fixed-point multiplication operations according to the sequence shown in FIG. 5 by means of time division multiplexing.
  • the accumulator 32 is coupled to the multiplier 31 , the multiplexer 2 and the state machine 4 .
  • the accumulator 32 is configured to: calculate a 0 +a 1 x 1 +a 2 y 1 +a 3 xy 1 +a 4 x 1 2 +a 5 y 1 2 , so as to output X G ; and calculate b 0 +b 1 x 1 +b 2 y 1 +b 3 x 1 y 1 +b 4 x 1 2 +b 5 y 1 2 , so as to output Y G . That is, for a frame of image of the human eye, the accumulator 32 may perform ten accumulation operations by means of time division multiplexing.
  • the multiplexer 2 may be controlled by the state machine 4 to transmit the intermediate values in the first cache register 1 to the accumulator 32 for an addition operation.
  • the state machine 4 may directly call the intermediate values in the second cache register for an addition operation, without using the multiplexer 2 .
  • step 5 it is delayed for n (n ⁇ 1) clock(s) based on step 4 , and then the multiplexer 2 , under the control of the state machine 4 , selects y 1 from the first cache register 1 , and outputs y 1 to both the multiplier port and the multiplicand port of the multiplier 31 . Then, the multiplier 31 obtains a result of y 1 2 through calculation, and the multiplexer 2 transmits the result of y 1 2 (i.e., an intermediate value) to the first cache register 1 for storage, ready for calling.
  • y 1 2 i.e., an intermediate value
  • step 8 it is delayed for n (n ⁇ 1) clock(s) based on step 7 , and then the multiplexer 2 , under the control of the state machine 4 , selects a 5 and the result of y 1 2 (i.e., the intermediate value) from the first cache register 1 , and outputs a 5 and the result of y 1 2 to the multiplier port and the multiplicand port of the multiplier 31 respectively. Then, the multiplier 31 obtains a result of a 5 (y 1 2 ) through calculation. It will be noted that, the result of a 5 (y 1 2 ) here may also be stored in the first cache register 1 as an intermediate value.
  • step 9 it is delayed for n (n ⁇ 1) clock(s) based on step 8 , and the multiplexer 2 , under the control of the state machine 4 , transmits a 0 , a 1 x 1 , a 2 y 1 , a 3 (x 1 y 1 ), a 4 (x 1 2 ) and a 5 (y 1 2 ) in the first cache register to the accumulator 32 for accumulation, so as to obtain and output X G .
  • At least one of the result of a 1 x 1 , the result of a 2 y 1 , the result of a 3 (x 1 y 1 ), the result of a 4 (x 1 2 ) and the result of a 5 (y 1 2 ) may be stored in the first cache register 1 as an intermediate value, or may be stored in the second cache register integrated in the accumulator 32 as an intermediate value, or may be directly output by the multiplier 31 to the accumulator 32 for accumulation.
  • An operation method of Y G may be similar to the above steps. Moreover, in a case where the operation result of x 1 2 , the operation result of x 1 y 1 and the operation result of y 1 2 have been stored in the first cache register, the multiplier 31 may directly perform five operations, i.e., calculating b 1 x 1 , b 2 y 1 , b 3 (x 1 y 1 ), b 4 (x 1 2 ) and b 5 (y 1 2 ).
  • the accumulator 32 accumulates b 0 , a result of b 1 x 1 , a result of b 2 y 1 , a result of b 3 (x 1 y 1 ), a result of b 4 (x 1 2 ) and a result of b 5 (y 1 2 ), so as to obtain and output Y G .
  • At least one of the result of b 1 x 1 , the result of b 2 y 1 , the result of b 3 (x 1 y 1 ), the result of b 4 (x 1 2 ) and the result of b 5 (y 1 2 ) may be stored in the first cache register 1 as an intermediate value, or may be stored in the second cache register integrated in the accumulator 32 as an intermediate value, or may be directly output by the multiplier 31 to the accumulator 32 for accumulation.
  • the second cache register may be integrated therein.
  • the second cache register may be integrated therein.
  • the state machine 4 controls the multiplexer 2 to select each time at least one value from the first cache register 1 and transmit the at least one value to the arithmetic unit assembly 3 .
  • the at least one value is selected from first coordinates and a plurality of calibration parameters required when second coordinates are obtained through calculation according to the first coordinates, and the first coordinates and the plurality of calibration parameters are stored in the first cache register.
  • the first coordinates are position coordinates of a pupil.
  • the display driver module 300 is configured to process a displayed image according to the second coordinates. For example, display of local rendering of an image around the gaze point (i.e., the second coordinates) may be achieved.
  • the displayed image may be divided into a first display portion located around the gaze point (i.e., the second coordinates) and a second display portion other than the first display portion.
  • a display effect may be improved by improving a rendering effect of the first display portion, or by controlling a display resolution of the first display portion to be greater than that of the second display portion, or by shrinking the displayed image to be an image only in the first display portion.
  • the camera is configured to acquire the image of the human eye.
  • the electronic device 400 may be a display device, which may be, for example, any device with a display function, such as the AR display device or the VR display device.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Pure & Applied Mathematics (AREA)
  • Ophthalmology & Optometry (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Image Processing (AREA)

Abstract

A gaze point calculation apparatus includes: a first cache register, a multiplexer, an arithmetic unit assembly, and a state machine. The first cache register is configured to receive and store first coordinates and a plurality of calibration parameters required when second required are obtained through calculation according to the first coordinates. The state machine is configured to control the multiplexer to select each time at least one value from the first cache register and transmit same to the arithmetic unit assembly. The arithmetic unit assembly is configured to perform a preset operation on the at least one value received each time until the second coordinates are obtained, and output the second coordinates under control of the state machine.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
This application is a national phase entry under 35 USC 371 of International Patent Application No. PCT/CN2021/093117, filed on May 11, 2021, which claims priority to Chinese Patent Application No. 202011045736.2, filed on Sep. 28, 2020, which are incorporated herein by reference in their entirety.
TECHNICAL FIELD
The present disclosure relates to the field of chip technologies, and in particular, to a gaze point calculation apparatus and a driving method therefor, and an electronic device.
BACKGROUND
Existing electronic devices, such as augmented reality (AR) display devices and virtual reality (VR) display devices, need to rely on central processing units (CPUs) of computers to realize gaze point recognition based on eye tracking, which may cause problems of huge related devices, inconvenience for carrying and being not conducive to human-computer interaction.
SUMMARY
In an aspect, a gaze point calculation apparatus is provided. The gaze point calculation apparatus includes a first cache register, a multiplexer, an arithmetic unit assembly and a state machine. The multiplexer is coupled to the first cache register. The arithmetic unit assembly is coupled to the multiplexer. The state machine is coupled to the multiplexer and the arithmetic unit assembly. The first cache register is configured to receive and store first coordinates and a plurality of calibration parameters required when second coordinates are obtained through calculation according to the first coordinates. The first coordinates are position coordinates of a pupil, and the second coordinates are position coordinates of a gaze point. The state machine is configured to control the multiplexer to select each time at least one value from the first cache register and transmit the at least one value to the arithmetic unit assembly. The at least one value is selected from the first coordinates and the plurality of calibration parameters. The arithmetic unit assembly is configured to perform a preset operation on the at least one value received each time until the second coordinates are obtained, and output the second coordinates under control of the state machine.
In some embodiments, the arithmetic unit assembly obtains the second coordinates through calculation according to following formulas:
X G =a 0 +a 1 x 1 +a 2 y 1 +a 3 x 1 y 1 +a 4 x 1 2 +a 5 y 1 2;
Y G =b 0 +b 1 x 1 +b 2 y 1 +b 3 x 1 y 1 +b 4 x 1 2 +b 5 y 1 2.
Where (x1, y1) is the first coordinates; a0 to a5 and b0 to b5 are the calibration parameters, and a0 to a5 and b0 to b5 are vectors; and (XG, YG) is the second coordinates.
In some embodiments, the arithmetic unit assembly includes a multiplier and an accumulator. The multiplier is coupled to the multiplexer, and the multiplier is configured to calculate x1 2, a1x1, a4(x1 2), x1y1, y1 2, a2y1, a3(x1y1), a5(y1 2), b1x1, b2y1, b3(x1y1), b4(x1 2) and b5(y1 2). The accumulator is coupled to the multiplier, the multiplexer and the state machine. The accumulator is configured to calculate a0+apx1+a2y1+a3xy1+a4x1 2+a5y1 2 to output XG, and calculate b0+b1x1+b2y1+b3x1y1+b4x1 2+b5y1 2 to output YG.
In some embodiments, the arithmetic unit assembly includes a first multiplier, a first accumulator, a second multiplier, and a second accumulator. The first multiplier is coupled to the multiplexer, and the first multiplier is configured to calculate a1x1, a4(x1 2), a2y1, a3(x1y1), and a5(y1 2). The first accumulator is coupled to the first multiplier, the multiplexer and the state machine. The first accumulator is configured to calculate a0+a1x1+a2y1+a3xy1+a4x1 2+a5y1 2 to output XG. The second multiplier is coupled to the multiplexer, and the second multiplier is configured to calculate b1x1, b4(x1 2), b2y1, b3(x1y1), and b5(y1 2). The second accumulator is coupled to the second multiplier, the multiplexer and the state machine. The second accumulator is configured to calculate b0+b1x1+b2y1+b3x1y1+b4x1 2+b5y1 2 to output YG. At least one of the first multiplier and the second multiplier is further configured to calculate x1 2, x1y1 and y1 2. Or, the first multiplier is further configured to calculate a part of x1 2, x1y1 and y1 2, and the second multiplier is further configured to calculate a remaining part of x1 2, x1y1 and y1 2.
In some embodiments, the arithmetic unit assembly is further configured to: convert the received at least one value from an original code into a complement code; calculate the at least one value converted into the complement code to obtain an intermediate calculation result; and convert the intermediate calculation result from another complement code into another original code to obtain an operation result.
In some embodiments, the arithmetic unit assembly is further coupled to the first cache register, and the arithmetic unit assembly is further configured to transmit, under the control of the state machine, at least a part of intermediate values that need to be stored in intermediate values generated by the arithmetic unit assembly to the first cache register for storage. Or, the arithmetic unit assembly further includes at least one second cache register, and the arithmetic unit assembly is further configured to transmit, under the control of the state machine, at least a part of intermediate values that need to be stored in the intermediate values generated by the arithmetic unit assembly to the at least one second cache register for storage.
In some embodiments, the arithmetic unit assembly is further coupled to the first cache register, and the arithmetic unit assembly is further configured to transmit, under the control of the state machine, some of at least a part of intermediate values that need to be stored in intermediate values generated by the arithmetic unit assembly to the first cache register for storage; and the arithmetic unit assembly further includes at least one second cache register, and the arithmetic unit assembly is further configured to transmit, under the control of the state machine, another of the part of intermediate values that need to be stored in the intermediate values generated by the arithmetic unit assembly to the at least one second cache register for storage.
In some embodiments, the gaze point calculation apparatus further includes a normalization processing module coupled to the first cache register. The normalization processing module is configured to: receive initial coordinates of the pupil in an acquired image of a human eye; and perform normalization processing on the initial coordinates to obtain the first coordinates, and transmit the first coordinates after the normalization processing to the first cache register.
In some embodiments, the normalization processing module performs the normalization processing on the initial coordinates according to following formulas:
x 1 =x′/a lateral resolution of the acquired image of the human eye;
y 1 =y′/a vertical resolution of the acquired image of the human eye.
Where (x′, y′) is the initial coordinates, and (x1, y1) is the first coordinates.
In some embodiments, the gaze point calculation apparatus is a field programmable gate array (FPGA) chip or an application specific integrated circuit (ASIC) chip.
In another aspect, an electronic device is provided. The electronic device includes the gaze point calculation apparatus according to any of the above embodiments.
In some embodiments, the electronic device includes a pupil coordinate extraction module and a display driver module that are coupled to the gaze point calculation apparatus. The pupil coordinate extraction module is configured to acquire an image of a human eye, and determine initial coordinates of the pupil in the image of the human eye according to the image of the human eye, the initial coordinates or initial coordinates after normalization processing being the first coordinates, and the first coordinates being the position coordinates of the pupil. The gaze point calculation apparatus is configured to obtain the second coordinates through calculation according to the first coordinates, the second coordinates being the position coordinates of the gaze point. The display driver module is configured to process a displayed image according to the second coordinates.
In some embodiments, the pupil coordinate extraction module includes a camera and a processing module. The camera is configured to acquire the image of the human eye. The processing module is configured to determine, according to the acquired image of the human eye, the initial coordinates of the pupil in the image of the human eye based on an image processing algorithm.
In some embodiments, the electronic device includes an augmented reality (AR) display device or a virtual reality (VR) display device.
In yet another aspect, a driving method for a gaze point calculation apparatus is provided. The gaze point calculation apparatus includes a first cache register, a multiplexer coupled to the first cache register, an arithmetic unit assembly coupled to the multiplexer, and a state machine coupled to the multiplexer and the arithmetic unit assembly. The driving method includes: controlling, by the state machine, the multiplexer to select each time at least one value from the first cache register and transmit the at least one value to the arithmetic unit assembly, the at least one value being selected from first coordinates and a plurality of calibration parameters required when second coordinates are obtained through calculation according to the first coordinates, both the first coordinates and the plurality of calibration parameters being stored in the first cache register; performing, by the arithmetic unit assembly, a preset operation on the at least one value received each time until the second coordinates are obtained; and outputting, by the arithmetic unit assembly, the second coordinates under control of the state machine.
In some embodiments, the second coordinates are obtained according to following formulas:
X G =a 0 +a 1 x 1 +a 2 y 1 +a 3 x 1 y 1 +a 4 x 1 2 +a 5 y 1 2;
Y G =b 0 +b 1 x 1 +b 2 y 1 +b 3 x 1 y 1 +b 4 x 1 2 +b 5 y 1 2.
Where (x1, y1) is the first coordinates; a0 to a5 and b0 to b5 are the calibration parameters, and a0 to a5 and b0 to b5 are vectors; and (XG, YG) is the second coordinates.
In some embodiments, the driving method further includes: converting the received at least one value from an original code into a complement code; calculating the at least one value converted into the complement code to obtain an intermediate calculation result; and converting the intermediate calculation result from another complement code into another original code to obtain an operation result.
In some embodiments, the driving method further includes: receiving initial coordinates of the pupil in an acquired image of a human eye; performing normalization processing on the initial coordinates to obtain the first coordinates, and transmitting the first coordinates after the normalization processing to the first cache register.
In some embodiments, the normalization processing is performed according to following formulas:
x 1 =x′/lateral resolution of the acquired image of the human eye;
y 1 =y′/vertical resolution of the acquired image of the human eye.
Where (x′, y′) is the initial coordinates, and (x1, y1) is the first coordinates.
BRIEF DESCRIPTION OF THE DRAWINGS
In order to describe technical solutions in the present disclosure more clearly, accompanying drawings to be used in some embodiments of the present disclosure will be introduced briefly below. Obviously, the accompanying drawings to be described below are merely accompanying drawings of some embodiments of the present disclosure, and a person of ordinary skill in the art may obtain other drawings according to these drawings. In addition, the accompanying drawings to be described below may be regarded as schematic diagrams, and are not limitations on actual sizes of products, actual processes of methods and actual timings of signals to which the embodiments of the present disclosure relate.
FIG. 1A is a block diagram showing a structure of a gaze point calculation apparatus, in accordance with some embodiments;
FIG. 1B is a block diagram showing a structure of another gaze point calculation apparatus, in accordance with some embodiments;
FIG. 1C is a block diagram showing a structure of yet another gaze point calculation apparatus, in accordance with some embodiments;
FIG. 1D is a block diagram showing a structure of yet another gaze point calculation apparatus, in accordance with some embodiments;
FIG. 1E is a block diagram showing a structure of yet another gaze point calculation apparatus, in accordance with some embodiments;
FIG. 2 is a diagram showing a positional relationship between a pupil and a gaze point, in accordance with some embodiments;
FIG. 3 is a block diagram showing a structure of yet another gaze point calculation apparatus, in accordance with some embodiments;
FIG. 4 is a block diagram showing a structure of yet another gaze point calculation apparatus, in accordance with some embodiments;
FIG. 5 is a diagram showing an order of operations of a multiplier, in accordance with some embodiments;
FIG. 6 is a block diagram showing a structure of yet another gaze point calculation apparatus, in accordance with some embodiments;
FIG. 7 is a flow diagram of a method for driving a gaze point calculation apparatus, in accordance with some embodiments;
FIG. 8 is a block diagram showing a structure of an electronic device, in accordance with some embodiments; and
FIG. 9 is a block diagram showing a structure of a processing module, in accordance with some embodiments.
 DETAILED DESCRIPTION
Technical solutions in some embodiments of the present disclosure will be described clearly and completely with reference to the accompanying drawings below. Obviously, the described embodiments are merely some but not all embodiments of the present disclosure. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present disclosure shall be included in the protection scope of the present disclosure.
Unless the context requires otherwise, throughout the description and the claims, the term “comprise” and other forms thereof such as the third-person singular form “comprises” and the present participle form “comprising” are construed in an open and inclusive sense, i.e., “including, but not limited to”. In the description, the terms such as “one embodiment”, “some embodiments”, “exemplary embodiments”, “example”, “specific example” or “some examples” are intended to indicate that specific features, structures, materials or characteristics related to the embodiment(s) or example(s) are included in at least one embodiment or example of the present disclosure. Schematic representation of the above term does not necessarily refer to the same embodiment(s) or example(s). In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any suitable manner.
Hereinafter, the terms “first” and “second” are only used for descriptive purposes, and are not to be construed as indicating or implying the relative importance or implicitly indicating the number of indicated technical features. Thus, features defined as “first” or “second” may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, the term “a plurality of” or “the plurality of” means two or more unless otherwise specified.
In the description of some embodiments, the expressions “coupled” and “connected” and derivatives thereof may be used. For example, the term “connected” may be used in the description of some embodiments to indicate that two or more components are in direct physical or electrical contact with each other. For another example, the term “coupled” may be used in the description of some embodiments to indicate that two or more components are in direct physical or electrical contact. However, the term “coupled” or “communicatively coupled” may also mean that two or more components are not in direct contact with each other, but still cooperate or interact with each other. The embodiments disclosed herein are not necessarily limited to the contents herein.
The phrase “at least one of A, B and C” has a same meaning as the phrase “at least one of A, B or C”, and they both include the following combinations of A, B and C: only A, only B, only C, a combination of A and B, a combination of A and C, a combination of B and C, and a combination of A, B and C.
The phrase “A and/or B” includes the following three combinations: only A, only B, and a combination of A and B.
As used herein, the term “if”, depending on the context, is optionally construed as “when”, “in a case where”, “in response to determining” or “in response to detecting”. Similarly, depending on the context, the phrase “if it is determined” or “if [a stated condition or event] is detected” is optionally construed as “in a case where it is determined”, “in response to determining”, “in a case where [the stated condition or event] is detected”, or “in response to detecting [the stated condition or event]”.
The use of the phrase “applicable to” or “configured to” herein means an open and inclusive language, which does not exclude devices that are applicable to or configured to perform additional tasks or steps.
In addition, the use of the phrase “based on” is meant to be open and inclusive, since a process, step, calculation or other action that is “based on” one or more of the stated conditions or values may, in practice, be based on additional conditions or values exceeding those stated.
Some embodiments of the present disclosure provide a gaze point calculation apparatus 100. As shown in FIG. 1A, the gaze point calculation apparatus 100 includes a first cache register 1, a multiplexer 2, an arithmetic unit assembly 3 and a state machine 4. The multiplexer 2 is coupled to the first cache register 1, the arithmetic unit assembly 3 is coupled to the multiplexer 2, and the state machine 4 is coupled to the multiplexer 2 and the arithmetic unit assembly 3. The state machine 4 may be a finite-state machine.
It will be noted that, the state machine 4 may determine, according to its own operation state, an overall operation progress of the gaze point calculation apparatus 100, so that it may be known that which values are stored in the first cache register 1, and in the stored values, which values are used, which values are not used, and which values require a second operation.
In some other examples, referring to FIG. 1B, the state machine 4 may be coupled to the first cache register 1. In this case, the state machine 4 may directly perform information interaction with the first cache register 1, so that it may be quickly known that which values are stored in the first cache register 1, and in the stored values, which values are used, which values are not used, and which values require a second operation.
The first cache register 1 is configured to receive and store first coordinates and a plurality of calibration parameters required when second coordinates are obtained through calculation according to the first coordinates. The first coordinates are position coordinates of a pupil, and the second coordinates are position coordinates of a gaze point. When the position coordinates of the pupil are determined, the pupil may be simplified into a point for positioning. For example, the point may be a center point of the pupil, which is not limited in the embodiments of the present disclosure. In some other examples, the point may be a point at a position other than the center point of the pupil, such as a point on a border of the pupil.
The state machine 4 is configured to control the multiplexer 2 to select each time at least one value from the first cache register 1 and transmit the at least one value to the arithmetic unit assembly 3.
The arithmetic unit assembly 3 is configured to perform a preset operation (e.g., a multiplication operation, a root operation, or an addition operation) on the at least one value received each time until the second coordinates are obtained, and output the second coordinates under control of the state machine 4. For example, in a case where the second coordinates are (XG, YG), under the control of the state machine 4, one of XG and YG may be obtained through calculation and output first, and then the other one is obtained through calculation and output; alternatively, XG and YG are sequentially obtained through calculation, and are then output at a same time.
It will be noted that, one or more intermediate values may be generated in a whole process of obtaining the second coordinates through calculation. When performing the preset operation, the arithmetic unit assembly 3 may perform a second operation on these intermediate values, so as to finally obtain the second coordinates. Processing of some intermediate values will be described below with reference to some examples.
In some examples, as shown in FIG. 1C, the arithmetic unit assembly 3 is further coupled to the first cache register 1. In this case, the arithmetic unit assembly 3 is further configured to transmit, under the control of the state machine 4, at least a part of intermediate values that need to be stored in intermediate values generated by the arithmetic unit assembly 3 to the first cache register 1 for storage. It can be understood that, in a case where the first cache register 1 has stored the intermediate values therein, when the multiplexer 2 subsequently selects the at least one value from the first cache register 1, the at least one value may include at least one of the intermediate values, the calibration parameters or the first coordinates.
In some other examples, as shown in FIG. 1D, the arithmetic unit assembly 3 further includes at least one second cache register 11. In this case, the arithmetic unit assembly 3 is further configured to transmit, under the control of the state machine 4, at least a part of intermediate values that need to be stored in the intermediate values generated by the arithmetic unit assembly 3 to the at least one second cache register 11 for storage. It can be understood that, intermediate values stored in different second cache registers 11 may be different.
The arithmetic unit assembly 3 may include one group of arithmetic units or at least two groups of arithmetic units connected in parallel, and each group of arithmetic units may include at least two arithmetic units connected in series. A second cache register 11 may be integrated in an arithmetic unit.
In yet some other examples, as shown in FIG. 1E, the arithmetic unit assembly 3 is couple to the first cache register 1, and the arithmetic unit assembly 3 further includes at least one second cache register 11. In this case, the intermediate values that need to be stored generated by the arithmetic unit assembly 3 may be divided into a plurality of parts, and are stored in different cache registers (i.e., the first cache register 1 and the at least one second cache register 11).
The gaze point calculation apparatus 100 includes a plurality of hardware (e.g., the first cache register 1, the multiplexer 2, the arithmetic unit assembly 3 and the state machine 4). That is, through a hardware-based implementation solution, the gaze point calculation apparatus 100 achieves processing of the first coordinates (i.e., the position coordinates of the pupil) to output the second coordinates (i.e., the position coordinates of the gaze point).
In some examples, the gaze point calculation apparatus 100 may be a field programmable gate array (FPGA) chip. In some other examples, the gaze point calculation apparatus 100 may be an application specific integrated circuit (ASIC) chip.
It is worth pointing out that, in terms of eye tracking, calculation of the gaze point achieved through a central processing unit (CPU) of a computer may result in problems of huge related devices, inconvenience for carrying, and being not conducive to human-computer interaction. However, the gaze point calculation apparatus 100 provided by the embodiments of the present disclosure may be well integrated into an all-in-one device (e.g., an augmented reality (AR) display device or a virtual reality (VR) display device) to achieve the calculation of the gaze point. As a result, it is possible to solve the technical problems of huge related devices, inconvenience for carrying, and being not conducive to the human-computer interaction. In addition, using the gaze point calculation apparatus 100 also has advantages of fast operation speed and low overall power consumption.
A process of achieving the calculation of the gaze point by using the gaze point calculation apparatus 100 will be described below through some embodiments.
In some embodiments, referring to FIGS. 1A to 1E and 2 , the gaze point calculation apparatus 100 may obtain the second coordinates through calculation according to the following formulas:
X G =a 0 +a 1 x 1 +a 2 y 1 +a 3 x 1 y 1 +a 4 x 1 2 +a 5 y 1 2;
Y G =b 0 +b 1 x 1 +b 2 y 1 +b 3 x 1 y 1 +b 4 x 1 2 +b 5 y 1 2.
Where (x1, y1) is the first coordinates (i.e., the position coordinates of the pupil); a0 to a5 and b0 to b5 are the calibration parameters, and a0 to a5 and b0 to b5 are vectors; and (XG, YG) is the second coordinates (i.e., the position coordinates of the gaze point, e.g., the coordinates of the gaze point in the display area B shown in FIG. 2 ). Therefore, the gaze point calculation apparatus 100 may process the first coordinates (x1, y1) according to the above formulas, and then output the second coordinates (XG, YG). That is, the gaze point calculation apparatus 100 may obtain the position coordinates of the gaze point through calculation according to the position coordinates of the pupil.
It will be noted that, (x1, y1) here may be initial coordinates (x′, y′) of a center of the pupil in an acquired image A of a human eye shown in FIG. 2 , or initial coordinates after normalization processing.
Based on this, in some examples, as shown in FIG. 3 , the gaze point calculation apparatus 100 may further include a normalization processing module 5, which is coupled to the first cache register 1. The normalization processing module 5 is configured to receive the initial coordinates (x′, y′) of the pupil in the acquired image of the human eye, perform the normalization processing on the initial coordinates (x′, y′) to obtain the first coordinates (x1, y1), and transmit the first coordinates (x1, y1) to the first cache register 1.
For example, the normalization processing module 5 may perform the normalization processing on the initial coordinates (x′, y′) according to the following formulas:
x 1 =x′/a lateral resolution of the acquired image of the human eye;
y 1 =y′/a vertical resolution of the acquired image of the human eye.
Where the lateral resolution of the acquired image of the human eye may be a resolution in an x-axis direction of the acquired image A of the human eye shown in FIG. 2 , and the vertical resolution of the acquired image of the human eye may be a resolution in a y-axis direction of the acquired image A of the human eye shown in FIG. 2 , and calculation results x1 and y1 are the first coordinates.
Through the above solution, it is possible to facilitate to obtain an accurate second coordinates through calculation, and in turn, the position of the gaze point may be accurately determined.
In some embodiments, the arithmetic unit assembly 3 is configured to: convert the received at least one value from an original code into a complement code; calculate the at least one value converted into the complement code to obtain an intermediate calculation result; and convert the intermediate calculation result from a complement code into an original code to obtain an operation result. With such a configuration, operations of signed (i.e., positive or negative) values may be accurately achieved. For example, in a case where at least one calibration parameter is a negative value, operations such as accumulation, multiplication and root may still be accurately achieved.
The arithmetic unit assembly 3 may be of various structures, which, for example, include but are not limited to the possible implementations described below.
In a possible implementation, as shown in FIG. 4 , the arithmetic unit assembly 3 may include a multiplier 31 and an accumulator 32. For example, the multiplier 31 may be a signed fixed-point multiplier, and the accumulator 32 may be a signed fixed-point accumulator.
During operation, the signed fixed-point multiplier (or the signed fixed-point accumulator) may first convert the received value from the original code into the complement code, and then obtain the intermediate calculation result through calculation, and finally convert the intermediate calculation result from the complement code into the original code to obtain the operation result. In this way, the operations of the signed (i.e., positive or negative) values may be accurately achieved. For example, in the case where at least one calibration parameter is a negative value, the operations such as accumulation, multiplication and root may still be accurately achieved.
In addition, it is worth pointing out that, compared with a hardware circuit structure of a signed floating-point multiplier (or a signed floating-point accumulator), a hardware circuit structure of the signed fixed-point multiplier (or the signed fixed-point accumulator) is simpler and easier to achieve. Therefore, using the signed fixed-point multiplier (or the signed fixed-point accumulator) may facilitate to simplify a circuit structure of the arithmetic unit assembly 3, and further facilitate to simplify a structure of the gaze point calculation apparatus 100.
The multiplier 31 is coupled to the multiplexer 2, and the multiplier 31 is configured to calculate x1 2, a1x1, a4(x1 2), x1y1, y1 2, a2y1, a3(x1y1), a5(y1 2), b1x1, b2y1, b3(x1y1), b4(x1 2) and b5(y1 2). For example, for a frame of image of the human eye, the multiplier 31 may sequentially perform thirteen fixed-point multiplication operations according to the sequence shown in FIG. 5 by means of time division multiplexing.
It will be noted that, the intermediate values that need to be stored may include an operation result of x1 2, an operation result of x1y1, and an operation result of y1 2. These intermediate values may be stored in the first cache register 1 (as shown in FIG. 4 ), or may be stored in a second cache register integrated in the multiplier 31. In a case where these intermediate values are stored in the first cache register 1, as shown in FIG. 4 , the multiplexer 2 may be controlled by the state machine 4 to transmit the intermediate values in the first cache register 1 again to the multiplier 31 for a multiplication operation. In a case where these intermediate values are stored in the second cache register integrated in the multiplier 31, the state machine 4 may directly call the intermediate values in the second cache register for a multiplication operation, without using the multiplexer 2.
The accumulator 32 is coupled to the multiplier 31, the multiplexer 2 and the state machine 4. The accumulator 32 is configured to: calculate a0+a1x1+a2y1+a3xy1+a4x1 2+a5y1 2, so as to output XG; and calculate b0+b1x1+b2y1+b3x1y1+b4x1 2+b5y1 2, so as to output YG. That is, for a frame of image of the human eye, the accumulator 32 may perform ten accumulation operations by means of time division multiplexing.
It will be noted that, the intermediate values may further include: an operation result of a1x1, an operation result of a2y1, an operation result of a3(x1y1), an operation result of a4(x1 2), an operation result of a5(y1 2), an operation result of b1x1, an operation result of b2y1, an operation result of b3(x1y1), an operation result of b4(x1 2), and an operation result of b5(y1 2). These intermediate values may be stored in the first cache register 1 (as shown in FIG. 4 ), or may be stored in a second cache register integrated in the accumulator 32, or may be directly transmitted to the accumulator 32 for accumulation. In a case where these intermediate values are stored in the first cache register 1, as shown in FIG. 4 , the multiplexer 2 may be controlled by the state machine 4 to transmit the intermediate values in the first cache register 1 to the accumulator 32 for an addition operation. In a case where these intermediate values are stored in the second cache register integrated in the accumulator 32, the state machine 4 may directly call the intermediate values in the second cache register for an addition operation, without using the multiplexer 2.
In practical applications, for example, according to the example shown in FIG. 4 , the multiplier 31 and the accumulator 32 may perform calculation according to the following steps.
In step 1, the multiplexer 2, under the control of the state machine 4, selects x1 from the first cache register 1, and outputs x1 to both a multiplier port and a multiplicand port of the multiplier 31. Then, the multiplier 31 obtains a result of x1 2 through calculation, and the multiplexer 2 transmits the result of x1 2 (i.e., an intermediate value) to the first cache register 1 for storage, ready for calling.
In step 2, it is delayed for n (n being greater than or equal to 1 (n≥1)) clock(s) based on step 1, and then the multiplexer 2, under the control of the state machine 4, selects a1 and x1 from the first cache register 1, and outputs a1 and x1 to a multiplier port and a multiplicand port of the multiplier 31 respectively. Then, the multiplier 31 obtains a result of a1x1 through calculation. It will be noted that, the result of a1x1 here may also be stored in the first cache register 1 as an intermediate value.
In step 3, it is delayed for n (n≥1) clock(s) based on step 2, and then the multiplexer 2, under the control of the state machine 4, selects a4 and the operation result of x1 2 (i.e., the intermediate value) from the first cache register 1, and outputs a4 and the operation result of x1 2 to the multiplier port and the multiplicand port of the multiplier 31 respectively. Then, the multiplier 31 obtains a result of a4(x1 2) through calculation. It will be noted that, the result of a4(x1 2) here may also be stored in the first cache register 1 as an intermediate value.
In step 4, it is delayed for n (n≥1) clock(s) based on step 3, and then the multiplexer 2, under the control of the state machine 4, selects x1 and y1 from the first cache register 1, and outputs x1 and y1 to the multiplier port and the multiplicand port of the multiplier 31 respectively. Then, the multiplier 31 obtains a result of x1y1 through calculation, and the multiplexer 2 transmits the result of x1y1 (i.e., an intermediate value) to the first cache register 1 for storage, ready for calling.
In step 5, it is delayed for n (n≥1) clock(s) based on step 4, and then the multiplexer 2, under the control of the state machine 4, selects y1 from the first cache register 1, and outputs y1 to both the multiplier port and the multiplicand port of the multiplier 31. Then, the multiplier 31 obtains a result of y1 2 through calculation, and the multiplexer 2 transmits the result of y1 2 (i.e., an intermediate value) to the first cache register 1 for storage, ready for calling.
In step 6, it is delayed for n (n≥1) clock(s) based on step 5, and then the multiplexer 2, under the control of the state machine 4, selects a2 and y1 from the first cache register 1, and outputs a2 and y1 to the multiplier port and the multiplicand port of the multiplier 31 respectively. Then, the multiplier 31 obtains a result of a2y1 through calculation. It will be noted that, the result of a2y1 here may also be stored in the first cache register 1 as an intermediate value.
In step 7, it is delayed for n (n≥1) clock(s) based on step 6, and then the multiplexer 2, under the control of the state machine 4, selects a3 and the result of x1y1 (i.e., the intermediate value) from the first cache register 1, and outputs a3 and the result of x1y1 to the multiplier port and the multiplicand port of the multiplier 31 respectively. Then, the multiplier 31 obtains a result of a3(x1y1) through calculation. It will be noted that, the result of a3(x1y1) here may also be stored in the first cache register 1 as an intermediate value.
In step 8, it is delayed for n (n≥1) clock(s) based on step 7, and then the multiplexer 2, under the control of the state machine 4, selects a5 and the result of y1 2 (i.e., the intermediate value) from the first cache register 1, and outputs a5 and the result of y1 2 to the multiplier port and the multiplicand port of the multiplier 31 respectively. Then, the multiplier 31 obtains a result of a5(y1 2) through calculation. It will be noted that, the result of a5(y1 2) here may also be stored in the first cache register 1 as an intermediate value.
In step 9, it is delayed for n (n≥1) clock(s) based on step 8, and the multiplexer 2, under the control of the state machine 4, transmits a0, a1x1, a2y1, a3(x1y1), a4(x1 2) and a5(y1 2) in the first cache register to the accumulator 32 for accumulation, so as to obtain and output XG.
It can be understood that, at least one of the result of a1x1, the result of a2y1, the result of a3(x1y1), the result of a4(x1 2) and the result of a5(y1 2) may be stored in the first cache register 1 as an intermediate value, or may be stored in the second cache register integrated in the accumulator 32 as an intermediate value, or may be directly output by the multiplier 31 to the accumulator 32 for accumulation.
An operation method of YG may be similar to the above steps. Moreover, in a case where the operation result of x1 2, the operation result of x1y1 and the operation result of y1 2 have been stored in the first cache register, the multiplier 31 may directly perform five operations, i.e., calculating b1x1, b2y1, b3(x1y1), b4(x1 2) and b5(y1 2). Thereafter, the accumulator 32 accumulates b0, a result of b1x1, a result of b2y1, a result of b3(x1y1), a result of b4(x1 2) and a result of b5(y1 2), so as to obtain and output YG.
Likewise, at least one of the result of b1x1, the result of b2y1, the result of b3(x1y1), the result of b4(x1 2) and the result of b5(y1 2) may be stored in the first cache register 1 as an intermediate value, or may be stored in the second cache register integrated in the accumulator 32 as an intermediate value, or may be directly output by the multiplier 31 to the accumulator 32 for accumulation.
It is worth pointing out that, in a case where the result of a1x1, the result of a2y1, the result of a3(x1y1), the result of a4(x1 2), and the result of a5(y1 2), as well as the result of b1x1, the result of b2y1, the result of b3(x1y1), the result of b4(x1 2), and the result of b5(y1 2) are directly transmitted by the multiplier 31 to the accumulator 32 for accumulation, the multiplier 31 and the accumulator 32 not only achieve time division multiplexing separately, but also achieve a serial-parallel combination as a whole (i.e., in this case, the multiplier 31 and the accumulator 32 may work in parallel at a same time, and the multiplier 31 may also output the calculation results to the accumulator 32 to achieve serial working). As a result, operation capabilities of the multiplier 31 and the accumulator 32 may be fully utilized, and the operation speed may be improved.
In another possible implementation, as shown in FIG. 6 , the arithmetic unit assembly 3 may include a first multiplier 31A, a first accumulator 32A, a second multiplier 31B and a second accumulator 32B. The first multiplier 31A and the second multiplier 31B may be signed fixed-point multipliers, and the first accumulator 32A and the second accumulator 32B may be signed fixed-point accumulators. For operation processes and beneficial effects of the signed fixed-point multipliers and the signed fixed-point accumulators, details will not be repeated here.
The first multiplier 31A is coupled to the multiplexer 2, and the first accumulator 32A is coupled to the first multiplier 31A, the multiplexer 2 and the state machine 4. The first multiplier 31A is configured to calculate x1 2, a1x1, a4(x1 2), a2y1, a3(x1y1), and a5(y1 2). The first accumulator 32A is configured to calculate a0+a1x1+a2y1+a3xy1+a4x1 2+a5y1 2, so as to output XG.
The second multiplier 31B is coupled to the multiplexer 2, and the second accumulator 32B is coupled to the second multiplier 31B, the multiplexer 2 and the state machine 4. The second multiplier 31B is configured to calculate x1 2, b1x1, b4(x1 2), x1y1, y9 2, b2y1, b3(x1y1), and b5(y1 2). The second accumulator 32B is configured to calculate b0+b1x1+b2y1+b3x1y1+b4x1 2+b5y1 2, so as to output YG.
In this possible implementation, XG may be obtained through calculation by using the first multiplier 31A and the first accumulator 32A, and at a same time, YG may be obtained through calculation by using the second multiplier 31B and the second accumulator 32B. That is, in this way, synchronous processing of an abscissa and an ordinate may be realized, thereby facilitating to improve an overall operation speed of the arithmetic unit assembly 3.
The intermediate values stored in the first cache register 1 include at least an operation result of x1 2, an operation result of x1y1, and an operation result of y1 2. In this possible implementation, it will be noted that, it is not limited herein to the example in which x1 2, x1y1 and y1 2 are calculated twice by using the above two multipliers (i.e., the first multiplier 31A and the second multiplier 31B). That is, x1 2, x1y1 and y1 2 may be calculated by using only one multiplier (e.g., the first multiplier 31A or the second multiplier 31B); alternatively, a part of x1 2, x1y1 and y1 2 (e.g., x1 2 and x1y1) may be calculated by using a multiplier (e.g., the first multiplier 31A), and a remaining part of x1 2, x1y1 and y1 2 (e.g., y1 2) may be calculated by using the other multiplier (e.g., the second multiplier 31B). As a result, the operation result of x1 2, the operation result of x1y1 and the operation result of y1 2 may be obtained, and then are transmitted to and stored in the first cache register 1 for sharing. In this way, it is possible to facilitate to simplify an operation process and improve an operation efficiency.
Likewise, it can be understood that, for at least one of the first multiplier 31A, the second multiplier 31B, the first accumulator 32A or the second accumulator 32B, the second cache register may be integrated therein. For a function and beneficial effect of integrating the second cache register, reference may be made to the above description, and details will not be repeated herein.
Some embodiments of the present disclosure provide a driving method for a gaze point calculation apparatus. As shown in FIGS. 1A to 1C, the gaze point calculation apparatus 100 includes a first cache register 1, a multiplexer 2 coupled to the first cache register 1, an arithmetic unit assembly 3 coupled to the multiplexer 2, and a state machine 4 coupled to the first cache register 1, the multiplexer 2 and the arithmetic unit assembly 3. As shown in FIG. 7 , the driving method includes the following steps.
In S1, the state machine 4 controls the multiplexer 2 to select each time at least one value from the first cache register 1 and transmit the at least one value to the arithmetic unit assembly 3. The at least one value is selected from first coordinates and a plurality of calibration parameters required when second coordinates are obtained through calculation according to the first coordinates, and the first coordinates and the plurality of calibration parameters are stored in the first cache register. The first coordinates are position coordinates of a pupil.
In S2, the arithmetic unit assembly 3 performs a preset operation on the at least one value received each time until second coordinates are obtained, and outputs the second coordinates under control of the state machine 4. The second coordinates are position coordinates of a gaze point (e.g., the position coordinates of the gaze point in the display area B shown in FIG. 2 ).
Beneficial effects of the driving method are same as those of the above gaze point calculation apparatus, and details will not be repeated here.
Some embodiments of the present disclosure provide an electronic device 400. As shown in FIG. 8 , the electronic device 400 includes a pupil coordinate extraction module 200, the gaze point calculation apparatus 100 as described in any of the above embodiments, and a display driver module 300 that are coupled in sequence.
The pupil coordinate extraction module 200 is configured to acquire an image of a human eye (e.g., the image A of the human eye shown in FIG. 2 ), and determine initial coordinates (x′, y′) of a pupil in the image of the human eye according to the image of the human eye. The initial coordinates (x′, y′) or initial coordinates after normalization processing are first coordinates (x1, y1), i.e., position coordinates of the pupil.
The gaze point calculation apparatus 100 is configured to obtain second coordinates (XG, YG) through calculation according to the first coordinates (x1, y1). The second coordinates (XG, YG) are position coordinates of a gaze point (e.g., the position coordinates of the gaze point in the display area B shown in FIG. 2 ).
The display driver module 300 is configured to process a displayed image according to the second coordinates. For example, display of local rendering of an image around the gaze point (i.e., the second coordinates) may be achieved. For example, the displayed image may be divided into a first display portion located around the gaze point (i.e., the second coordinates) and a second display portion other than the first display portion. During display, for example, a display effect may be improved by improving a rendering effect of the first display portion, or by controlling a display resolution of the first display portion to be greater than that of the second display portion, or by shrinking the displayed image to be an image only in the first display portion.
Based on this, for example, as shown in FIG. 8 , the pupil coordinate extraction module 200 may include a camera 201 and a processing module 202.
The camera is configured to acquire the image of the human eye.
The processing module is configured to determine, according to the acquired image of the human eye, the initial coordinates of the pupil in the image of the human eye based on an image processing algorithm.
For example, referring to FIG. 9 , the processing module 202 may include an image preprocessing module 2021 and a pupil positioning module 2022. The image preprocessing module 2021 may perform pre-processing (e.g., binarization processing) on the image of the human eye, and then the pupil positioning module 2022 may determine a position of the pupil based on a processed image of the human eye, i.e., the initial coordinates of the pupil in the image of the human eye.
It will be noted that, the electronic device 400 may be a display device, which may be, for example, any device with a display function, such as the AR display device or the VR display device.
The foregoing descriptions are merely specific implementations of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or replacements that a person skilled in the art could conceive of within the technical scope of the present disclosure shall be included in the protection scope of the present disclosure. Therefore, the scope of the present disclosure shall be subject to the protection scope of the claims.

Claims (20)

What is claimed is:
1. A gaze point calculation apparatus, comprising:
a first cache register configured to receive and store first coordinates and a plurality of calibration parameters required when second coordinates are obtained through calculation according to the first coordinates, the first coordinates being position coordinates of a pupil, and the second coordinates being position coordinates of a gaze point;
a multiplexer coupled to the first cache register;
an arithmetic unit assembly coupled to the multiplexer; and
a state machine coupled to the multiplexer and the arithmetic unit assembly, wherein
the state machine is configured to control the multiplexer to select each time at least one value from the first cache register and transmit the at least one value to the arithmetic unit assembly; the at least one value is selected from the first coordinates and the plurality of calibration parameters; and
the arithmetic unit assembly is configured to perform a preset operation on the at least one value received each time until the second coordinates are obtained, and output the second coordinates under control of the state machine;
wherein the gaze point calculation apparatus is integrated into an all-in-one device so as to realize gaze point recognition based on eye tracking.
2. The gaze point calculation apparatus according to claim 1, wherein the arithmetic unit assembly obtains the second coordinates through calculation according to following formulas:

X G =a 0 +a 1 x 1 +a 2 y 1 +a 3 x 1 y 1 +a 4 x 1 2 +a 5 y 1 2;

Y G =b 0 +b 1 x 1 +b 2 y 1 +b 3 x 1 y 1 +b 4 x 1 2 +b 5 y 1 2;
wherein (x1, y1) is the first coordinates; a0 to a5 and b0 to b5 are the calibration parameters, and a0 to a5 and b0 to b5 are vectors; and (XG, YG) is the second coordinates.
3. The gaze point calculation apparatus according to claim 2, wherein the arithmetic unit assembly includes:
a multiplier coupled to the multiplexer, the multiplier being configured to calculate x1 2, a1x1, a4(x1 2), x1y1, y1 2, a2y1, a3(x1y1), a5(y1 2), b1x1, b2y1, b3(x1y1), b4(x1 2) and b5(y1 2); and an accumulator coupled to the multiplier, the multiplexer and the state machine, the accumulator being configured to calculate a0+apx1+a2y1+a3xy1+a4x1 2+a5y1 2 to output XG, and calculate b0+b1x1+b2y1+b3x1y1+b4x1 2+b5y1 2 to output YG.
4. The gaze point calculation apparatus according to claim 2, wherein the arithmetic unit assembly includes:
a first multiplier coupled to the multiplexer, the first multiplier being configured to calculate a1x1, a4(x1 2), a2y1, a3(x1y1), and a5(y1 2);
a first accumulator coupled to the first multiplier, the multiplexer and the state machine, the first accumulator being configured to calculate a0+a1x1+a2y1+a3xy1+a4x1 2+a5y1 2 to output XG;
a second multiplier coupled to the multiplexer, the second multiplier being configured to calculate b1x1, b4(x1 2), b2y1, b3(x1y1), and b5(y1 2); and
a second accumulator coupled to the second multiplier, the multiplexer and the state machine, the second accumulator being configured to calculate b0+b1x1+b2y1+b3x1y1+b4x1 2+b5y1 2 to output YG, wherein
at least one of the first multiplier and the second multiplier is further configured to calculate x1 2, x1y1 and y1 2; or, the first multiplier is further configured to calculate a part of x1 2, x1y1 and y1 2, and the second multiplier is further configured to calculate a remaining part of x1 2, x1y1 and y1 2.
5. The gaze point calculation apparatus according to claim 1, wherein the arithmetic unit assembly is further configured to:
convert the received at least one value from an original code into a complement code;
calculate the at least one value converted into the complement code to obtain an intermediate calculation result; and
convert the intermediate calculation result from another complement code into another original code to obtain an operation result.
6. The gaze point calculation apparatus according to claim 1, wherein the arithmetic unit assembly is further coupled to the first cache register; the arithmetic unit assembly is further configured to transmit, under the control of the state machine, at least a part of intermediate values that need to be stored in intermediate values generated by the arithmetic unit assembly to the first cache register for storage.
7. The gaze point calculation apparatus according to claim 1, further comprising:
a normalization processing module coupled to the first cache register, the normalization processing module being configured to:
receive initial coordinates of the pupil in an acquired image of a human eye; and
perform normalization processing on the initial coordinates to obtain the first coordinates, and transmit the first coordinates after the normalization processing to the first cache register.
8. The gaze point calculation apparatus according to claim 7, wherein the normalization processing module performs the normalization processing on the initial coordinates according to following formulas:

x 1 =x′/a lateral resolution of the acquired image of the human eye;

y 1 =y′/a vertical resolution of the acquired image of the human eye;
wherein (x′, y′) is the initial coordinates, and (x1, y1) is the first coordinates.
9. The gaze point calculation apparatus according to claim 1, wherein
the gaze point calculation apparatus is a field programmable gate array (FPGA) chip or an application specific integrated circuit (ASIC) chip.
10. An electronic device, comprising the gaze point calculation apparatus according to claim 1.
11. The electronic device according to claim 10, wherein the electronic device comprises an augmented reality (AR) display device or a virtual reality (VR) display device.
12. The electronic device according to claim 10, further comprising a pupil coordinate extraction module and a display driver module that are coupled to the gaze point calculation apparatus, wherein
the pupil coordinate extraction module is configured to acquire an image of a human eye, and determine initial coordinates of the pupil in the image of the human eye according to the image of the human eye; the initial coordinates or initial coordinates after normalization processing are the first coordinates, and the first coordinates are the position coordinates of the pupil;
the gaze point calculation apparatus is configured to obtain the second coordinates through calculation according to the first coordinates; the second coordinates are the position coordinates of the gaze point; and
the display driver module is configured to process a displayed image according to the second coordinates.
13. The electronic device according to claim 12, wherein the pupil coordinate extraction module includes:
a camera configured to acquire the image of the human eye; and
a processing module configured to determine, according to the acquired image of the human eye, the initial coordinates of the pupil in the image of the human eye based on an image processing algorithm.
14. The gaze point calculation apparatus according to claim 1, wherein the arithmetic unit assembly further includes at least one second cache register; the arithmetic unit assembly is further configured to transmit, under the control of the state machine, at least a part of intermediate values that need to be stored in intermediate values generated by the arithmetic unit assembly to the at least one second cache register for storage.
15. The gaze point calculation apparatus according to claim 1, wherein the arithmetic unit assembly is further coupled to the first cache register; the arithmetic unit assembly is further configured to transmit, under the control of the state machine, some of at least a part of intermediate values that need to be stored in intermediate values generated by the arithmetic unit assembly to the first cache register for storage; and
the arithmetic unit assembly further includes at least one second cache register; the arithmetic unit assembly is further configured to transmit, under the control of the state machine, another of the part of intermediate values that need to be stored in the intermediate values generated by the arithmetic unit assembly to the at least one second cache register for storage.
16. A driving method for a gaze point calculation apparatus, the gaze point calculation apparatus including a first cache register, a multiplexer coupled to the first cache register, an arithmetic unit assembly coupled to the multiplexer, and a state machine coupled to the multiplexer and the arithmetic unit assembly, the driving method comprising:
controlling, by the state machine, the multiplexer to select each time at least one value from the first cache register and transmit the at least one value to the arithmetic unit assembly; the at least one value is selected from first coordinates and a plurality of calibration parameters required when second coordinates are obtained through calculation according to the first coordinates, both the first coordinates and the plurality of calibration parameters are stored in the first cache register;
performing, by the arithmetic unit assembly, a preset operation on the at least one value received each time until the second coordinates are obtained; and
outputting, by the arithmetic unit assembly, the second coordinates under control of the state machine;
wherein the gaze point calculation apparatus is integrated into an all-in-one device so as to realize gaze point recognition based on eye tracking.
17. The driving method according to claim 16, wherein the second coordinates are obtained according to following formulas:

X G =a 0 +a 1 x 1 +a 2 y 1 +a 3 x 1 y 1 +a 4 x 1 2 +a 5 y 1 2;

Y G =b 0 +b 1 x 1 +b 2 y 1 +b 3 x 1 y 1 +b 4 x 1 2 +b 5 y 1 2;
wherein (x1, y1) is the first coordinates; a0 to a5 and b0 to b5 are the calibration parameters, and a0 to a5 and b0 to b5 are vectors; and (XG, YG) is the second coordinates.
18. The driving method according to claim 13, further comprising:
converting the received at least one value from an original code into a complement code;
calculating the at least one value converted into the complement code to obtain an intermediate calculation result; and
converting the intermediate calculation result from another complement code into another original code to obtain an operation result.
19. The driving method according to claim 16, further comprising:
receiving initial coordinates of the pupil in an acquired image of a human eye;
performing normalization processing on the initial coordinates to obtain the first coordinates, and
transmitting the first coordinates after the normalization processing to the first cache register.
20. The driving method according to claim 19, wherein the normalization processing is performed according to following formulas:

x 1 =x′/lateral resolution of the acquired image of the human eye;

y 1 =y′/vertical resolution of the acquired image of the human eye;
wherein (x′, y′) is the initial coordinates, and (x1, y1) is the first coordinates.
US17/787,012 2020-09-28 2021-05-11 Gaze point calculation apparatus and driving method therefor, and electronic device Active 2042-06-18 US12271518B2 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
CN202011045736.2A CN114281184A (en) 2020-09-28 2020-09-28 Gaze calculation device and driving method thereof, and electronic device
CN202011045736.2 2020-09-28
PCT/CN2021/093117 WO2022062422A1 (en) 2020-09-28 2021-05-11 Gaze point calculation apparatus and driving method therefor, and electronic device

Publications (2)

Publication Number Publication Date
US20230015967A1 US20230015967A1 (en) 2023-01-19
US12271518B2 true US12271518B2 (en) 2025-04-08

Family

ID=80846168

Family Applications (1)

Application Number Title Priority Date Filing Date
US17/787,012 Active 2042-06-18 US12271518B2 (en) 2020-09-28 2021-05-11 Gaze point calculation apparatus and driving method therefor, and electronic device

Country Status (3)

Country Link
US (1) US12271518B2 (en)
CN (1) CN114281184A (en)
WO (1) WO2022062422A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12462729B2 (en) * 2024-01-08 2025-11-04 Apple Inc. Mux-free architecture for pixel data bus latching in foveated displays

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100103375A1 (en) 2008-10-27 2010-04-29 Utechzone Co., Ltd. Eye-tracking method and system for implementing the same
CN106066696A (en) 2016-06-08 2016-11-02 华南理工大学 The sight tracing compensated based on projection mapping correction and point of fixation under natural light
CN108196881A (en) 2017-12-01 2018-06-22 北京时代民芯科技有限公司 A kind of fixed-point calculation accelerator module based on configurable technology
CN110032271A (en) 2018-01-12 2019-07-19 京东方科技集团股份有限公司 Contrast control device and its method, virtual reality device and storage medium
CN110619303A (en) 2019-09-16 2019-12-27 Oppo广东移动通信有限公司 Method, device and terminal for tracking point of regard and computer readable storage medium
CN212433723U (en) 2020-09-28 2021-01-29 京东方科技集团股份有限公司 Gaze point calculation device and electronic apparatus
US20210088784A1 (en) * 2019-09-20 2021-03-25 Nvidia Corp. Driver gaze tracking system for use in vehicles
US20210350550A1 (en) * 2020-05-11 2021-11-11 Nvidia Corporation Gaze estimation using one or more neural networks

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108572735B (en) * 2018-04-24 2021-01-26 京东方科技集团股份有限公司 Pupil center positioning device and method, virtual reality device
CN110263745B (en) * 2019-06-26 2021-09-07 京东方科技集团股份有限公司 Method and device for positioning pupil of human eye

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100103375A1 (en) 2008-10-27 2010-04-29 Utechzone Co., Ltd. Eye-tracking method and system for implementing the same
CN106066696A (en) 2016-06-08 2016-11-02 华南理工大学 The sight tracing compensated based on projection mapping correction and point of fixation under natural light
CN108196881A (en) 2017-12-01 2018-06-22 北京时代民芯科技有限公司 A kind of fixed-point calculation accelerator module based on configurable technology
CN110032271A (en) 2018-01-12 2019-07-19 京东方科技集团股份有限公司 Contrast control device and its method, virtual reality device and storage medium
US20210341998A1 (en) 2018-01-12 2021-11-04 Boe Technology Group Co., Ltd. Gaze-point determining method, contrast adjusting method, and contrast adjusting apparatus, virtual reality device and storage medium
CN110619303A (en) 2019-09-16 2019-12-27 Oppo广东移动通信有限公司 Method, device and terminal for tracking point of regard and computer readable storage medium
US20210088784A1 (en) * 2019-09-20 2021-03-25 Nvidia Corp. Driver gaze tracking system for use in vehicles
US20210350550A1 (en) * 2020-05-11 2021-11-11 Nvidia Corporation Gaze estimation using one or more neural networks
CN212433723U (en) 2020-09-28 2021-01-29 京东方科技集团股份有限公司 Gaze point calculation device and electronic apparatus

Also Published As

Publication number Publication date
CN114281184A (en) 2022-04-05
US20230015967A1 (en) 2023-01-19
WO2022062422A1 (en) 2022-03-31

Similar Documents

Publication Publication Date Title
EP3951721B1 (en) Method and apparatus for determining occluded area of virtual object
US11443173B2 (en) Hardware-software co-design for accelerating deep learning inference
US12260556B2 (en) Image segmentation method, system, and device, and readable storage medium
CN107808397B (en) Pupil positioning device, pupil positioning method and sight tracking equipment
CN111291698B (en) High-speed recognition method and device for face images of dense crowd scene
US12271518B2 (en) Gaze point calculation apparatus and driving method therefor, and electronic device
CN118247617A (en) Information processing method, device, equipment and medium based on multi-mode large model
CN118351181A (en) Visual positioning method and electronic equipment
CN109242091B (en) Image recognition method, device, equipment and readable storage medium
CN109493349A (en) A kind of characteristics of image processing module, augmented reality equipment and angular-point detection method
CN115623339B (en) Image data exchange method, device, system and vehicle for multi-camera
CN212433723U (en) Gaze point calculation device and electronic apparatus
US20170069294A1 (en) Driving device for display panel, and display panel
Sada et al. A dataflow pipelining architecture for tile segmentation with a sparse MobileNet on an FPGA
CN109858341B (en) Rapid multi-face detection and tracking method based on embedded system
Huang et al. A high-efficiency fpga-based orb feature matching system
CN118519610A (en) Operation method, device, equipment and storage medium based on pulse array
CN114862658A (en) Image processing system, method, intelligent terminal and computer readable storage medium
CN119151770B (en) Image rendering method, device, equipment and medium in live camera
CN114254255B (en) Convolution calculation method and related device
CN120832010A (en) Eye tracking method, sight line detection method, device, equipment and storage medium
CN113255903B (en) Detection model adjustment method and device
TWI511088B (en) Method of generating azimuth images
Tsai et al. Hardware Design on Face Recognition System by Deep Neural Network for Access Control System
US20230122373A1 (en) Method for training depth estimation model, electronic device, and storage medium

Legal Events

Date Code Title Description
AS Assignment

Owner name: BOE TECHNOLOGY GROUP CO., LTD., CHINA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SUN, GAOMING;HU, GUOFENG;REEL/FRAME:060514/0549

Effective date: 20220318

Owner name: BEIJING BOE OPTOELECTRONICS TECHNOLOGY CO., LTD., CHINA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SUN, GAOMING;HU, GUOFENG;REEL/FRAME:060514/0549

Effective date: 20220318

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: AWAITING TC RESP., ISSUE FEE NOT PAID

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE