WO2024047990A1 - 情報処理装置 - Google Patents

情報処理装置 Download PDF

Info

Publication number
WO2024047990A1
WO2024047990A1 PCT/JP2023/020973 JP2023020973W WO2024047990A1 WO 2024047990 A1 WO2024047990 A1 WO 2024047990A1 JP 2023020973 W JP2023020973 W JP 2023020973W WO 2024047990 A1 WO2024047990 A1 WO 2024047990A1
Authority
WO
WIPO (PCT)
Prior art keywords
interest
information
user
line
sight
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2023/020973
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
秀 田中
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Canon Inc
Original Assignee
Canon Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Canon Inc filed Critical Canon Inc
Priority to DE112023003676.4T priority Critical patent/DE112023003676T5/de
Priority to CN202380062495.8A priority patent/CN119816797A/zh
Publication of WO2024047990A1 publication Critical patent/WO2024047990A1/ja
Priority to US19/061,432 priority patent/US20250199613A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis

Definitions

  • the present invention relates to an information processing device, and particularly to a technique for determining a person's level of interest in an object.
  • Patent Document 1 discloses a technique for measuring the number of people viewing advertising content displayed on an advertising display device based on the line of sight of people present in an area where the advertising display device is installed.
  • Patent Document 2 discloses a technique for determining the degree of interest of a passenger (passenger of a moving object) in a gaze target object based on the gaze duration of the passenger (passenger of a moving body) with respect to the gaze target object.
  • the person's degree of interest in the object is determined by regarding the act of a person looking at an object as an act of the person being interested in the object.
  • a person sees an object does not necessarily mean that the person is interested in the object. Therefore, with the conventional technology, it is not possible to judge the degree of interest with high precision.
  • An object of the present invention is to provide a technology that can accurately determine a person's level of interest in an object.
  • a first aspect of the present invention includes: a first acquisition unit that acquires line-of-sight information of a user; a detection unit that detects an object that the user is looking at based on the line-of-sight information; a second acquisition unit that acquires reference information for determining the user's degree of interest in the object from a storage unit; and a determination that determines the degree of interest based on the line-of-sight information and the reference information.
  • An information processing device characterized by having means.
  • a second aspect of the present invention includes a first acquisition unit that acquires line-of-sight information of a user, a detection unit that detects an object that the user is looking at based on the line-of-sight information, and an image of the object that is analyzed. and a determining means for determining the degree of interest based on the line-of-sight information and the reference information.
  • This is an information processing device characterized by:
  • a third aspect of the present invention is a step of acquiring line-of-sight information of a user, a step of detecting an object that the user is looking at based on the line-of-sight information, and predetermined information corresponding to the object.
  • the method includes the steps of: acquiring reference information for determining the user's degree of interest in the object from a storage unit; and determining the degree of interest based on the line-of-sight information and the reference information. This is a method for controlling an information processing device.
  • a fourth aspect of the present invention is to obtain the user's line-of-sight information, to detect the object that the user is looking at based on the line-of-sight information, and to analyze the image of the object.
  • An information processing device comprising: acquiring reference information for determining the user's degree of interest in an object; and determining the degree of interest based on the line-of-sight information and the reference information. This is a control method.
  • a fifth aspect of the present invention is a program for causing a computer to function as each means of the information processing device described above.
  • a sixth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each means of the information processing apparatus described above.
  • a person's degree of interest in an object can be determined with high accuracy.
  • FIG. 1A and 1B are external views of the display system.
  • FIG. 2 is a block diagram showing the electrical configuration of the display system.
  • FIG. 3 is a diagram for explaining the principle of the line of sight detection method.
  • FIG. 4A is a diagram showing an eye image
  • FIG. 4B is a diagram showing a brightness distribution of the eye image. It is a flowchart of line of sight detection processing. It is a flowchart of the whole process.
  • FIG. 1 is a block diagram showing the functional configuration of a display system.
  • FIG. 8A is a diagram showing an object of interest
  • FIG. 8B is a diagram showing interest level reference information.
  • FIG. 9 is a flowchart of the interest level determination process.
  • FIG. 10 is a flowchart of reference information acquisition processing according to the second embodiment.
  • Example 1 >> Example 1 of the present invention will be described.
  • the present invention is applicable to various electronic devices that can acquire line-of-sight information (information regarding line-of-sight) of a person.
  • the present invention is applicable to both video see-through type display devices and optical see-through type display devices.
  • a video see-through type display device combines an image captured in real space (virtual space) with graphics (e.g., virtual objects) as needed to create a display surface (a display surface that does not transmit light from the real space (external world)).
  • graphics e.g., virtual objects
  • the user cannot directly see the real space, but by looking at the displayed image, the user can indirectly see the real space or see graphics synthesized with the real space image.
  • an optical see-through type display device displays graphics on a display surface (a display surface that transmits light from real space). In this case, the user can directly view the real space through the display surface and view graphics displayed on the display surface.
  • the present invention is applicable to both head-mounted display devices and other display devices.
  • the present invention is also applicable to handheld display devices and stationary display devices.
  • the head-mounted display device is, for example, smart glasses (AR (Augmented Reality) glasses) or HMD (head-mounted display).
  • the handheld display device is, for example, a smartphone or a tablet terminal.
  • a display device that a user holds in his/her hand and attaches to his or her head is a type of handheld display device and also a type of head-mounted display device.
  • a smartphone attached to a head mount adapter for example, VR (Virtual Reality) goggles) is a type of head-mounted display device.
  • the present invention is applicable to both head-mounted display devices in which the user views images with both eyes, and head-mounted display devices in which the user views images with one eye.
  • the present invention is also applicable to a system in which only a virtual space is allowed to be viewed by the user without allowing the user to see the real space. That is, the present invention is applicable to various XR (Cross Realities), such as AR (Augmented Reality), MR (Mixed Reality), and VR (Virtual Reality).
  • XR Cross Realities
  • AR Algmented Reality
  • MR Magnetic Reality
  • VR Virtual Reality
  • the present invention is also applicable to electronic devices other than display devices.
  • the information processing device to which the present invention is applied may be provided in a display device, or may be provided in an electronic device separate from the display device.
  • the present invention is also applicable to a controller or a PC (personal computer) connected to a display device.
  • the present invention is also applicable to surveillance cameras.
  • a person's line-of-sight information is acquired, and the person's degree of interest in an object is determined based on the line-of-sight information.
  • the object is a real object
  • the object may be a virtual object.
  • the person's degree of interest in the object is determined by regarding the act of a person looking at an object as an act of the person being interested in the object.
  • a person sees an object does not necessarily mean that the person is interested in the object. Therefore, with the conventional technology, it is not possible to judge the degree of interest with high precision.
  • advertising media may have a large area for eye-catching (eye-catching area).
  • the eye-catching area is, for example, an area in which a photograph of a person (advertisement model) or a catchphrase is arranged in a large size.
  • the eye-catching area may have low relevance to the advertising target (for example, a product or service). Therefore, even if a person views the eye-catching area, the person may not be associated with the advertising target and may not be interested in the advertising target or advertising medium.
  • the area with low relevance to the advertising target is not limited to the eye-catching area.
  • a person's degree of interest in an object is determined with high accuracy using suitable criteria depending on the object.
  • the level of interest is determined with high accuracy using criteria that take into account the content of the object (the intention of the object's design).
  • FIG. 1A and 1B show the appearance of a display system 100 according to the first embodiment.
  • FIG. 1A is a front perspective view
  • FIG. 1B is a back perspective view.
  • display system 100 includes HMD 110, controller 120, and server 130.
  • the HMD 110 is used as a video see-through HMD.
  • the controller 120 is connected to the HMD 110 by wire and to the server 130 wirelessly. Controller 120 may be wirelessly connected to HMD 110.
  • FIG. 2 is a block diagram showing the electrical configuration of the display system 100.
  • the HMD 110 includes a photographic lens 111, a display section 112, light sources 113a and 113b, a light receiving lens 114, an eye image sensor 115, and a knob 140. Further, as shown in FIG. 2, the HMD 110 includes an image sensor 116, a photometry circuit 117, a light source drive circuit 118, and a display drive circuit 119. As shown in FIG. 1B, the display unit 112, light sources 113a and 113b, light receiving lens 114, and eye imaging device 115 are provided for each of the user's right eye and left eye.
  • the photographing lens 111 is a lens for capturing an image of the outside world
  • the image sensor 116 is an image sensor for capturing an image of the outside world.
  • the image sensor 116 is arranged on the intended imaging plane of the photographic lens 111.
  • the display unit 112 displays various images (information). For example, the display unit 112 displays an image of the outside world captured by the image sensor 116, or displays information regarding an object that the user is viewing.
  • the display unit drive circuit 119 is controlled by a controller 120 (CPU 121 described later) and drives the display unit 112.
  • Each of the light sources 113a and 113b is a light source that illuminates the user's eyes, and is, for example, an infrared light emitting diode that emits infrared light that is insensitive to the user.
  • the light source drive circuit 118 is controlled by the controller 120 (CPU 121) and drives the light sources 113a and 113b. A portion of the light emitted from the light sources 113a and 113b and reflected by the user's eyes is focused on the eye imaging device 115 by the light receiving lens 114.
  • the light receiving lens 114 is a lens for capturing an image of the user's eye
  • the eye image sensor 115 is an image sensor for capturing an image of the user's eye.
  • the knob 140 is a knob for adjusting the distance between the right eye display section 112 and the left eye display section 112 to match the interpupillary distance of the user.
  • the photometric circuit 117 amplifies, logarithmically compresses, and A/D converts a signal obtained from the image sensor 116 that also serves as a photometric sensor, specifically, a luminance signal corresponding to the brightness of the object.
  • the result is sent to the controller 120 (CPU 121) as field brightness information.
  • the controller 120 includes a CPU 121, a memory section 122, a line of sight detection circuit 123, an LPF (low pass filter) 124, a display section drive circuit 125, operating members 126 to 128, and a wireless communication circuit 129.
  • the CPU 121 is a central processing unit of a microcomputer built into the controller 120 and controls the entire display system 100.
  • the memory unit 122 has a function of storing video signals from the eye image sensor 115 and a function of storing line of sight correction parameters.
  • the line of sight correction parameter is a parameter for correcting individual differences in line of sight.
  • the memory unit 122 has a storage function of interest level reference information.
  • the interest level reference information is predetermined information corresponding to the object, and is reference information for determining the user's interest level in the object.
  • the memory unit 122 stores a plurality of pieces of interest level reference information respectively corresponding to a plurality of objects.
  • the line of sight detection circuit 123 A/D converts the output of the eye image sensor 115 (the eye image captured by the eye) in a state where the optical image of the eye is formed on the eye image sensor 115, and converts the result through the LPF 124. and transmits it to the CPU 121.
  • the CPU 121 extracts feature points necessary for line-of-sight detection from the eye image according to a predetermined algorithm described later, and detects the user's line-of-sight from the position of the feature point.
  • the operation members 126 to 128 accept operations from the user and output operation signals (signals corresponding to the operations performed by the user) to the CPU 121.
  • the operating member 126 is a touch panel that can accept touch operations
  • the operating member 127 is an operating lever that can be pushed down in each direction
  • the operating member 128 is a four-way key that can be pushed down in each of four directions.
  • the operation member 126 (touch panel) has a function of displaying images (information). In this way, the operation member 126 has the functions of a touch panel and a display section.
  • the display unit drive circuit 125 is controlled by the CPU 121 and drives the operating member 126 (display unit).
  • the user can perform various operations (instructions) using the operating members 126 to 128.
  • the user can use the operating members 126 to 128 to finely adjust the position of the UI (user interface, such as an indicator) displayed on the display unit 112.
  • the wireless communication circuit 129 is controlled by the CPU 121 and communicates with external devices. For example, the wireless communication circuit 129 transmits the degree of interest determined by the CPU 121 (the user's degree of interest in the object) to the server 130 via the Internet and records it.
  • FIG. 3 is a diagram for explaining the principle of the line of sight detection method, and is a schematic diagram of an optical system for detecting the line of sight.
  • the light sources 113a and 113b are arranged substantially symmetrically with respect to the optical axis of the light receiving lens 114, and illuminate the user's eyeball 200.
  • FIG. 4A is a schematic diagram of an eye image captured by the eye image sensor 115 (an optical image of the eye projected onto the eye image sensor 115), and FIG. 4B is a diagram showing the output intensity of the eye image sensor 115.
  • FIG. 5 is a flowchart of the line of sight detection process.
  • step S1 the CPU 121 uses the light source drive circuit 118 to drive the light sources 113a and 113b so as to emit infrared light toward the user's eyeball 200.
  • the optical image of the user's eye illuminated by the infrared light passes through the light-receiving lens 114 and forms an image on the eye image sensor 115, and is photoelectrically converted by the eye image sensor 115. This provides an electrical signal of the eye image that can be processed.
  • step S2 the CPU 121 acquires an eye image (image data, image signal) from the eye image sensor 115 via the line of sight detection circuit 123.
  • step S3 the CPU 121 detects the coordinates of a point corresponding to the corneal reflection images Pd and Pe of the light sources 113a and 113b and the pupil center c from the eye image obtained in step S2.
  • the infrared light emitted from the light sources 113a and 113b illuminates the cornea 201 of the user's eyeball 200.
  • corneal reflection images Pd and Pe formed by a portion of the infrared light reflected on the surface of the cornea 201 are condensed by the light receiving lens 114 and formed on the eye image sensor 115 to form an image in the eye image.
  • the corneal reflection images become Pd' and Pe'.
  • the light beams from the ends a and b of the pupil 202 also form images on the eye imaging device 115, forming pupil end images a' and b' in the eye image.
  • FIG. 4B shows brightness information (brightness distribution) of area ⁇ in the eye image of FIG. 4A.
  • the horizontal direction of the eye image is the X-axis direction
  • the vertical direction is the Y-axis direction
  • the brightness distribution in the X-axis direction is shown.
  • the coordinates of the corneal reflection images Pd' and Pe' in the X-axis direction are Xd and Xe
  • the coordinates of the pupil edge images a' and b' in the X-axis direction are Xa and Xb.
  • an extremely high level of brightness is obtained at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'.
  • a brightness intermediate between the above two types of brightness can be obtained.
  • the X coordinates Xd, Xe of the corneal reflection images Pd', Pe' and the X coordinates Xa, Xb of the pupil edge images a', b' can be obtained.
  • coordinates with extremely high brightness can be obtained as the coordinates of the corneal reflection images Pd', Pe'
  • coordinates with extremely low brightness can be obtained as the coordinates of the pupil edge images a', b'.
  • the pupil center image c' (of the pupil image
  • the coordinate Xc of the center) can be expressed as Xc ⁇ (Xa+Xb)/2. That is, the coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa, Xb of the pupil edge images a', b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be estimated.
  • step S4 the CPU 121 calculates the imaging magnification ⁇ of the eye image.
  • the imaging magnification ⁇ is determined by the position of the eyeball 200 with respect to the light receiving lens 114, and can be calculated using a function of the interval (Xd ⁇ Xe) between the corneal reflection images Pd' and Pe'.
  • step S5 the CPU 121 calculates the rotation angle of the optical axis of the eyeball 200 with respect to the optical axis of the light receiving lens 114.
  • the X coordinate of the midpoint between the corneal reflection image Pd and the corneal reflection image Pe substantially matches the X coordinate of the center of curvature O of the cornea 201. Therefore, if the standard distance from the center of curvature O of the cornea 201 to the center c of the pupil 202 is Oc, then the rotation angle ⁇ x of the eyeball 200 in the ZX plane (a plane perpendicular to the Y-axis) is as follows: It can be calculated using equation 1.
  • the rotation angle ⁇ y of the eyeball 200 within the ZY plane can also be calculated in the same manner as the rotation angle ⁇ x.
  • the CPU 121 reads parameters necessary for line of sight detection from the memory unit 122.
  • the CPU 121 reads parameters m, Ax, Bx, Ay, By, nx, and ny.
  • the parameter m is a constant determined by the configuration of the optical system for performing the line of sight detection processing, and is a conversion coefficient for converting the rotation angles ⁇ x and ⁇ y into coordinates corresponding to the pupil center c on the display unit 112. It is assumed that the parameter m is determined in advance and stored in the memory unit 122.
  • the parameters Ax, Bx, Ay, and By are line-of-sight correction parameters for correcting individual differences in line-of-sight, and are obtained by calibrating line-of-sight detection.
  • Parameter Ax is an offset value in the X-axis direction
  • parameter Bx is a sensitivity coefficient in the X-axis direction
  • parameter Ay is an offset value in the Y-axis direction
  • the parameter nRx is a correction coefficient in the X-axis direction for obtaining right-eye line-of-sight information
  • the parameter nRy is a correction coefficient in the Y-axis direction for obtaining right-eye line-of-sight information
  • the parameter nLx is a correction coefficient in the X-axis direction for obtaining left-eye line-of-sight information
  • the parameter nLy is a correction coefficient in the Y-axis direction for obtaining left-eye line-of-sight information. It is assumed that the parameters nRx, nRy, nLx, and nLy are determined in advance and stored in the memory unit 122.
  • step S7 the CPU 121 uses the rotation angles ⁇ x, ⁇ y calculated in step S5 and the parameters m, Ax, Bx, Ay, By, nx, ny read out in step S6 to Estimate the viewpoint.
  • the viewpoint can be considered as the position where the line of sight is focused, the position where the user is looking, or the position of the user's line of sight.
  • the coordinates (Hx, Hy) of the viewpoint are coordinates corresponding to the pupil center c
  • the coordinates (Hx, Hy) of the viewpoint can be calculated using equations 2 and 3 below.
  • Hx m ⁇ (Ax ⁇ x+Bx) ⁇ nx (Formula 2)
  • Hy m ⁇ (Ay ⁇ y+By) ⁇ ny...(Formula 3)
  • step S8 the CPU 121 stores the coordinates (Hx, Hy) of the viewpoint in the memory unit 122, and ends the line of sight detection process.
  • the line of sight detection method is not limited to the above method, and may be any method as long as it acquires line of sight information from an eye image, for example.
  • the line of sight may be detected by a method that does not use an eye image, such as a method of detecting an electrooculogram and detecting the line of sight based on the electrooculogram, without using an eye image.
  • information indicating the line-of-sight direction may be obtained instead of information indicating the viewpoint.
  • processing may be performed to obtain the rotation angles (Ax ⁇ x+Bx) ⁇ nx and (Ay ⁇ y+By) ⁇ ny without obtaining the coordinates (Hx, Hy) of the viewpoint.
  • the final line of sight information information indicating the viewpoint of both eyes (the intersection of the line of sight of the right eye and the line of sight of the left eye, the position viewed by both eyes) may be acquired.
  • FIG. 6 is a flowchart of the overall process. For example, when the user wears the HMD 110 and starts the HMD 110 and controller 120, the overall process shown in FIG. 6 starts.
  • FIG. 7 is a block diagram showing the functional configuration of the display system 100. Each functional unit in FIG. 7 is realized by the CPU 121 of the controller 120.
  • step S101 the external world image acquisition unit 302 acquires an image of the external world (external world image) captured by the image sensor 116.
  • the CPU 121 performs predetermined image processing on the acquired external world image and displays it on the display unit 112.
  • step S102 the line-of-sight detection unit 301 acquires the user's line-of-sight information by performing the line-of-sight detection process shown in FIG.
  • step S103 the CPU 121 detects the object that the user is looking at (object of interest) based on the line of sight information acquired in step S102.
  • the object detection unit 303 detects an object from the external world image acquired in step S101.
  • the object of interest detection unit 304 detects the object of interest from the objects detected by the object detection unit 303 based on the line of sight information acquired in step S102.
  • the object of interest is detected based on information obtained from multiple pieces of time-series gaze information (e.g., the trajectory of the viewpoint, the saccade of the eyeball, the length of time the viewpoint remains, and the number of times the object is fixed). do.
  • the object of interest information that is the detection result of the object of interest is acquired. Details of the attention object information will be described later.
  • step S104 the reference information acquisition unit 305 acquires interest level reference information corresponding to the object of interest based on the object of interest information obtained in step S103 (reference information acquisition process).
  • the reference information acquisition unit 305 acquires interest level reference information corresponding to the object of interest from the memory unit 122. Details of the interest level reference information will be described later.
  • the object of interest information indicates the position of the object of interest (the position in real space, the position of the object of interest in the coordinate system of real space (world coordinate system)). It is assumed that the interest level reference information stored in the memory unit 122 is associated with the position of the object corresponding to the interest level reference information. In that case, the reference information acquisition unit 305 may acquire interest level reference information associated with the position closest to the position indicated by the object of interest information from the memory unit 122.
  • the object of interest information may indicate the position and type of the object of interest.
  • the interest level reference information stored in the memory unit 122 may be associated with the position and type of the object corresponding to the interest level reference information.
  • the reference information acquisition unit 305 acquires interest level reference information associated with the position closest to the position indicated by the object of interest information from the memory unit 122 among the objects of the type indicated by the object of interest information. You may obtain it.
  • the attention object information may include identification information (for example, an identifier) that identifies (specifies) the attention object.
  • the interest level reference information stored in the memory unit 122 may be associated with the identification information of the object corresponding to the interest level reference information. In that case, the reference information acquisition unit 305 may acquire interest level reference information associated with the same identification information as the identification information included in the attention object information from the memory unit 122. .
  • a plurality of pieces of interest level reference information corresponding to a plurality of objects may be stored in the server 130.
  • the reference information acquisition unit 305 may obtain interest level reference information corresponding to the object of interest from the server 130.
  • the process of selecting interest level reference information corresponding to the object of interest from a plurality of pieces of interest level reference information based on the object of interest information may be performed by the reference information acquisition unit 305 or by the server 130.
  • the server 130 performs the process of selecting the interest level reference information corresponding to the object of interest, for example, the reference information acquisition unit 305 transmits the object of interest information to the server 130, and the server 130 selects the degree of interest reference information corresponding to the object of interest.
  • the reference information is returned to the reference information acquisition unit 305.
  • the object of interest detection unit 304 detects the three-dimensional position of the object of interest based on the line-of-sight information of the user's right eye and the line-of-sight information of the user's left eye, and generates the object of interest information indicating the three-dimensional position. good.
  • the object of interest detection unit 304 communicates with the user (HMD 110) based on the right eye's line of sight information, the left eye's line of sight information, and the interpupillary distance (the distance between the display section 112 for the right eye and the display section 112 for the left eye). Determine (estimate) the three-dimensional positional relationship with the object of interest.
  • the object of interest detection unit 304 detects (estimates) the three-dimensional position of the object of interest based on the three-dimensional position of the user (HMD 110) and the determined positional relationship.
  • the three-dimensional position of the user (HMD 110) is estimated using, for example, GPS (Global Positioning System).
  • GPS Global Positioning System
  • the three-dimensional position of the user (HMD 110) may be estimated by SLAM (Simultaneous Localization and Mapping).
  • the three-dimensional position of the user (HMD 110) may be estimated by odometry using an IMU (Inertial Measurement Unit).
  • step S104 may be performed. By doing so, the processing load can be reduced.
  • step S105 the interest level determining unit 306 determines the user's interest level with respect to the object of interest based on the line of sight information obtained in step S102, the object of interest information obtained in step S103, and the interest level reference information obtained in step S104. Determine level of interest. Details of the process in step S105 (interest level determination process) will be described later.
  • step S106 the interest level recording unit 307 transmits the interest level determined in step S105 to the server 130 via the Internet and records it.
  • the interest level may be recorded in the memory unit 122.
  • FIG. 8A shows an object of interest
  • FIG. 8B shows interest level reference information corresponding to the object of interest in FIG. 8A.
  • FIG. 9 is a flowchart of the interest level determination process.
  • the object of interest in FIG. 8A is an advertising medium and includes an eye-catching area, a text area, and an image area.
  • a catchphrase that has little relevance to the product being advertised is written.
  • a description of the product to be advertised and the company name of the product are written.
  • An image of the product to be advertised is drawn in the image area located at the lower right of the object of interest.
  • the degree of interest reference information in FIG. 8B indicates a standard for determining the user's degree of interest in the object of interest for each of the plurality of regions of the object of interest.
  • the interest level reference information in FIG. 8B indicates setting regions 1 to 3 (relative arrangement of each of setting regions 1 to 3 with respect to the entire object of interest).
  • Setting area 1 corresponds to an image area
  • setting area 2 corresponds to an eye-catching area
  • setting area 3 corresponds to a text area.
  • the interest level reference information in FIG. 8B indicates the type (image area/eye-catching area/text area), weight, and related area for each of setting areas 1 to 3.
  • setting area 2 is set as a related area to setting area 1, and no related area is set in setting areas 2 and 3.
  • step S201 the interest level determination unit 306 assigns 0 to the interest level I (initialization of the interest level I).
  • the interest level determination unit 306 sets the sensitivity level k of the interest level I.
  • the interest level determination unit 306 sets a value according to the user's attributes as the sensitivity level k so that it is easy to obtain a large value as the interest level I of a user who is likely to be interested (and who wants to be interested).
  • the object of interest is an advertising medium and the advertising target is cosmetics
  • the interest level determination unit 306 is likely to obtain a large value as the interest level of users of the age (age group) and gender targeted by the cosmetics.
  • the sensitivity k may be set as follows.
  • the interest level determination unit 306 determines whether the user's viewpoint remains in the setting area 1 for a threshold time T1 or more. This determination can also be interpreted as determining whether the user's line of sight is directed toward the setting area 1 for a predetermined threshold time T1 or longer.
  • the object of interest information indicates the area of the object of interest in the external world image (the area of the object of interest on the display unit 112).
  • the degree of interest determining unit 306 determines the setting area 1 on the display unit 112 based on the object of interest information and the reference level of interest information of the object of interest (relative arrangement of the setting area 1 with respect to the entire object of interest). .
  • the interest level determination unit 306 determines whether the viewpoint remains in the set area 1 on the display unit 112 for a threshold time T1 or more based on the line-of-sight information. If the interest level determining unit 306 determines that the user's viewpoint has remained in the setting area 1 for the threshold time T1 or more, the process proceeds to step S211, and if the user's viewpoint has not remained in the setting area 1 for the threshold time T1 or more. If determined, the process advances to step S204.
  • the interest level determination unit 306 updates the interest level I using the interest level reference information (weight w1 of setting area 1) of the object of interest.
  • the interest level determination unit 306 updates the interest level I to a larger value as the weight w1 is larger.
  • step S204 the interest level determining unit 306 determines whether a saccade that moves the viewpoint within the setting area 2 has occurred. If the interest level determining unit 306 determines that a saccade into the setting area 2 has occurred, the process proceeds to step S212; if it determines that a saccade into the setting area 2 has not occurred, the interest level determining unit 306 advances the process to step S212. The process advances to step S205.
  • step S205 the interest level determination unit 306 determines whether the user's viewpoint has remained in the setting area 1 for a threshold time T1' or more and the user's viewpoint has remained in the setting area 2 for a threshold time T2 or more. If the interest level determination unit 306 determines that the user's viewpoint has remained in the setting area 1 for the threshold time T1' or longer and the user's viewpoint has remained in the setting area 2 for the threshold time T2 or more, the process proceeds to step S213. , otherwise, the process advances to step S206.
  • the eye-catching area corresponding to setting area 2 includes a catchphrase that has low relevance to the product being advertised. Therefore, just because the user looks at the setting area 2, it does not necessarily mean that the user is interested in the object of interest.
  • steps S204 and S205 by combining setting area 2 with other parts of the object of interest, it is possible to determine whether or not the user is interested in the object of interest.
  • setting area 2 is set as a related area to setting area 1. Therefore, in step S205, setting area 1 and setting area 2 are combined.
  • step S206 the interest level determining unit 306 determines whether the user's viewpoint remains in the setting area 3 for a threshold time T3 or more. If the interest level determination unit 306 determines that the user's viewpoint has remained in the setting area 3 for the threshold time T3 or more, the process advances to step S214, and if the user's viewpoint has not remained in the setting area 3 for the threshold time T3 or more, If it is determined, the interest level determination process shown in FIG. 9 is ended.
  • the threshold times T1, T1', T2, and T3 may be predetermined fixed times, or may be times that are changed as appropriate.
  • the threshold times T1, T1', T2, and T3 may be changed as appropriate by the user, or may be changed as appropriate by the controller 120.
  • the controller may determine the visibility of the object of interest based on the external world image, and set a longer time as the lower the visibility of the object of interest as the threshold times T1, T1', T2, and T3.
  • the threshold times T1, T1', T2, and T3 may be the same or different.
  • the interest level determination process is not limited to the process shown in FIG.
  • the interest level reference information only needs to indicate a criterion for determining the user's interest level in the object, and is not limited to the information shown in FIG. 8B.
  • the interest level reference information may indicate the order in which the set area is viewed, or may indicate the time at which the set area is desired to be viewed.
  • the user's degree of interest in the object of interest can be determined with high accuracy by acquiring and using the predetermined interest level reference information corresponding to the object of interest from the storage unit. can. Furthermore, it is possible to provide a service that presents users with information about products that they are interested in with high precision, and to obtain the number of people who are interested in an object with high precision.
  • Example 2 of the present invention will be described.
  • description of the same points as Example 1 (for example, the same configuration and processing as Example 1) will be omitted, and points different from Example 1 will be described.
  • the overall processing of the second embodiment is the same as that of the first embodiment (FIG. 6).
  • the reference information acquisition process in step S104 is different between Example 1 and Example 2.
  • interest level reference information corresponding to the object of interest is selected from a plurality of interest level reference information prepared in advance.
  • degree-of-interest reference information corresponding to the object of interest is acquired by analyzing an image of the object of interest.
  • FIG. 10 is a flowchart of reference information acquisition processing according to the second embodiment.
  • the interest level determination process in the second embodiment is the same as the interest level determination process in the first embodiment (FIG. 9).
  • step S301 the reference information acquisition unit 305 extracts the region of the object of interest detected in step S103 from the external world image obtained in step S101 in FIG. As a result, an image of the object of interest (an image of the object of interest) is obtained.
  • the reference information acquisition unit 305 acquires interest level reference information corresponding to the object of interest by analyzing the object of interest image obtained in step S301.
  • step S302 the reference information acquisition unit 305 performs semantic region division (segmentation) of the object of interest image acquired in step S301.
  • Setting areas 1 to 3 in FIG. 8B are set by the process in step S302.
  • step S303 the reference information acquisition unit 305 detects characteristic parts (for example, characters or objects) from the setting area for each setting area set in step S302, and detects the type of setting area (image area/eye-catching area/ (text area).
  • characteristic parts for example, characters or objects
  • the reference information acquisition unit 305 determines the weight and related area of the setting area by evaluating the characteristic parts detected in step S303 for each setting area set in step S302.
  • the object of interest is an advertising medium
  • a large weight is determined for a setting area that is estimated to be in line with the purpose of the advertisement
  • a small weight is determined for a setting area that is estimated to be inconsistent with the purpose of the advertisement.
  • Good too. For example, a large weight may be determined for an image area where a product to be advertised is depicted, and a low weight may be determined for an eye-catching area where only a catchphrase that is less relevant to the product is described.
  • a medium weight may be determined for a text area in which a description of a product is written.
  • step S305 the reference information acquisition unit 305 obtains interest level reference information corresponding to the object of interest by integrating the processing results of steps S302 to S304.
  • the reference information acquisition unit 305 may use an arithmetic unit (a trained model) that inputs the object image of interest and outputs the analysis result of the object image of interest (for example, the processing result of any one of steps S302 to S305).
  • a computing unit may be used for one process among steps S302 to S305, or a computing unit may be used for two or more processes.
  • the CPU 121 may function as an arithmetic unit, or the controller 120 may include a GPU (Graphics Processing Unit) that functions as an arithmetic unit.
  • the server 130 may analyze the object of interest image and obtain interest level reference information corresponding to the object of interest.
  • the reference information acquisition unit 305 transmits the external world image or the object of interest image to the server 130, and the server 130 returns the degree of interest reference information corresponding to the object of interest to the reference information acquisition unit 305.
  • the degree of interest reference information corresponding to the object of interest is acquired by analyzing the object of interest image, and the obtained degree of interest reference information is used to improve the user's understanding of the object of interest.
  • the degree of interest can be determined with high accuracy.
  • the above embodiments are merely examples, and the present invention also includes configurations obtained by appropriately modifying or changing the configuration of the above embodiments within the scope of the gist of the present invention. It will be done.
  • the present invention also includes configurations obtained by appropriately combining the configurations of the above embodiments.
  • the present invention provides a system or device with a program that implements one or more of the functions of the above-described embodiments via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.
  • a circuit for example, ASIC
  • the disclosure of this embodiment includes the following configuration, method, program, and medium.
  • (Configuration 1) a first acquisition means for acquiring user's line of sight information; detection means for detecting an object that the user is looking at based on the line of sight information; a second acquisition means for acquiring, from a storage unit, reference information for determining the user's degree of interest in the object, which is predetermined information corresponding to the object;
  • An information processing device characterized by comprising: determining means for determining the degree of interest based on the line-of-sight information and the reference information.
  • (Configuration 2) a first acquisition means for acquiring user's line of sight information; detection means for detecting an object that the user is looking at based on the line of sight information; a second acquisition means for acquiring reference information for determining the user's degree of interest in the object by analyzing an image of the object;
  • An information processing device characterized by comprising: determining means for determining the degree of interest based on the line-of-sight information and the reference information.
  • (Configuration 3) The information processing device according to configuration 2, wherein the second acquisition unit acquires the reference information using a trained model that inputs the image and outputs an analysis result of the image.
  • the first acquisition means acquires line-of-sight information of the user's right eye and line-of-sight information of the user's left eye, According to any one of configurations 1 to 3, the detection means detects the three-dimensional position of the object based on the line-of-sight information of the user's right eye and the line-of-sight information of the user's left eye.
  • Information processing device (Configuration 5) 5.
  • the information processing apparatus according to any one of configurations 1 to 4 further comprising recording means for recording the degree of interest on a server.
  • Configuration 6) 6.
  • the information processing device according to any one of configurations 1 to 5, wherein the reference information indicates a criterion for determining the degree of interest for each of the plurality of regions of the object.
  • (Configuration 7) The information processing device according to any one of configurations 1 to 6, a display means;
  • a head-mounted display device comprising: (Method 1) a step of obtaining gaze information of the user; detecting an object that the user is looking at based on the line of sight information; acquiring from a storage unit reference information for determining the user's degree of interest in the object, which is predetermined information corresponding to the object;
  • a method for controlling an information processing apparatus comprising the step of determining the degree of interest based on the line-of-sight information and the reference information.
  • Method 2 a step of obtaining gaze information of the user; detecting an object that the user is looking at based on the line of sight information; acquiring reference information for determining the user's degree of interest in the object by analyzing an image of the object; A method for controlling an information processing apparatus, comprising the step of determining the degree of interest based on the line-of-sight information and the reference information.
  • program A program for causing a computer to function as each means of the information processing apparatus according to any one of configurations 1 to 6.
  • media A computer-readable storage medium storing a program for causing a computer to function as each means of the information processing apparatus according to any one of configurations 1 to 6.
  • Controller 121 CPU 301: Line of sight detection unit 304: Attention object detection unit 305: Reference information acquisition unit 306: Interest level determination unit

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • User Interface Of Digital Computer (AREA)
  • Image Analysis (AREA)
PCT/JP2023/020973 2022-09-02 2023-06-06 情報処理装置 Ceased WO2024047990A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
DE112023003676.4T DE112023003676T5 (de) 2022-09-02 2023-06-06 Informationsverarbeitungsvorrichtung
CN202380062495.8A CN119816797A (zh) 2022-09-02 2023-06-06 信息处理装置
US19/061,432 US20250199613A1 (en) 2022-09-02 2025-02-24 Information processing device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2022140049A JP2024035533A (ja) 2022-09-02 2022-09-02 情報処理装置
JP2022-140049 2022-09-02

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US19/061,432 Continuation US20250199613A1 (en) 2022-09-02 2025-02-24 Information processing device

Publications (1)

Publication Number Publication Date
WO2024047990A1 true WO2024047990A1 (ja) 2024-03-07

Family

ID=90099298

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/020973 Ceased WO2024047990A1 (ja) 2022-09-02 2023-06-06 情報処理装置

Country Status (5)

Country Link
US (1) US20250199613A1 (https=)
JP (1) JP2024035533A (https=)
CN (1) CN119816797A (https=)
DE (1) DE112023003676T5 (https=)
WO (1) WO2024047990A1 (https=)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022219877A1 (ja) * 2021-04-12 2022-10-20 ソニーグループ株式会社 情報処理装置、情報処理方法およびプログラム

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007286995A (ja) * 2006-04-19 2007-11-01 Hitachi Ltd 注目度計測装置及び注目度計測システム
WO2011074198A1 (ja) * 2009-12-14 2011-06-23 パナソニック株式会社 ユーザインタフェース装置および入力方法

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5981591B1 (ja) * 2015-03-17 2016-08-31 株式会社コロプラ 没入型仮想空間でオブジェクト操作を制御するためのコンピュータ・プログラムおよびコンピュータ・システム
US10539797B2 (en) * 2016-05-06 2020-01-21 Colopl, Inc. Method of providing virtual space, program therefor, and recording medium
JP6298874B1 (ja) * 2016-12-01 2018-03-20 株式会社コロプラ 情報処理方法及び当該情報処理方法をコンピュータに実行させるためのプログラム
WO2019093278A1 (ja) * 2017-11-10 2019-05-16 株式会社ソニー・インタラクティブエンタテインメント 情報処理装置、情報処理方法、及びプログラム

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007286995A (ja) * 2006-04-19 2007-11-01 Hitachi Ltd 注目度計測装置及び注目度計測システム
WO2011074198A1 (ja) * 2009-12-14 2011-06-23 パナソニック株式会社 ユーザインタフェース装置および入力方法

Also Published As

Publication number Publication date
DE112023003676T5 (de) 2025-06-26
CN119816797A (zh) 2025-04-11
JP2024035533A (ja) 2024-03-14
US20250199613A1 (en) 2025-06-19

Similar Documents

Publication Publication Date Title
US10488659B2 (en) Apparatus, systems and methods for providing motion tracking using a personal viewing device
US9710973B2 (en) Low-latency fusing of virtual and real content
US9105210B2 (en) Multi-node poster location
US9035970B2 (en) Constraint based information inference
US20200341284A1 (en) Information processing apparatus, information processing method, and recording medium
US20170344110A1 (en) Line-of-sight detector and line-of-sight detection method
US20130326364A1 (en) Position relative hologram interactions
CN110275603A (zh) 分布式人造现实系统、手镯设备和头戴式显示器
US11579449B2 (en) Systems and methods for providing mixed-reality experiences under low light conditions
US11024040B2 (en) Dynamic object tracking
KR101467529B1 (ko) 착용형 정보 제공 시스템
US11527004B2 (en) Electronic device and operation method thereof
JP3372926B2 (ja) ヘッドマウントディスプレイ装置およびヘッドマウントディスプレイシステム
KR20190004806A (ko) 헤드 마운트 디스플레이의 얼굴 센서를 사용한 얼굴과 안구 추적 및 얼굴 애니메이션
CN118394205A (zh) 利用眼动追踪技术的混合现实交互
US20180158242A1 (en) Information processing method and program for executing the information processing method on computer
WO2019130708A1 (ja) 情報処理装置、情報処理方法及びプログラム
US20250199613A1 (en) Information processing device
CN114581514B (zh) 一种双眼注视点的确定方法和电子设备
JP2018088604A (ja) 画像表示装置、画像表示方法、システム
KR101733519B1 (ko) 3차원 디스플레이 장치 및 방법
JP2017055233A (ja) 表示装置、表示システム、及び、表示装置の制御方法
US20250281085A1 (en) Systems and methods for performing a motor skills neurological test using augmented or virtual reality
JP2024022726A (ja) 電子機器
JP2024008006A (ja) 画像処理装置、表示装置、画像処理方法、およびプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23859760

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202380062495.8

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 112023003676

Country of ref document: DE

WWP Wipo information: published in national office

Ref document number: 202380062495.8

Country of ref document: CN

WWP Wipo information: published in national office

Ref document number: 112023003676

Country of ref document: DE

122 Ep: pct application non-entry in european phase

Ref document number: 23859760

Country of ref document: EP

Kind code of ref document: A1