WO2019130992A1 - Dispositif de traitement d'informations - Google Patents

Dispositif de traitement d'informations Download PDF

Info

Publication number
WO2019130992A1
WO2019130992A1 PCT/JP2018/044297 JP2018044297W WO2019130992A1 WO 2019130992 A1 WO2019130992 A1 WO 2019130992A1 JP 2018044297 W JP2018044297 W JP 2018044297W WO 2019130992 A1 WO2019130992 A1 WO 2019130992A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
gaze
sight
line
candidate
Prior art date
Application number
PCT/JP2018/044297
Other languages
English (en)
Japanese (ja)
Inventor
敬幸 古田
雄太 樋口
和輝 東
Original Assignee
株式会社Ｎｔｔドコモ
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社Ｎｔｔドコモ filed Critical 株式会社Ｎｔｔドコモ
Priority to US16/957,581 priority Critical patent/US11282481B2/en
Priority to JP2019562896A priority patent/JP6970757B2/ja
Publication of WO2019130992A1 publication Critical patent/WO2019130992A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
    • G09G5/37—Details of the operation on graphic patterns
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/147—Digital output to display device ; Cooperation and interconnection of the display device with other functional units using display panels
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
    • G09G5/38—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory with means for controlling the display position
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/04—Changes in size, position or resolution of an image
    • G09G2340/0464—Positioning
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2354/00—Aspects of interface with display user

Definitions

  • One aspect of the present invention relates to an information processing apparatus.
  • the virtual reality (VR: Virtual Reality) is embedded with the virtual space.
  • VR virtual reality
  • a user object avatar, character, etc.
  • the user's action for example, the action of a part of the body such as the head and hands
  • the user's action is controlled according to Then, by displaying an image showing a view seen from the user object on the HMD, the user is provided with an experience as if the user object exists in the virtual space.
  • Patent Document 1 in order to facilitate communication between users in a virtual space (communication via a user object), the user's actual head or eye movement and the head of a user object in the virtual space Or the method of interlocking with the movement of the eye is disclosed.
  • an aspect of the present invention is to provide an information processing apparatus capable of further facilitating communication between users in a virtual space.
  • An information processing apparatus is an information processing apparatus that controls an operation of a user object in a virtual space associated with a user who wears a display device, and the user's head detected in real space
  • the movement information acquisition unit for acquiring movement information of the unit and the eye
  • the provisional gaze determination unit for determining the provisional gaze of the user object based on the movement information, and the positional relationship between the provisional gaze and one or more objects in the virtual space
  • a gaze control unit for controlling the gaze of the user object based on the image generation unit for generating an image to be displayed on the display device, the gaze control unit reflecting the gaze of the user object controlled by the gaze control unit; And.
  • the provisional gaze of the user object is determined on the basis of the motion information of the head and eyes of the user actually detected. Then, the line of sight of the user object is controlled based on the positional relationship between the provisional line of sight and the one or more objects in the virtual space, and an image displayed on the display device is generated. According to such processing, the line of sight of the user object controlled so as to be directed more naturally as compared with the case where the line of sight (motion information) of the user detected actually is directly applied to the line of sight of the user object. Can be presented to the user. As a result, communication between users in the virtual space via user objects can be further facilitated.
  • FIG. 1 is a diagram showing a functional configuration of an information processing apparatus 10 according to an embodiment of the present invention.
  • the information processing apparatus 10 is an apparatus for providing a user with a virtual space in which arbitrary VR contents such as a game space and a chat space are expanded, via a head mounted display (HMD) 1 (display device) mounted to the user. is there.
  • the information processing apparatus 10 controls the operation (in particular, line of sight) of the user object in the virtual space.
  • HMD head mounted display
  • the line of sight of each user object controlled by the information processing apparatus 10 is reflected in an image (video) displayed on the HMD 1 worn by each user.
  • each user grasps the gaze of the other user based on the gaze of the user object displayed on the HMD 1 (for example, the gaze of the head of the user object, the gaze of the user object represented by the position of eyes, etc.) can do.
  • the processing of the information processing apparatus 10 described later can be applied even when only a user object of one user exists in the same virtual space.
  • the line of sight of the user object controlled by the information processing apparatus 10 is reflected on, for example, an image or the like obtained by photographing and recording the user object from a third party viewpoint in the virtual space.
  • the information processing apparatus 10 includes a communication unit 11, an operation information acquisition unit 12, a provisional gaze determination unit 13, a gaze control unit 14, and an image generation unit 15.
  • the information processing apparatus 10 is, for example, a game terminal, a personal computer, a tablet terminal or the like that can communicate with the plurality of HMDs 1 attached by each of the plurality of users.
  • the implementation form of the information processing apparatus 10 is not limited to a specific form.
  • the information processing device 10 may be a computer device incorporated in the same device as the HMD 1.
  • the information processing apparatus 10 may be a server device or the like that can communicate with each of the HMDs 1 (or each computer terminal that controls the operation of each HMD 1) of each of a plurality of users via a communication line such as the Internet. Further, the information processing apparatus 10 may be physically configured by a single device or may be configured by a plurality of devices. For example, in the information processing apparatus 10, a part of functions (for example, the function of the image generation unit 15) are realized by a computer terminal that controls the operation of each HMD 1, and other functions are transmitted by a server device that can communicate with the computer terminal. It may be configured as a distributed system to be realized.
  • the HMD 1 is a display device mounted on the body (for example, the head) of the user.
  • the HMD 1 includes, for example, a display unit that displays an image (an image for the left eye and an image for the right eye) in front of each eye of the user in a state of being worn on the head of the user.
  • a stereoscopic image three-dimensional image
  • the display unit described above may be a display integrally configured with a main unit mounted on the user's body, such as a glasses type or helmet type, or a device detachably attachable to the main unit of the HMD 1 (For example, a display of a terminal such as a smartphone attached to the main unit) may function as the display unit.
  • the HMD 1 includes, for example, a sensor (eg, an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, a gyro sensor, etc.) capable of detecting the position, orientation (tilt), velocity, acceleration, etc. of the user's head (ie, the HMD 1). There is.
  • the HMD 1 periodically transmits information on the motion (position, orientation, velocity, acceleration, etc.) of the head of the user detected by such a sensor to the information processing apparatus 10 as motion information on the head of the user.
  • the HMD 1 includes, for example, a sensor such as an infrared camera that detects an action of the user's eyes (for example, the position and the movement of the black eye portion, etc.).
  • the sensor is, for example, a sensor having a known eye tracking function.
  • the said sensor detects the operation
  • the HMD 1 periodically transmits the operation information of the user's eyes detected as described above to the information processing apparatus 10.
  • the HMD 1 also includes a microphone (not shown) for inputting the voice of the user wearing the HMD 1 and a speaker (not shown) for outputting voice and the like of each user as accessories.
  • the voice acquired by the microphone is transmitted to the information processing apparatus 10.
  • the speaker outputs the voice or the like of the other user received from the information processing device 10. With such a microphone and a speaker, it is possible to make conversation (chat) between a plurality of users.
  • the microphone and the speaker may be devices integrated with the HMD 1 or may be devices different from the HMD 1.
  • the motion information acquisition unit 12 acquires motion information of the head and eyes of the user detected in the real space.
  • the motion information acquisition unit 12 acquires motion information of the head and eyes of each user received from the HMD 1 by the communication unit 11.
  • the temporary line-of-sight determination unit 13 determines a temporary line-of-sight of the user object associated with the user based on the user's motion information acquired by the motion information acquisition unit 12.
  • the user object is, for example, a character and an avatar operated by the user.
  • the provisional gaze determination unit 13 determines the provisional gaze of the user object in the virtual space by associating the motion information of the head and eyes of the user as the movement of the head and eyes of the user object in the virtual space.
  • the provisional gaze of the user object is, for example, information including at least one of a gaze direction (visual axis) and a viewpoint (gaze point).
  • the process of the temporary gaze determination unit 13 is performed for each of a plurality of users. As a result, the provisional gaze of each user object associated with each user is determined.
  • FIG. 2 is a diagram schematically showing an example of a virtual space V including user objects 21 (21A to 21C) associated with a plurality of (here, three) users.
  • the process of the gaze control unit 14 described above will be described focusing on the case of controlling the gaze of the user object 21A.
  • the line-of-sight control unit 14 sets the candidate point P of the user object 21 to the candidate point P of the user object 21 as an index indicating the distance between the provisional line of sight 31 and the user object 21 (here, the user object 21C as an example).
  • An angle ⁇ 1 formed between the virtual line of sight 32 and the temporary line of sight 31 is calculated.
  • the line-of-sight control unit 14 directs the candidate point P of the object object 22 from the base point 31 a of the temporary line of sight 31 as an index indicating the distance between the temporary line of sight 31 and the object object 22 (here, the object object 22A as an example).
  • An angle ⁇ 2 between the virtual line of sight 33 and the temporary line of sight 31 is calculated.
  • the line-of-sight control unit 14 detects an angle ⁇ 2 h (that is, a horizontal component of the angle ⁇ 2) between the virtual line of sight 33 and the virtual line of sight 31 in the horizontal direction in the virtual space V and the vertical direction (vertical direction) in the virtual space V
  • the angle ⁇ 2 v that is, the vertical component of the angle ⁇ 2 formed by the virtual line of sight 33 and the virtual line of sight 31 is calculated, and the angle ⁇ 2 is calculated based on the angle ⁇ 2 h and the angle ⁇ 2 v .
  • the sight line control unit 14 compares these index values (angles ⁇ 1 and ⁇ 2) with a predetermined threshold value Th. Then, the line-of-sight control unit 14 extracts, as candidate objects, objects in which the angles ⁇ 1 and ⁇ 2 are equal to or smaller than the threshold value Th.
  • the object object 22A is extracted as a candidate object, while the user object 21C is not extracted as a candidate object.
  • the gaze control unit 14 controls the gaze of the user object 21A so as to face the object object 22A (candidate point P of the object object 22A). That is, the line-of-sight control unit 14 causes the line of sight of the user object 21A to coincide with the virtual line of sight 33.
  • the gaze control unit 14 selects one candidate object based on at least one of the type and the state of each candidate object, and selects the one candidate object.
  • the line of sight of the user object 21A is controlled to turn. For example, in each object, a priority based on at least one of the type and the state of the object is set.
  • the gaze control unit 14 determines a candidate object to which the gaze of the user object 21A is directed based on the priorities set for the candidate objects. Then, the gaze control unit 14 can control the gaze of the user object 21A so as to turn to the candidate object determined as the destination of the gaze of the user object 21A.
  • an object object 22 moving in the virtual space V (hereinafter also referred to as “moving object”) is higher than an object object 22 moving in the virtual space V (hereinafter also referred to as “stationary object”).
  • Priority is set. Such setting of the priority is based on the idea that objects activated in the virtual space V (moving objects) are more likely to be noticed by the user than stationary objects.
  • the user object 21 associated with the uttering user (hereinafter also referred to as “speech object”) is a user object 21 associated with the user not uttering (hereinafter also referred to as “non-speaking object” Higher priority than.) Is set.
  • speech object is a user object 21 associated with the user not uttering (hereinafter also referred to as “non-speaking object” Higher priority than.) Is set.
  • non-speaking object Higher priority than.
  • the gaze control unit 14 determines whether or not the user associated with the user object 21 is inputting a sound of a volume greater than or equal to a predetermined size into the microphone (a speech state). It can be determined whether the object 21 is a speech object. Here, whether or not it is in the speech state is, for example, whether or not a predetermined time (for example, a preset number of seconds, etc.) has elapsed since the sound of a volume greater than or equal to the predetermined size was input last time. It can be determined based on.
  • a predetermined time for example, a preset number of seconds, etc.
  • the sight-line control unit 14 determines that the user is not in the speech state, and the volume of the volume greater than or equal to the predetermined volume. If the predetermined time has not elapsed since the voice was previously input, it is determined that the user is in the speech state.
  • the method of determining whether or not in the speech state is not limited to the above method.
  • the gaze control unit 14 excludes the non-speaking object from the candidates for turning the gaze of the user object 21A.
  • the gaze control unit 14 may include the non-speaking object as a candidate for turning the gaze of the user object 21A.
  • Moving object Utterance object 3.
  • Static object 4.
  • the gaze control unit 14 controls so that the candidate object having high priority is preferentially selected as the destination of the gaze of the user object 21A.
  • An example of such control (see FIG. 4) will be described later.
  • the gaze control unit 14 determines the index values (angles ⁇ 1 and ⁇ 2 described above). The candidate object with the smallest) may be selected as the destination of the user object 21A.
  • the sight line control unit 14 determines whether or not the user associated with the user object 21A is speaking, and if the user is speaking, the user associated with the user speaking
  • the line of sight of the user object 21A may be controlled based on control content predetermined for an object (that is, a speech object).
  • the line of sight of the user object 21A may be controlled based on the preset special control content so as to behave appropriately as a speaker (for example, a presenter or the like).
  • the line of sight control unit 14 when the provisional line of sight 31 of the user object 21A is fixed within a certain range (for example, when the amount of change per unit time of the provisional line of sight 31 is within a predetermined threshold), the line of sight control unit 14 Alternatively, the provisional line of sight 31 of the user object 21A may be adopted as the line of sight after control of the user object 21A.
  • the line-of-sight control unit 14 may control the line of sight of the user object 21A so as to sequentially turn the line of sight to each of the user objects 21B and 21C.
  • the image generation unit 15 generates an image (an image displayed on each HMD 1) on which the line of sight of the user object 21A controlled by the line of sight control unit 14 is reflected.
  • the image generation unit 15 detects the line of sight (line of sight direction and / or viewpoint) of the user object 21A controlled by the line-of-sight control unit 14 and the face direction and eyes (black eyes) of the user object 21A displayed in the image in each HMD 1
  • An image adjusted so as to coincide with the line of sight (line of sight direction and / or viewpoint) specified by the position etc. is generated.
  • the image generated by the image generation unit 15 is output to each HMD 1 via the communication unit 11. Through such processing, an image in which the line of sight of the user object 21A controlled by the line-of-sight control unit 14 is reflected is presented to each user via each HMD 1.
  • FIGS. 3 to 6 show processing focused on the case of performing the gaze control of the user object 21A shown in FIG.
  • the line-of-sight control of the user objects 21B and 21C associated with other users is the same as the line-of-sight control of the user object 21A, and thus the description thereof is omitted.
  • the information processing apparatus 10 generates a virtual space V (see FIG. 2) shared by a plurality of users (step S1). Specifically, a virtual space V in which the user object 21 and the object object 22 or the like associated with each user are arranged at the initial position is generated. Subsequently, the operation information acquisition unit 12 acquires the head and eye operation information of each user received from each HMD 1 via the communication unit 11 (step S2). Subsequently, based on the motion information of the user acquired by the motion information acquisition unit 12, the provisional gaze determination unit 13 selects the provisional gaze 31 of the user object 21 (here, the user object 21A) associated with the user. decide.
  • the gaze control unit 14 determines the positional relationship between the provisional gaze 31 of the user object 21A determined by the provisional gaze determination unit 13 and one or more objects in the virtual space V (the other user object 21 and the object object 22).
  • the line of sight of the user object 21A is controlled based on (step S4).
  • step S4 An example of the detailed processing procedure of step S4 will be described with reference to FIG.
  • the processing procedure illustrated in FIG. 4 is an example of a determination procedure for selecting a candidate object to which the user object 21A is directed in accordance with the priority order described above.
  • the gaze control unit 14 determines whether the user associated with the user object 21A to be processed is speaking (step S41).
  • step S41 YES
  • the sight control unit 14 controls the sight of the user object 21A based on the predetermined control content as described above (step S42). For example, when the temporary line of sight 31 of the user object 21A is fixed within a certain range, the line-of-sight control unit 14 adopts the temporary line of sight 31 of the user object 21A as the line of sight after control of the user object 21A.
  • the line-of-sight control unit 14 sends the line of sight to each of the user objects 21B and 21C in order. Control the line of sight of 21A. By such processing, the line of sight of the user object 21A is controlled so as to behave appropriately as a speaker.
  • the gaze control unit 14 determines the angle between the virtual gaze 31 and the virtual gaze 33 (a line connecting the base point 31a and the candidate point P of the object object 22). It is determined whether there is a moving object for which ⁇ 2 is less than or equal to the threshold Th (step S43). If there is a moving object (candidate object) that satisfies the above determination condition (step S43: YES), the sight line control unit 14 moves a plurality of moving objects that satisfy the above determination condition (if there are multiple moving objects that satisfy the above determination condition , The gaze point of the user object 21 is directed to the candidate point P of the moving object with the smallest angle ⁇ 2. That is, the line-of-sight control unit 14 causes the line of sight of the user object 21 to coincide with the virtual line of sight 33 connecting the base point 31 a and the candidate point P of the moving object.
  • FIG. 5 shows a state in which the object object 22A is held by the user object 21C and the object object 22A is moved by moving the hand of the user object 21C.
  • the object 22A is determined to be a moving object.
  • the angle ⁇ 2 formed by the virtual line of sight 31 and the virtual line of sight 33 (the line segment connecting the base point 31a and the candidate point P of the object object 22A) is less than or equal to the threshold value Th, the object object 22A performs step S43. Satisfies the judgment condition of In that case, the line-of-sight control unit 14 causes the line of sight of the user object 21 to coincide with the virtual line of sight 33.
  • the sight line control unit 14 connects the provisional sight line 31 and the virtual sight line 32 (a line connecting the base point 31a and the candidate point P of the user object 21). It is determined whether there is a speech object whose angle ⁇ 1 formed with the minute) is equal to or smaller than the threshold Th (step S45).
  • step S45 the gaze control unit 14 determines that there are a plurality of speech objects that satisfy the above determination conditions (if there are multiple speech objects that satisfy the above determination conditions).
  • the gaze point of the user object 21 is directed to the candidate point P of the utterance object having the smallest angle ⁇ 1. That is, the line-of-sight control unit 14 causes the line of sight of the user object 21 to coincide with the virtual line of sight 32 connecting the base point 31 a and the candidate point P of the speech object.
  • the object 22A is a stationary object.
  • the object object 22A performs step S47. Satisfies the judgment condition of Therefore, in this case, the line-of-sight control unit 14 causes the line of sight of the user object 21 to coincide with the virtual line of sight 33.
  • an angle formed by a line connecting the base point 31a and the candidate point P of the object object 22B and the virtual line of sight 31 (hereinafter referred to as "angle A") may also be equal to or less than the threshold value Th.
  • the gaze control unit 14 selects the object object 22A as a destination to which the gaze of the user object 21 is directed.
  • the image generation unit 15 generates an image (an image displayed on each HMD 1) in which the line of sight of the user object 21A controlled by the line of sight control unit 14 is reflected (step S5).
  • the image for each user generated by the image generation unit 15 (that is, the image according to the field of view of the user object 21 associated with each user) is transmitted to each HMD 1 via the communication unit 11, for example, Displayed on the HMD1.
  • each user recognizes the line of sight of the user object 21A controlled by the line-of-sight control unit 14 through the image displayed on the HMD 1.
  • the virtual line of sight 31 of the user object 21A is once determined based on the motion information of the head and eyes of the user actually detected.
  • the line of sight of the user object 21A is controlled based on the positional relationship between the virtual line of sight 31 and one or more objects in the virtual space V (in the present embodiment, the other user object 21 and the object object 22).
  • An image to be displayed is generated.
  • the user object 21A controlled so as to turn in a more natural direction as compared with the case where the user's line of sight (action information) actually detected is directly applied to the line of sight of the user object 21A.
  • communication between users on the virtual space V via the user object 21A can be further facilitated.
  • the sight line control unit 14 calculates an index value (in the present embodiment, the angles ⁇ 1 and ⁇ 2) indicating the distance between the virtual line of sight 31 and the object, compares the index value with a predetermined threshold Th, and the index value is a threshold
  • the line of sight of the user object 21 is controlled to turn to the candidate object.
  • the line of sight of the user object 21 is controlled such that the distance between the provisional line of sight 31 and the object faces a candidate object closer than a predetermined reference.
  • a detection error may occur in the user's head or eye movement information detected by a sensor or the like mounted on the HMD 1.
  • the line of sight based on the motion information that is, the provisional line of sight 31
  • the detection error directly affects the line of sight of the user object 21.
  • an error occurs in the line of sight of the user object 21 by the amount of the detection error, and the line of sight of the user object 21 deviates from the direction toward the speech object.
  • the user in order to display the user object 21 associated with another user (second user) in a view image provided via the HMD 1 to a certain user (first user), the user generates the relevant view image.
  • the information processing apparatus 10 needs to receive the operation information of the second user, and control the operation of the user object 21 associated with the second user based on the operation information.
  • a delay communication delay, processing delay, etc.
  • the actual second user's operation and the user object 21 associated with the second user There may be a time difference between the operation and the operation.
  • the first user is provided with an uncomfortable video such as the face orientation and the eyes of the user object 21 changing discontinuously (suddenly changing greatly).
  • the processing of the information processing apparatus 10 the line-of-sight control unit 14
  • the line of sight of the user object 21 faces the candidate object even when the actual line of sight of the user It can be fixed in the direction.
  • the operation of the head or eyes of the second user and the operation (line of sight) of the user object 21 associated with the second user need to be completely interlocked Absent. That is, according to the control of the information processing apparatus 10, the line of sight of the user object 21 can be fixed in the direction toward the candidate object as long as the line of sight of the second user (temporary line of sight 31) does not deviate significantly from the candidate object. . During this time, it is not necessary to reflect the actual motion of the head or eyes of the second user on the motion (gaze line) of the user object 21. This can effectively reduce the processing load and usage of hardware resources such as processors and memories.
  • the angles ⁇ 1 and ⁇ 2 formed by the virtual line of sight 31 and the virtual lines of sight 32 and 33 are used as index values, but the index values may be other values.
  • the sight line control unit 14 may perform the above determination using only the horizontal component (angle ⁇ 1 h ) or the vertical component (angle ⁇ 1 v ) of the angle ⁇ 1 instead of the angle ⁇ 1. That is, instead of comparing the angle ⁇ 1 with the threshold value Th, the sight line control unit 14 compares the horizontal component (angle ⁇ 1 h ) or the vertical component (angle ⁇ 1 v ) of the angle ⁇ 1 with a predetermined threshold value. , Candidate objects may be extracted.
  • the sight control unit 14 may perform the above determination using only the horizontal component (angle ⁇ 2 h ) or the vertical component (angle ⁇ 2 v ) of the angle ⁇ 2 instead of the angle ⁇ 2. That is, instead of comparing the angle ⁇ 2 with the threshold value Th described above, the sight-line control unit 14 compares the horizontal component (angle ⁇ 2 h ) or the vertical component (angle ⁇ 2 v ) of the angle ⁇ 2 with a predetermined threshold value. , Candidate objects may be extracted.
  • provisional gaze determination unit 13 determines a provisional gaze including not only the gaze direction of the user but also the viewpoint (gaze point) based on the user's head and eye movement information described above.
  • a distance for example, a linear distance on the virtual space V
  • an object candidate point P set in the object
  • the gaze control unit 14 selects one candidate object based on at least one of the type and the state of each candidate object, and directs the user object 21 to face the one candidate object. Control your gaze.
  • the moving object (the object object 22 indicating an object moving in the virtual space V) is set to have a higher priority than the stationary object (the object object 22 indicating an object not moving in the virtual space V)
  • the gaze control unit 14 determines a candidate object to which the user object 21 is directed based on the priority, and the gaze control unit 14 of the user object 21 faces the determined candidate object. Control your gaze.
  • the user object 21 is directed to the object that is likely to be focused by the user.
  • the user object 21 associated with the uttering user a priority higher than that of the non-utterance object (the user object 21 associated with the user not uttering) is set.
  • the user object is directed to an object that is highly likely to be focused by the user, based on the present inventor's finding that the utterance object is more likely to attract the user's eyes than the non-speaking object. 21 lines of sight can be controlled.
  • each functional block is realized by one physically and / or logically coupled device, or directly and / or indirectly two or more physically and / or logically separated devices. It may be connected (for example, wired and / or wirelessly) and realized by the plurality of devices.
  • the term “device” can be read as a circuit, a device, a unit or the like.
  • the hardware configuration of the information processing device 10 may be configured to include one or more of the devices illustrated in FIG. 9 or may be configured without including some devices.
  • Each function in the information processing apparatus 10 causes the processor 1001 to perform an operation by reading predetermined software (program) on hardware such as the processor 1001, the memory 1002, etc., communication by the communication device 1004, the memory 1002 and the storage 1003. This is realized by controlling the reading and / or writing of data in
  • the processor 1001 operates, for example, an operating system to control the entire computer.
  • the processor 1001 may be configured by a central processing unit (CPU) including an interface with a peripheral device, a control device, an arithmetic device, a register, and the like.
  • CPU central processing unit
  • the processor 1001 reads a program (program code), a software module, and / or data from the storage 1003 and / or the communication device 1004 to the memory 1002, and executes various processing according to these.
  • a program a program that causes a computer to execute at least a part of the operations described in the above embodiments is used.
  • the line-of-sight control unit 14 of the information processing apparatus 10 may be realized by a control program stored in the memory 1002 and operated by the processor 1001, and may be realized similarly for other functional blocks shown in FIG. Good.
  • the various processes described above have been described to be executed by one processor 1001, but may be executed simultaneously or sequentially by two or more processors 1001.
  • the processor 1001 may be implemented by one or more chips.
  • the program may be transmitted from the network via a telecommunication line.
  • the memory 1002 is a computer readable recording medium, and includes, for example, at least one of a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), and a RAM (Random Access Memory). It may be done.
  • the memory 1002 may be called a register, a cache, a main memory (main storage device) or the like.
  • the memory 1002 can store a program (program code), a software module, etc. that can be executed to execute the information processing method (for example, the procedure shown in the flowcharts of FIGS. 3 and 4) according to the above embodiment. .
  • the storage 1003 is a computer readable recording medium, and is, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (for example, a compact disk, a digital versatile disk, Blu-ray A (registered trademark) disk, a smart card, a flash memory (for example, a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like may be used.
  • the storage 1003 may be called an auxiliary storage device.
  • the above-described storage medium may be, for example, a database including the memory 1002 and / or the storage 1003, a server, or any other suitable medium.
  • the communication device 1004 is hardware (transmission / reception device) for performing communication between computers via a wired and / or wireless network, and is also called, for example, a network device, a network controller, a network card, a communication module, or the like.
  • the input device 1005 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, a sensor, and the like) that receives external input.
  • the output device 1006 is an output device (for example, a display, a speaker, an LED lamp, etc.) that performs output to the outside.
  • the input device 1005 and the output device 1006 may be integrated (for example, a touch panel).
  • each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information.
  • the bus 1007 may be configured by a single bus or may be configured by different buses among the devices.
  • the information processing apparatus 10 includes hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA). And part or all of each functional block may be realized by the hardware.
  • processor 1001 may be implemented in at least one of these hardware.
  • the gaze control unit 14 may adjust the index value or the threshold based on the size of the object.
  • FIG. 7 shows a rectangular object object 22C having a width w1 and a height h1 and a rectangular object object 22D having a width w2 ( ⁇ w1) and a height h2 ( ⁇ h1) in the virtual space V. Are adjacent to each other.
  • FIG. 7 schematically shows an image (visual field) of the virtual space V as viewed from the user object 21 to be subjected to the gaze control.
  • the candidate points P are set at the center points of the object objects 22C and 22D.
  • the virtual line of sight 31 of the user object 21 includes information indicating the viewpoint (point of gaze) P0, and the gaze control unit 14 determines the distances d1 and d2 between the viewpoint P0 and the candidate points P of the respective objects. Is used as the index value described in the above embodiment.
  • the object 22D is determined as a candidate object to which the user object 21 is directed. Further, even when “d1 ⁇ Th” and “d2 ⁇ Th” hold, since “d1> d2”, the object object 22D is determined as a candidate object to which the user object 21 is directed. However, as shown in FIG. 7, the viewpoint P0 of the user object 21 is actually located at the edge of the object object 22C. That is, the user corresponding to the user object 21 has a high possibility of looking at the object object 22C instead of the object object 22D.
  • the index value d1 new adjusted in this manner, the possibility that “d1 new ⁇ Th” is established is increased. Further, the radius “ ⁇ (w1 2 + h1 2 ) / 2” of the object 22C is larger than the radius “ ⁇ (w2 2 + h2 2 ) / 2” of the object 22D, and “d1 new ⁇ d2 new ” is To establish. Therefore, even if “d1 new ⁇ Th” and “d2 new ⁇ Th” are satisfied, the object object 22C is a candidate object to which the user object 21 is directed at a higher priority than the object object 22D. It is determined.
  • the thresholds may be adjusted individually for the object objects 22C and 22D instead of the index values d1 and d2.
  • the gaze control unit 14 may compare the index value d1 of the object 22C with the threshold value Th1 instead of the threshold value Th, which is calculated by the following (Equation 3).
  • the gaze control unit 14 may compare the index value d2 of the object 22D with the threshold value Th2 calculated by the following (Expression 4) instead of the threshold value Th.
  • Th1 Th + ⁇ (w1 2 + h1 2 ) / 2 (Equation 3)
  • Th2 Th + ⁇ (w2 2 + h2 2 ) / 2 (Equation 4)
  • the determination condition is relaxed as the size of the object object 22 (here, the above radius) is larger. That is, the larger the size of the object object 22 is, the higher the possibility of satisfying the condition (“d1 ⁇ Th1” or “d2 ⁇ Th2”) for being extracted as a candidate object.
  • the sight control unit 14 compares the sight of the user object 21 by, for example, comparing “d1 ⁇ Th1” and “d2 ⁇ Th2”.
  • the candidate object to which to direct may be determined.
  • the sight line control unit 14 determines the object object 22C as a candidate object to which the line of sight of the user object 21 is directed, “d1 ⁇ Th1> In the case of “d 2 ⁇ Th 2”, the object object 22 D may be determined as a candidate object to which the user object 21 is directed. In this way, the same control result as in the case of using the index values d1 new and d2 new described above can be obtained.
  • the index value (here, d1, d2) or the threshold value Th is adjusted in consideration of the size of the object (here, the object object 22), and the candidate is selected based on the adjusted index value or threshold value It is possible to judge the object. As a result, it is possible to control the line of sight of the user object 21 so as to turn to a more appropriate object (candidate point P).
  • the gaze control unit 14 directs the gaze of the user object 21 to the candidate object based on the movement mode of the candidate object. It may be determined whether or not.
  • the candidate object which is a moving object is a movement aspect which moves a fixed distance or more per unit time (for example, when moving a predetermined threshold distance at a predetermined threshold speed or more)
  • the line of sight (temporary line of sight 31) and the line of sight after adjustment (line of sight facing the candidate object) will momentarily diverge greatly.
  • the gaze control unit 14 may not direct the gaze of the user object 21 to the candidate object. That is, the sight control unit 14 may not cause the sight of the user object 21 to follow the candidate object.
  • the gaze control unit 14 may derive the moving range of the latest candidate object from, for example, the position of the candidate object in a predetermined period in the past, and fix the viewpoint of the user object 21 at the center point of the moving range. .
  • the line of sight of the user object 21 is an unnatural line of sight (in the above example, a line of sight completely following the candidate object moving at high speed) based on the movement mode of the candidate object. It can be suppressed.
  • the sight line control unit 14 determines a range that configures a part of the virtual space V based on the tentative sight line 31, extracts an object included in the range, and a positional relationship between the tentative sight line 31 and the extracted object.
  • the line of sight of the user object 21 may be controlled based on FIG. 8 illustrates an example of an extraction range of an object determined based on the virtual line of sight 31.
  • the line-of-sight control unit 14 determines a spherical region having a radius r from the viewpoint P0 of the provisional line of sight 31 as the above-described range R.
  • the user object 21C and the object objects 22E, 22F, and 22G are extracted as candidates for turning the line of sight of the user object 21. Then, the other objects are excluded from the candidates to which the user object 21 looks at, and the processing such as the above-described threshold comparison is not performed.
  • the computational complexity of the gaze control unit 14 is achieved by narrowing down the objects to be calculated (candidates to which the gaze of the user object 21 is directed). Can be suppressed. As a result, it is possible to speed up the processing and reduce the amount of memory used.
  • range R is an example, and, for example, when an utterance object is present ahead of the viewpoint P0 of the virtual line of sight 31, a spherical region with a radius r centered on the candidate point P of the utterance object is It may be used instead of range R mentioned above.
  • the input / output information may be stored in a specific place (for example, a memory) or may be managed by a management table. Information to be input or output may be overwritten, updated or added. The output information etc. may be deleted. The input information or the like may be transmitted to another device.
  • the determination may be performed by a value (0 or 1) represented by one bit, may be performed by a true / false value (Boolean: true or false), or may be compared with a numerical value (for example, a predetermined value). Comparison with the value).
  • Software may be called software, firmware, middleware, microcode, hardware description language, or any other name, and may be instructions, instruction sets, codes, code segments, program codes, programs, subprograms, software modules. Should be interpreted broadly to mean applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, etc.
  • software, instructions and the like may be transmitted and received via a transmission medium.
  • software may use a wireline technology such as coaxial cable, fiber optic cable, twisted pair and digital subscriber line (DSL) and / or a website, server or other using wireless technology such as infrared, radio and microwave When transmitted from a remote source, these wired and / or wireless technologies are included within the definition of transmission medium.
  • wireline technology such as coaxial cable, fiber optic cable, twisted pair and digital subscriber line (DSL) and / or a website, server or other using wireless technology such as infrared, radio and microwave
  • data, instructions, commands, information, signals, bits, symbols, chips etc may be voltage, current, electromagnetic waves, magnetic fields or particles, light fields or photons, or any of these May be represented by a combination of
  • information, parameters, and the like described in the present specification may be represented by an absolute value, may be represented by a relative value from a predetermined value, or may be represented by corresponding other information. .
  • the phrase “based on” does not mean “based only on,” unless expressly stated otherwise. In other words, the phrase “based on” means both “based only on” and “based at least on.”
  • determining may encompass a wide variety of operations. “Decision” may be, for example, judging, calculating, computing, processing, deriving, investigating, looking up (eg table, database or other (Searching in the data structure of (a)), ascertaining it may be regarded as “decided”, and the like. Also, “determination” may be receiving (e.g., receiving information), transmitting (e.g., transmitting information), input (input), output (output), accessing (accessing) (e.g. For example, it can be regarded as “determining” access to data in memory. Also, “determining” may include considering “resolving", selecting, choosing, establishing, comparing, etc., as “determining”. That is, “determination” may include considering that some action is "decision”.
  • SYMBOLS 1 HMD (display apparatus), 10 ... Information processing apparatus, 12 ... Operation information acquisition part, 13 ... Temporary gaze determination part, 14 ... Line-of-sight control part, 15 ... Image generation part, 21, 21A, 21B, 21C ... User object 22, 22A, 22B, 22C, 22D, 22E, 22G, ... object object, 31 ... temporary line of sight, ⁇ 1, ⁇ 2 ... angle (index value), V ... virtual space.

Abstract

L'invention concerne un dispositif de traitement d'informations (10) qui est un dispositif de traitement d'informations commandant l'action d'un objet d'utilisateur (21), dans un espace virtuel (V), associé à un utilisateur qui porte un HMD (1). Le dispositif de traitement d'informations (10) est pourvu d'une unité d'acquisition d'informations d'action (12) acquérant des informations d'action concernant des actions de la tête et des yeux d'un utilisateur détectées dans un espace réel, une unité de détermination de ligne de visée virtuelle (13) qui détermine une ligne de visée virtuelle (31) d'un objet d'utilisateur (21) sur la base des informations d'action, une unité de commande de ligne de visée (14) commandant la ligne de visée de l'objet d'utilisateur (21) sur la base d'une relation de position entre la ligne de visée virtuelle (31) et un ou plusieurs objets dans l'espace virtuel (V), et une unité de génération d'image (15) générant une image reflétant la ligne de visée de l'objet d'utilisateur (21) commandée par l'unité de commande de ligne de visée (14), l'Image étant affichée sur le HMD (1).
PCT/JP2018/044297 2017-12-26 2018-11-30 Dispositif de traitement d'informations WO2019130992A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US16/957,581 US11282481B2 (en) 2017-12-26 2018-11-30 Information processing device
JP2019562896A JP6970757B2 (ja) 2017-12-26 2018-11-30 情報処理装置

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2017-249037 2017-12-26
JP2017249037 2017-12-26

Publications (1)

Publication Number Publication Date
WO2019130992A1 true WO2019130992A1 (fr) 2019-07-04

Family

ID=67067238

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2018/044297 WO2019130992A1 (fr) 2017-12-26 2018-11-30 Dispositif de traitement d'informations

Country Status (3)

Country Link
US (1) US11282481B2 (fr)
JP (1) JP6970757B2 (fr)
WO (1) WO2019130992A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113495624A (zh) * 2020-03-19 2021-10-12 马自达汽车株式会社 状态推断装置
JP2022526512A (ja) * 2019-11-28 2022-05-25 北京市商▲湯▼科技▲開▼▲發▼有限公司 インタラクティブオブジェクト駆動方法、装置、機器、及び記憶媒体
US11681364B1 (en) * 2021-06-29 2023-06-20 Amazon Technologies, Inc. Gaze prediction

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11393251B2 (en) 2018-02-09 2022-07-19 Pupil Labs Gmbh Devices, systems and methods for predicting gaze-related parameters
WO2019154511A1 (fr) * 2018-02-09 2019-08-15 Pupil Labs Gmbh Dispositifs, systèmes et procédés de prédiction de paramètres liés au regard à l'aide d'un réseau neuronal
WO2019154509A1 (fr) 2018-02-09 2019-08-15 Pupil Labs Gmbh Dispositifs, systèmes et procédés de prédiction de paramètres liés au regard
US11676422B2 (en) 2019-06-05 2023-06-13 Pupil Labs Gmbh Devices, systems and methods for predicting gaze-related parameters

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH09153146A (ja) * 1995-09-28 1997-06-10 Toshiba Corp 仮想空間表示方法
JPH11161815A (ja) * 1997-07-02 1999-06-18 Sega Enterp Ltd 立体視の表示装置及びその画像処理を行うプログラムを記録した記録媒体
JP2000020753A (ja) * 1998-07-07 2000-01-21 Mitsubishi Electric Corp 仮想空間制御装置

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6706473B2 (ja) 2015-09-14 2020-06-10 株式会社コーエーテクモゲームス 情報処理装置、表示制御方法、及び表示制御プログラム
US10762688B2 (en) * 2016-02-17 2020-09-01 Sony Corporation Information processing apparatus, information processing system, and information processing method
CN110024000B (zh) * 2016-11-24 2023-12-26 Groove X 株式会社 使瞳部变化的行为自主型机器人
US10438394B2 (en) * 2017-03-02 2019-10-08 Colopl, Inc. Information processing method, virtual space delivering system and apparatus therefor
JP6276882B1 (ja) * 2017-05-19 2018-02-07 株式会社コロプラ 情報処理方法、装置、および当該情報処理方法をコンピュータに実行させるためのプログラム
WO2019026617A1 (fr) * 2017-08-01 2019-02-07 ソニー株式会社 Dispositif de traitement d'informations et procédé de traitement d'informations

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH09153146A (ja) * 1995-09-28 1997-06-10 Toshiba Corp 仮想空間表示方法
JPH11161815A (ja) * 1997-07-02 1999-06-18 Sega Enterp Ltd 立体視の表示装置及びその画像処理を行うプログラムを記録した記録媒体
JP2000020753A (ja) * 1998-07-07 2000-01-21 Mitsubishi Electric Corp 仮想空間制御装置

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2022526512A (ja) * 2019-11-28 2022-05-25 北京市商▲湯▼科技▲開▼▲發▼有限公司 インタラクティブオブジェクト駆動方法、装置、機器、及び記憶媒体
CN113495624A (zh) * 2020-03-19 2021-10-12 马自达汽车株式会社 状态推断装置
US11681364B1 (en) * 2021-06-29 2023-06-20 Amazon Technologies, Inc. Gaze prediction

Also Published As

Publication number Publication date
JPWO2019130992A1 (ja) 2020-12-17
JP6970757B2 (ja) 2021-11-24
US20200335065A1 (en) 2020-10-22
US11282481B2 (en) 2022-03-22

Similar Documents

Publication Publication Date Title
WO2019130992A1 (fr) Dispositif de traitement d'informations
US11366516B2 (en) Visibility improvement method based on eye tracking, machine-readable storage medium and electronic device
JP7095030B2 (ja) 仮想現実コンテンツのビデオレンダリング速度の調整および立体画像の処理
EP2972559B1 (fr) Procédés et appareil d'affichage d'images sur un dispositif d'affichage monté sur tête (hmd)
EP3616050B1 (fr) Appareil et procédé pour contexte de commande vocale
US10223832B2 (en) Providing location occupancy analysis via a mixed reality device
US9798517B2 (en) Tap to initiate a next action for user requests
KR20190051028A (ko) 감각 안경류
JP7092108B2 (ja) 情報処理装置、情報処理方法、及びプログラム
KR20160111942A (ko) 자동 컨텐츠 스크롤링
US11327317B2 (en) Information processing apparatus and information processing method
US20210117048A1 (en) Adaptive assistive technology techniques for computing devices
US10423833B2 (en) Setting a personal status using augmented reality
WO2023058451A1 (fr) Dispositif de traitement d'informations, procédé de traitement d'informations et programme
WO2023037812A1 (fr) Système de prise en charge de dialogue en ligne
JP2018106644A (ja) 情報処理方法及び当該情報処理方法をコンピュータに実行させるためのプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18897564

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2019562896

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18897564

Country of ref document: EP

Kind code of ref document: A1