WO2024047720A1 - 仮想画像共有方法および仮想画像共有システム - Google Patents

仮想画像共有方法および仮想画像共有システム Download PDF

Info

Publication number
WO2024047720A1
WO2024047720A1 PCT/JP2022/032477 JP2022032477W WO2024047720A1 WO 2024047720 A1 WO2024047720 A1 WO 2024047720A1 JP 2022032477 W JP2022032477 W JP 2022032477W WO 2024047720 A1 WO2024047720 A1 WO 2024047720A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual image
display
user
display mode
display device
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/032477
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
伸悟 伊東
智和 足立
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Kyocera Corp
Original Assignee
Kyocera Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Kyocera Corp filed Critical Kyocera Corp
Priority to JP2024543628A priority Critical patent/JP7760742B2/ja
Priority to US19/106,925 priority patent/US20260037100A1/en
Priority to PCT/JP2022/032477 priority patent/WO2024047720A1/ja
Publication of WO2024047720A1 publication Critical patent/WO2024047720A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/16—Constructional details or arrangements
    • G06F1/1613—Constructional details or arrangements for portable computers
    • G06F1/163—Wearable computers, e.g. on a belt
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04845—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range for image manipulation, e.g. dragging, rotation, expansion or change of colour
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics

Definitions

  • the present disclosure relates to a virtual image sharing method and a virtual image sharing system.
  • VR virtual reality
  • MR mixed reality
  • AR augmented reality
  • the wearable terminal device has a display unit that covers the user's field of vision when the wearable terminal device is worn by the user.
  • a visual effect as if these exist is realized (for example, US Patent Application Publication No. 2019 /0087021 and US Patent Application Publication No. 2019/0340822).
  • MR is a technology that allows users to experience mixed reality, which is a fusion of real space and virtual images, by displaying a virtual image that appears to exist at a predetermined position in the real space while allowing the user to see the real space.
  • mixed reality which is a fusion of real space and virtual images
  • Patent Document 1 discloses a technique for allowing a plurality of users wearing see-through head-mounted displays to share a virtual object (for example, a virtual object of a building) displayed on the see-through head-mounted display.
  • Patent Document 1 when an operation (for example, a rotation operation or an enlargement/reduction operation) is performed on a virtual object, the way each user views the virtual object changes. Therefore, if the operation is performed blindly, there is a problem that the virtual object becomes difficult to see. Further, with the above technology, it is assumed that some users may want to operate a virtual object without being seen by other users.
  • an operation for example, a rotation operation or an enlargement/reduction operation
  • the virtual image sharing method and virtual image sharing system of the present disclosure have been made in view of the above problems, and aim to improve the usability of virtual images shared among multiple users.
  • one virtual image sharing method includes: A virtual image sharing method for sharing a virtual image between multiple display devices, the method comprising: The plurality of display devices includes at least a first display device and a second display device, A first virtual image located in a space and operable by a first user of the first display device and a second user of the second display device is displayed on each of the first display device and the second display device. display, A second virtual image located within the space, operable by the first user, and not displayed on the second display device is displayed on the first display device.
  • another virtual image sharing method includes: A virtual image sharing method for sharing a virtual image between multiple display devices, the method comprising: displaying a virtual image located in space and operable by a user of each of the plurality of display devices on each of the plurality of display devices; When a user of one of the plurality of display devices performs an operation to change the display mode of the virtual image, the display mode of the virtual image changed based on the operation is displayed on the one display device.
  • a first mode in which the virtual image is reflected in the display of the virtual image by another display device of the plurality of display devices, and also reflected in the display of the virtual image by other display devices of the plurality of display devices;
  • the display mode of the virtual image changed based on the operation is displayed on the one display device.
  • a second mode in which the virtual image is reflected in the display of the virtual image by another display device of the plurality of display devices, but not reflected in the display of the virtual image by other display devices of the plurality of display devices; Equipped with.
  • one virtual image sharing system includes: A virtual image sharing system that shares virtual images between multiple display devices,
  • the plurality of display devices includes at least a first display device and a second display device,
  • a first virtual image located in a space and operable by a first user of the first display device and a second user of the second display device is displayed on each of the first display device and the second display device.
  • display A second virtual image located within the space, operable by the first user, and not displayed on the second display device is displayed on the first display device.
  • another virtual image sharing system includes: A virtual image sharing system that shares virtual images between multiple display devices, displaying a virtual image located in space and operable by a user of each of the plurality of display devices on each of the plurality of display devices; When a user of one of the plurality of display devices performs an operation to change the display mode of the virtual image, the display mode of the virtual image changed based on the operation is displayed on the one display device.
  • a first mode in which the virtual image is reflected in the display of the virtual image by another display device of the plurality of display devices, and also reflected in the display of the virtual image by other display devices of the plurality of display devices;
  • the display mode of the virtual image changed based on the operation is displayed on the one display device.
  • a second mode in which the virtual image is reflected in the display of the virtual image by another display device of the plurality of display devices, but not reflected in the display of the virtual image by other display devices of the plurality of display devices; Equipped with.
  • FIG. 1 is a diagram showing a schematic configuration of a virtual image sharing system.
  • FIG. 2 is a block diagram showing the functional configuration of an information processing device.
  • FIG. 1 is a schematic perspective view showing the external configuration of a wearable terminal device.
  • FIG. 2 is a block diagram showing the functional configuration of a wearable terminal device.
  • 3 is a flowchart showing a control procedure of virtual image display control processing.
  • 7 is a flowchart showing a control procedure of a first reflection display process.
  • 12 is a flowchart showing a control procedure of a second reflection display process. It is a figure which shows the example of the visible area and 1st virtual image which are visually recognized by the 1st user wearing the wearable terminal device.
  • FIG. 1 is a diagram showing a schematic configuration of a virtual image sharing system.
  • FIG. 2 is a block diagram showing the functional configuration of an information processing device.
  • FIG. 1 is a schematic perspective view showing the external configuration of a wearable terminal device.
  • FIG. 3 is a diagram illustrating an example of a viewing area, a first virtual image, and a second virtual image that are viewed by a first user wearing a wearable terminal device. It is a figure which shows the example of a change of the display mode of a 1st virtual image. It is a figure which shows the example of a change of the display mode of a 1st virtual image.
  • FIG. 3 is a diagram illustrating an example of a viewing area, a first virtual image, and a second virtual image that are viewed by a first user wearing a wearable terminal device.
  • FIG. 3 is a diagram illustrating an example of a viewing area, a first virtual image, and a second virtual image that are viewed by a first user wearing a wearable terminal device.
  • FIG. 3 is a diagram illustrating an example of a viewing area, a first virtual image, and a second virtual image that are viewed by a first user wearing a wearable terminal device.
  • FIG. 3 is a diagram illustrating an example of a viewing area, a first virtual image, and a second virtual image that are viewed by a first user wearing a wearable terminal device.
  • FIG. 1 is a diagram showing a schematic configuration of a virtual image sharing system 100.
  • the virtual image sharing system 100 includes an information processing device 10 and a plurality of (for example, two) wearable terminal devices (display devices) 20 that are communicatively connected to the information processing device 10. Configured with the necessary features.
  • the information processing device 10 is a server device that performs display control of virtual images displayed on each wearable terminal device 20, etc.
  • the wearable terminal device 20 is an HMD (Head Mount Display) that is worn on the user's head. Specifically, the wearable terminal device 20 is a so-called MR/AR goggle that provides MR or AR to the user.
  • HMD Head Mount Display
  • MR/AR goggle that provides MR or AR to the user.
  • FIG. 2 is a block diagram showing the functional configuration of the information processing device 10. As shown in FIG.
  • the information processing device 10 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a storage section 13, a communication section 14, and a bus 15. Each part of the information processing device 10 is connected via a bus 15.
  • CPU Central Processing Unit
  • RAM Random Access Memory
  • the CPU 11 is a processor that performs various calculation processes and centrally controls the operation of each part of the information processing device 10.
  • the CPU 11 performs various control operations by reading and executing a program 131 stored in the storage unit 13. Note that although a single CPU 11 is illustrated in FIG. 2, the present invention is not limited to this. Two or more processors such as a CPU may be provided, and the processing executed by the CPU 11 of this embodiment may be shared and executed by these two or more processors.
  • the RAM 12 provides a working memory space for the CPU 11 and stores temporary data.
  • the storage unit 13 is a non-temporary recording medium that can be read by the CPU 11.
  • the storage unit 13 stores a program 131 executed by the CPU 11, various setting data, and the like.
  • the program 131 is stored in the storage unit 13 in the form of a computer-readable program code.
  • a nonvolatile storage device such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive) including a flash memory is used, for example.
  • the data stored in the storage unit 13 includes virtual image data related to virtual images.
  • the virtual image data includes data related to the display content of the virtual image, data on the display position, data on the orientation, and the like.
  • the communication unit 14 communicates with each wearable terminal device 20 to transmit and receive data.
  • the communication unit 14 receives data including part or all of the detection results by the sensor unit 25 of each wearable terminal device 20, information related to user operations detected by each wearable terminal device 20, and the like. Further, the communication unit 14 may be capable of communicating with other devices other than the wearable terminal device 20.
  • FIG. 3 is a schematic perspective view showing the external configuration of the wearable terminal device 20. As shown in FIG. 3
  • the wearable terminal device 20 includes a main body 20a, a visor 241 (display member) attached to the main body 20a, and the like.
  • the main body portion 20a is an annular member whose circumference can be adjusted.
  • Various devices such as a depth sensor 253 and a camera 254 are built inside the main body portion 20a.
  • the main body part 20a is worn on the head, the user's field of vision is covered by the visor 241.
  • the visor 241 has light transparency.
  • the user can view the real space through the visor 241.
  • An image such as a virtual image is projected and displayed on a display surface of the visor 241 facing the user's eyes from a laser scanner 242 (see FIG. 4) built into the main body 20a.
  • the user views the virtual image using reflected light from the display surface.
  • a visual effect as if the virtual image exists in the real space can be obtained.
  • FIG. 4 is a block diagram showing the functional configuration of the wearable terminal device 20. As shown in FIG. 4,
  • the wearable terminal device 20 includes a CPU 21, a RAM 22, a storage section 23, a display section 24, a sensor section 25, a communication section 26, etc., and these sections are connected by a bus 27. ing.
  • each part of the display unit 24 except for the visor 241 is built into the main body 20a, and is operated by power supplied from a battery also built into the main body 20a.
  • the CPU 21 is a processor that performs various calculation processes and centrally controls the operation of each part of the wearable terminal device 20.
  • the CPU 21 performs various control operations by reading and executing a program 231 stored in the storage unit 23.
  • the CPU 21 executes, for example, a visible area detection process.
  • the visible area detection process is a process that detects a user's visible area in space.
  • CPU 21 any processors such as a CPU may be provided, and the processing executed by the CPU 21 of this embodiment may be shared and executed by these two or more processors.
  • the RAM 22 provides a working memory space for the CPU 21 and stores temporary data.
  • the storage unit 23 is a non-temporary recording medium that can be read by the CPU 21 as a computer.
  • the storage unit 23 stores a program 231 executed by the CPU 21, various setting data, and the like.
  • the program 231 is stored in the storage unit 23 in the form of a computer-readable program code.
  • a nonvolatile storage device such as an SSD equipped with a flash memory is used.
  • the display unit 24 includes a visor 241, a laser scanner 242, and an optical system that guides the light output from the laser scanner 242 to the display surface of the visor 241.
  • the laser scanner 242 scans in a predetermined direction and irradiates the optical system with pulsed laser light whose on/off is controlled for each pixel in accordance with a control signal from the CPU 21 .
  • the laser light incident on the optical system forms a display screen consisting of a two-dimensional pixel matrix on the display surface of the visor 241.
  • the method of the laser scanner 242 is not particularly limited, for example, a method may be used in which a mirror is operated using MEMS (Micro Electro Mechanical Systems) to scan laser light.
  • MEMS Micro Electro Mechanical Systems
  • the laser scanner 242 has three light emitting parts that emit laser light of RGB colors, for example.
  • the display unit 24 can perform color display by projecting light from these light emitting units onto the visor 241.
  • the sensor section 25 includes an acceleration sensor 251, an angular velocity sensor 252, a depth sensor 253, a camera 254, an eye tracker 255, and the like. Note that the sensor section 25 may further include a sensor not shown in FIG. 4.
  • the acceleration sensor 251 detects acceleration and outputs the detection result to the CPU 21. From the detection result by the acceleration sensor 251, the translational movement of the wearable terminal device 20 in three orthogonal axes directions can be detected.
  • the angular velocity sensor 252 detects angular velocity and outputs the detection result to the CPU 21. From the detection result by the angular velocity sensor 252, the rotational movement of the wearable terminal device 20 can be detected.
  • the depth sensor 253 is an infrared camera that detects the distance to the subject using the ToF (Time of Flight) method, and outputs the distance detection result to the CPU 21.
  • the depth sensor 253 is provided on the front surface of the main body 20a so as to be able to photograph the visible area. By repeatedly performing measurements using the depth sensor 253 each time the user's position and orientation changes in space and synthesizing the results, it is possible to perform three-dimensional mapping of the entire space (that is, obtain a three-dimensional structure). .
  • the camera 254 photographs the space using a group of RGB image sensors, acquires color image data as a photographed result, and outputs it to the CPU 21.
  • the camera 254 is provided on the front surface of the main body 20a so as to be able to photograph the visible area.
  • the output image from the camera 254 is used to detect the position and orientation of the wearable terminal device 20, and is also transmitted from the communication unit 26 to an external device to display the visible area of the user of the wearable terminal device 20 on the external device. It is also used for
  • the eye tracker 255 detects the user's line of sight and outputs the detection result to the CPU 21.
  • the method of detecting the line of sight is not particularly limited, but for example, the point of reflection of near-infrared light in the user's eye is photographed with an eye tracking camera, and the photographing result and the image taken by the camera 254 are analyzed to detect the user's line of sight.
  • a method can be used to identify the object that the person is viewing.
  • a part of the structure of the eye tracker 255 may be provided at the periphery of the visor 241 or the like.
  • the communication unit 26 is a communication module that includes an antenna, a modulation/demodulation circuit, a signal processing circuit, and the like.
  • the communication unit 26 transmits and receives data by wireless communication to and from an external device according to a predetermined communication protocol.
  • the CPU 21 performs the following control operations.
  • the CPU 21 performs three-dimensional mapping of the space based on the distance data to the subject input from the depth sensor 253.
  • the CPU 21 repeatedly performs this three-dimensional mapping every time the user's position and orientation change, and updates the results each time. Further, the CPU 21 performs three-dimensional mapping using a continuous space as a unit. Therefore, when the user moves between a plurality of rooms partitioned by walls or the like, the CPU 21 recognizes each room as one space and performs three-dimensional mapping for each room separately.
  • the CPU 21 detects the user's visible area in the space. Specifically, the CPU 21 detects the user (wearable terminal determining the location and orientation of the device 20); Then, the visible area is detected (specified) based on the specified position and orientation and the predetermined shape of the visible area. Further, the CPU 21 continuously detects the user's position and orientation in real time, and updates the viewing area in conjunction with changes in the user's position and orientation. Note that the visual recognition area may be detected using detection results from some of the acceleration sensor 251, angular velocity sensor 252, depth sensor 253, camera 254, and eye tracker 255.
  • the first virtual image 30 (for example, the first virtual image having a cylindrical shape) is It is assumed that an image 30 (see FIG. 8) has been generated in advance.
  • This first virtual image 30 is an image that can be operated by a user wearing each wearable terminal device 20.
  • FIG. 5 is a flowchart showing the control procedure of virtual image display control processing.
  • the CPU 11 of the information processing device 10 connects each wearable terminal device 20 to the wearable terminal device 20 via the communication unit 14.
  • Information regarding the visible area of each user is acquired (step S101).
  • a user wearing one wearable terminal device (first display device) 20 of the two wearable terminal devices 20 constituting the virtual image sharing system 100 is referred to as a first user U1
  • the user wearing the other wearable terminal device 20 is referred to as a first user U1.
  • a user wearing the device (second display device) 20 is referred to as a second user U2.
  • the CPU 11 determines whether there is a user for whom the first virtual image 30 exists within the viewing area, based on the information regarding the viewing area of each user acquired in step S101 (step S102).
  • step S102 if it is determined that there is a user for whom the first virtual image 30 exists within the visible area (step S102; YES), the CPU 11 displays the first virtual image 30 on the wearable terminal device 20 (visor 241) of the user. is displayed (step S103).
  • FIG. 8 is a diagram illustrating an example of the visual recognition area and the first virtual image 30 that are viewed by the first user U1 wearing the wearable terminal device 20.
  • the first user U1 places the first virtual image 30 facing a predetermined direction at a predetermined position on the table T in the space (for example, a conference room).
  • Image 30 is visually recognized.
  • the first user U1 visually recognizes the second user U2 who is wearing the wearable terminal device 20 on the opposite side (opposite side) of the table T.
  • This space is a real space that the first user U1 visually recognizes through the visor 241. Since the first virtual image 30 is projected onto the light-transmitting visor 241, it is visually recognized as a semi-transparent image overlapping the real space.
  • the visible area visible to the first user U1 is indicated by a chain line.
  • the CPU 11 determines whether information instructing the display of the second virtual image 40 has been acquired from the wearable terminal device 20 via the communication unit 14 (step S104).
  • the second virtual image 40 is a virtual image that is a copy (including a reduced copy or an enlarged copy) of the first virtual image 30 displayed in the above space.
  • the second virtual image 40 is displayed only on the wearable terminal device 20 (visor 241) worn by the user who performed the operation to instruct the display of the second virtual image 40. That is, the second virtual image 40 displayed on the wearable terminal device 20 can be operated only by the user wearing the wearable terminal device 20.
  • the operation method of the second virtual image 40 for example, a so-called gesture operation performed by the wearable terminal device 20 detecting the movement of the user's hand, an operation using a controller (not shown) attached to the wearable terminal device, etc. (The same applies to the method of operating the first virtual image 30.)
  • step S104 if it is determined that information instructing the display of the second virtual image 40 has been acquired from the wearable terminal device 20 (step S104; YES), the CPU 11 instructs the display of the second virtual image 40.
  • the second virtual image 40 is displayed on the wearable terminal device 20 (visor 241) worn by the designated user (instruction user) (step S105). For example, if the user who gave the instruction to display the second virtual image 40 is the first user U1, the second virtual image 40 is displayed on the wearable terminal device 20 (visor 241) worn by the first user U1. The second virtual image 40 is not displayed on the wearable terminal device 20 (visor 241) worn by the second user U2.
  • the second virtual image 40 is displayed on the wearable terminal device 20 (visor 241) worn by the second user U2, and The second virtual image 40 is not displayed on the wearable terminal device 20 (visor 241) worn by one user U1.
  • a dedicated second virtual image is displayed on the wearable terminal device 20 (visor 241) worn by each user. Images 40 are displayed respectively.
  • each wearable terminal device 20 by enabling each wearable terminal device 20 to display the second virtual image 40 that is a copy of the first virtual image 30, each wearable terminal device 20 can simulate the operation of the first virtual image 30 using the second virtual image 40 displayed on the wearable terminal device 20 that the user is wearing. As a result, it is possible to prevent operations on the first virtual image 30 from being performed blindly, so it is possible to solve the problem that the first virtual image 30 becomes difficult to see due to the operation. Furthermore, since each user can freely simulate the operation of the first virtual image 30 without being seen or disturbed by other users, the 1. The usability of the virtual image 30 can be improved.
  • FIG. 9 is a diagram illustrating an example of the visible region viewed by the first user U1 wearing the wearable terminal device 20, as well as the first virtual image 30 and the second virtual image 40.
  • the first user U1 gives an instruction to display the second virtual image 40
  • the first user U1 visually recognizes the first virtual image 30 displayed on the table T
  • a second virtual image 40 facing the same direction as the first virtual image 30 is visually recognized at a predetermined position in front of the first virtual image 30 .
  • this second virtual image 40 is projected onto the light-transmissive visor 241, so it is visually recognized as a semi-transparent image that overlaps the real space.
  • FIG. 9 is a diagram illustrating an example of the visible region viewed by the first user U1 wearing the wearable terminal device 20, as well as the first virtual image 30 and the second virtual image 40.
  • the size of the first virtual image 30 is reduced by a predetermined magnification, but if the second virtual image 40 is displayed at the same magnification as the first virtual image 30, It may also be possible to display an image of the image or an image enlarged at a predetermined magnification.
  • step S104 if it is determined in step S104 that information instructing the display of the second virtual image 40 has not been acquired from the wearable terminal device 20 (step S104; NO), the CPU 11 skips step S105 and advances the process to step S106.
  • the CPU 11 determines whether information instructing to change the display mode of the first virtual image 30 has been acquired from the wearable terminal device 20 via the communication unit 14 (step S106).
  • step S106 if it is determined that information instructing to change the display mode of the first virtual image 30 has been acquired from the wearable terminal device 20 (step S106; YES), the CPU 11 controls the first virtual image based on the information.
  • the display mode of the image 30 is changed (step S107).
  • the information instructing to change the display mode of the first virtual image 30 acquired from the wearable terminal device 20 is, for example, information instructing to change the shape of the first virtual image 30 to a rectangular parallelepiped shape. If so, as shown in FIG. 10, the CPU 11 changes the shape of the first virtual image 30 to a rectangular parallelepiped shape based on the information.
  • the CPU 11 changes the shape of the first virtual image 30 to a rectangular parallelepiped shape and then to a triangular prism shape based on the information.
  • FIG. 12 is a diagram illustrating an example of a visible region viewed by the first user U1 wearing the wearable terminal device 20, as well as the first virtual image 30 and the second virtual image 40.
  • FIG. 12 for example, in a state where the shape of the first virtual image 30 is changed to a rectangular parallelepiped shape in response to an operation by the second user U2, the first user U1 While viewing the virtual image 30, a second virtual image 40 (cylindrical second virtual image 40) is also viewed at a predetermined position in front of the first virtual image 30.
  • the second user U2 who is wearing the wearable terminal device 20 also visually recognizes the first virtual image 30 that has been changed into a rectangular parallelepiped shape.
  • the CPU 11 executes the first reflection display process after executing the process of step S107 described above (step S108).
  • FIG. 6 is a flowchart showing the control procedure of the first reflection display process.
  • the CPU 11 of the information processing device 10 first receives the display mode of the first virtual image 30 from the wearable terminal device 20 via the communication unit 14. It is determined whether instruction information to be reflected on the second virtual image 40 has been acquired (step S121).
  • step S121 if it is determined that instruction information for reflecting the display mode of the first virtual image 30 on the second virtual image 40 has not been acquired from the wearable terminal device 20 (step S121; NO), the CPU 11 The process returns to the virtual image display control process (see FIG. 5), and processes from step S109 onwards are performed.
  • step S121 if it is determined in step S121 that instruction information for reflecting the display mode of the first virtual image 30 on the second virtual image 40 has been acquired from the wearable terminal device 20 (step S121; YES), the CPU 11 It is determined whether there are a plurality of change patterns of the display mode of the first virtual image 30 changed in step S107 of the virtual image display control process (see FIG. 5) (step S122).
  • step S122 if it is determined that there are multiple change patterns in the display mode of the first virtual image 30 (step S122; YES), the CPU 11 communicates with the instruction user (the change pattern of the first virtual image 30) via the communication unit 14. It is determined whether change pattern selection information has been acquired from the wearable terminal device 20 worn by the user (who gave the instruction to reflect the display mode on the second virtual image 40) (step S123).
  • a change pattern that adds a first additional image (not shown) to the first virtual image 30 in a certain display mode a change pattern that adds a first additional image (not shown) to the first virtual image 30 in a certain display mode
  • a change pattern that adds a first additional image (not shown) to the first virtual image 30 in a certain display mode a change pattern that adds a first additional image (not shown) to the first virtual image 30 in a certain display mode
  • step S123 If it is determined in step S123 that the selection information of the change pattern has not been acquired (step S123; NO), the CPU 11 repeatedly performs the determination process in step S123 until the selection information is acquired.
  • step S123 if it is determined in step S123 that the selection information of the change pattern has been acquired (step S123; YES), the CPU 11 selects the selected change pattern to be worn by the instruction user based on the acquired selection information. This is reflected in the second virtual image 40 displayed on the wearable terminal device 20 (step S124). For example, when the shape of the first virtual image 30 is changed to a rectangular parallelepiped shape (first change pattern) as described above and then changed to a triangular prism shape (second change pattern) (see FIG.
  • the CPU 11 selects the selected change pattern ( The rectangular parallelepiped shape that is the first change pattern) is reflected on the second virtual image 40 displayed on the wearable terminal device 20 of the instruction user.
  • FIG. 13 is a diagram illustrating an example of the visible region viewed by the first user U1 wearing the wearable terminal device 20, as well as the first virtual image 30 and the second virtual image 40.
  • the display mode rectangular parallelepiped shape
  • the first user U1 The user visually recognizes the first virtual image 30 in which the shape of the rectangular parallelepiped is reflected, and also visually recognizes the second virtual image 40 in which the rectangular parallelepiped shape is reflected at a predetermined position in front of the first virtual image 30.
  • the CPU 11 determines the reflection display from the wearable terminal device 20 worn by the above-mentioned instruction user. It is determined whether instruction information has been acquired (step S125).
  • step S125 if it is determined that the information instructing the determination of reflection display has been acquired (step S125; YES), the CPU 11 returns the process to the virtual image display control process (see FIG. 5), and the process from step S109 onwards. I do.
  • step S125 determines whether information instructing the determination of reflection display has not been acquired (step S125; NO). If it is determined in step S125 that information instructing the determination of reflection display has not been acquired (step S125; NO), the CPU 11 returns the process to step S123 and repeats the subsequent processes.
  • step S122 determines whether there are not multiple change patterns in the display mode of the first virtual image 30 (step S122; NO).
  • the CPU 11 instructs the changed display mode of the first virtual image 30.
  • the display mode of the first virtual image 30 is reflected in the second virtual image 40 displayed on the wearable terminal device 20 of the user (the user who gave the instruction to reflect the display mode of the first virtual image 30 on the second virtual image 40) (step S126).
  • the CPU 11 displays the changed rectangular parallelepiped shape on the wearable terminal device 20 of the instruction user. This is reflected on the second virtual image 40 (see FIG. 13). Then, the CPU 11 returns the process to the virtual image display control process (see FIG. 5) and performs the process from step S109 onwards.
  • step S109 if it is determined that information instructing to change the display mode of the second virtual image 40 has been acquired from the wearable terminal device 20 (step S109; YES), the CPU 11 controls the wearable terminal based on the information.
  • the display mode of the second virtual image 40 displayed on the device 20 is changed (step S110).
  • the information instructing to change the display mode of the second virtual image 40 acquired from the wearable terminal device 20 is, for example, information instructing to change the display color of the second virtual image 40 to red. If so, as shown in FIG. 14, the CPU 11 changes the display color of the second virtual image 40 to red (in the figure, red is represented by a diagonal line upward to the right) based on the information.
  • the above information acquired from the wearable terminal device 20 is, for example, information instructing to change the display color of the second virtual image 40 to red and then to yellow
  • the display color shown in FIG. Based on this information, the CPU 11 changes the display color of the second virtual image 40 to red, and then to yellow (in the figure, yellow is represented by a diagonal line downward to the right).
  • FIG. 16 is a diagram illustrating an example of the visible region viewed by the first user U1 wearing the wearable terminal device 20, as well as the first virtual image 30 and the second virtual image 40.
  • the first user U1 in a state where the display color of the second virtual image 40 is changed to red in accordance with the operation of the first user U1, the first user U1 can The virtual image 30 (the first virtual image 30 having a cylindrical shape) is visually recognized, and the second virtual image 40 whose color has been changed to red is also visually recognized.
  • the CPU 11 executes the second reflection display process after executing the process of step S110 described above (step S111).
  • FIG. 7 is a flowchart showing the control procedure of the second reflection display process.
  • the CPU 11 of the information processing device 10 first receives the display mode of the second virtual image 40 from the wearable terminal device 20 via the communication unit 14. It is determined whether instruction information to be reflected on the first virtual image 30 has been acquired (step S141).
  • step S141 if it is determined that instruction information for reflecting the display mode of the second virtual image 40 on the first virtual image 30 has not been acquired from the wearable terminal device 20 (step S141; NO), the CPU 11 The process returns to the virtual image display control process (see FIG. 5), and processes from step S112 onwards are performed.
  • step S141 If it is determined in step S141 that instruction information for reflecting the display mode of the second virtual image 40 on the first virtual image 30 has been acquired from the wearable terminal device 20 (step S141; YES), the CPU 11 It is determined whether there are a plurality of change patterns of the display mode of the second virtual image 40 changed in step S110 of the virtual image display control process (see FIG. 5) (step S142).
  • step S142 If it is determined in step S142 that there are a plurality of change patterns in the display mode of the second virtual image 40 (step S142; YES), the CPU 11 communicates with the instruction user (the change pattern of the second virtual image 40) via the communication unit 14. It is determined whether change pattern selection information has been acquired from the wearable terminal device 20 worn by the user who gave the instruction to reflect the display mode on the first virtual image 30 (step S143).
  • a change pattern that adds a third additional image (not shown) to the second virtual image 40 in a certain display mode a change pattern that adds a third additional image (not shown) to the second virtual image 40 in a certain display mode
  • a change pattern that adds a third additional image (not shown) to the second virtual image 40 in a certain display mode If the above change pattern includes a change pattern in which a fourth additional image (not shown) is added to the second virtual image 40 in the display mode, the second virtual image 40 in the certain display mode
  • a change pattern that adds both the third additional image and the fourth additional image may be added.
  • step S143 If it is determined in step S143 that the selection information of the change pattern has not been acquired (step S143; NO), the CPU 11 repeatedly performs the determination process in step S143 until the selection information is acquired.
  • step S143 if it is determined in step S143 that selection information of the change pattern has been acquired (step S143; YES), the CPU 11 transmits the selected change pattern to each wearable terminal device 20 based on the acquired selection information. This is reflected on the displayed first virtual image 30 (step S144). For example, when the display color of the second virtual image 40 is changed to red (first change pattern) and then changed to yellow (second change pattern) as described above (see FIG. 15), instruction When selection information for selecting red (first change pattern) is acquired as change pattern selection information from the wearable terminal device 20 worn by the user, the CPU 11 selects the selected change pattern (first change pattern). change pattern) is reflected in the first virtual image 30 displayed on each wearable terminal device 20.
  • FIG. 17 is a diagram illustrating an example of a visible region viewed by the first user U1 wearing the wearable terminal device 20, as well as the first virtual image 30 and the second virtual image 40.
  • the first user U1 when the display color (red) of the second virtual image 40 that has been changed according to the operation of the first user U1 is reflected in the first virtual image 40, the first user U1 The user visually recognizes the second virtual image 40 that has been changed to , and also visually recognizes the first virtual image 30 that reflects the red color.
  • step S144 the CPU 11 executes the determination of reflection display from the wearable terminal device 20 worn by the above-mentioned instruction user. It is determined whether instruction information has been acquired (step S145).
  • step S145 if it is determined that information instructing the determination of reflected display has been acquired (step S145; YES), the CPU 11 returns the process to the virtual image display control process (see FIG. 5), and returns the process to step S112 and subsequent steps. I do.
  • step S145 determines whether information instructing the determination of reflected display has not been acquired. If it is determined in step S145 that information instructing the determination of reflected display has not been acquired (step S145; NO), the CPU 11 returns the process to step S143 and repeats the subsequent processes.
  • step S142 if it is determined that there are not multiple change patterns in the display mode of the second virtual image 40 (step S142; NO), the CPU 11 changes the display mode of the second virtual image 40 to each of the changed display modes. This is reflected in the first virtual image 30 displayed on the wearable terminal device 20 (step S146). For example, when the display color of the second virtual image 40 is changed to red as described above (see FIG. 14), the CPU 11 transfers the changed red color to the first virtual image displayed on each wearable terminal device 20. This is reflected on the image 30 (see FIG. 17). Then, the CPU 11 returns the process to the virtual image display control process (see FIG. 5) and performs the process from step S112 onwards.
  • step S112 if it is determined that information instructing the end of the virtual image display control process has not been acquired from the wearable terminal device 20 (step S112; NO), the CPU 11 returns the process to step S101 and performs the subsequent steps. Repeat the process.
  • step S112 if it is determined that information instructing the end of the virtual image display control process has been acquired from the wearable terminal device 20 (step S112; YES), the CPU 11 ends the virtual image display control process.
  • the above embodiment is an example, and various changes are possible.
  • the first virtual image 30 displayed as if it existed in real space is shared among the users.
  • a VR type wearable terminal device is used as the wearable terminal device.
  • the action corresponding to the operation is not reflected on the avatar display. By doing so, it becomes possible to operate the second virtual image 40 without other users knowing.
  • step S107 of the virtual image display control process the first virtual image 30 is Although the display mode of the image 30 is changed, the information processing device 10 may be able to set a user who is allowed to make the change and a user who is not allowed to make the change.
  • the user who gave the instruction to display the second virtual image 40 wears the For example, on the condition that the first virtual image 30 is displayed on the wearable terminal device 20, the second virtual image 40 is displayed on the wearable terminal device 20. A virtual image 40 may also be displayed. Furthermore, even if the first virtual image 30 is not displayed on the wearable terminal device 20, a second virtual image is displayed on the wearable terminal device 20 on the condition that the first virtual image 30 is located near the display position of the first virtual image 30. The image 40 may also be displayed.
  • step S111 of the virtual image display control process the second reflection display process is executed, and an instruction is given to reflect the display mode of the second virtual image 40 on the first virtual image 30.
  • the display mode of the second virtual image 40 is reflected in the first virtual image 30, but there are two types of users: users who can display the reflected image and users who cannot display the reflected image. It may also be possible to set it in the information processing device 10.
  • the display manner of the first virtual image 30 and the second virtual image 40 described in the above embodiment is merely an example.
  • the usability of the first virtual image 30 is improved by using the first virtual image 30 and the second virtual image 40 together.
  • the first user U1 wearing the wearable terminal device performs an operation to change the display mode of the first virtual image 30
  • the display mode of the first virtual image 30 changed based on the operation is displayed on the wearable terminal device. 20 and also reflected in the display of the first virtual image 30 by other wearable terminal devices 20
  • the user U1 performs an operation to change the display mode of the first virtual image 30 the display mode of the first virtual image 30 changed based on the operation is changed to the first virtual image 30 by the wearable terminal device 20.
  • the display mode of the first virtual image 30 changed based on the operation of the first user U1 is changed only to the display of the first virtual image 30 by the wearable terminal device 20 of the first user U1.
  • information instructing the first user U1 to change the display mode of the first virtual image 30 is transmitted from the wearable terminal device 20 worn by the first user U1 to the information processing device 10. .
  • the information processing device 10 generates information regarding the display mode of the first virtual image 30 after the change based on the information, and transmits the information only to the wearable terminal device 20 worn by the first user U1.
  • One method is to do so.
  • the information processing device 10 may transmit virtual image data 132 related to the first virtual image 30 in advance to the wearable terminal device 20 worn by the first user U1. . Then, the wearable terminal device 20 that has acquired the virtual image data 132 changes the display mode of the first virtual image 30 by independently (standalone) performing display control processing in response to the operation of the first user U1.
  • the wearable terminal device 20 that has acquired the virtual image data 132 changes the display mode of the first virtual image 30 by independently (standalone) performing display control processing in response to the operation of the first user U1.
  • the present disclosure can be used in a virtual image sharing method and a virtual image sharing system.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Hardware Design (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • User Interface Of Digital Computer (AREA)
PCT/JP2022/032477 2022-08-30 2022-08-30 仮想画像共有方法および仮想画像共有システム Ceased WO2024047720A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2024543628A JP7760742B2 (ja) 2022-08-30 2022-08-30 仮想画像共有方法および仮想画像共有システム
US19/106,925 US20260037100A1 (en) 2022-08-30 2022-08-30 Virtual image sharing method and virtual image sharing system
PCT/JP2022/032477 WO2024047720A1 (ja) 2022-08-30 2022-08-30 仮想画像共有方法および仮想画像共有システム

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/032477 WO2024047720A1 (ja) 2022-08-30 2022-08-30 仮想画像共有方法および仮想画像共有システム

Publications (1)

Publication Number Publication Date
WO2024047720A1 true WO2024047720A1 (ja) 2024-03-07

Family

ID=90099205

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/032477 Ceased WO2024047720A1 (ja) 2022-08-30 2022-08-30 仮想画像共有方法および仮想画像共有システム

Country Status (3)

Country Link
US (1) US20260037100A1 (https=)
JP (1) JP7760742B2 (https=)
WO (1) WO2024047720A1 (https=)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2012168646A (ja) * 2011-02-10 2012-09-06 Sony Corp 情報処理装置、情報共有方法、プログラム及び端末装置
JP2015192436A (ja) * 2014-03-28 2015-11-02 キヤノン株式会社 送信端末、受信端末、送受信システム、およびそのプログラム
JP2016525741A (ja) * 2013-06-18 2016-08-25 マイクロソフト テクノロジー ライセンシング，エルエルシー 共有ホログラフィックオブジェクトおよびプライベートホログラフィックオブジェクト
WO2018225149A1 (ja) * 2017-06-06 2018-12-13 マクセル株式会社 複合現実表示システム及び複合現実表示端末
JP2021043476A (ja) * 2017-12-26 2021-03-18 株式会社Ｎｔｔドコモ 情報処理装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2012168646A (ja) * 2011-02-10 2012-09-06 Sony Corp 情報処理装置、情報共有方法、プログラム及び端末装置
JP2016525741A (ja) * 2013-06-18 2016-08-25 マイクロソフト テクノロジー ライセンシング，エルエルシー 共有ホログラフィックオブジェクトおよびプライベートホログラフィックオブジェクト
JP2015192436A (ja) * 2014-03-28 2015-11-02 キヤノン株式会社 送信端末、受信端末、送受信システム、およびそのプログラム
WO2018225149A1 (ja) * 2017-06-06 2018-12-13 マクセル株式会社 複合現実表示システム及び複合現実表示端末
JP2021043476A (ja) * 2017-12-26 2021-03-18 株式会社Ｎｔｔドコモ 情報処理装置

Also Published As

Publication number Publication date
JPWO2024047720A1 (https=) 2024-03-07
US20260037100A1 (en) 2026-02-05
JP7760742B2 (ja) 2025-10-27

Similar Documents

Publication Publication Date Title
US8570372B2 (en) Three-dimensional imager and projection device
JP2023507867A (ja) 人工現実コンテンツの可変焦点ディスプレイを有する人工現実システム
US9261959B1 (en) Input detection
KR20150093831A (ko) 혼합 현실 환경에 대한 직접 상호작용 시스템
CN111684336A (zh) 使用了视网膜扫描型显示部的图像显示装置及其方法
EP2926223A1 (en) Direct hologram manipulation using imu
KR102314822B1 (ko) 후방 산란된 레이저 스페클 패턴들의 광학 흐름을 추적함으로써 움직임의 6개의 자유도들을 검출하기 위한 시스템
JP2024137998A (ja) ウェアラブル端末装置、プログラム及び表示方法
JP7499945B2 (ja) ウェアラブル端末装置、プログラムおよび表示方法
US20240177436A1 (en) Wearable terminal device, program, and notification method
JP7802561B2 (ja) 電子機器、電子機器の制御方法、プログラム、および記録媒体
EP3867734B1 (en) Mobile platform as a physical interface for interaction
JP7633452B2 (ja) ウェアラブル端末装置、プログラムおよび表示方法
WO2024047720A1 (ja) 仮想画像共有方法および仮想画像共有システム
JP7660734B2 (ja) ウェアラブル端末装置、プログラムおよび表示方法
US12288305B2 (en) Wearable terminal apparatus to change display position of partial image
JP2023164452A (ja) 表示装置、表示システム、表示調整方法及び表示調整プログラム
JP7817407B2 (ja) 仮想空間画像生成方法および仮想空間画像生成システム
US12340462B2 (en) Wearable terminal device, program, and display method
WO2023275919A1 (ja) ウェアラブル端末装置、プログラムおよび表示方法
US20260095647A1 (en) Information processing apparatus, information processing system, method for controlling information processing apparatus, and storage medium
JP2026054720A (ja) 情報処理装置、情報処理装置の制御方法、およびプログラム
JP2023115505A (ja) 表示装置、表示方法、及びプログラム
CN121879556A (zh) 具有头戴式显示器的运算系统
CN111556310A (zh) 显示系统、记录介质以及信息处理装置的控制方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22957324

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2024543628

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22957324

Country of ref document: EP

Kind code of ref document: A1