WO2024095358A1 - 画像表示システム、ディスプレイ装置 - Google Patents

画像表示システム、ディスプレイ装置 Download PDF

Info

Publication number
WO2024095358A1
WO2024095358A1 PCT/JP2022/040854 JP2022040854W WO2024095358A1 WO 2024095358 A1 WO2024095358 A1 WO 2024095358A1 JP 2022040854 W JP2022040854 W JP 2022040854W WO 2024095358 A1 WO2024095358 A1 WO 2024095358A1
Authority
WO
WIPO (PCT)
Prior art keywords
display device
image
user
display
viewpoint position
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/040854
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
塁 佐藤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Virtualwindow Co Ltd
Original Assignee
Virtualwindow Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Virtualwindow Co Ltd filed Critical Virtualwindow Co Ltd
Priority to EP22964380.4A priority Critical patent/EP4614491A4/en
Priority to JP2024523997A priority patent/JP7527081B1/ja
Priority to PCT/JP2022/040854 priority patent/WO2024095358A1/ja
Priority to JP2024523998A priority patent/JP7511307B1/ja
Priority to EP23885798.1A priority patent/EP4614485A4/en
Priority to PCT/JP2023/039394 priority patent/WO2024096044A1/ja
Priority to US19/126,305 priority patent/US20260179513A1/en
Publication of WO2024095358A1 publication Critical patent/WO2024095358A1/ja
Priority to JP2024098185A priority patent/JP2024133508A/ja
Priority to JP2024113105A priority patent/JP2024138501A/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
    • G09G3/006—Electronic inspection or testing of displays and display drivers, e.g. of LED or LCD displays
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/128—Adjusting depth or disparity
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B27/0172—Head mounted characterised by optical features
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B30/00—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images
    • G02B30/20—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images by providing first and second parallax images to an observer's left and right eyes
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B30/00—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images
    • G02B30/20—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images by providing first and second parallax images to an observer's left and right eyes
    • G02B30/26—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images by providing first and second parallax images to an observer's left and right eyes of the autostereoscopic type
    • G02B30/27—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images by providing first and second parallax images to an observer's left and right eyes of the autostereoscopic type involving lenticular arrays
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/147—Digital output to display device ; Cooperation and interconnection of the display device with other functional units using display panels
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
    • G09G3/04—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of a single character by selection from a plurality of characters, or by composing the character by combination of individual elements, e.g. segments using a combination of such display devices for composing words, rows or the like, in a frame with fixed character positions
    • G09G3/16—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of a single character by selection from a plurality of characters, or by composing the character by combination of individual elements, e.g. segments using a combination of such display devices for composing words, rows or the like, in a frame with fixed character positions by control of light from an independent source
    • G09G3/18—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of a single character by selection from a plurality of characters, or by composing the character by combination of individual elements, e.g. segments using a combination of such display devices for composing words, rows or the like, in a frame with fixed character positions by control of light from an independent source using liquid crystals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/302—Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays
    • H04N13/305—Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays using lenticular lenses, e.g. arrangements of cylindrical lenses
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/341—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] using temporal multiplexing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0132—Head-up displays characterised by optical features comprising binocular systems
    • G02B2027/0134—Head-up displays characterised by optical features comprising binocular systems of stereoscopic type
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0179—Display position adjusting means not related to the information to be displayed
    • G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2320/00—Control of display operating conditions
    • G09G2320/02—Improving the quality of display appearance
    • G09G2320/0261—Improving the quality of display appearance in the context of movement of objects on the screen or movement of the observer relative to the screen
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2320/00—Control of display operating conditions
    • G09G2320/02—Improving the quality of display appearance
    • G09G2320/028—Improving the quality of display appearance by changing the viewing angle properties, e.g. widening the viewing angle, adapting the viewing angle to the view direction
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2320/00—Control of display operating conditions
    • G09G2320/06—Adjustment of display parameters
    • G09G2320/068—Adjustment of display parameters for control of viewing angle adjustment

Definitions

  • This disclosure relates to a technology for displaying virtual objects superimposed on real-space objects that are visible through a see-through display device.
  • MR mixed reality
  • Hololens registered trademark
  • HMD head-mounted display
  • Microsoft Corporation Microsoft Corporation
  • HMD head-mounted display
  • the user can view real space through the transparent eyepiece of the Hololens (HMD) worn on the head, while simultaneously superimposing a virtual CG object displayed in the eyepiece on the real space.
  • Hololens is also equipped with a distance sensor, and by acquiring the position and shape of real-space objects in the external world of the HMD while projecting a virtual CG object, it is possible to accurately superimpose real-space objects and virtual CG objects.
  • One objective of the present disclosure is to provide a methodology that enables a virtual object to be superimposed on an object in real space that is visible to a user through a see-through display device, under conditions where the relative position between the user's viewpoint and the see-through display device is not fixed.
  • An image display system includes a transparent display device capable of displaying an image, a storage unit that stores position information and three-dimensional shape information in a virtual space constructed on a predetermined reference coordinate system that is set in such a way that it is superimposed on the coordinate system in real space for each of the display device, objects in real space present around the display device, and any virtual object, and a processing unit that generates an image and displays it on the display device by performing a projective transformation or perspective projection transformation based on the viewpoint position, the position information and three-dimensional shape of the display device, and three-dimensional image data representing the virtual object, so that the virtual object is accurately superimposed on the real space seen through the display device when a user looks at the display device from a certain viewpoint position as it appears when the virtual space is viewed from the viewpoint position.
  • FIG. 1 is a schematic configuration diagram of an image display system according to a first embodiment of the present disclosure.
  • FIG. 1 is a schematic configuration diagram showing an example in which the image display system of the present embodiment is mounted on a vehicle.
  • 1A and 1B are diagrams for explaining how the image display system of the present embodiment displays an image according to a user's viewpoint position.
  • FIG. 11 is a conceptual diagram for explaining calculation of a viewpoint position.
  • 1 is a block diagram showing a configuration of an image generating apparatus according to a first embodiment; 4 is a flowchart showing an overall process performed by the image generating device of the first embodiment.
  • FIG. 11 is a diagram showing an example of a display device in a modified example of the system according to the first embodiment, and an example of a display image on the display device.
  • FIG. 11 is a diagram showing a display device and the like in a system according to a second embodiment.
  • FIG. 13 is a diagram showing an example of a display device in a modified example of the system according to the second embodiment, and an example of a display image on the display device.
  • FIG. 13 is a schematic diagram for explaining the configuration of a display device in a system according to a third embodiment. 13 is a diagram illustrating a more detailed configuration of a display device according to a third embodiment and display operation control by the display device.
  • FIG. 13 is a diagram illustrating a more detailed configuration of a display device according to a third embodiment and display operation control by the display device.
  • FIG. 7 is a flowchart showing a detailed example of the display process in step S104 of FIG. 6 described in the first embodiment.
  • FIG. 13 is a diagram showing a display device in a modified example of the system according to the third embodiment.
  • the image display system (hereinafter, sometimes simply referred to as the "system") 1 of this embodiment mainly includes an image generating device 10 that generates a virtual object image to be presented to a user, a measuring device 20 that measures the viewpoint position of a user 2, a transparent display device 30 that displays an image generated by the image generating device 10, and a real space information acquiring device 40 (hereinafter, sometimes simply referred to as the "acquisition device 40") that acquires environmental information of the real space (real world) that the user sees through the transparent display device 30 (information on the position, shape, size, distance to an object, etc. of an object existing in the real world).
  • the image generating device 10 is a computer that executes a software program with a processor, and also serves as a calculation unit (not shown) of the measuring device 20.
  • the transparent display device 30 in this embodiment is a so-called "transmissive" display device, and can be configured, for example, by a transparent organic EL (Electro Luminescence) display (transmissive OLED), a transparent liquid crystal display (transmissive LCD), or a transparent screen onto which an image is projected from an external projector.
  • the transparent display device 30 of this embodiment configured with such a transparent display, is transparent or semi-transparent, and allows the user to see the real space environment on the opposite side through the transparent display device 30. By displaying an image on the transparent display device 30, it is possible to provide a visual experience in which a virtual object represented by the displayed image exists in the real space seen through the transparent display device 30.
  • the transparent display device 30 of this embodiment can be used as a device that is fixedly installed at any position, such as a window of a building, a window of a vehicle such as a car, bus, ship, or airplane, or on an indoor table.
  • the virtual object can be superimposed on the outdoor or indoor real space scenery or landscape seen through the transparent display device 30.
  • the measurement device 20 (sensor 22) is provided below or above the transmissive display device 30 located in front of the user 2, and measures the user's viewpoint position from in front of the user 2.
  • the measuring device 20 measures the viewpoint position 2A of the user 2 in a specified reference coordinate system while the user 2 is within a range where the user 2 can see the transmissive display device 30.
  • the viewpoint position 2A is a position that corresponds to the position of the eyes.
  • the specific viewpoint position 2A used in the processing is not particularly limited, but for example, the midpoint between the eyes of the user 2, the center point of the head, or a position a specified distance inward from the center of the eyes can be used as the viewpoint position 2A.
  • a use case in which the transmissive display device 30 is mainly installed in the window of a vehicle such as an automobile will be described using a world coordinate system that is fixed and defined in real space, and a local coordinate system that is defined with the vehicle as the reference.
  • Both the world coordinate system and the local coordinate system are assumed to be Cartesian coordinate systems having three axes, xyz.
  • FIG. 2 is a schematic diagram showing an example in which the image display system of this embodiment is mounted on a vehicle.
  • an image display system 1 including an image generating device 10, a measuring device 20, a transmissive display device 30, and a real space information acquiring device 40 is mounted on a vehicle M, and provides a visual experience in which a real landscape and a virtual image are fused to a driver, passenger, or other occupant (user) 2 of the vehicle M.
  • the vehicle M of this embodiment is placed in a real space (real world) in which various real objects (hereinafter sometimes referred to as "real objects") RO exist, and is, for example, a small passenger vehicle for one person that runs within the real space.
  • the vehicle M is not limited to a passenger vehicle, and may be a passenger car, a bus, a railroad car, an airplane, a ship, a vehicle of an attraction facility, or the like.
  • an orthogonal coordinate system having an X-axis pointing to the right of the user 2, a Y-axis pointing upward, and a Z-axis pointing backward is shown as an example of a local coordinate system based on the position and direction of the vehicle.
  • the positions and orientations of the sensor 22 (not shown in FIG. 2) and the transmissive display device 30 in the measurement device 20 are fixed in this local coordinate system.
  • the orientation is represented by Pitch around the X-axis, Yaw around the Y-axis, and Roll around the Z-axis.
  • the image generating device 10 displays a virtual object VO in a virtual space on the transparent display device 30 based on the position and direction of the vehicle, the viewpoint position 2A of the user 2 in the vehicle, and the fixed position and orientation of the transparent display device 30.
  • the virtual object VO is placed in a three-dimensional virtual space defined by a world coordinate system, and the world coordinate system of the real space and the world coordinate system of the virtual space are defined by being superimposed so that their positional relationship and enlargement/reduction ratio match.
  • the user peers into the real space through the transparent display device 30 from the viewpoint position 2A, but at the same time, he or she also peers into the virtual space fixed to the same world coordinate system.
  • a calculation process is performed to generate an image to be displayed on the display screen.
  • three-dimensional CG data in a virtual space defined three-dimensionally based on the coordinates of viewpoint position 2A is transformed into a two-dimensional display screen using projective transformation, perspective projection transformation, or similar calculation processing to generate a two-dimensional image, which is then displayed on transmissive display device 30.
  • the three-dimensional information in the virtual space is converted into two-dimensional information with geometrical precision and presented to user 2.
  • the image generating device 10 stores in advance in an internal memory unit 12 (see FIG. 5) three-dimensional CG data representing a virtual object to be superimposed on the real space seen through the transparent display device 30, such as data of a cheetah drawn in three-dimensional CG, together with placement information in the virtual space, and generates a display image through computational processing to appropriately display the three-dimensional CG on the transparent display device 30 based on the viewpoint position 2A so that the image appears realistic and natural with geometrical precision to give the user 2 the illusion that the virtual object exists in real space.
  • an internal memory unit 12 see FIG. 5
  • image is used to include not only one or more still images, but also moving images (video) made up of multiple images that are consecutive in time series.
  • an image in which a virtual object is superimposed on a real-space view seen from a vehicle window is generated according to the viewpoint position 2A of the user 2 and displayed on the transmissive display device 30, so that the user 2 can feel as if the virtual object is placed together with a real-world object in real space within the real world and exists simultaneously.
  • this embodiment does not use a head-mounted display (HMD) that moves with the user's head, but instead uses a method of displaying an image on a screen fixed to a position other than the user's body part, including the user's head.
  • HMD head-mounted display
  • the display output unit 16 in the image generating device 10 can be used as a display device in mixed reality (MR) technology, and can allow the user to experience a realistic mixed reality in the same way as a head-mounted display, without causing the user 2 any inconvenience or discomfort due to the use of a head-mounted display.
  • MR mixed reality
  • the image display system 1 of this embodiment will be described in more detail below.
  • the measuring device 20 continuously measures the viewpoint position 2A, and the image generating device 10 generates a display image to follow the viewpoint position 2A and displays it on the screen of the transparent display device 30.
  • the display image displayed on the transparent display device 30 changes to follow the viewpoint position 2A that moves with it, so that the user 2 can be given the sensation that a virtual object exists in real space.
  • Image generation associated with following the viewpoint position 2A is realized by continuously generating images that look like the virtual object in the virtual space when viewed through the display device from the viewpoint position 2A at that time. The method of generating images at this time is as described above.
  • an image may be generated by predicting the viewpoint position 2A a little ahead.
  • the viewpoint position 2A moves, if an image is generated based on the current position of 2A and the result is displayed, a delay in image display will occur for the time required for calculation, which makes the user feel uncomfortable.
  • a method of predicting the position a little ahead while taking time series data of the viewpoint position 2A and generating an image in advance is also possible.
  • prediction is not limited to the viewpoint position; predictive calculations can also be used for the relative position of the local coordinate system with respect to the world coordinate system. For example, if we consider a car as the vehicle, the position of the vehicle some distance ahead can be predicted with high accuracy based on the direction of travel, speed information, and map information. This prediction result can also be reflected in the image generation described above.
  • FIG. 3 is a diagram for explaining how the image display system according to the present embodiment displays an image according to the user's viewpoint position.
  • the user space on the front side of the transparent display device 30 is defined as real space
  • the space on the back side is defined as virtual space, based on the user's viewpoint position.
  • the virtual space defined on the back side of the transparent display device 30 is displayed on the transparent display device 30 as an image seen from the position P1 where the head of the user 2 is located through a pseudo window (hereinafter also referred to as a "pseudo window") by the transparent display device 30.
  • the virtual object in the virtual space is defined by three-dimensional CG data to be described later. In the example of FIG. 3, six trees are arranged side by side as virtual objects.
  • the virtual space is defined on the back side of the transparent display device 30 in FIG. 3, but the virtual space may be defined on the front side of the transparent display device 30, or the space including the front side and the back side of the transparent display device 30 may be defined as the virtual space.
  • This makes it possible to display on the transparent display device 30 not only images in which a virtual object in a virtual space appears to be on the other side of a virtual window created by the transparent display device 30, but also images in which a virtual object in a virtual space appears to be protruding in front of the transparent display device 30.
  • the field of view FoV1 of the virtual space seen through the pseudo window created by the transparent display device 30 is wide, and all six trees are displayed on the transparent display device 30 as if they are included in the field of view FoV1 (display D1).
  • the field of view FoV2 of the virtual space seen through the pseudo window created by the transparent display device 30 narrows, and only the entire three trees and parts of the trees on either side of them are displayed in the field of view FoV2 (display D2).
  • the field of view FoV3 of the virtual space seen through the pseudo window created by the transparent display device 30 changes in the x direction. Only the three trees on the right side are included in the field of view FoV3.
  • the screen is viewed from an oblique direction rather than from the front of the transmissive display device 30, but the horizontal thickness of the tree seen through the pseudo window needs to be the same as when viewed from the front (display D3'). For this reason, when the tree is displayed on the transmissive display device 30, an image that has been appropriately stretched so that it looks like display D3' to the user is displayed (display D3).
  • a process for example, a projective transformation, a perspective projection transformation, or a calculation similar thereto
  • a process for example, a projective transformation, a perspective projection transformation, or a calculation similar thereto
  • each point of the three-dimensional CG data may be projected onto a point where a straight line connecting each point and the user's viewpoint position intersects with the transmissive display device 30 in the reference coordinate space.
  • Another processing method for generating an image to be displayed on the transmissive display device 30 may involve performing arithmetic operations on specific matrices or values based on empirical rules on the image or the three-dimensional parameters of the image.
  • the measuring device 20 of this embodiment is configured to have an imaging unit that images the user 2, and a calculation unit that determines the viewpoint position 2A based on the imaging information of the user 2 captured by the imaging unit.
  • the imaging unit is realized by a sensor 22 installed in the measuring device 20 shown in FIG. 1.
  • the calculation unit is realized by the processing unit 14 (see FIG. 5) of the image generating device 10 shown in FIG. 1.
  • the calculation unit may be realized by a processing unit (not shown) within the measuring device 20.
  • the sensor 22 is a depth sensor that measures the depth from the sensor 22 to an object (here, the body of user 2) at each pixel.
  • the calculation unit realized by the processing unit 14 of the image generating device 10 estimates the shape of the human body based on the depth of each pixel measured by the sensor 22, and calculates the viewpoint position 2A based on the position of the head on the body. In this way, because the human body shape is estimated from the depth of each pixel and the head position on the body shape is used, the viewpoint position 2A can be identified with high accuracy even if the position and orientation of the user 2's body changes in various ways.
  • FIG. 4 is a conceptual diagram for explaining the calculation of the viewpoint position.
  • the position ( Xs , Ys , Zs) and the attitude (Pitch s , Yaw s , Roll s ) of the sensor 22 in the reference coordinate system are set in advance.
  • the coordinates (Xh, Yh, Zh ) of the viewpoint position 2A can be calculated from the depth of each pixel acquired by the sensor 22 and the position and attitude of the sensor 22. As shown in FIG.
  • each transmissive display device 30 is a rectangle or a trapezoid, and the shape is expressed by a height (Height m ) and a width (Width m ).
  • the transmissive display device 30 may be of any polygonal shape, or may have a curved or spherical surface.
  • the image captured by the sensor 22 may be a depth image, or a depth image and a visible image.
  • the imaging unit may include a sensor that captures a depth image and a camera that captures a visible image, and the calculation unit may calculate the viewpoint position 2A using both the depth image and the visible image. It may also be two visible images captured at different positions. In this case, the viewpoint position 2A may be calculated using the parallax between the two visible images.
  • the viewpoint position 2A may also be estimated from a single visible image by image processing using AI.
  • the measuring device 20 (sensor 22) is installed above or below the transparent display device 30 in front of the user 2, but this is not limited to the above.
  • the measuring device 20 can also be placed behind the user 2 in a position that does not overlap with the transparent display device 30.
  • it can be installed behind the transparent display device 30 (the back side, rear side) and pass through the display portion of the transparent display device 30 to measure the user's viewpoint position.
  • it is possible to use multiple sensors and use an image captured by any of them to perform the measurement, or to use a viewpoint position obtained by integrating information from multiple sensors.
  • the real space information acquisition device 40 in this embodiment is a device that acquires environmental information (information on the position, shape, size, distance to an object, etc., of an object existing in the real world) as seen by the user through the transmissive display device 30.
  • the acquisition device 40 is equipped with a LiDAR sensor, as an example, and is capable of measuring the distance to an object and the shape of the object existing in the real space around the transmissive display device 30 as seen by the user.
  • LiDAR Light Detection and Ranging, or Laser Imaging Detection and Ranging
  • the acquisition device 40 uses such a LiDAR sensor to scan the real space around the transmissive display device 30 and acquires information about the distance to objects present in that space, their shapes, etc.
  • the acquisition device 40 may be equipped with an imaging camera equipped with an imaging element that generates images (visible images and depth images) in addition to or instead of the LiDAR sensor described above.
  • the real space visible through the transmissive display device 30 is captured by the imaging camera, and the captured image is analyzed using any image processing technology to obtain information regarding the position, size, shape, etc. of objects contained in the image.
  • Objects contained in the image include, for example, buildings such as buildings that exist in the real space, structures such as bridges, distant mountains, moving vehicles and living things, etc.
  • the acquisition device 40 may further include a GPS (Global Positioning System) sensor.
  • GPS Global Positioning System
  • the acquisition device 40 may further include a GPS (Global Positioning System) sensor.
  • GPS Global Positioning System
  • This position information is also, for example, position information in the world coordinate space of a vehicle or the like on which the system 1 including the acquisition device 40 is mounted.
  • the acquisition device 40 may also be configured to identify the position and direction of the vehicle using a Visual Positioning System (VPS).
  • VPS Visual Positioning System
  • the acquisition device 40 captures an image of the real space outside the vehicle, and the processing unit 14 (see FIG. 5) of the image generation device 10 identifies the position and direction of the vehicle based on the image.
  • many images whose captured positions and directions are identified are acquired in advance, and visual features such as the contours of objects such as buildings that appear in the images are extracted, and the visual features are stored in a database in the storage unit 12 (see FIG. 5) of the image generation device 10 together with information on the position and direction as a searchable index.
  • the processing unit 14 of the image generation device 10 extracts visual features from the acquired image and identifies the position and direction at which the image was captured by comparing the extracted visual features with the visual features stored in the database.
  • the identified position and direction at which the image was captured are converted into the position and direction of the vehicle. This allows the vehicle's position and orientation to be obtained with high accuracy in real time.
  • a time delay will occur in acquiring the information in accordance with the actual movement of the vehicle.
  • the speed and acceleration vectors are calculated using the current and past positions and orientations of the vehicle, and in addition, vehicle body information (accelerator information, brake information, speed and steering angle information, etc.) obtained directly from the vehicle, which can be obtained without delay, is added to these to predict the vehicle's future position and orientation, thereby solving the problem of time delay.
  • vehicle body information acceleration information, brake information, speed and steering angle information, etc.
  • visual features are stored as searchable indexes in a database in the storage unit 12 (see FIG. 5) of the image generating device 10 together with position and direction information
  • the visual features may also be stored in a database in an external device (not shown).
  • the image data acquired by the acquisition device 40 is transmitted from the acquisition device 40 to the external device.
  • the external device extracts visual features from the image and compares the extracted visual features with the visual features stored in the database to identify the position and direction at which the image was captured. Information regarding the identified position and direction for the image is transmitted from the external device to the acquisition device 40.
  • FIG. 5 is a block diagram showing the configuration of the image generating device in this embodiment.
  • the image generating device 10 has a memory unit 12, a processing unit 14, a display output unit 16, and a communication unit 18.
  • the image generating device 10 is depicted as a single element in FIG. 5, the image generating device 10 does not necessarily have to be a single physical element, and may be composed of multiple physically separated elements.
  • the storage unit 12 includes temporary or non-temporary storage media such as ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and SDD (Solid State Drive).
  • the storage unit 12 stores computer programs executed by the processing unit 14 and various data described below.
  • the computer programs stored in the non-temporary storage media of the storage unit 12 include instructions for carrying out each process of the image generation method by the processing unit 14, which will be described later with reference to FIG. 6 etc.
  • the storage unit 12 holds screen layout data indicating the position, attitude, and shape of the transparent display device 30 in a predetermined reference coordinate space, measuring device/acquisition device layout data indicating the positions and attitudes of the measuring device 20 and acquisition device 40 in the reference coordinate space, and three-dimensional CG data indicating a virtual object to be superimposed and displayed in real space seen through the transparent display device 30.
  • the reference coordinate space is a space represented by a coordinate system (reference coordinate system) having a predetermined origin O that serves as the basis for calculations in this embodiment, and is a Cartesian coordinate system space having three axes, x, y, and z. How to set the reference coordinate space and its origin O is arbitrary.
  • the reference coordinate space may be fixed to the transparent display device 30, may be fixed to the measuring device 20 and/or acquisition device 40, may be fixed to the transparent display device 30, the measuring device 20, and the acquisition device 40, or may not be fixed to any of them.
  • the reference coordinate space may be processed as a coordinate within the vehicle (local coordinate system) separately from the coordinates of the outside world (world coordinate system).
  • Three-dimensional CG is also called 3DCG because it is a virtual solid object in three-dimensional space that differs from two-dimensional images in that it contains depth (solid) information.
  • 3DCG depth (solid) information.
  • One example of three-dimensional CG is a virtual solid object (modeling) composed of multiple faces created with vertices at points placed on the three-dimensional coordinates of the virtual space, and it can be expressed by giving each face information that reproduces the material, etc., and illuminating the object from an arbitrary light intensity, light source position, etc.
  • Three-dimensional CG data includes information regarding the placement position in the virtual space of the virtual object displayed in the three-dimensional CG.
  • the memory unit 12 can store three-dimensional CG data for a variety of multiple virtual objects.
  • the processing unit 14 is composed of, for example, one or more CPUs (Central Processing Units).
  • the processing unit 14 may also include one or more GPUs (Graphics Processing Units).
  • the processing unit 14 executes a computer program stored in the storage unit 12, thereby carrying out each process of the image generation method described below with reference to FIG. 6 etc.
  • the display output unit 16 is an output unit to which the transparent display device 30 is connected and which outputs an image signal for displaying the image generated by the processing unit 14 on the transparent display device 30.
  • the display output unit 16 is configured with output terminals such as VGA, DVI, DisplayPort (trademark), HDMI (registered trademark), and USB (trademark) Type-C, and can be connected to the transparent display device 30 by wired connection.
  • the display output unit 16 may be configured to be wirelessly connected to the transparent display device 30 using any wireless communication technology.
  • the communication unit 18 has a function of transmitting and receiving data between the image generating device 10 and a device external to the image generating device 10.
  • the communication unit 18 may transmit and receive data of an image captured by the sensor 22 of the measuring device 20 and data of information acquired by the acquisition device 40.
  • the communication unit 18 may transmit image data acquired by the acquisition device 40 to the external device, or receive information from the external device regarding the position and direction at which the image was captured, which is specified in the external device.
  • the communication unit 18 may acquire only the necessary part of data of a three-dimensional real space model that models an object existing in real space, via a network. Note that the communication between the external device and the communication unit 18 may be either wired communication or wireless communication.
  • FIG. 6 is an example of a flowchart showing the overall processing by the image generating device of this embodiment.
  • Each process in FIG. 6 is executed by the processing unit 14.
  • the image generated by the image generating device 10 is a moving image made up of multiple frame images that are successive in time series
  • calculations may be made for each frame of the moving image.
  • predictive calculations may be made using past data in the time series to generate an image.
  • the processing unit 14 of the image generating device 10 first acquires information about the real space that the user 2 sees through the transmissive display device 30 (step S101).
  • the processing unit 14 controls the operation of the acquisition device 40 and causes the acquisition device 40 to acquire information about the real space behind the user 2 of the transmissive display device 30.
  • the processing unit 14 first acquires GPS position information from the acquisition device 40 to identify the approximate position of the vehicle equipped with the system 1 including the acquisition device 40, and further identifies VPS information (more accurate position and direction of the vehicle) from the acquisition device 40.
  • the processing unit 14 acquires data of a three-dimensional real space model in which an object existing in the real space is three-dimensionally modeled.
  • Objects existing in the real space are, for example, buildings, houses, bridges, railways, roads, and other structures in the real world.
  • the data of the three-dimensional real space model in which such an object is three-dimensionally modeled includes information on the position of each object in the real world and information on the shape and size of each object as information for reproducing the object in the virtual space.
  • 3D city model data in which real spaces are three-dimensionally modeled for some cities and natural shapes are provided by multiple businesses and institutions.
  • Well-known services that provide 3D city model data include, for example, "Project PLATEAU” by the Ministry of Land, Infrastructure, Transport and Tourism and “3D Map Data” by Zenrin Co., Ltd.
  • 3D model data for more cities and nature will be provided by various businesses and organizations.
  • three-dimensional real space model data can be obtained by using a service provided by a business or organization.
  • the communication unit 18 of the image generating device 10 connects to a service providing server of a business or organization via the Internet or the like, and downloads at least a necessary portion of the three-dimensional real space model data from the server, thereby obtaining the three-dimensional real space model data.
  • the data can be used temporarily or permanently without connecting to the Internet.
  • the obtained three-dimensional real space model data is stored in the storage unit 12.
  • the processing unit 14 may acquire information on the real space behind the user 2 of the transmissive display device 30 using a sensor (e.g., a LiDAR sensor) provided in the acquisition device 40, and acquire information on the distance from the acquisition device 40 to the object and the size and shape of the object for the object present in the real space. If there are multiple objects, information on the distance, size, shape, etc. of each object is acquired. If the acquisition device 40 is equipped with an imaging camera or a depth sensor, an image of the real space behind the user 2 of the transmissive display device 30 may be captured by the imaging camera or acquired as a depth image. The processing unit 14 can acquire or estimate information on the position, shape, etc.
  • a sensor e.g., a LiDAR sensor
  • objects included in the image e.g., buildings such as buildings present in the real space, structures such as bridges, traffic lights, and telephone poles, distant mountains, etc.
  • objects included in the image e.g., buildings such as buildings present in the real space, structures such as bridges, traffic lights, and telephone poles, distant mountains, etc.
  • the processing unit 14 then acquires the viewpoint position 2A of the user 2 (step S102).
  • the processing unit 14 uses the measurement device 20 to calculate the viewpoint position 2A of the user 2 by the method described with reference to FIG. 4 etc.
  • step S101 and the processing of step S102 by the processing unit 14 may be executed in a different order, or may be executed in parallel. Furthermore, if the processing of the subsequent steps is performed based on data acquired or estimated in the past in a time series, this part may be skipped since the necessary data has already been acquired without performing both processes.
  • the processing unit 14 then performs a process of generating a display image on the transparent display device 30, which displays the virtual object superimposed on the real space around the transparent display device 30 as seen by the user 2 through the transparent display device 30 (step S103).
  • the processing unit 14 continuously identifies the position and direction of the vehicle in the world coordinate system based on the real space environment information from the acquisition device 40, and also continuously identifies the viewpoint position 2A.
  • the processing unit 14 performs the above-mentioned calculation process based on the viewpoint position 2A of the user 2 on the screen of the transparent display device 30 to display an image displaying a virtual object so as to follow the position and direction of the vehicle and the viewpoint position 2A, thereby generating a display image with strict geometric accuracy that appears as if the virtual object exists together in the real world when viewed through the screen of the transparent display device 30 from the viewpoint position 2A of the user 2.
  • the display image When the display image is displayed on the transparent display device 30, it is an image that appears as if a virtual object in the virtual space seen through the transparent display device 30 from the viewpoint position 2A is viewed.
  • the display image displayed on the transparent display device 30 changes in accordance with the movement of the vehicle and the viewpoint position 2A that moves with the movement of the head of the user 2 inside the vehicle. This allows the user 2 to feel as if a virtual object exists in the real space.
  • Tracking to viewpoint position 2A is achieved by continuously generating images of how the drawn content appears when viewed from viewpoint position 2A at that time.
  • the display image of the virtual object is projected onto the screen of the transparent display device 30 with strict geometric accuracy, taking into account the user's viewpoint position 2A. Therefore, even when the user 2 looks at the screen of the transparent display device 30 from an oblique angle, when the user 2 views the virtual object from the viewpoint position 2A, a display image is generated and presented on the transparent display device 30 as a realistic and natural image that looks as if it actually exists at that position.
  • the processing performed here is not limited to projective transformation. It is also possible to use a perspective projection transformation method, a calculation method similar to that, or arithmetic processing of specific matrices and numerical values according to empirical rules.
  • the virtual object may be displayed on the transmissive display device 30 so as to be positioned at a position where it appears to overlap in part or in whole with a real-space object that exists in the real space surrounding the transmissive display device 30 when viewed from a certain viewpoint position 2A.
  • the virtual object is reflected in the display after a process of partially or completely blocking the virtual object is performed based on three-dimensional real-space model data corresponding to the object in real space, or three-dimensional data of the object constructed based on the distance to the object and the shape acquired in real time by the real-space information acquisition device 40, thereby accurately expressing the overlap between the two.
  • the process of occluding a part or all of a virtual object includes a process of making transparent (or deleting, scraping, or making invisible) an area of the virtual object that is occluded by a real-space object when a part or all of a real-space object is present in front of the virtual object when the virtual object is viewed from a certain viewpoint position 2A. More specifically, when the distance from the viewpoint position 2A to a part that constitutes the virtual object is greater than the distance from the viewpoint position 2A to the part that constitutes the real-space object, a display image of the virtual object is generated in which a part corresponding to the occluded area that is occluded by the part that constitutes the real-space object is occluded.
  • the processing unit 14 transmits the signal of the display image thus generated to the transparent display device 30 via the display output unit 16 of the image generating device 10, causing the transparent display device 30 to display the display image (step S104).
  • display images of virtual objects displayed in three-dimensional CG and superimposed on the real space seen through the transparent display device 30 are continuously generated following changes in the viewpoint position 2A of the user 2, and a composite realistic moving image is provided to the user 2 via the transparent display device 30.
  • the user 2 is given the illusion that the virtual objects exist together in the real space of the real world seen through the transparent display device 30.
  • the position information not only the position information but also information regarding the size, shape, etc., of objects in real space behind the transmissive display device 30 as seen by the user is obtained, so that it is possible to simulate the front-to-back relationship and relative positional relationship when a virtual object to be superimposed on an object in real space is virtually placed in the same space, and it is possible to generate an image that reflects the representation including the expression due to occlusion processing in which the virtual object is partially visible or hidden when the user's viewpoint position is used as the starting point.
  • the virtual object can be superimposed on the object in real space with higher quality, it is possible to provide the user 2 with a more realistic and natural image that gives the user 2 the illusion that the virtual object exists in real space.
  • uses of the system 1 include a use case in which the transparent display device 30 is used as a window of a building and a virtual object is superimposed on the outside scenery seen through the window, and a use case in which the transparent display device 30 is installed on an indoor table or the like and a virtual object is superimposed on a real space object (such as a shelf inside the room) seen through the transparent display device 30.
  • the system 1 including the transparent display device 30 is fixed and does not move in the world coordinate space. Therefore, the local coordinate system of the system 1 is fixed in the world coordinate system. Therefore, in these use cases, the GPS sensor and VPS-related functions for identifying the position of the system 1 in the local coordinate system may be omitted from the acquisition device 40 of this embodiment described above.
  • the acquisition device 40 may be omitted if environmental information of the real space (real world) is acquired in advance.
  • a device similar to the acquisition device 40 or a method such as photogrammetry may be used to acquire three-dimensional information of the surrounding real space in advance.
  • the content creator or the like may set the scenery seen through the transparent display device 30 and the positions and shapes of the real-space objects on the image generating device 10 and store them in the storage unit 12 as environmental information of the real space.
  • three-dimensional real-space model data such as 3D city model data may be accurately arranged in advance in the virtual space based on the relative position with the position of the transparent display device 30 fixedly arranged on the world coordinate system, and stored in the storage unit 12 as environmental information of the real space. Due to such advance processing, S101 in the flowchart of FIG. 6 may be omitted in these use cases.
  • FIG. 7 is a diagram showing a transmissive display device 30 in one modified example of the system according to the first embodiment, and an example of an image displayed on the transmissive display device 30.
  • the transparent display device 30 in this modified example has a relatively large, horizontally elongated shape so that multiple users 2 (four users 2 in the illustrated example) lined up side by side can view it simultaneously.
  • multiple transparent displays are arranged side by side and configured to function almost equivalently to a single large, indefinite transparent display 30.
  • the real space behind the transparent display device 30 in this modified example can be viewed through the transparent display device 30.
  • the transparent display device 30 in this modified example can be installed, for example, in the window facing a bench in a bus or train, so that multiple users 2 seated on the bench can view it simultaneously.
  • each user 2 looks in a predetermined line of sight (for example, in front of each user 2) from a predetermined position such as the bench.
  • the measurement device 20 determines a representative viewpoint position 2B based on a plurality of viewpoint positions 2A identified for each of the plurality of users 2.
  • the representative viewpoint position 2B is the center of gravity of the plurality of viewpoint positions 2A.
  • the representative viewpoint position 2B may be set in advance based on the position and shape of the seat.
  • the other configurations and operations of the system 1 in this modified example are the same as those of the system 1 in the first embodiment described above, but S102 in the flowchart of FIG. 6 may be skipped depending on the use case.
  • a display image of a virtual object displayed in 3D CG and superimposed on the real space seen through the transparent display device 30 is generated by projective transformation (including perspective projection transformation or a similar calculation method) using information such as the representative viewpoint position 2B and the arrangement, shape, and size of the transparent display device 30, and a composite realistic moving image in which the virtual object displayed in 3D CG exists in the real space of the real world is provided to each user 2 via the transparent display device 30.
  • Each user 2 who sees the image displayed on the transparent display device 30 can be given the illusion that the virtual object exists in the real space of the real world seen through the transparent display device 30.
  • the viewpoint position used in this modified example is the representative viewpoint position 2B, and the viewpoint position 2A of each user 2 is a fixed position, it is not possible to realize a superimposed representation with strict geometric accuracy as in the above-mentioned embodiment. Furthermore, when a real-space three-dimensional model is used to perform occlusion processing and superimposition, there is a lack of accuracy in how it appears to the user, as there is a shift in the outline of the overlapping real-space object and virtual object.
  • FIG. 7 shows how multiple cheetahs, which are virtual objects, are rendered on the transparent display device 30 as if they are running in the direction of the arrows in the figure, in front of a distant mountain range in real space that spreads out behind the transparent display device 30.
  • the mountain range and the cheetah do not overlap with each other in the front-to-back relationship with respect to the viewpoint position 2A of each user 2, and they exist independently, so there is no need to perform occlusion processing on the CG of the cheetah, which is a virtual object, and therefore, even if there is some lack of accuracy, the objective of creating an illusion as if a virtual object were simultaneously present in the real space of the real world seen through the transmissive display device 30 can be achieved to a certain extent.
  • the effect of creating an illusion for the user can be enhanced by devising the expression by adding real-world information as described in the above embodiment.
  • a virtual object cheetah displayed in 3D CG runs in a virtual space, and the cheetah in the virtual space appears to move backwards little by little in parallel as the vehicle moves forward, even if the configuration does not allow for strictly accurate superimposition, by reproducing the relative speed between the vehicle's moving speed and the cheetah's running speed and adding an appearance that the vehicle is overtaking the cheetah and running away, each user 2 can be given the illusion that the cheetah really exists in the real world.
  • calculation processing is performed to generate a virtual object to be displayed for the representative viewpoint position 2B, so it is not possible to provide a representation with geometric precision for all users 2 with different viewpoint positions 2A.
  • a virtual object located in real space behind the transmissive display device 30 is represented on the transmissive display device 30 based on the representative viewpoint position 2B and information on the position, size, orientation, etc. of the transmissive display device 30, so that a more natural-looking display image can be provided to each user 2, giving the illusion that the virtual object exists in real space.
  • the image generating device 10 stores in advance in the storage unit 12 three-dimensional CG data for generating a display image representing a virtual object, and generates a display image from the three-dimensional CG data through a calculation process that generates an image with geometric precision that looks plausible and natural to give the user 2 the illusion that the virtual object is actually there.
  • the second embodiment provides a means for presenting a display image of a virtual object as a three-dimensional image (stereoscopic image) consisting of two images that provide parallax between the left and right eyes.
  • FIG. 8 is a diagram showing a display device and the like in a system according to the second embodiment.
  • System 1 of this embodiment has a transmissive display device 30 that displays a display image of a virtual object on a screen, and a three-dimensional glasses device G worn by user 2.
  • Other configurations and operations of system 1 in this embodiment are similar to those of system 1 in the first embodiment described above.
  • the image generating device 10 generates two display images for the left and right eyes with parallax as a display image of a virtual object according to the viewpoint position 2A of the user 2, and displays them on the screen of the transmissive display device 30.
  • the three-dimensional glasses device G shows the two display images to the left and right eyes of the user 2, respectively.
  • the objects that make up the actual scenery seen by the user have various positions, especially in the depth direction, and the information that enters the left and right eyes is strictly different due to the difference in the positions of the left and right eyes, whereas the virtual object is ultimately displayed as a unique image on the transmissive display 30 in front of the eyes even after calculation, so the parallax between the left and right eyes was not taken into consideration.
  • a certain degree of discomfort is felt in the depth direction when recognizing the existence of a virtual object.
  • a virtual object Vobj the sun
  • a display image showing the virtual object Vobj is displayed on the transmissive display device 30 as two display images for the left and right eyes with parallax, and a user 2 wearing the three-dimensional glasses device G can obtain a visual experience as if a three-dimensional virtual sun object Vobj exists in real space visible beyond the transmissive display device 30.
  • display images of virtual objects displayed in three-dimensional CG and superimposed on the real space seen through the transparent display device 30 are continuously generated following changes in the viewpoint position 2A of the user 2, and a composite realistic moving image in which the virtual objects displayed in three-dimensional CG exist in the real space of the real world is provided to the user 2 via the transparent display device 30. Therefore, the user 2 who sees the image displayed on the transparent display device 30 is given a stronger illusion that the virtual objects exist in the real space of the real world as seen through the transparent display device 30, as the three-dimensional CG virtual objects are superimposed on the scenery of the real space, including the depth information due to the parallax between the left and right eyes.
  • the position information not only the position information but also information regarding the size, shape, etc., of objects in real space around the transmissive display device 30 as seen by the user is obtained, so that it is possible to simulate the front-to-back relationship and relative positional relationship when a virtual object to be superimposed on an object in real space is virtually placed in the same space, and it is possible to generate an image that reflects the representation including the expression due to the occlusion process in which the virtual object is partially visible or hidden when the user's viewpoint position is used as the starting point.
  • the virtual object can be superimposed on the object in real space with higher quality, it is possible to provide the user 2 with a more realistic and natural image that gives the user 2 the illusion that the virtual object exists in real space.
  • a display image that is continuously generated following changes in the viewpoint position 2A of the user 2 and displays the virtual object Vobj as if it exists in the real space of the real world is displayed on the transmissive display device 30 as two display images for the left and right eyes with parallax, so that the user 2 wearing the three-dimensional eyeglass device G can obtain a visual experience in which the virtual object Vobj exists three-dimensionally and naturally, including in the depth direction, in the real space visible beyond the transmissive display device 30.
  • the three-dimensional image system of the transmissive display device 30 and the three-dimensional glasses G according to this embodiment is not particularly limited.
  • it may be anaglyph type, polarized type, or liquid crystal shutter type.
  • the liquid crystal shutter type is adopted as the three-dimensional image system, a synchronization signal for alternately switching the opening and closing of the left and right liquid crystal shutters of the three-dimensional glasses G in synchronization with the timing of alternately switching and displaying the left and right display images on the transmissive display device 30 is transmitted to the three-dimensional glasses G.
  • the system 1 further includes, as an example, a synchronization signal sending device 50 that sends such a synchronization signal, and the synchronization signal sending device 50 is driven and controlled by the processing unit 14 of the image generating device 10.
  • the synchronization signal sending device 50 sends a synchronization signal composed of, for example, infrared light to the three-dimensional glasses G.
  • three-dimensional glasses are generally smaller and lighter than head-mounted displays, they do not cause anxiety or discomfort like head-mounted displays, and the viewing angle at which images can be seen is not narrow and limited like current goggle-type MR devices such as Hololens.
  • a viewpoint position 2A for one user 2 is detected, and a display image consisting of two display images for the left and right eyes with parallax of a virtual object is generated according to the viewpoint position 2A and displayed on the transmissive display device 30, and a three-dimensional virtual object superimposed on real space visible beyond the transmissive display device 30 is presented to the user 2 wearing a three-dimensional eyeglass device G.
  • a means is provided for presenting a three-dimensional virtual object superimposed on real space to multiple users 2 wearing three-dimensional eyeglass devices G.
  • FIG. 9 is a diagram showing a transmissive display device 30 in a modified example of the system according to the second embodiment, and an example of an image displayed on the transmissive display device 30.
  • the transparent display device 30 in this modified example has a relatively large, horizontally elongated shape so that multiple users 2 (four users 2 in the illustrated example) lined up side by side can view it simultaneously.
  • the transparent display device 30 in this modified example is also a transparent display, and the real space behind it can be viewed through the transparent display device 30.
  • the transparent display device 30 in this modified example can be installed, for example, in a window facing a bench in a bus or train, so that multiple users 2 sitting on the bench can view it simultaneously.
  • the measuring device 20 detects the viewpoint position 2A of each user 2, and performs projective transformation, perspective projection transformation, or similar calculation processing on the display image of the virtual object Vobj according to the viewpoint position 2A of each user 2.
  • this modified example employs a liquid crystal shutter type as the 3D image method, and further includes a synchronization signal sending device 50 that sends a synchronization signal for alternately opening and closing the left and right liquid crystal shutters of the 3D glasses G in synchronization with the timing at which the left and right display images are alternately switched and displayed on the transmissive display device 30.
  • the synchronization signal sending device 50 is driven and controlled by the processing unit 14 of the image generating device 10.
  • the synchronization signal sending device 50 sends a synchronization signal, for example composed of infrared light, to the 3D glasses G.
  • the processing unit 14 when multiple users 2 view the virtual object Vobj from their respective viewpoints, the processing unit 14 must perform display control of the display image of the virtual object Vobj for each user 2 so that the left and right display images having an appropriate parallax for each user 2 are displayed on the transmissive display device 30.
  • a transmissive display device 30 with a relatively high screen display rate e.g., 240 Hz
  • the processing unit 14 controls the display images for four users to be displayed in sequence in a time-division manner so that the left and right display images (two images) for each user 2 are displayed at a display rate of 30 Hz.
  • the display rate of 30 Hz is almost the same as the display rate of terrestrial digital television, and each user 2 can view the virtual object Vobj displayed at a display rate of 30 Hz without feeling uncomfortable.
  • a 480 Hz display device may be used to display at a display rate of 60 Hz for each eye of each user 2, or other frequencies may be used.
  • display images of virtual objects displayed in three-dimensional CG and superimposed on the real space seen through the transparent display device 30 are continuously generated following changes in the viewpoint position 2A of each user 2, and composite, realistic moving images in which the virtual objects displayed in three-dimensional CG exist in the real space of the real world are provided to each user 2 via the transparent display device 30. Therefore, each user 2 who sees the image displayed on the transparent display device 30 can be given a stronger illusion that the virtual objects exist in the real space of the real world as seen through the transparent display device 30, as the three-dimensional CG virtual objects are superimposed on the scenery of the real space, including the depth information due to the parallax between the left and right eyes.
  • a display image that is continuously generated following changes in the viewpoint position 2A of each user 2 and displays the virtual object Vobj as if it exists in real space in the real world is displayed on the transmissive display device 30 as two display images for the left and right eyes with parallax, so that each user 2 wearing the three-dimensional eyeglass device G can be individually provided with a visual experience in which the virtual object Vobj exists three-dimensionally and naturally, including in the depth direction, in the real space visible beyond the transmissive display device 30.
  • 3D glasses are generally smaller and lighter than head-mounted displays, so they do not cause the sense of anxiety or discomfort that head-mounted displays do, and the viewing angle of the image is not limited to a narrow angle, as is the case with current goggle-type MR devices such as Hololens.
  • a means for presenting a display image of a virtual object to a user wearing a three-dimensional glasses device as a three-dimensional image (stereoscopic image) consisting of two images that provide parallax between the left and right eyes.
  • a means is provided for presenting a display image of a virtual object to a naked-eye user who is not wearing a three-dimensional glasses device as a three-dimensional image (stereoscopic image) consisting of two images that provide parallax between the left and right eyes.
  • FIG. 10 is a schematic diagram illustrating the configuration of a display device in the system of the third embodiment.
  • the transmissive display device 30 of this embodiment is, for example, a display capable of simultaneously displaying different images in a plurality of display directions, for example, a type of display that realizes naked-eye stereoscopic vision by attaching a lenticular lens to the surface of a transmissive organic electroluminescent display (OLED).
  • the transmissive display device 30 of this embodiment includes a transparent substrate 301 having a plurality of pixel units 302 arranged in an array, and a plurality of lens elements 303 provided on each pixel unit 302 on the transparent substrate 301.
  • each lens element 303 in this embodiment is a semi-cylindrical lens called a lenticular lens.
  • the lens element 303 in addition to a lenticular lens, a lens of any three-dimensional shape such as a hemispherical semi-ellipsoid shape that controls the direction of light may be used. Also, instead of using a lens, a film or plate that generates a parallax barrier may be used.
  • the lens element 303 can be provided on each pixel unit 302 by, for example, individually manufacturing the lens element 303 and bonding it to the position of the transparent substrate 301 where each pixel unit 302 is provided with a transparent adhesive.
  • the lens element 302 can be provided on each pixel unit 302 by discharging a transparent material at the position of the transparent substrate 301 where each pixel unit 302 is provided using 3D printing technology to form the lens element 302.
  • the lens element 303 can be provided on each pixel unit 302 by forming a transparent film member in advance on which a plurality of lens elements 303 positioned with respect to each pixel unit 302 of the transparent substrate 301 are formed, and attaching the transparent film member in a predetermined position on the transparent substrate 301.
  • the transparent substrate 301 may be configured by inserting an array of pixel units and electrode wires that supply power and signals to the pixel units into the intermediate film portion of a plate having a structure in which a plurality of transparent substrates are bonded together, and at the same time, inserting a film or plate containing a lens element, or a film or plate that generates a parallax barrier.
  • a method may be used in which only the array-shaped pixel units are inserted into an intermediate film to form the transparent substrate 301, and then a film or plate containing lens elements or a film or plate that creates a parallax barrier is attached to the outside.
  • the transparent substrate may be a transparent resin plate including an acrylic plate, or glass. Any material may be used as long as it has a certain degree of transparency.
  • the pixel units 302 and lens elements 303 are arranged at a predetermined interval from each other. In the area where the pixel units 302 and lens elements 303 are not arranged, the real space behind the transmissive display device 30 can be seen through the transparent substrate 301. In the transmissive display device 30 of this embodiment, the pixel units 302 and lens elements 303 are arranged relatively sparsely. As an example, the proportion of the pixel units 302 and lens elements 302 in the entire screen of the transmissive display device 30 is several tens of percent to several percent, and if the size of the pixel units can be made smaller through future technological developments, it can be made less than several percent.
  • each pixel unit can be reduced or the light intensity of each pixel unit can be increased, it is possible to display an image that is visible to the user without being affected by surrounding light while maintaining transparency, and in an application in which a virtual object is superimposed on the real space seen through the transmissive display device 30 as in this case, sufficient resolution and clarity in the image can be provided to the user viewing the virtual object.
  • the transparent substrate 301 is provided with electrode wires made of a highly transparent conductive material or very fine metal electrode wires (not shown) that are electrically connected to each pixel unit 302, and each pixel unit 302 is individually driven to emit light or has light transmission control performed by power and drive signals supplied via these electrodes.
  • each pixel unit may be a self-emitting element such as an organic EL or micro LED, or an element that controls light transmission using a liquid crystal mechanism. It may also be an element or light-emitting material that reflects external projection light to allow the user to recognize differences in color visually.
  • each part of the system 1 according to this embodiment is generally similar to the system 1 described in the first embodiment, but the different configurations and operations are described below.
  • each pixel unit 302 is configured as a group of 45 pixels. That is, one pixel unit 302 is made up of 45 pixels, and an RGB display element 302' is arranged in each pixel.
  • the RGB display element 302' is made up of three sub-pixel display elements that display the three colors R (red), G (green), and B (blue).
  • the RGB display element 302' may be a display element for four colors.
  • the pixel unit 302 made up of 45 pixels corresponds to one pixel of a general display.
  • each pixel unit 302 is provided with a lens element 303, which is a semi-cylindrical lens called a lenticular lens, for refracting the directions of the optical paths of the 45 RGB lights displayed on the RGB display elements 302' for 45 pixels in that pixel unit 302 into the 45 display directions, for example 0 to 44, shown in FIG. 12 (hereinafter, these directions are referred to as "display directions 501").
  • a lens element 303 which is a semi-cylindrical lens called a lenticular lens, for refracting the directions of the optical paths of the 45 RGB lights displayed on the RGB display elements 302' for 45 pixels in that pixel unit 302 into the 45 display directions, for example 0 to 44, shown in FIG. 12 (hereinafter, these directions are referred to as "display directions 501").
  • the image generating device 10 inputs the display image data generated by the image generating device 10 corresponding to the viewpoint positions 2A1 to 2A4 of each of the users 21 to 24 to the RGB display element 500 with pixel numbers corresponding to directions 502A, 502B, 502C, and 502D from a predetermined origin O (e.g., the center position of the transmissive display device 30) in the reference coordinate space (FIG. 4) toward the viewpoint positions 2A1 to 2A4 of each of the users 21 to 24.
  • a predetermined origin O e.g., the center position of the transmissive display device 30
  • these directions are referred to as user viewpoint position directions 502A, 502B, 502C, and 502D.
  • these directions are collectively referred to as user viewpoint position directions 502.
  • the processing unit 14 of the image generating device 10 may generate a right-eye display image and a left-eye display image for each of the viewpoint positions 2A 1 to 2A 4 of each of the users 2 1 to 2 4 detected by the measurement device 20, so that when a three-dimensional virtual object in a virtual space is viewed from the right eye and the left eye of the user, who is located at a certain distance to the left and right of the viewpoint position, via the transmissive display device 30, the image appears as a realistic and natural image with geometric rigor to give each user 2 the illusion that the virtual object exists in real space.
  • the image generating device 10 may perform a projection transformation, a perspective projection transformation, or a calculation process similar thereto, such that a virtual object represented by three-dimensional CG data is projected onto the screen of the transmissive display device 30, i.e. , a two-dimensional surface, based on the viewpoint positions 2A 1 to 2A 4 of each of the users 2 1 to 2 4 .
  • the processing unit 14 displays the above-mentioned right eye display image and left eye display image data generated corresponding to each viewpoint position for each viewpoint position 2A1 to 2A4 of each user 21 to 24 on an RGB display element 302' with a pixel number corresponding to the right eye display direction and an RGB display element 302' with a pixel number corresponding to the left eye display direction, which are located on both sides of each user viewpoint position direction 502A, 502B, 502C, 502D corresponding to each viewpoint position.
  • the measurement device 20 detects the viewpoint position 2A of each user 2, and projective transformation, perspective projection transformation, or similar calculation processing is performed on the display image of the virtual object Vobj according to the viewpoint position 2A of each user 2.
  • the processing unit 14 sets the left-eye display image data generated corresponding to the left-eye display direction of the first viewpoint position 2A (e.g., 2A 1 in FIG. 12 ) to the RGB display element 302' of each pixel number corresponding to each display direction 501, for example, from 0th to 3rd in FIG. 12 between the 0th display direction 501 of the 45 display directions 501 from 0th to 44th described in FIG. 12 and the user viewpoint position direction 502 (e.g., 502A in FIG.
  • step S901 This set process is executed for the RGB display element 302' of each pixel number of each pixel unit 302 (see FIG. 11 ) constituting the transmissive display device 30.
  • the processing unit 14 increments the variable value indicating the viewpoint position 2A stored in, for example, a RAM or register (not shown) in the storage unit 12 of the image generating device 10 by +1 (step S905).
  • step S906 judges whether or not the variable value indicating the viewpoint position 2A has exceeded a predetermined last user viewpoint position.
  • the value indicating the last viewpoint position 2A can be determined in advance according to the number of users 2 using the system 1 and stored in the RAM or register.
  • step S906 it is judged whether or not the variable value indicating the viewpoint position 2A has exceeded the last user viewpoint position depending on whether or not the variable value indicating the viewpoint position 2A has exceeded the predetermined and stored value.
  • step S906 determines whether the last user viewpoint position has not yet been processed and the determination in step S906 is NO. If the last user viewpoint position has not yet been processed and the determination in step S906 is NO, the processing unit 14 transfers control to the process in step S902 and executes the processes from step S902 to S904.
  • the processing unit 14 calculates a midpoint direction 503 (for example, 503AB in FIG. 12) passing through the midpoint between the previous viewpoint position 2A (for example, 2A 1 in FIG. 12) and the current viewpoint position 2A (for example, 2A 2 in FIG. 12) (step S902).
  • a midpoint direction 503 for example, 503AB in FIG. 12
  • the processing unit 14 sets the right-eye display image data generated corresponding to the right-eye display direction of the previous user's viewpoint position 2A (e.g., 2A 1 in FIG. 12 ) to the RGB display elements 302' of each pixel number corresponding to each display direction 501, for example, from number 4 to number 10 in FIG. 12 , between the user viewpoint position direction 502 (e.g., 502A in FIG. 12 ) passing through the previous user's viewpoint position 2A (e.g., 2A 1 in FIG. 12 ) and the currently calculated midpoint direction 503 (e.g., 503AB in FIG. 12 ) (step S903).
  • This setting process is executed for the RGB display elements 302' of each pixel number of each pixel unit 302 (see FIG. 11 ) constituting the transmissive display device 30.
  • the processing unit 14 sets the left-eye display image data generated corresponding to the left-eye display direction of the current user's viewpoint position 2A (e.g., 2A 2 in FIG. 12 ) to the RGB display elements 302' of each pixel number corresponding to each display direction 501, for example, from No. 11 to No. 16 in FIG. 12 , between the midpoint direction 503 (e.g., 503AB in FIG. 12 ) calculated this time in step S902 and the user's viewpoint position direction 502 (e.g., 502B in FIG. 12 ) passing through the current user's viewpoint position 2A (e.g., 2A 2 in FIG. 12 ) (step S904).
  • This setting process is executed for the RGB display elements 302' of each pixel number of each pixel unit 302 (see FIG. 11 ) constituting the transmissive display device 30.
  • the processing unit 14 increments the variable value indicating the viewpoint position 2A by +1 (step S905), and determines whether the variable value exceeds the last predetermined user viewpoint position 2A (step S906).
  • step S906 the processing unit 14 executes the above-mentioned processes from steps S902 to S904 again.
  • step S902 similar to that described above, for example in FIG. 12, a midpoint direction 503BC passing through the midpoint between the previous viewpoint position 2A2 and the current user viewpoint position 2A3 is calculated.
  • step S903 the right-eye display image data generated corresponding to the right-eye display direction at the previous viewpoint position 2A2 is set to the RGB display elements 302' of each pixel number corresponding to each display direction 501, for example, from No. 17 to No. 21 in Fig. 12, between the user viewpoint position direction 502B passing through the previous viewpoint position 2A2 and the currently calculated midpoint direction 503BC in Fig. 12.
  • This setting process is executed for the RGB display elements 302' of each pixel number in each pixel unit 302 (see Fig. 11) constituting the transmissive display device 30.
  • step S904 the left-eye display image data generated corresponding to the left-eye display direction of the current viewpoint position 2A 3 is set to the RGB display elements 302' of each pixel number corresponding to each display direction 501, for example, from No. 22 to No. 26 in Fig. 12, between the midpoint direction 503BC calculated this time in step S902 and the user viewpoint position direction 502C passing through the current viewpoint position 2A 3.
  • This setting process is executed for the RGB display elements 302' of each pixel number in each pixel unit 302 (see Fig. 11) constituting the transmissive display device 30.
  • the processing unit 14 increments the variable value indicating the viewpoint position 2A by +1 (step S905), and determines whether the variable value exceeds the last predetermined user viewpoint position 2A (step S906).
  • step S906 the processing unit 14 executes the above-mentioned processes from steps S902 to S904 again.
  • step S902 similar to that described above, for example in FIG. 12, a midpoint direction 503CD passing through the midpoint between the previous viewpoint position 2A3 and the current viewpoint position 2A4 is calculated.
  • step S903 the right-eye display image data generated corresponding to the right-eye display direction of the previous viewpoint position 2A3 is set to the RGB display elements 302' of each pixel number corresponding to each display direction 501 of, for example, numbers 27 to 30 in Fig. 12 between a user viewpoint position direction 502C passing through the previous user viewpoint position 2A3 and the currently calculated midpoint direction 503CD in Fig. 12.
  • This setting process is executed for the RGB display elements 302' of each pixel number in each pixel unit 302 (see Fig. 11) constituting the transmissive display device 30.
  • step S904 the left-eye display image data generated corresponding to the left-eye display direction of the current viewpoint position 2A4 is set to the RGB display elements 302' of each pixel number corresponding to each display direction 501, for example, from No. 31 to No. 34 in Fig. 12, between the midpoint direction 503CD calculated this time in step S902 and the user viewpoint position direction 502D passing through the current user viewpoint position 2A4 .
  • This setting process is executed for the RGB display elements 302' of each pixel number in each pixel unit 302 (see Fig. 11) constituting the transmissive display device 30.
  • the processing unit 14 increments the variable value indicating the viewpoint position 2A by +1 (step S905), and determines whether the variable value exceeds the predetermined final viewpoint position 2A (step S906).
  • step S906 the processing unit 14 executes the processing in the final step S907.
  • step S907 the processing unit 14 sets the right-eye display image data generated corresponding to the right-eye display direction of the last user viewpoint position 2A (e.g., 2A 4 in FIG. 12 ) to the RGB display elements of each pixel number corresponding to each display direction 501, for example, from No. 35 to No. 44 in FIG. 12 , between the user viewpoint position direction 502 (e.g., 502D in FIG. 12 ) passing through the current viewpoint position 204 (e.g., 2A 4 in FIG. 12 ) and the 44th display direction 501.
  • the user viewpoint position direction 502 e.g., 502D in FIG. 12
  • processing unit 14 ends the display process of step S104 in FIG. 6, which is illustrated in the flowchart of FIG. 13.
  • the right-eye or left-eye display image data generated corresponding to the right-eye or left-eye display direction of the adjacent viewpoint position 2A is copied and set in each display direction 501 of the section divided by the user viewpoint position direction 502 of two viewpoint positions 2A in FIG. 12 and the midpoint direction 503 passing through the midpoint between those viewpoint positions 2A.
  • the algorithm of mapping the display pixels to each display direction 501 between the viewpoint position 2A of the user 2 close to the transmissive display device 30 and the viewpoint position 2A of the user 2 far from the transmissive display device 30 it is also possible to make the change sensitivity of the display pixels uniform according to the viewpoint position A of the user 2.
  • display pixels calculated by performing an interpolation process according to each display direction 501 between two adjacent display images may be set.
  • the display image of the virtual object can be presented as a three-dimensional image (stereoscopic image) consisting of two images that provide parallax between the left and right eyes, superimposed on the real space seen through the transmissive display device 30 so that the image appears realistic and natural with geometric rigor to give each user 2 the illusion that the virtual object exists in real space.
  • a three-dimensional image stereoscopic image
  • separate display images of the virtual object corresponding to the viewpoint position 2A of each user 2 can be presented to each user 2 simultaneously.
  • the transparent display device 30 of this embodiment it is possible to attach the transparent display device 30 of this embodiment to the glass part of an automobile or the like, and present a three-dimensional display image of a virtual object to each of a plurality of users 2 seated in the front and rear seats of the automobile.
  • a display image of a virtual object can be individually displayed to a plurality of users 2 who are at different distances and in different viewing directions from the transparent display device 30, so that a virtual object corresponding to the view seen from the viewpoint position of each user 2 can be superimposed and presented on the real space seen around the transparent display device 30 as the glass of a moving automobile.
  • FIG. 14 is a diagram showing a modification of the display device in the system of the third embodiment.
  • the lens element 303 is provided individually for each pixel unit 302.
  • the lens element 303 may have a vertically elongated shape that is arranged across multiple pixel units 302 in each column among the multiple pixel units 302 arranged in a matrix. This makes it possible to reduce the number of steps required to manufacture the lens element 303 compared to when the lens element 303 is provided individually for each pixel unit 302.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Optics & Photonics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Human Computer Interaction (AREA)
  • Crystallography & Structural Chemistry (AREA)
  • Chemical & Material Sciences (AREA)
  • Processing Or Creating Images (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Devices For Indicating Variable Information By Combining Individual Elements (AREA)
PCT/JP2022/040854 2022-11-01 2022-11-01 画像表示システム、ディスプレイ装置 Ceased WO2024095358A1 (ja)

Priority Applications (9)

Application Number Priority Date Filing Date Title
EP22964380.4A EP4614491A4 (en) 2022-11-01 2022-11-01 IMAGE DISPLAY SYSTEM AND DISPLAY DEVICE
JP2024523997A JP7527081B1 (ja) 2022-11-01 2022-11-01 画像表示システム
PCT/JP2022/040854 WO2024095358A1 (ja) 2022-11-01 2022-11-01 画像表示システム、ディスプレイ装置
PCT/JP2023/039394 WO2024096044A1 (ja) 2022-11-01 2023-11-01 ディスプレイ装置
EP23885798.1A EP4614485A4 (en) 2022-11-01 2023-11-01 DISPLAY DEVICE
JP2024523998A JP7511307B1 (ja) 2022-11-01 2023-11-01 ディスプレイ装置
US19/126,305 US20260179513A1 (en) 2022-11-01 2023-11-01 Display device
JP2024098185A JP2024133508A (ja) 2022-11-01 2024-06-18 ディスプレイ装置
JP2024113105A JP2024138501A (ja) 2022-11-01 2024-07-16 画像表示システム、ディスプレイ装置

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/040854 WO2024095358A1 (ja) 2022-11-01 2022-11-01 画像表示システム、ディスプレイ装置

Publications (1)

Publication Number Publication Date
WO2024095358A1 true WO2024095358A1 (ja) 2024-05-10

Family

ID=90930102

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/JP2022/040854 Ceased WO2024095358A1 (ja) 2022-11-01 2022-11-01 画像表示システム、ディスプレイ装置
PCT/JP2023/039394 Ceased WO2024096044A1 (ja) 2022-11-01 2023-11-01 ディスプレイ装置

Family Applications After (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/039394 Ceased WO2024096044A1 (ja) 2022-11-01 2023-11-01 ディスプレイ装置

Country Status (4)

Country Link
US (1) US20260179513A1 (cg-RX-API-DMAC7.html)
EP (2) EP4614491A4 (cg-RX-API-DMAC7.html)
JP (4) JP7527081B1 (cg-RX-API-DMAC7.html)
WO (2) WO2024095358A1 (cg-RX-API-DMAC7.html)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20250128533A (ko) * 2024-02-21 2025-08-28 엘지디스플레이 주식회사 입체 영상 표시　장치
JP2026069353A (ja) * 2024-10-11 2026-04-23 株式会社Dual Move ディスプレイ装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2012068481A (ja) * 2010-09-24 2012-04-05 Asia Air Survey Co Ltd 拡張現実表現システムおよび方法
US20130069933A1 (en) * 2011-09-19 2013-03-21 Disney Enterprises, Inc. Transparent multi-view mask for 3d display systems
WO2017094543A1 (ja) * 2015-12-02 2017-06-08 セイコーエプソン株式会社 情報処理装置、情報処理システム、情報処理装置の制御方法、及び、パラメーターの設定方法
JP2017191378A (ja) * 2016-04-11 2017-10-19 富士通テン株式会社 拡張現実情報表示装置および拡張現実情報表示方法
US20210134844A1 (en) * 2017-10-13 2021-05-06 Boe Technology Group Co., Ltd. Display panel and display device

Family Cites Families (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0389794A (ja) * 1989-09-01 1991-04-15 Nippon Telegr & Teleph Corp <Ntt> 多人数用三次元表示方式
JP3701355B2 (ja) * 1995-10-05 2005-09-28 株式会社半導体エネルギー研究所 複数の観察者が異なる画像を認識する装置
JPH09238369A (ja) * 1996-02-29 1997-09-09 Mitsubishi Electric Corp ３次元像表示装置
US6251566B1 (en) * 1996-12-09 2001-06-26 Scott Brosh Cylindrical lenticular image and method
JP3579242B2 (ja) * 1998-03-27 2004-10-20 三洋電機株式会社 眼鏡無し立体映像表示装置
JP2000276613A (ja) * 1999-03-29 2000-10-06 Sony Corp 情報処理装置および情報処理方法
JP4629838B2 (ja) * 2000-08-25 2011-02-09 株式会社バンダイナムコゲームス 立体視画像生成装置および立体視画像生成方法
JP2003228294A (ja) * 2002-02-04 2003-08-15 Fuji Photo Film Co Ltd 表示装置
JP2006201319A (ja) * 2005-01-19 2006-08-03 Hitachi Displays Ltd 表示装置
JP2006245680A (ja) * 2005-02-28 2006-09-14 Victor Co Of Japan Ltd 映像音響再生方法及び映像音響再生装置
KR101112735B1 (ko) * 2005-04-08 2012-03-13 삼성전자주식회사 하이브리드 위치 추적 시스템을 이용한 입체 디스플레이장치
JP4961929B2 (ja) * 2006-09-29 2012-06-27 セイコーエプソン株式会社 複数画面の画像を同時に表示する画像表示装置
CN101201498A (zh) * 2006-10-23 2008-06-18 Nec液晶技术株式会社 显示设备、终端设备、显示面板和光学构件
JP2008134617A (ja) * 2006-10-23 2008-06-12 Nec Lcd Technologies Ltd 表示装置、端末装置、表示パネル及び光学部材
GB2457692A (en) * 2008-02-21 2009-08-26 Sharp Kk A display device with a plurality of viewing modes
KR101615111B1 (ko) * 2009-06-16 2016-04-25 삼성전자주식회사 다시점 영상 표시 장치 및 방법
WO2012147363A1 (ja) * 2011-04-28 2012-11-01 パナソニック株式会社 画像生成装置
JP5720421B2 (ja) * 2011-05-27 2015-05-20 株式会社Ｊｖｃケンウッド 裸眼立体ディスプレイ装置
KR101274325B1 (ko) * 2011-11-01 2013-06-13 선문대학교 산학협력단 다시점 입체영상 표시 장치 및 그 방법
KR101887058B1 (ko) * 2011-11-11 2018-08-09 엘지전자 주식회사 입체영상 처리 장치 및 입체영상 처리 장치의 전력 제어 방법
US9152173B2 (en) * 2012-10-09 2015-10-06 Microsoft Technology Licensing, Llc Transparent display device
JP6326219B2 (ja) * 2013-11-26 2018-05-16 圭祐 戸田 表示装置および表示方法
DE102014001710A1 (de) * 2014-02-08 2014-08-14 Daimler Ag Vorrichtung und Verfahren zur augmentierten Darstellung eines virtuellen Bildobjektes in einer realen Umgebung
CN107483912B (zh) * 2017-08-31 2019-01-25 成都斯斐德科技有限公司 一种基于浮点型柱状透镜光栅的多视点图像融合方法
CN107678167A (zh) * 2017-10-17 2018-02-09 京东方科技集团股份有限公司 三维显示面板和显示装置
CN108761812A (zh) * 2018-05-31 2018-11-06 北京眸合科技有限公司 一种阶梯型透镜光栅及三维显示系统
US10854171B2 (en) * 2018-12-31 2020-12-01 Samsung Electronics Co., Ltd. Multi-user personal display system and applications thereof
JP2020181030A (ja) * 2019-04-24 2020-11-05 日本精機株式会社 表示装置
WO2021178397A1 (en) * 2020-03-03 2021-09-10 Google Llc Multiview autostereoscopic display using lenticular-based steerable backlighting
US11750795B2 (en) * 2020-05-12 2023-09-05 Apple Inc. Displays with viewer tracking

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2012068481A (ja) * 2010-09-24 2012-04-05 Asia Air Survey Co Ltd 拡張現実表現システムおよび方法
US20130069933A1 (en) * 2011-09-19 2013-03-21 Disney Enterprises, Inc. Transparent multi-view mask for 3d display systems
WO2017094543A1 (ja) * 2015-12-02 2017-06-08 セイコーエプソン株式会社 情報処理装置、情報処理システム、情報処理装置の制御方法、及び、パラメーターの設定方法
JP2017191378A (ja) * 2016-04-11 2017-10-19 富士通テン株式会社 拡張現実情報表示装置および拡張現実情報表示方法
US20210134844A1 (en) * 2017-10-13 2021-05-06 Boe Technology Group Co., Ltd. Display panel and display device

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
"Microsoft HoloLens 2", MICROSOFT CORPORATION, 19 October 2022 (2022-10-19), Retrieved from the Internet <URL:https://www.microsoft.com/ja-jp/hololens>
See also references of EP4614491A4

Also Published As

Publication number Publication date
US20260179513A1 (en) 2026-06-25
EP4614485A1 (en) 2025-09-10
EP4614485A4 (en) 2026-02-25
JP7511307B1 (ja) 2024-07-05
EP4614491A1 (en) 2025-09-10
JPWO2024095358A1 (cg-RX-API-DMAC7.html) 2024-05-10
JPWO2024096044A1 (cg-RX-API-DMAC7.html) 2024-05-10
WO2024096044A1 (ja) 2024-05-10
EP4614491A4 (en) 2025-11-12
JP7527081B1 (ja) 2024-08-02
JP2024138501A (ja) 2024-10-08
JP2024133508A (ja) 2024-10-02

Similar Documents

Publication Publication Date Title
JP7443602B2 (ja) 仮想コンテンツワーピングを伴う複合現実システムおよびそれを使用して仮想コンテンツを生成する方法
JP7329612B2 (ja) オブジェクトカテゴリモデリングのための生成潜在テクスチャプロキシ
JP4764305B2 (ja) 立体画像生成装置、方法およびプログラム
US10204452B2 (en) Apparatus and method for providing augmented reality-based realistic experience
US10672311B2 (en) Head tracking based depth fusion
CN103337095B (zh) 一种真实空间三维地理实体的立体虚拟显示方法
JP2024138501A (ja) 画像表示システム、ディスプレイ装置
CN106412556B (zh) 一种图像生成方法及装置
JP7426413B2 (ja) ブレンドモード３次元ディスプレイシステムおよび方法
KR20130097014A (ko) 확장형 3차원 입체영상 디스플레이 시스템
US10931938B2 (en) Method and system for stereoscopic simulation of a performance of a head-up display (HUD)
JP2025013901A (ja) 仮想、拡張、および複合現実システムおよび方法
CN120953534B (zh) 基于真实地理空间的数字孪生视频全息显示方法及系统
WO2024095356A1 (ja) グラフィックス生成装置、グラフィックス生成方法及びプログラム
Kovalev On personal-use displays for virtual environments with augmented reality
JP2026069353A (ja) ディスプレイ装置
WO2024095357A1 (ja) 映像システム
CN122008852A (zh) 一种悬浮式3d成像方法、成像系统和车辆

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2024523997

Country of ref document: JP

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22964380

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2022964380

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022964380

Country of ref document: EP

Effective date: 20250602

WWP Wipo information: published in national office

Ref document number: 2022964380

Country of ref document: EP