WO2023079623A1 - Système d'affichage d'image, dispositif de transmission d'image, dispositif de commande d'affichage et procédé d'affichage d'image - Google Patents

Système d'affichage d'image, dispositif de transmission d'image, dispositif de commande d'affichage et procédé d'affichage d'image Download PDF

Info

Publication number
WO2023079623A1
WO2023079623A1 PCT/JP2021/040577 JP2021040577W WO2023079623A1 WO 2023079623 A1 WO2023079623 A1 WO 2023079623A1 JP 2021040577 W JP2021040577 W JP 2021040577W WO 2023079623 A1 WO2023079623 A1 WO 2023079623A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
display
data
original image
event information
Prior art date
Application number
PCT/JP2021/040577
Other languages
English (en)
Japanese (ja)
Inventor
章男 大場
博之 勢川
Original Assignee
株式会社ソニー・インタラクティブエンタテインメント
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社ソニー・インタラクティブエンタテインメント filed Critical 株式会社ソニー・インタラクティブエンタテインメント
Priority to PCT/JP2021/040577 priority Critical patent/WO2023079623A1/fr
Publication of WO2023079623A1 publication Critical patent/WO2023079623A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
    • G09G3/20—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/513—Processing of motion vectors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering

Definitions

  • the present invention relates to image display technology for displaying moving images using transmitted original images.
  • UHDTV Ultra HDTV
  • HDTV High Definition Television
  • the resolution and viewing angle of images, as well as the number of pixels that make up one frame tend to increase more and more.
  • an increase in the amount of data causes various problems such as a tight transmission band, an increase in calculation cost, and an increase in power consumption.
  • the frame rate does not increase and the image quality deteriorates. .
  • the present invention has been made in view of these problems, and its purpose is to provide a technology that can easily display images with high resolution and wide viewing angles with high quality.
  • This image display system is an image display system comprising an image transmission device for transmitting original image data and a display control device for controlling image display using the original image data, wherein the image transmission device comprises: When a luminance change event occurs in any pixel in the original image, an event information generation unit that generates event information including position information of the pixel, and transmits data of the entire original image at a predetermined frame rate, and a data transmission unit that asynchronously transmits event information, and the display control device uses the data of the entire original image to generate display image data at a predetermined frame rate, and based on the latest event information,
  • the present invention is characterized by comprising a display image drawing section for locally updating a display image, and an output section for outputting data of the display image to a display device.
  • This image transmission device is an image transmission device that transmits data of an original image and realizes an image display using the original image at a transmission destination.
  • an event information generation unit that generates event information including the position information of the pixel, data of the entire original image are transmitted at a predetermined frame rate, and the event information is transmitted asynchronously, thereby generating a display image at the transmission destination.
  • a data transmission unit that locally updates the .
  • This display control device is a display control device that receives data of an original image and controls image display using the data of the original image. and an event information acquisition unit that acquires event information including position information of a pixel transmitted asynchronously with data of the entire original image in response to a luminance change event occurring in any pixel in the original image. a display image drawing unit that generates display image data at a predetermined frame rate using data of the entire original image and locally updates the display image based on recent event information; and an output unit for outputting to a display device.
  • This image display method comprises an image display system comprising an image transmission device for transmitting original image data and a display control device for controlling image display using the original image data. generating event information including position information of the pixel when a luminance change event occurs in one of the pixels; transmitting data of the entire original image at a predetermined frame rate; and the display control device generates display image data at a predetermined frame rate using the data of the entire original image, and locally updates the display image based on recent event information. and outputting data of the display image to a display device.
  • images with high resolution and wide viewing angles can be easily displayed with high quality.
  • FIG. 1 is a diagram showing an internal circuit configuration of a display control device according to this embodiment
  • FIG. 1 is a diagram showing functional block configurations of an image transmission device and a display control device according to the present embodiment
  • FIG. 10 is a diagram for explaining a problem related to the balance between the amount of provided information and the amount of received information when a flat-panel display is used as the display destination;
  • FIG. 3 is a diagram for explaining the relationship between data transmitted by an image transmission device and a display image generated by a display control device in the embodiment
  • FIG. 7 is a diagram showing another example of the relationship between data transmitted by the image transmission device and a display image generated by the display control device in the embodiment
  • 4 is a diagram illustrating data formats of an omnidirectional image and event information prepared by the image transmission device in the present embodiment
  • FIG. 5 is a diagram for explaining the relationship between the shape of the view screen and the amount of provided information
  • FIG. 4 is a diagram for explaining distortion image drawing processing in the display control device according to the embodiment
  • It is a figure which shows the modification of the image display system in this Embodiment.
  • FIG. 10 is a diagram for explaining the relationship between partial update and reprojection based on event information in the embodiment;
  • FIG. 7 is a diagram for explaining display image generation processing when an omnidirectional image is represented by a cube map in the present embodiment;
  • FIG. 10 is a diagram for explaining display image generation processing when an omnidirectional image is represented by equirectangular projection in the present embodiment;
  • FIG. 10 is a diagram for explaining display image generation processing when an omnidirectional image is represented by equidistant projection in the present embodiment;
  • FIG. 4 is a diagram for explaining a mode in which an omnidirectional image is represented by equidistant projection and a plurality of resolution levels are provided in the present embodiment;
  • FIG. 10 is a diagram for explaining a method of generating an equidistant projection image having a distribution in which the resolution decreases from the center to the periphery in the present embodiment;
  • FIG. 10 is a diagram for explaining display image generation processing when an omnidirectional image is represented on the plane of a regular octahedron in the present embodiment;
  • FIG. 1 shows a configuration example of an image display system to which this embodiment can be applied.
  • the image display system 1 includes display control devices 10a, 10b, and 10c that display images according to user operations, and an image transmission device 20 that provides image data used for display.
  • Input devices 14a, 14b, and 14c for user operations and display devices 16a, 16b, and 16c for displaying images are connected to the display control devices 10a, 10b, and 10c, respectively.
  • the display control devices 10a, 10b, 10c and the image transmission device 20 can establish communication via a network 8 such as a WAN (World Wide Network) or a LAN (Local Area Network).
  • a network 8 such as a WAN (World Wide Network) or a LAN (Local Area Network).
  • the display control devices 10a, 10b, 10c, the display devices 16a, 16b, 16c, and the input devices 14a, 14b, 14c may be connected either by wire or wirelessly. Alternatively, two or more of those devices may be integrally formed.
  • the display control device 10b is connected to a head-mounted display, which is the display device 16b. Since the head-mounted display can change the field of view of the displayed image according to the movement of the user wearing it on the head, it also functions as the input device 14b.
  • the display control device 10c is a mobile terminal, and is integrally configured with a display device 16c and an input device 14c, which is a touch pad covering the screen.
  • the external shape and connection form of the illustrated device are not limited.
  • the number of display control devices 10a, 10b, 10c and image transmission devices 20 connected to the network 8 is not limited either.
  • the image transmission device 20 may be a cloud server or content server in a system such as cloud gaming.
  • the display control devices 10a, 10b, and 10c may be collectively referred to as the display control device 10, the input devices 14a, 14b, and 14c as the input device 14, and the display devices 16a, 16b, and 16c as the display device 16 in some cases.
  • the input device 14 may be one or a combination of general input devices such as a controller, keyboard, mouse, touch pad, joystick, etc., and supplies the contents of user operations to the display control device 10 .
  • the display device 16 may be a general display such as a liquid crystal display, a plasma display, an organic EL display, a wearable display, or a projector, and displays images output from the display control device 10 .
  • the image transmission device 20 provides the display control device 10 with content data accompanied by image display.
  • the type of content is not particularly limited, and may be electronic games, images for viewing, television broadcasts, electronic conferences, video chats, or the like.
  • the display control device 10 basically acquires content data used for display from the image transmission device 20 and implements display processing. Alternatively, the image transmission device 20 and the display control device 10 may cooperate to generate content data, thereby distributing the load of drawing processing and the like.
  • FIG. 2 shows an example of an image transmitted and displayed in this embodiment.
  • (a) is an example of displaying the data of the captured image 130 as it is as the display image 132 in a live broadcast, video chat, or the like.
  • the source of the image data may be an imaging device connected to the display control device 10 as well as the image transmission device 20 .
  • the display device 16 is a flat display.
  • (b) is an example of preparing an omnidirectional image 134 and generating a display image 136 in a field of view corresponding to movement of the head of a user wearing a head-mounted display.
  • stereoscopic vision can be realized by displaying stereoscopic images with parallax on the left and right regions of the display panel corresponding to the left and right eyes of the user, as in the illustrated display image 136 .
  • the displayed image 136 has distortion and chromatic aberration corresponding to the eyepiece lens so that a distortion-free image can be viewed through the eyepiece lens. Give the opposite distortion.
  • the left and right images of the display image 136 are each curved in a barrel shape.
  • an image to which distortion corresponding to the eyepiece is applied is called a "distorted image”.
  • the omnidirectional image 134 may be an image rendered by computer graphics, a photographed image, or a combination thereof.
  • the image data transmission source may be a drawing mechanism inside the display control device 10, an imaging device connected to the display control device 10, or the like.
  • the delay time until display cannot be overlooked.
  • the delay time greatly affects the operability of the game.
  • the power consumption for data processing and transmission increases, especially in the case of a head-mounted display, a large-capacity storage battery may be required, and the feeling of wearing may deteriorate due to heat generation.
  • resources related to processing and transmission are saved by capturing images in units of pixels and transmitting information.
  • the data transmission source acquires the occurrence of the change event in units of pixels, and transmits only the information related to the local change event through the bypass line independently of the data of the entire image.
  • the display control device 10 locally updates the portion of the display image where the change event has occurred based on the most recently transmitted information.
  • a “change event” refers to a state in which a difference equal to or greater than a threshold occurs in observed luminance or pixel values (hereafter simply referred to as an "event").
  • a data transmission source such as the image transmission device 20 transmits data such as the captured image 130, the omnidirectional image 134, or a portion thereof at a predetermined rate, and at the time when a change event occurs, the pixel position and Send event information, including the content of the change, by bypass.
  • the display controller 10 generates frames of display images at a predetermined rate and locally updates the frames based on the transmitted event information.
  • the display control device 10 draws the display image using ray tracing technology.
  • Ray tracing is a method of generating a virtual ray that passes through each pixel on the view screen from the viewpoint and determining the pixel value based on the color information of the destination.
  • an image transmitted from the image transmission device 20 or the like is placed in front of the view screen, and the destination of the ray is obtained to acquire color information on the image. By adopting this method, it becomes possible to selectively draw only pixels where an event has occurred.
  • Ray tracing is generally regarded as a process with a large amount of computation and a heavy load.
  • most intermediate buffer memories such as depth buffers are unnecessary.
  • the efficiency of drawing can be improved while maintaining the quality.
  • by enabling the transmission and rendering of information in pixel units it is possible to minimize the transmission of data such as the background that does not contribute to the improvement of image quality, and reduce the processing and transmission load at the data transmission source.
  • FIG. 3 illustrates a specific configuration of the image display system according to this embodiment.
  • the image data that is the source of the display image is not limited to be generated by the image transmission device 20, and at least part of it is generated by a local device on the display side such as the display control device 10 or an imaging device. Although it may play a role, hereinafter, the image transmission device 20 will be mainly described. Further, the entity that forms the final display image from the image data using the ray tracing technique is not limited to the display control device 10, and at least a part thereof may be performed by the display device 16. The control device 10 will be described. Hereinafter, the image transmitted by the image transmission device 20 will be referred to as an "original image" to be distinguished from the "display image” generated by the display control device 10. FIG.
  • the figure shows a state in which two image transmission devices 20a and 20b and two display control devices 10a and 10b are connected via an arbitrary network such as LAN, WiFi (registered trademark), or 5G.
  • the image transmission device 20a transmits the moving image of the speaker 160 captured by the imaging device 140 to the display control devices 10a and 10b, as shown in FIG. 2(a).
  • the image transmission device 20b captures an omnidirectional image 142 with the imaging device 141, and transmits at least part of it to the display control devices 10a and 10b.
  • the omnidirectional image 142 is shown as a cross section of a spherical screen representing the image plane, but the representation method (data format) of the omnidirectional image 142 is not particularly limited as described later.
  • the imaging device 141 may be configured by a multi-lens camera in addition to a combination of fish-eye cameras on the front and back.
  • the image transmission devices 20a and 20b may render images equivalent to the images captured by the imaging devices 140 and 141 using computer graphics, or may render computer graphics on the captured images.
  • the display control device 10a uses the data of the original image 148 transmitted from the image transmission devices 20a and 20b to display the display image 150 on the flat panel display, which is the display device 16.
  • the display control device 10b uses the data of the original image 152 transmitted from the image transmission devices 20a and 20b to display the display image 154 on the head-mounted display, which is the display device 16.
  • the diagram shows the data of the original images 148 and 152 as part of a spherical screen representing the image plane
  • the data format is not limited to this.
  • the screen is of a flat plate type. Since the image displayed on the head-mounted display is a distorted image as described above, the screen (view screen) of the display image 154 is shown as a substantially spherical surface.
  • a display image 154 with distortion is directly generated from the original image 152.
  • the original image 152 is placed in front of the viewscreen in a curved screen shape corresponding to its distortion.
  • the original image 152 is then sampled from the curved viewscreen by ray tracing to generate the displayed image 154 .
  • the image transmission devices 20a and 20b From the image transmission devices 20a and 20b, data 166 of the entire original image is transmitted at a predetermined rate, as indicated by the thick line in the drawing.
  • the "whole original image” is the entire area of the frame of the original image, for example, the entire area of the image within the field of view.
  • the “frame” to be transmitted is not limited to the image within the field of view, and may include a range wider than the field of view used by the display control devices 10a and 10b to form the display image.
  • the image transmitting devices 20a and 20b also transmit event information 168, including the position of the pixel where the event occurred and the content of the change, via a bypass, as indicated by the dashed lines in the drawing.
  • the display control devices 10a and 10b basically use the data 166 of the entire original image transmitted from the image transmission devices 20a and 20b at a predetermined rate to generate the entire display image and cause the display device 16 to display it. At this time, the display control devices 10a and 10b increase the display frame rate by interpolating frames of the original image, or adjust the field of view according to the latest position and posture of the head of the user 162b viewing the head-mounted display. may be corrected. Then, the display control devices 10a and 10b use the event information 168 asynchronously transmitted from the image transmission devices 20a and 20b to selectively update necessary portions in the display image.
  • the display control devices 10a, 10b also transmit data 170a, 170b relating to the position or area on the screen that the user 162a, 162b looking at the display image is gazing to the image transmission device 20a, 20b at a predetermined rate. You may In this case, the image transmission devices 20a and 20b exclude, from among the events that have occurred in the original images, information on events outside the region that the users 162a and 162b are watching from the transmission targets. This makes it possible to further limit the update targets in transmission data and display images without notifying the users 162a and 162b.
  • FIG. 4 shows the internal circuit configuration of the display control device 10.
  • the display control device 10 includes a CPU (Central Processing Unit) 22 , a GPU (Graphics Processing Unit) 24 and a main memory 26 . These units are interconnected via a bus 30 .
  • An input/output interface 28 is also connected to the bus 30 .
  • the input/output interface 28 outputs data to a peripheral device interface such as USB and IEEE 1394, a communication unit 32 including a wired or wireless LAN network interface, a storage unit 34 such as a hard disk drive and a nonvolatile memory, and a display device 16.
  • An output unit 36, an input unit 38 for inputting data from the input device 14, an imaging device, etc., and a recording medium driving unit 40 for driving a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory are connected.
  • the CPU 22 controls the entire display control device 10 by executing the operating system stored in the storage unit 34 .
  • the CPU 22 also executes various programs read from a removable recording medium and loaded into the main memory 26 or downloaded via the communication section 32 .
  • the GPU 24 performs drawing processing according to a drawing command from the CPU 22, and stores the display image in a frame buffer (not shown). Then, the display image stored in the frame buffer is converted into a video signal and output to the output section 36 .
  • the main memory 26 is composed of a RAM (Random Access Memory) and stores programs and data necessary for processing.
  • the circuit configuration of the image transmission device 20 may also be substantially the same, and in this case, the acquired or generated image data is transmitted to the display control device 10 via the communication unit 32 .
  • FIG. 5 shows the configuration of functional blocks of the image transmission device 20 and the display control device 10 in this embodiment. Note that the image transmission device 20 and the display control device 10 may perform general information processing such as progressing an electronic game or outputting sound. is shown.
  • the functional blocks shown in FIG. 5 can be realized by the configuration of the CPU 22, GPU 24, main memory 26, etc. shown in FIG. It is realized by a program that exhibits various functions such as a data input function, a data holding function, an image processing function, and a communication function. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and are not limited to either one.
  • the image transmission device 20 includes a gaze area acquisition unit 50 that acquires information related to the user's gaze area, an image data acquisition unit 52 that acquires data of the original image, and data for transmitting the data of the original image to the display control device 10.
  • a transmitter 56 is provided.
  • the gaze area acquisition unit 50 acquires information indicating the position coordinates of the gaze point of the user on the screen of the display device 16 or the area of a predetermined range including the gaze point from the display control device 10 at a predetermined rate.
  • the image data acquisition unit 52 acquires data of the entire original image at a predetermined rate.
  • the data may be generated by the image data acquisition unit 52 itself, or may be acquired from an external device (not shown).
  • the image data acquisition unit 52 acquires captured images at a predetermined rate from an imaging device connected to the image transmission device 20 .
  • the image data acquisition unit 52 draws the original image at a predetermined rate by computer graphics based on the model data and game program stored therein.
  • the image data acquisition unit 52 may acquire the content of the user's operation on the game or the like from the display control device 10 .
  • the image data acquisition unit 52 also acquires the information related to the position and orientation of the head at a predetermined rate.
  • the head-mounted display is equipped with at least one of a motion sensor and a camera that captures images of the surrounding space.
  • the display control device 10 obtains the position and orientation of the head-mounted display and, in turn, the user's head, based on the measurement values of the motion sensor and the images captured by the camera, and transmits them to the image transmission device 20 at a predetermined rate.
  • Various techniques have been put into practical use as techniques for obtaining the position and orientation of the head-mounted display, and any of them may be adopted in the present embodiment.
  • the image data acquisition unit 52 preferentially draws the original image of the area within the user's field of view, or the area expected to be within the user's field of view, among the omnidirectional images representing the virtual world, for example.
  • the image data acquisition unit 52 can be regarded as an “original image drawing unit”. Note that the image data acquisition unit 52 may draw computer graphics on the captured image.
  • the image data acquisition unit 52 also includes an event information generation unit 54 that detects the occurrence of an event and generates event information to be transmitted.
  • the event information generation unit 54 may acquire event information from the imaging device.
  • EDS event-driven sensor
  • EDS also called neuromorphic camera, silicon retina, or dynamic vision sensor
  • EDS is an image sensor that responds to local change events in brightness. Specifically, when a change occurs in the intensity of incident light in any sensor of the sensor array, it outputs a signal indicating the position, time, and whether it increases or decreases. Since EDS locally responds only to pixels that have occurred at the time when an event occurs, it has a light load, low latency, and low latency compared to an image processing approach that extracts differences while scanning the entire image. Output with power consumption is possible. In other words, information can be output with higher temporal resolution than processing the entire image.
  • the event information generation unit 54 acquires an event signal from the EDS provided in the imaging device and immediately generates event information. For example, information is generated that associates the position of a pixel where the intensity of incident light has changed by a threshold value or more, the occurrence time, and the details of the change.
  • the position of a pixel may be represented by the direction of a ray passing through each pixel of the original image from the virtual viewpoint or the center of the optical axis of the camera.
  • the content of the change is not particularly limited as long as it is information that allows the display control device 10 to update the display image based on it.
  • it may be a pixel value difference or a pixel value after change.
  • the display control device 10 itself can draw the object by including information used to draw the object represented by the pixel where the event occurred, such as depth information and model data, in the event information. You may do so.
  • the imaging device may have a configuration in which an image sensor for capturing a general color image and an EDS are separately provided, or the device may output a color image by accumulating event data for a predetermined period of time. good too.
  • the positions of the two sensor arrays are associated with each other, and clocks for generating the imaging time and the event occurrence time are synchronized.
  • it is expected that the frame rate of color images will be reduced and the compression rate will be improved, making it easier to transmit depth information and model data of the object itself in which an event has occurred.
  • the event information generation unit 54 when the image data acquisition unit 52 draws the original image by itself, the event information generation unit 54 generates event information according to a program for the image data acquisition unit 52 to draw the original image. That is, the image data acquisition unit 52 determines the movement of the object in the image world before drawing the image by ray tracing or the like. Therefore, when drawing an image, it is possible to specify by itself at which pixel an event occurs. By acquiring the information, the event information generator 54 generates event information in the same way as when using the event signal from the EDS. When synthesizing a captured image with computer graphics, the event information generating section 54 may generate event information based on both an event signal from the EDS and event information found when the image is drawn.
  • the event information generation unit 54 may further select events to be transmitted as event information based on information related to the gaze point and the gaze area acquired by the gaze area acquisition unit 50 . For example, the event information generation unit 54 excludes events that have occurred outside the attention area from transmission targets by masking them.
  • the data transmission unit 56 transmits the data of the entire original image to the display control device 10 at a predetermined rate, and immediately transmits the event information when the event information generation unit 54 generates the event information.
  • the display control device 10 includes a gaze area transmission unit 60 that transmits information related to the user's gaze area, an image data acquisition unit 62 that acquires data of the entire original image from the image transmission device 20, and acquires event information from the image transmission device 20. , an event information acquisition unit 64 for processing, a display image drawing unit 66 for drawing a display image, and an output unit 74 for outputting display image data.
  • the gaze region transmission unit 60 acquires from the display device 16 the positional coordinates of the user's gaze point on the screen of the display device 16 or the information related to the region of a predetermined range including the gaze point, and transmits the information to the image transmission device 20 at a predetermined rate. Send. For this reason, the display device 16 is provided with a gaze point detector (not shown).
  • a point-of-regard detector is a general device that detects a point of an object that a person is gazing at by, for example, photographing the reflection of infrared rays irradiated to the eyeball.
  • the image data acquisition unit 62 acquires data of the entire original image from the image transmission device 20 at a predetermined rate.
  • the event information acquisition unit 64 acquires event information from the image transmission device 20 each time.
  • the display image drawing section 66 includes a frame image drawing section 70 for drawing frames of the display image, and an event information management section 72 for changing the processing of the frame image drawing section 70 according to the occurrence of an event.
  • the frame image rendering unit 70 uses the data of the entire original image acquired by the image data acquiring unit 62 to render the frames of the display image at a predetermined rate using a technique similar to ray tracing.
  • the frame image rendering unit 70 generates a ray passing through each pixel on the view screen from the user's viewpoint, and samples the pixel values of the arrival point in the original image placed in front of the ray, thereby obtaining the pixel values of the display image. decide.
  • the pixel array of the display screen and the pixel array of the original image may be the same.
  • the frame image drawing unit 70 performs local update by switching the processing content for the place where the event occurred based on the most recent event information.
  • the frame image drawing unit 70 determines the pixel values of the display image based on the information of the pixel values of the original image included in the event information for the location where the event has occurred.
  • the position where the event occurs in the original image is associated with the direction of the ray, whether or not the event has occurred at the destination can be determined from the relationship with the ray direction set in the ray tracing.
  • the frame image drawing unit 70 replaces the sampling destination with pixel value information included in the event information, thereby updating only the necessary portion of the display image.
  • the frame image drawing unit 70 may draw the image of the object in which the event occurred by 3DCG based on the depth information and model data of the object included in the event information. At this time, the frame image drawing unit 70 may perform general ray tracing that calculates reflection, refraction, and the like on the surface of the object. In any case, in the present embodiment, the rendering of the entire display image is performed based on ray tracing, so even with such local correspondence, there is little change in processing content and switching is easy.
  • the frame image drawing unit 70 draws the frames of the display image at a rate higher than the frame rate of the original image in order to reflect the occurrence of the event in the display with high time resolution.
  • the event information management section 72 switches the drawing process in the frame image drawing section 70 as described above based on the event information acquired by the event information acquisition section 64 . That is, it determines whether or not an event has occurred at the set destination of the ray, replaces the sampling destination, or draws the object in which the event has occurred by 3DCG.
  • the output unit 74 outputs data of the display image drawn by the display image drawing unit 66 to the display device 16 for display. As a result, at a frame rate higher than the transmission rate of the original image, an image showing details of changes in the portion where the event occurred is displayed.
  • FIG. 6 is a diagram for explaining a problem related to the balance between the amount of information provided and the amount of information received when a flat panel display is used as the display destination.
  • the amount of provided information is the amount of information represented by the display image
  • the amount of received information is the amount of information recognized by the brain of the user who is viewing the image.
  • the figure shows the angle of view (or FOV: Field Of View) of the displayed image at an appropriate distance from the user's viewpoint 80 .
  • FOV Field Of View
  • the angle of view ⁇ 2K is about 33°.
  • the angles of view ⁇ 4K and ⁇ 8K are approximately 61° and 100°, respectively.
  • the figure also shows a screen 86 of a head-mounted display for comparison, and in this case, the angle of view is about 100° like the screen 84.
  • the larger the screen of the display the wider the angle of view from the line of sight V becomes.
  • the number of pixels on the screen included per unit angle when viewed from the viewpoint 80 increases as the distance from the line of sight V increases. For example, when looking at the screen 84, even if the range of the angle ⁇ is the same, a wider range of pixels come into the field of view in a region 90b distant from the line of sight V than in a region 90a close to the line of sight V.
  • the human visual function has a distribution with respect to the visual field.
  • the central region on the retina that corresponds to the center of the visual field is called the fovea, and has the highest visual acuity and ability to discriminate colors and shapes.
  • the area within about 5° from the line of sight, including the area that is foveated out of the visual field, is called central vision, the area within about 20° from the line of sight is called the effective field of view, and the area outside it is called peripheral vision. It decreases in that order.
  • the amount of received information becomes smaller the further away from the line of sight V.
  • the density of the amount of information entering the field of view increases at positions away from the line of sight V as described above, displaying the entire image with a uniform spatial detail level will increase the amount of information provided and the amount of information received. It can be said that it is in a state of balance.
  • the line of sight V changes due to the movement of the image of the object, etc., so the area that can be captured with high visual performance moves on the screen. In order to deal with this, it is necessary to raise the frame rate and update the image in detail temporally at least for the relevant area.
  • the image quality is improved by paying particular attention to an area within a predetermined range from the line of sight V, and by extension, the point of gaze, thereby correcting the imbalance between the amount of provided information and the amount of received information.
  • Eliminate waste of processing and transmission for the area Specifically, as described above, information about an event occurring outside the region of interest is excluded from transmission targets. As a result, wasteful update processing in the display control device 10 can be eliminated, and display at a high frame rate is possible.
  • FIG. 7 is a diagram for explaining the relationship between the data transmitted by the image transmission device 20 and the display image generated by the display control device 10.
  • FIG. The figure shows the passage of time in the horizontal direction, and the middle part of the figure represents the arrangement of the displayed frames.
  • the updated portion is indicated by a thick line in the vertical line representing the entire frame.
  • FIG. 2(a) it is assumed that an image captured at a remote location such as a live broadcast or video chat is displayed on a flat panel display.
  • the data transmitted from the image transmission device 20 consists of data of the entire original images 180a, 180b, . . . and event information 182a, 182b, 182c, .
  • the figure shows the images of speakers on the image plane as event information 182a, 182b, 182c, .
  • event information 182a, 182b, 182c is transmitted with higher spatial and temporal resolution, such as speaking lips or blinking eyes.
  • the event information 182a, 182b, 182c is transmitted with higher spatial and temporal resolution, such as speaking lips or blinking eyes.
  • the data of the entire original images 180a, 180b, . is sent asynchronously with the timing of event occurrence.
  • the display control device 10 generates a display image frame using the transmitted data. That is, the display image 184a is generated using the data of the entire original image 180a. Display image 184a corresponds to frame 188 in the middle row.
  • the display control device 10 updates only the necessary part 190 using the event information 182a, 182b, 182c, .
  • Display images 186a, 186b, 186c, . . . are generated. Since the transmission timing of the event information 182a, 182b, 182c, . Then, the display control device 10 generates a corresponding display image 184b based on the data of the entire original image 180b transmitted next.
  • the display control device 10 interpolates between the frames of the entire original images 180a, 180b, . to achieve the display of
  • the display control device 10 uses the original image of 30 fps or 60 fps transmitted from the image transmission device 20 and displays the image at 120 fps or 240 fps.
  • FIG. 8 shows another example of the relationship between the data transmitted by the image transmission device 20 and the display image generated by the display control device 10.
  • FIG. 8 shows another example of the relationship between the data transmitted by the image transmission device 20 and the display image generated by the display control device 10.
  • FIG. 2(b) it is assumed that an area corresponding to the user's field of view in the omnidirectional image is displayed on the head-mounted display.
  • the data transmitted from the image transmission device 20 consists of data of the entire original images 200a, 200b, . . . and event information 202a, 202b, 202c, .
  • the image transmission device 20 transmits the data of the entire original images 200a, 200b, .
  • an animal character is moving, and event information 202a, 202b, 202c, .
  • omnidirectional images are shown as the entire original images 200a, 200b, . may be limited to The event information 202a, 202b, 202c, .
  • the display control device 10 generates display images 204a, 204b, . . . using data of the entire original images 200a, 200b, . , . . . are used to generate display images 206a, 206b, 206c, .
  • the display control device 10 may perform reprojection to correct the field of view so as to correspond to the position and posture of the user's head at each time, and then generate distorted images for the left eye and the right eye. .
  • the position and posture of the user's head can be acquired using the motion sensor and camera provided in the head-mounted display.
  • the display control device 10 draws pixels where events have occurred using the event information 202a, 202b, 202c, . Even in this case, even if the frame rate of the entire original image is lowered, a high-quality image can be displayed at a high frame rate such as 120 fps or 240 fps. In addition, it is possible to achieve both low delay in the change of the visual field with respect to the movement of the user's head and low delay in the movement of the image in the fixation area. Since the event detection is performed independently of the user's motion, the motion in the world coordinate system can be accurately detected by comparing the difference between the frames of the display image.
  • FIG. 9 exemplifies the data format of the omnidirectional image and event information prepared by the image transmission device 20 .
  • (a) of the figure shows the equidistant cylindrical projection
  • (b) shows the form of equidistant projection directly expressing the output of the fisheye camera.
  • an omnidirectional camera in which fisheye cameras having an angle of view of about 220° are combined on the front and back is used.
  • an image of equidistant projection in which an object is projected onto a spherical surface is obtained.
  • such an image of equidistant projection is once converted into data 210a in equirectangular projection, and then at least part of it is targeted for transmission.
  • the event information 212a represents the occurrence position of the event by the position coordinates on the image plane of the equirectangular projection.
  • equidistant projection data 210b as shown in (b) may be transmitted. By doing so, it is possible to realize low-delay data transmission without processing for converting to data 210a in the equirectangular projection or preparing a buffer memory for that purpose.
  • the event information 212b represents the occurrence position of the event with position coordinates on the image plane in equidistant projection.
  • event information can also be transmitted with low delay without coordinate conversion of the output from the EDS.
  • only the data of the visual field range 214 in the equidistant projection data 210b may be transmitted as the data of the entire original image.
  • only the information of the event occurring in the gaze area 216 within the visual field range 214 may be transmitted.
  • FIG. 9 focused on transmission processing from the image transmission device 20
  • the display format on the display device 16 side also affects the reduction in delay until display.
  • FIG. 10 is a diagram for explaining the relationship between the shape of the view screen and the amount of provided information.
  • the viewscreen 414 is a screen for presenting central projection images on a flat panel display.
  • a view screen 426 is a screen for representing a distorted image through the eyepiece. The figure shows both screens viewed from the side along with the user's viewpoint 424 .
  • the view screen 414 is composed of a plane having a predetermined angle of view around the axis C, for example.
  • the image of the object 425 is displayed in a uniformly reduced scale according to the distance from the view screen 414 regardless of the height from the axis C.
  • a distorted image premised on an eyepiece lens has the same properties as an image captured by a fisheye lens, and as a result the view screen 426 has a curved shape as shown.
  • the detailed shape of the viewscreen 426 depends on the lens design.
  • the area of the angular range 428 in the vicinity of the optical axis (axis C) of the viewscreen 426 has a small area difference from the corresponding area of the viewscreen 414, whereas the angular range is greater than the optical axis.
  • the image in the central projection image is significantly reduced in the distorted image. be done. In other words, it can be said that part of the image represented by the central projection contains useless information that is not reflected in the distorted image.
  • FIG. 11 is a diagram for explaining distortion image drawing processing in the display control device 10.
  • FIG. The figure shows the passage of time in the horizontal direction, and the upper part of the figure represents the arrangement of the displayed frames.
  • Each frame is represented here as a curved viewscreen of the distorted image, of which the updated portion is shown in bold.
  • the format of data transmitted from the image transmission device 20 is not limited.
  • the display image drawing unit 66 of the display control device 10 draws the entire frame 220a of the display image at the timing when the data of the entire original image is transmitted. At this time, a ray is generated that passes through each pixel on the curved view screen, and the pixel value is determined by acquiring the color information of the arrival point in the original image. This allows the distorted frame 220a to be drawn directly.
  • an image pair composed of left-eye and right-eye images is generated.
  • the display image drawing unit 66 generates frames 222a, 222b, and 222c in which only the parts where the event occurred are updated based on the event information until the data of the entire original image is transmitted next time. Although only the updated portions of the frames 222a, 222b, and 222c are shown in the drawing, in reality, the regions in which no events occur are also represented as images. The direction from the optical center of the camera to the pixel where the event occurred can be determined from the positional information of the pixel where the event occurred and the camera optical system included in the event information.
  • the display image drawing unit 66 performs ray tracing using event information only for necessary parts based on the correspondence between the direction and the ray set at the time of drawing, and determines the pixel value. Again, the strain image can be drawn directly using a curved viewing screen. Next, when the data of the entire original image is transmitted, the display image drawing section 66 generates a corresponding display image frame 220b. By these procedures, low-delay and high-definition image expression can be realized without unnecessary processing and transmission even for distorted images.
  • FIG. 12 shows a modification of the image display system according to this embodiment.
  • a server 252 that generates computer graphics using 3D model data 260 is introduced.
  • the server 252 may be connected to imaging devices 240a and 240b that capture omnidirectional images and wide-angle images, acquire these captured images, and generate images combined with computer graphics.
  • the server 252 may be a server in a cloud environment such as cloud gaming.
  • the server 252 transmits the data 266 of the entire original image at a predetermined rate to the display control devices 10d and 10d of the clients, and also transmits the event information 268a and 268b through the bypass line.
  • the figure shows an equidistant projection image 242 output by a pair of fisheye cameras and a cube map image 270 as an example of the data format of the omnidirectional image.
  • a cube map is a two-dimensional map obtained by projecting an omnidirectional image onto the inner surface of a cube that circumscribes a spherical screen that represents the omnidirectional image, and then developing the image.
  • Various data formats may be used when the server 252 represents the omnidirectional image in this manner.
  • the display control device 10c connected to the flat-panel display as the display device 16c generates and outputs a centrally-projected display image 272.
  • the display control device 10d connected to the head-mounted display as the display device 16d generates and outputs a display image 274 composed of distorted images for the left eye and the right eye.
  • the display control devices 10c and 10d update only the necessary portions 276a and 276b according to the transmission of the event information 268a and 268b, thereby achieving display at a high frame rate.
  • the display control device 10 determines the color information for each pixel by ray tracing using the transmitted image, thereby limiting the type and data format of the transmitted image and the type of the display device 16. It is possible to display with a unified scheme without In addition, it is possible to locally draw images based on event information, generate distorted images without converting to images of central projection, and also make partial areas on the image, such as the area corresponding to the fovea of the eye, Fine control is possible, such as drawing at a higher resolution than other areas.
  • FIG. 13 is a diagram for explaining the relationship between partial update and reprojection based on event information.
  • the display control device 10 performs reprojection to correct the image so as to correspond to the position and posture of the user's head immediately before display.
  • the image transmission device 20 acquires information related to the position and posture of the user's head from the display control device 10 to which data is transmitted, and generates an image in the corresponding field of view.
  • the display control device 10 makes the final adjustment of the field of view according to the movement of the head during the time from generation of the image to display. As a result, the image is displayed in a field of view that follows the movement of the head.
  • the image 280 in the upper part of the figure represents the image world generated by the image transmission device 20.
  • a spherical object 282 is moving against a belt-like object 283 like the ground below.
  • (t0), (t1), and (t2) attached to the object 282 represent the positions of the object 282 at times t0, t1, and t2.
  • the display field of view at each time corresponding to the movement of the user's head is assumed to change to fields of view 284a, 284b, and 284c.
  • the image transmission device 20 predicts the movement of the user's head and transmits image data in a range wider than the field of view 284a as original image data, thereby enabling reprojection. .
  • the display control device 10 performs reprojection based on the transmitted data, and generates and outputs a display image as shown in the lower part. That is, the image data transmitted corresponding to time t0 is used to generate the display frames at times t0 and t1 (S10), and the image data transmitted corresponding to time t2 is used to generate the time t2 and the next display frame. A display frame is generated (S14).
  • S10 the image data transmitted corresponding to time t0
  • the image data transmitted corresponding to time t2 is used to generate the time t2 and the next display frame.
  • a display frame is generated (S14).
  • the image transmission device 20 transmits, as event information, information about pixels whose brightness has changed due to movement of the object 282 at a timing independent of data transmission of the entire original image.
  • the display control device 10 can appropriately draw the image of the object 282 drawn using the latest event information in the display frame at time t1 (S12), as shown in (b).
  • the drawing shows a large change in whether or not the object 282 appears in the display image. is a large value.
  • minute changes can be expressed with high definition and low delay.
  • the display control device 10 refers to the color information included in the event information for the ray corresponding to the pixel where the event occurred in ray tracing. For other rays, the color information of the original image transmitted at time t0 is referred to. On the other hand, the display control device 10 itself may draw the object 282 without including the color information in the event information.
  • the server 252 In the image display system shown in FIG. 12, the server 252 generates virtual world data represented by computer graphics. In this case, the server 252 generates information related to the three-dimensional space to be drawn, such as depth information of the object, during the image drawing process. Therefore, the server 252 transmits the information related to the generated three-dimensional space, so that the movement of the object 282, for example, can be drawn from the model data on the display control device 10 side.
  • the display control device 10 directly draws an object set at a short distance from the user, an object that interacts with the user, and the like, so that a realistic image world can be efficiently expressed.
  • the display control device 10 directly draws an object set at a short distance from the user, an object that interacts with the user, and the like, so that a realistic image world can be efficiently expressed.
  • This aspect is applicable not only to computer graphics, but also to live-action images. That is, the image transmission device 20 separately acquires and transmits the depth information and material information of the subject, so that the display control device 10 can draw only objects with changes using them.
  • depth information and material information may be estimated using deep learning or artificial intelligence, as well as using on-site measurement values or pre-obtained registration information. Technologies for estimating subject material information and depth information from captured images have made rapid progress in recent years. Material Estimation", 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (USA), 2018, Vol. 1, p. 6315-6324).
  • an omnidirectional image captured by a fish-eye camera or a multi-view camera, or an omnidirectional image drawn by computer graphics can be used. Since an omnidirectional image cannot be expressed at once with a general central projection image, a three-dimensional image that contains orientation information, such as an image projected onto a polyhedron that combines planes, an equirectangular view, and an equidistant projection image. A special data format is used to represent .
  • the image transmission device 20 of the present embodiment transmits the original image in a data format other than central projection, and the display control device 10 reads the pixel values directly from the original image using a conversion formula corresponding to the data format. , produces a distorted display image.
  • the original azimuth information of the original image can be transmitted as it is, it is possible to easily generate a display image corresponding to the direction of the user's line of sight. For example, when viewing an omnidirectional image from the same position as the imaging device, since the coordinate system of the image centered on the viewpoint matches, it is sufficient to display the image in the same direction as the user's line of sight. It will be. Furthermore, if the distortion aberration of the lens of the fish-eye camera that captures the omnidirectional image and the eyepiece lens are substantially the same, the transformation for the distorted image can be minimized.
  • FIG. 14 is a diagram for explaining display image generation processing when an omnidirectional image is represented by a cube map.
  • the cube map is an image 292 obtained by projecting the image on the surface of the sphere 294 representing the omnidirectional circumference onto each surface of the cube 290 circumscribing the sphere 294 and developing the image 292 .
  • the image transmitting device 20 sets a ray from the center of the sphere 294 to each pixel of the six squares forming the cube 290 .
  • an image 292 is generated by using the color of the surface of the sphere 294 through which the ray (vector rd) passes as the pixel value of the pixel.
  • the resulting image 292 shows orientation information by means of the six squares and their positions.
  • the image transmission device 20 may transmit only the data of the area required for display in the image 292 . In this case also, the image data format is the same.
  • the display control device 10 acquires the data of the image 292 or part thereof and generates a display image. At this time, the display control device 10 first sets a ray passing through each pixel of the curved screen corresponding to the eyepiece. Then, an image 292 is arranged as a cube circumscribing a sphere 294 centered on the viewpoint, and the pixel value is determined by sampling the color on the image 292 corresponding to the set ray (vector nor). For this processing, for example, a cube map texture reading command in the cube mapping function of OpenGL, which is a well-known computer graphics library, can be used. On the other hand, in this embodiment, event information is also transmitted in the same format.
  • OpenGL which is a well-known computer graphics library
  • the image transmission device 20 may transmit the vector rd of rays reaching the pixel.
  • the display control device 10 generates a new display frame
  • the vector rd transmitted as event information matches the ray vector nor set in generating the display frame
  • the vector nor is included in the event information. partially updates the image by sampling the latest pixel information.
  • FIG. 15 is a diagram for explaining display image generation processing when an omnidirectional image is represented by equirectangular projection.
  • the equirectangular projection image 302 is data obtained by projecting the image on the surface of the sphere 294 representing the whole sky onto the surface of the cylinder 300 that is in contact with the equator and expanding it.
  • the image transmission device 20 first sets a ray that reaches each pixel of the cylinder 300 from the center of the sphere 294 . Then, the image 302 is generated by using the color of the surface of the sphere 294 through which the ray (vector rd) passes as the pixel value of the pixel.
  • the plane of image 302 represents longitude u in the horizontal direction and latitude v in the vertical direction.
  • the display control device 10 acquires the data of the image 302 or part thereof and generates a display image. At this time, the display control device 10 first sets a ray passing through each pixel of the curved screen corresponding to the eyepiece. Then, the image 302 is arranged as a cylinder circumscribing the equator of the sphere 294 centered on the viewpoint, and the pixel value is determined by sampling the color on the image 302 corresponding to the set ray (vector nor). The image transmission device 20 further transmits event information indicating the position of the pixel where the event occurred by position coordinates on the plane of the image 302 . The process of partially updating the image by the display control device 10 in response to this may be the same as described with reference to FIG. 14 .
  • FIG. 16 is a diagram for explaining display image generation processing when an omnidirectional image is represented by equidistant projection.
  • an imaging device is assumed in which fish-eye cameras having an angle of view of 220° are combined on the front and back. That is, the photographed image is data on the surface of the portion of the sphere 306 centered at the optical center, excluding the range outside 220°. Note that the drawing shows only the sphere 306 that constitutes the captured image of one of the pair of fisheye cameras. Also, the angle of view of each fisheye camera is not limited as long as it is 180° or more.
  • equidistant projection images 310a and 310b are obtained by representing the output of the sensor array as it is. However, it is not limited to the photographed image, and can be generated by drawing an image of equidistant projection by computer graphics.
  • the image transmission device 20 generates an image 308 consisting of equidistant projection images 310a and 310b representing the space in the opposite direction.
  • the plane of the image 308 represents longitude u in the horizontal direction and latitude v in the vertical direction.
  • the image transmitting device 20 sets rays at equal intervals from the center of the sphere 294 (306), and the color of the surface of the sphere 294 (306) through which the vector rd passes is displayed in pixels.
  • An image 308 can be generated by taking values.
  • the display control device 10 acquires the data of the image 308 or part thereof and generates a display image. At this time, the display control device 10 first sets a ray passing through each pixel of the curved screen corresponding to the eyepiece. Then, the pixel value is determined by sampling the color at the position corresponding to the set ray (vector nor) in the image 308 . The image transmission device 20 further transmits event information representing the position of the pixel where the event occurred by position coordinates on the plane of the image 308 . The process of partially updating the image by the display control device 10 in response to this may be the same as described with reference to FIG. 14 .
  • FIG. 17 is a diagram for explaining a mode in which an omnidirectional image is represented by equidistant projection and a plurality of resolution levels are provided.
  • the image to be used may be an image captured by a device in which fisheye cameras having an angle of view of 180° or more are combined on the front and back, as in FIG. 16, or may be an image rendered by computer graphics in a similar format. Therefore, the original image becomes data on the surface of the sphere 306 excluding the range outside the angle of view.
  • an image represented by a curved screen such as a fisheye lens is represented by compressing the image as the distance from the center of the image increases, compared to the center projection image.
  • This feature is compatible with the human visual characteristic that visual performance decreases with increasing distance from the region corresponding to the fovea. Developing this, it is considered that the image representing the space outside the user's field of view is less likely to be immediately captured by the fovea, so even if it is prepared with a low resolution, the visual impact will be small.
  • each image 314a, 314b has a mipmap structure representing a plurality of resolution levels. For example, when the user is looking in the direction represented by image 314b, most of the area of image 314a is outside the field of view. Therefore, the image transmission device 20 preferentially transmits the image 314a with a low resolution as necessary.
  • the overall transmission data size is reduced and low delay is guaranteed compared to the two images 314a and 314b having the same resolution.
  • the image world can be visually recognized without discomfort by increasing the resolution of the image 314a gradually as the user's line of sight moves closer to the user's line of sight.
  • the process of generating the display image in the display control device 10 and the process of partially updating the image based on the event information may be the same as those described above.
  • the omnidirectional image is represented by two images of equidistant projection, but the number of images of equidistant projection may be three or more depending on the angle of view.
  • their resolutions may be controlled independently.
  • FIG. 18 is a diagram for explaining a method of generating an equidistant projection image having a distribution in which the resolution decreases from the center to the periphery.
  • the equidistant projection image 318 is represented by two-dimensional coordinates of longitude u and latitude v, as shown in (a).
  • two fish-eye cameras are used as shown in FIGS. 16 and 17, two similar images are generated, but the object to which the resolution distribution is given may be both of them, or the user is mainly looking at it. An image only is also acceptable.
  • the image transmission device 20 sets a ray that reaches each pixel on the plane of the image 318 from the optical center of the camera, and sets the color of the surface of the sphere through which the vector rd passes as the pixel value of that pixel.
  • the area represented by one pixel is increased by decreasing the ray density as the distance from the center of the image increases.
  • the transformation from the position coordinates (u, v) to the vector rd on the plane of the image 318 with the angle of view of 220° is realized, for example, by the following calculation.
  • the function f(th) is such that the larger the absolute value of the angle th corresponding to the distance from the center of the image to the position coordinates (u, v), as indicated by the broken line 320a in FIG. is a monotonically increasing function.
  • the greater the distance from the center of the image the lower the density of the rays represented by the vector rd.
  • the image 318 becomes data in which the area represented by one pixel varies depending on the position on the image plane.
  • the display control device 10 performs sampling after converting the set ray vector nor into position coordinates (u, v) on the image 318 by, for example, the following calculation.
  • the function f(th) is the inverse function of the dashed line 320a used to generate the image 318, as indicated by the solid line 320b in (b) of the figure.
  • data representing different areas represented by one pixel can be represented as data of pixels arranged evenly on the display screen.
  • reading corresponding to the mipmap structure is performed by the "LODBIAS" parameter, but the gist of the present embodiment is not limited to that.
  • FIG. 19 is a diagram for explaining display image generation processing when an omnidirectional image is represented on the plane of a regular octahedron.
  • an image 324 is generated by projecting the image on the surface of the sphere 294 representing the omnidirectional circumference onto the surface of the regular octahedron 322 circumscribing it. That is, the image transmission device 20 sets a ray that reaches each pixel of the regular octahedron 322 from the center of the sphere 294, and converts the color of the surface of the sphere 294 through which the ray (vector rd) passes to the pixel value of the pixel.
  • An image 324 is generated by .
  • the display control device 10 acquires the data of the image 324 or part thereof and generates a display image. At this time, the display control device 10 first sets a ray passing through each pixel of the curved screen corresponding to the eyepiece. Then, the image 324 is arranged as a regular octahedron circumscribing the sphere 294 centered on the viewpoint, and the pixel value is determined by sampling the color on the image 324 corresponding to the set ray (vector nor). The image transmission device 20 further transmits event information indicating the position of the pixel where the event occurred by position coordinates on the plane of the image 324 . The process of partially updating the image by the display control device 10 in response to this may be the same as described with reference to FIG. 14 . Various methods other than those described above have been proposed for expressing an omnidirectional image with two-dimensional data, and any of them may be adopted in the present embodiment.
  • the image transmission device detects a change in brightness as an event on a pixel-by-pixel basis. , is transmitted asynchronously with the data of the entire original image.
  • the display control device that has acquired the data generates the entire area of the display frame using the data of the entire original image, and interpolates the display frame in which only the changed portion is updated pixel by pixel based on the event information.
  • the display control device enables updating for each pixel by using a ray tracing method of setting a ray corresponding to each pixel on the display image and acquiring color information.
  • a similar effect can also be obtained with a flat panel display by updating only the part where the event occurred within the gaze area.
  • the present embodiment can be easily applied regardless of the display format, such as a flat panel display or a head-mounted display, and regardless of the type of display image, such as a photographed image, computer graphics, or a composite image thereof. .
  • Display control device 16 Display device, 20 Image transmission device, 22 CPU, 24 GPU, 26 Main memory, 50 Gaze area acquisition unit, 52 Image data acquisition unit, 54 Event information generation unit, 56 Data transmission unit, 60 Gaze area Transmission unit 62 Image data acquisition unit 64 Event information acquisition unit 66 Display image drawing unit 70 Frame image drawing unit 72 Event information management unit 74 Output unit.
  • the present invention can be used for various information processing devices such as content servers, game devices, head-mounted displays, display devices, mobile terminals, personal computers, and image display systems including any of them.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Hardware Design (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

Un dispositif de transmission d'images (20a ou 20b) transmet, à un débit prédéterminé, les données (166) d'une image originale entière capturée par un dispositif d'imagerie (140 ou 141) à un dispositif de commande d'affichage (10a ou 10b), obtient un signal pour un événement de changement de luminosité détecté par le dispositif d'imagerie (140 ou 141) pour chaque pixel afin de générer des informations d'événement (168), et transmet de manière asynchrone les informations d'événement (168). Le dispositif de commande d'affichage (10a ou 10b) génère une image d'affichage par traçage de rayons à une fréquence d'images supérieure à celle de l'image originale transmise et met sélectivement à jour l'image correspondante en unités de pixels en utilisant les dernières informations d'événement.
PCT/JP2021/040577 2021-11-04 2021-11-04 Système d'affichage d'image, dispositif de transmission d'image, dispositif de commande d'affichage et procédé d'affichage d'image WO2023079623A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/040577 WO2023079623A1 (fr) 2021-11-04 2021-11-04 Système d'affichage d'image, dispositif de transmission d'image, dispositif de commande d'affichage et procédé d'affichage d'image

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/040577 WO2023079623A1 (fr) 2021-11-04 2021-11-04 Système d'affichage d'image, dispositif de transmission d'image, dispositif de commande d'affichage et procédé d'affichage d'image

Publications (1)

Publication Number Publication Date
WO2023079623A1 true WO2023079623A1 (fr) 2023-05-11

Family

ID=86240775

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/040577 WO2023079623A1 (fr) 2021-11-04 2021-11-04 Système d'affichage d'image, dispositif de transmission d'image, dispositif de commande d'affichage et procédé d'affichage d'image

Country Status (1)

Country Link
WO (1) WO2023079623A1 (fr)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010233185A (ja) * 2009-03-30 2010-10-14 Saxa Inc 画像処理装置、画像処理システム及びプログラム
JP2011087257A (ja) * 2009-10-19 2011-04-28 Panasonic Corp 半導体集積回路及び撮像装置
JP2014155168A (ja) * 2013-02-13 2014-08-25 Mitsubishi Electric Corp 画像合成装置及び画像合成方法
US20180098082A1 (en) * 2016-09-30 2018-04-05 Intel Corporation Motion estimation using hybrid video imaging system
WO2019003889A1 (fr) * 2017-06-26 2019-01-03 ソニー株式会社 Dispositif de génération et procédé de génération, ainsi que dispositif de lecture et procédé de lecture
JP2019134202A (ja) * 2018-01-29 2019-08-08 ソニーセミコンダクタソリューションズ株式会社 撮像装置および撮像素子の制御方法
JP2021057767A (ja) * 2019-09-30 2021-04-08 株式会社ソニー・インタラクティブエンタテインメント 画像データ転送装置および画像データ転送方法

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010233185A (ja) * 2009-03-30 2010-10-14 Saxa Inc 画像処理装置、画像処理システム及びプログラム
JP2011087257A (ja) * 2009-10-19 2011-04-28 Panasonic Corp 半導体集積回路及び撮像装置
JP2014155168A (ja) * 2013-02-13 2014-08-25 Mitsubishi Electric Corp 画像合成装置及び画像合成方法
US20180098082A1 (en) * 2016-09-30 2018-04-05 Intel Corporation Motion estimation using hybrid video imaging system
WO2019003889A1 (fr) * 2017-06-26 2019-01-03 ソニー株式会社 Dispositif de génération et procédé de génération, ainsi que dispositif de lecture et procédé de lecture
JP2019134202A (ja) * 2018-01-29 2019-08-08 ソニーセミコンダクタソリューションズ株式会社 撮像装置および撮像素子の制御方法
JP2021057767A (ja) * 2019-09-30 2021-04-08 株式会社ソニー・インタラクティブエンタテインメント 画像データ転送装置および画像データ転送方法

Similar Documents

Publication Publication Date Title
US11575876B2 (en) Stereo viewing
JP6632443B2 (ja) 情報処理装置、情報処理システム、および情報処理方法
JP7358448B2 (ja) 画像生成装置、ヘッドマウントディスプレイ、および画像生成方法
JP7150134B2 (ja) ヘッドマウントディスプレイおよび画像表示方法
WO2017086244A1 (fr) Dispositif de traitement d'image, dispositif de traitement d'informations et procédé de traitement d'image
JP7142762B2 (ja) 表示装置および画像表示方法
US11187895B2 (en) Content generation apparatus and method
JP6965439B2 (ja) 基準画像生成装置、表示画像生成装置、基準画像生成方法、および表示画像生成方法
US20240119557A1 (en) Image display system and image display method
WO2023079623A1 (fr) Système d'affichage d'image, dispositif de transmission d'image, dispositif de commande d'affichage et procédé d'affichage d'image
WO2019193698A1 (fr) Dispositif de génération d'images de référence, dispositif de génération d'images d'affichage, procédé de génération d'images de référence, et procédé de génération d'images d'affichage
WO2019193699A1 (fr) Dispositif de génération d'images de référence, système de génération d'images d'affichage, procédé de génération d'images de référence et procédé de génération d'images d'affichage
US11688124B2 (en) Methods and apparatus rendering images using point clouds representing one or more objects
WO2023136073A1 (fr) Dispositif d'affichage d'image et procédé d'affichage d'image
WO2022230253A1 (fr) Dispositif de traitement d'informations et procédé de traitement d'informations
JP2020167658A (ja) 画像生成装置、ヘッドマウントディスプレイ、コンテンツ処理システム、および画像表示方法
KR20180117867A (ko) 가상현실 콘텐츠 서비스를 위한 360도 vr 어안 렌더링 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21963220

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2023557492

Country of ref document: JP

Kind code of ref document: A