WO2023079623A1 - Image display system, image transmission device, display control device, and image display method - Google Patents

Abstract

An image transmission device 20a or 20b transmits, at a predetermined rate, data 166 of an entire original image captured by an imaging device 140 or 141 to a display control device 10a or 10b, obtains a signal for a brightness change event detected by the imaging device 140 or 141 for each pixel to generate event information 168, and asynchronously transmits the event information 168. The display control device 10a or 10b generates a display image by ray tracing at a frame rate higher than the frame rate of the transmitted original image and selectively updates the corresponding image in units of pixels by using the latest event information.

Description

Image display system, image transmission device, display control device, and image display method

The present invention relates to image display technology for displaying moving images using transmitted original images.

Display devices for UHDTV (Ultra HDTV), which have a resolution 4 times and 16 times that of HDTV (High Definition Television), have been put into practical use. In addition, a display system that gives a sense of immersion in the image world by showing an omnidirectional image in a free field of view corresponding to the line of sight of the user wearing the head-mounted display is also widespread. With the advent of 5G (fifth generation mobile communication system), wideband, low-delay data transmission has become possible, and it has become possible to enjoy high-quality image content regardless of the environment.

As described above, in various electronic contents, the resolution and viewing angle of images, as well as the number of pixels that make up one frame, tend to increase more and more. However, such an increase in the amount of data causes various problems such as a tight transmission band, an increase in calculation cost, and an increase in power consumption. In particular, because it takes time to process and transmit data for one frame, the frame rate does not increase and the image quality deteriorates. .

The present invention has been made in view of these problems, and its purpose is to provide a technology that can easily display images with high resolution and wide viewing angles with high quality.

In order to solve the above problems, one aspect of the present invention relates to an image display system. This image display system is an image display system comprising an image transmission device for transmitting original image data and a display control device for controlling image display using the original image data, wherein the image transmission device comprises: When a luminance change event occurs in any pixel in the original image, an event information generation unit that generates event information including position information of the pixel, and transmits data of the entire original image at a predetermined frame rate, and a data transmission unit that asynchronously transmits event information, and the display control device uses the data of the entire original image to generate display image data at a predetermined frame rate, and based on the latest event information, The present invention is characterized by comprising a display image drawing section for locally updating a display image, and an output section for outputting data of the display image to a display device.

Another aspect of the present invention relates to an image transmission device. This image transmission device is an image transmission device that transmits data of an original image and realizes an image display using the original image at a transmission destination. At this time, an event information generation unit that generates event information including the position information of the pixel, data of the entire original image are transmitted at a predetermined frame rate, and the event information is transmitted asynchronously, thereby generating a display image at the transmission destination. and a data transmission unit that locally updates the .

Yet another aspect of the present invention relates to a display control device. This display control device is a display control device that receives data of an original image and controls image display using the data of the original image. and an event information acquisition unit that acquires event information including position information of a pixel transmitted asynchronously with data of the entire original image in response to a luminance change event occurring in any pixel in the original image. a display image drawing unit that generates display image data at a predetermined frame rate using data of the entire original image and locally updates the display image based on recent event information; and an output unit for outputting to a display device.

Yet another aspect of the present invention relates to an image display method. This image display method comprises an image display system comprising an image transmission device for transmitting original image data and a display control device for controlling image display using the original image data. generating event information including position information of the pixel when a luminance change event occurs in one of the pixels; transmitting data of the entire original image at a predetermined frame rate; and the display control device generates display image data at a predetermined frame rate using the data of the entire original image, and locally updates the display image based on recent event information. and outputting data of the display image to a display device.

It should be noted that any combination of the above constituent elements, and any conversion of the expression of the present invention between methods, devices, systems, computer programs, data structures, recording media, etc. are also effective as embodiments of the present invention.

According to the present invention, images with high resolution and wide viewing angles can be easily displayed with high quality.

It is a figure which shows the structural example of the image display system which can apply this Embodiment. 4A and 4B are diagrams showing examples of images transmitted and displayed in the present embodiment; FIG. It is a figure which illustrates the concrete structure of the image display system in this Embodiment. 1 is a diagram showing an internal circuit configuration of a display control device according to this embodiment; FIG. 1 is a diagram showing functional block configurations of an image transmission device and a display control device according to the present embodiment; FIG. FIG. 10 is a diagram for explaining a problem related to the balance between the amount of provided information and the amount of received information when a flat-panel display is used as the display destination; FIG. 3 is a diagram for explaining the relationship between data transmitted by an image transmission device and a display image generated by a display control device in the embodiment; FIG. 7 is a diagram showing another example of the relationship between data transmitted by the image transmission device and a display image generated by the display control device in the embodiment; 4 is a diagram illustrating data formats of an omnidirectional image and event information prepared by the image transmission device in the present embodiment; FIG. FIG. 5 is a diagram for explaining the relationship between the shape of the view screen and the amount of provided information; FIG. 4 is a diagram for explaining distortion image drawing processing in the display control device according to the embodiment; It is a figure which shows the modification of the image display system in this Embodiment. FIG. 10 is a diagram for explaining the relationship between partial update and reprojection based on event information in the embodiment; FIG. 7 is a diagram for explaining display image generation processing when an omnidirectional image is represented by a cube map in the present embodiment; FIG. 10 is a diagram for explaining display image generation processing when an omnidirectional image is represented by equirectangular projection in the present embodiment; FIG. 10 is a diagram for explaining display image generation processing when an omnidirectional image is represented by equidistant projection in the present embodiment; FIG. 4 is a diagram for explaining a mode in which an omnidirectional image is represented by equidistant projection and a plurality of resolution levels are provided in the present embodiment; FIG. 10 is a diagram for explaining a method of generating an equidistant projection image having a distribution in which the resolution decreases from the center to the periphery in the present embodiment; FIG. 10 is a diagram for explaining display image generation processing when an omnidirectional image is represented on the plane of a regular octahedron in the present embodiment;

FIG. 1 shows a configuration example of an image display system to which this embodiment can be applied. The image display system 1 includes display control devices 10a, 10b, and 10c that display images according to user operations, and an image transmission device 20 that provides image data used for display. Input devices 14a, 14b, and 14c for user operations and display devices 16a, 16b, and 16c for displaying images are connected to the display control devices 10a, 10b, and 10c, respectively. The display control devices 10a, 10b, 10c and the image transmission device 20 can establish communication via a network 8 such as a WAN (World Wide Network) or a LAN (Local Area Network).

The display control devices 10a, 10b, 10c, the display devices 16a, 16b, 16c, and the input devices 14a, 14b, 14c may be connected either by wire or wirelessly. Alternatively, two or more of those devices may be integrally formed. For example, in the figure, the display control device 10b is connected to a head-mounted display, which is the display device 16b. Since the head-mounted display can change the field of view of the displayed image according to the movement of the user wearing it on the head, it also functions as the input device 14b.

The display control device 10c is a mobile terminal, and is integrally configured with a display device 16c and an input device 14c, which is a touch pad covering the screen. Thus, the external shape and connection form of the illustrated device are not limited. The number of display control devices 10a, 10b, 10c and image transmission devices 20 connected to the network 8 is not limited either. For example, the image transmission device 20 may be a cloud server or content server in a system such as cloud gaming. Hereinafter, the display control devices 10a, 10b, and 10c may be collectively referred to as the display control device 10, the input devices 14a, 14b, and 14c as the input device 14, and the display devices 16a, 16b, and 16c as the display device 16 in some cases.

The input device 14 may be one or a combination of general input devices such as a controller, keyboard, mouse, touch pad, joystick, etc., and supplies the contents of user operations to the display control device 10 . The display device 16 may be a general display such as a liquid crystal display, a plasma display, an organic EL display, a wearable display, or a projector, and displays images output from the display control device 10 .

The image transmission device 20 provides the display control device 10 with content data accompanied by image display. The type of content is not particularly limited, and may be electronic games, images for viewing, television broadcasts, electronic conferences, video chats, or the like. The display control device 10 basically acquires content data used for display from the image transmission device 20 and implements display processing. Alternatively, the image transmission device 20 and the display control device 10 may cooperate to generate content data, thereby distributing the load of drawing processing and the like.

FIG. 2 shows an example of an image transmitted and displayed in this embodiment. (a) is an example of displaying the data of the captured image 130 as it is as the display image 132 in a live broadcast, video chat, or the like. In this case, the source of the image data may be an imaging device connected to the display control device 10 as well as the image transmission device 20 . The display device 16 is a flat display.

(b) is an example of preparing an omnidirectional image 134 and generating a display image 136 in a field of view corresponding to movement of the head of a user wearing a head-mounted display. In the case of a head-mounted display, stereoscopic vision can be realized by displaying stereoscopic images with parallax on the left and right regions of the display panel corresponding to the left and right eyes of the user, as in the illustrated display image 136 . Further, in the case of a head-mounted display provided with an eyepiece lens for expanding the field of view, the displayed image 136 has distortion and chromatic aberration corresponding to the eyepiece lens so that a distortion-free image can be viewed through the eyepiece lens. Give the opposite distortion.

For example, in the case of a lens in which the four sides of the image appear to be pincushion-shaped, the left and right images of the display image 136 are each curved in a barrel shape. Hereinafter, an image to which distortion corresponding to the eyepiece is applied is called a "distorted image". The omnidirectional image 134 may be an image rendered by computer graphics, a photographed image, or a combination thereof. In addition to the image transmission device 20, the image data transmission source may be a drawing mechanism inside the display control device 10, an imaging device connected to the display control device 10, or the like.

In the field of flat panel displays shown in (a), it has become possible to realize high-definition image expression with a high number of pixels like UHDTV. Also, in the field of head-mounted displays, by preparing an image with a wide viewing angle like the omnidirectional image 134 shown in (b), it becomes possible to look around the image world from a free viewpoint, and a sense of immersion can be achieved. Increased. In either case, the number of pixels forming one frame tends to increase. When the amount of data for one frame increases in this way, the time required for processing and transmission increases, making compatibility with a high frame rate difficult.

In addition, in the mode of displaying images shot and drawn in real time, it is possible that the delay time until display cannot be overlooked. For example, in electronic games that generate images according to user operations, the delay time greatly affects the operability of the game. Furthermore, since the power consumption for data processing and transmission increases, especially in the case of a head-mounted display, a large-capacity storage battery may be required, and the feeling of wearing may deteriorate due to heat generation.

Therefore, in the present embodiment, resources related to processing and transmission are saved by capturing images in units of pixels and transmitting information. Specifically, the data transmission source acquires the occurrence of the change event in units of pixels, and transmits only the information related to the local change event through the bypass line independently of the data of the entire image. The display control device 10 locally updates the portion of the display image where the change event has occurred based on the most recently transmitted information.

Here, a "change event" refers to a state in which a difference equal to or greater than a threshold occurs in observed luminance or pixel values (hereafter simply referred to as an "event"). A data transmission source such as the image transmission device 20 transmits data such as the captured image 130, the omnidirectional image 134, or a portion thereof at a predetermined rate, and at the time when a change event occurs, the pixel position and Send event information, including the content of the change, by bypass. The display controller 10 generates frames of display images at a predetermined rate and locally updates the frames based on the transmitted event information.

Here, the display control device 10 draws the display image using ray tracing technology. Ray tracing is a method of generating a virtual ray that passes through each pixel on the view screen from the viewpoint and determining the pixel value based on the color information of the destination. In the present embodiment, an image transmitted from the image transmission device 20 or the like is placed in front of the view screen, and the destination of the ray is obtained to acquire color information on the image. By adopting this method, it becomes possible to selectively draw only pixels where an event has occurred.

Ray tracing is generally regarded as a process with a large amount of computation and a heavy load. However, compared to rasterization that projects polygons, there is an advantage that most intermediate buffer memories such as depth buffers are unnecessary. In addition, depending on the content of the image, it is possible to operate with light weight and low latency in terms of calculation amount and resources used. Furthermore, since the amount of processing is proportional to the drawing area, by limiting the update targets as described above, the efficiency of drawing can be improved while maintaining the quality. In addition, by enabling the transmission and rendering of information in pixel units, it is possible to minimize the transmission of data such as the background that does not contribute to the improvement of image quality, and reduce the processing and transmission load at the data transmission source.

FIG. 3 illustrates a specific configuration of the image display system according to this embodiment. As described above, in the present embodiment, the image data that is the source of the display image is not limited to be generated by the image transmission device 20, and at least part of it is generated by a local device on the display side such as the display control device 10 or an imaging device. Although it may play a role, hereinafter, the image transmission device 20 will be mainly described. Further, the entity that forms the final display image from the image data using the ray tracing technique is not limited to the display control device 10, and at least a part thereof may be performed by the display device 16. The control device 10 will be described. Hereinafter, the image transmitted by the image transmission device 20 will be referred to as an "original image" to be distinguished from the "display image" generated by the display control device 10. FIG.

The figure shows a state in which two image transmission devices 20a and 20b and two display control devices 10a and 10b are connected via an arbitrary network such as LAN, WiFi (registered trademark), or 5G. Of these, the image transmission device 20a transmits the moving image of the speaker 160 captured by the imaging device 140 to the display control devices 10a and 10b, as shown in FIG. 2(a). On the other hand, as shown in FIG. 2B, the image transmission device 20b captures an omnidirectional image 142 with the imaging device 141, and transmits at least part of it to the display control devices 10a and 10b.

In the drawing, the omnidirectional image 142 is shown as a cross section of a spherical screen representing the image plane, but the representation method (data format) of the omnidirectional image 142 is not particularly limited as described later. In addition, the imaging device 141 may be configured by a multi-lens camera in addition to a combination of fish-eye cameras on the front and back. Furthermore, as will be described later, the image transmission devices 20a and 20b may render images equivalent to the images captured by the imaging devices 140 and 141 using computer graphics, or may render computer graphics on the captured images.

The display control device 10a uses the data of the original image 148 transmitted from the image transmission devices 20a and 20b to display the display image 150 on the flat panel display, which is the display device 16. FIG. On the other hand, the display control device 10b uses the data of the original image 152 transmitted from the image transmission devices 20a and 20b to display the display image 154 on the head-mounted display, which is the display device 16. FIG.

Although the diagram shows the data of the original images 148 and 152 as part of a spherical screen representing the image plane, the data format is not limited to this. For example, when the central projection image captured by the imaging device 140 is used as the original image 148, the screen is of a flat plate type. Since the image displayed on the head-mounted display is a distorted image as described above, the screen (view screen) of the display image 154 is shown as a substantially spherical surface.

When an image with distortion such as an omnidirectional image is used as the original image 152, in the present embodiment, a display image 154 with distortion is directly generated from the original image 152. Specifically, as shown, the original image 152 is placed in front of the viewscreen in a curved screen shape corresponding to its distortion. The original image 152 is then sampled from the curved viewscreen by ray tracing to generate the displayed image 154 . As a result, there is no need to generate a central projection image once, and memory for developing the image and waiting time for developing can be saved.

From the image transmission devices 20a and 20b, data 166 of the entire original image is transmitted at a predetermined rate, as indicated by the thick line in the drawing. Here, the "whole original image" is the entire area of the frame of the original image, for example, the entire area of the image within the field of view. However, the “frame” to be transmitted is not limited to the image within the field of view, and may include a range wider than the field of view used by the display control devices 10a and 10b to form the display image. Each time an event occurs, the image transmitting devices 20a and 20b also transmit event information 168, including the position of the pixel where the event occurred and the content of the change, via a bypass, as indicated by the dashed lines in the drawing.

The display control devices 10a and 10b basically use the data 166 of the entire original image transmitted from the image transmission devices 20a and 20b at a predetermined rate to generate the entire display image and cause the display device 16 to display it. At this time, the display control devices 10a and 10b increase the display frame rate by interpolating frames of the original image, or adjust the field of view according to the latest position and posture of the head of the user 162b viewing the head-mounted display. may be corrected. Then, the display control devices 10a and 10b use the event information 168 asynchronously transmitted from the image transmission devices 20a and 20b to selectively update necessary portions in the display image.

The display control devices 10a, 10b also transmit data 170a, 170b relating to the position or area on the screen that the user 162a, 162b looking at the display image is gazing to the image transmission device 20a, 20b at a predetermined rate. You may In this case, the image transmission devices 20a and 20b exclude, from among the events that have occurred in the original images, information on events outside the region that the users 162a and 162b are watching from the transmission targets. This makes it possible to further limit the update targets in transmission data and display images without notifying the users 162a and 162b.

4 shows the internal circuit configuration of the display control device 10. FIG. The display control device 10 includes a CPU (Central Processing Unit) 22 , a GPU (Graphics Processing Unit) 24 and a main memory 26 . These units are interconnected via a bus 30 . An input/output interface 28 is also connected to the bus 30 . The input/output interface 28 outputs data to a peripheral device interface such as USB and IEEE 1394, a communication unit 32 including a wired or wireless LAN network interface, a storage unit 34 such as a hard disk drive and a nonvolatile memory, and a display device 16. An output unit 36, an input unit 38 for inputting data from the input device 14, an imaging device, etc., and a recording medium driving unit 40 for driving a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory are connected.

The CPU 22 controls the entire display control device 10 by executing the operating system stored in the storage unit 34 . The CPU 22 also executes various programs read from a removable recording medium and loaded into the main memory 26 or downloaded via the communication section 32 . The GPU 24 performs drawing processing according to a drawing command from the CPU 22, and stores the display image in a frame buffer (not shown). Then, the display image stored in the frame buffer is converted into a video signal and output to the output section 36 . The main memory 26 is composed of a RAM (Random Access Memory) and stores programs and data necessary for processing. The circuit configuration of the image transmission device 20 may also be substantially the same, and in this case, the acquired or generated image data is transmitted to the display control device 10 via the communication unit 32 .

FIG. 5 shows the configuration of functional blocks of the image transmission device 20 and the display control device 10 in this embodiment. Note that the image transmission device 20 and the display control device 10 may perform general information processing such as progressing an electronic game or outputting sound. is shown.

In terms of hardware, the functional blocks shown in FIG. 5 can be realized by the configuration of the CPU 22, GPU 24, main memory 26, etc. shown in FIG. It is realized by a program that exhibits various functions such as a data input function, a data holding function, an image processing function, and a communication function. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and are not limited to either one.

The image transmission device 20 includes a gaze area acquisition unit 50 that acquires information related to the user's gaze area, an image data acquisition unit 52 that acquires data of the original image, and data for transmitting the data of the original image to the display control device 10. A transmitter 56 is provided. The gaze area acquisition unit 50 acquires information indicating the position coordinates of the gaze point of the user on the screen of the display device 16 or the area of a predetermined range including the gaze point from the display control device 10 at a predetermined rate.

The image data acquisition unit 52 acquires data of the entire original image at a predetermined rate. The data may be generated by the image data acquisition unit 52 itself, or may be acquired from an external device (not shown). For example, the image data acquisition unit 52 acquires captured images at a predetermined rate from an imaging device connected to the image transmission device 20 . Alternatively, the image data acquisition unit 52 draws the original image at a predetermined rate by computer graphics based on the model data and game program stored therein. In this case, the image data acquisition unit 52 may acquire the content of the user's operation on the game or the like from the display control device 10 .

When the display device 16 is a head-mounted display and the field of view is changed according to the movement of the user's head, the image data acquisition unit 52 also acquires the information related to the position and orientation of the head at a predetermined rate. Get from In this case, the head-mounted display is equipped with at least one of a motion sensor and a camera that captures images of the surrounding space. The display control device 10 obtains the position and orientation of the head-mounted display and, in turn, the user's head, based on the measurement values of the motion sensor and the images captured by the camera, and transmits them to the image transmission device 20 at a predetermined rate. Various techniques have been put into practical use as techniques for obtaining the position and orientation of the head-mounted display, and any of them may be adopted in the present embodiment.

As a result, the image data acquisition unit 52 preferentially draws the original image of the area within the user's field of view, or the area expected to be within the user's field of view, among the omnidirectional images representing the virtual world, for example. When drawing an image by itself, the image data acquisition unit 52 can be regarded as an “original image drawing unit”. Note that the image data acquisition unit 52 may draw computer graphics on the captured image.

The image data acquisition unit 52 also includes an event information generation unit 54 that detects the occurrence of an event and generates event information to be transmitted. When the image to be displayed is a captured image, the event information generation unit 54 may acquire event information from the imaging device. There is known an event-driven sensor (EDS) that generates a signal indicating information when there is a change in the intensity of light incident on the imaging device (for example, International Publication No. 2021/033251, International See Publication No. 2021/033252).

EDS, also called neuromorphic camera, silicon retina, or dynamic vision sensor, is an image sensor that responds to local change events in brightness. Specifically, when a change occurs in the intensity of incident light in any sensor of the sensor array, it outputs a signal indicating the position, time, and whether it increases or decreases. Since EDS locally responds only to pixels that have occurred at the time when an event occurs, it has a light load, low latency, and low latency compared to an image processing approach that extracts differences while scanning the entire image. Output with power consumption is possible. In other words, information can be output with higher temporal resolution than processing the entire image.

Therefore, the event information generation unit 54 acquires an event signal from the EDS provided in the imaging device and immediately generates event information. For example, information is generated that associates the position of a pixel where the intensity of incident light has changed by a threshold value or more, the occurrence time, and the details of the change. Here, the position of a pixel may be represented by the direction of a ray passing through each pixel of the original image from the virtual viewpoint or the center of the optical axis of the camera.

Also, the content of the change is not particularly limited as long as it is information that allows the display control device 10 to update the display image based on it. For example, it may be a pixel value difference or a pixel value after change. Alternatively, as will be described later, the display control device 10 itself can draw the object by including information used to draw the object represented by the pixel where the event occurred, such as depth information and model data, in the event information. You may do so.

Note that the imaging device may have a configuration in which an image sensor for capturing a general color image and an EDS are separately provided, or the device may output a color image by accumulating event data for a predetermined period of time. good too. In the former case, the positions of the two sensor arrays are associated with each other, and clocks for generating the imaging time and the event occurrence time are synchronized. In the latter case, it is expected that the frame rate of color images will be reduced and the compression rate will be improved, making it easier to transmit depth information and model data of the object itself in which an event has occurred.

On the other hand, when the image data acquisition unit 52 draws the original image by itself, the event information generation unit 54 generates event information according to a program for the image data acquisition unit 52 to draw the original image. That is, the image data acquisition unit 52 determines the movement of the object in the image world before drawing the image by ray tracing or the like. Therefore, when drawing an image, it is possible to specify by itself at which pixel an event occurs. By acquiring the information, the event information generator 54 generates event information in the same way as when using the event signal from the EDS. When synthesizing a captured image with computer graphics, the event information generating section 54 may generate event information based on both an event signal from the EDS and event information found when the image is drawn.

The event information generation unit 54 may further select events to be transmitted as event information based on information related to the gaze point and the gaze area acquired by the gaze area acquisition unit 50 . For example, the event information generation unit 54 excludes events that have occurred outside the attention area from transmission targets by masking them. The data transmission unit 56 transmits the data of the entire original image to the display control device 10 at a predetermined rate, and immediately transmits the event information when the event information generation unit 54 generates the event information.

The display control device 10 includes a gaze area transmission unit 60 that transmits information related to the user's gaze area, an image data acquisition unit 62 that acquires data of the entire original image from the image transmission device 20, and acquires event information from the image transmission device 20. , an event information acquisition unit 64 for processing, a display image drawing unit 66 for drawing a display image, and an output unit 74 for outputting display image data.

The gaze region transmission unit 60 acquires from the display device 16 the positional coordinates of the user's gaze point on the screen of the display device 16 or the information related to the region of a predetermined range including the gaze point, and transmits the information to the image transmission device 20 at a predetermined rate. Send. For this reason, the display device 16 is provided with a gaze point detector (not shown). A point-of-regard detector is a general device that detects a point of an object that a person is gazing at by, for example, photographing the reflection of infrared rays irradiated to the eyeball.

The image data acquisition unit 62 acquires data of the entire original image from the image transmission device 20 at a predetermined rate. The event information acquisition unit 64 acquires event information from the image transmission device 20 each time. The display image drawing section 66 includes a frame image drawing section 70 for drawing frames of the display image, and an event information management section 72 for changing the processing of the frame image drawing section 70 according to the occurrence of an event. The frame image rendering unit 70 uses the data of the entire original image acquired by the image data acquiring unit 62 to render the frames of the display image at a predetermined rate using a technique similar to ray tracing.

That is, the frame image rendering unit 70 generates a ray passing through each pixel on the view screen from the user's viewpoint, and samples the pixel values of the arrival point in the original image placed in front of the ray, thereby obtaining the pixel values of the display image. decide. When an image is displayed on a flat-panel display using the central projection original image, the pixel array of the display screen and the pixel array of the original image may be the same. Also, the frame image drawing unit 70 performs local update by switching the processing content for the place where the event occurred based on the most recent event information.

For example, the frame image drawing unit 70 determines the pixel values of the display image based on the information of the pixel values of the original image included in the event information for the location where the event has occurred. When the position where the event occurs in the original image is associated with the direction of the ray, whether or not the event has occurred at the destination can be determined from the relationship with the ray direction set in the ray tracing. When an event occurs at the destination, the frame image drawing unit 70 replaces the sampling destination with pixel value information included in the event information, thereby updating only the necessary portion of the display image.

Alternatively, the frame image drawing unit 70 may draw the image of the object in which the event occurred by 3DCG based on the depth information and model data of the object included in the event information. At this time, the frame image drawing unit 70 may perform general ray tracing that calculates reflection, refraction, and the like on the surface of the object. In any case, in the present embodiment, the rendering of the entire display image is performed based on ray tracing, so even with such local correspondence, there is little change in processing content and switching is easy.

Note that the frame image drawing unit 70 draws the frames of the display image at a rate higher than the frame rate of the original image in order to reflect the occurrence of the event in the display with high time resolution. The event information management section 72 switches the drawing process in the frame image drawing section 70 as described above based on the event information acquired by the event information acquisition section 64 . That is, it determines whether or not an event has occurred at the set destination of the ray, replaces the sampling destination, or draws the object in which the event has occurred by 3DCG.

Note that the model data of the object may be stored inside the display control device 10. The output unit 74 outputs data of the display image drawn by the display image drawing unit 66 to the display device 16 for display. As a result, at a frame rate higher than the transmission rate of the original image, an image showing details of changes in the portion where the event occurred is displayed.

FIG. 6 is a diagram for explaining a problem related to the balance between the amount of information provided and the amount of information received when a flat panel display is used as the display destination. Here, the amount of provided information is the amount of information represented by the display image, and the amount of received information is the amount of information recognized by the brain of the user who is viewing the image. The figure shows the angle of view (or FOV: Field Of View) of the displayed image at an appropriate distance from the user's viewpoint 80 . For example, in the case of an HDTV screen 81 having about 2000 horizontal pixels, the angle of view _θ2K is about 33°. In the case of UHDTV screens 82 and 84 having approximately 4000 horizontal pixels and approximately 8000 horizontal pixels, the angles of view θ _4K and θ _8K are approximately 61° and 100°, respectively.

The figure also shows a screen 86 of a head-mounted display for comparison, and in this case, the angle of view is about 100° like the screen 84. In any case, as shown in the figure, the larger the screen of the display, the wider the angle of view from the line of sight V becomes. On the other hand, the number of pixels on the screen included per unit angle when viewed from the viewpoint 80 increases as the distance from the line of sight V increases. For example, when looking at the screen 84, even if the range of the angle θ is the same, a wider range of pixels come into the field of view in a region 90b distant from the line of sight V than in a region 90a close to the line of sight V.

That is, the further away from the line of sight V, the greater the density of the amount of information entering the field of view. On the other hand, it is known that the human visual function has a distribution with respect to the visual field. The central region on the retina that corresponds to the center of the visual field is called the fovea, and has the highest visual acuity and ability to discriminate colors and shapes. The area within about 5° from the line of sight, including the area that is foveated out of the visual field, is called central vision, the area within about 20° from the line of sight is called the effective field of view, and the area outside it is called peripheral vision. It decreases in that order.

In other words, the amount of received information becomes smaller the further away from the line of sight V. Considering that the density of the amount of information entering the field of view increases at positions away from the line of sight V as described above, displaying the entire image with a uniform spatial detail level will increase the amount of information provided and the amount of information received. It can be said that it is in a state of balance. In addition, in reality, the line of sight V changes due to the movement of the image of the object, etc., so the area that can be captured with high visual performance moves on the screen. In order to deal with this, it is necessary to raise the frame rate and update the image in detail temporally at least for the relevant area.

Therefore, in the present embodiment, the image quality is improved by paying particular attention to an area within a predetermined range from the line of sight V, and by extension, the point of gaze, thereby correcting the imbalance between the amount of provided information and the amount of received information. Eliminate waste of processing and transmission for the area. Specifically, as described above, information about an event occurring outside the region of interest is excluded from transmission targets. As a result, wasteful update processing in the display control device 10 can be eliminated, and display at a high frame rate is possible.

FIG. 7 is a diagram for explaining the relationship between the data transmitted by the image transmission device 20 and the display image generated by the display control device 10. FIG. The figure shows the passage of time in the horizontal direction, and the middle part of the figure represents the arrangement of the displayed frames. Here, the updated portion is indicated by a thick line in the vertical line representing the entire frame. In this example, as shown in FIG. 2(a), it is assumed that an image captured at a remote location such as a live broadcast or video chat is displayed on a flat panel display.

In this case, the data transmitted from the image transmission device 20 consists of data of the entire original images 180a, 180b, . . . and event information 182a, 182b, 182c, . For simplification, the figure shows the images of speakers on the image plane as event information 182a, 182b, 182c, . However, in practice, as described above, by detecting the occurrence of events on a pixel-by-pixel basis, information is transmitted with higher spatial and temporal resolution, such as speaking lips or blinking eyes. Also, the event information 182a, 182b, 182c, .

The data of the entire original images 180a, 180b, . is sent asynchronously with the timing of event occurrence. The display control device 10 generates a display image frame using the transmitted data. That is, the display image 184a is generated using the data of the entire original image 180a. Display image 184a corresponds to frame 188 in the middle row.

Then, the display control device 10 updates only the necessary part 190 using the event information 182a, 182b, 182c, . Display images 186a, 186b, 186c, . . . are generated. Since the transmission timing of the event information 182a, 182b, 182c, . Then, the display control device 10 generates a corresponding display image 184b based on the data of the entire original image 180b transmitted next.

In this way, the display control device 10 interpolates between the frames of the entire original images 180a, 180b, . to achieve the display of For example, the display control device 10 uses the original image of 30 fps or 60 fps transmitted from the image transmission device 20 and displays the image at 120 fps or 240 fps. By updating only the part that the user is looking at and the part that has changed in pixel units, it has good compatibility with the above-mentioned visual characteristics, and the reduction in the size of the data to be transmitted and processed results in low latency and high performance. Changes can be expressed precisely.

FIG. 8 shows another example of the relationship between the data transmitted by the image transmission device 20 and the display image generated by the display control device 10. FIG. In this example, as shown in FIG. 2(b), it is assumed that an area corresponding to the user's field of view in the omnidirectional image is displayed on the head-mounted display. In this case, the data transmitted from the image transmission device 20 consists of data of the entire original images 200a, 200b, . . . and event information 202a, 202b, 202c, .

By preparing an omnidirectional image as the original image, it is possible to show a continuous image world even if the user suddenly turns around. On the other hand, when limited to a certain time, the angle of view to be displayed on the head-mounted display is limited to about 100 to 120 degrees as shown in FIG. Furthermore, as described above, the visual function of the user deteriorates as the area is farther from the line-of-sight direction. Therefore, treating the entire image uniformly lacks a balance between the amount of information to be provided and the amount of information to be received.

The image transmission device 20 transmits the data of the entire original images 200a, 200b, . In the drawing, it is assumed that an animal character is moving, and event information 202a, 202b, 202c, . In the figure, omnidirectional images are shown as the entire original images 200a, 200b, . may be limited to The event information 202a, 202b, 202c, .

The display control device 10 generates display images 204a, 204b, . . . using data of the entire original images 200a, 200b, . , . . . are used to generate display images 206a, 206b, 206c, . Here, the display control device 10 may perform reprojection to correct the field of view so as to correspond to the position and posture of the user's head at each time, and then generate distorted images for the left eye and the right eye. . As described above, the position and posture of the user's head can be acquired using the motion sensor and camera provided in the head-mounted display.

In this reprojection, the display control device 10 draws pixels where events have occurred using the event information 202a, 202b, 202c, . Even in this case, even if the frame rate of the entire original image is lowered, a high-quality image can be displayed at a high frame rate such as 120 fps or 240 fps. In addition, it is possible to achieve both low delay in the change of the visual field with respect to the movement of the user's head and low delay in the movement of the image in the fixation area. Since the event detection is performed independently of the user's motion, the motion in the world coordinate system can be accurately detected by comparing the difference between the frames of the display image.

FIG. 9 exemplifies the data format of the omnidirectional image and event information prepared by the image transmission device 20 . (a) of the figure shows the equidistant cylindrical projection, and (b) shows the form of equidistant projection directly expressing the output of the fisheye camera. When acquiring an omnidirectional image by photographing, for example, an omnidirectional camera in which fisheye cameras having an angle of view of about 220° are combined on the front and back is used. When a fish-eye camera is used, as shown in (b), an image of equidistant projection in which an object is projected onto a spherical surface is obtained.

In the case of (a), such an image of equidistant projection is once converted into data 210a in equirectangular projection, and then at least part of it is targeted for transmission. At this time, the event information 212a represents the occurrence position of the event by the position coordinates on the image plane of the equirectangular projection. On the other hand, in the present embodiment, equidistant projection data 210b as shown in (b) may be transmitted. By doing so, it is possible to realize low-delay data transmission without processing for converting to data 210a in the equirectangular projection or preparing a buffer memory for that purpose.

At this time, the event information 212b represents the occurrence position of the event with position coordinates on the image plane in equidistant projection. In other words, event information can also be transmitted with low delay without coordinate conversion of the output from the EDS. Also in the case of (b), only the data of the visual field range 214 in the equidistant projection data 210b may be transmitted as the data of the entire original image. As for the event information 212b, only the information of the event occurring in the gaze area 216 within the visual field range 214 may be transmitted.

Although FIG. 9 focused on transmission processing from the image transmission device 20, the display format on the display device 16 side also affects the reduction in delay until display. FIG. 10 is a diagram for explaining the relationship between the shape of the view screen and the amount of provided information. The viewscreen 414 is a screen for presenting central projection images on a flat panel display. A view screen 426 is a screen for representing a distorted image through the eyepiece. The figure shows both screens viewed from the side along with the user's viewpoint 424 .

The view screen 414 is composed of a plane having a predetermined angle of view around the axis C, for example. In this case, the image of the object 425 is displayed in a uniformly reduced scale according to the distance from the view screen 414 regardless of the height from the axis C. On the other hand, a distorted image premised on an eyepiece lens has the same properties as an image captured by a fisheye lens, and as a result the view screen 426 has a curved shape as shown. However, the detailed shape of the viewscreen 426 depends on the lens design.

As is clear from the figure, the area of the angular range 428 in the vicinity of the optical axis (axis C) of the viewscreen 426 has a small area difference from the corresponding area of the viewscreen 414, whereas the angular range is greater than the optical axis. The farther away, the smaller the area ratio to the viewscreen 414 . For this reason, in the central region 434 of the image, there is almost no difference in image size between the central projection image and the distorted image, whereas in the peripheral regions 432 and 426, the image in the central projection image is significantly reduced in the distorted image. be done. In other words, it can be said that part of the image represented by the central projection contains useless information that is not reflected in the distorted image.

This is even more pronounced not only in central projection images, but also in equirectangular projection images in which the image is magnified above and below the image plane. Therefore, in the case of a head-mounted display, as described with reference to FIG. 9, by transmitting the equidistant projection image without converting it into equirectangular projection data, information is wasted in the distorted image that is finally displayed. can eliminate the generation and transmission of

FIG. 11 is a diagram for explaining distortion image drawing processing in the display control device 10. FIG. The figure shows the passage of time in the horizontal direction, and the upper part of the figure represents the arrangement of the displayed frames. Each frame is represented here as a curved viewscreen of the distorted image, of which the updated portion is shown in bold. The format of data transmitted from the image transmission device 20 is not limited.

The display image drawing unit 66 of the display control device 10 draws the entire frame 220a of the display image at the timing when the data of the entire original image is transmitted. At this time, a ray is generated that passes through each pixel on the curved view screen, and the pixel value is determined by acquiring the color information of the arrival point in the original image. This allows the distorted frame 220a to be drawn directly. Incidentally, in practice, as shown in FIG. 8, an image pair composed of left-eye and right-eye images is generated.

The display image drawing unit 66 generates frames 222a, 222b, and 222c in which only the parts where the event occurred are updated based on the event information until the data of the entire original image is transmitted next time. Although only the updated portions of the frames 222a, 222b, and 222c are shown in the drawing, in reality, the regions in which no events occur are also represented as images. The direction from the optical center of the camera to the pixel where the event occurred can be determined from the positional information of the pixel where the event occurred and the camera optical system included in the event information.

For example, the display image drawing unit 66 performs ray tracing using event information only for necessary parts based on the correspondence between the direction and the ray set at the time of drawing, and determines the pixel value. Again, the strain image can be drawn directly using a curved viewing screen. Next, when the data of the entire original image is transmitted, the display image drawing section 66 generates a corresponding display image frame 220b. By these procedures, low-delay and high-definition image expression can be realized without unnecessary processing and transmission even for distorted images.

FIG. 12 shows a modification of the image display system according to this embodiment. In this example, as the image transmission device 20, a server 252 that generates computer graphics using 3D model data 260 is introduced. Note that the server 252 may be connected to imaging devices 240a and 240b that capture omnidirectional images and wide-angle images, acquire these captured images, and generate images combined with computer graphics. The server 252 may be a server in a cloud environment such as cloud gaming.

In this case as well, the server 252 transmits the data 266 of the entire original image at a predetermined rate to the display control devices 10d and 10d of the clients, and also transmits the event information 268a and 268b through the bypass line. The figure shows an equidistant projection image 242 output by a pair of fisheye cameras and a cube map image 270 as an example of the data format of the omnidirectional image. A cube map is a two-dimensional map obtained by projecting an omnidirectional image onto the inner surface of a cube that circumscribes a spherical screen that represents the omnidirectional image, and then developing the image. Various data formats may be used when the server 252 represents the omnidirectional image in this manner.

In any case, of the display control devices 10c and 10d that have received either of these data, the display control device 10c connected to the flat-panel display as the display device 16c generates and outputs a centrally-projected display image 272. . The display control device 10d connected to the head-mounted display as the display device 16d generates and outputs a display image 274 composed of distorted images for the left eye and the right eye. Further, the display control devices 10c and 10d update only the necessary portions 276a and 276b according to the transmission of the event information 268a and 268b, thereby achieving display at a high frame rate.

In this way, the display control device 10 determines the color information for each pixel by ray tracing using the transmitted image, thereby limiting the type and data format of the transmitted image and the type of the display device 16. It is possible to display with a unified scheme without In addition, it is possible to locally draw images based on event information, generate distorted images without converting to images of central projection, and also make partial areas on the image, such as the area corresponding to the fovea of the eye, Fine control is possible, such as drawing at a higher resolution than other areas.

FIG. 13 is a diagram for explaining the relationship between partial update and reprojection based on event information. In the case of a head-mounted display, as described above, the display control device 10 performs reprojection to correct the image so as to correspond to the position and posture of the user's head immediately before display. For example, the image transmission device 20 acquires information related to the position and posture of the user's head from the display control device 10 to which data is transmitted, and generates an image in the corresponding field of view. The display control device 10 makes the final adjustment of the field of view according to the movement of the head during the time from generation of the image to display. As a result, the image is displayed in a field of view that follows the movement of the head.

The image 280 in the upper part of the figure represents the image world generated by the image transmission device 20. In this image world, it is assumed that a spherical object 282 is moving against a belt-like object 283 like the ground below. (t0), (t1), and (t2) attached to the object 282 represent the positions of the object 282 at times t0, t1, and t2. In this image world, the display field of view at each time corresponding to the movement of the user's head is assumed to change to fields of view 284a, 284b, and 284c.

On the other hand, if the frame rate of the entire original image by the image transmission device 20 is set lower than the display frame rate, and only the data at times t0 and t2 are transmitted, naturally the image of the field of view 284b including the object 282 at time t1 is not sent. In practice, as described above, the image transmission device 20 predicts the movement of the user's head and transmits image data in a range wider than the field of view 284a as original image data, thereby enabling reprojection. .

The display control device 10 performs reprojection based on the transmitted data, and generates and outputs a display image as shown in the lower part. That is, the image data transmitted corresponding to time t0 is used to generate the display frames at times t0 and t1 (S10), and the image data transmitted corresponding to time t2 is used to generate the time t2 and the next display frame. A display frame is generated (S14). Here, when generating a display frame at time t1 in which only the field of view is changed using the image data at time t0, as shown in (a), even if the area occupied by the background object 283 changes appropriately, will result in the object 282 at time t1 appearing in the field of view not appearing.

Therefore, as described above, the image transmission device 20 transmits, as event information, information about pixels whose brightness has changed due to movement of the object 282 at a timing independent of data transmission of the entire original image. As a result, the display control device 10 can appropriately draw the image of the object 282 drawn using the latest event information in the display frame at time t1 (S12), as shown in (b). To facilitate understanding, the drawing shows a large change in whether or not the object 282 appears in the display image. is a large value. As a result, minute changes can be expressed with high definition and low delay.

When the event information includes the color change in the pixel where the event occurred, the display control device 10 refers to the color information included in the event information for the ray corresponding to the pixel where the event occurred in ray tracing. For other rays, the color information of the original image transmitted at time t0 is referred to. On the other hand, the display control device 10 itself may draw the object 282 without including the color information in the event information.

In the image display system shown in FIG. 12, the server 252 generates virtual world data represented by computer graphics. In this case, the server 252 generates information related to the three-dimensional space to be drawn, such as depth information of the object, during the image drawing process. Therefore, the server 252 transmits the information related to the generated three-dimensional space, so that the movement of the object 282, for example, can be drawn from the model data on the display control device 10 side.

In this case, by also transmitting the material information of the target object, it is possible to perform accurate light source calculation processing based on the material, such as changes in specular reflection accompanying movement of the viewpoint. For example, the display control device 10 directly draws an object set at a short distance from the user, an object that interacts with the user, and the like, so that a realistic image world can be efficiently expressed. As a result, it is possible to respond to needs such as virtual stores that display virtual products such as jewelry, virtual try-on and virtual makeup that allows virtual try-on and experience.

This aspect is applicable not only to computer graphics, but also to live-action images. That is, the image transmission device 20 separately acquires and transmits the depth information and material information of the subject, so that the display control device 10 can draw only objects with changes using them. Here, depth information and material information may be estimated using deep learning or artificial intelligence, as well as using on-site measurement values or pre-obtained registration information. Technologies for estimating subject material information and depth information from captured images have made rapid progress in recent years. Material Estimation", 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (USA), 2018, Vol. 1, p. 6315-6324).

Next, the representation format of the omnidirectional image that can be used in the present embodiment will be exemplified. In this embodiment, an omnidirectional image captured by a fish-eye camera or a multi-view camera, or an omnidirectional image drawn by computer graphics can be used. Since an omnidirectional image cannot be expressed at once with a general central projection image, a three-dimensional image that contains orientation information, such as an image projected onto a polyhedron that combines planes, an equirectangular view, and an equidistant projection image. A special data format is used to represent .

In image display technology such as a head-mounted display that is viewed through an eyepiece, it is possible to cut out a part of such a peculiar image, convert it to a centrally projected image, and then give the corresponding distortion to the eyepiece. many. On the other hand, the image transmission device 20 of the present embodiment transmits the original image in a data format other than central projection, and the display control device 10 reads the pixel values directly from the original image using a conversion formula corresponding to the data format. , produces a distorted display image.

This saves the resources required to go through central projection. In addition, since the original azimuth information of the original image can be transmitted as it is, it is possible to easily generate a display image corresponding to the direction of the user's line of sight. For example, when viewing an omnidirectional image from the same position as the imaging device, since the coordinate system of the image centered on the viewpoint matches, it is sufficient to display the image in the same direction as the user's line of sight. It will be. Furthermore, if the distortion aberration of the lens of the fish-eye camera that captures the omnidirectional image and the eyepiece lens are substantially the same, the transformation for the distorted image can be minimized.

FIG. 14 is a diagram for explaining display image generation processing when an omnidirectional image is represented by a cube map. As described above, the cube map is an image 292 obtained by projecting the image on the surface of the sphere 294 representing the omnidirectional circumference onto each surface of the cube 290 circumscribing the sphere 294 and developing the image 292 . First, the image transmitting device 20 sets a ray from the center of the sphere 294 to each pixel of the six squares forming the cube 290 . Then, an image 292 is generated by using the color of the surface of the sphere 294 through which the ray (vector rd) passes as the pixel value of the pixel. The resulting image 292 shows orientation information by means of the six squares and their positions. Note that, as described above, the image transmission device 20 may transmit only the data of the area required for display in the image 292 . In this case also, the image data format is the same.

The display control device 10 acquires the data of the image 292 or part thereof and generates a display image. At this time, the display control device 10 first sets a ray passing through each pixel of the curved screen corresponding to the eyepiece. Then, an image 292 is arranged as a cube circumscribing a sphere 294 centered on the viewpoint, and the pixel value is determined by sampling the color on the image 292 corresponding to the set ray (vector nor). For this processing, for example, a cube map texture reading command in the cube mapping function of OpenGL, which is a well-known computer graphics library, can be used. On the other hand, in this embodiment, event information is also transmitted in the same format.

That is, the position of the pixel where the event occurred is represented by the position coordinates on the plane of the image 292. At this time, the image transmission device 20 may transmit the vector rd of rays reaching the pixel. In this case, for example, when the display control device 10 generates a new display frame, when the vector rd transmitted as event information matches the ray vector nor set in generating the display frame, the vector nor is included in the event information. , partially updates the image by sampling the latest pixel information.

FIG. 15 is a diagram for explaining display image generation processing when an omnidirectional image is represented by equirectangular projection. The equirectangular projection image 302 is data obtained by projecting the image on the surface of the sphere 294 representing the whole sky onto the surface of the cylinder 300 that is in contact with the equator and expanding it. The image transmission device 20 first sets a ray that reaches each pixel of the cylinder 300 from the center of the sphere 294 . Then, the image 302 is generated by using the color of the surface of the sphere 294 through which the ray (vector rd) passes as the pixel value of the pixel. As a result, the plane of image 302 represents longitude u in the horizontal direction and latitude v in the vertical direction.

The display control device 10 acquires the data of the image 302 or part thereof and generates a display image. At this time, the display control device 10 first sets a ray passing through each pixel of the curved screen corresponding to the eyepiece. Then, the image 302 is arranged as a cylinder circumscribing the equator of the sphere 294 centered on the viewpoint, and the pixel value is determined by sampling the color on the image 302 corresponding to the set ray (vector nor). The image transmission device 20 further transmits event information indicating the position of the pixel where the event occurred by position coordinates on the plane of the image 302 . The process of partially updating the image by the display control device 10 in response to this may be the same as described with reference to FIG. 14 .

FIG. 16 is a diagram for explaining display image generation processing when an omnidirectional image is represented by equidistant projection. In this example, an imaging device is assumed in which fish-eye cameras having an angle of view of 220° are combined on the front and back. That is, the photographed image is data on the surface of the portion of the sphere 306 centered at the optical center, excluding the range outside 220°. Note that the drawing shows only the sphere 306 that constitutes the captured image of one of the pair of fisheye cameras. Also, the angle of view of each fisheye camera is not limited as long as it is 180° or more. In the case of a fish-eye camera, equidistant projection images 310a and 310b are obtained by representing the output of the sensor array as it is. However, it is not limited to the photographed image, and can be generated by drawing an image of equidistant projection by computer graphics.

The image transmission device 20 generates an image 308 consisting of equidistant projection images 310a and 310b representing the space in the opposite direction. The plane of the image 308 represents longitude u in the horizontal direction and latitude v in the vertical direction. When the image 308 is generated by ray tracing, the image transmitting device 20 sets rays at equal intervals from the center of the sphere 294 (306), and the color of the surface of the sphere 294 (306) through which the vector rd passes is displayed in pixels. An image 308 can be generated by taking values.

The display control device 10 acquires the data of the image 308 or part thereof and generates a display image. At this time, the display control device 10 first sets a ray passing through each pixel of the curved screen corresponding to the eyepiece. Then, the pixel value is determined by sampling the color at the position corresponding to the set ray (vector nor) in the image 308 . The image transmission device 20 further transmits event information representing the position of the pixel where the event occurred by position coordinates on the plane of the image 308 . The process of partially updating the image by the display control device 10 in response to this may be the same as described with reference to FIG. 14 .

FIG. 17 is a diagram for explaining a mode in which an omnidirectional image is represented by equidistant projection and a plurality of resolution levels are provided. The image to be used may be an image captured by a device in which fisheye cameras having an angle of view of 180° or more are combined on the front and back, as in FIG. 16, or may be an image rendered by computer graphics in a similar format. Therefore, the original image becomes data on the surface of the sphere 306 excluding the range outside the angle of view.

As described with reference to FIG. 10, an image represented by a curved screen such as a fisheye lens is represented by compressing the image as the distance from the center of the image increases, compared to the center projection image. This feature is compatible with the human visual characteristic that visual performance decreases with increasing distance from the region corresponding to the fovea. Developing this, it is considered that the image representing the space outside the user's field of view is less likely to be immediately captured by the fovea, so even if it is prepared with a low resolution, the visual impact will be small.

Therefore, the image transmission device 20 independently controls the resolution of the two equidistant projection images representing the space in the opposite direction. Specifically, each image 314a, 314b has a mipmap structure representing a plurality of resolution levels. For example, when the user is looking in the direction represented by image 314b, most of the area of image 314a is outside the field of view. Therefore, the image transmission device 20 preferentially transmits the image 314a with a low resolution as necessary.

As a result, the overall transmission data size is reduced and low delay is guaranteed compared to the two images 314a and 314b having the same resolution. In addition, the image world can be visually recognized without discomfort by increasing the resolution of the image 314a gradually as the user's line of sight moves closer to the user's line of sight.

Note that the process of generating the display image in the display control device 10 and the process of partially updating the image based on the event information may be the same as those described above. In the figure, the omnidirectional image is represented by two images of equidistant projection, but the number of images of equidistant projection may be three or more depending on the angle of view. Again, by constructing a mipmap structure for each image represented in different orientations, their resolutions may be controlled independently.

Note that the image transmission device 20 may generate data such that the resolution of the central portion of the equidistant projection image is maximized, and the resolution decreases with increasing distance from the center. FIG. 18 is a diagram for explaining a method of generating an equidistant projection image having a distribution in which the resolution decreases from the center to the periphery. First, the equidistant projection image 318 is represented by two-dimensional coordinates of longitude u and latitude v, as shown in (a). When two fish-eye cameras are used as shown in FIGS. 16 and 17, two similar images are generated, but the object to which the resolution distribution is given may be both of them, or the user is mainly looking at it. An image only is also acceptable.

The image transmission device 20 sets a ray that reaches each pixel on the plane of the image 318 from the optical center of the camera, and sets the color of the surface of the sphere through which the vector rd passes as the pixel value of that pixel. However, in this case, the area represented by one pixel is increased by decreasing the ray density as the distance from the center of the image increases. The transformation from the position coordinates (u, v) to the vector rd on the plane of the image 318 with the angle of view of 220° is realized, for example, by the following calculation.

Here, the function f(th) is such that the larger the absolute value of the angle th corresponding to the distance from the center of the image to the position coordinates (u, v), as indicated by the broken line 320a in FIG. is a monotonically increasing function. As a result, the greater the distance from the center of the image, the lower the density of the rays represented by the vector rd. As a result, the image 318 becomes data in which the area represented by one pixel varies depending on the position on the image plane. The display control device 10 performs sampling after converting the set ray vector nor into position coordinates (u, v) on the image 318 by, for example, the following calculation.

Here, the function f(th) is the inverse function of the dashed line 320a used to generate the image 318, as indicated by the solid line 320b in (b) of the figure. As a result, data representing different areas represented by one pixel can be represented as data of pixels arranged evenly on the display screen. By such processing, it is possible to express the region corresponding to the fovea with higher definition, reduce the amount of transmission data, and improve low-delay performance. In the above calculation, reading corresponding to the mipmap structure is performed by the "LODBIAS" parameter, but the gist of the present embodiment is not limited to that.

FIG. 19 is a diagram for explaining display image generation processing when an omnidirectional image is represented on the plane of a regular octahedron. In this case, an image 324 is generated by projecting the image on the surface of the sphere 294 representing the omnidirectional circumference onto the surface of the regular octahedron 322 circumscribing it. That is, the image transmission device 20 sets a ray that reaches each pixel of the regular octahedron 322 from the center of the sphere 294, and converts the color of the surface of the sphere 294 through which the ray (vector rd) passes to the pixel value of the pixel. An image 324 is generated by .

The display control device 10 acquires the data of the image 324 or part thereof and generates a display image. At this time, the display control device 10 first sets a ray passing through each pixel of the curved screen corresponding to the eyepiece. Then, the image 324 is arranged as a regular octahedron circumscribing the sphere 294 centered on the viewpoint, and the pixel value is determined by sampling the color on the image 324 corresponding to the set ray (vector nor). The image transmission device 20 further transmits event information indicating the position of the pixel where the event occurred by position coordinates on the plane of the image 324 . The process of partially updating the image by the display control device 10 in response to this may be the same as described with reference to FIG. 14 . Various methods other than those described above have been proposed for expressing an omnidirectional image with two-dimensional data, and any of them may be adopted in the present embodiment.

According to the present embodiment described above, in a system that displays an image using data of an original image transmitted from an image transmission device, the image transmission device detects a change in brightness as an event on a pixel-by-pixel basis. , is transmitted asynchronously with the data of the entire original image. The display control device that has acquired the data generates the entire area of the display frame using the data of the entire original image, and interpolates the display frame in which only the changed portion is updated pixel by pixel based on the event information. . Here, the display control device enables updating for each pixel by using a ray tracing method of setting a ray corresponding to each pixel on the display image and acquiring color information.

As a result, even if the frame rate of the entire original image is lowered, the movement of the detailed image can be accurately represented on the displayed image, so the size of the data to be transmitted can be significantly reduced. By limiting the event to be transmitted within the user's attention area, the visual impact can be minimized and the data size can be reduced. As a result, even when displaying an image with a wide viewing angle, such as an omnidirectional image, on a head-mounted display or on a display device with a large number of pixels, such as a UHDTV, communication bandwidth and processing resources can be saved, resulting in low power consumption. High-quality image representation can be achieved with low power consumption and low latency.

Also, by adopting ray tracing, it is possible to directly generate a display image with distortion corresponding to the distortion and chromatic aberration of the eyepiece lens from an image in which the original image is distorted, such as an omnidirectional image. As a result, there is no need for a buffer memory for developing the central projection image, and processing for image conversion can be simplified. Furthermore, by directly generating a distorted display image from an equidistant projection image, it is possible to consistently maintain the amount of information in a distribution suitable for the human visual characteristics from transmission to display. there will be no room to enter.

A similar effect can also be obtained with a flat panel display by updating only the part where the event occurred within the gaze area. As described above, the present embodiment can be easily applied regardless of the display format, such as a flat panel display or a head-mounted display, and regardless of the type of display image, such as a photographed image, computer graphics, or a composite image thereof. .

The present invention has been described above based on the embodiment. It should be understood by those skilled in the art that the embodiments are examples, and that various modifications can be made to combinations of each component and each treatment process, and that such modifications are within the scope of the present invention. .

10 Display control device, 16 Display device, 20 Image transmission device, 22 CPU, 24 GPU, 26 Main memory, 50 Gaze area acquisition unit, 52 Image data acquisition unit, 54 Event information generation unit, 56 Data transmission unit, 60 Gaze area Transmission unit 62 Image data acquisition unit 64 Event information acquisition unit 66 Display image drawing unit 70 Frame image drawing unit 72 Event information management unit 74 Output unit.

As described above, the present invention can be used for various information processing devices such as content servers, game devices, head-mounted displays, display devices, mobile terminals, personal computers, and image display systems including any of them.

Claims (17)

1. An image display system comprising: an image transmission device for transmitting original image data; and a display control device for controlling image display using the original image data,
   The image transmission device is
   an event information generating unit that generates event information including position information of a pixel when a luminance change event occurs in any pixel in the original image;
   a data transmission unit that transmits data of the entire original image at a predetermined frame rate and asynchronously transmits the event information;
   with
   The display control device is
   a display image drawing unit that generates display image data at a predetermined frame rate using the data of the entire original image, and locally updates the display image based on the most recent event information;
   an output unit that outputs data of the display image to a display device;
   An image display system comprising:
2. The image display system according to claim 1, wherein the display image drawing unit generates the data of the display image at a frame rate higher than the frame rate of the entire original image to be transmitted.
3. The display image drawing unit determines the pixel value of the display image by ray tracing the original image, and determines the pixel value of the ray corresponding to the occurrence location of the event based on the event information. 3. The image display system according to claim 1 or 2.
4. The image transmission device further includes a gaze area acquiring unit that acquires information related to a user's gaze area with respect to the display image,
   4. The image display system according to any one of claims 1 to 3, wherein the event information generation unit selects an event occurring within the attention area and generates the event information related to the event.
5. The event information generation unit generates the event information by acquiring an event signal indicating the occurrence of the event from an imaging device that captures the original image and is equipped with an event-driven sensor. Item 5. The image display system according to any one of Items 1 to 4.
6. The image transmission device further includes an original image rendering unit that renders the original image,
   6. The event information generation unit according to claim 1, wherein the event information generation unit generates the event information by acquiring information on occurrence of an event specified in drawing processing by the original image drawing unit. image display system.
7. wherein the event information generation unit includes information related to a pixel value of a pixel in which the event has occurred in the event information;
   7. The image display system according to any one of claims 1 to 6, wherein the display image drawing unit locally updates the display image based on the information regarding the pixel values.
8. The event information generation unit includes information used for drawing an object represented by a pixel in which the event has occurred in the event information,
   7. The image display system according to claim 1, wherein the drawing unit draws the object using information used for the drawing.
9. The output unit outputs data of the display image to a head mounted display,
   The display image rendering unit corrects the field of view of the original image so as to correspond to the position and orientation of the head of a user wearing the head-mounted display in the display image, and localizes the display image based on the event information. 9. The image display system according to any one of claims 1 to 8, wherein updating is performed.
10. The data transmission unit transmits data of the entire original image in which at least part of the omnidirectional image is represented in a format different from central projection,
    10. The image display system according to claim 9, wherein the display image rendering unit renders the display image distorted corresponding to an eyepiece lens of the head-mounted display by ray tracing the original image.
11. The data transmission unit acquires a plurality of equidistant projection images representing different directions as the omnidirectional image, generates data of a plurality of resolutions for each image, and transmits data of a selected resolution for each image. 11. The image display system according to claim 10, wherein:
12. The data transmission unit transmits data of the entire original image having a distribution in an area represented by one pixel,
    12. The display image rendering unit according to any one of claims 9 to 11, wherein the display image drawing unit uses the data of the entire original image to generate the data of the display image having a resolution distribution in which the resolution of the central portion is maximized. The image display system described in .
13. An image transmission device that transmits data of an original image and realizes image display using the original image at a transmission destination,
    an event information generating unit that generates event information including position information of a pixel when a luminance change event occurs in any pixel in the original image;
    a data transmission unit that transmits data of the entire original image at a predetermined frame rate and asynchronously transmits the event information, thereby locally updating a display image at a transmission destination;
    An image transmission device comprising:
14. A display control device that receives data of an original image and controls image display using the data of the original image,
    an image data acquisition unit that acquires data of the entire original image at a predetermined frame rate;
    an event information acquisition unit that acquires event information including position information of a pixel transmitted asynchronously with data of the entire original image in response to a brightness change event occurring in one of the pixels in the original image;
    a display image drawing unit that generates display image data at a predetermined frame rate using the data of the entire original image, and locally updates the display image based on the most recent event information;
    an output unit that outputs data of the display image to a display device;
    A display control device comprising:
15. An image display system comprising an image transmission device for transmitting original image data and a display control device for controlling image display using the original image data,
    The image transmission device is
    a step of generating event information including position information of a pixel when a luminance change event occurs in any pixel in the original image;
    transmitting the data of the entire original image at a predetermined frame rate and asynchronously transmitting the event information;
    The display control device
    generating display image data at a predetermined frame rate using the data of the entire original image, and locally updating the display image based on the most recent event information;
    a step of outputting the display image data to a display device;
    An image display method comprising:
16. To a computer that transmits original image data and realizes image display using the original image at the destination,
    a function of generating event information including position information of a pixel when a luminance change event occurs in one of the pixels in the original image;
    a function of locally updating a display image at a transmission destination by transmitting data of the entire original image at a predetermined frame rate and transmitting the event information asynchronously;
    A computer program characterized by realizing
17. A computer that receives original image data and controls image display using the original image data,
    a function of acquiring data of the entire original image at a predetermined frame rate;
    a function of acquiring event information including position information of a pixel transmitted asynchronously with data of the entire original image in response to a luminance change event occurring in any pixel in the original image;
    A function of generating display image data at a predetermined frame rate using the data of the entire original image, and locally updating the display image based on the most recent event information;
    a function of outputting data of the display image to a display device;
    A computer program characterized by realizing

Images