KR102204212B1 - Apparatus and method for providing realistic contents - Google Patents

Abstract

A sensory content providing apparatus according to an embodiment of the present disclosure includes a motion detector configured to detect at least one of a movement of a user's head and a gaze, receiving an image, and based on at least one of a movement and gaze detected from the motion detector. An ROI image extraction module configured to determine a region of interest to be projected on the display of the device from an image, an image enhancement module configured to perform image enhancement processing on a target image corresponding to the determined region of interest, and image enhancement processing are performed. It may include a display device configured to output an image including the target image.

Description

A device and method for providing immersive content {APPARATUS AND METHOD FOR PROVIDING REALISTIC CONTENTS}

The present disclosure relates to an apparatus and method for providing immersive content, and more particularly, to determine a region of interest of immersive content based on detected user motion information, and perform image improvement processing for the determined region of interest. It relates to an apparatus and method for providing immersive content.

Recently, consumers prefer high-resolution and sized TVs, and thus, ultra-high resolution display panels are continuously being launched into the market rapidly. However, since most of the current image data sources are images captured using previously produced photographing equipment, there is a high possibility that the image has a relatively low resolution than the resolution supported by the ultra-high resolution display panel. In other words, it is a reality that a lot of content of high-definition video is insufficient. For example, in the case of a cable video service currently supplied to homes, assuming that there is one UHD channel, there are about 200 FHD channels.

Meanwhile, the media environment is evolving into immersive media beyond TV, the Internet, and smartphones due to the convergence and digitalization of broadcast communication. Realistic media can create new services and contents through convergence with existing media based on convergence between industries. Virtual Reality and/or Augmented Reality technologies are essential to realize such realistic media.

In order to implement immersive media using virtual reality and augmented reality technology, unlike a 3D world based on a conventional 2D display, 360-degree images in all directions of up, down, left, and right are created around a user and provided as a display. Since images are provided in all directions of the user, if the resolution of the display or the resolution of the image is low, a “screen door effect” phenomenon in which pixels look like mosquito nets may appear, resulting in a significant decrease in immersion. Therefore, in order to realize virtual reality with realistic three-dimensional effect and perspective, the importance of super-resolution technology (for example, 4K or 8K image conversion from FHD) that restores from low-resolution images to high-resolution images is increasing.

In addition, since virtual reality and augmented reality express a three-dimensional effect and perspective by utilizing the parallax of both eyes, when the reaction speed of an image is low, a user may feel dizzy due to an error between the user's vision and the image. Accordingly, there is a need for a technology that implements an image to move smoothly without an afterimage according to a user's perspective that changes in real time.

The embodiments disclosed in the present specification detect at least one of the user's head movement and gaze using a motion detector, and determine a user's region of interest in immersive content based on at least one of the detected movement and gaze, The present invention relates to an apparatus and method for providing immersive content capable of performing image improvement processing on a target image corresponding to a determined region of interest.

A sensory content providing apparatus according to an embodiment of the present disclosure includes a motion detector configured to detect at least one of a movement of a user's head and a gaze, receiving an image, and based on at least one of a movement and gaze detected from the motion detector. An ROI image extraction module configured to determine a region of interest to be projected on the display of the device from an image, an image enhancement module configured to perform image enhancement processing on a target image corresponding to the determined region of interest, and image enhancement processing are performed. It may include a display device configured to output an image including the target image.

A method of providing immersive content by a head mounted display device according to another exemplary embodiment of the present disclosure includes: receiving a virtual reality video, head movement of a user wearing a head mounted display And detecting at least one of the gaze, determining a region of interest based on at least one of the detected motion and the gaze, and performing image enhancement processing on a portion of the received virtual reality image corresponding to the determined region of interest. And outputting an image including a portion of the virtual reality image on which the image enhancement processing has been performed, on a display of the head worn display device.

According to various embodiments of the present disclosure, a region of interest of sensory content may be determined based on motion information of a user, and quality of the determined region of interest may be improved, thereby providing a high quality image to a user. While the improved image is provided to the user, the amount of computation required for this can be minimized.

In addition, since the amount of computation required for image processing is minimized, the computational speed for processing immersive content can be increased, and further, the reaction speed of the image can be increased. Accordingly, a smooth and natural image without an afterimage is provided to the user, thereby maximizing the user's immersion in tangible media.

Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, in which like reference numerals denote like elements, but are not limited thereto.
1 is a diagram illustrating an environment in which image processing and output are performed in real time on an image received by a sensory content providing apparatus according to an exemplary embodiment of the present disclosure.
2 is a block diagram illustrating a detailed configuration of an apparatus for providing immersive content according to an embodiment of the present disclosure.
3 is an exemplary diagram in which a head-worn display device according to an embodiment of the present disclosure is provided as a sensory content providing device.
FIG. 4 is an exemplary diagram when a sensory content provided by a sensory content providing apparatus according to an embodiment of the present disclosure is displayed in a three-dimensional space.
5 is an exemplary diagram showing a range of an ROI that can be defined based on an angle of a field of view viewed by a user according to an embodiment of the present disclosure.
6 is an exemplary diagram illustrating a process of image enhancement processing an ROI determined based on a user's gaze detected by a motion detector according to an exemplary embodiment of the present disclosure.
7 is a view of a user detected from a motion detector according to another embodiment of the present disclosure.
It is an exemplary diagram showing a process of image enhancement processing an ROI determined based on a gaze.
8 is an exemplary view showing a process of providing an augmented reality environment by synthesizing an augmented reality image to a region of interest determined based on a user's gaze sensed from a motion sensor of a sensory content providing apparatus according to an embodiment of the present disclosure to be.
9 is a flowchart illustrating a method of providing immersive content by a head-worn display device according to an embodiment of the present disclosure.

Hereinafter, with reference to the accompanying drawings, specific details for the implementation of the present disclosure will be described in detail. However, in the following description, when there is a concern that the subject matter of the present disclosure may be unnecessarily obscure, detailed descriptions of widely known functions or configurations will be omitted.

In the accompanying drawings, the same or corresponding elements are assigned the same reference numerals. In addition, in the description of the following embodiments, overlapping descriptions of the same or corresponding components may be omitted. However, even if description of a component is omitted, it is not intended that such component is not included in any embodiment.

Advantages and features of the disclosed embodiments, and a method of achieving them will become apparent with reference to the embodiments described below together with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, and only these embodiments make the present disclosure complete, and those skilled in the art to which the present disclosure pertains. It is provided only to fully inform the person of the scope of the invention.

The terms used in the present specification will be briefly described, and the disclosed embodiments will be described in detail.

The terms used in the present specification have selected general terms that are currently widely used as possible while considering functions in the present disclosure, but this may vary according to the intention or precedent of a technician engaged in a related field, the emergence of new technologies, and the like. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning of the terms will be described in detail in the description of the corresponding invention. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, not the name of a simple term.

In this specification, expressions in the singular include plural expressions, unless the context clearly specifies that they are singular. In addition, plural expressions include expressions in the singular, unless the context clearly specifies that they are plural.

When a part of the specification is said to "include" a certain component, it means that other components may be further included rather than excluding other components unless otherwise stated.

In addition, the terms "unit" or "module" used in the specification means software or hardware components, and "unit" or "module" performs certain roles. However, "unit" or "module" is not meant to be limited to software or hardware. The "unit" or "module" may be configured to be in an addressable storage medium or may be configured to reproduce one or more processors. Thus, as an example, "sub" or "module" refers to components such as software components, object-oriented software components, class components and task components, processes, functions, properties, It includes procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays and variables. Components and the functions provided in "sub" or "module" may be combined into a smaller number of components and "sub" or "module" or into additional components and "sub" or "module" Can be further separated.

According to an embodiment of the present disclosure, a "unit" or a "module" may be implemented with a processor and a memory. The term “processor” is to be interpreted broadly to include general purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and the like. In some circumstances, “processor” may refer to an application specific application (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), and the like. The term “processor” refers to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such configuration. You can also refer to it.

In addition, in this specification, the terms "server" or "client" may include "server device" or "client device", respectively.

Also, the term "memory" should be interpreted broadly to include any electronic component capable of storing electronic information. The term memory refers to random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erase-programmable read-only memory (EPROM), electrical May refer to various types of processor-readable media such as erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. The memory is said to be in electronic communication with the processor if it can read information from and/or write information to the memory. The memory integrated in the processor is in electronic communication with the processor.

In addition, the term "real time" indicates that the electronic switching system or common control system must continue to process information continuously without interruption, and that input cannot be waited or delayed even if the input processing cannot be handled. Can include. In addition, "real-time" may include a method of immediately performing necessary computational processing when data is generated in an information processing method by a computer and returning the result to a place where data is generated or where data is needed. In addition, "real-time" refers to the actual time that the physical process takes place, and means the time it takes to process the data and obtain the necessary results at the same time as the data is generated. It may include a method of processing and notifying the result immediately, and may include a method of processing a computer operation that processes data as soon as it is input. For example, the method of providing an image improvement process for a received image to the display in real time includes a method in which the received image is immediately image improved and provided to the display so that there is no afterimage and a smooth and natural image is provided to the user. can do.

The sensory content providing apparatus of the present disclosure may be any apparatus including an image sensor and/or a communication module. The sensory content providing device may include an electronic device capable of connecting to a network by having a communication module and performing image processing on an image for the sensory content. For example, the immersive content providing device may be an AR device and/or a VR device, but is not limited thereto, and the immersive content (for example, a virtual reality (VR) image, an augmented reality (AR) image, etc.) It may include any suitable device capable of performing image enhancement processing for and outputting to an output device in real time.

FIG. 1 is a diagram illustrating an environment in which image processing and output are performed in real time on an image received by a sensory content providing apparatus 120 according to an exemplary embodiment of the present disclosure.

In one embodiment, the external devices 110_1 to 110_n; where n is an integer greater than or equal to 1) is provided with a communication module to enable communication with other devices, and any device capable of storing at least one of one or more images and images Can be For example, the external devices 110_1 to 110_n; where n is an integer greater than or equal to 1) may be a server device, a smart phone, a tablet PC, a notebook computer, personal digital assistants (PDAs), or a mobile communication terminal. Each of the plurality of external devices 110_1 to 110_n may transmit one or more stored images to the sensory content providing device 120 through the communication network 130. Here, the at least one image may include a 360-degree image, and may include an image capable of supporting an augmented reality (“AR”) and/or a virtual reality (“VR”) environment. have. For example, the 360-degree image may include an image corresponding to an angle and direction that the user can see according to the movement of the user's head and/or the line of sight.

The sensory content providing device 120 may receive one or more AR images and/or VR images from at least one of the plurality of external devices 110_1 to 110_n through the communication network 130. Here, the sensory content providing device 120 may process or provide at least one of an AR image and a VR image, and may include an output device (eg, a display device, a speaker, etc.). For example, the sensory content providing device 120 may be configured to output an image according to the time of the user wearing the device 120 in real time. According to an embodiment, the received one or more images may include AR and/or VR images compressed with commercially available compression techniques (eg, MPEG, etc.). In addition or alternatively, the received one or more images are AR and/or VR having a lower resolution (e.g., full HD) than the resolution (e.g., 4K or 8K) supported by the display of the immersive content providing device 120 It may include an image.

According to an embodiment, the immersive content providing device 120 transmits all or part of the received image to motion information of the user wearing the device 120 or a device associated therewith (eg, the user's head movement and/or An image enhancement process may be performed based on the user's gaze movement), and an image on which the image processing operation was performed may be output to a display in real time and provided to the user. In one embodiment, the sensory content providing device 120 determines a region viewed by the user, that is, a region of interest (ROI) to be projected on the display, based on the user's motion information, and determines an image for the region of interest. Improvement processing can be performed.

According to an embodiment, when the received image is a low-resolution image, such image processing may perform a super-resolution operation to be output on a high-resolution display of the immersive content providing apparatus 120. According to another embodiment, when the received image is an image compressed with a compression technique, compression decoding may be required when displayed in the immersive content providing apparatus 120. According to another embodiment, the sensory content providing device 120 may perform an image quality improvement operation when outputting the received image to the display of the sensory content providing device 120. Here, a super-resolution operation, compression decoding, and image quality improvement operation have been described in different embodiments, but the immersive content providing apparatus 120 includes a plurality of super-resolution operation, compression decoding operation, and image quality improvement operation according to the received image. Can perform the arithmetic processing of Through such image processing, while the haptic image quality for maximizing the user's immersion is improved, the amount of computation required for this may be minimized. A process in which the immersive content providing apparatus 120 receives an image and performs image enhancement processing will be described in detail with reference to FIG. 2.

2 is a block diagram showing a detailed configuration of a sensory content providing apparatus 120 according to an embodiment of the present disclosure. As shown in FIG. 2, the sensory content providing device 120 includes a communication module 210, an image storage unit 220, a motion detector 230, an image sensor 240, a display device 250, and a control unit. It may include 260. Here, the control unit 260 may include an image generation module 262, an ROI image extraction module 264, and an image enhancement module 266.

The sensory content providing device 120 may communicate with an external device such as a server or another user terminal through the communication module 210. The communication module 210 is configured to communicate with an external device such as a user terminal or a server through a communication network, and an AR/VR image and/or image captured through an external device, or an AR/VR image and/or stored in an external device. You can receive video.

According to an embodiment, the communication module 210 may communicate with at least one of a plurality of external devices to receive one or more images (for example, a virtual reality (VR) image, an augmented reality (AR) image, etc.), and , These images may be provided to the controller 260. The communication module 210 described above is, for example, a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device and RS-serial communication, or a wireless LAN (WLAN), depending on the installation environment. Wireless networks such as Bluetooth and ZigBee can be variously selected and configured.

The controller 260 may output the received images to the display device 250 by performing image enhancement processing and provide the received images to the user. In an embodiment, the controller 260 may store the received images in the image storage unit 220. The image storage unit 220 may include any storage device configured to store image data for supporting various functions of the immersive content providing device 120. According to another embodiment, the image storage unit 220 of the immersive content providing device 120 includes a plurality of images (eg, a virtual reality (VR) image, an augmented reality (AR)) received from the communication module 210. Image, etc.), and the controller 260 may read or search at least one image from among a plurality of images stored from the image storage unit 220. In addition, the sensory content providing device 120 includes a connection terminal to insert a variety of storage devices capable of storing sensory content such as a memory card (eg, a compact flash card, an SD memory card, a USB memory, etc.). I can. The storage device insertable into the immersive content providing device 120 is not limited thereto, and may include any storage device capable of storing an image. The controller 260 may read or search at least one image from the storage device, and may store the corresponding image in the image storage unit 220.

The control unit 260 may receive an image from the communication module 210 or the image storage unit 220, and generate and display a virtual reality or augmented reality image based on the received image. Output to the device 250 may be provided to the user. The display device 250 may output an image or an image related to immersive content according to the control of the controller 260, or may reproduce an image by overlaying an image on the image. The display device 250 may receive a command signal (eg, a command signal related to an operation such as immersive content selection) through a preset interface (eg, a touch display, a microphone, a motion recognition sensor, etc.). . The display device 250 may include a display capable of displaying the received virtual reality or augmented reality image. According to an embodiment, the display device 250 may include transparent glass, and may be configured to output an augmented reality image on which image enhancement processing has been performed to an ROI on the transparent glass. The configuration of outputting the region of interest on the transparent glass will be described in detail with reference to FIG. 8.

The motion detector 230 may be configured to detect a user's motion, and may transmit the detected user's motion to the controller 260. According to an embodiment, the motion detector 230 may be configured to detect at least one of a movement of a user's head and a gaze. Here, the user's movement may include at least one of the user's gaze information (eg, the position of the user's pupil), the user's head direction information, and the head tilt information. In FIG. 2, it is disclosed that the sensory content providing device 120 is included in the motion detector 230, but is not limited thereto, and the motion sensor 230 is installed separately from the sensory content providing device 120 to detect It may be configured to provide the resulting movement to the controller 260.

The ROI image extraction module 264 of the controller 260 may determine a region of interest of the user based on motion information sensed from the motion detector 230. In an embodiment, the ROI image extraction module 264 may determine at least one of a user's head direction and a head tilt based on the user's motion information provided from the motion detector 230. The region of interest may be determined based on at least one of the determined user's head direction and head tilt. In another embodiment, the motion detector 230 detects the position of the pupil of the user wearing the sensory content providing device 120 and provides the detected pupil position information as motion information to the ROI image extraction module 264 can do. For example, the motion detector 230 may be disposed on the display device 250 and configured to detect the position of the user's pupil. The ROI image extraction module 264 may determine the location of the user's gaze on the display device 250 viewed by the user based on the location information of the pupil, and determine the area of interest to the user based on the determined location of the user's gaze. You can decide. According to another embodiment, the ROI image extraction module 264 is based on the user's movement information (head movement and/or gaze movement) detected from the motion detector 230, and the interest to be displayed on the display device made of transparent glass. It can be configured to determine the area. The region of interest described above may be changed in real time according to the user's motion information provided from the motion detector 230. This region of interest may be provided to the image enhancement module 266.

When the image enhancement module 266 receives the user's determined region of interest from the ROI image extraction module 264, the image enhancement module 266 may perform image processing on the received image based on the region of interest. According to an embodiment, the image enhancement module 266 may be configured to perform image enhancement processing on a target image corresponding to the received ROI. According to another embodiment, the image enhancement module 266 may be configured to receive an augmented reality image to be displayed in an ROI and perform image enhancement processing on the received augmented reality image. As shown in FIG. 2, the image enhancement module 266 may be configured to include a compression decoding operation processing module 270, an image preprocessing operation module 272, and a super-resolution operation processing module 274.

In one embodiment, the compression decoding operation processing module 270 is a region of interest determined by the ROI image extraction module 264 when the received image (for example, a virtual reality (VR) image) is a compressed image due to compression technology. The compression applied to the portion of the image corresponding to is decoded to be output to the display device 250. Here, the compressed video may be a video compressed by commercially available video compression techniques such as HEVC, H.265, and MPEG.

It has been described that the compression decoding operation processing module 270 decodes the compression applied to the portion of the image corresponding to the region of interest, but the present invention is not limited thereto, and the received image (eg, VR image, AR image, etc.) is a compressed image. In this case, prior to applying the image quality improvement process (eg, preprocessing operation processing, super-resolution operation processing, etc.), the compression decoding operation processing module 270 may perform compression decoding operation on the received image. For example, the entire received video can be decoded. In contrast, when the received image is a compressed or unencoded image, the decoding operation processing process need not be applied by the compression decoding operation processing module 270.

The image enhancement module 266 may be configured to improve image quality by processing an image corresponding to an ROI determined by the ROI image extraction module 264. The image preprocessing operation module 272 of the image enhancement module 266 may be configured to perform a preprocessing operation on a corresponding portion of an image (eg, virtual reality) in order to increase the quality of an image corresponding to a corresponding region of interest. have. For example, the image preprocessing module 272 may adjust an arbitrary image parameter representing an image corresponding to the region of interest. Here, the technology for adjusting the image parameter may include a technology for pre-processing the image, and the Demosaicing technique, WDR (Wide Dynamic Range) or HDR (High Dynamic Range) technique, Deblur technique, Denoise technique, Color Tone mapping technique, White It may include at least one of a balance technique and a decompression technique.

The technology for preprocessing an image may include at least one of an image signal processing technology for preprocessing the current image, a technology for preprocessing an image using a machine learning method, and a technology for preprocessing an image using a deep learning learning method. According to an embodiment, the image preprocessing operation module 272 may be configured to process an image signal on a plurality of image parameters representing an image for an ROI. For example, each of the plurality of image parameters may be sequentially image signal processed. That is, after one of the plurality of image parameters is pre-processed, it may be configured to perform pre-processing of other image parameters for an image having the processed parameter.

According to another embodiment, the image preprocessing operation module 272 may be configured to include a preprocessing probability model for preprocessing a plurality of image parameters representing an image for an ROI. Here, the preprocessing probability model may be learned through machine learning using various reference images or images, and may be configured to output an image obtained by preprocessing such an image when an input image for an ROI is input. According to another embodiment, the image preprocessing operation module 272 may include a learned artificial neural network (eg, a deep neural network) for preprocessing a plurality of image parameters representing an image for an ROI. Here, the artificial neural network may be trained using various reference images or images, and may be configured to output an image obtained by pre-processing such an image when an image of an input region of interest is input.

Although it has been described that the image pre-processing operation module 272 improves the image quality by pre-processing a portion of the image corresponding to the region of interest, it is not limited thereto. In order to improve the image quality and realize a visually clear image quality (for example, in the case of a compressed image, the portion lost due to compression is corrected to be close to the original image-Decompression technique). It is also possible to perform processing. According to an embodiment, the image preprocessing operation module 272 may perform a preprocessing operation on the entire area of the received image.

The image enhancement module 266 may be configured to increase the resolution of the region of interest determined by the ROI image extraction module 264 (ie, super-resolution). Here, such resolution may be implemented through an existing interpolation method or a recently introduced deep learning algorithm. For example, the resolution of the region of interest where the user's gaze is located can be rendered in high quality (e.g., 4K or 8K), and if the user's gaze is out of sight, it can be rendered in normal quality (e.g., Full HD). have.

According to an embodiment, the super-resolution operation processing module 274 of the image enhancement module 266 performs a super resolution operation on a portion of the image of the received virtual reality corresponding to the region of interest. I can. According to an embodiment, after the image quality is improved by performing a pre-processing operation on a portion of the image corresponding to the region of interest by the image pre-processing operation module 272, the super-resolution operation processing module 274 A super resolution operation may be performed to increase the resolution of a portion of an image corresponding to the region. Therefore, even if the image to be output to the display is not of high quality (for example, a low-resolution image), it is possible to minimize the amount of computation required by processing only the area of interest to the user by pre-processing and/or super-resolution operation. It is possible to provide an image that can maximize the user's immersion by improving it. In addition, as described above, even if the received image has a lower resolution (e.g., Full HD, etc.) than the resolution of the display that outputs the image (e.g., 4K, 8K, etc.), the region of interest Can be printed in high resolution.

In the present embodiment, a configuration in which a pre-processing operation and a super-resolution operation for a portion of an image of a received virtual reality corresponding to the region of interest are processed together is disclosed, but the received virtual reality corresponding to the region of interest according to the received image Any combination of a compression decoding operation, a preprocessing operation, and a super-resolution operation on the portion of the image of the image may be processed. The ROI image extraction module 264 determines a region of interest, and the image enhancement module 266 performs an image processing operation based on the determined region of interest. This will be described in detail with reference to FIG. 8. The virtual reality image or augmented reality image subjected to the image improvement processing may be provided to the display device 250.

The image sensor 240 may be configured to capture an image and/or an image of an actual reality to be displayed to the user. For example, the image sensor 240 may generate an image or an image by capturing an area of interest to the user (eg, part of a direction the user is looking at). In an embodiment, the image sensor 240 may be configured to be photographed according to motion information of the user. The image or image generated in this way may be provided to the image generating module 262 of the controller 260. The image generation module 262 may receive an image or an image captured from the image sensor 240 to generate immersive content (eg, an AR image). In one embodiment, the image generation module 262 receives an image and/or an image captured from an image sensor, and the augmented reality content (for example, in an area corresponding to an image and/or an image included in the image , An augmented reality image and/or augmented reality image, etc.) may be synthesized to generate an augmented reality image. Such augmented reality content may include a target image processed by the image enhancement module 266 to improve the image. As described above, the image-improved target image is an image by at least one of the compression decoding operation processing module 270, the image preprocessing operation module 272, and the super-resolution operation processing module 274. It may include processed images. The image and/or image generated through the image generation module 262 may be transmitted to the image storage unit 220 and stored, or may be provided to the display device 250 to provide to a user.

The sensory content providing apparatus 120 may provide immersive sensory content to a user by determining an ROI based on motion information of the user and performing an image processing operation based on the determined ROI. Furthermore, by processing only the region of interest in high resolution, the required amount of computation is minimized, thereby reducing the load on digital rendering.

3 is an exemplary view in which a head mounted display (HMD) device 310 according to an embodiment of the present disclosure is provided as a sensory content providing device. In this embodiment, the head-worn display device 310 may be configured to include the same or similar configuration of the sensory content providing device 120. The head-worn display device 310 may be configured to support an AR image and/or a VR image, and may include a display capable of displaying such an image. Accordingly, a user wearing such a head-worn display device 310 can apply for immersive content using a display provided by the head-worn display device 310. The head-worn display device 310 may receive a command signal (eg, a command signal related to an operation such as immersive content selection) through a preset interface (eg, a touch display, a microphone, a motion recognition sensor, etc.). And can be operated according to the received command signal. The head-worn display device 310 is an example of a sensory content providing device, and is not limited thereto, and may be configured in various forms that can be worn on the head of a human body, such as a glasses type, a helmet type, and a hat type.

As shown in FIG. 3, the display of the sensory content providing apparatus may be arranged to correspond to at least one of the user's right eye and the left eye, and thus an image may be directly output in front of the user's eyes. As described with reference to FIG. 2, the head-worn display device 310 is configured to include a motion detector to detect the movement of a user's head wearing the head-worn display device 310. In an embodiment, the motion detector of the head-worn display device 310 may detect movement of the user's head moving in the x-axis, y-axis and z-axis with respect to the center of the user's head. Here, the user's head movement may include at least one of a head direction and a head tilt. The user's region of interest may be determined based on the measured head movement.

FIG. 4 is an exemplary diagram when the sensory content provided by the sensory content providing apparatus 410 according to an embodiment of the present disclosure is displayed in a three-dimensional space. The immersive content providing device 410 receives images (eg, virtual reality (VR) images, augmented reality (AR) images, etc.) provided by a plurality of external devices, or stores images of the immersive content providing device Realistic content can be provided to the user by using images stored in the unit. As shown in FIG. 4, a user may wear a sensory content providing device 410 (eg, a head mounted display (HMD)) to view an image provided on the display. In this embodiment, the sensory content providing device 410 may be configured to include the same or similar configuration of the sensory content providing device 120.

The sensory content provided by the sensory content providing device 410 may be a virtual reality (VR) image. Such a virtual reality (VR) image may be a panoramic image and/or an image in order to provide a user with maximized sense of vitality and immersion. According to an embodiment, the VR image may be a hemispherical panoramic image 420 that supports viewing in all directions (up/down/left/right directions) with the user as a central axis. For example, the panoramic image 420 may be a 360 degree image supporting a 360 degree viewer. A 360-degree image supporting a 360-degree viewer may be output to a user through a display device of the immersive content providing device 410 and may include a target image 440 corresponding to the region of interest 430. Here, the ROI 430 may correspond to a target image 440 which is a partial image of an image output on the display. For example, the 360-degree image may include a target image 440 that is a partial image of a 360-degree image corresponding to the region of interest 430 determined through the user's movement, as illustrated. The target image 440 may include an image that has been image-improved by the image enhancement module of the sensory content providing apparatus 410. In addition, the target image 440 may be changed and displayed in real time based on the head movement and/or gaze information of a person wearing the sensory content providing device 410. In this case, the image enhancement module of the sensory content providing apparatus 410 may perform an image enhancement process on the partial image corresponding to the region of interest 430 in real time and provide it to the display device. The virtual reality (VR) image described above may be configured to give a user a feeling of being in a virtual reality space, such as a hemispherical shape, a spherical shape, and a cylindrical shape, depending on the scene presentation.

5 is an exemplary view showing a range 510 of a region of interest that can be defined based on an angle of a field of view viewed by a user according to an embodiment of the present disclosure. In one embodiment, the range 510 of the region of interest may be defined based on the angle of view viewed by the user, that is, the viewing angle. Here, the range 510 of the region of interest is defined by the position of the head (eg, head direction and/or head inclination) and/or the position of the line of sight (eg, the position of the pupil) detected by the motion detector. Can be. As illustrated in FIG. 5, a user viewing an image may have a certain viewing angle. Typically, the range 520 of the viewing angle may vary according to the positions of both eyes, and thus the viewing angle is different for each individual. Accordingly, the range 520 of the viewing angle may be configured to be defined based on the position of both eyes (eg, the position of the pupil). In one embodiment, the range 520 of the viewing angle may be defined based on the position of the user's head (eg, head direction, head tilt) and/or the position of both eyes. As an example, the range of the viewing angle in which both eyes of a person are combined may have 180 degrees in a horizontal direction and 120 degrees in a vertical direction as shown, but is not limited thereto and may be defined as various angles.

The range 510 of the region of interest may be determined through the position of the head and/or both eyes detected by the motion detector, and may be defined equal to or smaller than the range 520 of the viewing angle. For example, the range 510 of the ROI may be defined to be smaller than the range 520 of the viewing angle (here, 180 degrees in the horizontal direction and 120 degrees in the vertical direction). The motion detector may detect the position of the head and/or the position of the pupil, and the position of the user's gaze and the direction of the gaze, and the range 510 of the region of interest through the detected position of the head and/or the pupil. You can decide. According to an embodiment, as shown in FIG. 5, a rectangle 530 circumscribed to the shape of the face may be generated by detecting the position of the user's head (eg, skull), and Positions (a, b, c, d) can be detected. An extension of a line connecting the detected four vertices (a, b, c, d) and the center point (e) of the user's occipital region and a location (f, g, h, i) meeting on the display may be detected. The range 510 of the region of interest may be determined based on the detected positions f, g, h, and i. As illustrated, a region connecting four points f, g, h, and i on the display may be determined as the range 510 of the region of interest. The process of defining the range 520 of the viewing angle and the range 510 of the region of interest is not limited to the examples described above, and can be defined in various ways.

6 is an exemplary diagram illustrating a process of image enhancement processing an ROI determined based on a user's gaze detected by a motion detector according to an exemplary embodiment of the present disclosure. In FIG. 6, the sensory content device 120 is worn on the user's head, but is omitted for explanation of the gaze. The ROI image extraction module 264 may determine the position of the user's gaze on the display device based on the position of the pupil detected by the motion detector 230.

In one embodiment, the ROI image extraction module 264 is the right eye in the eye line direction 630 of the left eye 610 and the pupil position 622 of the right eye 620 at the position 612 of the pupil of the left eye 610. A point j where the gaze direction 640 of 620 meets on the display may be detected to determine the location point j of the user's gaze. Here, the gaze direction 630 of the left eye 610 indicates the direction in which the left eye 610 gazes, and the gaze direction 640 of the right eye 620 indicates the direction in which the right eye 620 gazes. The ROI image extraction module 264 may determine the region of interest 652 by designating a position of the center point of the range 650 of the region of interest 652 as a position point j of the line of sight. As shown in FIG. 6A, the region of interest 652 to be projected on the display may be determined based on the position (j) of the user's gaze, and the image enhancement module 266 has a purpose corresponding to the determined region of interest 652. Image enhancement processing may be performed on an image. The image enhancement module 266 may, for example, increase the resolution of the ROI 652. Referring to FIG. 6B, it can be seen that the resolution of the ROI 662 is increased compared to the ROI 652 illustrated in FIG. 6A.

On the other hand, as shown in FIG. 6C, when it is determined that the user's head and/or pupil moves from the left direction to the right direction (here, when the position of the user's gaze moves from point j to point k) , Based on the user's movement, the ROI 682 may be newly determined. Even in this case, as described in FIG. 6A, the point k may be determined as the location point of the user's gaze based on the gaze direction 660 of the left eye 610 and the gaze direction 670 of the right eye 620, The region of interest 682 may be determined by designating the position k of the center point of the determined range 680 of the region of interest 682 as the position point k of the line of sight. In contrast, the range of the ROI 682 is not predetermined and may be changed according to the user's gaze.

As described above, the image enhancement module 266 may perform image enhancement processing on the target image corresponding to the newly determined ROI 682. For example, a super-resolution operation may be performed on a target image corresponding to the region of interest 682. Referring to FIG. 6D, it can be seen that the resolution of the region of interest 692 subjected to the super-resolution operation is higher than that of the region of interest 682 of FIG. 6C and thus becomes clearer. By improving only the image quality of the determined region of interest, the amount of computation required for image processing can be minimized, and accordingly, the reaction speed of an image (for example, immersive content) provided to a user can be increased. Accordingly, it is possible to provide a user with realistic content that is natural and highly immersive.

In the above, it has been described that the image enhancement module 266 performs image enhancement processing to increase the resolution of the target image corresponding to the region of interest, but is not limited thereto, and various images related to image enhancement such as compression decoding operations and preprocessing operations described above are described. The operation can be processed. According to an embodiment, the image enhancement module 266 does not perform image enhancement processing only on the target image corresponding to the region of interest, but performs image enhancement processing on the entire image. Image enhancement can be performed on the.

7 is an exemplary diagram illustrating a process of image enhancement processing an ROI determined based on a user's gaze sensed by the motion detector 230 according to another exemplary embodiment of the present disclosure. In FIG. 7, the sensory content device is worn on the user's head, but is omitted for explanation of the gaze.

In an embodiment, the sensory content device may determine an ROI based on motion and gaze information detected by a motion detector, and perform image enhancement processing according to each ROI. According to an embodiment, the ROI image extraction module 264 may determine the region of interest based on the motion detected by the motion detector. As described in FIG. 5, the ROI image extraction module 264 detects the position of the user's head (eg, skull) based on at least one of a head direction and a head inclination, and displays the region of interest 752 on the display. Range 750 can be determined. Based on the range 750 of the ROI 752, as shown in FIG. 7A, the ROI 752 may be determined based on the head direction and the head tilt.

In addition, the ROI image extraction module 264 is the right eye 720 at the position 712 of the pupil of the left eye 710 and the gaze direction 730 of the left eye 710 and the position 722 of the pupil of the right eye 720. Regions of interest 760 and 770 of the left eye 710 and the right eye 720 may be determined by detecting the points l and m where each of the gaze directions 740 meet on the display.

In one embodiment, the image enhancement module 266 determines a ranking of each ROI, and performs an image enhancement processing operation (eg, super-resolution calculation, compression decoding) for each ROI in stages based on the determined ranking. Calculation, preprocessing calculation, etc.) can be performed. For example, the image enhancement module 266 includes a region 780 where the regions of interest of the left and right eyes overlap, regions of interest 760 and 770 of the left and right eyes, respectively, and the region of interest 752 based on head direction and head tilt. You can determine the ranking of the user's interest area in the order of.

According to an embodiment, as shown in FIG. 7B, based on the determined ranking, the image enhancement module 266 sets the resolution of the region 780 where the regions of interest of the left and right eyes overlap the highest quality (e.g., 8K). ), each of the regions of interest 760 and 770 of the left and right eyes is of higher quality (e.g., 4K) lower than the resolution of the region 780 where the regions of interest overlap, and the region of interest 752 based on the head direction and head tilt The resolution of) may be rendered with a higher quality (eg, 4K) lower than that of the ROI 760 and 770 of the left and right eyes. The image enhancement processing operation is performed according to each region of interest to provide a user with immersive content maximizing a sense of vitality and immersion.

FIG. 8 illustrates a process of providing an augmented reality environment by synthesizing an augmented reality image to a region of interest determined based on a user's gaze sensed from a motion sensor of the sensory content providing apparatus 800 according to an embodiment of the present disclosure. It is an exemplary diagram shown. In this embodiment, the sensory content providing device 800 may be configured to include the same or similar configuration of the sensory content providing device 120.

The ROI image extraction module of the immersive content providing apparatus 800 includes an ROI image that is to be displayed on the display device from an image received based on at least one of the user's head movement and gaze detected by the motion detector. Region of Interest) can be determined. In an embodiment, by detecting the locations 812 and 822 of the user's pupil, a point l where the gaze directions of the left eye 810 and the right eye 820 meet on the display may be detected. As illustrated in FIG. 8B, a point on the display that is skewed to the left may be determined as a position point (l) of the line of sight, and an ROI 852 may be determined based on the position point (l). In FIG. 8B, it has been described that the region of interest is determined based on the user's gaze, but the present invention is not limited thereto, and the region of interest may be determined based on the user's head movement or the head movement and the gaze.

In one embodiment, the sensory content providing apparatus 800 may include an image sensor configured to capture an image of an actual reality to be displayed to a user. Here, as illustrated in FIG. 8A, the image sensor may change the actual reality to be displayed to the user according to the movement of the head and/or the line of sight of the person. In addition, the sensory content providing apparatus 800 may extract an area corresponding to the ROI image extraction module 852 received from the ROI image extraction module as a target image from the captured image. The image generation module of the immersive content providing apparatus 800 synthesizes the augmented reality image 860 corresponding to the target image with the captured real reality image, and displays the synthesized image as shown in FIG. 8C. Can be provided to the device. According to an embodiment, before synthesizing the augmented reality image 860 with an image of a captured real reality, the augmented reality image 860 may be processed for image improvement by an image enhancement module of the immersive content providing apparatus.

According to another embodiment, the sensory content providing apparatus 800 may be configured to include a transparent glass configured to see real reality through the human eye. Here, the display device of the sensory content providing apparatus 800 may be implemented on transparent glass. That is, the user can not only see the actual reality through the transparent glass using his/her eyes, but also view the augmented reality image output by the display device on the transparent glass. In this embodiment, the region of interest 852 determined by the ROI image extraction module of the sensory content providing apparatus 800 may include a region to be displayed on the transparent glass. The image generation module of the immersive content providing device 800 generates an augmented reality image 860 and displays the generated augmented reality image 860 in the region of interest determined through the display device to display the augmented reality image 860 in the general field of view. Can overlap. For example, the augmented reality image 860 may be processed for image enhancement by the image enhancement module of the sensory content providing apparatus 800 before being displayed on the display device.

9 is a flowchart illustrating a method of providing immersive content by a head-worn display device according to an embodiment of the present disclosure. Here, the head worn display device may be configured to include the same or similar configuration of the sensory content providing device 120.

In the sensory content providing method 900, the sensory content providing apparatus 120 may perform an operation 910 of receiving a virtual reality video. According to an embodiment, the controller of the sensory content providing device may receive a virtual reality image from an external device by a communication module based on a request to provide sensory content. According to another embodiment, the controller may read the stored virtual reality image from the image storage unit. Here, the request for providing sensory content may be an operation of selecting the sensory content that the user wants to watch through a preset interface.

After receiving the virtual reality image, an operation 920 of detecting at least one of a head movement and a gaze of a user wearing a head worn display may be performed. At least one of a movement of a user's head and a gaze may be detected using a motion detector provided in the head worn display.

Thereafter, in step 930, the region of interest may be determined based on at least one of the sensed movement and gaze. In an embodiment, the motion detector may detect the movement of the user's head and transmit information on the movement of the head to the ROI image extraction module. The ROI image extraction module may determine at least one of a user's head direction and a head inclination based on the received head motion information, and determine an ROI based on at least one of the determined user's head direction and head inclination. According to another embodiment, the motion detector may be configured to detect the user's pupil position, determine the position of the user's gaze on the display based on the detected pupil position, and based on the determined user's gaze position Area can be determined. According to another embodiment, the motion detector may be configured to determine the region of interest in consideration of both the movement of the user's head and the position of the pupil.

After the ROI is determined, the image enhancement module of the sensory content providing apparatus 120 may perform an image improvement process 940 on a portion corresponding to the ROI in the received virtual reality image. According to an embodiment, when the received virtual reality image is a compressed image, the image enhancement module may perform compression decoding processing on a portion of the virtual reality image corresponding to the determined region of interest. According to another embodiment, pre-processing of a portion of a virtual reality image corresponding to the determined region of interest (eg, Denoise, Deblur, High Dynamic Range, Color Tone Mapping, Defog, Brightness, Contrast, Auto White Balance, Back Light Compensation, etc.) Can perform operations. According to another embodiment, a super resolution operation may be performed on a portion of an image of a received virtual reality corresponding to an ROI. It is possible to provide a high-quality image to the user by image enhancement processing only in the region of interest of interest to the user. Alternatively, the image enhancement module may be configured to perform a plurality of operation processing among compression decoding operation processing, pre-processing, and super-resolution operation processing for a portion corresponding to the region of interest.

Lastly, the control unit may perform step 950 of providing immersive content to the user by outputting an image including a portion of the virtual reality image on which the image enhancement processing has been performed to the head worn display device.

The above-described sensory content providing apparatus may be implemented as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable recording medium is distributed over a computer system connected through a network, so that computer-readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the above embodiments can be easily inferred by programmers in the technical field to which the present invention belongs.

The techniques described herein may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those of skill in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, It may be implemented in a computer, or a combination thereof.

Accordingly, various illustrative logic blocks, modules, and circuits described in connection with the disclosure herein may be a general purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or It may be implemented or performed in any combination of those designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in connection with the DSP core, or any other such configuration.

In the firmware and/or software implementation, the techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), PROM ( on a computer-readable medium such as programmable read-only memory), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage device, etc. It can also be implemented as stored instructions. The instructions may be executable by one or more processors, and may cause the processor(s) to perform certain aspects of the functionality described herein.

When implemented in software, the functions may be stored on a computer readable medium as one or more instructions or codes or transmitted through a computer readable medium. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a computer. By way of non-limiting example, such computer-readable medium may contain RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or the desired program code in the form of instructions or data structures. It may include any other medium that may be used for transfer or storage to and accessible by a computer. Also, any connection is properly termed a computer-readable medium.

For example, if the software is transmitted from a website, server, or other remote source using wireless technologies such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or infrared, wireless, and microwave, coaxial cable , Fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, wireless, and microwave are included within the definition of the medium. As used herein, disks and disks include CDs, laser disks, optical disks, digital versatile discs (DVDs), floppy disks, and Blu-ray disks, where disks are usually magnetic It reproduces data optically, while discs reproduce data optically using a laser. Combinations of the above should also be included within the scope of computer-readable media.

The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other type of storage medium known in the art. An exemplary storage medium may be coupled to a processor such that the processor can read information from or write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and storage medium may also reside within the ASIC. The ASIC may exist in the user terminal. Alternatively, the processor and storage medium may exist as separate components in the user terminal.

The previous description of the present disclosure is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications of the present disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to various modifications without departing from the spirit or scope of the present disclosure. Accordingly, this disclosure is not intended to be limited to the examples described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Although exemplary implementations may refer to utilizing aspects of the currently disclosed subject matter in the context of one or more standalone computer systems, the subject matter is not so limited, but rather is associated with any computing environment, such as a network or distributed computing environment. It can also be implemented. Furthermore, aspects of the presently disclosed subject matter may be implemented in or across multiple processing chips or devices, and storage may be similarly affected across multiple devices. Such devices may include PCs, network servers, and handheld devices.

Although the subject matter has been described in language specific to structural features and/or methodological actions, it will be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and acts described above are described as an exemplary form of implementing the claims.

Although the method mentioned in this specification has been described through specific embodiments, it is possible to implement it as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable recording medium is distributed over a computer system connected through a network, so that computer-readable codes can be stored and executed in a distributed manner. Further, functional programs, codes, and code segments for implementing the embodiments can be easily inferred by programmers in the technical field to which the present invention belongs.

In the present specification, the present disclosure has been described with reference to some embodiments, but various modifications and changes may be made without departing from the scope of the present disclosure as understood by those of ordinary skill in the art to which the present disclosure belongs. In addition, such modifications and changes should be considered to fall within the scope of the claims appended to this specification.

110_1 to 110_n: external device
120: immersive content providing device
130: communication network
210: communication module
220: image storage unit
230: motion detector
240: image sensor
250: display device
260: control unit
262: image generation module
264: ROI image extraction module
266: image enhancement module
270: compression decoding operation processing module
272: image preprocessing operation module
274: super-resolution operation processing module

Claims (10)

In the apparatus for providing immersive content,
A motion detector configured to detect at least one of a user's head movement and a gaze;
An image sensor configured to take an image of an actual reality to be displayed to the user;
Receives an image photographed through the image sensor, determines a region of interest in the image based on at least one of motion and gaze detected from the motion detector, and corresponds to the determined region of interest from the image An ROI image extraction module configured to extract a target image;
Receives an image including an augmented reality image, performs a pre-processing operation and a super resolution operation on the augmented reality image corresponding to the extracted target image, and a plurality of image parameters of the extracted target image At least one of the Demosaicing technique, Wide Dynamic Range (WDR), High Dynamic Range (HDR) technique, Deblur technique, Denoise technique, Color Tone mapping technique, White Balance technique, and Decompression technique using the learned artificial neural network for preprocessing An image enhancement module configured to perform image enhancement processing on the augmented reality through the augmented reality; And
And a display device configured to output the image including a target image obtained by synthesizing the augmented reality image on which the image enhancement processing has been performed. The apparatus of claim 1, wherein the image sensor is arranged to change the actual reality to be displayed to the user according to the movement of the user's head.
delete delete The method of claim 1,
The ROI image extraction module,
Determine at least one of the user's head direction and head tilt based on the sensed movement,
The apparatus for providing immersive content, further configured to determine the region of interest based on at least one of the determined user's head direction and head tilt.
The method of claim 1,
The motion detector,
Further configured to detect the location of the user's pupil,
The ROI image extraction module,
Determine the position of the user's gaze on the display based on the detected position of the pupil,
The apparatus for providing immersive content, further configured to determine the region of interest based on the determined position of the user's gaze.
The method of claim 1,
The image enhancement module includes a super resolution operation processing module configured to perform a super resolution operation processing on an augmented reality image corresponding to the target image.
delete The method of claim 1,
The image enhancement module includes an image preprocessing operation module configured to perform a preprocessing operation on an augmented reality image corresponding to the target image.
In a method of providing immersive content by a head mounted display device,
Detecting at least one of a head movement and a gaze of a user wearing the head-worn display device;
Photographing an image of an actual reality to be displayed to the user;
Determining a region of interest based on at least one of the sensed motion and gaze in the captured image;
Extracting a target image corresponding to the determined ROI from the image;
Receives an image including an augmented reality image, performs a pre-processing operation and a super resolution operation on the augmented reality corresponding to the extracted target image, and determines a plurality of image parameters of the extracted target image. Using the learned artificial neural network for preprocessing, at least one of the Demosaicing technique, WDR (Wide Dynamic Range), HDR (High Dynamic Range) technique, Deblur technique, Denoise technique, Color Tone mapping technique, White Balance technique, and Decompression technique And performing image enhancement processing on the augmented reality image for the augmented reality; And
And outputting, on a display of the head-worn display device, an image including a target image obtained by synthesizing the augmented reality image on which the image enhancement processing has been performed.

Images