KR20200028067A - Image processing method and apparatus using region-of-interest information in video contents - Google Patents

Abstract

An image display method disclosed in the present specification can comprise: receiving stream data including image data of a plurality of tile images generated based at least in part on a region of interest of a user; performing low-pass filtering on at least a part of the tile images based on the region of interest; and instructing to display the entire tile images on a display device after performing low-pass filtering.

Description

IMAGE PROCESSING METHOD AND APPARATUS USING REGION-OF-INTEREST INFORMATION IN VIDEO CONTENTS}

The present specification relates to image processing using information on a region of interest in image content.

Recently, with the development of virtual reality (VR) technology and equipment, wearable devices such as Head-Mounted Display (HMD) are being released through various companies. Service scenarios through HMD may include movie viewing, gaming, telesurgery, and video conferencing.

A virtual reality service is a service that makes a specific environment or situation into a computer and makes the user feel as if they are interacting with the actual surrounding situation and environment.

There is a need for a technology that provides high-quality natural media content at a high speed on a device that provides a virtual reality service.

In addition, in addition to the high-quality media content required to provide a virtual reality service, there is an increasing demand for ultra-high resolution images such as 8K and 16K, and thus the need to efficiently transmit these ultra-high resolution images has emerged.

In the prior art virtual reality image providing apparatus, the entire image is provided by dividing it into tile images, and the image for the region of interest is provided by differently the image quality from the image outside the region of interest. However, due to a difference in image quality between tile images, a problem that causes inconvenience to a user's viewing occurs, and it is necessary to solve this.

In addition, it is necessary to provide a natural image while processing the quality of the displayed screen differently by solving a problem in that an image display speed is reduced due to a change in a region of interest of the user and an unnatural image is provided.

In addition, there is a need for a method and apparatus for differently processing the quality of the screen displayed to the user according to the intention of the creator of the media content.

This specification presents an image display method. The image display method includes receiving stream data including image data of a plurality of tile images generated based at least in part on a user's region of interest, and low-pass filtering on at least a portion of the tile image based on the region of interest And an operation of instructing to display the entire tile image on the display device after low-pass filtering.

The method and other embodiments may include the following features.

The low-pass filtering operation may perform low-pass filtering of image data of a tile image not related to the region of interest.

Further, the degree of low-pass filtering may be set based on a distance from the region of interest.

Also, the low-pass filtering operation may be an operation performed in units of tiles or pixels.

In addition, the method may further include determining a boundary of the user's field of view based on the region of interest.

Further, the low-pass filtering operation may be performed based on the viewing boundary line, but may be an operation of performing low-pass filtering on tiles or pixels in an area outside the viewing boundary line.

Further, the degree of low-pass filtering may be set based on a distance from the field boundary.

In addition, the plurality of tile images may include at least one extended tile image having image data of at least a part of the expanded region than the tile region.

In addition, the image display method may further include compressing the expanded tile image into the interior of the tile region so that the image quality continuously changes at the boundary of the tile region.

Also, in the extended tile image, neighboring extended tile images may not overlap extended regions.

In addition, the operation of compressing the extended tile image into the interior of the tile area may include image data of the extended area and image data of a tile area symmetric with the extended area based on a boundary between the extended area and the tile area. Based on the may be compressed into the symmetrical region.

Meanwhile, this specification presents an image processing apparatus. The image processing apparatus is a communication module that receives stream data including image data of a plurality of tile images generated based on a user's region of interest, and performs low-pass filtering on at least a portion of the tile image based on the region of interest. It may include a low-pass filter, and a control unit instructing to display all tile images on the display module after low-pass filtering.

The device and other embodiments may include the following features.

The low-pass filter performs low-pass filtering on image data of a tile image not related to the region of interest, and the degree of the low-pass filtering may be set based on a distance from the region of interest.

In addition, the image processing apparatus may further include a region of interest information setting unit that determines a user's field of view boundary based at least in part on the region of interest.

In addition, the plurality of tile images includes at least one extended tile image having image data of at least a part of an extended region than the tile region, and the image processing apparatus expands the image quality to continuously change at a boundary of the tile region. A decoder that compresses the tile image into the interior of the tile area may be further included.

In addition, the extended tile image may have image data of an area extended by a predetermined portion from at least a part of a boundary of the tile area, and neighboring extended tile images may be tile images in which the extended areas do not overlap.

According to the present invention, tile images of different qualities may be connected through fitting of an extended tile image to a tile image to generate an entire image capable of reducing user discomfort.

In addition, according to the present invention, by setting different image quality inside and outside the user's field of view, it is possible to generate a whole image capable of reducing the discomfort that a user feels when viewing.

In addition, according to the present invention, when tiles having different resolutions are adjacent to each other, the unnaturalness of an image at a tile boundary may be removed to provide a natural image to a user.

In addition, the present invention extracts the movement of the pupil of the user and the movement of the sound in the virtual reality space, and predicts the intention of the user's region of interest based on the media content creator's intention, user movement, and motion vector statistics information. Video can be provided.

In addition, the present invention can solve the discomfort and discomfort of the service for the virtual reality, augmented reality, or hologram of the user by transmitting and providing high-resolution natural media content based on the region of interest.

1 is a diagram illustrating an electronic system according to an embodiment.
2 is a block diagram illustrating an electronic device according to an embodiment.
3 is a block diagram of a program module according to an embodiment.
4 is a diagram illustrating a configuration of a system for transmitting and receiving media content through an IP network according to an embodiment.
5 is a diagram showing the configuration of a server according to an embodiment.
6 is a diagram illustrating an entire image encoded with various qualities according to an embodiment.
7 is a diagram for describing motion vector statistical information, producer intention information, and region of interest ranking information according to an embodiment.
8 is a conceptual diagram for receiving tile images encoded with different qualities according to an embodiment.
9 is a view showing a method of removing an edge effect according to an embodiment.
10 is a diagram illustrating a method of compressing an extended tile image into the interior of a tile area.
11 is a diagram illustrating an example of a method of spatially compressing an extended region by a weighted distribution method.
12 is a diagram illustrating an example of displaying a tile image based on a region of interest and a boundary line of view.
13 is a diagram illustrating a procedure of a method of displaying a tile image based on a region of interest and a viewing boundary.
14 is a diagram illustrating a configuration of an electronic device (or a media content receiving device) according to an embodiment.
15 is a diagram illustrating a method for receiving media content according to an embodiment.
16 is a diagram illustrating a sequence of a method for displaying tile images of different qualities through tile image expansion.

The techniques disclosed herein can be applied to electronic processing systems. However, the technology disclosed in this specification is not limited thereto, and may be applied to any encryption and decryption apparatus and method to which the technical spirit of the technology can be applied.

It should be noted that the technical terms used in this specification are only used to describe specific embodiments, and are not intended to limit the spirit of the technology disclosed herein. In addition, technical terms used in this specification should be interpreted as meanings generally understood by a person having ordinary knowledge in the field to which the technology disclosed in this specification belongs, unless defined otherwise. It should not be interpreted as a comprehensive meaning or an excessively reduced meaning. In addition, when the technical terms used in this specification are incorrect technical terms that cannot accurately represent the spirit of the technology disclosed in the present specification, technical terms that can be correctly understood by those skilled in the art to which the technology disclosed in this specification belongs Should be understood. In addition, the general terms used in the present specification should be interpreted as defined in the dictionary or in context before and after, and should not be interpreted as an excessively reduced meaning.

Terms including ordinal numbers such as first and second used in the present specification may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from other components. For example, the first component may be referred to as a second component without departing from the scope of rights disclosed herein, and similarly, the second component may also be referred to as a first component.

Hereinafter, exemplary embodiments disclosed herein will be described in detail with reference to the accompanying drawings, but the same or similar elements will be given the same reference numbers regardless of the reference numerals, and redundant descriptions thereof will be omitted.

In addition, in the description of the technology disclosed in the present specification, when it is determined that the detailed description of the related known technology may obscure the gist of the technology disclosed herein, the detailed description will be omitted. In addition, it should be noted that the accompanying drawings are only for facilitating understanding of the spirit of the technology disclosed in this specification, and should not be interpreted as limiting the spirit of the technology by the accompanying drawings.

1 is a diagram illustrating an electronic system according to an embodiment.

Referring to FIG. 1, the electronic system 100 may include at least one electronic device 101, 102, 104, a server 106 and / or a network 162.

The electronic device according to the present disclosure may be a device including a communication function. For example, the electronic device includes a smart phone, a tablet personal computer (PC), a mobile phone, a video phone, an ebook reader, a desktop personal computer, and a laptop PC. (laptop personal computer), netbook computer, personal digital assistant (PDA), portable multimedia player (PMP), MP3 player, mobile medical device, camera, or wearable device (e.g. electronics) It may include at least one of a head-mounted device (HMD) such as glasses, an electronic garment, an electronic bracelet, an electronic necklace, an electronic appcessory, an electronic newspaper, or a smartwatch.

According to some embodiments, the electronic device may be a smart home appliance with a communication function. For smart appliances, for example, electronic devices are televisions, digital video disk (DVD) players, audio, refrigerators, air conditioners, vacuum cleaners, ovens, microwave ovens, washing machines, air cleaners, set-top boxes, TVs. It may include at least one of a box (eg, Samsung HomeSync, Apple TV, or Google TV), game consoles, electronic dictionary, electronic key, camcorder, or electronic picture frame.

According to some embodiments, the electronic device may include various medical devices (eg, magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), computed tomography (CT), imaging device, ultrasound device, etc.), navigation device, GPS receiver ( Global positioning system receiver (EDR), event data recorder (EDR), flight data recorder (FDR), automotive infotainment devices, marine electronic equipment (e.g. navigational devices and gyro compasses, etc.), avionics, security It may include at least one of a device, a head unit for a vehicle, an industrial or household robot, an automated teller machine (ATM) of a financial institution, or a point of sales (POS) of a store.

According to some embodiments, the electronic device is a furniture or part of a building / structure including a communication function, an electronic board, an electronic signature receiving device, a projector, or various measurement It may include at least one of a device (for example, water, electricity, gas, or radio wave measurement devices). The electronic device according to the present disclosure may be a combination of one or more of the various devices described above. Also, the electronic device according to the present disclosure may be a flexible device. In addition, it is apparent to those skilled in the art that the electronic device according to the present disclosure is not limited to the aforementioned devices.

The electronic device 101 may include a bus 110, a processor 120, a memory 130, an input / output interface 150, a display 160, and / or a communication interface 170.

At this time, the network 162 may include a LAN connected by a wire, a cable channel network, and all possible methods wirelessly connected.

In some embodiments, the electronic device 101 may omit at least one of the components or additionally include other components.

The bus 110 may include, for example, circuits that connect the components 110-170 to each other and transfer communication (eg, control messages and / or data) between the components.

The processor 120 may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor 120 may perform, for example, calculation or data processing related to control and / or communication of at least one other component of the electronic device 101.

The memory 130 may include volatile and / or non-volatile memory. The memory 130 may store, for example, commands or data related to at least one other component of the electronic device 101. According to one embodiment, the memory 130 may store software and / or programs 140. The program 140 includes, for example, a kernel 141, a middleware 143, an application programming interface (API) 145, and / or an application program (or "application") 147, etc. It may include. At least a portion of the kernel 141, the middleware 143, or the API 145 may be referred to as an operating system (OS).

The kernel 141 is, for example, a system used to execute operations or functions implemented in other programs (eg, the middleware 143, the API 145, or the application program 147). Resources (eg, the bus 110, the processor 120, or the memory 130, etc.) may be controlled or managed. In addition, the kernel 141 may control or manage system resources by accessing individual components of the electronic device 101 in the middleware 143, the API 145, or the application program 147. Interface.

The middleware 143, for example, may act as an intermediary so that the API 145 or the application program 147 communicates with the kernel 141 to exchange data.

Further, the middleware 143 may process one or more job requests received from the application program 147 according to priority. For example, the middleware 143 is a system resource (eg, the bus 110, the processor 120, or the memory 130) of the electronic device 101 in at least one of the application programs 147. Etc.) can be given priority. For example, the middleware 143 may perform scheduling or load balancing for the one or more job requests by processing the one or more job requests according to the priority given to the at least one.

The API 145 is, for example, an interface for the application 147 to control functions provided by the kernel 141 or the middleware 143, for example, file control, window control, and video. It may include at least one interface or function (eg, a command) for processing or character control.

The input / output interface 150 may serve as an interface capable of transmitting commands or data input from a user or other external device to other component (s) of the electronic device 101. Also, the input / output interface 150 may output commands or data received from other component (s) of the electronic device 101 to a user or another external device.

The display 160 may include, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light-emitting diode (OLED) display, or a microelectromechanical machine. Micro-Electro-Mechanical Systems (MEMS) displays, or electronic paper displays. The display 160 may display various contents (for example, text, images, videos, icons, or symbols) to a user. The display 160 may include a touch screen, and may receive a touch, gesture, proximity, or hovering input using, for example, an electronic pen or a user's body part.

The communication interface 170 may establish communication between the electronic device 101 and an external device (eg, the electronic device 102, the electronic device 104, or the server 106). For example, the communication interface 170 may be connected to the network 162 through wireless communication or wired communication to communicate with an external device (eg, the electronic device 104 or the server 106).

The wireless communication may include, for example, cellular communication. The cellular communication is, for example, long-term evolution (LTE), LTE Advance (LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), wireless (WiBro) Broadband), or GSM (Global System for Mobile Communications). Further, the wireless communication may include, for example, short-range communication 164. The short-range communication 164 may include, for example, at least one of wireless fidelity (Wi-Fi), Bluetooth, or near field communication (NFC). In addition, the wireless communication may include a global positioning system (GPS).

The wired communication may include, for example, at least one of a universal serial bus (USB), a high definition multimedia interface (HDMI), a recommended standard232 (RS-232), or a plain old telephone service (POTS). The network 162 may include at least one of a telecommunications network, for example, a computer network (eg, LAN or WAN), the Internet, or a telephone network.

Each of the electronic device 102 and the electronic device 104 may be the same or a different type of device from the electronic device 101. According to one embodiment, the server 106 may include a group of one or more servers. According to various embodiments of the present disclosure, all or some of operations performed in the electronic device 101 may be executed in another one or a plurality of external devices (eg, the electronic devices 102, 104, or the server 106). According to one embodiment, when the electronic device 101 needs to perform a function or service automatically or by request, the electronic device 101 may instead or additionally execute the function or service itself, At least some of the functionality associated therewith may be requested from an external device (eg, electronic device 102, 104, or server 106). The external device (eg, the electronic device 102 or 104, or the server 106) may execute a requested function or an additional function, and deliver the result to the electronic device 101. The electronic device 101 may process the received result as it is or additionally to provide the requested function or service. To this end, for example, cloud computing, distributed computing, or client-server computing technology can be used.

2 is a block diagram illustrating an electronic device according to an embodiment.

The electronic device 201 may include, for example, all or part of the electronic device 101 shown in FIG. 1. The electronic device 200 includes one or more processors (eg, an application processor (AP) 210), a communication module 220, the memory 230, a sensor module 240, an input device 250, and a display 260 ), An interface 270, an audio module 280, a camera module 291, a power management module 295, a battery 296, an indicator 297, and a motor 298.

The processor 210 may control, for example, a plurality of hardware or software components connected to the processor 210 by driving an operating system or an application program, and perform various data processing and calculations. The processor 210 may be implemented with, for example, a System on Chip (SoC). According to one embodiment, the processor 210 may further include a graphic processing unit (GPU) and / or an image signal processor. The processor 210 may include at least some (eg, a cellular module 221) of the components illustrated in FIG. 2. The processor 210 loads and processes instructions or data received from at least one of other components (eg, non-volatile memory) into a volatile memory, and stores various data in a non-volatile memory. You can.

The communication module 220 may have the same or similar configuration to the communication interface 170 of FIG. 1. The communication module 220 includes, for example, a cellular module 221, a Wi-Fi module 223, a Bluetooth module 225, a GPS module 227, an NFC module 228, and a radio frequency (RF) module It may include (229).

The cellular module 221 may provide, for example, a voice call, a video call, a text service, or an internet service through a communication network. According to an embodiment, the cellular module 221 may distinguish and authenticate the electronic device 201 within a communication network using a subscriber identification module (eg, a SIM card) 224. According to one embodiment, the cellular module 221 may perform at least some of the functions that the processor 210 can provide. According to an embodiment, the cellular module 221 may include a communication processor (CP).

Each of the Wi-Fi module 223, the Bluetooth module 225, the GPS module 227, or the NFC module 228 includes, for example, a processor for processing data transmitted and received through a corresponding module. It can contain. According to some embodiments, at least some of the cellular module 221, the Wi-Fi module 223, the Bluetooth module 225, the GPS module 227, or the NFC module 228 (for example, two) The above) may be included in one integrated chip (IC) or IC package.

The RF module 229 may transmit and receive, for example, communication signals (eg, RF signals). The RF module 229 may include, for example, a transceiver, a power amp module (PAM), a frequency filter, a low noise amplifier (LNA), and the like. In addition, the RF module 229 may further include a component for transmitting and receiving electromagnetic waves in a free space in wireless communication, for example, a conductor or a conductor. 2, the cellular module 221, the Wi-Fi module 223, the Bluetooth module 225, the GPS module 227, or the NFC module 228 share one RF module 229 with each other. According to one embodiment, at least one of the cellular module 221, the Wi-Fi module 223, the Bluetooth module 225, the GPS module 227 or the NFC module 228 One can transmit and receive RF signals through separate RF modules.

The SIM card 224 may include, for example, a card including a subscriber identification module and / or an embedded SIM (SIM), unique identification information (eg, an integrated circuit card identifier (ICCID)), or It may include subscriber information (eg, international mobile subscriber identity (IMSI)).

The memory 230 (eg, the memory 130) may include, for example, an internal memory 232 or an external memory 234. The internal memory 232 may include, for example, volatile memory (eg, dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (eg : OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g. NAND flash or NOR flash, etc.) , A hard drive, or at least one of a solid state drive (SSD).

The external memory 234 may be a flash drive, for example, compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (SD-SD), xD ( extreme digital), MMC (MultiMediaCard) or a memory stick. The external memory 234 may be functionally and / or physically connected to the electronic device 201 through various interfaces.

The sensor module 240 may, for example, measure a physical quantity or sense an operating state of the electronic device 201, and convert the measured or sensed information into an electrical signal. The sensor module 240 includes, for example, a proximity sensor 241, a gyro sensor 242, an illuminance sensor 243, a magnetic sensor 244, an acceleration sensor 245, a grip sensor 246, a gesture sensor 247, or at least one of the biological sensors 248. Additionally or alternatively, the sensor module 240 may include, for example, an E-nose sensor, an EMG sensor (electromyography sensor), an EEG sensor (electroencephalogram sensor), or an ECG sensor (electrocardiogram sensor). ), An infrared (IR) sensor, an iris sensor, and / or a fingerprint sensor. The sensor module 240 may further include a control circuit for controlling at least one sensor included therein. In some embodiments, the electronic device 201 further includes a processor configured to control the sensor module 240 as a part of the processor 210 or separately, so that the processor 210 is in a sleep state While in, it is possible to control the sensor module 240.

The input device 250 may include, for example, a touch panel 252, a (digital) pen sensor 254, or a key 256. The touch panel 252 may use, for example, at least one of capacitive, pressure-sensitive, infrared, and ultrasonic methods. Also, the touch panel 252 may further include a control circuit. The touch panel 252 may further include a tactile layer to provide a tactile reaction to the user.

The pen sensor 254 may be, for example, a part of the touch panel or may include a separate sheet for recognition. The key 256 may include, for example, a physical button, an optical key, or a keypad.

The display 260 (eg, the display 160) may include a panel 262, a hologram device 264, or a projector 266. The panel 262 may include the same or similar configuration to the display 160 of FIG. 1. The panel 262 may be implemented, for example, to be flexible, transparent, or wearable. The panel 262 may be configured as one module with the touch panel 252. The hologram device 264 may show a stereoscopic image in the air using interference of light. The projector 266 may display an image by projecting light on a screen. The screen may be located, for example, inside or outside the electronic device 201. According to an embodiment, the display 260 may further include a control circuit for controlling the panel 262, the hologram device 264, or the projector 266.

The interface 270 may include, for example, a high-definition multimedia interface (HDMI), or a universal serial bus (USB). The interface 270 may be included, for example, in the communication interface 170 shown in FIG. 1. Additionally or alternatively, the interface 270 is, for example, an optical interface, D-sub (D-subminiature), mobile high-definition link (MHL) interface, secure (SD) digital (card) / MMC (multi-media card) interface, or an IrDA (infrared data association) compliant interface.

The audio module 280 may convert, for example, a sound and an electrical signal in both directions. At least some components of the audio module 280 may be included, for example, in the input / output interface 150 shown in FIG. 1. The audio module 280 may process sound information input or output through, for example, a speaker 282, a receiver 284, an earphone 286, or a microphone 288.

The camera module 291 is, for example, a device capable of photographing still images and videos. According to an embodiment, one or more image sensors (eg, front sensors or rear sensors), lenses, and an image signal processor (ISP) ), Or a flash (eg, LED or xenon lamp, etc.).

The power management module 295 may manage power of the electronic device 201, for example. According to an embodiment, the power management module 295 may include a power management integrated circuit (PMIC), a charger integrated circuit (IC), or a battery or fuel gauge. The PMIC may have a wired and / or wireless charging method. The wireless charging method includes, for example, a magnetic resonance method, a magnetic induction method, or an electromagnetic wave method, and may further include an additional circuit for wireless charging, for example, a coil loop, a resonance circuit, or a rectifier. have. The battery gauge may measure, for example, the remaining amount of the battery 296, a voltage, current, or temperature during charging. The battery 296 may include, for example, a rechargeable battery and / or a solar battery.

The indicator 297 may display a specific state of the electronic device 201 or a part of the electronic device 201 (for example, the processor 210), for example, a booting state, a message state, or a charging state. The motor 298 may convert an electrical signal into mechanical vibration, and may generate a vibration or haptic effect. Although not illustrated, the electronic device 201 may include a processing device (eg, GPU) for mobile TV support. The processing device for mobile TV support may process media data according to standards such as digital multimedia broadcasting (DMB), digital video broadcasting (DVB), or MediaFlo ^TM .

Each of the components described in this document may be composed of one or more components, and the name of the component may vary depending on the type of electronic device. In various embodiments, the electronic device may include at least one of the components described in this document, and some components may be omitted or further include other components. In addition, since some of the components of the electronic device according to various embodiments are combined and configured as one entity, functions of the corresponding components before being combined may be performed in the same manner.

3 is a block diagram of a program module according to an embodiment.

According to an embodiment of the present disclosure, the program module 300 (eg, the program 140) is an operating system (OS) and / or operating system that controls resources related to an electronic device (eg, the electronic device 101). Various applications (eg, the application program 147) running on the system may be included. Operating systems are based on, for example, Android, iOS, Windows, Symbian, RTOS (Realtime OS), Linux, Linux, WebOS, Peer to Peer (P2P) It may be an OS having characteristics, or a tizen.

The program module 300 may include a kernel 320, middleware 330, an application programming interface (API) 360, and / or an application 370. At least a portion of the program module 300 may be preloaded on an electronic device or downloadable from an external electronic device (eg, external electronic devices 102, 104, server 106, etc.).

The kernel 320 (eg, the kernel 141) may include, for example, a system resource manager 321 and / or a device driver 323. The system resource manager 321 may perform control, allocation, or recovery of system resources. According to an embodiment, the system resource manager 321 may include a process management unit, a memory management unit, or a file system management unit. The device driver 323 may include, for example, a display driver, camera driver, Bluetooth driver, shared memory driver, USB driver, keypad driver, Wi-Fi driver, audio driver, or inter process communication (IPC) driver. You can.

The middleware 330 may provide, for example, a function commonly required by the application 370, or the API 360 to enable the application 370 to efficiently use limited system resources inside the electronic device. ) To provide various functions to the application 370. According to one embodiment, the middleware 330 (eg, middleware 143) includes a runtime library 335, an application manager 341, a window manager 342, and a multimedia manager. manager (343), resource manager (resource manager) 344, power manager (power manager) 345, database manager (database manager) 346, package manager (package manager) 347, connection manager (connectivity) At least one of a manager 348, a notification manager 349, a location manager 350, a graphic manager 351, or a security manager 352 It can contain.

The runtime library 335 may include, for example, a library module used by a compiler to add new functions through a programming language while the application 370 is running. The runtime library 335 may perform input / output management, memory management, or functions for arithmetic functions.

The application manager 341 may manage, for example, the lifecycle of at least one application among the applications 370. The window manager 342 may manage GUI resources used on the screen. The multimedia manager 343 may grasp a format required for reproduction of various media files, and may perform encoding or decoding of a media file using a codec suitable for the format. The resource manager 344 may manage resources such as source code, memory, or storage space of at least one of the applications 370.

The power manager 345 may operate, for example, with a basic input / output system (BIOS) to manage a battery or power, and provide power information necessary for the operation of the electronic device. . The database manager 346 may create, search, or change a database to be used by at least one of the applications 370. The package manager 347 may manage installation or update of an application distributed in the form of a package file.

The connection manager 348 may manage, for example, a wireless connection such as Wi-Fi or Bluetooth. The notification manager 349 may display or notify an event such as an arrival message, an appointment, and proximity notification in a manner that does not disturb the user. The location manager 350 may manage location information of the electronic device. The location information disclosed herein includes not only GPS (Global Positioning System) information, but also acceleration sensor information, gyro sensor information, indoor coordinate sensor information for Location Based Service (LBS), location information estimated based on a network, and location using a communication network. Information, and other location information related to the movement received from the new electronic device may be included. The graphic manager 351 may manage graphic effects to be provided to a user or a user interface related thereto. The security manager 352 may provide various security functions required for system security or user authentication. According to an embodiment, when the electronic device (for example, the electronic device 101) includes a telephone function, the middleware 330 includes a telephony manager for managing a voice or video call function of the electronic device. It may further include.

The middleware 330 may include a middleware module forming a combination of various functions of the above-described components. The middleware 330 may provide modules specialized for each type of operating system in order to provide differentiated functions. Also, the middleware 330 may dynamically delete some of the existing components or add new components.

The API 360 (eg, the API 145) is, for example, a set of API programming functions, and may be provided in different configurations according to an operating system. For example, in the case of Android or iOS, one API set may be provided per platform, and in the case of Tizen, two or more API sets may be provided per platform.

The application 370 (eg, the application program 147) includes, for example, a home 371, a dialer 372, an SMS / MMS 373, an instant message (IM) 374, Browser 375, Camera 376, Alarm 377, Contact 378, Voice Dial 379, Email 380, Calendar 381, Media Player 382, Album 383, or Watch ( 384), one or more applications that can perform functions such as health care (e.g. measuring exercise or blood sugar), or providing environmental information (e.g. providing air pressure, humidity, or temperature information) It can contain.

According to an embodiment of the present disclosure, the application 370 is an application that supports information exchange between an electronic device (for example, the electronic device 101) and an external device (for example, electronic devices 102 and 104). For convenience, it may include "information exchange application"). The information exchange application may include, for example, a notification relay application for delivering specific information to an external device, or a device management application for managing an external device.

For example, the notification delivery application may transmit notification information generated by other applications of the electronic device (eg, SMS / MMS application, email application, health care application, or environmental information application) to an external device (eg, the electronic device 102 , 104)). Also, the notification delivery application may receive notification information from an external electronic device and provide it to a user.

The device management application may include, for example, turn-on / at least one function of an external device (eg, the electronic device 102, 104) communicating with the electronic device (eg, the external device itself (or some component parts)). Turn-off or adjust the brightness (or resolution) of the display, manage applications (e.g. install, delete, or update) applications running on external devices or services provided by external devices (e.g. call service or message service). You can.

According to an embodiment of the present disclosure, the application 370 may include an application (for example, a health care application of a mobile medical device, etc.) designated according to attributes of an external device (eg, the electronic devices 102 and 104). According to an embodiment, the application 370 may include an application received from an external device (eg, the server 106 or the electronic devices 102 and 104. According to an embodiment, the application 370 ) May include a preloaded application or a third party application that can be downloaded from a server.The names of the components of the program module 310 according to the illustrated embodiment may include operating system It may vary depending on the type.

According to various embodiments, at least a portion of the program module 310 may be implemented in software, firmware, hardware, or a combination of at least two or more of them. At least a part of the program module 300 may be implemented (eg, executed) by, for example, a processor (eg, the processor 210). At least a part of the program module 300 may include, for example, a module, program, routine, sets of instructions, or process for performing one or more functions.

As used herein, the term "module" may mean a unit including one or a combination of two or more of, for example, hardware, software, or firmware. The term "module" may be used interchangeably with terms such as, for example, a unit, logic, logical block, component, or circuit. The "module" may be a minimum unit of an integrally configured component or a part thereof. The "module" may be a minimum unit performing one or more functions or a part thereof. The "module" can be implemented mechanically or electronically. For example, a "module" is either an application-specific integrated circuit (ASIC) chip, field-programmable gate arrays (FPGAs) or programmable-logic devices (ASICs) that perform certain operations, known or to be developed in the future. It may include at least one.

At least a part of an apparatus (eg, modules or functions thereof) or a method (eg, operations) according to various embodiments of the present disclosure is, for example, a computer-readable storage media in the form of a program module. It can be implemented as a command stored in. When the instruction is executed by a processor (eg, the processor 120), the one or more processors may perform a function corresponding to the instruction. The computer-readable storage medium may be, for example, the memory 130.

Computer-readable recording media include hard disks, floppy disks, magnetic media (eg magnetic tape), optical media (eg compact disc read only memory (CD-ROM), DVD ( Digital Versatile Discs, magneto-optical media (such as floptical disks), hardware devices (such as read only memory (ROM), random access memory (RAM), or flash memory) In addition, the program instructions may include machine language codes such as those produced by a compiler, as well as high-level language codes executable by a computer using an interpreter, etc. It may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The module or program module according to various embodiments of the present disclosure may include at least one or more of the above-described components, some of them may be omitted, or additional other components may be further included. Operations performed by a module, a program module, or other components according to various embodiments of the present disclosure may be executed in a sequential, parallel, repetitive or heuristic manner. Also, some operations may be executed in a different order, omitted, or other operations may be added.

4 is a diagram illustrating a configuration of a system for transmitting and receiving media content through an IP network according to an embodiment.

Referring to FIG. 4, the media content transmission / reception system 400 according to an embodiment disclosed herein may include a content server 420 and / or a DASH client 440. For example, the content server 420 may be included in the server 106 of FIG. 1. Also, the DASH client 440 may be included in the processors 120 and 210 of the electronic device 101 of FIGS. 1 and / or 2.

The system 400 for transmitting and receiving media content through an IP network according to an embodiment disclosed in the present specification is divided into transmission and reception of transport packets including actual media content and transmission and reception of media content playback information. The DASH client 440 receives media content presentation information and receives a transport packet including media content. At this time, the media content presentation information indicates information necessary for media content reproduction. The media content presentation information may include at least one of spatial information and temporal information required for media content reproduction. The DASH client 440 plays media content based on the media content presentation information.

In a specific embodiment, media content may be transmitted and received through an IP network according to the MMTP (MPEG Media Transport Protocol) standard. At this time, the content server 420 transmits a presentation information document (PI document) including media content presentation information. In addition, the content server 420 transmits an MMT protocol (MMTP) packet including media content based on the request of the DASH client 440. The DASH client 440 receives the PI document. The DASH client 440 receives a transport packet including media content. The DASH client 440 extracts media content from a transport packet including media content. The DASH client 440 plays media content based on the PI document.

In another specific embodiment, media content may be transmitted and received through an IP network according to the MPEG-DASH standard. At this time, the content server 420 transmits a media presentation description (MPD) including media content presentation information. However, according to a specific embodiment, the MPD may be transmitted by an external server other than the content server 50. For example, an external server may include a server for a Content Delivery Network (CDN). CDN refers to a system that stores and provides data in a network having multiple nodes to efficiently deliver large-capacity service data, media data, and / or signaling data. In addition, the content server 420 transmits segments including media content based on the request of the DASH client 440. The DASH client 440 receives the MPD. The DASH client 440 requests media content from the content server 420 based on the MPD. The DASH client 440 receives a transport packet including media content based on the request. The DASH client 440 plays media content based on the MPD.

The DASH client 440 transmits an HTTP request message through the MPD Parser parsing the MPD, the Segment Parser parsing the Segment, the communication interface 170 and / or the communication module 220, and receives an HTTP response message. It may include an HTTP client, a media engine that plays media.

5 is a diagram showing the configuration of a server according to an embodiment.

The server 500 according to an embodiment disclosed in the present disclosure may encode media content differently according to quality, and selectively transmit it to an electronic device. For example, the server 500 may include at least one of the control unit 510 and / or the communication module 520. The controller 510 may include all the contents of the above-described processor. Also, the control unit 510 may be included in the server 500 or may exist independently of the server 500. The details of the communication module 520 may include all of the above-described communication module. Hereinafter, the control unit 510 will be mainly described.

The control unit 510 according to an embodiment disclosed in the present specification may include at least one of an image generation unit 511, an image division unit 513, an encoder 515, and / or a signaling data generation unit 517. have.

The image generator 511 may generate an entire image in which a 3D virtual space is converted into a 2D plane. For example, the image generator 511 may receive image information from at least one electronic device and / or external device, and generate a 3D virtual space. In addition, the image generator 511 may convert the generated 3D virtual space into a 2D plane.

The image division unit 513 may divide the entire image of the two-dimensional plane into a plurality of tile regions. For example, the tile area may be a rectangular area. The tile image may represent an image corresponding to a tile area in the entire image.

The encoder 515 may encode a tile image of each tile area with a plurality of qualities. For example, the encoder 515 may encode the entire image and / or tile image at various bit rates. In addition, the encoder 1615 can use H.264 / AVC and / or HEVC, and in addition to the video format referred to in the existing MPEG-DASH, 3D virtual reality such as hologram and data for augmented reality are also matched. Can transmit.

In addition, the encoder 515 can generate a motion vector. A motion vector is information represented by a relative position of this dimension from a reference picture to a reference block based on the same position as the current block. In inter-frame prediction, the encoder transmits a motion vector of a referenced block to a decoder, so that the encoder and the decoder can generate the same prediction signal.

Also, the encoder 515 may generate an extended tile image. The extended tile image is a tile image (overscan) having image data not only for the original tile region of the tile image but also for the region extended by a predetermined portion (for example, the margin ratio 927 of FIG. 9) in the boundary of the tile region. Image), and may be a tile image in which the size of the image is expanded larger than the original size of the tile image by the preset portion. In addition, the size and shape of the expanding region may be variously modified according to a preset value, and may be variously modified within a range in which the expanded region of the tile to be expanded and the expanded region of the neighboring tile do not overlap. That is, the extended tile images may be variously modified in shape and size of the extended region so that the expanded regions do not overlap each other.

The signaling data generation unit 517 may generate signaling data for acquiring service data of a service for virtual reality, augmented reality, or hologram.

The signaling data may include a media presentation description (MPD) that describes tile images that are encoded into a plurality of qualities for each tile region and an image expanded in various shapes. Also, the service data may include a selected tile image corresponding to a tile image encoded with a specific quality for each tile region.

For example, the media presentation description may include a plurality of adaptation set elements describing a plurality of tile areas. Also, each adaptation set element may include a plurality of representation elements that describe tile images encoded with a plurality of qualities for each tile area. Further, each representation element may include at least one of location information of a tile region, quality information of a tile image, and URL information of a tile image.

The signaling data may further include producer intention information indicating a rank of a tile area that is expected to be viewed by the user or must be viewed by the user based on the intention of the producer of the service for virtual reality, augmented reality, or hologram. . The producer may set a high rank of the tile area corresponding to the region of interest and transmit a tile image encoded with high quality. In addition, the rank of the tile region corresponding to the region other than the region of interest may be set low and a tile image encoded with low quality may be transmitted.

The signaling data may further include motion vector statistical information indicating a representative value of a motion vector for each tile area.

The motion vector statistics information may be added to the file name of the media presentation description, or may be included in elements and / or attributes included in the media presentation description. In addition, the motion vector statistics information may be added to the file name of the metadata table existing separately from the media presentation description, or may be included in elements and / or attributes included in the metadata table.

For example, the motion vector statistical information may exist per time zone and / or per macro block. In addition, the motion vector statistics information may be in a format such as [TimeSeconds, Poi_x, Poi_y, DeltaOf (MV)]. Here, TimeSeconds indicates time information, Poi_x and / or Poi_y indicates coordinate information, and DeltaOf (MV) may indicate a representative value (or average value) of at least one motion vectors in each tile.

The communication module 520 may transmit and / or receive service data, signaling data, and / or additional data for image display to at least one sensor device, electronic device, and / or external device. For example, the communication module 520 may transmit service data including image data for a tile image and an extended tile image based on the request information.

For example, the request information may describe a tile image selected based on signaling data and region of interest information. Also, the region of interest information may indicate a region of interest that is expected to be viewed or viewed by the user in the entire image.

6 is a diagram illustrating an entire image encoded with various qualities according to an embodiment.

Referring to FIG. 6 (a), an entire image 610 divided into at least one tile area is shown.

The server may divide the entire image 610 for a service for virtual reality, augmented reality, or hologram into at least one tile area having a rectangular shape using the image division unit.

For example, the entire image 610 may be an image representing a 360 degree range. Further, the number and / or size of tile areas may be predefined by a user, an electronic device, and / or a server.

Referring to FIG. 6 (b), entire images encoded with various qualities are shown.

The server may encode the entire image and / or at least one tile image with various qualities using an encoder.

For example, the server may encode the entire image 610 to generate a low quality first full image 621, a medium quality second full image 623, and / or a high quality third full image 625. have.

A plurality of tile images included in the same tile area may be described by an adaptation set element. Further, each tile image included in the same tile area may be described by a representation element.

The server according to an embodiment disclosed in the present specification may selectively and / or adaptively transmit a tile image encoded with a specific quality to each tile area based on network conditions and / or user requests.

7 is a diagram for description of motion vector statistical information, producer intention information, and region of interest ranking information according to an embodiment.

Referring to FIG. 7 (a), the entire image 710 and the motion vector 713 are shown.

The motion vector statistical information according to an embodiment disclosed in this specification may indicate a representative value of the motion vector 713 for each tile area. For example, the representative value may be an average value of a plurality of motion vectors present in each tile area.

The electronic device according to an embodiment disclosed in this specification may select a tile image encoded with a specific quality for each tile region based on motion vector statistical information. For example, the electronic device arranges tile regions in order of the highest value of the motion vector statistics information, and receives tile images encoded in high quality for the tile regions 711, 715, and 717 where the values of the motion vector statistics information are large. can do.

An electronic device according to an embodiment disclosed in the present disclosure obtains statistics on a motion vector of an image by using deep learning using a convolutional neural network (CNN) to tile images for each tile The mobility of objects within can be extracted.

In addition, when discriminating 2D / 3D images, it is possible to improve the accuracy of motion / high-frequency image detection and systematize the data to be processed by employing a fractal encoding technique to vectorize existing 2D images.

Statistical accuracy can be increased through a convolutional neural network (CNN) based on vectorized image information data through fractal encoding between frames.

Referring to FIG. 7 (b), the entire image 720, the region of interest 721 viewed by the user, the tile region 723 expected to be viewed by the user based on the user's movement, and the intention of the creator of the media content Based on the, a tile area 725 that the user is expected to look at is shown.

Currently, the user is looking at the region of interest 721, but may move in the direction of the tile region 723 that the user is expected to look at. At this time, the region of interest ranking information may have a high ranking value for the tile region 723 based on the user's movement.

Also, the intention of the creator of the media content may be that the user looks at the tile area 725. At this time, the producer intention information may have a high priority value for the tile area 725 based on the intention of the producer.

The electronic device according to an exemplary embodiment disclosed herein may select a tile image encoded with a specific quality for each tile area based on the region of interest ranking information and / or the maker's intention information. For example, the electronic device arranges the tile regions in the order of the highest value of the region of interest ranking information and / or the producer intention information, and the tile region of the big region of interest region information and / or the producer intention information (723, 725) With respect to, it is possible to receive a tile image encoded in high quality.

8 is a conceptual diagram for receiving tile images encoded with different qualities according to an embodiment.

Referring to the drawings, an entire image 810 encoded with low quality, an entire image 820 encoded with high quality, and an entire image 830 received by the electronic device are shown.

The areas of interest 811, 821, and 831 are areas currently viewed by the user. The first regions of interest 813, 823, and 833 are regions having the highest value of the region of interest ranking information. The second regions of interest 814, 824, and 834 are regions having the highest value of producer intention information. The third regions of interest 815, 825, and 835 are regions having the highest value of motion vector statistical information. The first region of interest, the second region of interest, and the third region of interest may all be expressed as regions of interest.

The electronic device according to an embodiment disclosed herein may selectively receive a tile image based on region of interest ranking information, producer intention information, and / or motion vector statistical information.

For example, the electronic device may receive a tile image encoded in high quality with respect to the ROI 831, the first ROI 833, the second ROI 834, and the third ROI 835. The electronic device may receive a tile image encoded with low quality for the remaining areas.

Hereinafter, with reference to FIGS. 9 to 12, an image display method capable of providing an image in which the heterogeneity of a user is reduced by an edge effect in an entire image composed of tile images having different qualities will be described.

The edge effect means a phenomenon in which two or more adjacent tile images of different qualities have an unnatural border due to a rapid quality difference, or the tile images appear disconnected and disconnected, and the edge effect causes the user (viewer) to It will be inconvenient to watch the entire video composed of video.

An image receiving apparatus (electronic device) according to an embodiment disclosed in the present specification may receive tile image data for all images encoded with a plurality of tile images of multiple quality. The plurality of tile images may include at least one extended tile image having image data for an area extended by a portion of the tile area.

In addition, the image receiving apparatus fits the expanded tile image into the original tile region (the tile region before expansion) of the expanded tile so that the image quality between neighboring tiles at the border of the tile continuously changes, and the tile image Edge effects can be eliminated by combining them to generate an entire image. Here, the method of fitting the extended tile image into the original tile area of the extended tile includes: image information (pixel information) in the original tile area and image information of the extended area in the extended tile image into the tile area before expansion. It may be image processing compressed and displayed.

The above-described image quality may be determined by a bit depth indicating a resolution, color depth, and the like of the image, a frame rate, a bit rate, and a gradation difference.

9 is a diagram illustrating in detail a method of removing an edge effect according to an embodiment.

Referring to FIG. 9A, a region of interest 915 may exist in four tile regions 911, 912, 913, and 914. Also, the image quality of the four tile regions may be the same or different.

If the quality and / or resolution of two adjacent tile regions in the region of interest 915 are different, an edge effect 916 may occur at the boundary portion.

Hereinafter, a method of removing the edge effect 916 according to an embodiment disclosed in the present specification will be described.

9B, a tile image 923 and an extended tile image 925 are shown.

The at least one tile image 923 according to the exemplary embodiment disclosed herein is actually transmitted and / or received as the extended tile image 925 scanned by a margin ratio 927 wider than the tile area (image of the original tile). Can be. That is, the quality may be the same in the area of the extended tile image 925.

The electronic device may receive at least one extended tile image 925 and combine the at least one extended tile image 925 to generate an entire image. In this case, the extended tile images 925 combined with each other may overlap each other by a margin ratio 927. Accordingly, the electronic device may remove edge effects by processing image information so that two tile images are naturally connected to overlapping portions, that is, the quality of the two adjacent tile images is continuously changed at neighboring boundary portions. You can.

Referring to FIG. 9 (c), a processing method for the boundary portion is shown.

The tile image 931 encoded in low quality and the tile image 932 encoded in high quality are adjacent. The extended tile image for the high quality encoded tile image 932 overlaps the low quality encoded tile image 931 as much as the preset portion 933. In addition, the extended tile image for the tile image 931 encoded with low quality overlaps with the tile image 932 encoded with high quality as much as the preset portion 934.

The electronic device according to an embodiment disclosed in the present disclosure may perform a proportional average method, a logistic model applied weight average method, and / or a filtering method on preset portions 933 and 934 in which two extended tile images overlap. .

The proportional average method is a method of obtaining a proportional average value of pixel image values of two adjacent or neighboring tile images by varying pixel weights by an extended distance based on a high-quality tile image. In this case, the boundary portion may have a continuous value, and two tile images may be naturally combined.

The logistic model-applied weight average method is a regression expressed by a logistic regression model equation (940), which is applied when a high-resolution tile overlaps a low-resolution tile and / or applies a weight of a pixel by a predetermined distance to an extended tile area. It is an equation model.

Hereinafter, the logistic model application weight average method will be mainly described with respect to a high-resolution tile and a low-resolution tile, but the method can be equally applied to high-quality tiles and low-quality tiles. The shape of the graph 941 is the most standard slope distribution model. For example, a method of obtaining weights when a tile is a high resolution tile will be described. First, the electronic system can obtain a weight value 942 according to the% position of the high resolution pixel. At this time, the electronic system may grasp the weight 943 according to the% position of the low resolution pixel, which is symmetric to the% position of the low resolution pixel, of the low resolution tile overlapping the high resolution tile. The electronic system multiplies the weight value 942 according to the% position of the high resolution pixel by the pixel corresponding to the distance of the high resolution extended area, and the weight 943 according to the% position of the low resolution pixel is multiplied by the pixel corresponding to the distance of the low resolution extended area, The average pixel value between the two can be obtained. In this way, in the pixel corresponding to the extended area distance in the low-resolution tile, the weight of the low-resolution pixel has a high value, and the weight of the high-resolution pixel has a low value.

The filtering method is a method of calculating and / or correcting a preset portion 933 and 934 in which two extended tile images overlap with a high pass filter. In this case, it is possible to improve the image quality of the border portion.

Accordingly, the electronic device according to an embodiment disclosed herein may process to have continuous image quality in preset portions 933 and 934 in which two extended tile images overlap, thereby eliminating edge effects (935) ).

The range of the extended area of the extended tile image is a preset portion of the tile image (margin ratio 927) for four boundary lines of one tile image when the tile image is rectangular as shown in FIG. 9 (b). It may be an area extended by), or an area extended by the preset portion only for a boundary line 953 neighboring a tile image having a different quality in the tile image 951 as shown in FIG. 9 (d). . The electronic device receives image data corresponding to an extended area on the entire screen.

In the case of FIG. 9 (d), only the left and bottom border lines of the tile image are adjacent to tiles of different quality, and the expanded region is obtained by acquiring image data of the expanded region only for borders adjacent to tiles of different quality. As shown in FIG. 9 (b), it can be reduced by about 50% compared to acquiring data of an extended area for all four boundary lines.

Further, the range of an area for acquiring image data may be extended only for a part of the boundary of the tile image according to the quality of the neighboring tile image or the quality of the tile containing the region of interest, and FIG. 9 ( As shown in e), the range 961 of the extended area may be variously selected, and the tile area may be extended in various shapes so that the range 961 of the extended area does not overlap between different tiles.

Therefore, as shown in FIGS. 9 (d) and 9 (e), when the image to be expanded or the range to be expanded is reduced without acquiring the image data of the extended area for all four boundary lines of one tile image, , Since the amount of data to be processed can be reduced, the image processing speed for removing the edge effect can be increased.

10 is a diagram illustrating a method of compressing an extended tile image into the interior of a tile area.

A method of fitting an extended area to a tile image according to an embodiment disclosed in the present disclosure may be to fit the original tile into the original tile by compressing and compressing the image information of the extended area in a preset ratio in the image of the original tile. For example, the image area occupied by the extended area may be fitted into the original tile in a form of folding and folding based on a boundary line. That is, the image data of the tiled area that is symmetrical to the expanded area may be compressed and inserted into the symmetrical area based on the boundary between the extended area and the tiled area.

Referring to FIG. 10 (a), as shown, the entire area (w + ov) of the original area (w) and the extended area (ov) of the tile image is fit into the original area (w) of the tile image. You can.

The method of folding the space occupied by the extended area to fit the size of the original tile image will be described in detail as follows.

First, a line-symmetrical area is set inside the original tile area based on the boundary of the tile area of the extended tile. Next, the line symmetry area inside the tile area is divided into equal parts as shown. Finally, in the left area of the divided area, image information of the line-symmetrical area, that is, the image information of the area corresponding to the portion where the expanded tile area is folded into the original tile image is spatially 2 to 1 It is compressed and displayed. Also, in the right area of the divided area, image information of the expanded tile area is spatially compressed 2 to 1 and displayed.

Referring to FIG. 10 (b), for example, the width w of the original tile image area 1001 of one tile image is 100 pixels, and the width ov of the extended region 1003 is 5 pixels. Then, an image (w + ov) having a total width of 105 pixels (a picture obtained by combining an area of an original tile image with an extended tile region) is spatially compressed into a 100-pixel area. The area of the portion 1005 in which the expanded area 1003 is folded into the original tile image area is the same as the area ov of the extended area. At this time, 95 pixels of the total width of the original tile image are inserted 1: 1, and the remaining 10 pixel portions (ov + ov) are spatially compressed to 2: 1 in a region of 5 pixels (ov) to fit. can do.

The above-described spatial compression method of the extended region includes a uniform fit method and a weighted fit method incorporating a weight graph, and the weight distribution method will be described below with reference to FIG. 11.

11 is a diagram illustrating an example of a method of spatially compressing an extended region by a weighted distribution method.

For example, as shown in FIG. 11 (a), if the width of the original tile image area is 100 pixels and the width of the extended area is 5 pixels, the 96th pixel is 85% pixel information compared to the original pixel information. Display, the 97th pixel is displayed with 82% pixel information, the 98th pixel is displayed with 75% pixel information, the 99th pixel is displayed with 61% pixel information, and the 100th pixel is 52% pixel information. Pixel fitting weights such as information display are applied, the 101st pixel, which is the area where expansion starts, is displayed as 45% pixel information, the 102th pixel is displayed as 35% pixel information, and the 103rd pixel is 27 A compressed image of a tile image is generated by spatial interpolation using a method of applying pixel fitting weights, such as displaying in% pixel information. Displaying the specific% of the pixel information may mean displaying the image information of the target pixel in the corresponding% image quality or pixel size.

FIG. 11 (b) is a graph showing pixel weights of the spatial interpolation method of FIG. 11 (a).

At this time, adjacent tiles are similarly spaced by applying pixel fitting weights to the left, right, top, or bottom, etc., centering on the tiles. After increasing the original size, the boundary area between tiles is restored. The saved and saved pixels have small pixel energy, but since they are basic information for restoring the original image, it is possible to perform a restoration operation that minimizes the boundary effect.

12 is a diagram illustrating an example of displaying a tile image based on a region of interest and a viewing boundary.

Referring to FIG. 12 (a), when a whole image is displayed with a plurality of tile images generated with a plurality of quality or resolutions, tiles including a region of interest of the user and tiles not including the region of interest are adjacent to each other. A method of displaying a natural image for a case will be described. "High", "Medium", and "Low" indicated on the tiles in the drawing have high image quality such as resolution, frame rate, bit rate, and color depth (bit depth) of the displayed image of the tile, respectively. It represents medium and low, and based on medium, high is higher than medium and low is lower than medium.

For example, when an object 1201 of interest to the user exists in an area in the center of the entire image displayed to the user, a specific region including the object 1201 becomes the user's region of interest, and the region of interest The boundary line 1203 of the user's field of view may be set around. In this case, as illustrated, the inside and outside of the view boundary line are distinguished based on the boundary line 1203 of the user's view field in the entire image.

The region of interest of the user may be determined as an area viewed by the user wearing the HMD or an area expected to be viewed, or may be determined based on a pupil image, a pupil vector, a head vector, etc. generated by measuring with a sensor device provided in the HMD. have.

When the user's region of interest is determined, the tile image of the region not related to the region of interest may be displayed with low quality.

In addition, a user's visual field boundary 1203 may be set based on the region of interest, and a low-quality image may be displayed on an image outside the set visual field boundary 1203.

The viewing boundary line 1203 may be set as a shape surrounding a region of interest with a predetermined shape and / or size centering on the region of interest. The predetermined shape may be various shapes such as a shape having rounded corners in a circle, ellipse, or rectangle. The predetermined size takes into account factors related to the user's field of view (eg, viewing angle, distance from the eye to the display, size of the eye, etc.) so that the user can comfortably perceive the image within the field of view. It can be set to include a range that can be, it can be set based on the average user's visual field related data derived separately. In addition, the viewing boundary may be manually set by the user to reflect the unique characteristics of the user. The user's field of view may be continuously changed according to the head movement information of the HMD wearer, or may be set by changing within the entire image according to a change in the direction of the user's eyes. In addition, the viewing boundary line 1203 may be set to include a specific range based on a point viewed by the user regardless of whether or not the object 1201 in the region of interest exists.

At this time, a high quality or high resolution image is displayed on a tile including the object 1201 in the entire image, and a tile including the view boundary line 1203 but not the object 1201 is of medium quality or medium resolution. An image can be displayed. In addition, a low quality image may be displayed on a tile that does not include the viewing boundary line 1203 among tiles outside the viewing boundary line 1203. In this case, due to a sharp difference in image quality between tiles displaying the medium-quality or high-quality image including the viewing boundary line 1203 and tiles displaying the low-quality image outside the viewing boundary line 1203, the user may be in a neighborhood. You can feel the unnaturalness of the connecting portion of the tile video and the inconvenience of viewing.

Therefore, in order to reduce the inconvenience of the user due to the sudden difference in the above-described image quality, the image outside the viewing boundary line 1203 is subjected to low-pass filtering processing, that is, lowering the resolution or displaying the original image as a blurred image. As a method of reducing the image quality of the back, an image outside the user's field of view can be displayed by blurring, and an image in the user's field of view can be displayed with the original quality of the tile image. Therefore, it is possible to allow the user to watch a natural image, and to speed up the processing of image data for displaying a region of interest.

In addition, the low-pass filtering can adjust the degree (intensity) in proportion to the distance from the boundary 1320 of the region of interest or the field of view. For example, as illustrated in FIG. 12B, the electronic device is proportional to the distance D between the boundary line 1209 of tile A 1205 and tile B 1207 and the boundary line 1203 of the user's field of view. You can adjust the degree of blurring (intensity) of the image.

Referring to FIG. 12 (b), the low pass filtering degree of the image is proportional to the distance D between the boundary 1209 of the tile A 1205 and the tile B 1207 and the boundary 1203 of the user's field of view. How to adjust the will be described in detail. As illustrated, an image of an original tile image quality is displayed for a region of interest inside the boundary line 1203 of the user's field of view, and the boundary line 1203 of the user's field of view is displayed for an external display image of the boundary line 1203 of the user's field of view. The distance from the image processing can gradually increase the level of low-pass filtering, thereby increasing the speed of image processing for a user's region of interest, and reducing the heterogeneity or discomfort of viewing a user's image.

In addition, the low-pass filtering method may vary the degree of low-pass filtering in proportion to a difference in image quality between tiles inside the view boundary line and outside the view boundary line.

On the other hand, when the quality of neighboring tile images is different for the image displayed inside the user's field of view boundary line 1203, the image quality at the tile boundary is continuously changed through the pixel compression method of the extended tile image described above. You can do it.

For example, the electronic device, for a tile image that is located inside the user's field of view boundary 1203 or passes through the field boundary line 1203, is an original size of the tile image, that is, an area that is expanded by a preset portion than the tile area. After receiving data for at least one extended tile image having image information from outside, such as a content server, the image at the tile boundary is displayed to the user so that the user does not feel uncomfortable through a pixel compression method for the tile boundary. You can.

13 is a diagram illustrating a procedure of a method of displaying a tile image based on a region of interest and a viewing boundary.

Hereinafter, an image display method capable of providing an entire image composed of tile images of different qualities with reduced viewing disparity of a user will be described with reference to the drawings.

An electronic device (or a media content receiving device) according to an embodiment disclosed in the present specification may include a control unit and / or a communication module. The detailed content of the control unit may include all of the above-described media content reception and processing, and may include all of the above-described processor and the electronic device of FIG. 14 to be described later. Detailed contents of the communication module may include all of the contents of the communication module disclosed herein.

The electronic device according to an embodiment disclosed in the present disclosure may receive stream data including image data of a plurality of tile images from a server or an external content transmission device through a communication module (1310). The plurality of tile images may be generated based at least in part on a user's region of interest. The plurality of tile images may be images obtained by converting an image for a 3D stereoscopic space into a 2D plane space, and then dividing an image for the converted flat space into multiple tiles.

Also, the electronic device may perform low pass filtering on at least a portion of the tile image based on the region of interest (1320). The electronic device may perform low-pass filtering through a low-pass filter provided on some of the plurality of tile images selected based on the region of interest. The electronic device may perform low-pass filtering of image data of a tile image not related to a region of interest. The low-pass filtered tile image or a part of the tile image may be blurred and blurred, or a low quality image may be displayed.

In addition, the electronic device may instruct to display the entire tile image on the display module after low-pass filtering (1330). The electronic device may combine the low-pass filtered tile image and the unfiltered tile image to display the entire image. The electronic device may display the entire combined image on a provided display module, or transmit the data for the entire image to a separate display device to display the entire image.

The display module may be provided in the image receiving device, or may be a separate device. When the display module is provided in a separate device, the separate device and the image receiving device may transmit and receive image data through a wired or wireless network. In this case, the instruction to display may be to transmit image data to the separate device. As an embodiment, the video receiving device may be a PC or a smart phone, and a device equipped with a display module may be a goggle type VR device connected to the PC or smart phone through a connection means such as a wired or wireless network or a USB terminal.

The degree of low-pass filtering (intensity), that is, the degree of blurring of the image, may be set based on the distance from the region of interest. Also, the low-pass filtering operation may be performed in units of tiles or in units of pixels of tile images.

Therefore, the electronic device does not need to receive an extended tile image from an external device such as a content server for all tile images having different resolutions or quality with respect to the entire image of the 2D plane, and is based on the user's interest area and the user's view. Since the extended tile image can be received, the bandwidth for image transmission can be secured, and the image processing speed in the electronic device can also be increased.

14 is a diagram illustrating a configuration of an electronic device (or a media content receiving device) according to an embodiment.

The electronic device 1400 (or media content transmission device) according to an embodiment disclosed in the present specification may include a control unit 1410 and / or a communication module 1420. The controller 1410 may include all the contents of the above-described processor. In addition, the communication module 1420 may include all the contents of the above-described communication module. In addition, the controller 1410 may include a region of interest information generator 1411, a request information generator 1413, a decoder 1415, a low-pass filter 1417, and / or an image combiner 1419.

The communication module 1420 may receive signaling data for obtaining service data of a service for virtual reality, augmented reality, or hologram.

The signaling data may include a media presentation description (MPD) describing tile images encoded with a plurality of qualities for each tile area. Also, the service data may include a selected tile image corresponding to a tile image encoded with a specific quality for each tile region. In addition, the service data may include an extended tile image with respect to the tile image.

The region of interest information generating unit 1411 may generate region of interest information indicating a region of interest that is expected to be viewed or viewed by the user in the entire image. In addition, the region of interest information generating unit 1411 may set a user's field of view boundary in the entire image based at least in part on the region of interest of the user. The region of interest may be set to an area expected to be viewed by the user, or may be set by tracking a user's pupil vector using a sensor provided in the electronic device. In addition, a method of setting a user's visual field boundary by the region of interest information generating unit 1411 has been described in the detailed description of FIG. 12 (a).

The entire image may indicate an image in which a 3D virtual space is converted into a 2D plane.

The request information generation unit 1413 may generate request information describing the selected tile image based on the signaling data and the region of interest information.

The quality of the selected tile image for the region of interest may be higher than the quality of the selected tile image for the region other than the region of interest.

The communication module 1420 may receive the service data based on the request information.

The decoder 1415 may decode the service data into the selected tile image of each tile area.

The low-pass filter 1417 performs low-pass filtering on image data of a tile image not related to the region of interest or an image outside the viewing boundary, thereby improving image quality for an image that is not of the user's interest or an image outside the viewing boundary. It can decrease. The low-pass filter can increase the degree of filtering in proportion to the distance from the region of interest or / and the field boundary.

Accordingly, the decoder 1415 can decode image data such that an image inside the viewing boundary is displayed in original quality and an image outside the viewing boundary is displayed lower than the original quality.

Meanwhile, the decoder 1415 may spatially compress an expanded region and an image of the expanded tile into the expanded tile image so that the image quality continuously changes at a tile boundary. Accordingly, when the quality of neighboring tile images is different for the images inside the viewing boundary line, the decoder 1415 uses the aforementioned extended tile image generation method, so that the quality of the image at the border portion of the tile is continuous. Image processing can be performed to change to.

The image combining unit 1419 may combine the selected tile images of a plurality of tile regions into a whole image.

Then, the electronic device (or the media content receiving device) may provide the user with a service for virtual reality, augmented reality, or hologram.

The media presentation description may include a plurality of adaptation set elements describing the plurality of tile areas. Also, each adaptation set element may include a plurality of representation elements describing tile images encoded with the plurality of qualities for each tile area. In addition, each representation element may include at least one of location information of the tile area, quality information of the tile image, and URL information of the tile image.

The region of interest information may include region of interest ranking information indicating a rank of the tile region that is expected to be viewed or looked at by the user based on the user's movement. In addition, the region of interest ranking information may be generated based on a pupil vector describing the movement of the user's pupil and a head vector describing the movement of the user's head.

The pupil vector may include at least one of a left pupil pupil coordinate vector indicating coordinates of the pupil included in the left pupil of the user, and a right pupil pupil coordinate vector indicating coordinates of the pupil included in the right pupil of the user. You can.

The signaling data further includes producer intention information indicating the rank of the tile area that is expected to be viewed by the user or must be viewed by the user based on the intention of the producer of the service for the virtual reality, augmented reality, or hologram. It can contain.

The signaling data may further include motion vector statistical information indicating a representative value of a motion vector for each tile area.

The selected tile image may indicate a tile image encoded with a specific quality selected for each tile region based on at least one of the region of interest ranking information, the producer intention information, and the motion vector statistics information.

The request information may include location information of the tile area, quality information of the selected tile image, and URL information of the selected tile image.

The tile image for each tile area may be expanded at a predetermined rate. Also, pixel values included in a boundary between tile images of different quality may indicate a proportional average value of pixel values of tile images of different quality.

15 is a diagram illustrating a method for receiving media content according to an embodiment.

An electronic device (or a media content receiving device) according to an embodiment disclosed herein may include a control unit (not shown) and / or a communication module. Details of the controller may include all of the above-described processor. Details of the communication module may include all of the above-described communication module.

The electronic device may receive signaling data for obtaining service data of a service for virtual reality, augmented reality, or hologram using a communication module (1510).

The signaling data may include a media presentation description describing tile images encoded with a plurality of qualities for each tile area. Also, the service data may include a selected tile image corresponding to a tile image encoded with a specific quality for each tile region.

The electronic device may generate the region of interest information indicating the region of interest expected to be viewed or looked at by the user in the entire image using the region of interest information generating unit (1520).

The entire image may indicate an image in which a 3D virtual space is converted into a 2D plane.

The electronic device may generate request information describing the selected tile image based on the signaling data and the region of interest information using the request information generator (1530).

The quality of the selected tile image for the region of interest may be higher than the quality of the selected tile image for the region other than the region of interest.

The electronic device may further receive the service data based on the request information by using a communication module (1540).

The electronic device may decode the service data into the selected tile image of each tile area using a decoder (1550).

The electronic device may combine the selected tile images of the plurality of tile regions into a whole image using an image combiner (1560).

Then, the electronic device (or the media content receiving device) may provide the user with a service for virtual reality, augmented reality, or hologram.

16 is a diagram illustrating a sequence of a method for displaying tile images of different qualities through tile image expansion.

Hereinafter, an image display method capable of providing an entire tile image in which tile images of different quality received from a server with reference to the drawings are processed by reducing image heterogeneity due to edge effects by processing image data of a tile boundary area. Explain.

An electronic device (or a media content receiving device) according to an embodiment disclosed in the present specification may include a control unit and / or a communication module. The specific content of the control unit may include all of the above-described media content reception and processing, and may include both the above-described processor content and the electronic device of FIG. 14. Details of the communication module may include all of the above-described communication module.

The electronic device according to an embodiment disclosed in the present disclosure may receive an image data stream for a plurality of tile images in which the entire image is displayed in a plurality of quality through a communication module from a server or an external content transmission device. The plurality of tile images may include at least one extended tile image having image data for an area extended by a predetermined portion than an original area of the image tile (1610).

In addition, the electronic device compresses an image of an extended portion of the expanded tile image into the interior of the tile area before expansion so that the image quality is continuously changed at a boundary between neighboring tiles through a decoder. Can (1620).

In addition, the electronic device may generate the entire tile image by combining the plurality of tile images through the image combiner (1630), and display the generated whole image through the display module.

In the extended tile image, a tile image may be overscanned to have image data of an area expanded by a predetermined portion from a boundary of the original tile image to a predetermined area than the tile area of the original size.

In addition, the extended region may be generated only for a part of the entire boundary of the tile image, and when the expanded region is generated, it is created so as not to overlap with the expanded region of the neighboring tile image. That is, when a plurality of extended tile images are adjacent to each other, each extended region of the plurality of extended tile images may be generated so as not to overlap each other.

In addition, the extended tile image may be generated at a tile boundary having a different quality only when the quality is different from a neighboring tile image.

Hereinafter, an embodiment of an image display system capable of providing an image in which a user's heterogeneity is reduced due to an edge effect in an entire image composed of a plurality of tile images having different qualities will be described.

An image display system according to an embodiment disclosed in the present specification includes a server for generating an image for transmission and an image receiving device for displaying an image.

The image server may divide the entire image into a plurality of tile images, and transmit stream data including image data of each divided tile image to the image receiving apparatus.

Also, the image server may generate image data of a plurality of tile images based at least in part on a user's region of interest.

The image receiving device may receive stream data from the image server, process the received stream data, and display the displayed data on the display module.

In addition, the image receiving apparatus may instruct low-pass filtering on at least a part of the tile image based on the region of interest, and display tile images on the display module after low-pass filtering. The display module may be provided in an image receiving device, or may be configured as a separate device. When the display module is provided in a separate device, the separate device and the image receiving device can transmit and receive image data through a wired or wireless network. have. In this case, the instruction to display may be to transmit image data to the separate device.

As an embodiment, the video receiving device may be a PC or a smart phone, and a device equipped with a display module may be a goggle type VR device connected to the PC or smart phone through a connection means such as a wired or wireless network or a USB terminal.

The plurality of tile images may include at least one extended tile image having image data of an area extended by a predetermined portion than a tile area.

The image receiving apparatus may compress the expanded tile image into the interior of the tile region so that the image quality continuously changes at the boundary of the tile region, and combine the tile images to generate the entire tile image.

Each of the above-described components of the electronic device according to the present disclosure may be composed of one or more components, and the name of the corresponding component may vary depending on the type of the electronic device. The electronic device according to the present disclosure may include at least one of the above-described components, and some components may be omitted or further include other components. In addition, since some of the components of the electronic device according to the present disclosure are combined and configured as one entity, functions of the corresponding components before being combined may be performed in the same manner.

The scope disclosed herein is not limited to the embodiments disclosed herein, and the present invention may be modified, changed, or improved in various forms within the scope described in the spirit and claims disclosed in this specification.

Claims (16)

Receiving stream data including image data of a plurality of tile images generated based at least in part on a user's region of interest;
Performing low pass filtering on at least a portion of the tile image based on the region of interest; And
Instructing to display the entire tile image on the display device after low-pass filtering
Video display method comprising a. According to claim 1, The low-pass filtering operation,
An image display method of low-pass filtering image data of a tile image not related to the region of interest. According to claim 2,
The degree of low-pass filtering is set based on the distance from the region of interest. According to claim 1, The low-pass filtering operation,
An image display method that is an operation performed in units of tiles or pixels. According to claim 1,
And determining a boundary of the user's field of view based on the region of interest. The method of claim 5, wherein the low-pass filtering operation,
It is performed based on the visual field boundary,
An image display method that is an operation of performing low-pass filtering on tiles or pixels in an area outside the viewing boundary. The method of claim 6,
The degree of low-pass filtering is set based on the distance from the viewing boundary. The method of claim 1, wherein the plurality of tile images,
An image display method including at least one extended tile image having image data of at least a part of an extended region than a tile region. The method of claim 8,
And compressing the expanded tile image into the interior of the tile region so that the image quality continuously changes at the boundary of the tile region. The method of claim 8, wherein the extended tile image,
A method of displaying an image in which neighboring extended tile images do not overlap extended regions. 10. The method of claim 9, Compressing the expanded tile image into the interior of the tile area,
Compressing into the symmetrical area based on image data of the extended area and image data of a tile area symmetric with the extended area based on a boundary between the extended area and the tile area . A communication module for receiving stream data including image data of a plurality of tile images generated based on a user's region of interest;
A low-pass filter for low-pass filtering on at least a portion of the tile image based on the region of interest; And
And a control unit instructing the display module to display all tile images after low-pass filtering. The method of claim 12, wherein the low-pass filter,
Low-pass filtering the image data of the tile image not related to the region of interest,
The degree of low-pass filtering is set based on the distance from the region of interest. The method of claim 13,
And a region of interest information setting unit configured to determine a user's field of view boundary based on at least a portion of the region of interest. The method of claim 12,
The plurality of tile images includes at least one extended tile image having image data of at least a part of an extended region than the tile region,
And a decoder for compressing the extended tile image into the interior of the tile region so that the image quality continuously changes at the boundary of the tile region. The extended tile image of claim 15,
The image data of the region extended by a predetermined portion from at least a part of the boundary of the tile region,
The neighboring expanded tile images are tile images in which the extended regions do not overlap.

Images