KR102164686B1 - Image processing method and apparatus of tile images - Google Patents

Abstract

The image processing method disclosed in the present specification provides an image data stream of a plurality of tile images including at least one extended tile image having image data of an area extending from the tile area by a predetermined portion at a boundary of at least a portion of the tile area. Receiving operation; Compressing an image of the extended region of the extended tile image into the inside of the tile region such that image quality continuously changes at the boundary of the tile region; And generating an entire tile image by combining the plurality of tile images, wherein the extended tile image may have an image quality different from that of at least one neighboring tile image.

Description

Image processing method and apparatus of tile image TECHNICAL FIELD

The present specification relates to generating an entire image by processing a tile image in image content.

With the recent development of virtual reality (VR) technology and equipment, devices that can be worn on the body such as Head-Mounted Display (HMD) are being released through various companies. Service scenarios through the HMD may include watching movies, games, remote surgery, and video conferencing.

Virtual reality service refers to a service that makes a specific environment or situation into a computer and makes it as if the user is interacting with the actual surrounding situation and environment.

There is a need for a technology that provides high-quality natural media content at a high speed in a device that provides virtual reality services.

In addition, in addition to the high-quality media content required to provide virtual reality services, demand for ultra-high resolution images such as 8K and 16K is increasing, and thus the need for efficient transmission of these ultra-high resolution images is emerging.

In the prior art virtual reality image providing apparatus, the entire image is divided into tile images and provided, while the image for the region of interest is provided by having an image quality different from that of an image other than the region of interest. However, due to the difference in image quality between tiled images, there is a problem that causes inconvenience to the user's viewing and needs to be solved.

In addition, it is necessary to provide a natural image while differently processing the quality of a displayed screen by solving the problem that the image display speed is reduced and an unnatural image is provided due to the user's change of the region of interest.

In addition, there is a need for a method and a method for providing an apparatus and a method for differently processing the quality of a screen displayed to a user according to the intention of a media content producer.

This specification proposes an image processing method. The image processing method includes an operation of receiving an image data stream of a plurality of tile images including at least one extended tile image having image data of an area extending from the tile area by a predetermined portion at a boundary of at least a portion of the tile area. ; Compressing an image of the extended region of the extended tile image into the inside of the tile region such that image quality continuously changes at the boundary of the tile region; And generating an entire tile image by combining the plurality of tile images, wherein the extended tile image may have an image quality different from that of at least one neighboring tile image.

The method and other embodiments may include the following features.

The operation of compressing the image of the extended area into the tile area includes the image data of the extended area and a tile area symmetrical to the expanded area based on a boundary between the expanded area and the tile area. It may be compressed into the symmetrical area based on image data.

In addition, when a plurality of extended tile images are adjacent to each other, each extended region of the plurality of extended tile images may not overlap with each other.

In addition, performing low-pass filtering of image data of a tile image not related to the ROI for at least a part of the entire tile image based on the ROI; And instructing to display the entire tile image on the display device after low-pass filtering.

Also, the degree of the low-pass filtering may be set based on a distance from the ROI.

In addition, the low-pass filtering operation may be performed in units of tiles or pixels.

In addition, an operation of determining a boundary line of the user's view based on the region of interest may be further included.

Further, the low-pass filtering operation may be performed based on the viewing boundary line, but may be an operation of performing low-pass filtering on a tile or pixel outside the viewing boundary line.

Also, the degree of the low-pass filtering may be set based on a distance from the viewing boundary.

Meanwhile, an image processing apparatus is presented in the present specification. The image processing apparatus is a communication for receiving an image data stream of a plurality of tile images including at least one extended tile image having image data of an area extending from the tile area by a predetermined portion at a boundary of at least a portion of the tile area. module; A decoder for compressing an image of the extended region of the extended tile image into the tile region such that image quality continuously changes at the boundary of the tile region; And an image combiner configured to combine the plurality of tile images to generate an entire tile image, wherein the extended tile image may have an image quality different from that of at least one neighboring tile image.

The device and other embodiments may include the following features.

The decoder may compress the image data of the extended region into the symmetrical region based on the image data of the tile region symmetrical with the extended region based on the boundary between the extended region and the tile region. have.

In addition, when a plurality of extended tile images are adjacent to each other, each extended region of the plurality of extended tile images may not overlap with each other.

Meanwhile, another image processing method is proposed in this specification. The image processing method includes receiving an image data stream of a plurality of tile images including at least one extended tile image having image data of an area extending from the tile area by a predetermined portion at at least a portion of a boundary of the tile area. ; Compressing an image of the extended region of the extended tile image into the inside of the tile region such that image quality continuously changes at the boundary of the tile region; And generating a full tile image by combining the plurality of tile images, wherein the extended tile image has an image quality different from that of at least one neighboring tile image, and extended portions of the neighboring tile images do not overlap each other. It can be formed to avoid.

According to the present invention, tile images of different qualities are connected through fitting of an extended tile image to a tile image to generate an entire image capable of reducing user viewing discomfort.

In addition, according to the present invention, by differently setting the image quality inside and outside the user's field of view, it is possible to create an entire image capable of reducing discomfort that the user feels when viewing.

In addition, according to the present invention, when tiles of different resolutions are adjacent to each other and are joined, unnaturalness of an image at a tile boundary is removed, thereby providing a natural image to a user.

In addition, the present invention predicts the user's intention for the region of interest based on the result of extracting the motion of the user's pupils and the motion of sound in the virtual reality space, the intention of the media content creator, the user's motion, and motion vector statistical information. Video can be provided.

In addition, the present invention transmits and provides high-resolution natural media content based on a region of interest, so that a user's feeling of rejection and discomfort for services for virtual reality, augmented reality, or hologram can be solved.

1 is a diagram illustrating an electronic system according to an exemplary embodiment.
2 is a block diagram illustrating an electronic device according to an exemplary embodiment.
3 is a block diagram of a program module according to an embodiment.
4 is a diagram illustrating a configuration of a system for transmitting and receiving media content through an IP network according to an exemplary embodiment.
5 is a diagram showing the configuration of a server according to an embodiment.
6 is a diagram illustrating an entire image encoded with various qualities according to an embodiment.
7 is a diagram for describing motion vector statistics information, producer intention information, and ROI ranking information according to an exemplary embodiment.
8 is a conceptual diagram of receiving tile images encoded with different qualities according to an embodiment.
9 is a diagram illustrating a method of removing an edge effect according to an exemplary embodiment.
10 is a diagram illustrating a method of compressing an extended tile image into a tile area.
11 is a diagram illustrating an example of a method of spatially compressing an extended area using a weighted distribution method.
12 is a diagram illustrating an example of displaying a tile image based on a region of interest and a boundary line of view.
13 is a diagram illustrating a procedure of a method of displaying a tile image based on an ROI and a boundary line of view.
14 is a diagram illustrating a configuration of an electronic device (or a media content receiving device) according to an exemplary embodiment.
15 is a diagram illustrating a method of receiving media content according to an exemplary embodiment.
16 is a diagram illustrating a procedure of a method for displaying tile images of different qualities through tile image expansion.

The technology disclosed herein can be applied to an electronic processing system. However, the technology disclosed in this specification is not limited thereto, and may be applied to all encryption and decryption apparatuses and methods to which the technical idea of the technology is applicable.

It should be noted that the technical terms used in the present specification are only used to describe specific embodiments and are not intended to limit the spirit of the technology disclosed in the present specification. In addition, the technical terms used in the present specification should be interpreted in the meaning generally understood by those of ordinary skill in the field to which the technology disclosed in the present specification belongs, unless otherwise defined in the specification. It should not be construed in a comprehensive or excessively reduced sense. In addition, when a technical term used in the present specification is an incorrect technical term that does not accurately express the spirit of the technology disclosed in the present specification, a technical term that can be correctly understood by those of ordinary skill in the field to which the technology disclosed in the present specification belongs. Should be replaced with In addition, general terms used in the present specification should be interpreted as defined in the dictionary or according to the context before and after, and should not be interpreted as an excessively reduced meaning.

Terms including ordinal numbers such as first and second used in the present specification may be used to describe various components, but the components should not be limited by the terms. These terms are used only for the purpose of distinguishing one component from another component. For example, without departing from the scope of the rights disclosed in this specification, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

Hereinafter, exemplary embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but identical or similar components are denoted by the same reference numerals regardless of reference numerals, and redundant descriptions thereof will be omitted.

In addition, in describing the technology disclosed in the present specification, when it is determined that a detailed description of a related known technology may obscure the gist of the technology disclosed in the present specification, a detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are for easy understanding of the spirit of the technology disclosed in the present specification, and should not be construed as limiting the spirit of the technology by the accompanying drawings.

1 is a diagram illustrating an electronic system according to an exemplary embodiment.

Referring to FIG. 1, the electronic system 100 may include at least one electronic device 101, 102, 104, a server 106 and/or a network 162.

The electronic device according to the present disclosure may be a device including a communication function. For example, electronic devices include smart phones, tablet personal computers (PCs), mobile phones, video phones, ebook readers, desktop personal computers (desktop personal computers), and laptop PCs. (laptop personal computer), netbook computer, personal digital assistant (PDA), portable multimedia player (PMP), MP3 player, mobile medical device, camera, or wearable device (e.g., electronic It may include at least one of a head-mounted device (HMD) such as glasses, an electronic clothing, an electronic bracelet, an electronic necklace, an electronic appcessory, an electronic newspaper, or a smartwatch.

According to some embodiments, the electronic device may be a smart home appliance having a communication function. Smart home appliances, for example, include televisions, digital video disk (DVD) players, audio, refrigerators, air conditioners, vacuum cleaners, ovens, microwave ovens, washing machines, air purifiers, set-top boxes, and TVs. It may include at least one of a box (eg, Samsung HomeSync, Apple TV, or Google TV), game consoles, electronic dictionary, electronic key, camcorder, or electronic frame.

According to some embodiments, the electronic device includes various medical devices (e.g., magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), computed tomography (CT)), an imager, an ultrasound device, etc.), a navigation device, a GPS receiver ( global positioning system receiver), EDR (event data recorder), FDR (flight data recorder), automobile infotainment device, marine electronic equipment (e.g., marine navigation equipment and gyro compass, etc.), avionics, security It may include at least one of a device, a vehicle head unit, an industrial or domestic robot, an automatic teller machine (ATM) of a financial institution, or a point of sales (POS) of a store.

According to some embodiments, the electronic device is a piece of furniture or building/structure including a communication function, an electronic board, an electronic signature receiving device, a projector, or various measurement devices. It may include at least one of devices (eg, water, electricity, gas, or radio wave measurement devices). The electronic device according to the present disclosure may be a combination of one or more of the aforementioned various devices. Also, the electronic device according to the present disclosure may be a flexible device. In addition, it is obvious to those skilled in the art that the electronic device according to the present disclosure is not limited to the above-described devices.

The electronic device 101 may include a bus 110, a processor 120, a memory 130, an input/output interface 150, a display 160, and/or a communication interface 170.

In this case, the network 162 may include a LAN connected by wire, a cable channel network, and all possible methods connected by wireless.

In some embodiments, the electronic device 101 may omit at least one of the elements or may additionally include other elements.

The bus 110 may include, for example, a circuit that connects the components 110 to 170 to each other and transmits communication (eg, control messages and/or data) between the components.

The processor 120 may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor 120 may control at least one other component of the electronic device 101 and/or perform an operation or data processing related to communication.

The memory 130 may include a volatile and/or non-volatile memory. The memory 130 may store, for example, a command or data related to at least one other component of the electronic device 101. According to an embodiment, the memory 130 may store software and/or a program 140. The program 140 is, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and/or an application program (or "application") 147, etc. It may include. At least a portion of the kernel 141, the middleware 143, or the API 145 may be referred to as an operating system (OS).

The kernel 141 is, for example, a system used to execute an operation or function implemented in other programs (eg, the middleware 143, the API 145, or the application program 147) Resources (eg, the bus 110, the processor 120, or the memory 130) may be controlled or managed. In addition, the kernel 141 can control or manage system resources by accessing individual components of the electronic device 101 from the middleware 143, the API 145, or the application program 147. Interface can be provided.

The middleware 143 may, for example, play an intermediary role so that the API 145 or the application program 147 communicates with the kernel 141 to exchange data.

In addition, the middleware 143 may process one or more job requests received from the application program 147 according to priority. For example, the middleware 143 is a system resource (for example, the bus 110, the processor 120, or the memory 130) of the electronic device 101 in at least one of the application programs 147 Etc.) can be given priority. For example, the middleware 143 may perform scheduling or load balancing for the one or more work requests by processing the one or more work requests according to the priority assigned to the at least one.

The API 145 is an interface for controlling functions provided by the kernel 141 or the middleware 143 by the application 147, for example, file control, window control, image It may include at least one interface or function (eg, command) for processing or character control.

The input/output interface 150 may serve as an interface through which commands or data input from a user or another external device can be transmitted to other component(s) of the electronic device 101, for example. In addition, the input/output interface 150 may output commands or data received from other component(s) of the electronic device 101 to a user or another external device.

The display 160 is, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light-emitting diode (OLED) display, or a microelectromechanical It may include a Micro-Electro-Mechanical Systems (MEMS) display, or an electronic paper display. The display 160 may display, for example, various contents (eg, text, images, videos, icons, symbols, etc.) to a user. The display 160 may include a touch screen, and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a user's body part.

The communication interface 170 may establish communication between the electronic device 101 and an external device (eg, the electronic device 102, the electronic device 104, or the server 106 ), for example. For example, the communication interface 170 may be connected to the network 162 through wireless communication or wired communication to communicate with an external device (eg, the electronic device 104 or the server 106).

The wireless communication may include, for example, cellular communication. The cellular communication is, for example, LTE (long-term evolution), LTE-A (LTE Advance), CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal mobile telecommunications system), WiBro (Wireless Broadband), or Global System for Mobile Communications (GSM), or the like. In addition, the wireless communication may include, for example, short-range communication 164. The short-range communication 164 may include at least one of, for example, wireless fidelity (Wi-Fi), Bluetooth, or nearfield communication (NFC). In addition, the wireless communication may include a global positioning system (GPS).

The wired communication may include at least one of, for example, a universal serial bus (USB), a high definition multimedia interface (HDMI), a recommended standard 232 (RS-232), or a plain old telephone service (POTS). The network 162 may include at least one of a telecommunications network, for example, a computer network (eg, LAN or WAN), the Internet, or a telephone network.

Each of the electronic device 102 and the electronic device 104 may be a device of the same or different type as the electronic device 101. According to an embodiment, the server 106 may include a group of one or more servers. According to various embodiments, all or part of the operations executed by the electronic device 101 may be executed by another one or a plurality of external devices (eg, the electronic devices 102 and 104 or the server 106 ). According to an embodiment, when the electronic device 101 needs to perform a certain function or service automatically or by request, the electronic device 101 instead of executing the function or service itself or additionally, At least some functions related thereto may be requested from an external device (eg, the electronic devices 102 and 104 or the server 106). An external device (for example, the electronic devices 102 and 104, or the server 106) may execute the requested function or additional function and transmit the result to the electronic device 101. The electronic device 101 may provide a requested function or service by processing the received result as it is or additionally. For this, for example, cloud computing, distributed computing, or client-server computing technology may be used.

2 is a block diagram illustrating an electronic device according to an exemplary embodiment.

The electronic device 201 may include, for example, all or part of the electronic device 101 illustrated in FIG. 1. The electronic device 200 includes one or more processors (eg, an application processor (AP)) 210, a communication module 220, the memory 230, a sensor module 240, an input device 250, and a display 260. ), an interface 270, an audio module 280, a camera module 291, a power management module 295, a battery 296, an indicator 297, and a motor 298.

The processor 210 may control a plurality of hardware or software components connected to the processor 210 by driving an operating system or an application program, for example, and may perform various data processing and operations. The processor 210 may be implemented with, for example, a system on chip (SoC). According to an embodiment, the processor 210 may further include a graphic processing unit (GPU) and/or an image signal processor. The processor 210 may also include at least some of the components shown in FIG. 2 (eg, a cellular module 221). The processor 210 loads and processes commands or data received from at least one of other components (eg, nonvolatile memory) into a volatile memory, and stores various data in the nonvolatile memory. I can.

The communication module 220 may have the same or similar configuration as the communication interface 170 of FIG. 1. The communication module 220 is, for example, a cellular module 221, a Wi-Fi module 223, a Bluetooth module 225, a GPS module 227, an NFC module 228, and a radio frequency (RF) module (229) may be included.

The cellular module 221 may provide, for example, a voice call, a video call, a text service, or an Internet service through a communication network. According to an embodiment, the cellular module 221 may distinguish and authenticate the electronic device 201 in a communication network using a subscriber identification module (eg, a SIM card) 224. According to an embodiment, the cellular module 221 may perform at least some of the functions that can be provided by the processor 210. According to an embodiment, the cellular module 221 may include a communication processor (CP).

Each of the Wi-Fi module 223, the Bluetooth module 225, the GPS module 227, or the NFC module 228 includes, for example, a processor for processing data transmitted and received through a corresponding module. Can include. According to some embodiments, at least some of the cellular module 221, the Wi-Fi module 223, the Bluetooth module 225, the GPS module 227, or the NFC module 228 (for example, two The above) may be included in one integrated chip (IC) or IC package.

The RF module 229 may transmit and receive, for example, a communication signal (eg, an RF signal). The RF module 229 may include, for example, a transceiver, a power amp module (PAM), a frequency filter, a low noise amplifier (LNA), and the like. In addition, the RF module 229 may further include a component for transmitting and receiving an electromagnetic wave in a free space in wireless communication, for example, a conductor or a conducting wire. In FIG. 2, the cellular module 221, the Wi-Fi module 223, the Bluetooth module 225, the GPS module 227, or the NFC module 228 share one RF module 229 with each other. However, according to an embodiment, at least one of the cellular module 221, the Wi-Fi module 223, the Bluetooth module 225, the GPS module 227, or the NFC module 228 One can transmit and receive RF signals through separate RF modules.

The SIM card 224 may include, for example, a card including a subscriber identification module and/or an embedded SIM (embedded SIM), and unique identification information (eg, integrated circuit card identifier (ICCID)) or Subscriber information (eg, international mobile subscriber identity (IMSI)) may be included.

The memory 230 (for example, the memory 130) may include, for example, an internal memory 232 or an external memory 234. The internal memory 232 is, for example, a volatile memory (eg, dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), a non-volatile memory (eg : OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash, etc.) , A hard drive, or a solid state drive (SSD).

The external memory 234 is a flash drive, for example, compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), xD ( extreme digital), MMC (MultiMediaCard) or memory stick (memory stick) may further include. The external memory 234 may be functionally and/or physically connected to the electronic device 201 through various interfaces.

The sensor module 240 may measure a physical quantity or detect an operating state of the electronic device 201, for example, and convert the measured or detected information into an electrical signal. The sensor module 240 is, for example, a proximity sensor 241, a gyro sensor 242, an illuminance sensor 243, a magnetic sensor 244, an acceleration sensor 245, a grip sensor 246, a gesture sensor. It may include at least one of 247 or the biometric sensor 248. Additionally or alternatively, the sensor module 240 includes, for example, an E-nose sensor, an electromyography sensor, an electroencephalogram sensor, an electrocardiogram sensor. ), an IR (infrared) sensor, an iris sensor, and/or a fingerprint sensor. The sensor module 240 may further include a control circuit for controlling at least one or more sensors included therein. In some embodiments, the electronic device 201 further includes a processor configured to control the sensor module 240 as part of the processor 210 or separately, and the processor 210 is in a sleep state. While in, it is possible to control the sensor module 240.

The input device 250 may include, for example, a touch panel 252, a (digital) pen sensor 254 or a key 256. The touch panel 252 may use at least one of, for example, a capacitive type, a pressure sensitive type, an infrared type, or an ultrasonic type. In addition, the touch panel 252 may further include a control circuit. The touch panel 252 may further include a tactile layer to provide a tactile reaction to a user.

The pen sensor 254 may be, for example, a part of a touch panel, or may include a separate sheet for recognition. The key 256 may include, for example, a physical button, an optical key, or a keypad.

The display 260 (for example, the display 160) may include a panel 262, a hologram device 264, or a projector 266. The panel 262 may have the same or similar configuration as the display 160 of FIG. 1. The panel 262 may be implemented, for example, to be flexible, transparent, or wearable. The panel 262 may be configured with the touch panel 252 as a single module. The hologram device 264 may show a 3D image in the air using interference of light. The projector 266 may display an image by projecting light onto a screen. The screen may be located inside or outside the electronic device 201, for example. According to an embodiment, the display 260 may further include a control circuit for controlling the panel 262, the hologram device 264, or the projector 266.

The interface 270 may include, for example, a high-definition multimedia interface (HDMI) or a universal serial bus (USB). The interface 270 may be included in the communication interface 170 shown in FIG. 1, for example. Additionally or alternatively, the interface 270 may include, for example, an optical interface, a D-sub (D-subminiature), a mobile high-definition link (MHL) interface, and a secure SD (SD) interface. digital) card/multi-media card (MMC) interface, or an Infrared Data Association (IrDA) standard interface.

The audio module 280 may bidirectionally convert a sound and an electric signal, for example. At least some components of the audio module 280 may be included in, for example, the input/output interface 150 illustrated in FIG. 1. The audio module 280 may process sound information input or output through, for example, a speaker 282, a receiver 284, earphones 286, or a microphone 288.

The camera module 291 is, for example, a device capable of photographing still images and moving pictures, and according to an embodiment, one or more image sensors (eg, a front sensor or a rear sensor), a lens, an image signal processor (ISP) ), or flash (eg, LED or xenon lamp).

The power management module 295 may manage power of the electronic device 201, for example. According to an embodiment, the power management module 295 may include a power management integrated circuit (PMIC), a charger integrated circuit (IC), or a battery or fuel gauge. The PMIC may have a wired and/or wireless charging method. The wireless charging method includes, for example, a magnetic resonance method, a magnetic induction method, or an electromagnetic wave method, and may further include an additional circuit for wireless charging, for example, a coil loop, a resonance circuit, or a rectifier. have. The battery gauge may measure, for example, the remaining amount of the battery 296, a voltage, current, or temperature during charging. The battery 296 may include, for example, a rechargeable battery and/or a solar battery.

The indicator 297 may display a specific state of the electronic device 201 or a part thereof (eg, the processor 210), for example, a booting state, a message state, or a charging state. The motor 298 may convert an electrical signal into mechanical vibration, and may generate a vibration or a haptic effect. Although not shown, the electronic device 201 may include a processing device (eg, a GPU) for supporting mobile TV. A processing device for supporting mobile TV may process media data according to standards such as digital multimedia broadcasting (DMB), digital video broadcasting (DVB), or MediaFlo ^™ .

Each of the components described in this document may be composed of one or more components, and the name of the component may vary according to the type of electronic device. In various embodiments, the electronic device may be configured to include at least one of the components described in this document, and some components may be omitted or additional other components may be further included. In addition, some of the components of the electronic device according to various embodiments of the present disclosure are combined to form a single entity, so that functions of the corresponding components before the combination may be performed in the same manner.

3 is a block diagram of a program module according to an embodiment.

According to an embodiment, the program module 300 (eg, program 140) is an operating system (OS) and/or operating system that controls resources related to an electronic device (eg, the electronic device 101). Various applications (eg, the application program 147) running on the system may be included. Operating system, for example, Android (Android), iOS, Windows (Windows), Symbian (Symbian), RTOS (Realtime OS), Linux (Linux), WebOS (WebOS), P2P (Peer to Peer) based It may be an OS having a feature, or a Tizen.

The program module 300 may include a kernel 320, middleware 330, an application programming interface (API) 360, and/or an application 370. At least a portion of the program module 300 may be preloaded on an electronic device or downloaded from an external electronic device (eg, external electronic devices 102 and 104, server 106, etc.).

The kernel 320 (for example, the kernel 141) may include, for example, a system resource manager 321 and/or a device driver 323. The system resource manager 321 may perform control, allocation, or recovery of system resources. According to an embodiment, the system resource manager 321 may include a process management unit, a memory management unit, or a file system management unit. The device driver 323 may include, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a Wi-Fi driver, an audio driver, or an IPC (inter process communication) driver. I can.

The middleware 330 may provide, for example, a function commonly required by the application 370 or the API 360 so that the application 370 can efficiently use limited system resources inside the electronic device. ) To provide various functions to the application 370. According to an embodiment, the middleware 330 (for example, the middleware 143) is a runtime library 335, an application manager 341, a window manager 342, and a multimedia manager. manager(343), resource manager(344), power manager(345), database manager(346), package manager(347), connectivity manager(connectivity) Manager (348), notification manager (notification manager) (349), location manager (location manager) 350, graphic manager (graphic manager) 351, or security manager (security manager) 352 Can include.

The runtime library 335 may include, for example, a library module used by a compiler to add a new function through a programming language while the application 370 is being executed. The runtime library 335 may perform input/output management, memory management, or a function for an arithmetic function.

The application manager 341 may manage, for example, a lifecycle of at least one application among the applications 370. The window manager 342 may manage GUI resources used on a screen. The multimedia manager 343 may grasp a format required for reproduction of various media files, and may perform encoding or decoding of a media file using a codec suitable for the corresponding format. The resource manager 344 may manage resources such as source code, memory, or storage space of at least one of the applications 370.

The power manager 345 may operate together with, for example, a BIOS (basic input/output system), to manage a battery or power, and provide power information necessary for the operation of an electronic device. . The database manager 346 may create, search, or change a database to be used by at least one of the applications 370. The package manager 347 may manage installation or update of an application distributed in the form of a package file.

The connection manager 348 may manage a wireless connection such as Wi-Fi or Bluetooth. The notification manager 349 may display or notify an event such as an arrival message, an appointment, and a proximity notification in a manner that does not interfere with the user. The location manager 350 may manage location information of an electronic device. The location information disclosed in this specification includes not only GPS (Global Positioning System) information, but also acceleration sensor information, gyro sensor information, indoor coordinate sensor information for LBS (Location Based Service), network-based location information, and location using a communication network. It may include information and other location information related to motion transmitted from the new electronic device. The graphic manager 351 may manage a graphic effect to be provided to a user or a user interface related thereto. The security manager 352 may provide all security functions required for system security or user authentication. According to an embodiment, when an electronic device (for example, the electronic device 101) includes a phone function, the middleware 330 provides a telephony manager for managing a voice or video call function of the electronic device. It may contain more.

The middleware 330 may include a middleware module that forms a combination of various functions of the above-described components. The middleware 330 may provide modules specialized for each type of operating system in order to provide differentiated functions. Further, the middleware 330 may dynamically delete some of the existing components or add new components.

The API 360 (for example, the API 145) is a set of API programming functions, for example, and may be provided in different configurations according to an operating system. For example, in the case of Android or iOS, one API set may be provided for each platform, and in the case of Tizen, two or more API sets may be provided for each platform.

The application 370 (for example, the application program 147) is, for example, a home 371, a dialer 372, an SMS/MMS 373, an instant message (IM) 374, Browser (375), camera (376), alarm (377), contact (378), voice dial (379), email (380), calendar (381), media player (382), album (383), or clock ( 384), health care (e.g., measuring exercise volume or blood sugar), or providing environmental information (e.g., providing information on barometric pressure, humidity, or temperature), etc. Can include.

According to an embodiment, the application 370 is an application that supports information exchange between an electronic device (eg, the electronic device 101) and an external device (eg, the electronic device 102, 104) For convenience, "information exchange application") may be included. The information exchange application may include, for example, a notification relay application for delivering specific information to an external device, or a device management application for managing an external device.

For example, the notification delivery application transmits notification information generated from another application (eg, SMS/MMS application, email application, health management application, environment information application, etc.) of the electronic device to an external device (eg, electronic device 102). , 104)). In addition, the notification delivery application may receive notification information from an external electronic device and provide it to a user.

The device management application is, for example, at least one function of an external device (eg, electronic device 102, 104) communicating with the electronic device (eg, turn-on of the external device itself (or some component parts))/ Turn-off or adjust the brightness (or resolution) of the display), manage (e.g., install, delete, or update) an application running on an external device or a service (e.g., call service or message service, etc.) I can.

According to an embodiment, the application 370 may include properties of external devices (eg, electronic devices 102 and 104) (applications designated according to (eg, a health management application of a mobile medical device)). According to one embodiment, the application 370 may include an application received from an external device (eg, the server 106 or the electronic devices 102 and 104). According to an embodiment, the application 370 ) May include a preloaded application or a third party application that can be downloaded from the server. Names of the components of the program module 310 according to the illustrated embodiment are It may vary depending on the type.

According to various embodiments, at least a portion of the program module 310 may be implemented as software, firmware, hardware, or a combination of at least two or more of them. At least a part of the program module 300 may be implemented (eg, executed) by, for example, a processor (eg, the processor 210). At least a portion of the program module 300 may include, for example, a module, a program, a routine, a set of instructions or a process for performing one or more functions.

The term "module" used in this document may mean, for example, a unit including one or a combination of two or more of hardware, software, or firmware. “Module” may be used interchangeably with terms such as unit, logic, logical block, component, or circuit, for example. The "module" may be the smallest unit of integrally configured parts or a part thereof. The "module" may be a minimum unit or a part of one or more functions. The "module" can be implemented mechanically or electronically. For example, a "module" is one of known or future developed application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), or programmable-logic devices that perform certain operations. It may include at least one.

At least a part of a device (eg, modules or their functions) or a method (eg, operations) according to various embodiments is, for example, a computer-readable storage media in the form of a program module. It can be implemented as a command stored in. When the command is executed by a processor (eg, the processor 120), the one or more processors may perform a function corresponding to the command. The computer-readable storage medium may be, for example, the memory 130.

Computer-readable recording media include hard disks, floppy disks, magnetic media (e.g. magnetic tape), optical media (e.g. compact disc read only memory (CD-ROM)), DVD ( Digital Versatile Disc), magnetic-optical media (e.g. floptical disk), hardware devices (e.g. read only memory (ROM), random access memory (RAM)), or flash memory, etc. ), etc. In addition, the program instruction may include not only machine language code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may include various types of hardware devices. It may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

A module or a program module according to various embodiments of the present disclosure may include at least one or more of the above-described components, some of which are omitted, or may further include other additional components. Operations performed by a module, a program module, or other components according to various embodiments may be executed sequentially, in parallel, repeatedly, or in a heuristic manner. Also, some operations may be executed in a different order, omitted, or other operations may be added.

4 is a diagram illustrating a configuration of a system for transmitting and receiving media content through an IP network according to an exemplary embodiment.

Referring to FIG. 4, a media content transmission/reception system 400 according to an embodiment disclosed in the present specification may include a content server 420 and/or a DASH client 440. For example, the content server 420 may be included in the server 106 of FIG. 1. Also, the DASH client 440 may be included in the processors 120 and 210 of the electronic device 101 of FIGS. 1 and/or 2.

The system 400 for transmitting/receiving media content through an IP network according to an embodiment disclosed in the present specification is divided into transmission/reception of a transport packet including actual media content and transmission/reception of media content presentation information. The DASH client 440 receives media content presentation information and receives a transport packet including media content. In this case, the media content presentation information indicates information necessary for media content playback. The media content presentation information may include at least one of spatial information and temporal information required for media content playback. The DASH client 440 plays media content based on the media content presentation information.

In a specific embodiment, media content may be transmitted/received through an IP network according to the MMTP (MPEG Media Transport Protocol) standard. At this time, the content server 420 transmits a presentation information document (PI document) including media content presentation information. In addition, the content server 420 transmits an MMT protocol (MMTP) packet including media content based on the request of the DASH client 440. The DASH client 440 receives a PI document. The DASH client 440 receives a transport packet including media content. The DASH client 440 extracts media content from a transport packet including the media content. The DASH client 440 plays media content based on the PI document.

In another specific embodiment, media content may be transmitted/received through an IP network according to the MPEG-DASH standard. In this case, the content server 420 transmits a media presentation description (MPD) including media content presentation information. However, according to a specific embodiment, the MPD may be transmitted by an external server other than the content server 50. For example, the external server may include a server for a content delivery network (CDN). CDN refers to a system that stores and provides data in a network having multiple nodes in order to efficiently deliver large-capacity service data, media data, and/or signaling data. In addition, the content server 420 transmits segments including media content based on the request of the DASH client 440. The DASH client 440 receives the MPD. The DASH client 440 requests media content from the content server 420 based on the MPD. DASH client 440 receives a transport packet including media content based on the request. The DASH client 440 plays media content based on the MPD.

The DASH client 440 transmits an HTTP request message through the MPD Parser that parses the MPD, the Segment Parser that parses the Segment, and the communication interface 170 and/or the communication module 220 and receives an HTTP response message. It may include an HTTP client and a media engine that plays media.

5 is a diagram showing the configuration of a server according to an embodiment.

The server 500 according to an embodiment disclosed in the present specification may encode media content differently according to quality and may selectively transmit the media content to the electronic device. For example, the server 500 may include at least one of the control unit 510 and/or the communication module 520. The control unit 510 may include all the contents of the above-described processor. Further, the control unit 510 may be included in the server 500 or may exist independently of the server 500. Detailed contents of the communication module 520 may include all contents of the above-described communication module. Hereinafter, a description will be given focusing on the control unit 510.

The control unit 510 according to an embodiment disclosed in the present specification may include at least one of an image generation unit 511, an image division unit 513, an encoder 515, and/or a signaling data generation unit 517. have.

The image generator 511 may generate an entire image in which a 3D virtual space is converted into a 2D plane. For example, the image generator 511 may receive image information from at least one electronic device and/or an external device, and may generate a 3D virtual space. Also, the image generator 511 may convert the generated 3D virtual space into a 2D plane.

The image dividing unit 513 may divide the entire image of the 2D plane into a plurality of tile regions. For example, the tile area may be a rectangular area. The tile image may represent an image corresponding to the tile area within the entire image.

The encoder 515 may encode the tile image of each tile region with a plurality of qualities. For example, the encoder 515 may encode the entire image and/or the tile image at various bit rates. In addition, the encoder 1615 may use H.264/AVC and/or HEVC, and in addition to the video format previously referred to in MPEG-DASH, data for 3D virtual reality and augmented reality such as holograms are newly matched. Can be transmitted.

Further, the encoder 515 may generate a motion vector. The motion vector is information expressed as a relative position in this dimension from the reference picture to the reference block based on the same position as the current block. In inter prediction, the encoder and the decoder can generate the same prediction signal by transmitting the motion vector of the referenced block to the decoder.

Also, the encoder 515 may generate an extended tile image. The extended tile image is a tile image (overscan) having image data not only for the original tile area of the tile image, but also for the area extended by a preset portion (for example, the margin ratio 927 in FIG. 9) at the boundary of the tile area. Image), and the size of the image may be a tile image whose size is larger than the original size of the tile image by the preset portion. Also, the size and shape of the extended area may be variously modified according to a preset value, and variously modified within a range in which the extended area of the tile to be expanded and the extended area of the adjacent tile do not overlap. That is, the extended tile images may have various shapes and sizes of the extended areas so that the extended areas do not overlap each other.

The signaling data generation unit 517 may generate signaling data for acquiring service data of a service for virtual reality, augmented reality, or hologram.

The signaling data may include a Media Presentation Description (MPD) that describes tile images encoded as an image extended in a plurality of qualities and various shapes for each tile region. In addition, the service data may include a selected tile image corresponding to a tile image encoded with a specific quality for each tile region.

For example, the media presentation description may include a plurality of adaptation set elements describing a plurality of tile regions. In addition, each adaptation set element may include a plurality of representation elements that describe tile images encoded with a plurality of qualities for each tile area. In addition, each representation element may include at least one of location information of a tile region, quality information of a tile image, and URL information of a tile image.

The signaling data may further include maker intention information indicating the ranking of the tile area that the user is expected to see or must see based on the intention of the maker of the service for virtual reality, augmented reality, or hologram. . The producer may set a higher order of the tile region corresponding to the region of interest, and transmit a tile image encoded with high quality. In addition, the order of tile regions corresponding to regions other than the region of interest may be set to be low, and a tile image encoded with low quality may be transmitted.

The signaling data may further include motion vector statistical information indicating a representative value of a motion vector for each tile region.

Motion vector statistics information may be added to the file name of the media presentation description, or may be included in an element and/or attribute included in the media presentation description. In addition, motion vector statistics information may be added to a file name of a meta data table that exists separately from the media presentation description, or may be included in an element and/or attribute included in the meta data table.

For example, motion vector statistics information may exist for each time zone and/or for each macro block. In addition, the motion vector statistics information may be in a format such as [TimeSeconds, Poi_x, Poi_y, DeltaOf(MV)]. Here, TimeSeconds indicates time information, Poi_x and/or Poi_y indicates coordinate information, and DeltaOf(MV) may indicate a representative value (or average value) of at least one motion vector in each tile.

The communication module 520 may transmit and/or receive service data, signaling data, and/or additional data for image display to at least one sensor device, an electronic device, and/or an external device. For example, the communication module 520 may transmit service data including image data for a tile image and an extended tile image based on the request information.

For example, the request information may describe a tile image selected based on signaling data and ROI information. Also, the ROI information may indicate an ROI that is expected to be viewed or viewed by the user in the entire image.

6 is a diagram illustrating an entire image encoded with various qualities according to an embodiment.

Referring to FIG. 6A, the entire image 610 divided into at least one tile area is shown.

The server may divide the entire image 610 for a service for virtual reality, augmented reality, or hologram into at least one rectangular tile area using the image segmentation unit.

For example, the entire image 610 may be an image representing a range of 360 degrees. Also, the number and/or size of the tile areas may be predefined by a user, an electronic device, and/or a server.

Referring to FIG. 6(b), all images encoded with various qualities are shown.

The server may encode the entire image and/or at least one tile image with various qualities using the encoder.

For example, the server may generate a first full image 621 of low quality, a second full image 623 of medium quality, and/or a third full image 625 of high quality by encoding the entire image 610. have.

A plurality of tile images included in the same tile area may be described by an adaptation set element. Also, each tile image included in the same tile area may be described by a representation element.

The server according to an embodiment disclosed in the present specification may selectively and/or adaptively transmit a tile image encoded with a specific quality for each tile region to a user based on a network condition and/or a user's request.

7 is a diagram for describing motion vector statistics information, producer intention information, and ROI ranking information according to an embodiment.

Referring to FIG. 7(a), an entire image 710 and a motion vector 713 are shown.

The motion vector statistical information according to an embodiment disclosed in the present specification may indicate a representative value of the motion vector 713 for each tile area. For example, the representative value may be an average value of a plurality of motion vectors existing in each tile area.

The electronic device according to an embodiment disclosed in the present specification may select a tile image encoded with a specific quality for each tile region based on motion vector statistics information. For example, the electronic device arranges tile regions in the order in which the value of the motion vector statistics information is large, and receives a tile image encoded with high quality for the tile regions 711, 715, and 717 having a large value of the motion vector statistics information. can do.

The electronic device according to an embodiment disclosed in the present specification obtains statistics on a motion vector of an image using a deep learning using a convolutional neural network (CNN), thereby obtaining a tile image for each tile. You can extract the mobility of an object inside.

In addition, when discriminating 2D/3D of an image, it is possible to increase the accuracy of motion/high frequency image detection and systemize the data to be processed by employing a fractal encoding technique to vectorize the existing 2D image.

Statistical accuracy can be improved through a convolutional neural network (CNN) based on image information data vectorized through fractal encoding between frames and frames.

Referring to FIG. 7(b), the entire image 720, the region of interest 721 viewed by the user, the tile region 723 expected to be viewed by the user based on the user's movement, and the intention of the creator of the media content A tile area 725 expected to be viewed by the user is shown based on.

Currently, the user is looking at the region of interest 721, but may move in the direction of the tile region 723 expected to be viewed by the user. In this case, the ROI priority information may have a high priority value for the tile area 723 based on the user's movement.

Also, the media content creator's intention may be that the user looks at the tile area 725. In this case, the maker intention information may have a high priority value for the tile area 725 based on the maker's intention.

The electronic device according to an embodiment disclosed in the present specification may select a tile image encoded with a specific quality for each tile region based on the ROI priority information and/or the maker intention information. For example, the electronic device arranges the tile regions in the order of the values of the region of interest priority information and/or the creator intention information, and the tile regions 723 and 725 having the higher values of the region of interest priority information and/or the creator intention information It is possible to receive a tile image encoded with high quality for.

8 is a conceptual diagram of receiving tile images encoded with different qualities according to an embodiment.

Referring to the drawing, an entire image 810 encoded with low quality, an entire image 820 encoded with high quality, and an entire image 830 received by an electronic device are illustrated.

The regions of interest 811, 821, and 831 are regions that the user is currently viewing. The first ROI 813, 823, and 833 is an area having the highest value of the ROI ranking information. The second regions of interest 814, 824, and 834 are regions in which the value of producer intention information is the highest. The third ROI 815, 825, and 835 is an area having the highest motion vector statistical information. The first region of interest, the second region of interest, and the third region of interest may all be expressed as regions of interest.

The electronic device according to an embodiment disclosed in the present specification may selectively receive a tile image based on ROI ranking information, maker intention information, and/or motion vector statistics information.

For example, the electronic device may receive high quality encoded tile images for the ROI 831, the first ROI 833, the second ROI 834, and the third ROI 835. The electronic device may receive a tile image encoded with low quality for the remaining area.

Hereinafter, an image display method capable of providing an image in which a user's sense of heterogeneity due to an edge effect is reduced in an entire image composed of tile images of different quality will be described with reference to FIGS. 9 to 12.

The edge effect refers to a phenomenon in which the boundary between two adjacent tile images of different qualities is unnatural due to a sharp difference in quality, or the tile images are not connected and appear broken. It is possible to feel uncomfortable in viewing the entire image formed by combining the images.

The image receiving apparatus (electronic device) according to the exemplary embodiment disclosed in the present specification may receive tile image data for an entire image encoded as a plurality of tile images of a plurality of qualities. The plurality of tile images may include at least one extended tile image having image data for an area extended by a portion of the tile area.

In addition, the image receiving apparatus fits the extended tile image into the original tile area (the tile area before expansion) of the expanded tile so that the image quality between neighboring tiles at the boundary of the tile continuously changes, and the tile image The edge effect can be removed by combining them to create an entire image. Herein, the method of fitting the extended tile image into the original tile region of the extended tile includes image information (pixel information) in the original tile region and image information of the extended region in the tile region before expansion in the extended tile image. It may be image processing that is compressed and displayed.

The above-described image quality may be determined by a bit depth representing an image resolution, a color depth, etc., a frame rate, a bit-rate, a grayscale difference, and the like.

9 is a diagram illustrating in detail a method of removing an edge effect according to an exemplary embodiment.

Referring to FIG. 9A, an ROI 915 may exist in four tile regions 911, 912, 913, and 914. In addition, images of the four tile regions may all have the same image quality or may be different from each other.

If the quality and/or resolution of two adjacent tile regions within the ROI 915 are different, an edge effect 916 may occur at the boundary.

Hereinafter, a method of removing the edge effect 916 according to an embodiment disclosed in the present specification will be described.

Referring to FIG. 9B, a tile image 923 and an extended tile image 925 are shown.

At least one tile image 923 according to an embodiment disclosed in the present specification is actually transmitted and/or received as an extended tile image 925 scanned wider by a margin ratio 927 than a tile area (an image of an original tile). Can be. That is, the quality may be the same within the region of the extended tile image 925.

The electronic device may receive at least one extended tile image 925 and combine the at least one extended tile image 925 to generate an entire image. In this case, the extended tile images 925 combined with each other may overlap with each other by a margin ratio 927. Therefore, the electronic device can remove the edge effect by processing the image information so that the two tile images are naturally connected to the overlapping part, that is, the quality of the two neighboring tile images continuously changes at the neighboring boundary. I can.

Referring to Fig. 9(c), a processing method for a boundary portion is shown.

A tile image 931 encoded with low quality and a tile image 932 encoded with high quality are adjacent to each other. The extended tile image for the tile image 932 encoded with high quality overlaps the tile image 931 encoded with low quality by a preset portion 933. In addition, the extended tile image of the tile image 931 encoded with low quality overlaps the tile image 932 encoded with high quality by a preset portion 934.

The electronic device according to an embodiment disclosed in the present specification may perform a proportional averaging method, a logistic model applied weight averaging method, and/or a filtering method on preset portions 933 and 934 where two extended tile images overlap. .

The proportional average method is a method of obtaining a proportional average value of pixel image values of two adjacent or adjacent tile images by varying the weight of pixels by an extended distance based on a high-quality tile image. In this case, the boundary portion may have a continuous value, and two tile images may be naturally combined.

The logistic model-applied weight averaging method is a regression expressed by the logistic regression model equation (940), which is applied when a high-resolution tile and a low-resolution tile overlap and/or when a pixel weight is applied by a preset distance to an extended tile area. It is an equation model.

Hereinafter, the logistic model application weight averaging method will be described centering on high-resolution tiles and low-resolution tiles, but the method can be applied equally to high-quality tiles and low-quality tiles. The shape of the graph 941 is the most standard slope distribution model. For example, when a tile is a high-resolution tile, a method of obtaining a weight will be described. First, the electronic system can obtain a weight value 942 according to the% position of the high-resolution pixel. In this case, the electronic system may grasp the weight 943 according to the% position of the low-resolution pixel, which is symmetric to the% position of the low-resolution pixel, of the low-resolution tile overlapping the high-resolution tile. The electronic system multiplies the weight value 942 according to the% position of the high resolution pixel by the pixel corresponding to the high resolution extended area distance, and the weight 943 according to the% position of the low resolution pixel multiplies the pixel corresponding to the low resolution extended area distance, You can get the average pixel value between the two. In this way, in a pixel corresponding to an extended area distance in a low-resolution tile, a low-resolution pixel has a high weight and a high-resolution pixel has a low weight.

The filtering method is a method of calculating and/or correcting the preset portions 933 and 934 where two extended tile images overlap with a high pass filter. In this case, the image quality of the boundary portion can be improved.

Accordingly, the electronic device according to the exemplary embodiment disclosed in the present specification may process the two extended tile images to have continuous image quality in the overlapping preset portions 933 and 934, thereby removing the edge effect (935). ).

When the tile image has a rectangular shape as shown in Fig. 9(b), the extent of the extended area of the extended tile image is a preset portion (margin ratio 927) in the tile image for the four boundaries of one tile image. ), or may be an area extended by the preset portion only for the boundary line 953 adjacent to the tile image of different quality in the tile image 951 as shown in FIG. 9(d). . The electronic device receives image data corresponding to an extended area on the full screen.

In the case of FIG. 9(d), a tile of different quality is adjacent to a tile with different quality only at the left and lower borders of the tile image. As shown in Fig. 9(b), it can be reduced by about 50% compared to the case of acquiring data of the extended area for all four border lines.

In addition, the range of the region for obtaining image data according to the quality of the neighboring tile image or the quality of the tile including the region of interest can be extended to only a part of the boundary of the tile image, and Fig. 9 ( As shown in e), the range 961 of the extended area may be selected in various ways, and the tile area may be expanded in various shapes so that the range 961 of the extended area does not overlap between different tiles.

Therefore, as shown in Figs. 9(d) and 9(e), in the case of reducing the target to be expanded or the range to be expanded without acquiring the image data of the extended area for all four borders of one tile image, However, since the amount of data to be processed can be reduced, the image processing speed to remove the edge effect can be increased.

10 is a diagram illustrating a method of compressing an extended tile image into a tile area.

A method of fitting an extended area to a tile image according to an exemplary embodiment disclosed in the present specification may be fitting into an original tile by compressing and putting image information of the extended area into an image of the original tile at a preset ratio. For example, the image area occupied by the extended area may be fitted in a form in which the image area occupied by the extended area is folded into the inside of the original tile based on a boundary line. That is, the image data of the expanded area and the image data of the tile area symmetrical with the expanded area based on the boundary between the expanded area and the tile area may be compressed into the symmetrical area.

Referring to FIG. 10(a), as shown, the entire area (w + ov), which is the sum of the original area (w) and the extended area (ov) of the tile image, is fitted into the original area (w) of the tile image. I can.

A method of folding the space occupied by the extended area to fit the size of the original tile image will be described in detail as follows.

First, a line-symmetric area based on the boundary of the tile area of the expanded tile is set inside the original tile area. Next, the line symmetrical area inside the tile area is divided into equal parts as shown. Finally, in the left area of the divided area, the image information of the line-symmetric area, that is, the image information of the area corresponding to the portion where the extended tile area is folded into the original tile image is spatially 2 to 1 Compressed and displayed. In addition, the image information of the expanded tile area is spatially compressed and displayed in a 2 to 1 area on the right area of the divided area.

Referring to FIG. 10(b), for example, an original tile image area 1001 of one tile image has an area w of 100 pixels and an extended area 1003 having an area ov of 5 pixels. Then, an image with a total area of 105 pixels (w + ov) (a picture of the sum of the original tile image area and the extended tile area) is spatially compressed and inserted into a 100 pixel area. The area of the portion 1005 in which the extended area 1003 is folded into the original tile image area is the same as the area ov of the extended area. At this time, 95 pixels of the total width of the original tile image are inserted as 1:1, and the remaining 10 pixel portions (ov + ov) are spatially compressed to a 5 pixel area (ov) by 2:1, and then fit. can do.

The above-described spatial compression method of the extended area includes a linear fit method and a weighted fit method in which a weight graph is combined. Hereinafter, a weighted distribution method will be described with reference to FIG. 11.

11 is a diagram illustrating an example of a method of spatially compressing an extended area using a weighted distribution method.

For example, as shown in FIG. 11(a), if the area of the original tile image area is 100 pixels and the area of the extended area is 5 pixels, the 96th pixel is 85% of the original pixel information. The 97th pixel is displayed with 82% pixel information, the 98th pixel is displayed with 75% pixel information, the 99th pixel is displayed with 61% pixel information, and the 100th pixel is displayed with 52% pixel information A pixel fitting weight, such as displaying as information, is applied, and the 101st pixel, which is the area where expansion starts, is displayed with 45% of pixel information, the 102nd pixel is displayed with 35% of pixel information, and the 103rd pixel is 27 A compressed image of the tiled image is generated by spatial interpolation in which pixel fitting weights are applied by displaying as% pixel information. Displaying the specific percentage of pixel information may mean displaying the image information of the target pixel with the corresponding percentage of image quality or pixel size.

FIG. 11(b) is a graph showing pixel weight values of the spatial interpolation method of FIG. 11(a).

At this time, the adjacent tiles are similarly performed with a spatial interpolation method applying pixel fitting weights to the left, right, upper, or lower part of the tile, and the boundary area between the tiles is restored after increasing to the original size. In this case, the spatial interpolation method is performed. The pixels stored by performing have small pixel energy, but since they become basic information for restoring the original image, a restoration operation to minimize the boundary effect can be performed.

12 is a diagram illustrating an example of displaying a tile image based on a region of interest and a boundary line of view.

Referring to FIG. 12(a), when the entire image is displayed with a plurality of tile images generated with a plurality of qualities or resolutions, tiles including the user's region of interest and tiles not including the region of interest are adjacent. A method of displaying a natural image for a case will be described. "High", "Medium", and "Low" indicated on the tiles in the drawing have high image quality such as resolution, frame rate, bit rate, and color depth (bit depth) of the display image of the corresponding tile. It indicates medium and low, and based on the medium, high indicates higher than medium, and low indicates lower than medium.

For example, if an object 1201 that the user is interested in exists in the middle area of the entire image displayed to the user, a specific area including the object 1201 becomes the user's interest area, and the interest area A boundary line 1203 of the user's field of view may be set around. In this case, as shown, the inside and outside of the view boundary line are divided based on the boundary line 1203 of the user's view in the entire image.

The user's region of interest may be determined as an area viewed by a user wearing an HMD or an area expected to be viewed, and may be determined based on a pupil image, a pupil vector, a head vector, etc. generated by measuring with a sensor device provided in the HMD. have.

When the user's region of interest is determined, an image may be displayed in low quality with respect to a tile image of a region not related to the region of interest.

In addition, a user's view boundary line 1203 may be set based on the region of interest, and an image of an area outside the set view boundary line 1203 may be displayed in low quality.

The visual field boundary line 1203 may be set as a shape surrounding the ROI in a predetermined shape and/or size with the ROI in the center. The predetermined shape may be various shapes such as a circle, an ellipse, and a shape in which the corners are rounded in a rectangle. The predetermined size takes into account factors related to the user's field of view (e.g., viewing angle, distance from the eye to the display, eye size, etc.) so that the user can comfortably perceive the image within the field of view. It may be set to include a possible range, and may be set based on the separately derived average user's field-related data. In addition, the viewing boundary may be manually set by the user to reflect the user's individual characteristics. The user's field of view may be continuously changed according to the head movement information of the HMD wearer, and may be changed and set within the entire image according to the change in the direction of the user's pupils. In addition, the viewing boundary line 1203 may be set to include a specific range based on a point viewed by the user regardless of the existence of the object 1201 in the region of interest.

In this case, a high-quality or high-resolution image is displayed on a tile including the object 1201 in the entire image, and a tile including the viewing boundary line 1203 but not including the object 1201 has a medium or medium resolution. An image can be displayed. In addition, among tiles outside the viewing boundary line 1203, a low-quality image may be displayed on a tile that does not include the viewing boundary line 1203. In this case, due to a sharp difference in image quality between tiles displaying a medium-quality or high-quality image including the viewing boundary line 1203 and tiles displaying a low-quality image outside the viewing boundary line 1203 You may feel unnatural about the connection part of the tile video and uncomfortable viewing.

Therefore, in order to reduce user discomfort due to the sudden difference in image quality described above, the image outside the field of view 1203 is processed by low-pass filtering, that is, the resolution is lowered or the original image is displayed as a blurred image. As a method of lowering the image quality, such as, an image outside the user's field of view may be blurred and displayed, and an image within the user's field of view may be displayed in the original quality of the tile image. Accordingly, it is possible to allow a user to view a natural image, and to increase a processing speed of image data for displaying an ROI.

In addition, the low-pass filtering can adjust the degree (intensity) in proportion to the distance from the boundary line 1203 of the region of interest or the field of view. For example, as shown in Fig. 12(b), the electronic device is in proportion to the distance D between the boundary line 1209 of the tile A 1205 and the tile B 1207 and the boundary line 1203 of the user's field of view. You can adjust the blurring level (intensity) of the image.

Referring to FIG. 12(b), the degree of low-pass filtering of an image in proportion to the distance D between the boundary line 1209 of the tile A 1205 and the tile B 1207 described above and the boundary line 1203 of the user's field of view. It will be described in detail how to adjust. As shown, the image is displayed in the original tile image quality for the region of interest inside the boundary line 1203 of the user's field of view, and the boundary line 1203 of the user's field of view for the external display image of the boundary line 1203 of the user's field of view. By performing image processing that gradually increases the degree of low-pass filtering as the distance increases from, it is possible to increase the image processing speed of the user's interest area, and reduce discomfort or discomfort from the user's viewing of the image.

In addition, the low-pass filtering method may vary a degree of low-pass filtering in proportion to a difference in image quality between tiles inside the viewing boundary and outside the viewing boundary.

On the other hand, with respect to the image displayed inside the user's view boundary line 1203, when the quality of neighboring tile images is different, the image quality at the tile boundary continuously changes through the pixel compression method of the expanded tile image described above. You can do it.

For example, the electronic device, for a tile image located inside the user's view boundary line 1203 or passing the view boundary line 1203, the original size of the tile image, that is, an area that is extended by a preset portion from the tile area. After receiving data on at least one extended tile image having image information from outside the content server, etc., the image at the tile boundary is displayed to the user so that the user does not feel uncomfortable through the pixel compression method for the tile boundary. I can.

13 is a diagram illustrating a procedure of a method of displaying a tile image based on an ROI and a boundary line of view.

Hereinafter, an image display method capable of providing an entire image including tile images of different qualities in which a user's viewing disparity is reduced will be described with reference to the drawings.

An electronic device (or a media content receiving device) according to an embodiment disclosed in the present specification may include a control unit and/or a communication module. Details of the control unit may include all of the above-described media content reception and processing, and may include both the above-described processor and the electronic device of FIG. 14 to be described later. Detailed contents of the communication module may include all contents of the communication module disclosed in the present specification.

The electronic device according to the exemplary embodiment disclosed in the present specification may receive stream data including image data of a plurality of tile images from a server or an external content transmission device through a communication module (1310 ). The plurality of tile images may be generated based at least in part on the user's region of interest. The plurality of tile images may be images obtained by converting an image of a three-dimensional three-dimensional space into a two-dimensional plane space, and then dividing the image of the converted planar space into a plurality of tiles.

Also, the electronic device may perform low pass filtering on at least a portion of the tile image based on the ROI (1320). The electronic device may perform low-pass filtering through a low-pass filter provided for some of the plurality of tile images selected based on the ROI. The electronic device may perform low-pass filtering of image data of a tile image that is not related to an ROI. The low-pass filtered tile image or a part of the tile image may be blurred and displayed, or an image of low quality may be displayed.

In addition, the electronic device may instruct to display the entire tile image on the display module after low-pass filtering (1330). The electronic device may combine the low-pass filtered tile image and the unfiltered tile image to display the entire image. The electronic device may display the combined entire image on an equipped display module, and may transmit data on the entire image to a separate display device to display the entire image.

The display module may be provided in the image receiving device or may be a separate device. When the display module is provided in a separate device, the separate device and the image receiving device may transmit and receive image data through a wired or wireless network. In this case, instructing to display may be transmitting the image data to the separate device. As an embodiment, the image receiving device may be a PC or a smart phone, and the device having a separate display module may be a goggle-type VR device connected to the PC or smart phone through a connection means such as a wired/wireless network or a USB terminal.

The degree (intensity) of low-pass filtering, that is, the degree of blurring of the image may be set based on a distance from the ROI. In addition, the low-pass filtering operation may be performed in units of tiles or may be performed in units of pixels of a tile image.

Therefore, the electronic device does not need to receive an extended tile image from an external device such as a content server for all tile images having different resolutions or qualities for the entire image of the 2D plane, and is based on the user's region of interest and the user's field of view. Accordingly, since the extended tile image can be received, a bandwidth for image transmission can be secured, and an image processing speed in the electronic device can be increased.

14 is a diagram illustrating a configuration of an electronic device (or a media content receiving device) according to an exemplary embodiment.

The electronic device 1400 (or media content transmission device) according to the exemplary embodiment disclosed in the present specification may include a control unit 1410 and/or a communication module 1420. The control unit 1410 may include all contents of the above-described processor. In addition, the communication module 1420 may include all contents of the above-described communication module. In addition, the control unit 1410 may include an ROI information generation unit 1411, a request information generation unit 1413, a decoder 1415, a low pass filter 1417 and/or an image combiner 1419.

The communication module 1420 may receive signaling data for obtaining service data of a service for virtual reality, augmented reality, or hologram.

The signaling data may include a media presentation description (MPD) describing tile images encoded with a plurality of qualities for each tile region. In addition, the service data may include a selected tile image corresponding to a tile image encoded with a specific quality for each tile region. In addition, the service data may include a tile image extended with respect to the tile image.

The ROI information generator 1411 may generate ROI information indicating an ROI that is expected to be viewed or viewed by the user in the entire image. Also, the ROI information generator 1411 may set a boundary line of the user's view in the entire image based on at least a part of the ROI information of the user. The ROI may be set as an area expected to be viewed by the user, or may be set by tracking the user's pupil vector using a sensor provided in the electronic device. In addition, a method of setting the boundary line of the user's visual field by the region of interest information generation unit 1411 has been described in the detailed description of FIG. 12A.

The entire image may indicate an image in which a 3D virtual space is converted into a 2D plane.

The request information generator 1413 may generate request information describing the selected tile image based on the signaling data and the ROI information.

The quality of the selected tile image for the region of interest may be higher than the quality of the selected tile image for regions other than the region of interest.

The communication module 1420 may receive the service data based on the request information.

The decoder 1415 may decode the service data into the selected tile image of each tile region.

The low-pass filter 1417 performs low-pass filtering on the image data of the tile image that is not related to the region of interest or the image outside the field of view boundary, thereby improving the image quality of the region not of interest to the user or the image outside the field of view boundary. Can lower it. The low-pass filter may increase the degree of filtering in proportion to the distance from the region of interest or/and the viewing boundary.

Accordingly, the decoder 1415 may decode the image data so that the image inside the viewing boundary is displayed in original quality and the image outside the viewing boundary is displayed lower than the original quality.

Meanwhile, the decoder 1415 may spatially compress the extended region and the image of the extended tile into the inside of the extended tile image so that the image quality continuously changes at the tile boundary. Accordingly, the decoder 1415 uses the above-described extended tile image generation method to ensure that the quality of the image at the boundary of the tile is continuous when the quality of the neighboring tile image is different for the image inside the viewing boundary. You can process the image to change into.

The image combiner 1419 may combine the selected tile image of a plurality of tile regions into an entire image.

Then, the electronic device (or media content receiving device) may provide a service for virtual reality, augmented reality, or hologram to the user.

The media presentation description may include a plurality of adaptation set elements describing the plurality of tile regions. In addition, each adaptation set element may include a plurality of representation elements describing tile images encoded with the plurality of qualities for each tile area. In addition, each representation element may include at least one of location information of the tile region, quality information of the tile image, and URL information of the tile image.

The ROI information may include ROI ranking information indicating the ranking of the tile area that the user is expected to see or look at based on the user's movement. In addition, the ROI ranking information may be generated based on a pupil vector describing the movement of the user's pupil and a head vector describing the movement of the user's head.

The pupil vector may include at least one of a left pupil pupil coordinate vector indicating coordinates of a pupil included in the user's left pupil, and a right pupil pupil coordinate vector indicating coordinates of a pupil included in the user's right pupil. I can.

The signaling data further includes maker intention information indicating the ranking of the tile area that the user is expected to see or the user must see based on the intention of the maker of the service for the virtual reality, augmented reality, or hologram. Can include.

The signaling data may further include motion vector statistical information indicating a representative value of a motion vector for each tile region.

The selected tile image may indicate a tile image encoded with a specific quality selected for each tile region based on at least one of the ROI ranking information, the maker intention information, and the motion vector statistical information.

The request information may include location information of the tile area, quality information of the selected tile image, and URL information of the selected tile image.

The tile image for each tile area may be expanded at a predetermined ratio. Also, pixel values included in a boundary portion between tile images of different qualities may indicate a proportional average value of pixel values of tile images of different qualities.

15 is a diagram illustrating a method of receiving media content according to an exemplary embodiment.

An electronic device (or a media content receiving device) according to an embodiment disclosed in the present specification may include a control unit (not shown) and/or a communication module. Detailed contents of the control unit may include all contents of the above-described processor. Detailed contents of the communication module may include all contents of the above-described communication module.

The electronic device may receive signaling data for acquiring service data of a service for virtual reality, augmented reality, or a hologram using a communication module (1510).

The signaling data may include a media presentation description describing tile images encoded with a plurality of qualities for each tile region. In addition, the service data may include a selected tile image corresponding to a tile image encoded with a specific quality for each tile region.

The electronic device may generate ROI information indicating an ROI that is expected to be viewed or viewed by the user in the entire image using the ROI information generator (1520).

The entire image may indicate an image in which a 3D virtual space is converted into a 2D plane.

The electronic device may generate request information describing the selected tile image based on the signaling data and the ROI information using the request information generator (1530).

The quality of the selected tile image for the region of interest may be higher than the quality of the selected tile image for regions other than the region of interest.

The electronic device may further receive the service data based on the request information by using the communication module (1540).

The electronic device may decode the service data into the selected tile image of each tile region by using a decoder (1550).

The electronic device may combine the selected tile image of a plurality of tile regions into an entire image using an image combiner (1560).

Then, the electronic device (or media content receiving device) may provide a service for virtual reality, augmented reality, or hologram to the user.

16 is a diagram illustrating a procedure of a method for displaying tile images of different qualities through tile image expansion.

Hereinafter, referring to the drawings, an image display method capable of providing an entire tile image in which a user's sense of heterogeneity due to an edge effect is reduced by processing image data of a tile boundary region from tile images of different qualities received from a server is described. Explain.

An electronic device (or a media content receiving device) according to an embodiment disclosed in the present specification may include a control unit and/or a communication module. Details of the control unit may include all of the above-described reception and processing of the media content, and may include both the above-described processor and the electronic device of FIG. 14. Detailed contents of the communication module may include all contents of the above-described communication module.

The electronic device according to the exemplary embodiment disclosed in the present specification may receive an image data stream for a plurality of tile images in which the entire image is displayed in a plurality of qualities through a communication module from a server or an external content transmission device. The plurality of tile images may include at least one extended tile image having image data for a region that is extended by a predetermined portion from an original region of the image tile (1610).

In addition, through a decoder, the electronic device compresses the image of the extended portion of the extended tile image into the interior of the tile area before expansion so that the image quality is continuously changed at the boundary between neighboring tiles although the image quality is different. Can (1620).

In addition, the electronic device may generate an entire tile image by combining the plurality of tile images through an image combiner (1630 ), and may display the entire generated image through a display module.

The tile image may be overscanned so that the extended tile image has image data of an area that is extended by a predetermined portion from the boundary line of the original tile image than the tile area of the original size.

In addition, the extended area may be generated for only a part of the entire boundary line of the tile image, and when the extended area is generated, it is generated so as not to overlap with the extended area of a neighboring tile image. That is, when a plurality of extended tile images are adjacent to each other, each extended region of the plurality of extended tile images may be generated so as not to overlap each other.

In addition, the extended tile image may be generated at a tile boundary portion having a different quality only when the quality of the extended tile image is different from that of the neighboring tile image.

Hereinafter, an embodiment of an image display system capable of providing an image in which a user's sense of heterogeneity due to an edge effect is reduced in an entire image composed of a plurality of tile images of different qualities will be described.

An image display system according to an exemplary embodiment disclosed in the present specification includes a server for generating an image for transmission and an image receiving device for displaying an image.

The image server may divide the entire image into a plurality of tile images and transmit stream data including image data of each of the divided tile images to the image receiving device.

Also, the image server may generate image data of a plurality of tile images based at least in part on the user's region of interest.

The image receiving apparatus may receive stream data from an image server, process the received stream data, and display the received stream data on the display module.

Also, the image receiving apparatus may perform low-pass filtering on at least a part of the tile image based on the ROI, and instruct to display the tile images on the display module after low-pass filtering. The display module may be provided in the image receiving device or may be configured as a separate device, and when the display module is provided in a separate device, the separate device and the image receiving device may transmit and receive image data through a wired or wireless network. have. In this case, instructing to display may be transmitting the image data to the separate device.

As an embodiment, the image receiving device may be a PC or a smart phone, and the device having a separate display module may be a goggle-type VR device connected to the PC or smart phone through a connection means such as a wired/wireless network or a USB terminal.

The plurality of tile images may include at least one extended tile image having image data of an area that is extended by a predetermined portion than the tile area.

The image receiving apparatus may compress the extended tile image into the inside of the tile region so that the image quality continuously changes at the boundary of the tile region and combine the tile images to generate the entire tile image.

Each of the above-described components of the electronic device according to the present disclosure may be composed of one or more components, and the name of the corresponding component may vary according to the type of the electronic device. The electronic device according to the present disclosure may be configured to include at least one of the above-described components, and some components may be omitted or additional other components may be further included. In addition, some of the constituent elements of the electronic device according to the present disclosure are combined to form a single entity, so that functions of the corresponding constituent elements before the combination may be performed in the same manner.

The scope disclosed in the present specification is not limited to the embodiments disclosed herein, and the present invention may be modified, changed, or improved in various forms within the scope described in the spirit and claims disclosed in the present specification.

Claims (13)

Receiving an image data stream of a plurality of tile images including at least one extended tile image having image data of an area extending from the tile area by a predetermined portion at a boundary of at least a portion of the tile area;
Compressing an image of the extended region of the extended tile image into the inside of the tile region so that the image quality continuously changes at the boundary of the tile region; And
Including an operation of combining the plurality of tile images to generate an entire tile image,
The extended tile image has an image quality different from at least one neighboring tile image,
Compressing the image of the extended area into the tile area,
The image of the expanded area is transferred into the symmetrical area based on the image data of the expanded area and the image data of the tile area symmetrical with the expanded area based on the boundary between the expanded area and the tile area. Is to compress,
An image processing method in which the extended area of the extended tile image does not overlap with the extended area of a neighboring extended tile image. delete delete The method of claim 1,
Low-pass filtering image data of a tile image not related to the ROI for at least a part of the entire tile image based on the ROI; And
The image processing method further comprising instructing to display the entire tile image on a display device after low-pass filtering. The method of claim 4,
The degree of the low-pass filtering is set based on a distance from the ROI. The method of claim 4, wherein the low pass filtering operation comprises:
An image processing method that is an operation performed in units of tiles or pixels. The method of claim 4,
The image processing method further comprising determining a boundary line of a user's view based on the region of interest. The method of claim 7, wherein the low-pass filtering operation,
It is performed based on the visual field boundary line,
An image processing method comprising performing low-pass filtering on a tile or a pixel outside the viewing boundary. The method of claim 8,
The level of the low-pass filtering is set based on a distance from the viewing boundary. A communication module configured to receive an image data stream of a plurality of tile images including at least one extended tile image having image data of an area extending from the tile area by a predetermined portion at a boundary of at least a portion of the tile area;
A decoder for compressing an image of the extended region of the extended tile image into the tile region such that image quality continuously changes at the boundary of the tile region; And
Including an image combiner configured to combine the plurality of tile images to generate an entire tile image,
The extended tile image has an image quality different from at least one neighboring tile image,
The decoder makes the image of the extended area symmetrical based on the image data of the extended area and the image data of the tile area symmetrical with the extended area based on the boundary between the extended area and the tile area. Compress it inside the area,
An image processing apparatus in which the extended area of the extended tile image does not overlap with the extended area of a neighboring extended tile image. delete delete delete

Images