KR102603220B1 - The method and apparatus for playing tile based multiview video - Google Patents

Abstract

The present invention discloses a method for a terminal to play a multi-view video. In particular, the method includes receiving a multi-view image from a video management device, dividing the multi-view image into a plurality of images, receiving a request message for changing the location of the plurality of images, and responding to the request message. The method is characterized in that the positions of the plurality of images are changed accordingly.

Description

Method for playing tile-based multi-view video and device therefor {THE METHOD AND APPARATUS FOR PLAYING TILE BASED MULTIVIEW VIDEO}

The present invention relates to a method and device for playing a tile-based multi-view video, and more specifically, to freely change the arrangement of a plurality of images included in a multi-view video without being affected by the decoding performance of the terminal. It relates to a method of reproducing and a device for this.

Conventionally, in order to provide a multi-view video to a terminal, a screen in which the multi-view video is synthesized is provided using production equipment (e.g., a broadcasting car) at the filming site, or the terminal receives a plurality of images and the terminal itself Multi-view playback was performed.

However, this conventional method has the following problems.

First, when providing multi-view video to a terminal using production equipment in the field, the customer could not view the desired video on a relatively large multi-view screen. In other words, since the production equipment synthesizes multiple videos into one video and provides it to the terminal, the video designated by the production equipment can only be viewed larger than other videos on the multi-view screen.

Therefore, even if the user using the terminal does not want to view the video designated by the production equipment in large size, the user has no choice but to view the designated video in large size, and the user's request cannot be satisfied.

Second, when the terminal itself combines multiple images into one multi-view screen, the number of images that can be included in the multi-view screen may be limited depending on the decoding performance of each terminal. For example, if the decoding performance of a specific terminal allows up to 4 screens to be included in the multi-view screen, even if the user wants to watch 8 screens through the multi-view screen, a problem may arise that cannot satisfy this demand. You can.

In addition, in order for the terminal to synchronize multiple videos on its own, a separate player for synchronization must be installed, which may lead to an increase in the cost of producing the terminal.

Specifically, in order to play multiple videos included in a multi-view video, the terminal downloads multiple streams and synchronizes them based on PTS (Presentation Time Stamp) or UTC (Coordinate Universal Time) to play the multi-view video. was played.

However, in the above-described method, the maximum number of video playbacks is limited depending on the decoding performance of the terminal, and the operation to synchronize each video is continuously performed, so the number of multi-views provided is limited depending on the terminal performance.

Therefore, there is a need for a method that can provide a multi-view video containing multiple images regardless of the terminal's decoding performance.

Korean Patent Publication No. 10-2017-0029725 (Name: Method and device for generating lecture images using image synthesis, 2017.03.16)

The present invention seeks to provide a method and device for synthesizing tile-based multi-view images.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

In a method for a terminal to play a multi-view video according to an embodiment of the present invention, a multi-view video is received from a video management device, the multi-view video is divided into a plurality of images, and the positions of the plurality of images are determined. It may be characterized by receiving a request message for change and changing the positions of the plurality of images according to the request message.

At this time, among the plurality of images in the multi-view image, the first image has a first size, the second image and the third image have a second size, and the first size is larger than the second size. It can be big.

Additionally, if the request message includes changing the positions of the second video and the third video, the terminal may change the positions of the second video and the third video.

Additionally, if the request message includes changing the positions of the first video and the second video, the terminal may reduce the first video to the second size and change the position to the second video. .

Additionally, the terminal may receive the second image corresponding to the first size from the image management device, and synthesize the second image corresponding to the first size at the position of the first image.

Additionally, the multi-view image may be one in which the plurality of images are arranged in a tile-based format.

Additionally, dividing the plurality of images into the plurality of images may mean dividing the plurality of images in units of the tiles.

Additionally, it may include receiving size information and location information of tiles corresponding to each of the plurality of images from the image management device.

Additionally, the plurality of images may be inserted frame by frame into a canvas for each of the plurality of images.

Additionally, the frequency with which each of the plurality of images is updated on the canvas for each of the plurality of images may be based on the number of frames per second (FPS).

Additionally, the canvas may be based on HTML 5 (HyperText Markup Language 5).

In addition, the method may further include performing decoding and rendering on the multi-view image before dividing the multi-view image into the plurality of images.

A terminal for playing multi-view video according to the present invention, comprising: an input unit for receiving a message from a user; A communication unit that transmits and receives data to and from the video management device; a storage unit that stores the data; and a control unit that controls the operation of the terminal, wherein the control unit controls the communication unit to receive a multi-view image from the image management device, divides the multi-view image into a plurality of images, and inputs the A request message for changing the location of the plurality of images may be received through the unit, and the location of the plurality of images may be changed according to the request message.

At this time, among the plurality of images in the multi-view image, the first image has a first size, the second image and the third image have a second size, and the first size is larger than the second size. It can be big.

Additionally, when the request message includes changing the positions of the second video and the third video, the control unit can change the positions of the second video and the third video.

Additionally, if the request message includes changing the positions of the first image and the second image, the control unit may reduce the first image to the second size and change the position to the second image. .

In addition, the control unit controls the communication unit to receive the second image corresponding to the first size from the image management device, and receives the second image corresponding to the first size at the location of the first image. can be synthesized.

Additionally, the multi-view image may be one in which the plurality of images are arranged in a tile-based format.

Additionally, dividing the plurality of images into the plurality of images may mean dividing the plurality of images in units of the tiles.

Additionally, the control unit may include controlling the communication unit to receive size information and location information of tiles corresponding to each of the plurality of images from the image management device.

Additionally, the plurality of images may be inserted frame by frame into a canvas for each of the plurality of images.

Additionally, the frequency with which each of the plurality of images is updated on the canvas for each of the plurality of images may be based on the number of frames per second (FPS).

Additionally, the canvas may be based on HTML 5 (HyperText Markup Language 5).

Additionally, the control unit may further include performing decoding and rendering on the multi-view image before dividing the multi-view image into the plurality of images.

A multi-view video playback system for playing multi-view video according to the present invention, comprising: an image providing device that generates each of a plurality of images and transmits each of the plurality of images to an image synthesis device; the image synthesis device that receives the plurality of images from the image providing device and synthesizes the plurality of images into the multi-view image; a video management device that stores the multi-view video and transmits the multi-view video to the terminal according to a request from the terminal; and receiving the multi-view image from the video management device, dividing the multi-view image into a plurality of images, receiving a request message for changing the location of the plurality of images, and receiving the plurality of images according to the request message. It may include a terminal that changes the location of the images.

According to the present invention, a tile-based synthesized multi-view image is divided into tile units, and each image is played using the HTML5 Canvas function to play the multi-view. Since only one synthesized image is decoded, terminal performance is limited. You can use Canvas to smoothly move between images and create an intuitive movement effect.

In addition, the present invention reproduces multi-view video on a tile basis, so that it is not limited by terminal decoding performance and can provide multi-view video to a terminal capable of playing a single video without performance restrictions, even if there are more than N videos.

For example, the invention can be applied not only to PC environments where terminal performance is very diverse, but also to web-based smart TVs where terminal performance is very limited.

In addition, by using HTML5 Canvas, the video divided into tiles can be freely moved, providing an intuitive and free multi-view experience that allows users to arrange the screen as they wish, rather than providing a fixed multi-view.

The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

1 is a diagram for explaining a multi-view screen playback system according to an embodiment of the present invention.
2 and 3 are diagrams for explaining a multi-view screen playback method according to an embodiment of the present invention.
Figure 4 is a diagram for explaining a method of changing a plurality of images included in a multi-view screen according to an embodiment of the present invention.
Figure 5 is a diagram for explaining a method for a terminal to play a multi-view screen according to an embodiment of the present invention.
Figure 6 is a diagram for explaining the configuration of a terminal according to an embodiment of the present invention.

In order to make the characteristics and advantages of the problem-solving means of the present invention clearer, the present invention will be described in more detail with reference to specific embodiments of the present invention shown in the attached drawings.

However, detailed descriptions of known functions or configurations that may obscure the gist of the present invention are omitted in the following description and attached drawings. Additionally, it should be noted that the same components throughout the drawings are indicated by the same reference numerals whenever possible.

Terms or words used in the following description and drawings should not be construed as limited to their usual or dictionary meanings, and the inventor may appropriately define the concept of the term to explain his or her invention in the best way. It must be interpreted with meaning and concept consistent with the technical idea of the present invention based on the principle that it is. Therefore, the embodiments described in this specification and the configurations shown in the drawings are only one of the most preferred embodiments of the present invention, and do not represent the entire technical idea of the present invention, so at the time of filing the present application, various alternatives may be used to replace them. It should be understood that equivalents and variations may exist.

In addition, terms containing ordinal numbers, such as first, second, etc., are used to describe various components, and are used only for the purpose of distinguishing one component from other components and to limit the components. Not used. For example, the second component may be referred to as the first component without departing from the scope of the present invention, and similarly, the first component may also be referred to as the second component.

Additionally, the terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In addition, terms such as "comprise" or "have" used in the specification are intended to indicate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are intended to indicate the presence of one or more of the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification. It should be understood that this does not exclude in advance the presence or addition of other features, numbers, steps, operations, components, parts, or combinations thereof.

Additionally, terms such as “unit,” “unit,” and “module” used in the specification refer to a unit that processes at least one function or operation, and may be implemented as hardware, software, or a combination of hardware and software. In addition, the terms "a or an", "one", "the", and similar related terms are used in the context of describing the invention (particularly in the context of the claims below) as used herein. It may be used in both singular and plural terms, unless indicated otherwise or clearly contradicted by context.

In addition to the terms described above, specific terms used in the following description are provided to aid understanding of the present invention, and the use of these specific terms may be changed to other forms without departing from the technical spirit of the present invention.

Additionally, embodiments within the scope of the present invention include computer-readable media having or transmitting computer-executable instructions or data structures stored on the computer-readable media. Such computer-readable media may be any available media that can be accessed by a general-purpose or special-purpose computer system. By way of example, such computer-readable media may include RAM, ROM, EPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, or in the form of computer-executable instructions, computer-readable instructions or data structures. It may be used to store or transmit certain program code means, and may include, but is not limited to, a physical storage medium such as any other medium that can be accessed by a general purpose or special purpose computer system. .

In addition, the present invention is applicable to personal computers, laptop computers, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, and pagers. It can be implemented in a network computing environment with various types of computer system configurations including (pager) and the like. The invention may also be practiced in distributed system environments where tasks are performed by both local and remote computer systems that are linked over a network via wired data links, wireless data links, or a combination of wired and wireless data links. In a distributed system environment, program modules may be located in local and remote memory storage devices.

Additionally, it will be understood that each block of the processing flow diagrams and combinations of the flow diagram diagrams may be performed by computer program instructions. These computer program instructions can be mounted on a processor of a general-purpose computer, special-purpose computer, or other programmable data processing equipment, so that the instructions performed through the processor of the computer or other programmable data processing equipment are described in the flow chart block(s). It creates the means to perform functions. These computer program instructions may also be stored in computer-usable or computer-readable memory that can be directed to a computer or other programmable data processing equipment to implement a function in a particular manner, so that the computer-usable or computer-readable memory It is also possible to produce manufactured items containing instruction means that perform the functions described in the flowchart block(s). Computer program instructions can also be mounted on a computer or other programmable data processing equipment, so that a series of operational steps are performed on the computer or other programmable data processing equipment to create a process that is executed by the computer, thereby generating a process that is executed by the computer or other programmable data processing equipment. Instructions that perform processing equipment may also provide steps for executing the functions described in the flow diagram block(s).

Additionally, each block may represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical function(s). Additionally, it should be noted that in some alternative execution examples it is possible for the functions mentioned in the blocks to occur out of order. For example, it is possible for two blocks shown in succession to be performed substantially at the same time, or it is possible for the blocks to be performed in reverse order depending on the corresponding function.

In describing the embodiments of the present disclosure in detail, the main focus will be on the example of a specific system, but the main point claimed in the present specification is that the scope disclosed in the present specification is applicable to other communication systems and services with similar technical background. It can be applied within a range that does not deviate significantly, and this can be done at the discretion of a person with skilled technical knowledge in the relevant technical field.

Hereinafter, we will look in detail at the method and system for playing multi-view images according to an embodiment of the present invention.

1 is a diagram illustrating the structure of an image synthesis system for reproducing multi-view images according to an embodiment of the present invention.

Referring to FIG. 1, an image synthesis system for playing multi-view images according to an embodiment of the present invention includes a communication network 1, an image providing device 100, an image synthesis device 200, an image management device 300, and It may be configured to include a terminal 400.

Here, the terminal 400 and the video management device 300 may be linked through the communication network 1.

At this time, the communication network 1 may be a wireless communication network or a wired communication network. For example, the communication network 1 may be a wireless communication network using LTE (Long Term Evolution), 5G (5 Generation), and WiFi.

Specifically, the communication network 1 serves to transmit data for data transmission and reception between the terminal 400 and the video management device 300, and depending on the system implementation method, Ethernet, xDSL (ADSL, VDSL), and HFC You can also use wired communication methods such as (Hybrid Fiber Coaxial Cable), FTTC (Fiber to The Curb), and FTTH (Fiber To The Home), as well as WLAN (Wireless LAN), Wi-Fi, Wibro, Wireless communication methods such as Wimax, HSDPA (High Speed Downlink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution Advanced), and 5G (5-Generation) can also be used.

Meanwhile, looking at 5G, a communication field to which the present invention can be typically applied, the three main requirements areas of 5G are (1) improved mobile broadband (eMBB) area, and (2) large number of machine types. (3) Massive Machine Type Communication (mMTC) area and (3) Ultra-reliable and Low Latency Communications (URLLC) area.

Some use cases may require multiple areas for optimization, while others may focus on just one Key Performance Indicator (KPI). 5G supports these diverse use cases in a flexible and reliable way.

eMBB goes far beyond basic mobile Internet access and covers rich interactive tasks, media and entertainment applications in the cloud or augmented reality. Data is one of the key drivers of 5G, and we may not see dedicated voice services for the first time in the 5G era. In 5G, voice is expected to be processed simply as an application using the data connection provided by the communication system. The main reasons for the increased traffic volume are the increase in content size and the number of applications requiring high data rates. Streaming services (audio and video), interactive video and mobile Internet connections will become more prevalent as more devices are connected to the Internet. Many of these applications require always-on connectivity to push real-time information and notifications to users. Cloud storage and applications are rapidly increasing mobile communication platforms, and this can apply to both work and entertainment. And cloud storage is a particular use case driving growth in uplink data rates. 5G will also be used for remote work in the cloud and will require much lower end-to-end latency to maintain a good user experience when tactile interfaces are used. Entertainment, for example, cloud gaming and video streaming are other key factors driving increased demand for mobile broadband capabilities. Entertainment is essential on smartphones and tablets anywhere, including high mobility environments such as trains, cars and airplanes. Another use case is augmented reality for entertainment and information retrieval. Here, augmented reality requires very low latency and instantaneous amounts of data.

Additionally, one of the most anticipated 5G use cases concerns the ability to seamlessly connect embedded sensors in any field, or mMTC. By 2020, the number of potential IoT devices is expected to reach 20.4 billion. Industrial IoT is one area where 5G will play a key role in enabling smart cities, asset tracking, smart utilities, agriculture and security infrastructure.

URLLC includes new services that will transform industries through ultra-reliable/available low-latency links, such as remote control of critical infrastructure and self-driving vehicles. Levels of reliability and latency are essential for smart grid control, industrial automation, robotics, and drone control and coordination.

5G can complement fiber-to-the-home (FTTH) and cable-based broadband (or DOCSIS) as a means of delivering streams rated at hundreds of megabits per second to gigabits per second. These high speeds are required to deliver TV at resolutions above 4K (6K, 8K and beyond) as well as virtual and augmented reality. Virtual Reality (VR) and Augmented Reality (AR) applications include nearly immersive sporting events. Certain applications may require special network settings. For example, for VR games, gaming companies may need to integrate core servers with a network operator's edge network servers to minimize latency.

Automotive is expected to be an important new driver for 5G, with many use cases for mobile communications for vehicles. For example, entertainment for passengers requires simultaneous, high capacity and high mobility mobile broadband. That's because future users will continue to expect high-quality connections regardless of their location and speed. Another use case in the automotive sector is augmented reality dashboards. It identifies objects in the dark and superimposes information telling the driver about the object's distance and movement on top of what the driver is seeing through the front window. In the future, wireless modules will enable communication between vehicles, information exchange between vehicles and supporting infrastructure, and information exchange between cars and other connected devices (eg, devices accompanied by pedestrians). Safety systems can reduce the risk of accidents by guiding drivers through alternative courses of action to help them drive safer. The next step will be remotely controlled or self-driven vehicles. This requires highly reliable and very fast communication between different self-driving vehicles and between cars and infrastructure. In the future, self-driving vehicles will perform all driving activities, leaving drivers to focus only on traffic abnormalities that the vehicles themselves cannot discern. The technical requirements of self-driving vehicles call for ultra-low latency and ultra-high reliability, increasing traffic safety to levels unachievable by humans.

Smart cities and smart homes, referred to as smart societies, will be embedded with high-density wireless sensor networks. A distributed network of intelligent sensors will identify conditions for cost-effective and energy-efficient maintenance of a city or home. A similar setup can be done for each household. Temperature sensors, window and heating controllers, burglar alarms and home appliances are all connected wirelessly. Many of these sensors are typically low data rate, low power, and low cost. However, real-time HD video may be required in certain types of devices for surveillance, for example.

Consumption and distribution of energy, including heat or gas, is highly decentralized, requiring automated control of distributed sensor networks. A smart grid interconnects these sensors using digital information and communications technologies to collect and act on information. This information can include the behavior of suppliers and consumers, allowing smart grids to improve the efficiency, reliability, economics, sustainability of production and distribution of fuels such as electricity in an automated manner. Smart grid can also be viewed as another low-latency sensor network.

Mission critical applications (e.g., e-health) are one of the 5G usage scenarios. The health sector has many applications that can benefit from mobile communications. Communications systems can support telemedicine, providing clinical care in remote locations. This can help reduce the barrier of distance and improve access to health services that are consistently unavailable in remote rural areas. It is also used to save lives in critical care and emergency situations. Mobile communications-based wireless sensor networks can provide remote monitoring and sensors for parameters such as heart rate and blood pressure.

Wireless and mobile communications are becoming increasingly important in industrial applications. Wiring is expensive to install and maintain. Therefore, the possibility of replacing cables with reconfigurable wireless links is an attractive opportunity for many industries. However, achieving this requires that wireless connections operate with similar latency, reliability and capacity as cables, and that their management be simplified. Low latency and very low error probability are new requirements needed for 5G connectivity.

Logistics and freight tracking are important examples of mobile communications that enable inventory and tracking of packages anywhere using location-based information systems. Use cases in logistics and cargo tracking typically require low data rates but require wide range and reliable location information.

In addition, this communication network 1 includes, for example, a plurality of access networks (not shown) and a core network (not shown), and may be configured to include an external network, such as an Internet network (not shown). Here, the access network (not shown) may be an access network that performs wired and wireless communication with the terminal 400 and the video management device 300. For example, it can be implemented with multiple base stations such as BS (Base Station), BTS (Base Transceiver Station), NodeB, eNodeB, gNodeB, and base station controllers such as BSC (Base Station Controller) and RNC (Radio Network Controller). there is.

In addition, as described above, the digital signal processing unit and the wireless signal processing unit integrated in the base station are divided into a digital unit (hereinafter referred to as DU) and a radio unit (hereinafter referred to as RU), respectively. , it can be configured by installing multiple RUs (not shown) in multiple areas and connecting multiple RUs (not shown) with a centralized DU (not shown).

In addition, the core network (not shown) that constitutes the mobile network together with the access network (not shown) serves to connect the access network (not shown) with an external network, such as an Internet network (not shown).

As described above, this core network (not shown) is a network system that performs major functions for mobile communication services such as mobility control and switching between access networks (not shown), and is a network system that performs circuit switching or packet switching. switching) and manages and controls packet flow within the mobile network. In addition, the core network (not shown) manages mobility between frequencies and plays a role in linking traffic within the access network (not shown) and the core network (not shown) with other networks, such as the Internet network (not shown). It may be possible. This core network (not shown) further includes Serving GateWay (SGW), PDN GateWay (PGW), Mobile Switching Center (MSC), Home Location Register (HLR), Mobile Mobility Entity (MME), and Home Subscriber Server (HSS). It may also be configured to include.

In addition, the Internet network (not shown) refers to a typical open communication network, that is, a public network, in which information is exchanged according to the TCP/IP protocol. It is connected to the terminal 400 and the video management device 300, and the terminal ( Information provided from 400) can be provided to the video management device 300 through a core network (not shown) and an access network (not shown), and conversely, information provided to the video management device 300 can be provided to the core network (not shown). ) and an access network (not shown) may be provided to the terminal 400. However, it is not limited to this, and the video management device 300 may be implemented integrally with a core network (not shown).

In addition, in addition to the above-mentioned communication methods, it may include all other types of communication methods that are widely known or will be developed in the future. Additionally, the communication network 1 may be configured as a combination of two or more of the wired and wireless communication networks described above. For example, the communication network 1 may be implemented as a combination of a wired communication network and a wireless communication network, or a combination of two or more wireless communication networks.

Meanwhile, the image management device 300, the image synthesis device 200, and the image providing device 100 may be configured as one device. If the video management device 300, the video synthesis device 200, and the video provision device 100 are configured as separate devices, the video management device 300, the video synthesis device 200, and the video provision device 100 ) can be linked through a wired or wireless interface.

The terminal 400 according to FIG. 1 may refer to a user's device that can transmit and receive various data through the communication network 100 according to the user's operation.

This terminal 400 can perform voice or data communication through the communication network 1, and can transmit and receive information with the video management device 300 through the communication network 1. For this purpose, the terminal 400 of the present invention may be equipped with a memory for storing programs and protocols for transmitting and receiving data through the communication network 1, and a microprocessor for calculating and controlling various programs.

This terminal 400 first connects to an application (hereinafter referred to as 'app') providing device (not shown) connected to the communication network (1), such as an app store, and provides an app for providing insurance coverage details from the app store. You can receive and install it.

Additionally, this terminal 400 may be implemented in various forms. For example, the terminal 400 described in this specification includes mobile terminals such as smart phones, tablet PCs, personal digital assistants (PDAs), portable multimedia players (PMPs), and MP3 players. , fixed terminals such as Smart TVs, desktop computers, etc. may also be used.

In addition, the terminal 400 of the present invention has so many variations of portable devices according to the convergence trend of digital devices that it is not possible to list them all, but units equivalent to the above-mentioned units according to the present invention. Any device that can be used as the terminal 400 and can download and install an app related to the multi-view video according to the present invention can be used as the terminal 400 according to an embodiment of the present invention.

According to an embodiment of the present invention, the terminal 400 may request the video management device 300 to stream a multi-view video through the communication network 1. At this time, the terminal 400 may request streaming of a specific multi-view video among a plurality of multi-view videos stored in the video management device 300.

At this time, the specific multi-view video may mean a multi-view video in which the video that the user of the terminal 400 wants to view in the largest size occupies the largest proportion of the multi-view video.

That is, in a multi-view image, among a plurality of images, one image is arranged at the largest size, and the remaining images are arranged at the same size, but may be arranged based on a tile shape.

When a specific multi-view video is streamed from the video management device 300 to the terminal 400 at the request of the terminal 400, the terminal 400 can play the specific multi-view video based on streaming. Meanwhile, while the terminal 400 is playing a specific multi-view video, if it receives a request from the user to play another multi-view video, it can request the video management device 300 to stream another multi-view video.

When another multi-view video is streamed from the video management device 300, the terminal 400 plays the other multi-view video based on streaming. At this time, the terminal 400 requests another multi-view video and specifies a specific multi-view video. The time required to change from one image to another multi-view image may be less than a certain amount of time (for example, 300 ms).

Meanwhile, the multi-view video that the terminal 400 is streaming is one in which the video synthesis device 200 synthesizes a plurality of images into one image. Accordingly, when receiving streaming multi-view video from the video management device 300, the terminal 400 streams one video file.

At this time, the terminal 400 can perform decoding and rendering on the streaming multi-view image. Additionally, the terminal 400 may divide the multi-view image into a plurality of images in tile units. For example, multi-view images can be divided into tiles so that one image is arranged at the largest size and the remaining images are arranged at the same size.

Information on the size of each tile and the location of each tile for arranging a plurality of images in a multi-view image is received in advance through the image management device 300, or the terminal 400 is used to stream a specific multi-view image. You can receive it together when you request it.

The terminal 400 may divide the multi-view image into a plurality of images in units of tiles based on the information about the size and location of each tile described above. As a result, among the plurality of images, one image will be divided into the largest size, and the remaining images will be divided into the same size.

The terminal 400 can insert a plurality of divided images into each HTML 5 (HypeText Markup Language) canvas on a frame-by-frame basis. That is, each HTML 5 canvas can be mapped one-to-one with each tile, and each of the plurality of images divided into tile units can be inserted frame by frame into the HTML 5 canvas that is mapped to the tile related to each of the plurality of images. You can.

Meanwhile, HTML 5 is a next-generation web document standard and is a "web programming language" that has evolved to express and provide various applications such as multimedia compared to existing HTML. By providing various additional functions such as audio, video, graphics processing, and location information provision in the existing HTML4, the web's own processing capabilities have been significantly improved.

The basic structure of HTML 5 is the same as existing HTML. In HTML, all content is expressed using various tags. And, the content of HTML is divided into a HEAD part and a BODY part.

The big difference between HTML 5 and existing HTML lies in graphics processing. In HTML5, Canvas and SVG can be used to replace graphics processing implemented using Flash and Silverlight.

Canvas does not itself implement graphics, but defines an area for graphic implementation in a web document, and actually implements graphics using code created with the Canvas API through JavaScript. And the pixel-based immediate mode graphics system allows the screen to be redrawn every frame. There is the advantage of being able to control bitmaps directly on a web page, but it is also cumbersome because pixels are handled at a low level.

Through Canvas, you can draw anything in the browser window, edit images interactively, and control videos. Additionally, video compositing is possible using a green screen. Additionally, you can provide users with real-time graphs, animations, or interactive games without having to download separate plug-ins.

Meanwhile, SVG (Scalable Vector Graphic) is an XML-based open standard vector graphics file format developed by W3C in 1999. SVG is a shape-based maintenance mode graphics system. The list of objects to be drawn on the screen is in the graphics renderer, and objects can be drawn on the screen according to the characteristics (x-coordinate, y-coordinate, transparency) of the object specified in the code. Images in SVG format and their operations are defined as XML text files, which can be searched, cataloged, scripted, and compressed if necessary.

This makes SVG suitable for all mobile devices. Additionally, because SVG is an XML-based graphic, it can be edited with a plain text editor. However, it is generally more productive to produce with a graphic editor (tool) that is saved in SVG format, so it is common to work mainly with a graphic editor. SVG, like canvas, can be provided to graphic users without a separate plug-in.

Meanwhile, the terminal 400 can stream a plurality of images inserted in frame units and output them to the user. At this time, a request may be made to change the positions of two images among a plurality of images by user input.

In this case, the terminal 400 may directly change the position between the two requested images without requesting the video management device 300 to replace the multi-view image. For example, the terminal 400 may change the positions of the two requested images by changing the positions between canvases for each of the two requested images.

That is, the terminal 400 can directly change the position between the two requested videos without separate stream replacement.

However, in this case, the size of the canvas and/or the size of the tile for each of the two images may be limited to being the same.

In other words, when the size of the tile and/or the size of the canvas for each of the two requested images are the same, the terminal 400 directly determines the position between the canvases for each of the two requested images without separate stream replacement. By performing a change, the positions of the two images can be changed.

If the tile size and/or canvas size for each of the two requested images are different, if the small size image is changed to a large size canvas and/or tile, the screen area where the small size image must be drawn As this increases, image quality may deteriorate.

Therefore, the terminal 400 requests the video management device 300 to provide a stream of video of a size matching the canvas and/or tile based on the large-sized canvas and/or tile that needs to be changed from the small-sized video. ) can be requested. When the video management device 300 provides the video stream again, the terminal 400 can insert the video into a large canvas and complete the process.

Meanwhile, inserting a large-sized video into a small-sized canvas and/or tile does not cause image quality deterioration, and re-receiving a separate stream only causes a time delay in video playback. Accordingly, the change of the large-sized image can be completed by reducing its size to match the small-sized canvas and/or tile and inserting it into the small-sized canvas and/or tile.

For example, in the multi-view screen where the terminal 400 is streaming, the image corresponding to the largest canvas and/tile is the first image, and the images corresponding to the canvas and/tile of the same size are the second image and It is called the third video.

If the terminal 400 receives a request from the user to replace the second video and the third video, the terminal 400 does not request separate video transmission from the video management device 300 without separate stream replacement. Instead, the image replacement task can be completed by directly changing the position between the second canvas of the requested second image and the third canvas of the third image.

If the terminal 400 receives a request from the user to replace the first image and the second image, the terminal 400 informs the image management device 300 of the size of the first canvas and/or first tile of the first image. A request may be made to transmit a stream of a second video having a quality and size corresponding to . When the terminal 400 receives a stream of the second video from the video management device 300, the received second video can be inserted into the first canvas and/or the first tile. Additionally, the terminal 400 changes the size and quality of the first image to match the second canvas and/or the second tile, inserts the changed first image into the second canvas and/or the second tile, and The position change between the image and the second image may be completed.

Meanwhile, the plurality of images streaming to the terminal 400 may be updated and/or changed on a frame-by-frame basis on the canvas corresponding to each of the plurality of images. In other words, the image on the canvas can be updated and/or changed on a per second (FPS) basis. For example, if the video is at 30 fps, the image on the canvas can be updated and/or changed 30 times per second. That is, when streaming to the terminal 400, 30 frames per second are received through the canvas corresponding to each video, and the frames 30 times per second are continuously changed and/or updated through the canvas and transmitted through the terminal 400. can be printed.

The video management device 300 stores a plurality of multi-view images received from the video synthesis device 200, and, at the request of the terminal 400, selects one of the stored multi-view images based on a streaming method. It is a device for transmission.

In addition, while providing the first multi-view image in a streaming manner at the request of the terminal 400, if the terminal 400 requests to provide a second multi-view image, streaming of the first multi-view image is stopped and , the second multi-view video can be streamed.

At this time, the video management device 300 sets the time required from receiving the request to provide the second multi-view video to changing the streaming from the first multi-view video to the second multi-view video to less than a certain amount of time (e.g., 300ms).

Additionally, the video management device 300 may transmit each of a plurality of images included in the multi-view image to the terminal 400.

For example, during the initial request, the terminal 400 requests a specific multi-view image, and while streaming the specific multi-view image, the terminal 400 changes the position between a plurality of images included in the specific multi-view image. To this end, when requesting streaming of a specific video with an image quality and size corresponding to the canvas and/or tile of the largest size, the video management device 300 streams the specific video to the canvas and/or tile of the largest size. A specific video can be converted to quality and size and streamed and/or transmitted to the terminal 400.

The image providing device 100 is a device for transmitting a plurality of images to the image synthesis device 200. For example, the video providing device 100 may be a device that includes an MPEG2-TS Demuxer. Additionally, the plurality of images may be images encoded according to the H.265 standard.

The image synthesis device 200 is a device for synthesizing a plurality of images into a plurality of multi-view images. Specifically, the image synthesis device 200 outputs a plurality of images received from the image providing device 100 through a multi-encoder. Additionally, a plurality of videos can be decoded and synchronized based on the Presentation Time Stamp (PTS) or Coordinated Universal Time (UTC) of each of the plurality of videos.

Here, PTS (Presentation Time Stamp) is a timestamp metadata field that indicates the separated element streams of the program (separate) when the MPEG transport stream or MPEG program stream (e.g. video, audio, subtitles) is provided to the viewer. It is used to achieve synchronization of Elementary Streams.

PTS is provided in units related to the overall clock reference of the program, i.e., program clock reference (PCR) or system clock reference (SCR), and can be transmitted over the transport stream or program stream.

The resolution of PTS is 90kHz, suitable for presentation synchronization tasks. PCR or SCR has a resolution of 27 MHz. The resolution of these PTS, PCR and SCR includes driving TV signals such as frame and line sync timing, color sub carrier, etc., synchronizing the decoder's overall clock with that of a typical remote encoder. suitable for

Decoding of N common streams involves adjusting and synchronizing the decoding of the streams based on the common master time, which is more efficient than adjusting the decoding of one stream to match the decoding of another stream. At this time, the common master time may be one of the clocks of N decoders, a clock of a data source, or an external clock.

A transport stream can contain multiple programs, and each program can have its own time base. The time base of each program in the transport stream may be different. Since PTS applies to the decoding of individual elementary streams, both the transport stream and the program stream are in the PES packet layer. End-to-end synchronization can occur when the encoder stores timestamps at capture time, when timestamps are propagated to the decoder along with the associated coded data, and when the decoder uses these timestamps to schedule presentation.

Synchronization of channels and decoding systems is achieved using SCR in the program stream and its analog, PCR, in the transport stream. SCR and PCR are timestamps that encode the timing of the bit stream itself and can be derived from the same time base used for audio and video PTS values in the same program. Because each program may have its own time base, a transport stream containing multiple programs will include a separate PCR field for each program. In some cases, it is possible for programs to share PCR fields.

Additionally, UTC is a time scale held by time research institutes around the world, and is determined to be a very accurate atomic clock. UTC time is accurate to approximately nanoseconds (billionths of a second) per day. Additionally, UTC is based on the current Gregorian year and is located at 0 degrees latitude, the prime meridian.

Gregorian UTC is used as the starting point for system time calculations. To calculate the system time from UTC, a certain offset from the system's UTC may be added to UTC. The offset can specify the number of hours and minutes the system is west or east of UTC. With respect to the prime meridian, west has a negative offset and east has a positive offset. And, the offset can be applied to UTC to calculate the system time.

Meanwhile, the image synthesis device 200 can synthesize a plurality of images output simultaneously into a plurality of multi-view images. At this time, a plurality of multi-view images can be synthesized based on the tile shape. Additionally, the number of multi-view images may be equal to the number of images.

In other words, a plurality of multi-view images are synthesized based on the tile shape, meaning that among the plurality of images, one image is included in the multi-view image in the largest size, and the remaining images are placed in the multi-view image with the same size. You can.

For example, in the first multi-view image, among a plurality of images, the first image is placed at the top left so as to occupy 70% of the entire screen of the first multi-view image, and the remaining images are located on the right and bottom of the first image. They can be listed in the same size.

In addition, in the same way, among the plurality of images in the second multi-view image, the second image is placed in the upper left corner so as to occupy 70% of the entire screen of the second multi-view image, and the remaining images are located on the right and left sides of the second multi-view image. They can be listed in the same size at the bottom.

In other words, all of the plurality of images can be arranged on the largest screen once in the plurality of multi-view images.

The image synthesis device 200 can encode a plurality of multi-view images and package them to enable service. At this time, in the above-described encoding and packaging process, DASH (Dynamic Adaptive Streaming over HTTP) and/or CMAF (Common Media Application Format) can be used.

DASH is one of the typical bitrate streaming technologies that enables high-quality streaming of media delivered from traditional HTTP web servers. Similar to Apple's HTTP Live Streaming (HLS) solution, DASH breaks content into small HTTP-based file segments. Additionally, each segment includes a section of video of a certain length (for example, a short interval) during the playback time of potentially long content such as a movie, sports event, and/or live broadcast.

At this time, the content may use various bitrates. When content is played by a DASH clarient, the client can use a bitrate adaptation algorithm to select segments with the highest possible bitrate for download without causing pauses or rebuffering during playback.

The current MPEG-DASH reference client dash.js provides both buffer-based (BOLA) bitrate adaptation algorithm and hybrid (DYNAMIC) bitrate adaptation algorithm.

As a result, MPEG-DASH clients can adapt to changing network conditions and provide high-quality playback with little stopping or re-buffering.

MPEG-DASH is the first internationally standardized adaptive bitrate HTTP-based streaming solution. MPEG-DASH should not be confused with the transport protocol. The transport protocol used by MPEG-DASH is TCP . MPEG-DASH uses the existing HTTP web server infrastructure used for the delivery of all World Wide Web content. It allows devices such as televisions, TV set-top boxes, desktop computers, smartphones, tablets, etc. connected to the Internet to consume multimedia content (e.g. video, TV, radio, etc.) delivered via the Internet and provides various Internet reception capabilities. Deal with the situation.

Meanwhile, CMAF is a new standard for providing adaptive bit transmission at high speeds.

CMAF can reduce complexity and latency by using a common media format in video streaming, and can reduce investment costs for encoding and storing multiple copies of the same content.

Additionally, complexity can be minimized by simplifying the workflow, and video delay can be reduced by improving streaming efficiency.

Prior to CMAF, the HLS protocol used the MPEG transport stream container format, or .ts (MPEG-TS). Additionally, other HTTP-based protocols such as DASH used the fragmented MP4 format, or .mp4 (fMP4).

However, Microsoft and Apple have agreed to distribute content to more users through HLS and DASH protocols using standardized transport containers in the form of fragmented MP4s. In other words, the purpose of CMAF is to distribute content using only mp4. CMAF has two encoding methods to reduce latency: chunk encoding and chunk transfer encoding.

This process allows video to be divided into smaller chunks of set duration, allowing the separated chunks to be output immediately after encoding. This way, preceding chunks can be delivered in near real-time (e.g., within 3 to 5 seconds) while subsequent chunks are being processed.

Additionally, the image synthesis device 200 may transmit a plurality of encoded multi-view images to the image management device 300.

Now, with reference to FIGS. 2 to 5, let us look at a multi-view image playback method according to an embodiment of the present invention.

Referring to Figures 2 and 3, the terminal 400 may request the video management device 300 to stream a multi-view video. At this time, the terminal 400 may request streaming of a specific multi-view video among a plurality of multi-view videos stored in the video management device 300.

At this time, the specific multi-view video may mean a multi-view video in which the video that the user of the terminal 400 wants to view in the largest size occupies the largest proportion of the multi-view video.

That is, in a multi-view image, among a plurality of images, one image is arranged at the largest size, and the remaining images are arranged at the same size, but may be arranged based on a tile shape.

According to the request of the terminal 400, the terminal 400 may receive a specific multi-view video from the video management device 300 based on a streaming method (S201).

At this time, referring to FIG. 3, the multi-view image provided to the terminal 400 may be generated through the following process. A plurality of images encoded by the image providing device 100 may be transmitted to a multi-encoder and transmitted to the image synthesis device 200 through the multi-encoder. The image synthesis apparatus 200 may decode a plurality of images, synchronize each of the multiple images based on a Presentation Time Stamp (PTS) or Coordinated Universal Time (UTC), and synthesize them.

At this time, a plurality of multi-view images are synthesized based on the tile shape, so that among the plurality of images, one image is included in the multi-view image in the largest size, and the remaining images are arranged in the same size in the multi-view image. It can be.

Additionally, a plurality of images may be arranged on the largest screen once from a plurality of multi-view images.

The plurality of synthesized multi-view images can be transmitted to and stored in the video management device 300 and transmitted to the terminal at the request of the terminal 400.

Meanwhile, the multi-view video that the terminal 400 is streaming is one in which the video synthesis device 200 synthesizes a plurality of images into one image. Accordingly, when receiving streaming multi-view video from the video management device 300, the terminal 400 streams one video file.

At this time, the terminal 400 can perform decoding and rendering on the streaming multi-view image (S203). Additionally, the terminal 400 may divide the multi-view image into a plurality of images in tile units (S205). For example, multi-view images can be divided into tiles so that one image is arranged at the largest size and the remaining images are arranged at the same size.

Information on the size of each tile and the location of each tile for arranging a plurality of images in a multi-view image is received in advance through the image management device 300, or the terminal 400 is used to stream a specific multi-view image. You can receive it together when you request it.

The terminal 400 may divide the multi-view image into a plurality of images in units of tiles based on the information about the size and location of each tile described above. As a result, among the plurality of images, one image will be divided into the largest size, and the remaining images will be divided into the same size.

The terminal 400 may insert a plurality of divided images into each HTML 5 (HypeText Markup Language) canvas on a frame-by-frame basis (S207). That is, as can be seen in Figure 3, each HTML 5 canvas can be mapped one-to-one with each tile, and each of the plurality of images divided into tile units is mapped to a tile related to each of the plurality of images. It can be inserted into the HTML 5 canvas on a frame-by-frame basis.

Meanwhile, the terminal 400 can stream a plurality of images inserted in frame units and output them to the user. At this time, a request may be made to change the positions of two images among a plurality of images by user input.

In this case, the terminal 400 may directly change the position between the two requested images without requesting the video management device 300 to replace the multi-view image (S209). For example, the terminal 400 may change the positions of the two requested images by changing the positions between canvases for each of the two requested images.

That is, the terminal 400 can directly change the position between the two requested videos without separate stream replacement.

However, in this case, the size of the canvas and/or the size of the tile for each of the two images may be limited to being the same.

In other words, if the size of the tile and/or the size of the canvas for each of the two requested videos are the same, the terminal 400 can replace the sub-videos without separate stream replacement, as can be seen in the replacement scenario between sub-videos in FIG. 5. The positions of the two images can be changed by directly changing the positions between the canvases for each of the two requested images.

If the tile size and/or canvas size for each of the two requested images are different, if the small size image is changed to a large size canvas and/or tile, the screen area where the small size image must be drawn As this increases, image quality may deteriorate.

Therefore, as can be seen in the main-sub video replacement scenario in Figure 5, the terminal 400 changes the small-sized video to the canvas and/or tile based on the large-sized canvas and/or tile that needs to be changed. The video management device 300 may be requested to provide a video stream of a matching size. When the video management device 300 provides the video stream again, the terminal 400 can insert the video into a large canvas and complete the process.

Meanwhile, inserting a large-sized video into a small-sized canvas and/or tile does not cause image quality deterioration, and re-receiving a separate stream only causes a time delay in video playback. Accordingly, the change of the large-sized image can be completed by reducing its size to match the small-sized canvas and/or tile and inserting it into the small-sized canvas and/or tile.

For example, in the multi-view screen where the terminal 400 is streaming, the image corresponding to the largest canvas and/tile is the first image, and the images corresponding to the canvas and/tile of the same size are the second image and It is called the third video.

If the terminal 400 receives a request from the user to replace the second video and the third video, like the sub-video replacement scenario in FIG. 5, the terminal 400 does not require separate stream replacement, that is, the video management device ( The image replacement task can be completed by directly changing the position between the second canvas of the requested second image and the third canvas of the third image, without requesting separate image transmission to 300).

If the terminal 400 receives a request from the user to replace the first video and the second video, like the main-sub video replacement scenario in FIG. 5, the terminal 400 sends the first video to the video management device 300. A request may be made to transmit a stream of a second video having image quality and size corresponding to the size of the first canvas and/or the first tile. When the terminal 400 receives a stream of the second video from the video management device 300, the received second video can be inserted into the first canvas and/or the first tile. Additionally, the terminal 400 changes the size and quality of the first image to match the second canvas and/or the second tile, inserts the changed first image into the second canvas and/or the second tile, and The position change between the image and the second image may be completed.

Meanwhile, the plurality of images streaming to the terminal 400 may be updated and/or changed on a frame-by-frame basis on the canvas corresponding to each of the plurality of images. In other words, the image on the canvas can be updated and/or changed on a per second (FPS) basis. For example, if the video is at 30 fps, the image on the canvas can be updated and/or changed 30 times per second. That is, when streaming to the terminal 400, 30 frames per second are received through the canvas corresponding to each video, and the frames 30 times per second are continuously changed and/or updated through the canvas and are transmitted through the terminal 400. can be printed.

Figure 6 is a diagram for explaining the configuration of the terminal 400 according to the present invention.

Referring to FIG. 6, the terminal 400 according to the present invention may be configured to include a control unit 410, a communication unit 420, a storage unit 430, an input unit 440, and an output unit 450.

The input unit 440 receives various information such as number and character information, and transmits input signals related to setting various functions and controlling functions of the terminal 400 to the control unit 410. Additionally, the input unit 440 may be configured to include at least one of a keypad and a touchpad that generate an input signal according to a user's touch or manipulation. At this time, the input unit 440 is configured in the form of a touch panel (or touch screen) together with the output unit 450 and can perform input and display functions simultaneously. Additionally, the input unit 440 may use any type of input means that may be developed in the future, in addition to input devices such as a keyboard, keypad, mouse, joystick, etc.

In particular, the input unit 440 according to the present invention can receive a request for changing the position between a plurality of images included in the multi-view screen from the user.

The output unit 450 displays information about a series of operation states and operation results that occur while the terminal 400 performs its functions. Additionally, the output unit 450 can display menus of the terminal 400 and user data input by the user. Here, the output unit 450 includes a liquid crystal display (LCD), an ultra-thin film transistor LCD (TFT-LCD), a light emitting diode (LED), and an organic light emitting diode (OLED). Organic LED), active organic light emitting diode (AMOLED, Active Matrix OLED), Retina Display, flexible display, and 3 Dimensional display. At this time, when the output unit 450 is configured in the form of a touch screen, the output unit 450 may perform some or all of the functions of the input unit 440.

In particular, the output unit 450 according to the present invention can display, play, and/or output a multi-view image including a plurality of images.

The storage unit 430 is a device for storing data, includes a main memory device and an auxiliary memory device, and stores application programs necessary for functional operation of the terminal 400. This storage unit 430 may largely include a program area and a data area. Here, when the terminal 400 activates each function in response to the user's request, the terminal 400 executes the corresponding application programs under the control of the control unit 410 to provide each function.

In particular, the storage unit 430 according to the present invention can temporarily or permanently store the entire streaming multi-view video and/or a plurality of images divided from the multi-view video.

Here, temporary storage may mean storing the entire multi-view image and/or multiple images in a buffer for streaming. Additionally, permanently storing may mean downloading the entire multi-view video and/or multiple videos.

The communication unit 420 can transmit and receive data with the video management device 300 through the communication network 1. In addition, the communication unit 420 includes an RF transmitting means for up-converting and amplifying the frequency of the transmitted signal, an RF receiving means for low-noise amplifying the received signal and down-converting the frequency, and data for processing a communication protocol according to a specific communication method. Including processing means, etc. This communication unit 420 may include at least one of a wireless communication module (not shown) and a wired communication module (not shown). In addition, the wireless communication module is configured to transmit and receive data according to a wireless communication method, and when the terminal 400 uses wireless communication, it uses any one of the wireless network communication module, wireless LAN communication module, and wireless fan communication module. Data can be transmitted and received with the video management device 300. Here, the communication unit 420 may include a plurality of communication modules. When a plurality of communication modules are included in the communication unit 42, one communication module can perform PAN (Personal Area Network) type communication including Bluetooth.

Additionally, other communication modules may use wireless communication methods such as Wireless LAN (WLAN), Wi-Fi, Wibro, Wimax, and High Speed Downlink Packet Access (HSDPA).

In particular, the communication unit 420 according to the present invention may request the video management device 300 to stream a specific video among the multi-view video and/or a plurality of videos included in the multi-view video. Additionally, the communication unit 420 may receive streaming for at least one image among the multi-view image and/or a plurality of images included in the multi-view image from the image management device 300.

The control unit 410 may be a process device that drives an operating system (OS) and each component.

That is, the control unit 410 can control the overall operation process of the terminal 400.

Specifically, let's look at the function of the control unit 410 according to an embodiment of the present invention.

The control unit 410 may control the communication unit 420 to request the video management device 300 to stream a multi-view video. At this time, the terminal 400 may control the communication unit 420 to request streaming of a specific multi-view video among the plurality of multi-view videos stored in the video management device 300.

At this time, the specific multi-view video may mean a multi-view video in which the video that the user of the terminal 400 wants to view in the largest size occupies the largest proportion of the multi-view video.

That is, in a multi-view image, among a plurality of images, one image is arranged at the largest size, and the remaining images are arranged at the same size, but may be arranged based on a tile shape.

In response to a request from the control unit 410, the control unit 410 may control the communication unit 420 to receive a specific multi-view image from the video management device 300 based on a streaming method.

Meanwhile, the multi-view image that the control unit 410 receives as a stream is one in which the image synthesis device 200 synthesizes a plurality of images into one image. Accordingly, when receiving streaming multi-view video from the video management device 300, the control unit 410 streams one video file.

At this time, the control unit 410 may perform decoding and rendering on the streaming multi-view image. Additionally, the control unit 410 may divide the multi-view image into a plurality of images in tile units. For example, multi-view images can be divided into tiles so that one image is arranged at the largest size and the remaining images are arranged at the same size.

The control unit 410 controls the communication unit 420 to receive information about the size and location of each tile for arranging multiple images in the multi-view image in advance through the image management device 300. , When the control unit 410 requests streaming of a specific multi-view video, the communication unit 420 can be controlled to receive it together.

The control unit 410 may divide the multi-view image into a plurality of images in units of tiles based on the above-described information about the size and location of each tile. As a result, among the plurality of images, one image will be divided into the largest size, and the remaining images will be divided into the same size.

The control unit 410 may insert a plurality of divided images into each HTML 5 (HypeText Markup Language) canvas on a frame-by-frame basis. That is, each HTML 5 canvas can be mapped one-to-one with each tile, and each of the plurality of images divided into tile units can be inserted frame by frame into the HTML 5 canvas that is mapped to the tile related to each of the plurality of images. You can.

Meanwhile, the control unit 410 can control the output port 450 to stream a plurality of images inserted in frame units and output them to the user. At this time, the control unit 410 may receive a request through the input unit 440 to change the positions of two images among the plurality of images by a user's input.

In this case, the control unit 410 may directly change the position between the two requested images without requesting the image management device 300 to replace the multi-view image. For example, the control unit 410 may change the positions of the two requested images by performing a position change between the canvases for each of the two requested images.

That is, the control unit 410 can directly change the position between the two requested videos without separate stream replacement.

However, in this case, the size of the canvas and/or the size of the tile for each of the two images may be limited to being the same.

In other words, if the tile size and/or canvas size for each of the two requested images are the same, the control unit 410 directly determines the position between the canvases for each of the two requested images without separate stream replacement. By performing a change, the positions of the two images can be changed.

If the tile size and/or canvas size for each of the two requested images are different, if the small size image is changed to a large size canvas and/or tile, the screen area where the small size image must be drawn As this increases, image quality may deteriorate.

Therefore, the control unit 410 instructs the communication unit 420 to provide a stream of images of a size matching the canvas and/or tile based on the large size canvas and/or tile that needs to be changed from the small size image. A request can be made to the video management device 300 through. When the video management device 300 provides the video stream again, the control unit 410 can complete the video by inserting it into a large canvas.

Meanwhile, inserting a large-sized video into a small-sized canvas and/or tile does not cause image quality deterioration, and re-receiving a separate stream only causes a time delay in video playback. Accordingly, the change of the large-sized image can be completed by reducing its size to match the small-sized canvas and/or tile and inserting it into the small-sized canvas and/or tile.

For example, in the multi-view screen where the control unit 410 is streaming, the image corresponding to the largest size canvas and/tile is the first image, and the images corresponding to the canvas and/tile of the same size are the second image and It is called the third video.

If the control unit 410 receives a request from the user to replace the second video and the third video through the input unit 440, the control unit 410 sends a separate stream to the video management device 300 without separate stream replacement. The image replacement task can be completed by directly changing the position between the second canvas of the requested second image and the third canvas of the third image, without requesting video transmission.

If the control unit 410 receives a request from the user to replace the first image and the second image through the input unit 440, the control unit 410 sends the first canvas and/or the first image to the image management device 300. Alternatively, a request may be made through the communication unit 420 to transmit a second video stream with image quality and size corresponding to the size of the first tile. When the control unit 410 receives a stream of the second video from the video management device 300 through the communication unit 420, the control unit 410 inserts the received second video into the first canvas and/or the first tile. You can. Additionally, the control unit 410 changes the size and quality of the first image to match the second canvas and/or the second tile, inserts the changed first image into the second canvas and/or the second tile, and The position change between the image and the second image may be completed.

Meanwhile, the control unit 410 may update and/or change the plurality of streaming images on a canvas corresponding to each of the plurality of images on a frame-by-frame basis. In other words, the image on the canvas can be updated and/or changed on a per second (FPS) basis. For example, if the video is at 30 fps, the image on the canvas can be updated and/or changed 30 times per second. That is, when a plurality of videos are streamed, 30 frames per second are received through the canvas corresponding to each video, and the 30 frames per second are continuously changed and/or updated through the canvas and output through the terminal 400. The control unit 410 can control as much as possible.

As described above, although this specification contains details of numerous specific implementations, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather may be specific to particular embodiments of a particular invention. It should be understood as a description of features. Certain features described herein in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination. Furthermore, although features may be described as operating in a particular combination and initially claimed as such, one or more features from a claimed combination may in some cases be excluded from that combination, and the claimed combination may be a sub-combination. It can be changed to a variant of a sub-combination.

Likewise, although operations are depicted in the drawings in a particular order, this should not be construed as requiring that those operations be performed in the specific order or sequential order shown or that all of the depicted operations must be performed to obtain desirable results. In certain cases, multitasking and parallel processing may be advantageous. Additionally, the separation of various system components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and the described program components and systems may generally be integrated together into a single software product or packaged into multiple software products. You must understand that it is possible.

Specific embodiments of the subject matter described herein have been described. Other embodiments are within the scope of the following claims. For example, the operations recited in the claims can be performed in a different order and still achieve desirable results. As an example, the process depicted in the accompanying drawings does not necessarily require the specific depicted order or sequential order to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

The present description sets forth the best mode of the invention and provides examples to illustrate the invention and to enable any person skilled in the art to make and use the invention. The specification prepared in this way does not limit the present invention to the specific terms presented. Accordingly, although the present invention has been described in detail with reference to the above-described examples, those skilled in the art may make modifications, changes, and variations to the examples without departing from the scope of the present invention.

Therefore, the scope of the present invention should not be determined by the described embodiments, but by the scope of the patent claims.

The present invention relates to a method and device for playing a tile-based multi-view video, and more specifically, to freely change the arrangement of a plurality of images included in a multi-view video without being affected by the decoding performance of the terminal. It relates to a method of reproducing and a device for this.

According to the present invention, a tile-based synthesized multi-view image is divided into tile units, and each image is played using the HTML5 Canvas function to play the multi-view. Since only one synthesized image is decoded, terminal performance is limited. You can use Canvas to smoothly move between images and create an intuitive movement effect.

In addition, the present invention reproduces multi-view video on a tile basis, so that it is not limited by terminal decoding performance and can provide multi-view video to a terminal capable of playing a single video without performance restrictions, even if there are more than N videos.

For example, the invention can be applied not only to PC environments where terminal performance is very diverse, but also to web-based smart TVs where terminal performance is very limited.

In addition, by using HTML5 Canvas, the video divided into tiles can be freely moved, providing an intuitive and free multi-view experience that allows users to arrange the screen as they wish, rather than providing a fixed multi-view.

Therefore, the present invention can contribute to the development of the overall media industry through a method for playing tile-based multi-view video and a device for the same, and not only has sufficient potential for commercialization or sales, but also can be clearly implemented in reality, so it can be used industrially. There is availability.

1: Communication network 100: Video providing device 200: Video synthesis device
300: Video management device 400: Terminal

Claims (25)

In a method for a terminal to play a multi-view video,
Receive multi-view video from a video management device,
Splitting the multi-view image into a plurality of images,
Receiving a request message to change the positions of a first image and a second image among the plurality of images,
When the size of the first area including the first image and the size of the second area including the second image in the multi-view image are the same:
Moving the first image to the second area, moving the second image to the first area,
In the multi-view image, the size of the first area including the first image and the size of the second area including the second image are different, and the size of the first area is larger than the size of the second area. case:
Reduce the size of the first image so that the first image becomes the same size as the second area while maintaining the quality of the first image, and then move the first image to the second area,
Receiving the second image having an image quality corresponding to the size of the first area from the image management device, and compositing the second image into the first area,
How to play multi-view video. delete delete delete delete According to claim 1,
The multi-view image is one in which the plurality of images are arranged in a tile-based format,
How to play multi-view video. According to claim 6,
Splitting into the plurality of images,
Splitting the plurality of images on a tile basis,
How to play multi-view video. According to claim 6,
Including receiving size information and location information of a tile corresponding to each of the plurality of images from the image management device,
How to play multi-view video. According to claim 1,
The plurality of images are inserted frame by frame into a canvas for each of the plurality of images,
How to play multi-view video. According to clause 9,
The frequency with which each of the plurality of images is updated on the canvas for each of the plurality of images is,
Based on Frames per Second (FPS),
How to play multi-view video. According to clause 9,
The canvas is based on HTML 5 (HyperText Markup Language 5),
How to play multi-view video. According to claim 1,
Before dividing the multi-view image into the plurality of images,
Further comprising performing decoding and rendering on the multi-view image,
How to play multi-view video. In a terminal that plays multi-view video,
An input unit that receives a message from the user;
A communication unit that transmits and receives data to and from the video management device;
a storage unit that stores the data; and
It includes a control unit that controls the operation of the terminal,
The control unit,
Controlling the communication unit to receive a multi-view image from the video management device,
Splitting the multi-view image into a plurality of images,
Receiving a request message to change the positions of a first image and a second image among the plurality of images through the input unit,
When the size of the first area including the first image and the size of the second area including the second image in the multi-view image are the same:
Moving the first image to the second area, moving the second image to the first area,
In the multi-view image, the size of the first area including the first image and the size of the second area including the second image are different, and the size of the first area is larger than the size of the second area. case:
Reduce the size of the first image so that the first image becomes the same size as the second area while maintaining the quality of the first image, and then move the first image to the second area,
Receiving the second image having an image quality corresponding to the size of the first area from the image management device through the communication unit, and compositing the second image into the first area,
Terminal. delete delete delete delete According to claim 13,
The multi-view image is one in which the plurality of images are arranged in a tile-based format,
Terminal. According to claim 18,
Splitting into the plurality of images,
Splitting the plurality of images on a tile basis,
Terminal. According to claim 18,
The control unit,
Including controlling the communication unit to receive size information and location information of tiles corresponding to each of the plurality of images from the image management device,
Terminal. According to claim 13,
The plurality of images are inserted frame by frame into a canvas for each of the plurality of images,
Terminal. According to claim 21,
The frequency with which each of the plurality of images is updated on the canvas for each of the plurality of images is,
Based on Frames per Second (FPS),
Terminal. According to claim 21,
The canvas is based on HTML 5 (HyperText Markup Language 5),
Terminal. According to claim 13,
The control unit,
Before dividing the multi-view image into the plurality of images,
Further comprising performing decoding and rendering on the multi-view image,
Terminal. In a multi-view video playback system for playing multi-view video,
an image providing device that generates each of a plurality of images and transmits each of the plurality of images to an image synthesis device;
the image synthesis device that receives the plurality of images from the image providing device and synthesizes the plurality of images into the multi-view image;
a video management device that stores the multi-view video and transmits the multi-view video to the terminal according to a request from the terminal; and
Receiving the multi-view image from the video management device, dividing the multi-view image into a plurality of images, and receiving a request message for changing the positions of the first image and the second image among the plurality of images. do,
When the size of the first area including the first image and the size of the second area including the second image in the multi-view image are the same:
Moving the first image to the second area, moving the second image to the first area,
In the multi-view image, the size of the first area including the first image and the size of the second area including the second image are different, and the size of the first area is larger than the size of the second area. case:
Reduce the size of the first image so that the first image becomes the same size as the second area while maintaining the quality of the first image, and then move the first image to the second area,
A terminal that receives the second image having an image quality corresponding to the size of the first area from the image management device and synthesizes the second image in the first area.
Multi-view video playback system.