KR102414377B1 - method of decoding plenoptic video by use of pre-processing of color component decomposition - Google Patents

Abstract

According to the present invention, the pre-processing streaming server for the plenoptic image sets image rendering conditions through interaction with the client terminal, and performs pre-processing of dividing and transcoding the plenoptic image for each production element in response thereto, and then provides it to the client terminal. It relates to a color element division preprocessing-based plenoptic image decoding technology that enables a client terminal to quickly decode a plenoptic image and display it to a user by configuring it. According to the present invention, by reducing the amount of streaming data for plenoptic video, the burden on the transmission network (eg, 5G infrastructure) can be reduced, and there is an advantage that the processing load of the client terminal can be reduced through pre-processing. Through this, it is possible to reduce the burden of implementation of the plenoptic video content service and to lower the product cost by lowering the hardware specification of the client terminal. In addition, according to the present invention, since it is possible to reduce the data processing burden of the client terminal handling the plenoptic image, there is an advantage that a high-quality plenoptic image can be applied to the image content service.

Description

Method of decoding plenoptic video by use of pre-processing of color component decomposition

FIELD OF THE INVENTION The present invention relates generally to techniques for rapidly decoding plenoptic images.

In particular, according to the present invention, the pre-processing streaming server sets image rendering conditions through interaction with the client terminal for the plenoptic image, and performs pre-processing of dividing and transcoding the plenoptic image by production elements in response to the plenoptic image. It relates to a plenoptic image decoding technology based on color component division pre-processing that enables a client terminal to quickly decode a plenoptic image and display it to a user by providing it.

Unlike a general two-dimensional image, a plenoptic image provides additional information about light. That is, it is very suitable for realistically depicting objects as it provides light intensity and color information in various sampled directions for each pixel constituting a two-dimensional image. Accordingly, the plenoptic image can be subjected to various image processing such as perspective viewing change, refocusing, and 3D depth of field extraction.

There are technical challenges to be solved in order to utilize these functions of plenoptic images in the application area.

In particular, the plenoptic data is very large because it includes light information (intensity of light, color) in various directions for each pixel of a two-dimensional image. In order to provide a plenoptic image to a client terminal in a streaming method, the load of a data transmission network (eg, a mobile communication network) is quite large, and accordingly, the plenoptic image is generally unsuitable for a streaming service. In addition, since a plenoptic image has to perform a complex operation to render each pixel, a processing load on the client terminal is very high, and accordingly, the plenoptic image is not suitable for a real-time application.

If you try to view a plenoptic image by accessing a server that provides Internet plenoptic images using a general personal computer or smartphone, you have to wait a very long time until the image is rendered and output. Plenoptic video is not suitable for use in real-time streaming services, and if it is to be applied to some applications, the client terminal must have very high performance.

Republic of Korea Patent Publication No. 10-2020-0052206 (published date: May 14, 2020) "Plenoptic image processing apparatus, plenoptic image processing system including same, and object segmentation method thereof"

Son Uk-ho et al., 2016, "Plenoptic Image Processing Technology Trend", Electronic Communication Trend Analysis, Vol. 31, No. 4, pp. 1 to 12 J. G. Marichal-Hernandez, “Obtencion de Infor macion Tridimensional de una Escena a partir de Sensores Plenopticos usando Procesamiento de Senales con Hardware Grafico,” PhD Thesis, University of La Laguna, 2012

SUMMARY OF THE INVENTION It is generally an object of the present invention to provide a technique for rapidly decoding plenoptic images.

In particular, an object of the present invention is to perform pre-processing of a plenoptic image by a pre-processing streaming server setting an image rendering condition through interaction with a client terminal, and dividing and transcoding the plenoptic image by production elements in response to the plenoptic image. It is to provide a plenoptic image decoding technology based on color element division preprocessing that enables a client terminal to quickly decode a plenoptic image and display it to a user by configuring it to be provided to a terminal.

On the other hand, the problem to be solved of the present invention is not limited to these matters, and other problems to be solved can be understood from the description of the present specification.

In order to achieve the above object, in the method for decoding a plenoptic image based on color element division preprocessing according to the present invention, the preprocessing streaming server 300 separates and extracts the plenoptic image for each color element to obtain an image for each color element. first step; a second step of extracting, by the pre-processing streaming server 300, an ROI (Region of Interest) by reflecting the target focus position for the plenoptic image; a third step of applying differential transcoding so that the pre-processing streaming server 300 separates the ROI part and the non-ROI part in the image for each color element so that the ROI part is relatively high-quality; A fourth step of the pre-processing streaming server 300 providing an image for each color element to the client terminal 400; and a fifth step in which the client terminal 400 renders a display screen by individually decoding and synthesizing images for each color element.

The second step in the present invention, the pre-processing streaming server 300 to identify the target focus position; The pre-processing streaming server 300 extracting a DOI (Depth of Interest) based on the target focus position; extracting, by the pre-processing streaming server 300, depth information of the plenoptic image; The pre-processing streaming server 300 extracts a pixel mass having depth information corresponding to the DOI from the plenoptic image and sets it as an ROI; may be configured to include.

The second step in the present invention, the pre-processing streaming server 300 dividing the foreground part and the background part based on the depth information of the plenoptic image; The pre-processing streaming server 300 extracts the pixel mass corresponding to the foreground part and sets it as an ROI; may be configured to include.

A plenoptic video decoding method according to the present invention includes the steps of: a preprocessing streaming server 300 identifying a display specification through interaction with a client terminal 400; The pre-processing streaming server 300 resizing the image for each color element by reflecting the display specifications; may be configured to further include.

The method for decoding a plenoptic image according to the present invention may further include, by the preprocessing streaming server 300, removing the empty region noise for each image for each color element.

On the other hand, the computer program according to the present invention is stored in the medium in order to execute the plenoptic image decoding method based on the color element division preprocessing as described above in combination with hardware.

According to the present invention, by reducing the amount of streaming data for plenoptic video, the burden on the transmission network (eg, 5G infrastructure) can be reduced, and there is an advantage that the processing load of the client terminal can be reduced through pre-processing. Through this, it is possible to reduce the burden of implementation of the plenoptic video content service and to lower the product cost by lowering the hardware specification of the client terminal.

In addition, according to the present invention, since it is possible to reduce the data processing burden of the client terminal handling the plenoptic image, there is an advantage that a high-quality plenoptic image can be applied to the image content service.

1 is a diagram illustrating the concept of a plenoptic image.
2 is a diagram illustrating an example of obtaining depth information from a plenoptic image.
[Fig. 3] is a diagram showing the overall system configuration for processing the color element division preprocessing-based plenoptic image decoding technology according to the present invention.
[Fig. 4] is a flowchart showing the entire process of a method for decoding a plenoptic image based on color element division preprocessing according to the present invention.
[Fig. 5] is a diagram showing the concept of color element division and transcoding process in the present invention.
6 is a diagram showing an example of color element division and transcoding process in the present invention.

Hereinafter, the present invention will be described in detail with reference to the drawings.

1 is a diagram illustrating the concept of a plenoptic image. In general, as a result of shooting an object (subject) in the real world with a camera, a two-dimensional image is in the form of a two-dimensional image recorded on a focal plane (canvas plane). It is possible to express a sense of depth or process a three-dimensional effect through a combination of RGB (Red Green Blue) and depth information.

On the other hand, the plenoptic image dealt with in the present invention includes information on the light to which an actual object is exposed in a real space. For each pixel constituting a two-dimensional image, it is suitable for realistically depicting an object on a display device as in real space, including light intensity and color information in various directions.

The plenoptic image information additionally includes angular domain information related to a light irradiation direction in addition to spatial domain information provided in a general two-dimensional image. In a plenoptic image by using depth information of a spatial domain and additional direction information of light rays by each domain, such as perspective viewing change, refocusing, 3D depth of field extraction, etc. Various image processing can be performed.

Since the plenoptic image information has both spatial information and angular information in a three-dimensional space, it is possible to reproduce light in the real world on a subject. Plenoptic information in three-dimensional space can be expressed as intensity with respect to the coordinate system (θ, Φ, λ, t, x, y, z,). Among them, θ and Φ are the direction of light, and λ is the wavelength. (color), t is time, and (x, y, z) is the 3D position of the observer. Through this, the intensity of light at each three-dimensional point of the subject is digitized and mathematically interpreted.

The feature of the plenoptic image is that it can be viewed by selecting a focal point and a viewing point through interaction with the user during playback. The former is a function that displays by focusing on a location desired by the user and is called refocusing. All-in-focus, which focuses not only on a feature location but also on all locations, is also available. It is possible. The latter is a function of displaying a viewpoint as if there is a viewpoint at a location desired by the user, and this viewpoint shifting image is obtained through a sub-aperture image.

In addition, the plenoptic image can extract depth information relatively accurately and use it. [Fig. 2] is a diagram illustrating an example of obtaining depth information from a plenoptic image. First, images (focal images) at various focal positions are generated using plenoptic image refocusing, and the result is called a focal stack. A focus value is obtained for each pixel (x, y) of a focal image, and a focal position of the focal image having the highest focus value in the focus stack is selected as the depth of the corresponding pixel. An object can be segmented in a plenoptic image using a depth map.

On the other hand, as the plenoptic technology, there exist an integral imaging method by a camera array, a volume type method in which actual light emitting points are physically formed in a volume, and a holographic method using the interference effect of light. The present invention is not limited in any one of these ways.

[Fig. 3] is a diagram showing the overall system configuration for processing a color element division preprocessing-based plenoptic image decoding technology according to the present invention, and [Fig. 4] is a color element division preprocessing-based plenoptic image decoding technology according to the present invention. It is a flowchart showing the entire process of the method.

First, referring to [Fig. 3], the entire system processing the plenoptic image decoding technology according to the present invention is a plenoptic camera 100, an image storage 200, a preprocessing streaming server 300, and a client terminal 400. includes

The plenoptic camera 100 is a device for generating a plenoptic image by photographing a subject in real space. In the present invention, as the plenoptic camera 100, a conventional dedicated camera product, for example, Lytro's Lytroilum product or Raytrix's light field camera may be used.

The image storage 200 is a device for collecting and storing plenoptic images. Since the plenoptic image includes additional image information from various angles for each pixel at each location, it has an average data capacity of 10 times or more compared to the existing single-focal-based image. Accordingly, the image storage 200 may include a mass storage device and additionally include compression storage technology.

The preprocessing streaming server 300 is a device that provides an image content device to a plurality of client terminals 400 using a plenoptic image stored in the image storage 200 and a data transmission network (eg, 5G). Pre-processing The streaming server 300 performs pre-processing on the plenoptic image so that the plenoptic image content service can be smoothly provided based on the normal client terminal 400 and the data transmission network. In the present invention, the pre-processing is performed based on color element division and differential transcoding, which will be described later in detail with reference to FIG. 4 .

The client terminal 400 is a device that displays a plenoptic image to a user, and may be a personal computer, a smart phone, a digital signage device, an over-the-top (OTT) box, an IP set-top box, or the like. It may be configured as a dedicated device, or may be configured by installing dedicated software on a general-purpose device.

Next, with reference to [Fig. 4], the entire process of the color element division preprocessing-based plenoptic image decoding method according to the present invention will be described.

Step (S100): First, the pre-processing streaming server 300 separates and extracts the plenoptic image for each color element to obtain an image for each color element. As an example of color component division, it may be considered that a plenoptic image is divided into a plenoptic R (Red) image, a plenoptic G (Green) image, and a plenoptic B (Blue) image. A series of image frames (30 images per second) constituting a plenoptic image can be separated and extracted by R, G, and B to compose a series of frame sequences for each color element. On the other hand, the present invention is not limited to RGB, and it is also possible to configure other types of color element division. In addition, the number of color elements is not limited to three, and it is possible to implement using fewer or more color elements than that.

Step (S110): The pre-processing streaming server 300 identifies the target focus position for the plenoptic image, and extracts the ROI (Region of Interest) from the plenoptic image by reflecting the viewing target focus position.

A first embodiment of step S110 will be described. First, the pre-processing streaming server 300 identifies the target focus position for the plenoptic image, and may identify the target focus position through interaction with the client terminal 400, or may use an auto-focus function. Then, a depth of interest (DOI), which is a depth value corresponding to the focus position of the viewer, is extracted from the plenoptic image. Then, the pre-processing streaming server 300 extracts the depth information of the plenoptic image in the same manner as described above with reference to, for example, [Fig. 2], extracts the pixel mass having the depth information corresponding to the DOI, and extracts the ROI (interest area) is set.

A second embodiment of step S110 will be described. The preprocessing streaming server 300 extracts the depth information of the plenoptic image in the same manner as described above with reference to, for example, [Fig. 2], and automatically divides the plenoptic image into a foreground part and a background part based on the depth information. . Then, a pixel mass corresponding to this foreground part is extracted and set as an ROI (region of interest).

Steps (S120, S130): And, the preprocessing streaming server 300 identifies a display specification through interaction with the client terminal 400, and resizes the image for each color element by reflecting this. In the case of a high-definition plenoptic image, the pre-processing streaming server 300 performs such resizing in advance, thereby remarkably reducing the processing burden of the client terminal 400 .

Step (S140): The pre-processing streaming server 300 applies differential transcoding by reflecting the ROI to the image for each color element. That is, the ROI part and the non-ROI part are divided in the image for each color element, and differential transcoding is applied so that the ROI part is relatively high-quality. For example, the density value of the color element may be set differentially in the ROI part and the non-ROI part, and the encoding loss rate may be differentially applied to the ROI part and the non-ROI part. Through this configuration, the user perceives the image as a high-definition image as a whole, while the data capacity felt by the data transmission network and the client terminal 400 is reduced.

Step (S150): The pre-processing streaming server 300 removes the empty region noise from the image for each color element. In the image for each color element, there is a lump of pixels in which nothing exists, which is referred to as an empty region in the present specification. Referring to [FIG. 6], an empty area is widely formed in the image for each color element. When the image for each color element is divided and acquired in step S100, the empty region is stored. When image processing is performed in the resizing or transcoding process, a non-zero data value is generated in the empty area. Leaving this value as it is may cause discoloration in a later composite image. Accordingly, in step S150, color fading is prevented by deleting noise (non-zero data) generated in the empty region of the image for each color element.

Steps (S160, S170): Then, the pre-processing streaming server 300 provides the images for each color element to the client terminal 400 through a data transmission network (eg, 5G), and the client terminal 400 provides these color elements The display screen is rendered by synthesizing the star images individually after decoding.

[Fig. 5] is a diagram illustrating the concept of a color element division and transcoding process performed by the preprocessing streaming server 300 in the present invention, and [Fig. 6] is a diagram illustrating an example of a color element division and transcoding process. .

Referring to [Fig. 6], when a plenoptic image including a tortoise and a cat as subjects has been provided, the pre-processing streaming server 300 divides the plenoptic image into color element units and divides the plenoptic image into a plurality of color element images. to get [Fig. 6] shows images obtained for each of the N color elements. And looking at the example in which differential transcoding is applied by reflecting the ROI in [Fig. 6], the case where the ROI is set in the turtle part and the case where the ROI is set in the cat part are shown. In the real space, if the cat is in front and the turtle is in the back, you can use depth information to separate these objects and set the ROI.

Meanwhile, the present invention can be implemented in the form of computer-readable codes on a computer-readable non-volatile recording medium. Various types of storage devices exist as such non-volatile recording media. For example, hard disks, SSDs, CD-ROMs, NAS, magnetic tapes, web disks, cloud disks, etc. The form in which it is made and executed can also be implemented. In addition, the present invention may be implemented in the form of a computer program stored in a medium to execute a specific procedure in combination with hardware.

100: plenoptic camera
200: video storage
300: preprocessing streaming server
400: client terminal

Claims (6)

a first step in which the pre-processing streaming server 300 separates and extracts a series of image frames constituting the plenoptic image for each of a plurality of color elements to obtain an image for each color element composed of a sequence of frame sequences for each color element;
identifying, by the pre-processing streaming server 300, an empty region that is a pixel mass in which an object does not exist for each image for each color element;
a second step of extracting, by the pre-processing streaming server 300, an ROI (Region of Interest) by reflecting the viewing target focus position with respect to the plenoptic image;
a third step of applying differential transcoding so that the pre-processing streaming server 300 separates the ROI part and the non-ROI part to the image for each color element so that the ROI part is relatively high quality;
removing, by the pre-processing streaming server 300, a non-zero data value generated in the empty region in the differential transcoding process for each image for each color element;
a fourth step of providing, by the pre-processing streaming server 300, the images for each of the plurality of color elements to the client terminal 400;
a fifth step of rendering the display screen by synthesizing, by the client terminal 400 after individually decoding the images for each of the plurality of color elements;
A plenoptic image decoding method based on color component division preprocessing, including
The method according to claim 1,
The second step is
The pre-processing streaming server 300 identifying the target focus position;
extracting, by the pre-processing streaming server 300, a depth of interest (DOI) based on the focus position of the audience;
extracting, by the pre-processing streaming server 300, depth information of the plenoptic image;
extracting, by the pre-processing streaming server 300, a pixel mass having depth information corresponding to the DOI from the plenoptic image and setting it as an ROI;
Plenoptic image decoding method based on color element division preprocessing, characterized in that it comprises a.
The method according to claim 1,
The second step is
Separating, by the pre-processing streaming server 300, a foreground part and a background part based on the depth information of the plenoptic image;
The pre-processing streaming server 300 extracts the pixel mass corresponding to the foreground part and sets it as an ROI;
Plenoptic image decoding method based on color element division preprocessing, characterized in that it comprises a.
The method according to claim 1,
performed between the second step and the third step,
The pre-processing streaming server 300 identifying the display specifications through the interaction with the client terminal (400);
Resizing, by the pre-processing streaming server 300, the image for each color element by reflecting the display specification;
Color component division preprocessing-based plenoptic image decoding method, characterized in that it further comprises a.
delete A computer program stored in a medium in order to execute the plenoptic image decoding method based on the color element division preprocessing according to any one of claims 1 to 4 in combination with hardware.

Images