KR20170143339A - Apparatus and Method for Providing Lightfield Motion Image - Google Patents

Abstract

The present invention relates to a server for providing a light field moving image. The server includes a light field image sequence acquiring part for acquiring a plurality of light field image sequences constituting a moving image, a refocused image sequence generation part for generating a light field refocused image sequence focused on each depth position from the light field image sequence acquired by the light field image sequence acquiring part; and a moving image segment generation part for performing moving image encoding with a moving image streaming file using a refocused image sequence for each focus generated in the refocused image sequence. It is possible to generate a still image obtained based on a light field in a moving image format.

Description

TECHNICAL FIELD [0001] The present invention relates to a light field motion image providing apparatus and method,

Field of the Invention [0002] The present invention relates to a light field technique, and more particularly, to a refocusable light field video streaming apparatus and method.

The light field-based image service is an image technology that acquires the field information of the light in the space at once and reproduces it again. It can reproduce the field of light in the same form as the real physical space, It is the next generation image acquisition method which can provide the same experience.

As a representative service using the light field-based image acquisition method, there is a post refocusing service, which can freely re-adjust the focus to a desired position after photographing Service.

On the other hand, most of the light field-based services are mostly commercial products for still images, and video services are still being studied.

The present invention provides a light field moving picture providing apparatus and method for generating a still picture based on a light field on a moving picture basis.

The present invention provides an apparatus and method for providing a light field moving picture capable of streaming a video sequence photographed based on a light field to a remote site in a moving image form.

The present invention provides a light field moving picture providing apparatus and method in a terminal that receives streaming, in which a user can refocus in real time at any time within a moving picture at a desired time at any time.

The present invention provides a light field moving image providing server, comprising: a light field image sequence acquiring unit for acquiring a plurality of light field image sequences constituting a moving image; and a light field image acquiring unit for acquiring, from the light field image sequence acquired by the light field image sequence acquiring unit, And a moving image segment generating unit for performing moving image encoding using a focus refocusing image sequence generated in the refocusing image sequence, as a moving image streaming file, based on the refocusing image sequence generated by the refocusing image sequence .

According to one embodiment, the light field image sequence acquiring unit acquires a light field image using a light field camera or physically combining a plurality of cameras.

According to one embodiment, the moving picture segment generation unit encodes a streaming file using a video codec including H.264 and MPEG.

According to one embodiment, the moving image segment generation unit time-divides a focus-based refocused image sequence by a predetermined time unit, and generates each of the divided images as a moving image streaming file.

According to an embodiment, the mobile communication terminal further includes a video segment transmission unit for streaming the video streaming file to a network-connected terminal.

According to one embodiment, the moving picture transmitter streams a file corresponding to a requested focal point position in response to receiving a signal requesting a focal position change request from a terminal in the course of streaming.

According to an embodiment of the present invention, the moving picture segment storing unit may further include a moving picture segment storing unit for storing the moving picture file by the moving picture segment generating unit, and the moving picture transmitting unit may receive the moving picture segment storing unit The moving image file corresponding to the changed focal position is streamed.

According to an embodiment, the moving picture transmitter selectively uses at least one of a RTP-based media streaming method, a broadcasting standard-based streaming method using a TS file, and a progressive download-type streaming method.

A moving picture segment receiving unit for receiving a moving picture segment from a light field moving picture providing server; a moving picture reproducing unit for decoding the received moving picture segment and reproducing the received moving picture segment as a moving picture on a screen; And a refocus request unit for requesting a focus position change request from the light field moving image providing server.

According to an embodiment, the moving picture segment receiving unit receives a moving picture streaming file generated by time-division-based images of a focus-by-focus-based image sequence in units of a predetermined time.

The present invention relates to a method of providing a light field moving picture, comprising the steps of: obtaining a plurality of light field image sequences constituting a moving picture; obtaining a light field image sequence obtained by the light field image sequence obtaining section Generating a focused image sequence, and performing a moving picture encoding of the moving image streaming file using the refocused image sequence generated by the refocused image sequence.

According to one embodiment, the acquiring step uses a light field camera or physically combines a plurality of cameras to acquire a light field image.

According to one embodiment, the generating step is a step of encoding a streaming file using a video codec including H.264 and MPEG.

According to an exemplary embodiment, the generating step may include time-dividing a focus-based refocused image sequence by a predetermined time unit, and generating each divided image as a moving image streaming file.

According to one embodiment, the method further comprises the step of streaming the moving picture streaming file to a network-connected terminal.

According to one embodiment, the streaming step further includes a step of streaming a file corresponding to a requested focal position as a signal requesting a focal position change from the terminal is received during streaming.

According to an embodiment of the present invention, the method further comprises storing the moving image file by the moving image segment generating unit.

According to an exemplary embodiment, the streaming may selectively use at least one of an RTP-based media streaming method, a broadcast standard-based streaming method using a TS file, and a streaming method of a progressive download type.

The present invention provides a method of providing a light field moving picture, comprising the steps of: receiving a moving picture segment from a light field moving picture providing server; decoding the received moving picture segment and reproducing it as a moving picture on a screen; And requesting a focus position change from the light field moving image providing server.

According to one embodiment, the receiving step receives a moving picture streaming file generated by time-division-based images of the focus-based refocused video sequence in units of a predetermined time.

The present invention is fully compatible with the existing streaming system and can be used again by adding only the signaling part for informing the refocusing point, Is very easy to expand.

1 is a configuration diagram of a light field moving picture providing system according to an embodiment of the present invention.
2 is an example of a light field refocus image according to the present invention.
3 is an example of a time-division refocus video segment according to the present invention.
4 is a flowchart illustrating a method of providing a light field moving picture according to an embodiment of the present invention.
FIG. 5 is a flowchart for illustrating a step of generating a light field moving picture according to an embodiment of the present invention.
6 is a flowchart illustrating a step of streaming a light field moving picture according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

The terms used throughout the specification are defined in consideration of the functions in the embodiments of the present invention and can be sufficiently modified according to the intentions and customs of the user or the operator. It should be based on the contents of.

FIG. 1 is a configuration diagram of a light field moving image providing system according to an embodiment of the present invention, FIG. 2 is an example of a light field refocus image according to the present invention, FIG. 3 is a time- .

Referring to FIG. 1, a light field moving picture providing system includes a light field moving picture providing server (hereinafter referred to as a 'server') 100 for generating a light field moving picture as a refocusable moving picture and remotely streaming the same, And a terminal 200 capable of real-time refocusing according to a user's operation while reproducing a light field moving picture received from the server 100 through the terminal 100. [

The server 100 includes a light field image sequence acquisition unit 110, a refocused image sequence generation unit 120, a moving image segment generation unit 130, and a moving image segment transmission unit 140. In addition, it further includes a moving picture segment storage unit 150.

The light field image sequence acquiring unit 110 acquires a plurality of light field images photographed from an external physics system and obtains an image sequence capable of composing a moving image. This can be achieved by using a light field camera or by physically combining several cameras to obtain a light field image.

영상으로부터 화면으로부터 순차적으로 멀어지는 방향의 깊이 위치들에 초점이 맞추어진

,

, ...,

영상들로 구성되어 있다. The refocused image sequence generation unit 120 generates a light field refocused image sequence focused on a specific depth position within the screen from the light field image sequence obtained by the image sequence acquisition unit 110. Referring to FIG. 2, an example in which a refocused image is generated from an image having n total focal positions is shown.

Focused on depth positions in a direction away from the screen sequentially from the screen

,

, ...,

It consists of images.

The moving image segment generation unit 130 performs moving image encoding using a focus-based refocused image sequence generated in the light field refocused image sequence 120 as a moving image streaming file. At this time, a file for streaming is encoded using a video codec including H.264 and MPEG.

~

)별로 생성된 라이트필드 재초점 영상이 특정 시간 길이로 분할되어 생성된다. 예컨대, 도 3을 참조하면, 각각의 깊이 위치(

~

)별로 생성된 라이트필드 재초점 영상은 Ts의 시간 구간 단위로 분할되어 동영상 파일이 생성된다. 이때, i번째 시간 구간에서 생성된 n번째 깊이 위치의 동영상 파일을

로 표시한다. 또한, 각각의 시간 구간 단위별로 동영상 파일은 초점 위치의 개수만큼인 n 개가 생성되는데, 예컨대, i번째 시간 구간(

)의 경우,

에서

까지 n 개의 동영상 파일이 생성된다. In addition, the generated video streaming file includes a depth position (

~

) Is generated by dividing the light field refocused image into a specific time length. For example, referring to FIG. 3,

~

) Is divided into Ts time units to generate a moving image file. At this time, the video file of the n-th depth position generated in the i-th time interval

. Also, n pieces of moving picture files are generated for each time interval unit by the number of focus positions. For example, the i th time period (

)In the case of,

in

N video files are generated.

의 구간에서

에 초점이 맞추어진 동영상을 스트리밍하고 있었다면,

파일을 송신하고 있는 도중이다. 이때, 단말(200)에서 초점의 위치를

으로 재조정 요청하는 신호가 수신되면, 동영상 세그먼트 송신부(140)는 이를 저장하고 있다가 다음

구간이 도래한 경우,

의 초점 영역에 해당하는

파일을 전송한다. The video segment transmission unit 140 streams the payload video to the terminal 200 capable of being connected to the network through the streaming technology. Also, according to the present invention, as the refocus request signal is received from the terminal 200 in the course of streaming, the user resumes the streaming to the desired focal position and continues the streaming. That is, it starts to stream the file corresponding to the adjusted focus position at the time when the next time interval area arrives, so that the focus corresponding to the request of the user can be readjusted. For example, referring to FIG. 3,

In the interval of

If you were streaming videos focused on,

It is in the middle of sending a file. At this time, the position of the focal point in the terminal 200

The moving picture segment transmission unit 140 stores the signal,

When the section comes,

Corresponding to the focus region of

Transfer the file.

The video segment storage unit 150 stores a video file generated by the video segment generation unit 130 for streaming. Accordingly, as the signal requesting the focal position change is received from the terminal 200, the moving image segment transmission unit 140 streams the moving image file corresponding to the focal point requested to be changed from the moving image segment storage unit 150. That is, the present invention provides a streaming service in which re-focusing is performed based on file transfer and file transfer.

In addition, it is configured to store streaming data in the video segment storage unit 150 in the form of a file once without streaming in real time, and then streaming the streaming data. Thus, various streaming methods can be supported. That is, since it is a file-based operation structure, various kinds of streaming techniques can be used. Here, the streaming technique includes a conventional media streaming system based on RTP, a streaming system based on a broadcasting standard using a TS file, a streaming system of a progressive download type, and the like.

Meanwhile, the terminal 200 includes a moving picture segment receiving unit 210, a moving picture reproducing unit 220, and a refocusing request unit 230.

The video segment receiving unit 210 receives a video segment from the server 100 via the network. The moving picture playback unit 210 decodes the received moving picture segment and plays back the moving picture segment on the screen. The re-focus requesting unit 230 transmits a signal for requesting a focus position change to the server 100 as the focus position change request is made from the user. At this time, the changed focal position information is transmitted.

4 is a flowchart illustrating a light field moving picture streaming method according to an embodiment of the present invention.

Referring to FIG. 4, the server 100 includes a step of generating a light field moving picture (S410, see FIG. 5) and a step S420 of streaming the generated light field moving picture to a remote terminal 200 through a network 6).

FIG. 5 is a flowchart for illustrating a step of generating a light field moving picture according to an embodiment of the present invention.

Referring to FIG. 5, the server 100 acquires a plurality of light field images photographed from an external physical system and obtains an image sequence capable of composing a moving image (S510). This can be achieved by using a light field camera or by physically combining several cameras to obtain a light field image. Then, the server 100 determines the number n of the focus positions in advance (S520), and then the server 100 determines a predetermined time during which the focus is maintained during real-time video streaming (S530).

) 동안의 영상 시퀀스를 생성한다. Then, the server 100 generates a light field refocused image sequence focused on specific depth positions within the screen from the obtained light field image sequence (S540). That is, as shown in FIG. 2, a predetermined time (for example,

) ≪ / RTI >

~

)별로 생성된 라이트필드 재초점 영상이 특정 시간 길이로 분할되어 생성된다. 즉, 각각의 깊이 위치(

~

)별로 생성된 라이트필드 재초점 영상은 Ts의 시간 구간 단위로 분할되어 동영상 파일이 생성된다. 또한, H.264 및 MPEG을 포함하는 비디오 코덱을 이용하여 스트리밍용 파일이 부호화한다. 서버(100)는 부호화된 동영상 파일을 저장한다(S560). 그런 후, 서버(100)는 이번 시간 구간(

)이 마지막 시간 구간인지를 판단(S570)하여, 마지막이 아닐 경우 S540 단계로 진행한다. Next, the server 100 performs a moving picture encoding process on the moving image streaming file using the generated refocusing image sequence for each focus (S550). That is,

~

) Is generated by dividing the light field refocused image into a specific time length. That is, each depth position (

~

) Is divided into Ts time units to generate a moving image file. In addition, a streaming file is encoded using a video codec including H.264 and MPEG. The server 100 stores the encoded moving image file (S560). Then, the server 100 determines whether or not this time interval (

(S570). If it is not the last time, the process proceeds to step S540.

6 is a flowchart illustrating a step of streaming a light field moving picture according to an embodiment of the present invention.

6, the terminal 200 requests streaming of the light field video to the server 100 (S610), and the server 100 streams the video data to the remote terminal 200 capable of being connected to the network through the streaming technology (S620).

The terminal 200 decodes the video segment received from the server 100 and reproduces the video segment as a moving picture on the screen (S630).

Thereafter, the terminal 200 transmits a signal requesting the focal point change to the server 100 (S650) according to the focal position change request from the user (S640). At this time, the changed focal position information is transmitted.

In response to receiving the focus position change request signal from the terminal 200 during the streaming, the server 100 changes the position of the focal point change request signal to the focal point position and continues the streaming operation in operation S660. That is, it starts to stream the file corresponding to the changed focus position at the time when the next time zone area arrives, thereby changing the focus position corresponding to the request of the user. At this time, the moving image file corresponding to the requested focal position is extracted from the moving image segment storage unit 150 and streamed. That is, since it is a file-based operation structure, various kinds of streaming techniques can be used. Here, the streaming technique includes a conventional media streaming system based on RTP, a streaming system based on a broadcasting standard using a TS file, a streaming system of a progressive download type, and the like.

Then, the terminal 200 decodes the focal point-changed moving image segment received from the server 100 and reproduces the moving image segment on the screen (S670).

Claims (20)

A light field image sequence acquiring unit for acquiring a plurality of light field image sequences constituting a moving picture,
A re-focus image sequence generator for generating a light field refocused image sequence focused by depth positions from the light field image sequence acquired by the light field image sequence acquiring unit;
And a moving image segment generation unit that performs moving image encoding using a focus refocusing image sequence generated in the refocusing image sequence as a moving image streaming file.
2. The apparatus of claim 1, wherein the light field image sequence obtaining unit
Wherein a light field image is obtained by using a light field camera or physically combining a plurality of cameras.
The apparatus of claim 1, wherein the moving image segment generation unit
Wherein a file for streaming is encoded using a video codec including H.264 and MPEG.
The apparatus of claim 1, wherein the moving image segment generation unit
Focus image sequence is time-divided by a predetermined time unit, and each divided image is generated as a moving image streaming file.
The method according to claim 1,
And a video segment transmission unit for streaming the video streaming file to a network-connected terminal.
[6] The apparatus of claim 5,
And a file corresponding to the requested focal point is streamed as a signal requesting the focal position change request is received from the terminal in the course of streaming.
The method according to claim 6,
And a moving picture segment storage unit for storing the moving picture file by the moving picture segment generation unit,
The moving picture transmitter
Wherein the controller is configured to stream the moving image file corresponding to the requested focus position from the moving image segment storage unit upon receiving a signal requesting the focus position change request from the terminal.
8. The apparatus of claim 7, wherein the moving picture transmitter
Wherein at least one of a media streaming method based on RTP, a broadcasting standard based streaming method using a TS file, and a streaming method using a progressive download type is selectively used.
A moving picture segment receiving unit for receiving a moving picture segment from the light field moving picture providing server;
A moving picture reproducing unit for decoding the received moving picture segment and playing the decoded moving picture segment as a moving picture on a screen,
And a focus refusal request unit for requesting a focus position change request from the light field moving image providing server in response to a focus position change request from a user.
The apparatus of claim 9, wherein the moving image segment receiving unit
Wherein the mobile terminal receives the moving picture streaming file generated by time-division-based images of the focus-based refocused video sequence on a predetermined time unit basis.
Obtaining a plurality of light field image sequences constituting a moving picture;
Generating a light field refocused image sequence focused on the depth position from the light field image sequence acquired by the light field image sequence acquiring unit;
And performing a motion picture encoding using the focus-based refocused image sequence generated in the refocused image sequence as a moving image streaming file.
12. The method of claim 11, wherein obtaining
Wherein a light field camera is used or a plurality of cameras are physically combined to acquire a light field image.
12. The method of claim 11, wherein the generating comprises:
Wherein a file for streaming is encoded using a video codec including H.264 and MPEG.
12. The method of claim 11, wherein the generating comprises:
Wherein each of the plurality of divided images is generated as a moving image streaming file by time-dividing the refocused image sequence by a predetermined time unit.
12. The method of claim 11,
And streaming the moving picture streaming file to a network-connected terminal.
16. The method of claim 15, wherein the streaming comprises:
Further comprising the step of streaming a file corresponding to the requested focal point as the signal requesting the focal position change is received from the terminal in the middle of the streaming.
17. The method of claim 16,
And storing the moving picture file by the moving picture segment generating unit.
17. The method of claim 16, wherein streaming comprises:
Wherein at least one of a RTP-based media streaming method, a broadcasting standard-based streaming method using a TS file, and a streaming method of a progressive download type is selectively used.
Receiving a video segment from a light field moving image providing server;
Decoding the received moving picture segment and reproducing it as a moving picture on a screen,
And requesting a focal position change from the light field moving image providing server according to a focal position change request from a user.
20. The method of claim 19, wherein the receiving comprises:
And receiving a video streaming file generated by time-division-based images of a focus-based refocused video sequence on a predetermined time unit basis.

Images