KR101573676B1 - Method of providing metadata-based object-oriented virtual-viewpoint broadcasting service and computer-readable recording medium for the same - Google Patents

Abstract

The present invention relates to a method for providing a customized (or specific object-centric) image service to a client terminal such as an IPTV. When a broadcast server provides object-centric image data implemented to be oriented to individual objects to the client terminal, a user can control the menu of the client terminal to receive a recorded image focusing on a desired object. When the broadcast server provides virtual viewpoint image data recording the specific object at various angles to the client terminal, the user can control the menu of the client terminal to watch a virtual viewpoint image of the desired object recorded at the various angles. The present invention enables a user to select a specific player through a menu control operation and focus on the corresponding player in a broadcast content if the user wants to focus on the corresponding player when the client terminal plays sports broadcast content by using the image data provided by the broadcast server.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a metadata-based object-based virtual point-of-view broadcasting service method and a recording medium therefor,

The present invention is a technique for providing a customized (specific object-oriented) video service to a client terminal (e.g., IPTV). More specifically, the present invention provides a technique for providing a client terminal with object-centered image data in which a broadcasting server implements individual objects, thereby allowing a user to operate a menu of a client terminal to receive a broadcast image of a desired object .

In addition, the present invention provides a client terminal with virtual viewpoint image data captured at various angles with respect to a specific object by a broadcasting server, and a user operates a menu of a client terminal to view virtual viewpoint images at various angles with respect to a desired object Technology.

2. Description of the Related Art [0002] With the recent development of multimedia technology, a plurality of multimedia contents produced from various copyright owners are supplied from a server to a user's client terminal (for example, IPTV) through various media such as the Internet, air waves, cable and satellite.

However, it is a reality that the image provided to the client terminal in the server is mostly provided in one pattern. For example, when a short-track game is broadcasted, several cameras are placed on the short-track stadium to shoot all the players in the track, but prioritizing the broadcast service is the first group leader, occasionally falling players, Is provided.

For example, while a viewer is watching a short-track game, they would like to continue to see their favorite player, but if the player does not belong to the leader group or does not fit the preference of the broadcaster, . As described above, the broadcast image of the prior art is sorted according to the priorities of the broadcasting company and has a disadvantage that it can not reflect various symbols of various viewers.

In addition, although it is desired to view the operation of a specific player from various angles, it is difficult to accommodate such a point in a video stream of a limited broadcasting service. Especially, when a user has difficulty in achieving such a purpose by menu operation of a client terminal This is the current situation.

delete

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and it is an object of the present invention to provide an object-centered image To the client terminal.

According to an embodiment of the present invention, there is provided an object-based broadcast service system based on metadata for an object-oriented image, the method comprising: generating an image based on an individual object in one event in which a plurality of objects operate; A technology for providing a broadcast service based on an individual object through metadata, comprising: a plurality of cameras providing a captured video stream; A memory unit for receiving and storing a plurality of video streams captured by a plurality of cameras in a single event in which a plurality of objects operate and receiving and storing feature data for one or more objects participating in an event; An information processing unit for searching an object-oriented image component of an individual object from a plurality of video streams, an object-centered image generating unit for generating an object-oriented image for each object by connecting the object-oriented image- And a broadcast server having a data generation unit for inserting metadata and generating a broadcast stream by mobilizing an object-oriented image, and a data transmission unit for providing a broadcast stream to a client terminal.

At this time, the data generation unit generates a virtual viewpoint image viewed from a plurality of angles with respect to an individual object based on the object-centered image, inserts metadata for image identification into the generated virtual viewpoint image, and the data transfer unit , And the generated virtual view image may be muxed together with the object center image to generate a broadcast stream and provide the generated broadcast stream to the client terminal.

The object-based broadcast service method based on metadata of an object-oriented image according to the present invention implements an image centered on an individual object in one event in which a plurality of objects operate, (A) a broadcast server receiving a plurality of video streams captured by a plurality of cameras for one event; (b) the broadcast server is provided with feature data for one or more objects participating in the event; (c) the broadcast server refers to the feature data to search for object-oriented image components for an individual object from a plurality of video streams; (d) generating an object-oriented image for an individual object by connecting the object-oriented image components detected by the broadcast server; (e) inserting metadata for image identification into the object-oriented image generated by the broadcast server.

The method may further include: (f) generating a broadcast stream by mobilizing the object-oriented image generated by the broadcast server; (g) providing a broadcast stream to a client terminal; (h) the client terminal identifying a selection input for a particular object in response to a user operation; (i) extracting an object-oriented image of a selected object by referring to metadata from a broadcast stream of the client terminal; (j) displaying and outputting the object-oriented image extracted by the client terminal.

According to another aspect of the present invention, there is provided a method of providing layout information for a plurality of cameras, the method comprising the steps of: (a) And to search for object-oriented image components for an individual object from a selected portion of the video stream.

The present invention also provides a method of generating a virtual viewpoint image between a step (d) and a step (e) in which a broadcast server views a virtual object image at a plurality of angles with respect to an individual object based on an object- And inserting meta data for image identification into the virtual viewpoint image generated by the broadcast server, wherein the virtual viewpoint image generated in step (f) is muxed together with the object centroid image to generate a broadcast stream Or the like.

According to the present invention, when a user wants to watch a particular player (object) while playing a sports broadcast of a client terminal (e.g., IPTV) through video data provided from a broadcasting server, There is an advantage that the player can mainly watch the broadcast.

In addition, according to the present invention, when a user selects a virtual view image with respect to an operation of a specific athlete by menu operation of a client terminal with respect to image data provided from a broadcasting server, There are also advantages.

1 is a block diagram showing a broadcast service system of the present invention;
Fig. 2 is an exemplary view showing a state in which a plurality of cameras are arranged in a short-track field.
[Fig. 3] is an exemplary view showing a video stream taken by an individual camera when four athletes race in the stadium as shown in [Fig. 2].
FIG. 4 is an exemplary view showing object-oriented image components for one of four players in the time domain in FIG. 3;
FIG. 5 is a diagram illustrating a process of generating an object-centered image by connecting object-oriented image components from FIG. 4;
FIG. 6 is a flowchart illustrating an object-based broadcast service process based on metadata for object-oriented images according to the present invention. FIG.
FIG. 7 is an exemplary view illustrating a state in which an image is displayed on a specific object (player) on a client terminal based on an object-centered image generated through the process of FIG. 6;
FIG. 8 is an exemplary view illustrating an object-oriented image component extracted from a plurality of video streams provided from a plurality of cameras installed to provide a virtual viewpoint; FIG.
FIG. 9 is an example of a state in which a virtual viewpoint image is generated and implemented in a client terminal; FIG.

Hereinafter, the present invention will be described in detail with reference to the drawings.

1 is a block diagram showing a video service system of the present invention. 1, the object-based broadcasting service system according to the present invention is a technique for providing a specific object-oriented image requested by client terminals 300, 310, 320, 330, and 340. In this case, A plurality of cameras 110, 120, 130 and 140 and a broadcasting server 200 installed in a broadcasting station (for example, a short-track stadium, a golf course, a soccer field). The medium connecting the broadcast server 200 and the client terminal 310 may be terrestrial, satellite, cable, internet, or the like.

When a plurality of cameras 110, 120, 130, and 140 are installed in a short track stadium, they may be installed at predetermined intervals along a track, and are preferably arranged so as to photograph all the players of all ranks along the track. Therefore, if it is desired to view an image from a client terminal 300, 310, 320, 330 or 340 around a specific player irrespective of the ranking, the object center image photographed from the individual camera can be selectively edited as shown in FIG. 7, Players will be able to watch mainly.

In addition, when a plurality of cameras 110, 120, 130, and 140 are installed in a golf course or a football stadium, it is preferable to have a configuration for shooting the player's action at various angles as shown in FIG. As described above, virtual viewpoint images of various angles can be obtained from multiple cameras and a virtual viewpoint image can be implemented based on the acquired virtual viewpoint images as shown in FIG.

The broadcast server 200 generates an object center image and a virtual view image from a plurality of video streams obtained from a plurality of cameras and transmits the object center image and the virtual view image to the client terminal. The broadcast server 200 includes a memory unit 210, an information processing unit 220, And a data transfer unit 240. [

The memory unit 210 receives and stores a plurality of video streams captured by a plurality of cameras in a single event in which a plurality of objects operate and stores feature data (e.g., name data, , Uniform design, uniform color, back number, country, age, sex, etc.). The memory unit 210 is a module for providing a space for storing these feature data and may be a RAM, a ROM, a flash memory, an MRAM (magnetic material RAM), a PRAM (phase change RAM), a FRAM (ferroelectric RAM) A Web disk, a cloud disk, and the like.

The information processing unit 220 refers to feature data stored in the memory unit 210 to search for object-oriented image components in which individual objects are captured from a plurality of video streams.

The data generation unit 230 generates an object-oriented image for an individual object by appropriately connecting the object-oriented image components detected in the above manner in time. Then, metadata for image identification is inserted into the generated object-centered image, and the object-oriented image is muxed to generate a broadcast stream.

Also, the data generation unit 230 generates virtual viewpoint images viewed from a plurality of angles with respect to the individual objects based on the object centered image, and inserts metadata for image identification into the generated virtual viewpoint images And generates a broadcast stream by middling the virtual viewpoint image.

The data transmission unit 240 provides the generated broadcast stream to the client terminal.

At this time, the client terminals 300, 310, 320, 330, and 340 receive broadcast streams for the object-centered image and the virtual viewpoint image from the broadcast server 200. Then, after identifying a selection input for a specific object in response to a user operation, an object-centered image of the selected object is extracted by referring to the metadata from the broadcast stream, and the object- do. The client terminals 300, 310, 320, 330 and 340 may be implemented in various products such as a digital TV, a digital set-top box, a smart phone, a smart box, a smart pad, and an HDMI dongle.

Meanwhile, the broadcast server 200 can provide the entire image according to the conventional technique to the client terminal in addition to the object-centered image and the virtual viewpoint image, and the client terminal can display the entire image without any special menu operation.

2 is an exemplary view illustrating a state in which a plurality of cameras are arranged in a short track stadium according to an embodiment of the present invention. Referring to FIG. 2, a plurality of cameras are installed along a track T, and the broadcast server 200 can receive and store layout information of cameras arranged along the track T in advance.

In the state that the layout information for the plurality of cameras is provided in this way, a camera close to the object is selected based on the layout information among the plurality of video streams, and an object-oriented image component for the individual object is searched do.

[Fig. 3] is an exemplary view showing a video stream taken by an individual camera when four athletes race in the stadium as shown in [Fig. 2]. As shown in Fig. 2, when cameras are sequentially arranged from "CAM01" to "CAM16 ", each camera in each time frame shoots a state in which each player (objects 1, 2, 3, 4) And obtains an object (player) center image as shown in FIG.

Here, it is assumed that the numbers "1", "2", "3" and "4" are the player's cabin number, and "t = 0" to "t = 15"

First, in the "CAM01", all the players 1, 2, 3 and 4 are shot from t = 0 to t = 4, and the three athletes before the t = 4 to t = It disappeared and only the fourth player was shot behind.

In the "CAM02" placed immediately after "CAM01", players 1, 2 and 3 are shot from t = 3 to t = 7, and player 4 is added from t = 7 to t = 8 All of the players 1, 2, 3, and 4 are taken on the screen. The fourth player who lags behind in the interval of t = 3 to t = 7 speeds up and shows the situation that four players are competing in the interval of t = 7 to t = 8.

It can be seen that the players 1, 2, and 3 are simultaneously shot at "CAM02" and "CAM03" at t = 7 time zone. Since the objects (the players) are photographed in the same time zone between the adjacent cameras, the broadcasting server 200 must select which camera to receive the object-centered image component when the object-centered image component is provided in real time.

As described above, the broadcast server 200 searches for individual objects based on feature data for a plurality of broadcast streams generated by a plurality of cameras, and thereby derives object-oriented image components for individual objects. In addition, object-centered images of individual objects can be obtained by classifying and handling these object-oriented image components by object. At this time, if the broadcasting server 200 is provided with the layout information of each camera in advance and performs an object searching process based on the proximity of the object from the object at the previous time based on the broadcasting stream of the adjacent camera, This makes it relatively easy to ensure real-time.

FIG. 4 is an exemplary view illustrating an object-oriented image component for the first player among the four players in the embodiment of FIG. 3 according to a time zone, and FIG. 5 illustrates the object- And generating an object-oriented image for the first player by connecting the object-centered image.

3, an object-oriented image component is displayed for each time zone as shown in FIG. 4, for example.

Then, as shown in FIG. 5, if an image in which the first player (object) is found, that is, the object-centered image component for the first player is successively selected by time zone, the object-centered image for the first player is generated .

In this case, when both of "CAM01" and "CAM02" are found in the broadcast stream at t = 3 to t = 4, a predetermined evaluation evaluation scale is applied to select one of the object- And techniques for presenting both in a multi-angle screen.

Then, the object-centered image of the player 2, player 3, and player 4 can be generated in this pattern.

In this way, the broadcasting server 200 inserts the metadata for image identification for each object-centered image while generating the object-oriented image for the individual object. The generated object-centered images are muxed to generate a broadcast stream and transmitted to the client terminal through a delivery medium.

FIG. 6 is a flowchart illustrating an object-based broadcasting service process based on metadata for object-oriented images according to the present invention. Referring to FIG. 6, a metadata-based object-based broadcast service process for object-oriented images according to the present invention will be described in detail.

Step S110: A plurality of cameras installed in a predetermined place (e.g., a short-track stadium, a golf course, a football stadium) in which a plurality of objects operate, , A soccer player), and acquires individual images as shown in FIG.

Then, the broadcast server 200 transmits a plurality of video streams captured by a plurality of cameras to one event and feature data (e.g., name, height, uniform design, uniform color, Country, age, sex, etc.).

Step S120: Next, the broadcast server 200 refers to the feature data and searches for object-oriented image components for individual objects from a plurality of video streams as shown in FIG.

Step S130: The broadcasting server 200 creates an object-oriented image for the individual object by connecting the searched object-oriented image components as shown in [Fig. 5]. Then, the metadata for the image identification is inserted into the generated object-oriented image. On the other hand, the broadcasting server 200 generates a virtual viewpoint image viewed from a plurality of angles with respect to an individual object based on the object-centered image generated in step S130, and identifies the corresponding image to the generated virtual viewpoint image And inserts the metadata for.

Step S140: The broadcasting server 200 generates a broadcasting stream by middling the generated object-oriented image, and provides it to the client terminal.

At this time, the broadcast server 200 can generate a broadcast stream by mixing the object-centered image and the virtual viewpoint image generated in step S140 together.

Steps S150, S160, and S170: The client terminal identifies a selection input for a specific object in response to a user operation, and then extracts an object-centered image for the selected object by referring to the metadata from the broadcast stream. Then, the extracted object-centered image is displayed and output.

On the other hand, the broadcast server 200 receives layout information on a plurality of cameras, and extracts, based on the layout information among a plurality of video streams provided from a plurality of cameras, object center It is preferable to be configured to search for an image component.

2, since the position of the camera disposed on the track T is known from the layout information, after receiving the video stream from "CAM03 "," CAM04 "or" CAM05 " It is possible to control to search for object-oriented image components.

FIG. 7 is an exemplary view illustrating a state in which an image (object centered image) is displayed on a client terminal based on a specific object (player) based on the object center image generated through the process of FIG.

[7] Referring to FIG. 7, when the broadcast server 200 generates an object-oriented image by connecting object-oriented image components as shown in FIG. 5 and transmits the object-oriented image to a client terminal in a broadcast stream, And displays the object-centered image of the selected object. At this time, the process of dividing the object-oriented image of the specific object selected by the user from the plurality of object-oriented images included in the broadcast stream is performed by the client referring to the metadata inserted by the broadcast server 200.

Thus, the client user can view the video on the basis of a specific player (object) desired during playing a sports broadcast on a client terminal (e.g., IPTV).

FIG. 8 is an exemplary view illustrating an object-oriented image component extracted from a plurality of video streams provided from a plurality of cameras installed to provide a virtual viewpoint, FIG. FIG. 8 is an exemplary diagram of a state implemented in a client terminal; FIG.

Referring to FIG. 8, a plurality of cameras can simultaneously photograph a swinging motion of a "Tiger Woods" player on a golf course at various angles. The broadcast server 200 generates a virtual viewpoint image of the corresponding player through the video images captured by the plurality of cameras, and transmits the virtual viewpoint image to the client terminal in a broadcast stream. The client terminal refers to the metadata of the virtual viewpoint image from the broadcast stream, The virtual view image of the object can be extracted and displayed as shown in FIG.

FIG. 8 shows that the swing motion of the "Tiger Woods" player on the golf course was simultaneously photographed by a plurality of cameras at various angles, FIG. 9 shows a ball touch operation of a specific player in a soccer field, It is display output by video. In other words, the virtual viewpoint image for the "Tiger Woods" player can be displayed and output on the basis of FIG. 8 (FIG. 9) . ≪ / RTI >

The present invention can also be embodied in the form of computer readable code on a computer readable recording medium. At this time, the computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored.

Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage, and the like, and may be implemented in the form of a carrier wave . The computer-readable recording medium can also be stored and executed by a computer-readable code in a distributed manner on a networked computer system. And functional programs, codes, and code segments for implementing the present invention can be easily deduced by programmers skilled in the art to which the present invention belongs.

As described above, the embodiments of the present invention have been disclosed in the present specification and drawings, and although specific terms have been used, they have been used only in a general sense to easily describe the technical contents of the present invention and to facilitate understanding of the invention. And is not intended to limit the scope of the invention. It will be apparent to those skilled in the art that other modifications based on the technical idea of the present invention are possible in addition to the embodiments disclosed herein.

110 to 140: camera
200: Broadcast server
210:
220: Information processor
230:
240: Data transfer unit
300 to 340: client terminal

Claims (7)

delete delete There is provided a method of providing an image based on an individual object in a single event in which a plurality of objects operate and providing an individual object-based virtual point-of-view broadcasting service through metadata about the object-
Receiving a plurality of video streams each of which is shot from a plurality of cameras arranged to divide and photograph the entire space in which the event occurs according to a predetermined layout;
Receiving the layout information for the plurality of cameras from the broadcasting server;
The broadcast server receiving feature data on a feature of a visual for one or more objects participating in the event;
Acquiring a series of object-oriented image components for each individual object by time in the broadcast server by searching individual objects based on the feature data for the plurality of image streams;
Performing a search for an individual object based on a broadcast stream of a nearby camera from a camera in which an individual object is found at a previous time based on the layout information;
Generating an object-oriented image for the individual object by connecting the set of object-oriented image components obtained for the individual object with the operation time according to the broadcast server;
Inserting first metadata for identifying the object centered image by the broadcast server;
Generating a virtual viewpoint image in which the broadcast server views virtual views of the individual objects at a plurality of angles based on the object centered images of the individual objects;
Inserting second metadata for identifying the virtual viewpoint image by the broadcast server;
Generating a broadcast stream by broadcasting the object-centered image and the virtual viewpoint image;
Based virtual point-of-view broadcast service method.
The method of claim 3,
Providing the broadcast stream to a client terminal;
The client terminal identifying a selection input for a particular object in response to a user operation;
The client terminal acquiring an object-oriented image of the selected object from the broadcast stream by referring to the first metadata included in the broadcast stream;
Displaying the obtained object-oriented image on the client terminal;
The client terminal identifying a selection input for a particular time point in response to a user operation;
Obtaining a virtual viewpoint image of the selected object and the selected viewpoint from the broadcast stream by referring to the second metadata included in the broadcast stream;
Displaying the obtained object-oriented image on the client terminal;
Based virtual-point broadcasting service method.
delete delete A computer-readable recording medium having recorded thereon a program for causing a computer to execute a metadata-based object-based virtual point-of-view broadcasting service method according to claim 3 or claim 4.

Images