JP2013239803A - Multi-viewpoint image separation method, multi-viewpoint image separation device and multi-viewpoint image separation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To separate a desired image from a multi-viewpoint image with PCR being left so that synchronization of a clock with a content distribution server can be established on a receiver side.SOLUTION: A multi-viewpoint image separation device 10 includes: acquisition means 14 which acquires a first packet identifier of a packet including a separation object image in a multi-viewpoint image, and a second packet identifier of a packet including clock information; comparison means 14 which compares a processing object packet identifier of a processing object packet of the multi-viewpoint image with the first packet identifier and the second packet identifier; and separation means 14 which separates the processing object packet when the processing object packet identifier matches the first packet identifier and does not match the second packet identifier, and separates the separation object image from the processing object packet when the processing object packet identifier matches the first packet identifier and the second packet identifier.

Description

  The present invention relates to a multi-view video separation method, a multi-view video separation device, and a multi-view video separation program for separating a video of a desired viewpoint from a multi-view video.

  In recent years, in video viewing, not only so-called terrestrial digital broadcasting transmitted by broadcast waves (hereinafter abbreviated as “RF”) but also video distribution services (hereinafter abbreviated as “IPTV”) transmitted by an IP network. It has become widespread. As an IPTV transmission method, the MPEG2-TS method (see Non-Patent Document 1) is mainly adopted, and handling of section data and video data is regulated.

  In addition, H.264 AVC, which can compress only one viewpoint video, has been adopted as the IPTV video compression method so far, but it is called H.264 MVC that can compress multi-viewpoint video with the progress of video compression technology. A compression method (see Non-Patent Document 2) has been established.

Overview of MPEG2-TS, [online], Internet <URL: http://www.mpeg.co.jp/libraries/video_it/video_03.html> "Keywords you should know, multi-view video coding", Journal of the Institute of Image Information and Television Engineers Vol.62, No.3, pp.343-345, 2008

  In contrast to H.264 AVC, which is a video compression method used in the past, the concept of multi-viewpoint is introduced in H.264 MVC. On the other hand, the capacity of content to be distributed (here, the generic name of files including video, audio, and subtitles necessary for viewing video) is increased by the number of viewpoints in H.264MVC compared to H.264 AVC To do. Based on this assumption, the following services are assumed.

  (1) Video ES (Elementary Stream) and audio / caption ES shot from a plurality of viewpoints are arranged on the distribution server as one TS (Transport Stream) file.

  (2) The viewer selects the video of the viewpoint that he / she wishes to view and requests it from the distribution server.

  (3) The distribution server multiplexes and distributes only necessary viewpoint videos in response to requests from viewers for the purpose of reducing the NW bandwidth to be used.

  It is assumed that the TS (Program Clock Reference), which is information for synchronizing the clock of the distribution server and the receiver, is already embedded in the TS file placed on the distribution server. If only the required viewpoint video is extracted from the source file and multiplexed as in (), if the video packet that was not extracted contains PCR, the required PCR is missing and a discontinuous clock is generated. End up.

  As a result, the receiver cannot synchronize the clock with the distribution server, and the video cannot be reproduced smoothly, and an unpleasant video is provided to the viewer.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to allow a receiver to synchronize a clock with a distribution server when separating a desired video from a multi-view video. Thus, it is to provide a multi-view video separation method, a multi-view video separation device, and a multi-view video separation program for separating a desired video by leaving PCR.

  In order to achieve the above object, the present invention provides a multi-view video separation method for separating a video of a desired viewpoint from a multi-view video, and includes a first packet including a video to be separated in the multi-view video. An acquisition step of acquiring the packet identifier of the packet and the second packet identifier of the packet including the clock information; reading each packet of the multi-view video; processing target packet identifier of the read processing target packet; A comparing step for comparing the packet identifier and the second packet identifier; and a first packet separating the processing target packet when the processing target packet identifier matches the first packet identifier and does not match the second packet identifier. The separation step and the processing target packet identifier matches the first packet identifier and the second packet identifier, If serial processing target packet to the clock information and the separation subject image exists, performing a second separation step of separating the image of the separation subject from the processed packet.

  The present invention is a multi-view video separation device that separates a video of a desired viewpoint from a multi-view video, the first packet identifier of a packet including a video to be separated in the multi-view video, and clock information Acquisition means for acquiring a second packet identifier of a packet including the packet, a packet to be processed for reading each packet of the multi-view video, a packet identifier to be processed, a first packet identifier and a second packet The comparison means for comparing the identifier and the processing target packet identifier when the processing target packet identifier matches the first packet identifier and does not match the second packet identifier, the processing target packet is separated and the processing target packet identifier is The packet identifier and the second packet identifier, and the processing target packet includes the clock information and If serial separation target image is present, having a separating means for separating the image of the separation subject from the processed packet.

  The present invention is a multi-view video separation program for causing a computer to execute the multi-view video separation method.

  According to the present invention, when separating a desired video from a multi-view video, the multi-view that separates the desired video by leaving the PCR so that the receiver can synchronize the clock with the distribution server. A video separation method, a multi-view video separation device, and a multi-view video separation program can be provided.

1 is a diagram illustrating an overall configuration of a multi-view video distribution system according to an embodiment of the present invention. It is explanatory drawing for demonstrating the component of a multiview video content, and remultiplexing. It is a figure which shows the pattern and structure of TS packet. It is a flowchart which shows operation | movement of a demultiplexing part.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a configuration diagram showing the overall configuration of the multi-view video distribution system of the present embodiment. The illustrated communication system includes a content distribution server 10 and a receiving terminal 30 that can be connected to the content distribution server 10 via a network 20 (for example, an IP network). In FIG. 1, one receiving terminal 30 is shown, but a plurality of receiving terminals 30 may be provided.

  The content distribution server 10 (multi-view video separation device) is a server that distributes multi-view video content, and includes a multi-view video content storage unit 11, a distribution management unit 12, a content acquisition unit 13, and demultiplexing and remultiplexing. Unit 14, content distribution unit 15, work buffer memory 16, distribution control unit 17, and communication unit 18.

  The multi-view video content storage unit 11 is a storage device such as a server disk, and stores at least one multi-view video content arranged in the content distribution server 10. The distribution management unit 12 controls and manages the content acquisition unit 13, the demultiplexing unit 14, the content distribution unit 15, and the distribution control unit 17. The content acquisition unit 13 reads the multi-view video content managed by the distribution management unit 12 from the multi-view video content storage unit 11 and writes it in the work buffer memory 16 in response to a request from the receiving terminal 30.

  The demultiplexing / remultiplexing unit 14 (acquiring unit, comparing unit, demultiplexing unit, and generating unit) separates and remultiplexes video unnecessary for the viewer of the receiving terminal 30 from the multi-view video content read by the content acquisition unit 13. Distribution contents are generated and output to the work buffer memory 16. Note that the distribution content of this embodiment is in the TTS format.

  The content distribution unit 15 converts the distribution content (TTS file) output by the demultiplexing / remultiplexing unit 14 into an IP packet and distributes it to the receiving terminal 30 via the communication unit 18 and the network 20. The work buffer memory 16 is a work memory area in which the content acquisition unit 13, the demultiplexing / multiplexing unit 14, and the content distribution unit 15 exchange files. The distribution control unit 17 outputs a content distribution request from the receiving terminal 30 to the distribution management unit 12. The communication unit 18 performs communication processing with the receiving terminal 30 via the network.

  The receiving terminal 30 is a device that receives and displays the distribution content distributed from the content distribution server 1.

  The content distribution server 10 and the receiving terminal 30 described above can use, for example, a general-purpose computer system including a CPU, a memory, an external storage device such as an HDD, an input device, and an output device. . In this computer system, each function of each device is realized by the CPU executing a predetermined program loaded on the memory. For example, the functions of the content distribution server 10 and the receiving terminal 30 are the CPU of the content distribution server 10 in the case of the program for the content distribution server 10 and the CPU of the receiving terminal 30 in the case of the program for the reception terminal 30. It is realized by executing each. The program for the content distribution server 10 and the program for the receiving terminal 30 can be stored in a computer-readable recording medium such as a hard disk, a flexible disk, a CD-ROM, an MO, or a DVD-ROM via a network. It can also be delivered.

  FIG. 2 is an explanatory diagram for explaining multi-view video content and a re-multiplexed image. In FIG. 2, the components 41 to 44 of the multi-view video content 40 stored in the multi-view video content storage unit 11 of the content distribution server 10 and the operation of the viewer (user) of the receiving terminal 30 can change the desired viewpoint. The components 51 to 53 of the distribution content 50, which is video content in which only video is selected and re-multiplexed, are shown.

  Here, regarding the multi-view video 43 and 44 of the multi-view video content 40, the two viewpoints with the smallest number of viewpoints are used. However, the number of viewpoints of the multi-view video content 40 is not limited to two, but two or more viewpoints. The number of viewpoints.

  The multi-view video content 40 is a video source arranged in the content distribution server 10 and is a TS (Transport Stream) file including video ES (Elementary Stream) shot from a plurality of viewpoints. The illustrated multi-view video content 40 includes a caption ES41, an audio ES42, a video ES43 at a first viewpoint, and a video ES44 at a second viewpoint.

  At the request of the viewer of the receiving terminal 30, the demultiplexing / remultiplexing unit 14 of the content distribution server 10 separates only the predetermined video ES, remultiplexes the remaining components of the multi-view video content, and generates distribution content. . In the illustrated example, it is assumed that the viewer needs only the video ES43 of the first viewpoint and separates (discards / deletes) the video ES44 of the unnecessary second viewpoint.

  The distribution content 50 is content that has been separated / remultiplexed by the separation / remultiplexing unit 14 and is a TS file in which only the video ES53 of the first viewpoint designated by the viewer is remultiplexed. This distribution content 50 (TS file) is distributed to the receiving terminal 30. The distribution content 50 includes a caption ES51, an audio ES52, and a first viewpoint video ES53 requested by the user. Note that the caption ES51, the audio ES52, and the first viewpoint video ES53 are the same as the caption ES41, the audio ES42, and the first viewpoint video ES53 of the multi-view video content 40, respectively.

  FIG. 3 is an explanatory diagram for explaining the pattern and configuration of a TS packet.

  Here, before describing a sequence for preventing the lack of PCR (Program Clock Reference), which is clock information for synchronizing the clocks of the content distribution server 10 and the receiving terminal 30, the PCR is originally arranged in the TS packet. Touch on

  The TS packet is a TS file transmission packet. The components of the TS packet are roughly divided into three parts: a TS header, an extension adaptation field, and a payload including a video ES, caption ES, audio ES, and the like. Three patterns of TS packets A to C are conceivable.

  Here, the PID (Packet ID, an identifier for identifying the packet type) is included in the TS header. PCR is included in the adaptation field.

  Pattern A is a TS packet composed of a TS header and a payload (part of PES). PES (Packetized Elementary Stream) is a packetized video ES or the like.

  Pattern B is a TS packet including a TS header, an adaptation field, and a payload.

  The pattern C is a TS packet composed of a TS header and an adaptation field.

  Next, the operation of this embodiment will be described.

  First, the receiving terminal 30 receives an instruction from the viewer and transmits a multi-view video content distribution request to the content distribution server 10. This distribution request includes identification information (content ID, etc.) of desired multi-view video content, identification information of desired viewpoint video (video to be distributed), or identification of unnecessary viewpoint video (video to be separated). Information is specified.

  The distribution control unit 17 of the content distribution server 10 sends the distribution request received from the receiving terminal 30 to the distribution management unit 12. The distribution management unit 12 controls the content acquisition unit 13 to read out the multi-view video content of the content identification information designated by the distribution request from the multi-view video content storage unit 11. The content acquisition unit 13 reads the instructed multi-view video content from the multi-view video content storage unit 11 and stores it in the work buffer memory 16. Then, the distribution management unit 12 instructs the demultiplexing / remultiplexing unit 14 to demultiplex and remultiplex the multi-view video content. At that time, the distribution management unit 12 instructs the separation / remultiplexing unit 14 on the video ES to be separated based on the distribution request. The demultiplexing unit 14 demultiplexes and remultiplexes the multi-view video content stored in the work buffer memory 16 by the content acquisition unit 13 as will be described later with reference to FIG. To remember. The distribution management unit 12 controls the content distribution unit 15 and the communication unit 18 to transmit the distribution content to the receiving terminal 30.

  FIG. 4 is a flowchart for explaining the operation of the demultiplexing / remultiplexing unit 14 of this embodiment. The specific processing contents in each step are as described later, but the main flow is as follows.

1. In response to a distribution request from the receiving terminal 30, the PID of the video ES to be separated (hereinafter abbreviated as “PID _filter ”) and the PID of the _PCR (hereinafter abbreviated as “PID _PCR ”) are acquired.

2. When the PID _filter and the PID _PCR are the same, the adaptation field of the TS packet of the separation target PID (= PID _filter = PID _PCR ) and the presence / absence of the payload are confirmed. (Determining whether TS packet needs filtering)
3. If the TS packet requires filtering, only the payload that is the video ES is replaced with null data and separated.

  4). Delete the replaced null data and generate TTS.

  Hereinafter, the operation of the demultiplexing / remultiplexing unit 14 will be described in detail with reference to FIG.

  In S001, the demultiplexing / remultiplexing unit 14 receives a demultiplexing / remultiplexing instruction (distribution file generation instruction) from the distribution managing unit 12. This instruction includes an instruction as to which target multi-view video content is to be separated from which viewpoint video ES of the multi-view video content.

  In S002, the demultiplexing / multiplexing unit 14 determines whether or not PMT (Program Map Table), which is PID index information for analyzing the PID of the TS packet, has been acquired.

  When the PMT has not been acquired (S002: NO), in S003, the demultiplexing / remultiplexing unit 14 acquires the PMT from the multi-view video content and analyzes it.

In S004, the demultiplexing / multiplexing unit 14 refers to the PMT, and acquires the PID (PID _filter ) of the separation target video ES specified in S001.

In S005, the separation / remultiplexing unit 14 refers to the PMT and acquires the PID _PCR of the _PCR . The PCR is clock information used as a reference when time synchronization between the distribution server (transmitter) and the receiving terminal 30 (receiver) is performed, and is embedded in a TS packet at a predetermined interval (for example, 100 ms interval). Yes. Then, the process returns to S002.

When PMT, PID _filter, and PID _PCR are acquired by performing S003 to S005 (S002: YES), in S006, the separation / remultiplexing unit 14 determines whether or not the separation target video ES is changed. To change the separation target video ES, a request from the receiving terminal 30 is received, and the distribution management unit 12 instructs the separation / remultiplexing unit 14. For example, there is a case where the receiving terminal 30 (viewer) that has requested only the video of the first viewpoint until now requests only the video of the second viewpoint on the way. If there is a change, the process returns to step S004 to perform the subsequent processing.

  In S007, the demultiplexing / multiplexing unit 14 reads each TS packet of the multi-view video content instructed to the distribution management unit 12 in S001 from the work buffer memory 16, analyzes the read TS packet to be processed, and determines the PID. get.

  In S008, it is determined whether or not the PID of the TS packet acquired in S007 is the PID of the PMT.

  If the PID of the TS packet is the PID of the PMT (S008: YES), in S009, the demultiplexing / remultiplexing unit 14 updates the work table stored in the work buffer memory 16, and proceeds to S016.

When the PID of the TS packet is not the PID of the PMT (S008: NO), in S010, the demultiplexing / remultiplexing unit 14 compares the PID of the TS packet acquired in S007 with the PID _filter, and the PID of the TS packet is It is determined whether or not it matches the PID _filter acquired in S004. When the PID of the TS packet does not match the PID _filter (S010: NO), the process proceeds to S016.

When the PID of the TS packet matches the PID _filter (S010: YES), the demultiplexing unit 14 compares the PID of the TS packet with the PID _PCR in S011, and the PID of the TS packet is acquired in S005. It is determined whether the PID _PCR matches.

When the PID of the TS packet does not match the PID _PCR (S011: NO), that is, the TS packet is a TS packet of the separation target video ES and does not include the PCR. Specifically, it is a TS packet of the pattern A in FIG. 3, and the payload stores the PES of the video ES to be separated. In this case, in S013, the demultiplexing / multiplexing unit 14 demultiplexes all the TS packets by nulling or the like.

If the PID of the TS packet matches the PID _PCR (S011: YES), in S012, the demultiplexing / remultiplexing unit 14 determines whether the adaptation field (PCR) is included in the TS packet. When the adaptation field (PCR) is not included in the TS packet (S012: NO), the process proceeds to S013, and the demultiplexing / multiplexing unit 14 separates all the TS packets by nulling or the like. In this case, the TS packet is the TS packet of the pattern A in FIG. 3, and the payload stores the PES of the video ES to be separated. Note that S012 considers the case where the same PID is set in the PID _filter and the PID _PCR .

  When the adaptation field (PCR) is included in the TS packet (S012: YES), in S014, the demultiplexing / remultiplexing unit 14 determines whether a payload exists in the TS packet. If no payload is included in the TS packet (S014: NO), specifically, in the case of the TS packet of the pattern C in FIG. 3, the process proceeds to S016 without performing separation processing such as nullification.

  When the payload is included in the TS packet (S014: YES), specifically, in the case of the TS packet of the pattern B in FIG. 3, the demultiplexing / remultiplexing unit 14 nullifies the payload portion in S015. The TS header and the adaptation field are left as they are.

 In S016, the demultiplexing / remultiplexing unit 14 determines whether or not an unprocessed TS packet exists in the multi-view video content in the work buffer memory 16, and if an unprocessed TS packet exists (S016: YES). , The process returns to S002, and the subsequent processes are repeated.

When there is no unprocessed TS packet (S016: NO), that is, when all TS packets have been processed, the demultiplexing / remultiplexing unit 14 is not separated in S013 and S015 by being nullified in S013. Use the remaining TS packets to generate distribution content. That is, the TS packet (S010: NO) whose PID does not match the PID _filter , the TS packet (S014: NO) that does not include the payload, and the payload portion are separated, and the TS header and the adaptation field portion are left as they are. Using the TS packet (S015), the distribution content from which the video ES to be separated is separated is generated.

  In this embodiment, in S017, the demultiplexing / remultiplexing unit 14 reads the processed TS packet from the work buffer memory 16, adds the time stamp information to the first 4 bytes of the TS packet, and makes the packet a total of 192 bytes. (Timestamped TS) Packetized to generate distribution content. The demultiplexing unit 14 uses the TTS packet in which the TS packet replaced with null in S013 is deleted and the portion replaced with null in S015 is deleted when the TSS packet is formed.

  In S018, the demultiplexing / multiplexing unit 14 ends the demultiplexing / multiplexing process.

  In the present embodiment described above, when separating a desired video from a multi-view video, the video is left with the PCR so that the receiving terminal 30 can synchronize the clock with the content distribution server 10. To separate. Thus, in the present embodiment, it is possible to correctly decode and operate on the receiving terminal 30 side without affecting the receiving terminal 30 during the distribution of multi-view video, and to provide a richer video distribution service to the viewer. Is possible.

  In other words, based on one video source (multi-view video content), the receiving terminal holds clock information (time information) necessary for real-time separation / re-multiplexing of video in accordance with the needs of the viewer The distribution of the re-multiplexed content can be enabled without affecting 30.

  As a result, in the present embodiment, it is possible to separate and remultiplex contents that can be decoded without giving a shock to the receiving terminal 30 even if filtering is performed on a specific video ES. In addition, H.264 MVC multi-viewpoint video can be flexibly distributed according to the viewer's intention, and one-source multi-use content management reduces server disk resources / management resources. NW bandwidth reduction at the time becomes possible.

  In addition, this invention is not limited to the said embodiment, Many deformation | transformation are possible within the range of the summary.

1: Content distribution server 11: Multi-view video content storage unit 12: Distribution management unit 13: Content acquisition unit 14: Separation / remultiplexing unit 15: Content distribution unit 16: Work buffer memory 17: Distribution control unit 18: Communication unit 20: Network 30: receiving terminal

Claims (8)

A multi-view video separation method for separating a video of a desired viewpoint from a multi-view video,
An obtaining step of obtaining a first packet identifier of a packet including a video to be separated in the multi-view video and a second packet identifier of a packet including clock information;
A comparison step of reading each packet of the multi-viewpoint video and comparing the processing target packet identifier of the read processing target packet with the first packet identifier and the second packet identifier;
A first separation step of separating the processing target packet when the processing target packet identifier matches the first packet identifier and does not match the second packet identifier;
If the processing target packet identifier matches the first packet identifier and the second packet identifier, and the clock information and the separation target video are present in the processing target packet, the separation target video is And a second separation step of separating from the packet to be processed. The multi-view video separation method according to claim 1,
Generation for generating a distribution video from which the video to be separated is deleted, using a packet whose processing target packet identifier does not match the first packet identifier and a packet remaining without being separated in the second separation step A multi-view video separation method characterized by further performing steps. The multi-view video separation method according to claim 1 or 2,
In the second separation step, when the processing target packet identifier matches the first packet identifier and the second packet identifier, and when the clock information does not exist in the processing target packet, the processing target packet is A multi-view video separation method characterized by separation. The multi-view video separation method according to any one of claims 1 to 3,
A receiving step of receiving a distribution request specifying a video to be separated or a video to be distributed in the multi-view video;
A distribution step of distributing the distribution video generated in the generation step to a receiving terminal that has transmitted the distribution request; A multi-view video separation device that separates a video of a desired viewpoint from a multi-view video,
Obtaining means for obtaining a first packet identifier of a packet including a video to be separated in the multi-view video and a second packet identifier of a packet including clock information;
A comparison unit that reads each packet of the multi-view video and compares the processing target packet identifier of the read processing target packet with the first packet identifier and the second packet identifier;
When the processing target packet identifier matches the first packet identifier and does not match the second packet identifier, the processing target packet is separated, and the processing target packet identifier is the first packet identifier and the second packet identifier. A separation means for separating the separation target video from the processing target packet when the processing target packet includes the clock information and the separation target video. Multi-viewpoint video separation device. The multi-view video separation device according to claim 5,
A multi-viewpoint video separation device further comprising: generation means for generating a distribution video from which a video to be separated is deleted, using a packet remaining without being separated by the separation means. The multi-view video separation device according to claim 5 or 6,
The separation means separates the processing target packet when the processing target packet identifier matches the first packet identifier and the second packet identifier, and the clock information does not exist in the processing target packet. Multi-viewpoint video separation device characterized by   A multi-view video separation program for causing a computer to execute the multi-view video separation method according to any one of claims 5 to 7.

Images