JP2018207288A - Redistribution system, redistribution method and program - Google Patents

Abstract

To provide a redistribution system, a redistribution method and a program which can reduce transmission cost and equipment material cost.SOLUTION: The redistribution system includes a receiver unit, an edition unit, an integration unit and a distribution unit. The receiver unit receives a first package which includes at least one type of contents encoded to an HTTP streaming form. The edition unit generates and outputs a new content which is based on at least a part of types among the contents included in the first package received by the receiver unit. The integration unit integrates and outputs the content included in the first package received by the receiver unit, and the new content generated by the edition unit, as one second package. The distribution unit redistributes the second package which is output from the integration unit.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a redistribution system, a redistribution method, and a program.

  A technique for distributing (streaming) various contents (video, audio, etc.) in real time using a communication line such as the Internet has become widespread. Distribution of video, audio, etc. using the Internet can be realized with relatively low cost facilities and relatively low information transmission costs, and is expected to be used more and more in the future.

  By the way, for example, when the content created by the first operator is received and redistributed by the second operator, the content may be added. In a typical example, when content is redistributed for a specific region or a specific language area, it may be desired to add content specific to the region or language. In such a case, in the conventional technology, first, content (for example, video and audio) produced by the first operator is transmitted to the second operator. Then, the second business operator added content to be added (for example, commentary sound in a specific language or commentary sound for a specific region), and then encoded it into a format for Internet distribution.

  In such a method according to the prior art, in order for the second operator to process and distribute the content, it is necessary to transmit high-quality content using a dedicated line or the like. In order to add the above voice), a lot of equipment and processes were required.

  Specifically, when the conventional technology is used, the video and audio transmitted from the first operator are decoded by a decoder, the video and audio are separated by DeMUX (demultiplexer), and then the original audio is restored. The voice that should be added was added. Then, the original video and the added audio are encoded using an encoder, and the video and the audio are combined again by a MUX (multiplexer) to be encoded into a format for Internet distribution.

  For example, Non-Patent Document 1 describes a system configuration when a broadcaster actually distributes all games and events of all sports via a live streaming over a large-scale sports event. According to this document, SDV image quality of IPVandA was transmitted from a local city center (Brazil) where an event was held to a broadcasting center in Tokyo using an international line. Note that SD quality video is based on a bit rate of about 2.5 Mbps. At the broadcast center, the IPVandA video was re-coded into a lower bit rate video and distributed via the Internet. In addition, for some competition videos, a simple audio booth was built in the above broadcasting center, and distributed with explanations and live audio specific to online distribution.

Kenji Shimanishi, Mune Endo, Yuki Kokubo, Shinya Orishita, Junichi Sakai, Aya Maeda, “On the production of digital content in the Rio de Janeiro Olympics”, Broadcast Technology, November 2016, p. 104-106.

In the prior art, when content is added and redistributed, it is necessary to reproduce at least a part of the original content. In addition, it is necessary to redistribute the original content and the added content at the same timing.
For this reason, it is necessary to first reproduce the original content in a baseband signal (uncompressed signal) state and add the content. For this reason, it is necessary to transmit a baseband signal (uncompressed signal) or a signal including a high-resolution video encoded at a high bit rate in order to transmit the original content. A costly communication line was required. Further, in order to configure such a system, a multi-stage process is required, that is, a lot of expensive equipment is required. In particular, it has been difficult in terms of cost to enable simultaneous delivery of content from a plurality of bases using conventional technology.

  The present invention has been made on the basis of the above problem recognition. When receiving original contents, adding new contents, and redistributing them together, transmission costs and equipment costs are reduced. It is an object of the present invention to provide a redistribution system, a redistribution method, and a program that can be lowered.

  [1] In order to solve the above problem, a redistribution system according to an aspect of the present invention includes a receiving unit that receives a first package including at least one type of content encoded in an HTTP streaming format, and the receiving unit includes: An editing unit that generates and outputs new content based on at least some types of content included in the received first package, and the first package received by the receiving unit. An integration unit that integrates and outputs at least a part of the content and the new content generated by the editing unit as one second package, and the second output from the integration unit And a distribution unit that redistributes the package.

  [2] According to another aspect of the present invention, in the redistribution system, the receiving unit receives the first package including at least one type of video content and at least one type of audio content. The editing unit reproduces the first audio that is at least one type of audio content included in the first package, and the first audio and another audio input corresponding to the first audio Are generated and output as the new content, and the integration unit includes the video content and the audio content included in the first package, and the new content. Are integrated and output so that the reproduction timing is consistent.

  [3] Further, according to one aspect of the present invention, in the above redistribution system, the editing unit sets matching timing information based on timing information held by the content included in the first package to the new content. And the integration unit makes the reproduction timing match based on the timing information held by the content included in the first package and the timing information given to the new content. It is characterized by that.

  [4] Further, according to one aspect of the present invention, in the redistribution system, the integration unit may be configured to perform the first audio based on similarity between the waveform of the first audio and the waveform of the second audio. By moving either one of the content of the first package including the content and the second audio that is the new content in the time direction, the output is integrated and output so that the reproduction timing is matched. It is characterized by that.

  [5] Further, according to one aspect of the present invention, in the redistribution system, the reception unit receives the first package including at least one type of audio content, and the editing unit includes the first package. The subtitle text content corresponding to the audio content is generated as the new content by performing audio recognition processing of at least one type of audio content included in the content, and the integration unit is included in the audio content Based on the correspondence in the time direction between the audio signal and the generated subtitle text, the audio content reproduction timing and the subtitle text presentation timing are integrated and output so as to match It is characterized by.

  [6] Further, according to one aspect of the present invention, a reception process of receiving a first package including at least one type of content encoded in an HTTP streaming format, and the content included in the first package received in the reception process An editing process for generating and outputting new content based on at least some types of content, and at least a part of the content included in the first package received in the receiving process; and An integration process in which the new content generated in the editing process is integrated and output as one second package; and a distribution process in which the second package output in the integration process is redistributed. This is a redistribution method.

  [7] Further, according to one aspect of the present invention, a computer receives a first package including at least one type of content encoded in an HTTP streaming format, and the first package received by the reception unit. An editing unit that generates and outputs new content based on at least some types of content included in the content, and at least a portion of the content included in the first package received by the receiving unit And the new content generated by the editing unit are integrated and output as one second package, and the distribution unit redistributes the second package output from the integration unit And a program for functioning as a redistribution system.

  According to the present invention, it is possible to redistribute after adding new content to streaming content at low transmission cost and low equipment cost.

It is the schematic which shows the schematic function structure of the redistribution system (redistribution apparatus) by 1st Embodiment of this invention, and the flow of the content data in the system. It is a block diagram which shows the structural example of the whole system containing the redistribution system by the embodiment. It is the schematic which shows the schematic function structure of the redistribution system (redistribution apparatus) by 2nd Embodiment, and the flow of the content data in the system. In the same embodiment, it is the schematic which shows the flow of the content at the time of the redistribution system redistributing the content distributed from the distribution server apparatus. It is the schematic which shows the structural example of the data stream-distributed from the delivery server apparatus side in the same embodiment. It is the schematic which shows the structural example of the data streamed from the redistribution system in the same embodiment. It is the schematic which shows the structural example of the index file of the highest layer used in the embodiment. It is the schematic which shows the structural example of the index file of a relatively lower layer used in the embodiment. It is the schematic which shows the schematic function structure of the redistribution system (redistribution apparatus) by 3rd Embodiment, and the flow of the content data in the system.

[First Embodiment]
FIG. 1 is a schematic diagram illustrating a schematic functional configuration of a redistribution system (redistribution apparatus) according to the present embodiment and a flow of content data in the system. As illustrated, the redistribution system 1 includes a receiving unit 120, an editing unit 140, an integration unit 160, and a distribution unit 180.

  The redistribution system 1 receives content distributed by, for example, HTTP streaming from an external distribution server. Note that HTTP is an abbreviation of HyperText Transfer Protocol. The content received by the redistribution system 1 includes a plurality of types of content such as video, audio, and text. Further, the content received by the redistribution system 1 may include, for example, a plurality of video contents, a plurality of audio contents, and the like. Then, the redistribution system 1 generates new content based on at least a part of the received content. The redistribution system 1 redistributes the received original content and the generated new content as a single content package.

The receiving unit 120 receives a first package including at least one type of content encoded in, for example, an HTTP streaming format. The receiving unit 120 may receive a plurality of types of content. In the illustrated example, a package including (m + n) types of contents from C (1) to C (m + n) is received. Here, m is an integer of 0 or more, and n is an integer of 1 or more. The receiving unit 120 receives these contents by, for example, HLS. HLS is an abbreviation for “HTTP Live Streaming” and is known as a method (protocol) for streaming video and the like via the Internet or the like.
The receiving unit 120 passes C (1) to C (m + n), which are the received contents, to the integrating unit 160. In addition, the receiving unit 120 passes C (m + 1) to C (m + n) of the received content to the editing unit 140.

  The editing unit 140 generates and outputs new content based on at least some types of content included in the first package received by the receiving unit 120. More specifically, the editing unit 140 receives n types of content from C (m + 1) to C (m + n) among the content received by the receiving unit 120 from the receiving unit 120. Then, the editing unit 140 generates new contents related to these contents based on the received contents C (m + 1) to C (m + n). The new contents generated by the editing unit 140 are k kinds of contents from C (m + n + 1) to C (m + n + k). However, k is an integer of 1 or more. There are various relationships between C (m + 1) to C (m + n), which are contents received by the editing unit 140, and C (m ++ n + 1) to C (m + n + k), which are contents generated by the editing unit. Have a relationship. In addition, since both are related to each other, the timing should be matched in reproduction or the like (more generally, presentation). The editing unit 140 passes the generated content to the integration unit 160.

The integration unit 160 integrates and outputs the content included in the first package received by the reception unit 120 and the new content generated by the editing unit 140 as one second package. The integration unit 160 integrates the contents C (1) to C (m + n) passed from the receiving unit 120 and the contents C (m + n + 1) to C (m + n + k) passed from the editing unit 140. To do. Note that the integration unit 160 receives C (1) to C (m + n) in an encoded state, and performs integration from C (m + n + 1) to C (m + n + k) as the contents as they are. Then, the integration unit 160 passes the entire content to the distribution unit 180 as one package. At this time, the integration unit 160 integrates the content delivered from the reception unit 120 and the content delivered from the editing unit 140 so that the reproduction timing is consistent.
Note that the integration unit 160 does not include all the contents C (1) to C (m + n), which are the contents passed from the reception unit 120, but only some of them from C (m + n + 1) to C (m + n + k). You may make it integrate. In this case, which of C (1) to C (m + n) is integrated with C (m + n + 1) to C (m + n + k) is appropriately determined.
That is, the integration unit 160 integrates at least a part of the content included in the first package received by the reception unit 120 and the new content generated by the editing unit 140 as one second package. And output.

  The distribution unit 180 redistributes the content (second package) passed from the integration unit 160.

  FIG. 2 is a block diagram illustrating a configuration example of the entire system including the redistribution system 1. As shown in the figure, this system includes a distribution server device 2, a redistribution system 1, and a client device 3. The redistribution system 1 is connected to the distribution server device 2 and the client device 3 via a communication line such as the Internet. Although only one client device 3 is shown in this figure, a large number of client devices 3 may actually be connected to the redistribution system 1. The point that the redistribution system 1 includes the receiving unit 120, the editing unit 140, the integration unit 160, and the distribution unit 180 is as described with reference to FIG.

The distribution server device 2 is a server computer that distributes original content. The content distributed by the distribution server device 2 is content composed of video and audio, for example. The distribution server device 2 uses, for example, the aforementioned HLS for content distribution.
The client device 3 receives content (redistributed content) sent from the redistribution system 1. The client device 3 is realized using, for example, a personal computer (PC), a smartphone (smartphone), a wristwatch-type information terminal, a glasses-type information terminal, other information devices, or the like. The client device 3 has, for example, a web browser function, and the web browser functions as an HTTP client. Thereby, the content redistributed by the HLS from the redistribution system 1 can be viewed.

  According to the configuration of the present embodiment, redistribution of content is realized without adding content based on a baseband signal (uncompressed signal) and adding new content related to the content received in the streaming format. It becomes possible. In other words, since the processes and equipment can be greatly reduced, a redistribution system can be realized at a low cost.

[Second Embodiment]
Next, a second embodiment will be described. In addition, about the matter already demonstrated in previous embodiment, description may be abbreviate | omitted below. Here, the description will focus on matters specific to the present embodiment.

  FIG. 3 is a schematic diagram illustrating a schematic functional configuration of a redistribution system (redistribution apparatus) according to the present embodiment and a flow of content data in the system. As illustrated, the redistribution system 11 includes a receiving unit 220, an editing unit 240, an integration unit 260, and a distribution unit 280.

  The redistribution system 11 receives video and audio content (referred to as “audio A”), generates another audio content (referred to as “audio B”) based on the received audio content, and receives the received original content. Content (including audio A) and the generated audio content (audio B) are redistributed.

  The receiving unit 220 receives a first package including at least one type of video content and at least one type of audio content. Specifically, the receiving unit 220 receives content of streaming video and audio (referred to as “audio A”) distributed via a communication line such as the Internet. The video and audio A received by the receiving unit 220 are encoded and transmitted from, for example, an external distribution server. As an example, the receiving unit 220 can be realized by using a reception-only computer device or the like.

The editing unit 240 reproduces the first audio that is at least one type of audio content included in the first package, and outputs the first audio and another audio input corresponding to the first audio. The second audio obtained by superimposing is generated and output as new content. That is, the editing unit 240 receives and reproduces at least the content of the audio A among the content received by the receiving unit 220 from the outside. Note that the editing unit 240 may also receive and play back video content. Then, the editing unit 240 mixes the content of the audio A and the audio collected from the microphone or the like connected to the editing unit 240 in the audio band, encodes it with a predetermined encoding method, and creates a new Output as audio content (referred to as “audio B”).
As an example, an announcer and commentators perform the actual situation or explanation while viewing the content (video and audio A) reproduced by the editing unit 240. That is, announcers and commentators utter their voices toward a microphone or the like, and the editing unit 240 generates the content of the voice B including the voices. As described above, when the decoded first sound is reproduced and the announcer or commentator speaks the first sound in real time, the delay time for processing the sound does not occur. Or the delay time is negligibly small. Therefore, the first sound and the new sound are mixed at an appropriate timing, and the sound B is generated.
Note that the voice B may be created by other methods. In that case, when creating the voice B, if necessary, appropriate processing for timing adjustment is performed so that the timing of the first voice and another voice (voices from announcers and commentators) are matched. You may go.
Note that the editing unit 240 can be realized using a computer. As an example, the editing unit 240 can be realized by using a personal information device such as a personal computer or a smartphone (smartphone).

The integration unit 260 integrates and outputs the video content and audio content included in the first package so that the playback timing is consistent with the new content. The integration unit 260 integrates the content (video and audio A) originally received by the reception unit 220 and the content (audio B) passed from the editing unit 240 and passes the integrated content to the distribution unit 280. When integrating the content, the integration unit 260 matches the timing between the content (video and audio A) received by the reception unit 220 and the content (audio B) passed from the editing unit 240. Adjust to Further, the integration unit 260 adjusts the level between the audio A passed from the receiving unit 220 and the audio B passed from the editing unit 240. The integration unit 260 receives the video and audio A from the reception unit 220 in an encoded state. Then, integration with the voice B is performed as it is.
Note that the integration unit 260 integrates not only all of the content (video and audio A) passed from the receiving unit 220 but also the content (audio B) passed from the editing unit 240. Also good. In that case, for example, the integration unit 260 may integrate only the video passed from the reception unit 220 and the audio B passed from the editing unit 240. Further, for example, only the voice A passed from the receiving unit 220 and the voice B passed from the editing unit 240 may be integrated. In any of these cases, when integrating the content, the integration unit 260 adjusts the timing so that the content received by the reception unit 220 and the content passed from the editing unit 240 are mutually matched. To do.
That is, the integration unit 260 integrates at least a part of the content included in the first package received by the reception unit 220 and the new content generated by the editing unit 240 as one second package. And output.
Note that the timing adjustment and level adjustment processing by the integration unit 260 is automatically performed. Details of how the integration unit 260 adjusts the timing will be described later.

  The distribution unit 280 distributes the content output from the integration unit 260. The distribution unit 280 distributes content via the Internet or the like.

  FIG. 4 is a schematic diagram showing the flow of content when the redistribution system redistributes the content distributed from the distribution server device in the present embodiment. In FIG. 3, a receiving unit 220, an editing unit 240, an integration unit 260, and a distribution unit 280 are devices (or a part of functions) constituting the redistribution system 11 as shown in FIG. Further, these functions constituting the redistribution system 11, the distribution server device 2, and the client device 3 are connected to the Internet and can communicate with each other. Note that means other than the Internet may be used for communication. In FIG. 4, only one client device 3 is shown, but in reality, a large number of client devices 3 may receive distribution from the distribution unit 280.

  As illustrated, the distribution server device 2 distributes content including video and audio via the Internet. For example, the above-mentioned HLS is used for content distribution. The receiving unit 220 receives the content distributed from the distribution server device 2. The receiving unit 220 passes the received video and audio (audio A) as content to the integration unit 260 via the Internet or other line. In addition, the receiving unit 220 passes at least the audio A of the received content (also video if necessary) to the editing unit 240 via the Internet or other line. The editing unit 240 generates audio B, which is audio content different from the audio A, based on the content received from the receiving unit 220. Note that voice A may be mixed in voice B. In a typical application example, the voice A is a live broadcast voice from the site where an event or the like is performed, and the voice B is an utterance matched with the voice A by an announcer or commentator using the editing unit 240. Are mixed. The editing unit 240 refers to the timing information included in the file having the sound A, adds the timing information to the sound B, encodes it, and outputs it as a file. Here, the timing information is, for example, a PTS (presentation time stamp). Furthermore, the editing unit 240 generates metadata (data such as a file name associated with the sound A and the sound B) for matching the reproduction timing between the sound A and the sound B. Then, the editing unit 240 passes the audio B to the integrating unit 260 as new audio content. Note that the editing unit 240 may transmit the audio content (audio B) to the integration unit 260 via the Internet or other line.

  The integration unit 260 integrates the content received from the reception unit 220 and the content received from the editing unit 240 based on metadata (data such as file name correspondences) for matching the reproduction timing. One of the important processes performed by the integration unit 260 is that the timing between the content from the receiving unit 220 side and the content from the editing unit 240 side is determined based on the above-described metadata (file name correspondence, etc.). To match. That is, when the integration unit 260 synchronizes between the contents, the content of the audio B generated by the editing unit 240 is matched with each of the video and audio (audio A) from the receiving unit 220 side. It becomes possible to deliver. The integration unit 260 includes a buffer area for accumulating content in consideration of a case where the arrival time of the content from the reception unit 220 and the content arrival time from the editing unit 240 that are to be matched in timing does not match. The integration unit 260 passes the integrated content to the distribution unit 280. The distribution unit 280 distributes the entire content delivered from the integration unit 260 via the Internet. For example, the above-mentioned HLS is used for content distribution. The client device 3 receives the content redistributed from the distribution unit 280, decodes and reproduces it. Note that the client device 3 may reproduce the video content and, at the same time, reproduce the audio content of either the audio A or the audio B as appropriate.

Next, the format of data distributed in the present embodiment will be described.
FIG. 5 is a schematic diagram illustrating a configuration example of streaming distribution data received by the receiving unit 220 in the present embodiment. As shown in the figure, data distributed from the distribution server device 2 side has a hierarchical structure. In the figure, the leftmost layer represents the highest layer, the middle represents the middle layer, and the rightmost represents the lowest layer. In the highest hierarchy, there is one index file, and the file name is “IndexFile.m3u8”. This index file “IndexFile.m3u8” holds location information (file name, path name, etc.) of another three types of index files in the lower layer (intermediate layer). The three types of index files are “Alternate-LowIndex”, “Alternate-MidIndex”, and “Alternate-HiIndex”. These three types of index files can be properly used according to the communication bandwidth that can be secured. For example, the user on the client device side receiving the delivery can specify one of three types of low bandwidth, medium bandwidth, and high bandwidth. Each of “Alternate-LowIndex”, “Alternate-MidIndex”, and “Alternate-HiIndex” holds a list of moving image location information for each predetermined time length (for example, 6 seconds). As an example, the index file “Alternate-LowIndex” holds the location information of four video files “Low — 01.ts”, “Low — 02.ts”, “Low — 03.ts”, and “Low — 04.ts”. doing. Note that “Low — 01.ts”, “Low — 02.ts”, “Low — 03.ts”, and “Low — 04.ts” are moving image files to be sequentially played back. The index file “Alternate-LowIndex” is not limited to four, and can have location information of an arbitrary number of moving image files. Here, the index file “Alternate-LowIndex” has been described as an example, but each of “Alternate-MidIndex” and “Alternate-HiIndex” also holds the location information of the moving image file according to the bandwidth.

In the case of the data configuration shown in FIG. 5, the sound (sound A) is included in each moving image file (Low_01.ts, Mid_01.ts, Hi_01.ts, etc.).
On the other hand, the audio (audio A) may be distributed as an independent file from the distribution server device 2 side, and the receiving unit 220 may also receive the audio file. In this case, the audio is divided into appropriate lengths and distributed as a plurality of files over time. These audio files are indexed by the same index file that indexes the moving image file.

  FIG. 6 is a schematic diagram illustrating a configuration example of streaming distribution data output from the integration unit 260 and distributed by the distribution unit 280 in the present embodiment. As shown in the figure, the data distributed by the integration unit 280 is also configured in a hierarchical structure. Similar to the data configuration described with reference to FIG. 5, the leftmost side represents the highest hierarchy, the middle represents the middle hierarchy, and the rightmost represents the lowest hierarchy. In the highest hierarchy, there is one index file, and the file name is “IndexFile.m3u8”. This index file “IndexFile.m3u8” holds the location information (file name, path name, etc.) of another five types of index files in the lower layer (intermediate layer). These five types of index files are “Alternate-LowIndex”, “Alternate-MidIndex”, “Alternate-HiIndex”, “mixed”, and “original”.

  Among these, three types of “Alternate-LowIndex”, “Alternate-MidIndex”, and “Alternate-HiIndex” are indexes related to moving image files, as in the data configuration described with reference to FIG. The subordinate moving image files of these three types of index files have the same data structure as described with reference to FIG.

  Of the five types of index files in the middle layer, two types, “mixed” and “original”, index audio files, respectively. Each of “mixed” and “original” holds a list of audio file location information for each predetermined time length (for example, 6 seconds). As an example, the index file “mixed” holds information on the location of four audio files “mixed — 01.ts”, “mixed — 02.ts”, “mixed — 03.ts”, and “mixed — 04.ts”. Yes. Note that “mixed — 01.ts”, “mixed — 02.ts”, “mixed — 03.ts”, and “mixed — 04.ts” are audio files to be sequentially played back. Note that the index file “mixed” is not limited to four, and can have location information of an arbitrary number of audio files. Just like “mixed”, “original” also holds a list of location information of another audio file for each predetermined time length (for example, 6 seconds). In other words, “original” includes the location information of the four audio files to be sequentially played, “original — 01.ts”, “original — 02.ts”, “original — 03.ts”, and “original — 04.ts”. Hold.

  Note that the audio file (such as mixed_01.ts) indexed by the index file “mixed” includes the audio (audio B) output by the editing unit 240. The audio file (original_01.ts, etc.) indexed by the index file “original” includes the original audio (audio A) received by the receiving unit 220 from the distribution server device 2 side.

When an independent file of the sound A is distributed from the original distribution server device 2, the integration unit 260 may output the file as it is as an audio file indexed by “original”.
When the sound A distributed from the original distribution server device 2 exists only in the moving image file to be distributed, the integration unit 260 extracts the sound from these moving image files and generates an audio file. Then, the integration unit 260 may output the generated audio file as an audio file indexed by “original”.

  FIG. 7 is a schematic diagram illustrating a configuration example of an index file used in the present embodiment. The file exemplified here is the highest index file in the hierarchical structure. The file name of this index file is “playlist.m3u8”. As shown in the figure, the index file “playlist.m3u8” is an extended M3U format file, and includes text representing index information therein. In FIG. 7, a line number corresponding to each line of text is given for convenience. The contents of the index file “playlist.m3u8” will be described below.

The first line is a header indicating that the file is an extended M3U format file.
The second and third lines hold information related to audio content. Both the second and third lines contain the description “TYPE = AUDIO”, which indicates that the second and third lines are indexes of audio content. The second and third lines both contain the description “GROUP-ID =“ audio ””, which is the identification information that the second and third lines are both “audio”. It belongs to the group identified by.
Among these, the second line includes a description “NAME =“ mixed ””, which is a mixed sound, that is, an additional sound added by the editing unit 240 (that is, a sound B). ). The second line includes a description “DEFAULT = YES”, which indicates that it is a default voice. The second line holds the location information of the lower index file related to the sound. “URI =” mixed / playlist. The description “m3u8 ″” corresponds to the location information.
On the other hand, the third line includes a description “NAME =“ original ””, which indicates the original sound before being mixed. That is, it indicates that the voice (voice A) is received by the receiving unit 220 with no additional voice added. The third line includes a description “DEFAULT = NO”, which indicates that this is not the default voice. The third line holds the location information of the lower index file related to the sound. “URI =” original / playlist. The description “m3u8 ″” corresponds to the location information.

  The fourth line is information indicating that the segment of the content is an independent segment. That is, it represents that information from another segment is not required to decode the content of the segment.

The 5th to 16th lines include the index information of 6 types of video files.
The fifth and sixth lines hold the index information of the first video. In the first video, the bandwidth (BANDWIDTH) and the average bandwidth (AVERAGE-BANDWIDTH) are both “545600” (the unit is bits per second). Also, codecs (codec) for decoding the video stream are “avc1.66.30” and “mp4a.40.2”. Further, the resolution of this video is “480 × 270”. The index file of this video is “stream1 / playlist.m3u8”.
The seventh and eighth lines hold the index information of the second video. In the second video, both the bandwidth (BANDWIDTH) and the average bandwidth (AVERAGE-BANDWIDTH) are “765600” (bits per second). Also, codecs (codec) for decoding the video stream are “avc1.66.30” and “mp4a.40.2”. The resolution of this video is “640 × 360”. The index file of this video is “stream2 / playlist.m3u8”.
The ninth and tenth lines hold the index information of the third video. In the third video, both the bandwidth (BANDWIDTH) and the average bandwidth (AVERAGE-BANDWIDTH) are “1425600” (bits per second). Further, codecs (codec) for decoding the video stream are “avc1.42c01f” and “mp4a.40.2”. The resolution of this video is “640 × 360”. The index file of this video is “stream3 / playlist.m3u8”.

The eleventh and twelfth lines hold the fourth video index information. In the fourth video, the bandwidth (BANDWIDTH) and the average bandwidth (AVERAGE-BANDWIDTH) are both “3955600” (bits per second). Further, codecs (codec) for decoding the video stream are “avc1.4d401f” and “mp4a.40.2”. Further, the resolution of this video is “960 × 540”. The index file of this video is “stream4 / playlist.m3u8”.
The 13th and 14th lines hold the index information of the fifth video. In the fifth video, the bandwidth (BANDWIDTH) and the average bandwidth (AVERAGE-BANDWIDTH) are both “5640800” (bits per second). Further, codecs (codec) for decoding the video stream are “avc1.4d401f” and “mp4a.40.2”. Further, the resolution of this video is “1280 × 720”. The index file of this video is “stream5 / playlist.m3u8”.
The 15th and 16th lines hold the index information of the sixth video. In the sixth video, the bandwidth (BANDWIDTH) and the average bandwidth (AVERAGE-BANDWIDTH) are both “7290800” (bits per second). Further, codecs (codec) for decoding the video stream are “avc1.4d401f” and “mp4a.40.2”. Further, the resolution of this video is “1280 × 720”. The index file of this video is “stream6 / playlist.m3u8”.

It should be noted that the frame rate (FRAME-RATE) is common to the first to sixth images.
) Is defined as “30.000”. Further, the description “AUDIO =“ audio ”” is included for all the videos from the first video to the sixth video. This indicates that the audio content associated with each video is identified by the group ID “audio”. That is, the audio content associated with each video is defined in the second row or the third row.

  FIG. 8 is a schematic diagram illustrating an example of an index file used in the present embodiment. The file shown here is a lower-order index file that is referenced from the highest-order index file shown in FIG. The file name of this index file is “mixed / playlist.m3u8”. In the description of the second line of the top-level index file shown in FIG. 7, “URI =” mixed / playlist. The description “m3u8 ″” indicates the location of the file in FIG. Here, “mixed” is a directory name, and this directory is a directory for storing a file for mixed audio (audio B). That is, the index file “mixed / playlist.m3u8” holds index information related to mixed audio. This index file “mixed / playlist.m3u8” is also an extended M3U format file. In FIG. 8 as well, a line number corresponding to each line of text is given. The contents of the index file “mixed / playlist.m3u8” will be described below.

The first line is a header indicating that the file is an extended M3U format file.
The second line is file format version information. Specifically, this indicates that the version of this file format is “3”.
“# EXT-X-TARGETDURATION” on the third line indicates an estimated time length of a media file to be added next. In the example of this data, the predicted time length is 6 seconds.
“# EXT-X-MEDIA-SEQUENCE” in the fourth line represents the sequence number of the first media file included in the index file. In this data example, the first sequence number is “417554” (this number is included in the file name of the file specified in the eighth line).
Description of “# EXT-X-DISCONTINITY-SEQUENCE” in the fifth line is omitted.

  From the 6th line to the 35th line, a group having a set of 3 lines is repeated 10 times (30 lines in total). The first line in each set associates a media file with a date / time. The second row represents the length of the media segment in seconds. The third line is information for referring to the media file itself.

Here, as an example, a group from the sixth line to the eighth line will be described.
“# EXT-X-PROGRAM-DATE-TIME” on the sixth line associates the referenced media file with the date and time. In this data example, the first media file is “2017-05-11T16: 19: 02.866 + 09: 00” (year / month / hour / minute / second and thousandth of a second notation). Date and time in a time zone 9 hours preceding standard time).
“#EXTINF” in the seventh row represents the length of the media segment corresponding to this set. Specifically, it is specified that the length is 6.000 seconds. Note that a title can be specified next to the comma following “6.00”, but the description of the title is omitted in this data.
The eighth line describes the file names of this set of media files (here, the audio file of mixed audio (audio B)). In this data, specifically, “test2 — 270 — 417554.ts”.

  Similarly, the nine sets subsequent to this set describe date and time information, media segment length information, and file name information of media files in the media segment. Since the specific date and time, the length of the media segment, and the file name are as described in the drawing, the description is omitted here.

  As described above, the index file exemplified here holds information for 10 segments for a mixed audio file. The length of each segment is 6 seconds, and the total length of 10 segments is 60 seconds.

  The integration unit 260 generates and outputs the index file illustrated in FIG. That is, the integration unit 260 passes the content including both the audio A (NAME = “original”) and the audio B (NAME = “mixed”) to the distribution unit 280. The distribution unit 280 distributes the content integrated by the integration unit 260 to the client device 3.

  Next, details of a method in which the integration unit 260 matches the timing of the (audio) audio B generated by the editing unit 240 with the timing of the video and audio (audio A) passed from the reception unit 220 will be described. In the method of this embodiment, presentation time information included in a file is used. That is, the video and audio (audio A) file received by the receiving unit 220 includes reproduction timing information (PTS, presentation time stamp) and information on the length of the reproduction time. When HLS is used, the receiving unit 220 can obtain timing information (PTS) from a TS (Transport Stream) file including video / audio data. Further, it is possible to acquire information on the length of the reproduction time from the description of “#EXTINF” of the index file (M3U8 file) distributed from the distribution server device 2. The editing unit 240 generates the sound B (mixed sound) while reproducing the original sound A. At this time, the timing information and the length of the reproduction time included in the sound A file are directly sounded. Embed in B. For example, the timing information (PTS-1) at the time when the input of the voice is started is acquired, and when the voice B is generated, the timing information at the head of the output stream is output so as to be the PTS-1. Furthermore, when the length of the playback time acquired from the M3U8 file is 5 seconds, the output stream is generated by dividing the output stream into files every 5 seconds. That is, the editing unit 240 generates and outputs a sound B having the same timing information and playback time length information as the individual files constituting the sound A. Then, the integration unit 260 confirms that the timing information and the playback time length information in the audio A and audio B files are the same, and the video and audio (audio A) passed from the receiving unit 220. A new M3U8 file including information on video, audio A, and audio B is generated so that the reproduction timing of the generated audio B is matched, and as an HLS content, an M3U8 file, a video TS file, an audio A TS file, The TS file of voice B is distributed.

  That is, the editing unit 240 gives matching timing information to the new content to be generated based on the timing information held by the content included in the first package. Further, the integration unit 260 matches the playback timing based on the timing information held by the content included in the first package and the timing information given to the new content.

  As a result, the content can be redistributed in a state in which the original content received by the redistribution system 11 and the content added by the redistribution system 11 are matched.

[Second Embodiment: Modification]
Next, a modification of the second embodiment will be described. The basic configuration of this modification is the same as that in the second embodiment, but the part of the method in which the integration unit 260 matches the timing between the voice A and the voice B is different from the second embodiment.

  In this modification, the integration unit 260 matches the timings of the voice A and the voice B as follows. The editing unit 240 generates a mixed sound (sound B) in which the original sound (sound A) is mixed with the utterance of an announcer or the like. That is, the audio B data generated by the editing unit 240 includes the audio A information. The integration unit 260 acquires the voice A (also referred to as “comparison voice”) and the voice B (also referred to as “comment voice” since a comment by an utterance is added). The integrating unit 260 may acquire the voice A directly from the receiving unit 220 or may acquire it from the editing unit 240. The integration unit 260 uses the fact that the signal of the sound A is included in the sound B, and executes processing for matching the timings of the sound A and the sound B.

As an example, the integration unit 260 performs the following calculation. The voice A and the voice B are represented by S _A (t) and S _B (t), respectively. S _A (t) and S _B (t) are signal values (for example, the amplitude of an audio signal) at time t, respectively. When the voice A and the voice B arrive at the integration unit 260, there is a possibility that the timings of the voice A and the voice B are shifted due to the difference in the process route up to that point. The shift amount is assumed to be Δt (delta · t). When the processing shown in FIG. 3 or the like is configured as an apparatus, it is reasonable to assume that the amount of timing deviation between the voice A and the voice B is normally less than one second at most, and within a few seconds at most in special cases. is there. Then, the integration unit 260 calculates a cross-correlation value between the signal S _A (t) and the signal S _B (t + Δt) in a predetermined time interval including the time t. The cross-correlation value is expressed as c = corr (S _A (t), S _B (t + Δt)). Here, corr () is a function for obtaining a cross-correlation value between two signals. Then, the integration unit 260 obtains a deviation amount Δt that maximizes the cross-correlation value c. Then, the integration unit 260 changes the value of the timing information (PTS) based on the obtained shift amount Δt, and outputs the contents of the audio A and audio B contents at the same timing.

Note that when the cross-correlation value is calculated by the function corr (), the signal level of the sound A and the signal level of the sound B may be appropriately adjusted. Further, the signal level adjustment amount here may be automatically obtained based on, for example, machine learning.
In addition, here, a method of matching the timings of the voice A and the voice B using the cross-correlation value is described as an example. For example, the signal waveforms of the sound A and the sound B may be compared by image processing, and the shift amount Δt that maximizes the degree of coincidence between the waveforms may be obtained.

  When organized, the integration unit 260 creates the new content generated by the editing unit 240 and the content of the first package including the content of the first audio based on the similarity between the waveform of the first audio and the waveform of the second audio. By moving either one of the second sounds as the contents in the time direction, the first sound and the second sound are integrated and output so that the reproduction timing is matched.

[Third Embodiment]
Next, a third embodiment will be described. Note that description of matters already described before the previous embodiment may be omitted below. Here, the description will focus on matters specific to the present embodiment.

  FIG. 9 is a schematic diagram illustrating a schematic functional configuration of a redistribution system (redistribution apparatus) according to the present embodiment and a flow of content data in the system. As illustrated, the redistribution system 12 includes a reception unit 320, an editing unit 340, an integration unit 360, and a distribution unit 380.

  The redistribution system 12 receives video and audio content. Then, the redistribution system 12 generates subtitle text content based on the received audio content. The redistribution system 12 redistributes the received original content and the generated subtitle text content in a state where there is a timing to reproduce and present.

  The receiving unit 320 receives a first package including at least one type of audio content. Specifically, for example, the receiving unit 320 receives content composed of video and audio from an external distribution server device in a streaming format. The reception unit 320 transmits the received video file and audio file to the integration unit 360. In addition, the reception unit 320 transmits the received audio file to the editing unit 340.

The editing unit 340 performs a speech recognition process on at least one type of audio content included in the first package, thereby generating subtitle text content corresponding to the audio content as new content. The editing unit 340 includes a voice recognition engine inside, and has a function of converting input voice into a character string. In addition, the editing unit 340 further shapes the character string converted from the sound into a subtitle text data format, and outputs the character string as a file that can be displayed as part of the video in live streaming. At this time, the editing unit 340 performs timing for each piece of subtitle text data based on the timing information (PTS, presentation time stamp) included in the original audio file and the relative position of the time in the file. Give information. Note that the editing unit 340 can output subtitle text in a data format to which timing information can be added, such as Timed Text Markup Language (TTML). The editing unit 340 transmits the subtitle text data file generated based on the sound to the integration unit 360.
Note that existing technology can be applied to the speech recognition engine itself. As a basic process, the speech recognition engine extracts the acoustic features of the input speech, and considers the features of the language as necessary. Is output as

  The integration unit 360 matches the reproduction timing of the audio content and the presentation timing of the subtitle text based on the correspondence in the time direction between the audio signal included in the audio content and the generated subtitle text. And output it. That is, the integration unit 360 integrates the video and audio content file received from the reception unit 320 and the subtitle text file received from the editing unit 340 as a package, and passes the package to the distribution unit 380. More specifically, the integration unit 360 outputs the content data in a state where the timing information between the audio content and the subtitle text content is consistent. The integration unit 360 receives the video and audio content from the reception unit 320 in an encoded state. Then, in the state as it is, integration with the content of the subtitle text is performed.

  The distribution unit 380 distributes the content file passed from the integration unit 360 via the Internet or the like. Specifically, the distribution unit 380 distributes video, audio, and subtitle text content.

[Third Embodiment: Modification 1]
Next, Modification 1 of the third embodiment will be described. The basic configuration of this modification is the same as that in the second embodiment, but is characterized in that the integration unit 360 further performs language translation.

  In Modification 1 of the third embodiment, the editing unit 340 includes a language translation engine. The language translation engine translates text in a natural language into another language. For example, the integration unit 360 translates Japanese text obtained as a result of the speech recognition processing into English, and outputs English subtitle text data. Alternatively, the editing unit 340 translates the French text obtained as a result of the speech recognition processing into Japanese and outputs Japanese subtitle text data. Note that the language of the translation source and the translation destination may be other than those exemplified here. Note that the timing information is also given to the translated subtitle text based on the timing information added to the original voice. The editing unit 340 transmits the translated subtitle text to the integration unit 360. The subsequent processing is the same as the processing in the already described form.

[Third Embodiment: Modification 2]
Next, Modification 2 of the third embodiment will be described. The basic configuration of this modification is the same as that in the second embodiment, but the characteristic is that the integration unit 360 further translates into sign language.
Note that existing technology may be applied to the language translation function itself.

  In the second modification of the third embodiment, the editing unit 340 has a function of translating into sign language. The language translation engine translates text data obtained as a result of the speech recognition processing into a sign language expression. Then, the editing unit 340 generates and outputs video content corresponding to the translated sign language expression. The sign language is represented as an image using, for example, computer graphics (CG). Note that timing information is also given to the output sign language video based on the timing information added to the original voice. The editing unit 340 transmits the generated sign language video data to the integration unit 360. The integration unit 360 passes the sign language video data to the distribution unit 380 instead of the caption text data described in the third embodiment. The distribution unit 380 distributes the original video and audio content and the sign language video generated by the editing unit 340.

  In addition, you may make it implement | achieve the function of the redistribution system in embodiment mentioned above and its modification, and the function of some apparatuses which comprise a redistribution system with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” is a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, a DVD-ROM, a USB memory, or a storage device such as a hard disk built in a computer system. That means. Furthermore, a “computer-readable recording medium” dynamically holds a program for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. The program may be a program for realizing a part of the above-described functions, or may be a program that can realize the above-described functions in combination with a program already recorded in a computer system.

  As described above, according to any of the embodiments described above or the modifications thereof, the redistribution system receives content distributed using means such as HLS via the Internet or the like. In other words, the redistribution system does not receive content composed of baseband signals (uncompressed signals). Then, the redistribution system generates another new content based on at least a part of the received content. Then, the redistribution system integrates the received original content and the generated new content, and redistributes it. Redistribution also uses, for example, HLS. As a result, the client device can receive streaming distribution of content in a state where new content is added.

  And according to each embodiment or its modification, it becomes possible to implement | achieve a re-distribution system with the minimum process and equipment, and the cost of constructing and operating a system can be held down. Further, for example, it is possible to realize a service in which new contents such as audio are added to a streaming video in a distribution format and redistributed basically anywhere as long as there is an environment that can be connected to the Internet.

  In terms of cost, expensive special equipment for processing baseband signals (uncompressed signals) is not required, and video transmission is possible over the Internet, so a significant reduction in transmission costs can be expected. Furthermore, since processing can be performed only with a general-purpose computer and software running on the computer, it is possible to realize a service for adding content from anywhere that can be connected to the Internet. In addition, the timing of the original content (such as video and audio) and the content to be added (for example, audio) can be automatically synchronized within the redistribution system. As a result, the flow of adding new content (audio or the like) in real time to the existing stream video and audio can be integrated into one and the service can be easily realized. By applying the configuration described in the above embodiment or the like to a service such as content distribution, it is possible to provide a variety of services with high mobility.

  Note that content newly added by the redistribution system is not limited to audio content. In the example already described, text (including so-called caption text) and video (for example, sign language video) can be generated and added. It is also possible to generate and add contents other than those exemplified here.

  As mentioned above, although embodiment of this invention has been explained in full detail with reference to drawings, a concrete structure is not restricted to this embodiment. You may make it implement in the further modification. Moreover, you may design within the range which does not deviate from the summary of this invention.

  For example, in the above embodiment, HLS is used as a format for distributing video and audio content, but it may be distributed in other formats. For example, formats such as MPEG-DASH, HDS, and MS Smooth Streaming can also be used.

  The present invention can be used, for example, in a business for distributing content. However, industrial applicability is not limited to the field illustrated here.

1 Redistribution system (redistribution device)
2 Distribution server device 3 Client device 11, 12 Redistribution system (Redistribution device)
120 receiving unit 140 editing unit 160 integrating unit 180 distributing unit 220 receiving unit 240 editing unit 260 integrating unit 280 distributing unit 320 receiving unit 340 editing unit 360 integrating unit 380 distributing unit

Claims (7)

A receiving unit for receiving a first package including at least one type of content encoded in an HTTP streaming format;
An editing unit that generates and outputs new content based on at least some types of content included in the first package received by the receiving unit;
At least a part of the content included in the first package received by the receiving unit and the new content generated by the editing unit are integrated and output as one second package. The integration department;
A distribution unit that redistributes the second package output from the integration unit;
A redistribution system comprising: The receiving unit receives the first package including at least one type of video content and at least one type of audio content;
The editing unit reproduces the first sound, which is at least one kind of sound content included in the first package, and the first sound and another sound input corresponding to the first sound; To generate a second sound obtained by superimposing and outputting as the new content,
The integration unit integrates and outputs the video content and the audio content included in the first package so that the playback timing is consistent between the new content,
The redistribution system according to claim 1. The editing unit gives matching timing information to the new content based on timing information held by the content included in the first package,
The integration unit matches reproduction timing based on timing information held by content included in the first package and timing information given to the new content.
The redistribution system according to claim 2. The integration unit is based on the similarity between the waveform of the first audio and the waveform of the second audio, the content of the first package including the content of the first audio, and the first content that is the new content By moving either one of the two voices in the time direction, they are integrated and output so that the timing of playback is consistent,
The redistribution system according to claim 2. The receiving unit receives the first package including at least one type of audio content;
The editing unit generates, as the new content, subtitle text content corresponding to the audio content by performing audio recognition processing of at least one type of audio content included in the first package;
The integration unit, based on a time direction correspondence between the audio signal included in the audio content and the generated subtitle text, the reproduction timing of the audio content and the presentation timing of the subtitle text Are integrated and output so that
The redistribution system according to claim 1. Receiving a first package including at least one type of content encoded in an HTTP streaming format;
An editing process for generating and outputting new content based on at least some types of content included in the first package received in the receiving process;
At least a part of the content included in the first package received in the reception process and the new content generated in the editing process are integrated and output as one second package. Integration process,
A distribution process of redistributing the second package output in the integration process;
A redistribution method characterized by including: Computer
A receiving unit for receiving a first package including at least one type of content encoded in an HTTP streaming format;
An editing unit that generates and outputs new content based on at least some types of content included in the first package received by the receiving unit;
At least a part of the content included in the first package received by the receiving unit and the new content generated by the editing unit are integrated and output as one second package. The integration department;
A distribution unit that redistributes the second package output from the integration unit;
A program for functioning as a redistribution system.

Images