JP6971059B2 - Redelivery system, redelivery method, and program - Google Patents

Description

The present invention relates to a redelivery system, a redelivery method, and a program.

Technology for delivering (streaming) various contents (video, audio, etc.) in real time using a communication line such as the Internet has become widespread. Distribution of video and audio using the Internet can be realized with relatively low-cost equipment and relatively low information transmission costs, so it is expected that it will be used more and more in the future.

By the way, for example, when the content produced by the first business operator is received and redistributed by the second business operator, the content may be added. In a typical example, when redistributing content for a specific region or a specific language area, it may be desired to add content specific to that region, language, or the like. In such a case, in the conventional technology, the content (for example, video and audio) produced by the first business operator is first transmitted to the second business operator. Then, the second business operator added the content to be added (for example, the commentary voice in a specific language or the commentary voice for a specific area), and then encoded it into a format for Internet distribution.

In such a method using the conventional technology, in order to make it easier for the second company to process and distribute the content, it is necessary to transmit high-quality content using a dedicated line or the like, and the second company needs to transmit the content ( It took a lot of equipment and processes to add the above voice).

Specifically, when the conventional technology is used, the video and audio transmitted from the first operator are decoded by a decoder, the video and audio are separated by a DeMUX (demultiplexer), and then the original audio is restored. The voice to be added was added. Then, the original video and the added audio were encoded using an encoder, and the video and audio were combined again with a MUX (multiplexer) to encode into a format for Internet distribution.

For example, Non-Patent Document 1 describes a system configuration when a broadcaster actually distributes videos of all competitions and all events by live streaming for a large-scale sporting event. According to this document, IPVandA's SD quality video resources were transmitted from the center of the local city (Brazil) where the event was held to the broadcasting center in Tokyo using an international line. The SD image quality video is based on a bit rate of about 2.5 Mbps. Then, at the broadcasting center, the IPVandA video was recoded into a lower bit rate video and distributed via the Internet. In addition, for the video of some competitions, a simple audio booth was constructed in the above broadcasting center, and the commentary and live audio unique to online distribution were added and distributed.

Kenji Shimanishi, Mune Endo, Yuki Kokubo, Shinya Orishita, Shunichi Sakai, Aya Maeda, "Rio de Janeiro Olympic Digital Content Production", Broadcasting Technology, November 2016, p. 104-106.

In the prior art, when adding and redistributing content, it is necessary to reproduce at least a part of the original content. In addition, it is necessary to redistribute the original content and the added content at the same timing.
Therefore, it is necessary to first play the original content in the state of a baseband signal (uncompressed signal) and add the content. Therefore, in order to transmit the original content, it is necessary to transmit a signal including a baseband signal (uncompressed signal) or high-resolution video encoded at a high bit rate, that is, a wideband and stable line, that is, It required a high-cost communication line. Further, in order to configure such a system, a multi-stage process is required, that is, a lot of expensive equipment and the like are required. In particular, it has been difficult in terms of cost to enable simultaneous distribution of content from a plurality of bases using conventional technology.

The present invention has been made based on the above-mentioned problem recognition, and when receiving the original content, adding new content, and redistributing them all together, the cost of transmission and the cost of equipment are reduced. It seeks to provide redelivery systems, redelivery methods, and programs that can be lowered.

[1] In order to solve the above problems, in the redistribution system according to one aspect of the present invention, a receiving unit that receives a first package containing at least one type of content encoded in an HTTP streaming format, and the receiving unit. An editorial unit that generates and outputs new content based on at least a part of the contents included in the received first package, and the said unit included in the first package received by the receiving unit. An integrated unit that integrates and outputs at least a part of the content and the new content generated by the editorial unit as one second package, and the second unit that is output from the integrated unit. It is characterized by having a distribution unit for redistributing the package.

[2] Further, in one aspect of the present invention, in the above-mentioned redistribution system, the receiving unit receives the first package including at least one kind of video content and at least one kind of audio content. The editorial unit reproduces the first voice, which is the content of at least one type of voice included in the first package, and at the same time, the first voice and another voice input corresponding to the first voice. The second audio obtained by superimposing the above is generated and output as the new content, and the integrated unit includes the video content and the audio content included in the first package, and the new content. It is characterized in that it is integrated and output so that the timing of reproduction is matched between the two.

[3] Further, in one aspect of the present invention, in the above-mentioned redistribution system, the editorial unit obtains the matching timing information based on the timing information held by the contents included in the first package, the new contents. The integrated unit makes the timing of reproduction consistent based on the timing information held by the content included in the first package and the timing information assigned to the new content. , Characterized by that.

[4] Further, in one aspect of the present invention, in the above-mentioned redistribution system, the integrated unit is based on the similarity between the waveform of the first voice and the waveform of the second voice, and the first voice. By moving one of the content of the first package including the content and the second audio which is the new content in the time direction, the playback timing is integrated and output. It is characterized by that.

[5] Further, in one aspect of the present invention, in the redistribution system, the receiving unit receives the first package containing at least one type of audio content, and the editorial unit receives the first package. By performing voice recognition processing of at least one type of voice content included in the above, the content of the subtitle text corresponding to the voice content is generated as the new content, and the integrated unit is included in the voice content. Based on the temporal correspondence between the audio signal and the generated subtitle text, the timing of reproduction of the audio content and the timing of presentation of the subtitle text are integrated and output. It is characterized by.

[6] Further, one aspect of the present invention is a reception process for receiving a first package containing at least one type of content encoded in an HTTP streaming format, and the content included in the first package received in the reception process. An editing process for generating and outputting new content based on at least a part of the contents, at least a part of the contents included in the first package received in the receiving process, and the above. It is characterized by including an integration process of integrating and outputting the new content generated in the editing process as one second package, and a distribution process of redistributing the second package output in the integration process. It is a redelivery method.

[7] Further, in one aspect of the present invention, a computer is provided with a receiving unit that receives a first package containing at least one type of content encoded in an HTTP streaming format, and the first package received by the receiving unit. An editorial unit that generates and outputs new content based on at least a part of the included contents, and at least a part of the contents included in the first package received by the receiving unit. Content and the new content generated by the editorial unit are integrated and output as one second package, and the distribution unit that redistributes the second package output from the integrated unit. It is a program for functioning as a redistribution system equipped with.

According to the present invention, it is possible to add new content to the streaming format content and redistribute it at a low transmission cost and a low device cost.

It is a schematic diagram which shows the schematic functional structure of the redistribution system (redistribution apparatus) by 1st Embodiment of this invention, and the flow of the content data in the system. It is a block diagram which shows the configuration example of the whole system including the redistribution system by the same embodiment. It is a schematic diagram which shows the schematic functional configuration of the redistribution system (redistribution apparatus) by 2nd Embodiment, and the flow of the content data in the system. In the same embodiment, it is a schematic diagram which shows the flow of the content when the redistribution system redistributes the content distributed from the distribution server device. It is a schematic diagram which shows the structural example of the data stream-distributed from the distribution server apparatus side in the same embodiment. It is a schematic diagram which shows the structural example of the data stream-delivered from the re-delivery system in the same embodiment. It is a schematic diagram which shows the structural example of the index file of the uppermost layer used in the same embodiment. It is a schematic diagram which shows the structural example of the index file of the relatively lower layer used in the same embodiment. It is a schematic diagram which shows the schematic functional structure of the redistribution system (redistribution apparatus) by 3rd Embodiment, and the flow of the content data in the system.

[First Embodiment]
FIG. 1 is a schematic diagram showing a schematic functional configuration of a redistribution system (redistribution device) according to the present embodiment and a flow of content data in the system. As shown in the figure, the redistribution system 1 includes a reception unit 120, an editorial unit 140, an integration unit 160, and a distribution unit 180.

The redistribution system 1 receives, for example, content distributed by HTTP streaming or the like from an external distribution server. Note that HTTP is an abbreviation for HyperText Transfer Protocol. The content received by the redistribution system 1 includes a plurality of types of content such as video, audio, and text. Further, the content received by the redistribution system 1 may include, for example, a plurality of video contents, a plurality of audio contents, and the like. Then, the redistribution system 1 generates new content based on at least a part of the received content. Then, the redistribution system 1 redistributes the received original content and the generated new content together as one content package.

The receiving unit 120 receives, for example, a first package containing at least one type of content encoded in the HTTP streaming format. The receiving unit 120 may receive a plurality of types of contents. In the illustrated example, a package containing (m + n) types of content from C (1) to C (m + n) is received. Here, m is an integer of 0 or more, and n is an integer of 1 or more. The receiving unit 120 receives these contents by, for example, HLS. HLS is an abbreviation for "HTTP Live Streaming", and is known as a method (protocol) for streaming and distributing video or the like via the Internet or the like.
The receiving unit 120 passes C (m + n), which is the received content, from C (1) to the integrated unit 160. Further, the receiving unit 120 passes C (m + n) to C (m + n) of the received contents to the editing unit 140.

The editorial unit 140 generates and outputs new content based on at least a part of the contents included in the first package received by the receiving unit 120. More specifically, the editorial unit 140 receives n types of contents from C (m + 1) to C (m + n) among the contents received by the receiving unit 120 from the receiving unit 120. Then, the editorial unit 140 generates new contents related to these contents based on the received contents C (m + 1) to C (m + n). The new content generated by the editorial unit 140 is k types of content from C (m + n + 1) to C (m + n + k). However, k is an integer of 1 or more. There are various relationships between the content received by the editorial unit 140 from C (m + 1) to C (m + n) and the content generated by the editorial unit C (m ++ n + 1) to C (m + n + k), but both are contents. Has a relationship as. Moreover, since both are related to each other, the timing should be adjusted in the reproduction or the like (more generally, the presentation). The editorial unit 140 passes the generated content to the integrated unit 160.

The integration unit 160 integrates and outputs the content included in the first package received by the reception unit 120 and the new content generated by the editorial unit 140 as one second package. The integration unit 160 integrates the contents C (1) to C (m + n) passed from the reception unit 120 and the contents C (m + n + 1) to C (m + n + k) passed from the editorial unit 140. do. The integration unit 160 receives C (1) to C (m + n) in the encoded state, and integrates the contents C (m + n + 1) to C (m + n + k) as they are. Then, the integration unit 160 passes the entire contents as one package to the distribution unit 180. At this time, the integration unit 160 integrates the content passed from the receiving unit 120 and the content passed from the editing unit 140 so that the reproduction timing is consistent.
It should be noted that the integrated unit 160 does not include all of the contents from C (1) to C (m + n) passed from the receiving unit 120, but only a part of them from C (m + n + 1) to C (m + n + k). It may be integrated. In this case, which of C (1) to C (m + n) is to be integrated with C (m + n + 1) to C (m + n + k) is appropriately determined.
That is, the integration unit 160 integrates at least a part of the content included in the first package received by the reception unit 120 and the new content generated by the editorial unit 140 as one second package. And output.

The distribution unit 180 redistributes the content (second package) passed from the integration unit 160.

FIG. 2 is a block diagram showing a configuration example of the entire system including the redistribution system 1. As shown in the figure, this system includes a distribution server device 2, a redistribution system 1, and a client device 3. The redistribution system 1 is connected to the distribution server device 2 and the client device 3 via a communication line such as the Internet. Although only one client device 3 is shown in this figure, a large number of client devices 3 may actually be connected to the redistribution system 1. The point that the redistribution system 1 includes the reception unit 120, the editing unit 140, the integration unit 160, and the distribution unit 180 is as described with reference to FIG.

The distribution server device 2 is a server computer that distributes original contents. The content distributed by the distribution server device 2 is, for example, content composed of video and audio. The distribution server device 2 uses, for example, the above-mentioned HLS for distributing the content.
The client device 3 receives the content (content to be redistributed) transmitted by the redistribution system 1. The client device 3 is realized by using, for example, a personal computer (PC), a smartphone (smartphone), a wristwatch-type information terminal, a glasses-type information terminal, other information devices, and the like. The client device 3 has, for example, the function of a web browser, and the web browser functions as an HTTP client. As a result, the content re-distributed by HLS from the re-distribution system 1 can be viewed.

According to the configuration of the present embodiment, new content related to the content received in the streaming format is added without receiving the content by the baseband signal (uncompressed signal), and then the content is redistributed. It becomes possible. In other words, since the number of processes and equipment can be significantly reduced, it is possible to realize a redistribution system at low cost.

[Second Embodiment]
Next, the second embodiment will be described. The matters already described in the previous embodiment may be omitted below. Here, the matters peculiar to the present embodiment will be mainly described.

FIG. 3 is a schematic diagram showing a schematic functional configuration of a redistribution system (redistribution device) according to the present embodiment and a flow of content data in the system. As shown in the figure, the redistribution system 11 includes a reception unit 220, an editorial unit 240, an integration unit 260, and a distribution unit 280.

The redistribution system 11 receives video and audio content (referred to as "audio A"), generates another audio content (referred to as "audio B") based on the received audio content, and receives the source. The content (including the voice A) and the generated voice content (voice B) are integrated and re-distributed.

The receiving unit 220 receives the first package including at least one type of video content and at least one type of audio content. Specifically, the receiving unit 220 receives streaming video and audio (referred to as “audio A”) content delivered via a communication line such as the Internet. The video and audio A received by the receiving unit 220 are, for example, transmitted from an external distribution server or the like in an encoded state. As an example, it is possible to realize the receiving unit 220 by using a computer device dedicated to receiving.

The editorial unit 240 reproduces the first voice, which is the content of at least one type of voice included in the first package, and the first voice and another voice input corresponding to the first voice. The second voice obtained by superimposing is generated and output as new content. That is, the editing unit 240 receives and reproduces at least the content of the voice A among the contents received from the outside by the receiving unit 220. The editorial unit 240 may also receive and reproduce the video content. Then, the editorial unit 240 mixes the content of the voice A and the voice collected from the microphone or the like connected to the editorial unit 240 in the voice band, encodes it by a predetermined coding method, and newly encodes it. It is output as audio content (referred to as "audio B").
As an example, announcers and commentators give live commentary or commentary while watching the content (video and audio A) played by the editorial unit 240. That is, the announcer and the commentator emit their own voice toward the microphone or the like, and the editorial unit 240 generates the content of the voice B including the voice. In this way, when the decoded first voice is reproduced and the announcer or the commentator utters his / her own voice while listening to the first voice in real time, there is no delay time for processing the voice. Or, the delay time is negligibly small. Therefore, the first voice and the new voice are mixed at an appropriate timing, and the voice B is generated.
The voice B may be created by another method. In that case, when creating voice B, if necessary, perform appropriate processing for timing adjustment so that the timing of the first voice and another voice (voice emitted by announcers and commentators) matches. You may go.
It is also possible to realize the editorial unit 240 by using a computer. As an example, it is also possible to realize the editorial unit 240 by using a personal information device such as a personal computer or a smartphone (smartphone).

The integration unit 260 integrates and outputs the video content and audio content included in the first package and the above-mentioned new content so that the reproduction timing is matched. The integration unit 260 integrates the content (video and audio A) originally received by the reception unit 220 and the content (audio B) passed from the editorial unit 240, and delivers the content (audio B) to the distribution unit 280. When the integration unit 260 integrates these contents, the timings of the content received by the reception unit 220 (video and audio A) and the content passed from the editorial unit 240 (audio B) are mutually matched. Adjust to do. Further, the integration unit 260 adjusts the level between the voice A passed from the receiving unit 220 and the voice B passed from the editing unit 240. The integrated unit 260 receives the video and audio A from the receiving unit 220 in the encoded state. Then, the integration with the voice B is performed as it is.
It should be noted that the integration unit 260 integrates not all of the content (video and audio A) passed from the reception unit 220, but only a part of them with the content (audio B) passed from the editorial unit 240. May be good. In that case, for example, the integration unit 260 may integrate only the video passed from the receiving unit 220 and the audio B passed from the editing unit 240. Further, for example, only the voice A passed from the receiving unit 220 and the voice B passed from the editing unit 240 may be integrated. In any of these cases, the integration unit 260 adjusts the timing so that the content received by the reception unit 220 and the content passed from the editorial unit 240 are mutually matched when integrating the contents. do.
That is, the integration unit 260 integrates at least a part of the content included in the first package received by the reception unit 220 and the new content generated by the editorial unit 240 as one second package. And output.
The timing adjustment and level adjustment processing by the integration unit 260 is automatically performed. Details of how the integration unit 260 adjusts the timing will be described later.

The distribution unit 280 distributes the content output from the integration unit 260. The distribution unit 280 distributes the content via the Internet or the like.

FIG. 4 is a schematic diagram showing the flow of content when the redistribution system redistributes the content distributed from the distribution server device in the present embodiment. In the figure, the receiving unit 220, the editing unit 240, the integrated unit 260, and the distribution unit 280 are devices (or a part of the functions thereof) constituting the redistribution system 11 as shown in FIG. Further, these functions constituting the redistribution system 11, the distribution server device 2 and the client device 3 are each connected to the Internet and can communicate with each other. In addition, a means other than the Internet may be used for communication. Although only one client device 3 is shown in FIG. 4, in reality, a large number of client devices 3 may receive distribution from the distribution unit 280.

As shown in the figure, the distribution server device 2 distributes content including video and audio via the Internet. For the distribution of the content, for example, the above-mentioned HLS is used. The receiving unit 220 receives the above-mentioned contents distributed from the distribution server device 2. The receiving unit 220 passes the received video and audio (audio A) to the integrated unit 260 via the Internet or another line or the like. Further, the receiving unit 220 passes at least the audio A (including the video if necessary) of the received content to the editorial unit 240 via the Internet or another line or the like. The editing unit 240 generates audio B, which is audio content different from audio A, based on the content received from the reception unit 220. The voice A may be mixed in the voice B. In a typical application example, the voice A is a live broadcast voice from the site where an event or the like is performed, and the voice B is an utterance matched to the voice A by an announcer or a commentator using the editorial unit 240. Is a mixture of. The editorial unit 240 refers to the timing information included in the file having the voice A, adds the timing information to the voice B, encodes it, and outputs it as a file. Here, the timing information is, for example, PTS (presentation time stamp). Further, the editorial unit 240 generates metadata (data such as a file name associated between the voice A and the voice B) for matching the reproduction timing between the voice A and the voice B. Then, the editorial unit 240 passes the audio B to the integrated unit 260 as new audio content. When the audio content (audio B) is passed to the integrated unit 260, the editorial unit 240 may transmit it via the Internet or may transmit it via another line or the like.

The integration unit 260 integrates the content received from the reception unit 220 and the content received from the editorial unit 240 based on the metadata (data such as the correspondence of file names) for matching the reproduction timing. One of the important processes performed by the integration unit 260 is to set the timing between the content from the receiving unit 220 side and the content from the editing unit 240 side based on the above metadata (correspondence of file names, etc.). It is to match. That is, the content of the audio B generated by the editorial unit 240 is synchronized with the video and audio (audio A) from the receiving unit 220 side by synchronizing the contents with the integrated unit 260. It will be possible to deliver. The integration unit 260 includes a buffer area for accumulating the content in consideration of the case where the arrival time of the content from the reception unit 220 side and the content from the editing unit 240 side to be matched in timing do not match. The integration unit 260 passes the integrated content to the distribution unit 280. Then, the distribution unit 280 distributes the entire content passed from the integration unit 260 via the Internet. For the distribution of the content, for example, the above-mentioned HLS is used. The client device 3 receives the content redistributed from the distribution unit 280, decodes it, and reproduces it. The client device 3 may reproduce the video content and appropriately reproduce the audio content of either the audio A or the audio B.

Next, the format and the like of the data to be delivered in this embodiment will be described.
FIG. 5 is a schematic diagram showing a configuration example of streaming distribution data received by the receiving unit 220 in the present embodiment. As shown in the figure, the data distributed from the distribution server device 2 side has a hierarchical structure. In the figure, the leftmost layer represents the highest level, the middle level represents the middle level, and the rightmost side represents the lowest level. At the highest level, one index file exists, and its file name is "IndexFile.m3u8". This index file "IndexFile.m3u8" holds information (file name, path name, etc.) of the locations of the other three types of index files in the lower layer (middle layer). These three types of index files are "Alternate-LowIndex", "Alternate-MidIndex", and "Alternate-HiIndex". These three types of index files can be appropriately used according to the available communication bandwidth. For example, the user on the client device side who receives the distribution can specify one of three types of low bandwidth, medium bandwidth, and high bandwidth. Each of "Alternate-LowIndex", "Alternate-MidIndex", and "Alternate-HiIndex" holds a list of location information of a moving image file for each predetermined time length (for example, 6 seconds). As an example, the index file "Alternate-LowIndex" holds information on the locations of four video files, "Low_01.ts", "Low_02.ts", "Low_03.ts", and "Low_04.ts". doing. In addition, "Low_01.ts", "Low_02.ts", "Low_03.ts", and "Low_04.ts" are moving image files to be sequentially played. The index file "Alternate-LowIndex" is not limited to four, and can have the location information of any number of moving image files. Here, the index file "Alternate-LowIndex" has been described as an example, but each of "Alternate-MidIndex" and "Alternate-HiIndex" also retains the location information of the moving image file according to the bandwidth.

In the case of the data structure shown in FIG. 5, the audio (audio A) is included in each moving image file (Low_01.ts, Mid_01.ts, Hi_01.ts, etc.).
On the other hand, the voice (voice A) may be distributed as an independent file from the distribution server device 2 side, and the receiving unit 220 may also receive the voice file. In this case, the audio is divided into appropriate lengths and delivered as a plurality of files over time. Also, those audio files are indexed by the same index file that indexes the video files.

FIG. 6 is a schematic diagram showing a configuration example of streaming distribution data output by the integrated unit 260 and distributed by the distribution unit 280 in the present embodiment. As shown in the figure, the data distributed by the integration unit 280 is also configured in a hierarchical structure. Similar to the data structure described with reference to FIG. 5, the leftmost layer represents the highest layer, the middle layer represents the middle layer, and the rightmost layer represents the lowest layer. At the highest level, one index file exists, and its file name is "IndexFile.m3u8". This index file "IndexFile.m3u8" holds information (file name, path name, etc.) of the locations of another five types of index files in the lower layer (middle layer). These five types of index files are "Alternate-LowIndex", "Alternate-MidIndex", "Alternate-HiIndex", "mixed", and "original".

Of these, three types, "Alternate-LowIndex", "Alternate-MidIndex", and "Alternate-HiIndex", are indexes related to moving image files, similar to the data structure described with reference to FIG. The lower moving image files of these three types of index files have the same data structure as described with reference to FIG.

Further, among the above five types of index files in the middle layer, two types, "mixed" and "original", respectively, index audio files. Each of "mixed" and "original" holds a list of the location information of the audio file for each predetermined time length (for example, 6 seconds). As an example, the index file "mixed" holds information on the locations of four audio files, "mixed_01.ts", "mixed_02.ts", "mixed_03.ts", and "mixed_04.ts". There is. In addition, "mixed_01.ts", "mixed_02.ts", "mixed_03.ts", and "mixed_04.ts" are audio files to be sequentially reproduced. The index file "mixed" is not limited to four, and can have the location information of any number of audio files. Just like "mixed", "original" holds a list of the location information of another audio file for each predetermined time length (for example, 6 seconds). That is, "original" refers to information on the locations of four audio files to be sequentially reproduced, "original_01.ts", "original_02.ts", "original_03.ts", and "original_04.ts". Hold.

The audio file (mixed_01.ts, etc.) indexed by the index file "mixed" includes the audio (audio B) output by the editorial unit 240. Further, the voice file (original_01.ts or the like) indexed by the index file "original" includes the original voice (voice A) received by the receiving unit 220 from the distribution server device 2 side.

When an independent file of audio A is distributed from the original distribution server device 2, the integration unit 260 may output the file as it is as an audio file indexed by "original".
When the audio A distributed from the original distribution server device 2 exists only in the video files to be distributed, the integration unit 260 extracts audio from those video files to generate an audio file. Then, the integration unit 260 may output the generated audio file as an audio file indexed by "original".

FIG. 7 is a schematic diagram showing a configuration example of the index file used in this embodiment. The file exemplified here is the top-level index file in the hierarchical structure. The file name of this index file is "playlist.m3u8". As shown in the figure, the index file "playlist.m3u8" is an extended M3U format file, and contains text representing index information inside the index file "playlist.m3u8". In FIG. 7, line numbers corresponding to each line of the text are added for convenience. Hereinafter, the contents of the index file "playlist.m3u8" will be described.

The first line is a header indicating that this file is an extended M3U format file.
The second and third lines hold information about the audio content. The second and third lines both contain the description "TYPE = AUDIO", which indicates that the second and third lines are indexes of audio content, respectively. Further, the second line and the third line both include the description "GROUP-ID =" audio "", which is the identification information that both the second line and the third line are "audio". Indicates that it belongs to the group identified by.
Of these, the second line includes the description "NAME =" mixed "", which is a mixed voice, that is, a voice to which an additional voice is added in the editorial unit 240 (that is, voice B). ). The second line also includes the description "DEFAULT = YES", which indicates that it is the default voice. In addition, the second line holds the location information of the lower index file related to the voice. "URI =" mixed / playlist. The description "m3u8" corresponds to the location information.
On the other hand, the third line contains the description "NAME =" original "", which indicates that it is the original audio before being mixed. That is, it indicates that the voice (voice A) is received by the receiving unit 220 without the additional voice added. Further, the third line includes the description "DEFAULT = NO", which indicates that it is not the default voice. In addition, the third line holds the location information of the lower index file related to the voice. "URI =" original / playlist. The description "m3u8" corresponds to the location information.

The fourth line is information indicating that the segment of the content is an independent segment. That is, it means that information from other segments is not required to decode the content of the segment.

The fifth to sixteenth lines include index information of six types of video files.
The fifth and sixth lines hold the information of the index of the first video. In the first video, the bandwidth (BANDWIDTH) and the average bandwidth (AVERAGE-BANDWIDTH) are both "545600" (unit: bits per second). The codecs for decoding this video stream are "avc1.66.30" and "mp4a.40.2". The resolution of this video is "480x270". The index file of this video is "stream1 / playlist.m3u8".
The seventh line and the eighth line hold the information of the index of the second video. In the second video, the bandwidth (BANDWIDTH) and the average bandwidth (AVERAGE-BANDWIDTH) are both "765600" (bits per second). The codecs for decoding this video stream are "avc1.66.30" and "mp4a.40.2". The resolution of this video is "640x360". The index file of this video is "stream2 / playlist.m3u8".
The 9th and 10th lines hold the information of the index of the third video. In the third video, the bandwidth (BANDWIDTH) and the average bandwidth (AVERAGE-BANDWIDTH) are both "1425600" (bits per second). The codecs for decoding this video stream are "avc1.42c01f" and "mp4a.40.2". The resolution of this video is "640x360". The index file of this video is "stream3 / playlist.m3u8".

The eleventh line and the twelfth line hold the information of the index of the fourth video. In the fourth video, the bandwidth (BANDWIDTH) and the average bandwidth (AVERAGE-BANDWIDTH) are both "3955600" (bits per second). The codecs for decoding this video stream are "avc1.4d401f" and "mp4a.40.2". The resolution of this video is "960x540". The index file of this video is "stream4 / playlist.m3u8".
The thirteenth line and the fourteenth line hold the information of the index of the fifth video. In the fifth video, the bandwidth (BANDWIDTH) and the average bandwidth (AVERAGE-BANDWIDTH) are both "5640800" (bits per second). The codecs for decoding this video stream are "avc1.4d401f" and "mp4a.40.2". The resolution of this video is "1280x720". The index file of this video is "stream5 / playlist.m3u8".
The 15th and 16th lines hold the information of the index of the sixth video. The sixth video has both a bandwidth (BANDWIDTH) and an average bandwidth (AVERAGE-BANDWIDTH) of "7290800" (bits per second). The codecs for decoding this video stream are "avc1.4d401f" and "mp4a.40.2". The resolution of this video is "1280x720". The index file of this video is "stream6 / playlist.m3u8".

It should be noted that the frame rate (FRAME-RATE) is common to the above first to sixth images.
) Is defined as "30.000". Further, all the images from the first image to the sixth image include the description "AUDIO =" audio "". This means that the audio content associated with each video is identified by the group ID "audio". That is, the audio content associated with each video is defined in the second or third line.

FIG. 8 is a schematic diagram showing an example of an index file used in this embodiment. The file shown here is a lower index file referenced from the highest index file shown in FIG. 7. The file name of this index file is "mixed / playlist.m3u8". In the description of the second line of the top-level index file shown in FIG. 7, "URI =" mixed / playlist. The description "m3u8" "indicates the location of the file in FIG. Here, "mixed" is a directory name, and this directory is a directory for storing files for mixed voice (voice B). That is, this index file "mixed / playlist.m3u8" holds the index information regarding the mixed voice. This index file "mixed / playlist.m3u8" is also an extended M3U format file. Also in FIG. 8, line numbers corresponding to each line of the text are assigned. Hereinafter, the contents of the index file "mixed / playlist.m3u8" will be described.

The first line is a header indicating that this file is an extended M3U format file.
The second line is the version information of the file format. Specifically, it indicates that the version of this file format is "3".
The third line "# EXT-X-TARGETDURATION" indicates the estimated time length of the media file to be added next. In the example of this data, the predicted time length is 6 seconds.
The fourth line "# EXT-X-MEDIA-SEQUENCE" represents the sequence number of the first media file included in this index file. In this data example, the first sequence number is "417554" (this number is included in the file name of the file specified in the eighth line).
The description of "# EXT-X-DISCONTINUITY-SEQUENCE" on the fifth line will be omitted.

From the 6th line to the 35th line, a set having a group of 3 lines is repeated 10 times (30 lines in total). The first line in each set associates the media file with the date / time. The second line represents the length of the media segment in seconds. The third line is information for referring to the media file itself.

Here, as an example, the set from the 6th line to the 8th line will be described.
The sixth line "# EXT-X-PROGRAM-DATE-TIME" associates the referenced media file with the date and time. In this data example, the first media file is the date and time (world) indicated by "2017-05-11T16: 19: 02.866 + 09: 00" (year, month, day, hour, minute, second, and one-thousandth of a second notation). It is associated with the date and time in the time zone 9 hours ahead of standard time).
The 7th line "#EXTINF" represents the length of the media segment corresponding to this set. Specifically, the length is specified to be 6,000 seconds. A title can be specified after the comma following "6.00", but the description of the title is omitted in this data.
The eighth line describes the file name of this set of media files (here, the audio file of mixed audio (audio B)). In this data, specifically, it is "test2_270_417554.ts".

Similarly, in the nine sets following this set, information on the date and time, information on the length of the media segment, and information on the file name of the media file in the media segment are described. Since the specific date and time, the length of the media segment, and the file name are as described in the drawings, the description thereof is omitted here.

As described above, the index file exemplified here holds information for 10 segments of the mixed voice file. The length of each segment is 6 seconds, and the total length of 10 segments is 60 seconds.

The integration unit 260 generates and outputs the index file illustrated in FIG. 7 above. That is, the integration unit 260 passes the content including both the voice A (NAME = "original") and the voice B (NAME = "mixed") to the distribution unit 280. The distribution unit 280 distributes the content thus integrated by the integration unit 260 to the client device 3.

Next, the integration unit 260 will explain in detail a method of matching the timing of the (audio) audio B generated by the editorial unit 240 with the timing of the video and audio (audio A) delivered from the reception unit 220. In the method of this embodiment, the presentation time information included in the file is used. That is, the video and audio (audio A) files received by the receiving unit 220 include reproduction timing information (PTS, presentation time stamp) and reproduction time length information. When HLS is used, the receiving unit 220 can acquire timing information (PTS) from a TS (Transport Stream) file including video / audio data. In addition, information on the length of the playback time can be obtained from the description of "#EXTINF" in the index file (M3U8 file) distributed from the distribution server device 2. The editorial unit 240 generates the voice B (mixed voice) while playing the original voice A, and at that time, the timing information and the information of the length of the playback time included in the file of the voice A are used as they are. Embed in B. For example, the timing information (PTS-1) at the time when the voice input is started is acquired, and when the voice B is generated, the timing information at the head of the output stream is output so as to be the PTS-1. Further, when the length of the reproduction time acquired from the M3U8 file is 5 seconds, the output stream is divided into files every 5 seconds and generated. That is, the editorial unit 240 generates and outputs the voice B having the same timing information and information of the length of the reproduction time as the individual files constituting the voice A. Then, the integration unit 260 confirms that the timing information and the playback time length information in the audio A and audio B files are the same, and the video and audio (audio A) passed from the reception unit 220. A new M3U8 file containing video, audio A, and audio B information is generated so that the playback timing of the generated audio B matches, and the HLS contents include the M3U8 file, the video TS file, and the audio A TS file. Deliver the TS file of voice B.

That is, the editorial unit 240 adds matching timing information to the new content to be generated based on the timing information held by the content included in the first package. Further, the integration unit 260 makes the timing of reproduction consistent based on the timing information held by the content included in the first package and the timing information given to the new content.

This makes it possible to redistribute the content in a state where the original content received by the redistribution system 11 and the content added by the redistribution system 11 are in time with each other.

[Second Embodiment: Modification Example]
Next, a modified example of the second embodiment will be described. The basic configuration of this modification is the same as that in the second embodiment, but the part of the method in which the integration unit 260 adjusts the timing between the voice A and the voice B is different from the second embodiment.

In this modification, the integration unit 260 adjusts the timing of the voice A and the voice B as follows. The editorial unit 240 generates a mixed voice (voice B) in which the original voice (voice A) is mixed with the utterance of an announcer or the like. That is, the data of the voice B generated by the editorial unit 240 also includes the information of the voice A. The integration unit 260 acquires voice A (also referred to as "comparison voice") and voice B (also referred to as "comment voice" because a comment by utterance is added). The integrated unit 260 may acquire the voice A directly from the receiving unit 220 or may acquire the voice A from the editing unit 240. The integration unit 260 executes a process for matching the timing of the voice A and the voice B by utilizing the fact that the signal of the voice A is included in the voice B.

As an example, the integration unit 260 performs the following calculation. The audio A and audio B, and _is represented by _S A (t) and _S B (t). S _A (t) and _S B (t) are respectively the signal value at time t (for example, the amplitude of the audio signal). When the voice A and the voice B reach the integrated unit 260, the timings of the two may be different due to the difference in the process route up to that point. Let the deviation amount be Δt (delta t). It is reasonable to assume that the amount of timing deviation between voice A and voice B when the process shown in FIG. 3 or the like is configured as a device is usually less than 1 second at the maximum, and within several seconds at the most even in a special case. be. Then, integrating unit 260, at a predetermined time interval including the time t, it calculates a cross-correlation value of the signal _S A (t) and the signal _S B (t + _Δt). Its cross-correlation value c = corr represented as _{(S A (t), S} B (t + Δt)). Here, corr () is a function for obtaining the cross-correlation value of the two signals. Then, the integration unit 260 obtains a deviation amount Δt that maximizes the cross-correlation value c. Then, the integration unit 260 changes the value of the timing information (PTS) based on the obtained deviation amount Δt, matches the timing of the contents of the voice A and the voice B, and outputs the timing.

When calculating the cross-correlation value by the above function corr (), the signal level of the voice A and the signal level of the voice B may be adjusted as appropriate. Further, the adjustment amount of the signal level here may be automatically obtained based on, for example, machine learning.
Further, although the method of matching the timings of the voice A and the voice B using the cross-correlation value is given as an example, the integration unit 260 may adjust the timings of both by another method. For example, the signal waveforms of the voice A and the voice B may be compared by image processing to obtain the deviation amount Δt at which the degree of coincidence between the two waveforms is the highest.

To summarize, the integration unit 260 is based on the similarity between the waveform of the first voice and the waveform of the second voice, the content of the first package including the content of the first voice, and the new content generated by the editorial unit 240. By moving either one of the second audio as the content in the time direction, the first audio and the second audio are integrated and output so that the timing of reproduction is matched.

[Third Embodiment]
Next, the third embodiment will be described. The matters already explained before the previous embodiment may be omitted below. Here, the matters peculiar to the present embodiment will be mainly described.

FIG. 9 is a schematic diagram showing a schematic functional configuration of a redistribution system (redistribution device) according to the present embodiment and a flow of content data in the system. As shown in the figure, the redistribution system 12 includes a reception unit 320, an editorial unit 340, an integration unit 360, and a distribution unit 380.

The redistribution system 12 receives video and audio content. Then, the redistribution system 12 generates the content of the subtitle text based on the content of the received audio. Then, the redistribution system 12 redistributes the received original content and the generated subtitle text content at the timing of reproduction / presentation.

The receiving unit 320 receives the first package containing at least one type of audio content. Specifically, for example, the receiving unit 320 receives content composed of video and audio from an external distribution server device in a streaming format. The reception unit 320 transmits the received video file and audio file to the integration unit 360. Further, the receiving unit 320 transmits the received audio file to the editing unit 340.

The editorial unit 340 generates the content of the subtitle text corresponding to the content of the voice as new content by performing the voice recognition processing of the content of at least one type of voice included in the first package. The editorial unit 340 has a voice recognition engine inside, and has a function of converting the input voice into a character string. Further, the editorial unit 340 further formats the character string converted from the audio into the format of subtitle text data, and outputs the file as a file that can be displayed as a part of the video in live streaming. At this time, the editorial unit 340 determines the timing for each fragment of the subtitle text data based on the timing information (PTS, presentation time stamp) contained in the original audio file and the relative position of the time in the file. Give information. The editorial unit 340 can output the subtitle text in a data format to which timing information can be added, such as, for example, a timed text markup language (TTML, Timed Text Markup Language). The editorial unit 340 transmits the file of the subtitle text data generated based on the voice to the integrated unit 360.
The existing technology can be applied to the speech recognition engine itself. As a basic process, the speech recognition engine extracts the acoustic characteristics of the input speech, and if necessary, takes into account the characteristics as a language, and produces a statistically probable character string as the text of the speech recognition result. Is output as.

The integration unit 360 ensures that the timing of playing the audio content and the timing of presenting the subtitle text are aligned based on the temporal correspondence between the audio signal contained in the audio content and the generated subtitle text. It is integrated into and output. That is, the integration unit 360 integrates the video and audio content files received from the reception unit 320 and the subtitle text file received from the editorial unit 340 as a package and passes them to the distribution unit 380. More specifically, the integration unit 360 outputs the content data in a state where the timing information between the audio content and the subtitle text content is consistent. The integrated unit 360 receives the video and audio contents from the receiving unit 320 in the encoded state. Then, as it is, the subtitle text is integrated with the content.

The distribution unit 380 distributes a file of the content passed from the integration unit 360 via the Internet or the like. Specifically, the distribution unit 380 distributes video, audio, and subtitle text content.

[Third Embodiment: Modification 1]
Next, a modification 1 of the third embodiment will be described. The basic configuration of this modification is the same as that in the second embodiment, but the characteristic configuration is that the integration unit 360 further performs language translation.

In the first modification of the third embodiment, the editorial unit 340 includes a language translation engine. The language translation engine translates texts in natural language into other languages. For example, the integration unit 360 translates the Japanese text obtained as a result of the voice recognition process into English and outputs English subtitle text data. Alternatively, the editorial unit 340 translates the French text obtained as a result of the voice recognition process into Japanese and outputs the Japanese subtitle text data. The languages of the translation source and the translation destination may be other than those exemplified here. The timing information is also added to the translated subtitle text based on the timing information added to the original voice. The editorial unit 340 transmits the translated subtitle text to the integration unit 360. Subsequent processing is the same as the processing in the above-described form.

[Third Embodiment: Modification 2]
Next, a modification 2 of the third embodiment will be described. The basic configuration of this modification is the same as that in the second embodiment, but the characteristic configuration is that the integration unit 360 further translates into sign language.
The existing technology may be applied to the language translation function itself.

In the second modification of the third embodiment, the editorial unit 340 has a function of translating into sign language. The language translation engine translates the text data obtained as a result of the speech recognition process into a sign language expression. Then, the editorial unit 340 generates and outputs video content corresponding to the translated sign language expression. Sign language is represented as an image using, for example, computer graphics (CG). The timing information is also added to the output sign language video based on the timing information added to the original audio. The editorial unit 340 transmits the generated sign language video data to the integrated unit 360. The integration unit 360 passes the sign language video data to the distribution unit 380 instead of the subtitle text data described in the third embodiment. The distribution unit 380 distributes the original video and audio content and the sign language video generated by the editorial unit 340.

It should be noted that the functions of the redistribution system in the above-described embodiment and its modifications and the functions of some of the devices constituting the redistribution system may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by a computer system and executed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The "computer-readable recording medium" is a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, a DVD-ROM, or a USB memory, or a storage device such as a hard disk built in a computer system. That means. Further, a "computer-readable recording medium" is a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line, and dynamically holds the program for a short period of time. It may also include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or a client in that case. Further, the above-mentioned program may be for realizing a part of the above-mentioned functions, and may be further realized for realizing the above-mentioned functions in combination with a program already recorded in the computer system.

According to any of the above-described embodiments or variations thereof, the redistribution system receives the content distributed by means such as HLS via the Internet or the like. In other words, the redelivery system does not receive content composed of baseband signals (uncompressed signals). The redelivery system then generates another new content based on at least a portion of the received content. Then, the redistribution system integrates the received original content and the generated new content, and then redistributes the content. Redelivery also uses, for example, HLS and the like. As a result, the client device can receive the streaming distribution of the content with the new content added.

Then, according to each embodiment or a modification thereof, it is possible to realize a redistribution system with a minimum number of processes and equipment, and it is possible to suppress the cost of constructing and operating the system. Further, for example, it is possible to realize a service of adding new contents such as audio to a distribution format streaming video and redistributing it basically anywhere as long as there is an environment that can connect to the Internet.

In terms of cost, there is no need for expensive special equipment that processes baseband signals (uncompressed signals), and video can be transmitted via the Internet, so a significant reduction in transmission costs can be expected. Furthermore, since processing can be performed only with a general-purpose computer and software running on it, it is possible to realize a service for adding content from any place where an Internet connection is possible. Further, the timing of the original content (video, audio, etc.) and the content to be added (for example, audio, etc.) can be automatically synchronized in the redistribution system. This makes it possible to easily realize the service by unifying the flow of adding new contents (audio, etc.) in real time to the existing stream video / audio. By applying the configuration described in the above embodiment to services such as content distribution, it becomes possible to provide various and agile services.

The content newly added by the redistribution system is not limited to audio content. In the example already described, text (including so-called subtitle text) and video (sign language video as an example) can be generated and added. It is also possible to generate and add content other than those illustrated here.

Although the embodiment of the present invention has been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment. It may be carried out by a further modification. Further, the design may be performed within a range that does not deviate from the gist of the present invention.

For example, in the above embodiment, HLS is used as a format for delivering video or audio content, but it may be delivered in another format. For example, formats such as MPEG-DASH, HDS, and MS Streaming can also be used.

The present invention can be used, for example, in a business of distributing contents. However, industrial applicability is not limited to the fields exemplified here.

1 Redelivery system (redelivery device)
2 Distribution server device 3 Client device 11,12 Redelivery system (redelivery device)
120 Receiving unit 140 Editing unit 160 Integration unit 180 Distribution unit 220 Reception unit 240 Editing unit 260 Integration unit 280 Distribution unit 320 Reception unit 340 Editing unit 360 Integration unit 380 Distribution unit

Claims (3)

A receiver for receiving a first package comprising at least one content encoded in the HTTP Live Streaming (HLS) format,
An editorial unit that generates and outputs new content based on at least a part of the contents included in the first package received by the receiving unit, and an editorial unit.
At least a part of the contents included in the first package received by the receiving unit and the new contents generated by the editorial unit are integrated and output as one second package. With the integration department
A distribution unit that redistributes the second package output from the integration unit, and a distribution unit.
Equipped with
The receiver receives the first package including at least one type of video content, at least one type of audio content, and an index file.
The editorial unit reproduces the first voice, which is the content of at least one kind of voice included in the first package, and inputs the first voice and the sound collected by a microphone corresponding to the first voice. A second voice obtained by superimposing another voice to be generated is generated and output as the new content.
The integrated unit integrates and outputs the video content and the audio content included in the first package and the new content so that the reproduction timing is matched .
The editing unit, wherein the presentation time stamp is a timing information content holds included in the first package, based on the length of the information written playtime in the index file, included in the first package The presentation time stamp held by the content is added to the new content as matching timing information , and the new content is divided into lengths according to the length of the playback time described in the index file. Content is generated
The integrated unit includes a length of the presentation time stamp and the playback time contents included in the first package held, the length of the timing information added to the new content and and its playback time are the same By confirming that there is, and outputting the content included in the first package, the new content, and a new index file containing information on the new content , the playback timing is made consistent. ,
Re-delivery system that is characterized in that. At least one receiving process of receiving a first package including a content encoded in the HTTP Live Streaming (HLS) format,
An editing process of generating and outputting new content based on at least a part of the contents included in the first package received in the receiving process.
At least a part of the contents included in the first package received in the receiving process and the new contents generated in the editing process are integrated and output as one second package. Integration process,
The distribution process of redistributing the second package output in the integration process,
The A including re-distribution method,
The receiving process receives the first package containing at least one type of video content, at least one type of audio content, and an index file.
In the editing process, the first voice, which is the content of at least one kind of voice included in the first package, is reproduced, and the first voice and the first voice are collected and input by a microphone corresponding to the first voice. A second voice obtained by superimposing another voice to be generated is generated and output as the new content.
The integration process integrates and outputs the video content and the audio content included in the first package and the new content so that the reproduction timing is matched.
The editing process is included in the first package based on the presentation time stamp which is the timing information held by the content included in the first package and the information on the length of the playback time described in the index file. The presentation time stamp held by the content is added to the new content as matching timing information, and the new content is divided into lengths according to the length of the playback time described in the index file. Content is generated
In the integration process, the presentation time stamp and the length of the playback time held by the content included in the first package, the timing information given to the new content, and the length of the playback time are the same. By confirming that, by outputting the content included in the first package, the new content, and the new index file containing the information of the new content, the playback timing is made consistent.
Redelivery method. Computer,
A receiver for receiving a first package comprising at least one content encoded in the HTTP Live Streaming (HLS) format,
An editorial unit that generates and outputs new content based on at least a part of the contents included in the first package received by the receiving unit, and an editorial unit.
At least a part of the contents included in the first package received by the receiving unit and the new contents generated by the editorial unit are integrated and output as one second package. With the integration department
A distribution unit that redistributes the second package output from the integration unit, and a distribution unit.
Equipped with
The receiver receives the first package including at least one type of video content, at least one type of audio content, and an index file.
The editorial unit reproduces the first voice, which is the content of at least one kind of voice included in the first package, and inputs the first voice and the sound collected by a microphone corresponding to the first voice. A second voice obtained by superimposing another voice to be generated is generated and output as the new content.
The integrated unit integrates and outputs the video content and the audio content included in the first package and the new content so that the reproduction timing is matched.
The editorial unit is included in the first package based on the presentation time stamp which is the timing information held by the content included in the first package and the information of the length of the reproduction time described in the index file. The presentation time stamp held by the content is added to the new content as matching timing information, and the new content is divided into lengths according to the length of the playback time described in the index file. Content is generated
In the integrated unit, the presentation time stamp and the length of the playback time held by the content included in the first package, the timing information given to the new content, and the length of the playback time are the same. By confirming that, by outputting the content included in the first package, the new content, and the new index file containing the information of the new content, the playback timing is made consistent.
A program to function as a redelivery system.

Images