JP5487074B2 - Video transmission method and video transmission system - Google Patents

Description

The present invention, video transmission method for transmitting within a predetermined transmission band in accordance with high-resolution panoramic image on fixation region designated by the user, regarding 及 Beauty video transmission system.

  As an encoding method for compressing a plurality of camera images together, the international standard encoding method H.264 is used. Multi-view video coding (MVC) is standardized as H.264 Annex H (see Non-Patent Document 1, for example). In MVC, a plurality of encoded video data is compressed and encoded in a multiplexed form. Encoded data of frames having the same time stamp are continuously multiplexed. That is, the encoded data is a stream in which a plurality of video data is multiplexed. Since frames with the same time stamp are continuously multiplexed, the decoder can decode and output MVC encoded data, thereby successively decoding and outputting a plurality of camera images with the same time stamp. it can.

  FIG. 18 is a block diagram showing the configuration of a video transmission system according to the prior art. In the figure, the conventional video transmission system includes a multiplexing device 1, a video transmission server 2, and a video reproduction client 3. In the multiplexing apparatus 1, an encoding unit 101 is an MVC encoding unit, and encodes and multiplexes M input streams at the same fixed encoding rate to create multiplexed data. The data storage unit 102 stores multiplexed data including the M streams.

  In the video transmission server 2, the data transmission unit 103 extracts N (≦ M) streams from the M multiplexed streams stored in the data storage unit 102 and sends them to the video reproduction client 3. Here, in order to transmit N streams, a transmission rate corresponding to N × fixed coding rate is required.

  In the video reproduction client 3, the data receiving unit 104 receives N streams. The decoding unit 105 decodes N streams. The video display unit 106 displays a partial video.

  Here, a video composed of M streams (whole stream) is referred to as “whole video”, a video composed of N (≦ M) streams (partial streams) is referred to as “partial video”, A video composed of streams is defined as “unit video”. That is, the whole video is composed of M unit videos, and the partial video is composed of N unit videos.

Hideaki Kimata, Shinya Shimizu, Yutaka Kunita, Megumi Isogai, and Yoshimitsu Ohtani, "Panorama video coding for user-driven interactive video application," IEEE International Symposium on Consumer Electronics (ISCE) 2009, 2009.

  The above-described prior art has a problem that high-resolution panoramic images exceeding the HD size cannot be transmitted within a limited transmission band such as a real network. Specifically, when transmission rate <N × fixed coding rate, there is a problem in that a partial video composed of N partial streams cannot be transmitted.

  Further, when the encoding rate is lowered too much so that the transmission rate ≧ N × fixed encoding rate, there is a problem that the image quality is deteriorated.

  Furthermore, even if a partial video composed of N partial streams can be obtained with sufficient quality, since it is a partial video transmission, a transmission request following a sudden change operation of the user's gaze area and reception of the partial video are received. There was a problem that I could not. That is, there is a problem that the video cannot be displayed following the user's gaze area operation. Note that the gaze area operation here includes “zoom in (gaze area reduction)”, “zoom out (gaze area expansion)”, “upward movement”, “downward movement”, “leftward movement”, “rightward movement”, etc. Means.

  FIG. 19 is a conceptual diagram illustrating an example of changing a gaze area. As shown in FIG. 19, the video data is converted into a plurality of resolutions, for example, high resolution, medium resolution, and low resolution, and then each of the resolutions includes 32 tiles each having a tile size of 640 × 320 pixels. It is divided into. The number given to each tile is tile division information (tile ID). The high resolution tiles 0, 1, 2, 8, 9, 10, 16, 17, and 18 are the gaze area 1, and 4, 5, 6, 12, 13, 14, 20, 21, and 22 are the gaze area 2. Suppose that 1, 2, 3, 9, 10, 11, 17, 18, 19 are the gaze region 3.

  For example, in FIG. 19, when an abrupt change operation is performed from the gaze area 1 to the gaze area 2, the display is displayed due to a transmission delay until the partial video of the gaze area 2 is requested and received. There was a problem that there was no video.

  Further, when an abrupt change operation is performed from the gaze area 1 to the gaze area 3 → the gaze area 2 in FIG. 19, the double request for the partial video in the gaze area is prohibited (request for the previous partial video). In the case where a new request is made after waiting for reception), when a request for a partial image of the gaze area 3 is made immediately when the position of the gaze area 3 is reached during the change operation, the gaze area 3 There is a problem in that the partial video request and reception processing of the gaze area 2 that the user originally intended is further delayed.

  FIG. 20 is a flowchart for explaining the operation when a sudden change of the gaze area occurs in the video transmission system in which the double request for the partial video in the gaze area is prohibited in the prior art. Specifically, in the prior art method, if there is a change in the overlapping tile due to the movement of the gaze area, the tile image is requested immediately, so a request for a partial video in the gaze area 3 (step S1001), After reception (step S1002), a request for a partial video of the gaze area 2 (step S1003) and reception (step S1004) were made.

The present invention has been made in consideration of such circumstances, and its purpose is to transmit high-resolution panoramic video at a transmission rate or lower without excessively reducing the encoding rate, thereby maximizing video quality. Video that can be displayed at a transmission rate or less even when the user is suddenly operating the gaze area, and that the user can accelerate reception of the target gaze area when moving the gaze area transmission method is to provide a 及 beauty video transmission system.

  In order to solve the above-described problems, the present invention is a video transmission method for encoding and multiplexing input video and transmitting the video in a predetermined transmission band. The input video is resolution-converted, and a plurality of resolution-converted videos are displayed. A first step of dividing the plurality of small regions, a second step of assigning a tile ID to each of the plurality of small regions, and a parameter for specifying a plurality of encoding rates for each stream of the plurality of small regions A third step of encoding and multiplexing with, a fourth step of assigning a data ID to the encoding result for each stream of the plurality of small areas, and a tile assigned to each of the plurality of small areas A fifth step of creating a correspondence table in which an ID is associated with a data ID assigned to an encoding result for each stream of the plurality of small areas; and for each stream of the plurality of small areas, A video transmission method characterized by comprising a sixth step of calculating the evaluation value for granted data ID to Goka results.

  According to the present invention, in the above invention, the seventh step of receiving the correspondence table and the evaluation value, the eighth step of determining a target resolution from a gaze area designated by a user, and the correspondence table A ninth step of creating a tile ID list of a small area corresponding to the gaze area, and a data ID corresponding to a coding result of a stream of each small area included in the tile ID list of the small area And a tenth step of calculating a correction evaluation value for each data ID based on the evaluation value, and the encoding result of the small area stream from the correction evaluation value and the tile ID of the small area An eleventh step of determining a data ID.

  According to the present invention, in the above invention, the eighth step includes a twelfth step of setting different threshold values for the case where the gaze area is enlarged and the case where the gaze area is reduced, and the vertical direction of the gaze area. And a thirteenth step of determining the resolution of interest based on a magnitude relationship between a width or a horizontal width and the threshold value.

  According to the present invention, in the above invention, the eleventh step includes a fourteenth step of setting a transmission bit rate, and subtracting a bit rate based on data unrelated to the video from the transmission bit rate, thereby A fifteenth step of calculating a usable bandwidth and a sixteenth step of determining a data ID for the encoding result of the stream of the small area based on the usable bandwidth of the video.

  According to the present invention, in the above invention, the sixteenth step is a tile ID having a transmission bit rate smaller than the usable bandwidth from the tile ID list of the small area based on the priority based on the correction evaluation value. The eighteenth step of narrowing down candidates, and the selection quality of the tile ID of the small area are determined based on the target resolution in the tile ID candidate and the resolution N lower than the target resolution. And a nineteenth step of determining a data ID for the encoding result of the stream of the small area based on the tile ID of the small area and the selection quality.

  According to the present invention, in the above invention, the seventeenth step is a twentieth step of selecting a tile ID of the corresponding small area from the tile ID list of the small area, and transmission of the lowest quality tile ID of the small area. A twenty-first step for determining whether the bit rate is equal to or lower than the usable bandwidth; and if it is determined that the transmission bit rate is equal to or smaller than the usable bandwidth, the minimum quality of the tile ID of the corresponding small area from the usable bandwidth A 22nd step of subtracting the transmission bit rate, changing the tile ID of the corresponding small area to the tile ID of the next small area and calculating the number of tile ID candidates of the small area after narrowing down; If it is determined that the transmission bit rate is not equal to or lower than the usable bandwidth, the tile I of the small area having a priority equal to or lower than the tile ID of the corresponding small area from the tile ID list of the small area Characterized in that it further comprises a first 23 of the step of deleting.

  In the present invention according to the twenty-fourth aspect, in the eighteenth step, the tile ID having the lowest correction evaluation value in the tile ID list of the small area is determined as the tile ID of the small area to be improved. Step, and when the tile ID of the small area to be improved exists and the bit rate of the increase from the current selection quality to the improvement quality is equal to or lower than the usable bandwidth, the incremented bit from the usable bandwidth A twenty-fifth step of changing the selection quality to the improved quality and updating the correction evaluation value of the tile ID of the small area to be improved; If the tile ID does not exist or the bit rate of the increase from the current selection quality to the improvement quality is not less than the usable bandwidth, the selection quality determination process is terminated. Characterized in that it comprises a first 26 steps of.

  In order to solve the above-described problem, the present invention provides a video transmission apparatus for transmitting a video encoded and multiplexed by a multiplexing apparatus to a video reproduction client in a predetermined transmission band, wherein a plurality of the input videos are transmitted. A resolution converting means for converting the image into a plurality of resolutions, a video dividing means for dividing the video of each resolution converted into a plurality of resolutions by the resolution converting means into a plurality of small regions, and a plurality of divided by the video dividing means. Encoding means for encoding and multiplexing each of the small area streams with a parameter specifying a plurality of encoding rates, and a quality evaluation value for each of the plurality of small area streams encoded by the encoding means The evaluation value calculating means for calculating the image, the small area divided by the video dividing means, and the plurality of small area streams encoded by the encoding means are associated with each other. A correspondence table creating means for creating a correspondence table; a determining means for determining a predetermined number of small area streams to be transmitted based on a gaze area designated by the video reproduction client; and a coding by the encoding means. A first selection unit that selects a predetermined number of small-area streams to be transmitted, determined by the determination unit, from the plurality of converted streams; and the predetermined unit that is selected by the first selection unit The predetermined number of small regions so that the sum of the coding rates of the small number of small regions is within a predetermined transmission band and the minimum quality evaluation value of the predetermined number of small regions is higher. A second selection unit that selects a video stream having a plurality of resolutions from the stream of the area; and a video stream having the plurality of resolutions that is finally selected by the second selection unit. A video transmission apparatus characterized by comprising a transmitting means for transmitting to the image reproduction client the stream.

  In order to solve the above-described problems, the present invention provides a computer of a video transmission apparatus that transmits a video encoded and multiplexed by a multiplexing apparatus to a video reproduction client in a predetermined transmission band. A resolution conversion function for converting the resolution into a plurality of resolutions, a video division function for dividing the video of each resolution converted into the plurality of resolutions into a plurality of small areas, and a plurality of the divided streams of the plurality of small areas respectively. An encoding function that encodes and multiplexes with a parameter that specifies an encoding rate, an evaluation value calculation function that calculates a quality evaluation value for each of the plurality of encoded small region streams, and the divided small regions; A correspondence table creation function for creating a correspondence table in which the encoded streams of a plurality of small areas are associated with each other, based on a gaze area designated by the video reproduction client A determination function for determining a predetermined number of small area streams to be transmitted; a first selection function for selecting the predetermined number of small area streams to be transmitted from the plurality of encoded streams The sum of the encoding rates of the selected predetermined number of small area streams is within a predetermined transmission band, and the minimum quality evaluation value of the predetermined number of small area streams is higher. A second selection function for selecting video streams of a plurality of resolutions from the predetermined number of small area streams; and transmitting the finally selected video streams of the plurality of resolutions to the video playback client. This is a program characterized by realizing a transmission function.

  According to the present invention, in the above-described invention, the gaze area is determined by the user based on the twenty-seventh step of accumulating the encoded streams of the plurality of small areas in a data accumulation unit and the determined data ID. After a predetermined time has elapsed since the designation, a command requesting a stream of a small area corresponding to the gaze area, and the command out of the encoded streams of the plurality of small areas A 29th step of retrieving a stream of a small area corresponding to the gaze area corresponding to the determined data ID, and a stream of the small area corresponding to the gaze area that has been retrieved, And 30th step of transmitting as a typical video stream.

  The present invention is characterized in that, in the above invention, the determination means determines a predetermined number of small area streams to be transmitted after a predetermined time or more has elapsed since the watch area was designated by the video reproduction client. And

  According to the present invention, it is possible to transmit a high-resolution panoramic video at a transmission rate or lower without excessively reducing the encoding rate, to maximize the video quality, and for abrupt operation of the user's gaze area, Video can be displayed at a transmission rate or lower, and reception of the gaze area targeted by the user can be accelerated when the gaze area moves.

It is a block diagram which shows the structure of the video transmission system by embodiment of this invention. It is a conceptual diagram which shows the example of the tile division | segmentation of the video data by the resolution conversion & image | video division | segmentation part 200, and the provision example of tile division information (tile ID). It is a conceptual diagram which shows the example of provision of the data ID for high resolution tiles. It is a conceptual diagram which shows an example of the correspondence table of each tile image | video and image | video encoded data. It is a conceptual diagram which shows the example of the evaluation value for every frame unit (or predetermined frame unit). 3 is a flowchart showing a processing procedure in the multiplexing apparatus 10; 10 is a flowchart showing a processing procedure from designation of a user's gaze area to a data ID determination method in a command conversion unit 211. It is a conceptual diagram for demonstrating the determination method of the attention resolution from a gaze area | region. It is a conceptual diagram which shows the correspondence of tile ID in each resolution. It is a conceptual diagram for demonstrating the determination method of tile ID. It is a flowchart for demonstrating the determination procedure of data ID by correction | amendment evaluation value and tile ID. It is a flowchart for demonstrating the determination procedure of data ID in consideration of the useable band. It is a flowchart for demonstrating the narrowing-down procedure (by resolution) of tile ID candidate by priority. It is a conceptual diagram which shows the example after the change of selection quality is implemented once after the ascending order sort of tile ID which used the correction evaluation value in selection quality as a key. It is a flowchart for demonstrating the selection quality determination procedure (by resolution) of tile ID. In this embodiment, in a video transmission system in which a double request for a partial video in a gaze area is prohibited, an operation for avoiding a reception delay that occurs when a sudden change in the gaze area occurs is described. It is a flowchart. In this embodiment, in a video transmission system in which a double request for a partial video in a gaze area is prohibited, an operation for avoiding a reception delay that occurs when a sudden change in the gaze area occurs is described. It is a flowchart. It is a block diagram which shows the structure of the video transmission system by a prior art. It is a conceptual diagram which shows the example of a change of a gaze area | region. 10 is a flowchart for explaining an operation when a sudden change of a gaze area occurs in a video transmission system in which a double request for a partial video of a gaze area is prohibited in the related art.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

A. Configuration of Embodiment In the present invention, when it is desired to receive N unit videos encoded in a plurality of resolutions including a gaze area within a predetermined transmission band (X), A feature is that the code rates of N regions are determined so that the minimum quality among the quality (higher quality becomes higher as the rate is higher) corresponding to each code rate Y (y1, y2, y3, etc.).

  FIG. 1 is a block diagram showing a configuration of a video transmission system according to an embodiment of the present invention. In the figure, the video transmission system according to the present embodiment includes a multiplexing device 10, a video transmission server 20, and a video reproduction client 30. In the multiplexing apparatus 10, the resolution conversion & video dividing unit 200 converts the video data into a plurality of resolutions, then divides the video data of each resolution into tiles (see, for example, FIG. 2 to be described later), and outputs a small area video Create a tile video consisting of data.

  The encoding unit 201 encodes and multiplexes these tile videos with parameters (for example, bit rate and QP value) that specify a plurality of encoding rates, and creates video data. The data storage unit 202 is created in the course of tile division information (for example, see FIG. 2 to be described later), tile video data before encoding, encoded and multiplexed video data for each tile video, and encoding processing. Tile video local decode video data is stored.

  The evaluation value calculation unit 203 uses the video data stored in the data storage unit 202 to calculate an evaluation value for each frame unit (or a predetermined frame unit), and these evaluation values (for example, FIG. 5 described later). And a correspondence table (for example, see FIG. 4 described later) between each tile video and video encoded data is stored in the correspondence table & evaluation value storage unit 204.

  In the video transmission server 20, the data management unit 205 sends N (≦ M) copies to the data storage unit 202 according to the command (stream request using the data ID) received from the command receiving unit 206 from the video playback client 30. By performing the stream request, N streams are extracted from the M streams stored in the data storage unit 202 and transferred to the data transmission unit 207.

  The data transmission unit 207 transmits N streams to the video reproduction client 30. The command receiving unit 206 receives a command from the video playback client 30. Before transmitting data to the video reproduction client 30, the data transmission unit 207 transmits tile division information (for example, see FIG. 2 described later) to the data reception unit 212. Also, the correspondence table & evaluation value transmission unit 208 takes out the correspondence table and evaluation value information from the evaluation value storage unit 204 and sends them to the evaluation value reception unit 215 of the video reproduction client.

  In the video reproduction client 30, the data receiving unit 212 receives tile division information and N streams. The decoding unit 213 decodes N streams. The video display unit 214 reconstructs and displays the decoded partial video based on the tile division information. The correspondence table & evaluation value receiving unit 215 receives the correspondence table and evaluation value information transmitted from the video transmission server 20 and stores them in the correspondence table & evaluation value storage unit 216. The gaze area designating unit 217 receives a change of the gaze area from the user.

  The tile ID list creation unit 218 determines the tile ID list from the gaze area when the designation of the gaze area of the user is changed, determines the data ID corresponding to the corresponding tile ID from the correspondence table, The evaluation value related to the data ID is extracted from the correspondence table & evaluation value storage unit 216 and sent to the corrected evaluation value calculation unit 210. The corrected evaluation value calculation unit 210 performs correction of the evaluation value related to the corresponding data ID (for example, correction processing such as averaging the evaluation value of each frame unit for each fixed section). This has the effect of avoiding frequent command requests.

  The command conversion unit 211 determines a data ID to be requested to the video transmission server 20 based on the correction evaluation value and the tile ID when the gaze area designation is changed and the correction evaluation value is changed. The command transmission unit 209 transmits a command based on the data ID determined by the command conversion unit 211.

  The data storage unit 202 and the correspondence table & evaluation value storage unit 204 in the multiplexing apparatus 10 are functionally integrated, and the data transmission unit 207 and the correspondence table & evaluation value transmission unit 208 in the video transmission server 20 are functionally integrated. A configuration in which the data reception unit 212 and the correspondence table & evaluation value reception unit 215 in the video reproduction client 30 are functionally integrated is also possible.

  FIG. 2 is a conceptual diagram illustrating an example of tile division of video data by the resolution conversion & video division unit 200 and an example of providing tile division information (tile ID). As shown in FIG. 2, the video data is converted into a plurality of resolutions, for example, high resolution, medium resolution, and low resolution, and then divided into tiles for each resolution of video data. In the example of FIG. 2, high-resolution video data is divided into 32 tiles each having a tile size of 640 × 320 pixels, and medium-resolution video data has a tile size of one having 640 × 320 pixels. Divided into 8 tiles, low-resolution video data is divided into 2 tiles each having a tile size of 640 × 320 pixels. The number given to each tile is tile division information (tile ID). The high resolution tiles 0, 1, 2, 8, 9, 10, 16, 17, and 18 are the gaze area 1, and 4, 5, 6, 12, 13, 14, 20, 21, and 22 are the gaze area 2. Assume that there is.

  FIG. 3 is a conceptual diagram showing an example of data ID assignment for a high resolution tile. For tiles included in the high-resolution video, a data ID is assigned to the encoding result at each bit rate. For example, video encoded data IDs when video encoding is performed at a bit rate of 500 Kbps are 100, 101,. Also, the video encoded data ID when the video is encoded at a bit rate of 1 Mbps is 200, 201,..., 230, 231. Similarly, video encoded data IDs when video encoding is performed at a bit rate of 1.5 Mbps are 300, 301,..., 330, 331.

  FIG. 4 is a conceptual diagram showing an example of a correspondence table between each tile video and video encoded data. The correspondence table associates the tile ID assigned to the high resolution tile shown in FIG. 2 with the data ID at the bit rate shown in FIG. That is, tile IDs 0, 1,..., 31 are associated with data IDs 100, 101,..., 131 at a bit rate of 500 Kbps, and 200, 201,. , 231 are associated with each other, and data IDs 300, 301,..., 331 with a bit rate of 1.5 Mbps are associated with each other.

  FIG. 5 is a conceptual diagram illustrating an example of an evaluation value for each frame unit (or a predetermined frame unit). The SNR (Signal Nose Ratio) of the Y component of the local decoded image when the stream with the tile ID is encoded at 500 kbps, 1 Mbps, and 1.5 Mbps for the high resolution tile is used as the evaluation value.

Next, the video multiplexing / transmission method of this embodiment will be described in detail.
Here, the resolution and the encoding bit rate are set in three stages, input video size (high resolution video size): 5120 × 1280 pixels, medium resolution video size: 2560 × 640 pixels, low resolution video size: 1280 × 320 pixels, tile The case where the size is 640 × 320 pixels and the bit rate is 500 kbps, 1 Mbps, and 1.5 Mbps will be described.

B. Processing of Multiplexing Device 10 FIG. 6 is a flowchart showing a processing procedure in the multiplexing device 10. First, the resolution conversion & video dividing unit 200 converts the resolution of the input video data and then divides the tile (step S1101), and assigns a tile ID to each tile as shown in FIG. 2 (step S1102). Next, the encoding unit 201 encodes each tile at a plurality of bit rates (step S1103), and assigns a data ID to the encoding result at each bit rate (step S1104). For example, for tiles included in a high-resolution video, a data ID is assigned to the encoding result at each bit rate as shown in FIG.

  Subsequently, the evaluation value calculation unit 203 creates a correspondence table between tile IDs and data IDs as shown in FIG. 4 (step S1105), and finally calculates an evaluation value for each data ID (step S1106). For example, when the Y component SNR of a local decoded image created at the time of encoding is used as the evaluation value, it can be expressed as a frame function SNR (f), where f indicates a frame number. This is because the SNR varies from frame to frame. Thereby, the video quality for every frame of each tile can be grasped.

  Here, the SNR (f) of each f-th frame is calculated by the following equation (1).

SNR (f) = 10 * log ₁₀ (MAX ² / MSE (f)) (1)

  The unit is decibel [dB]. MAX is the maximum pixel value that the original image can take. For example, the value of MAX when the bit depth is 8 bits is 255. MSE (f) in Equation (1) is a mean square error at frame number f, and the sum of squares of the difference between the tile-size original image and the local decoded image It is obtained by dividing by the number of pixels of size.

  In the case of an encoding mode (for example, full intra mode) in which a local decoded image cannot be obtained by the encoding unit 201, local decoded image creation (decoding) needs to be performed separately from the encoding procedure. .

C. Processing of Video Playback Client 30 Next, the processing procedure of the video playback client 30 according to the present embodiment will be described.
FIG. 7 is a flowchart illustrating a processing procedure from the designation of the user's gaze area to the data ID determination method in the command conversion unit 211 in the video playback client 30 according to the present embodiment. First, the title ID list creation unit 218 determines the resolution of interest from the designation of the user's gaze area from the gaze area designation unit 217 (step S1201), and the title ID list creation unit 218 creates the tile ID list from the gaze area. In step S1202, the correction evaluation value calculation unit 210 calculates a correction evaluation value for each data ID included in the tile ID list (step S1203). Then, the command conversion unit 211 determines a data ID from the correction evaluation value and the tile ID (step S1204).

  Next, a method for determining the target resolution from the gaze area in step S1201 will be described. FIG. 8 is a conceptual diagram for explaining a method of determining the target resolution from the gaze area. Here, for high-resolution panoramic images, the threshold is not matched between when the gaze area is enlarged (zoom-in) and when the gaze area is reduced (zoom-out). This has the effect of avoiding frequent resolution changes.

  For example, as shown in FIG. 8, when there is an enlargement operation from the gaze area 1 to the gaze area 2, when the lateral width W of the gaze area is equal to or greater than the threshold Th_out1, the resolution of the video data to which the transmission request is made is high. The resolution is changed to medium resolution. If there is a reduction operation from the gaze area 2 to the gaze area 1 following this enlargement operation, the value of the horizontal width W does not cross the threshold Th_in1, so that the change from the medium resolution to the high resolution does not occur. In this way, by using different threshold values when the gaze area is enlarged or reduced, the resolution is frequently changed when the gaze area is frequently enlarged or reduced in order to adjust the position of the user near the threshold Th_out1. There is an effect that does not occur. A similar resolution conversion process may be added to the value of the vertical width H.

  FIG. 9 is a conceptual diagram showing the correspondence between tile IDs at each resolution. In the figure, the high-resolution video size reduced to 1/2 in the vertical and horizontal directions is the medium-resolution video size, and the video image reduced to 1/2 in the vertical and horizontal directions is the low-resolution video size, and each resolution is 640 × 320 pixels. The tiles are divided. Here, the two-stage reduction rate is set to the same rate as 50%, but different reduction rates may be used. Further, although the tile size at each resolution is 640 × 320 pixels, it may be a different tile size.

  With respect to the correspondence relationship between the tile IDs of each resolution by the above division, the medium resolution tile ID = 40 corresponds to the high resolution tile IDs = 0, 1, 8, and 9. The low resolution tile ID = 80 corresponds to the high resolution tile ID = 0, 1, 2, 3, 8, 9, 10, 11, 16, 17, 18, 19, 24, 25, 26, 27. To do. The same relationship holds for other tiles.

  A high-resolution video size (input video size) and a reduced version of the video are prepared on the video transmission server 20 side, and a low-resolution video is requested on the client side as the user's gaze area expands (zooms out). Thus, efficient video transmission suitable for video display becomes possible. That is, as the zoom-out is performed, the panoramic image is reduced and displayed, so that a low-resolution image is sufficient.

  As described above, after determining the resolution in step S1201, the title ID list creation unit 218 creates a tile ID list from the gaze area in step S1202. Here, FIGS. 10A and 10B are conceptual diagrams for explaining a tile ID determination method. For example, in step S1201, the resolution is determined to be high, and as shown in FIG. 10A, the upper left vertex coordinate of the gaze area is (x, y) = (960, 100), and the gaze area is 1920 pixels × 1080 pixels. In this case, the tile ID list is a tile in the shaded portion in FIG. That is, the tile ID list is 1, 2, 3, 4, 9, 10, 11, 12, 17, 18, 19, 20, 25, 26, 27, 28. These are tiles that are transmission request targets.

  Next, in step S1203, the correction evaluation value calculation unit 210 extracts each data ID included in the tile ID list from the correspondence table (see FIG. 4), and the evaluation value (FIG. 4) transmitted from the video transmission server 20. 5), a corrected evaluation value for each data ID is calculated. Note that, as the correction evaluation value here, an average value (average SNR) of a certain section or the like is used. As a result, the evaluation value frequently fluctuates, and there is an effect of avoiding a change in frequent data ID determination in the following steps.

  Next, in step S1204, the command conversion unit 211 determines a data ID from the correction evaluation value and the tile ID. Note that when the corrected evaluation value is not used, the evaluation value SNR (f) varies from frame to frame, and thus step S1204 is processed for each frame.

D. Data ID Determination Processing Next, the detailed procedure of step S1204 performed by the command conversion unit 211 will be described.

  FIG. 11 is a flowchart for explaining the procedure for determining the data ID based on the correction evaluation value and the tile ID. As shown in FIG. 11, when there is information to be transmitted other than the high-resolution panoramic video, the band that can be used for transmission of the high-resolution panoramic video does not become the maximum transmission bit rate. In addition, it is necessary to calculate the usable bandwidth B.

  For example, when transmitting panoramic video with audio, first, the transmission bit rate is set (step S1301), and the transmission bit rate is substituted into B (step S1302). Next, the bit rate such as the transmission protocol header is subtracted from B (step S1303), and the audio bit rate is further subtracted (step S1304). Then, the data ID is determined in consideration of the obtained usable bandwidth B (step S1305). Considering the usable bandwidth B means that the data ID is determined so that the panoramic video can be passed through the remaining usable bandwidth. This has the effect of enabling panoramic video below the transmission rate.

  FIG. 12 is a flowchart for explaining the detailed processing procedure of the above-described step S1305 (procedure for determining the data ID in consideration of the usable bandwidth). Here, the reason why the same processing is performed for the target resolution and the next lower resolution is to transmit two levels of resolution. Furthermore, by expanding the transmission target area of the resolution one level lower than the target resolution, there is a problem that there is no data to be displayed in the video playback client even when the user's sudden operation in the gaze area is performed. Solve. In FIG. 12, the two levels of resolution are one resolution lower than the target resolution. However, the resolution may be N resolutions lower than the target resolution.

  For example, referring to FIG. 2, the high-resolution tile ID list determined by the gaze area 1 is tile ID = (0, 1, 2, 8, 9, 10, 16, 17, 18). The tile ID list of medium resolution is tile ID = (40, 41, 44, 45). In such transmission of tile ID, when the resolution of interest is high resolution, the user's gaze area 1 to gaze area 2 is changed. When there is an abrupt operation, there is a situation where there is no data to be displayed due to a transmission delay. For this reason, if the tile ID list of medium resolution is set to a wide area and tile ID = (40, 41, 42, 43, 44, 45, 46, 47), the entire video is transmitted at medium resolution. Therefore, it is possible to avoid a situation in which there is no video to be displayed even when the user performs a rapid operation.

  In FIG. 12, the command conversion unit 211 first narrows down tile IDs based on priority (step S1401). Details of step S1401 will be described with reference to FIGS.

  FIG. 13 is a flowchart for explaining a tile ID candidate narrowing-down procedure (by resolution) according to priority. FIG. 14 is a conceptual diagram showing an example after the selection quality is changed once after the tile IDs are sorted in ascending order using the correction evaluation value in the selection quality as a key.

  For example, when the processing resolution in step S1501 is high and the tile ID list is as shown in FIG. 10 from the user's gaze area, the evaluation value is obtained from the correspondence table & evaluation value storage unit 216 based on the tile ID list. Extraction and correction evaluation values are calculated, and a table composed of the tile ID, the current selection quality, and the correction evaluation value of the local decoded image at each bit rate as shown in the left table of FIG. 14 is created. In the selection quality column before sorting using the correction evaluation value in selection quality as a key, the correction evaluation value of the local decoded image at the lowest bit rate is set.

  Next, the processing priority is increased from the top with respect to the result of sorting in ascending order using the correction evaluation value for the selected quality as a key. As a result, the corresponding tile IDs are selected in descending order of priority (step S1501). Subsequently, it is determined whether or not the transmission bit rate with the lowest quality of the corresponding tile ID is equal to or lower than the usable bandwidth (step S1502). The lowest quality transmission bit rate is subtracted (step S1503). Next, the tile ID having the next priority is assigned to the tile ID (step S1504), and the process returns to step S1502. When the corresponding tile ID list disappears from the tile ID list, the number of tile ID candidates after narrowing down is calculated (step S1505).

  On the other hand, if the transmission bit rate of the lowest quality of the corresponding tile ID is larger than the usable bandwidth (NO in step S1502), it means that the tile ID cannot be added in the usable bandwidth. A tile ID with a priority equal to or lower than the corresponding tile ID is deleted from the list (S1506). Next, after deleting the tile ID, the number of tile ID candidates after narrowing down is calculated by counting the remaining tile IDs (step S1505).

  Thus, the process of step S1401 in FIG. 12 ends. Subsequently, it is determined whether or not the number of tile ID lists matches the number of tile ID candidates (step S1402). As can be seen from the flowchart shown in FIG. 13, the fact that the tile ID list matches the narrowed-down tile ID candidate means that the determination in step S1502 is all YES for all tile IDs in the tile ID list. This means that transmission is possible with the lowest quality for all tiles of processing resolution.

  On the other hand, when the number of tile ID lists does not match the number of tile ID candidates (NO in step S1402), it is determined whether the target resolution matches the minimum resolution (step S1410). If the target resolution and the minimum resolution do not match (NO in step S1410), the target resolution is changed to the next lower resolution (step S1411), and the process proceeds to step S1412, that is, step S1205 in FIG. To return.

  On the other hand, if the resolution of interest matches the minimum resolution (YES in step S1410), the process advances to step S1403. When the number of tile ID lists matches the number of tile ID candidates (YES in step S1402), or when the number of tile ID lists does not match the number of tile ID candidates but the eye resolution matches the minimum resolution (step In S1410 (YES), it is determined whether or not the target resolution does not match the minimum resolution (S1403).

  If the target resolution does not match the minimum resolution (YES in step S1403), the tile ID candidates are narrowed down by the priority of the next lower resolution (step S1405), and the process proceeds to step S1406. On the other hand, when the target resolution matches the minimum resolution (NO in step S1403), the process of step S1405 is skipped and the process proceeds to step S1406. Detailed processing in step S1406 will be described with reference to FIG.

  FIG. 15 is a flowchart for explaining the tile ID selection quality determination procedure (by resolution). In the tile ID selection quality determination procedure, first, an improvement target tile ID is determined (step S1601). In step S1601, first, as shown in FIG. 14, after the tile IDs are sorted in ascending order using the correction evaluation value in the selection quality as a key, the tile ID having the lowest correction evaluation value is determined as an improvement target. When there are two levels of improvement quality, one level of quality improvement is performed each time the improvement is targeted. For example, the right table of FIG. 14 shows a case where the SNR 128 shows the lowest value, and the ascending sort is performed once using the correction evaluation value of the selection quality as a key, and the selection quality is increased by one step, from SNR 128 to SNR 228. A modified example is shown. After that, after repeating sorting and changing the selection quality a plurality of times, if the SNR 228 shows the lowest quality again, the selection quality is raised again by one step and changed from the SNR 228 to the SNR 328. Raise the bottom.

  Normally, if there is a variation in the video quality of the entire screen, the video quality of the entire screen is pulled to the poor image quality, and the quality of the entire video sequence is considered to be an integral of the quality of each frame, Raising the quality from the lowest quality part is considered to have an efficient quality improvement effect and maximize the video quality.

  Subsequently, it is determined whether or not a tile ID to be improved exists (step S1602). If there is a tile ID to be improved (YES in step S1602), the selected quality is determined from the transmission bit rate RI of improved quality. The transmission bit rate RC is subtracted, and it is determined whether or not the result is equal to or less than the usable bandwidth B (step S1603). If it is equal to or less than the usable bandwidth B (YES in step S1603), (RI-RC) is subtracted from B (step S1604). Subsequently, the selected quality is changed to improved quality (step S1605). Finally, the correction evaluation value of the corresponding tile ID is updated (step S1606).

  On the other hand, if there is no improvement target tile ID (NO in step 1602), there is no improvement target, and the process ends. Further, when the result of subtracting the transmission bit rate RC of the selected quality from the transmission bit rate RI of the improved quality is larger than the usable bandwidth B (NO in step S1603), the usable bandwidth is insufficient. The process ends.

  When the processing in step S1406 in FIG. 12 is completed, it is determined whether or not the target resolution and the minimum resolution do not match (step S1407), and the target resolution and the minimum resolution do not match. In step S1407, the selection quality of the tile ID having the next lower resolution is determined (step S1408). Finally, a data ID is determined from the tile ID and the selected quality (step S1409).

  On the other hand, if the target resolution matches the minimum resolution (NO in step S1407), the process of step S1408 is skipped, and the process proceeds to step S1409 described above. For example, when the processing result as shown in the right table of FIG. 14 is obtained, since tile ID = 28 and selection quality = SNR 228, the data ID is determined to be 228.

  The command conversion unit 211 creates a video data request command using the data ID determined as described above, and transmits a command from the command transmission unit 209.

  As described above, as shown in FIG. 19, when an abrupt change operation is performed from the gaze area 1 to the gaze area 3 to the gaze area 2, the double request for the partial video in the gaze area is prohibited ( In the case of waiting for reception from the previous partial video request and making a new request), when the position of the gaze area 3 is reached in the middle of the change operation, the partial video request for the gaze area 3 is immediately requested. In other words, since it is necessary to wait for the reception process of the partial video in the gaze area 3, the request for the partial video in the gaze area 2 originally intended by the user and the reception process are further delayed. Therefore, in the present embodiment, the method is changed to a method in which a new request is made only when the gaze area moves and stops for N seconds or longer.

  FIG. 16 and FIG. 17 show that in this embodiment, in a video transmission system in which a double request for a partial video in the gaze area is prohibited, a reception delay that occurs when a sudden change in the gaze area occurs is avoided. It is a flowchart for demonstrating the operation | movement for. In this embodiment, when the gaze area starts to move (step S1701), it is determined whether or not the gaze area has stopped for N seconds or longer (step S1702), and only when the gaze area has stopped for N seconds or longer (step S1702). YES), a request for a partial image of the gaze area is made (step S1703). As a result, a request for a partial video in the gaze area 2 is made (step S1801), and a partial video in the gaze area 2 is received (step S1802). In this way, by changing to a method in which a new request is made, a new request is made only at the position after the gaze area movement (gaze area 2).

  According to the above-described embodiment, the high-resolution panoramic image is divided into a plurality of synchronized tile images, each tile image is encoded at a plurality of bit rates, and the bit rate suitable for the transmission rate from the multiplexed video data and The stream is extracted at the resolution, and the resolution one lower than the target resolution is transmitted.

  There are two types of decoding methods: a method of simultaneously decoding a target resolution and a resolution one level lower than that, and a method of decoding only the target resolution. In the latter method, when the resolution of interest becomes one lower resolution, only the resolution is decoded.

This allows the following:
1) High-resolution panoramic video can be transmitted at a transmission rate or lower without excessively reducing the encoding rate.
2) Video quality can be maximized below the transmission rate.
3) Video can be displayed at a transmission rate or less even when the user rapidly operates the gaze area.
4) It is possible to speed up the reception of the gaze area that the user intends when moving the gaze area.

DESCRIPTION OF SYMBOLS 10 Multiplexer 20 Video transmission server 30 Video reproduction client 200 Resolution conversion & video division part 201 Coding part 202 Data storage part 203 Evaluation value calculation part 204 Correspondence table & evaluation value storage part 205 Data management part 206 Command reception part 207 Data Transmission unit 208 Correspondence table & evaluation value transmission unit 209 Command transmission unit 210 Correction evaluation value calculation unit 211 Command conversion unit 212 Data reception unit 213 Decoding unit 214 Video display unit 215 Correspondence table & evaluation value reception unit 216 Correspondence table & evaluation value Storage unit 217 Gaze area designation unit 218 Tile ID list creation unit

Claims (8)

A video transmission method in which a first device encodes and multiplexes an input video and transmits it to a second device in a predetermined transmission band,
The first device includes :
A first step of converting the resolution of the input video and dividing the resolution-converted video into a plurality of small areas;
A second step of assigning a tile ID to each of the plurality of small areas;
A third step of encoding and multiplexing each stream of the plurality of small regions with a parameter specifying a plurality of encoding rates;
A fourth step of assigning a data ID to the encoding result for each stream of the plurality of small areas;
A fifth step of creating a correspondence table in which a tile ID assigned to each of the plurality of small areas is associated with a data ID assigned to an encoding result for each stream of the plurality of small areas;
The evaluation value for the encoding result for each stream of the plurality of small areas corresponding to the data ID for each predetermined frame unit, and the stream of the plurality of small areas before the third step encodes and multiplexes them, A sixth step of calculating as a value representing quality from a difference from each stream of the plurality of small regions encoded and multiplexed;
A seventh step of transmitting the correspondence table and the evaluation value;
The second device includes:
A step 7-1 for receiving the correspondence table and the evaluation value;
An eighth step of determining the target resolution from the gaze area designated by the user;
A ninth step of creating a tile ID list of a small area corresponding to the gaze area based on the correspondence table;
A data ID corresponding to the encoding result of each small area stream included in the tile ID list of the small area is extracted from the correspondence table, and an evaluation value for each data ID is averaged for each predetermined section based on the evaluation value. A tenth step of calculating as a corrected evaluation value that is an average value of the fixed interval
The encoding rate is identified from correspondence information between the correction evaluation value and the tile ID of the small area, and the data ID of the encoding result of the stream of the small area corresponding to the encoding rate is determined based on the correspondence table. An eleventh step of determining;
An eleventh step of transmitting a stream request including the data ID determined in the eleventh step;
The first device includes:
A step 11-2 for receiving the stream request transmitted in the step 11-1;
A step 11-3 for transmitting a stream of the data ID included in the stream request;
Including
The eighth step includes
A twelfth step of setting a threshold value for the case where the gaze area is enlarged and the case where the gaze area is reduced to different values;
A thirteenth step of determining the resolution of interest based on the magnitude relationship between the vertical or horizontal width of the gaze area and the threshold ;
A video transmission method comprising: The eleventh step includes
A fourteenth step of setting a transmission bit rate;
A fifteenth step of calculating a usable bandwidth of the video by subtracting a bit rate based on data unrelated to the video from the transmission bit rate;
A sixteenth step of determining a data ID for the encoding result of the small area stream based on the usable bandwidth of the video;
Including
The sixteenth step includes
A seventeenth step of narrowing down tile ID candidates having a transmission bit rate smaller than the usable bandwidth from the tile ID list of the small area based on the priority based on the correction evaluation value;
Based on the evaluation value of the target resolution in the tile ID candidate, sorting is performed for each priority to generate correspondence information between the tile ID candidate and the evaluation value of the target resolution, and the tile of the generated correspondence information An eighteenth step of determining the selection quality of the tile ID of the small area from the evaluation value of the target resolution for each ID;
A nineteenth step of determining a data ID for an encoding result of the small area stream based on the tile ID of the small area and the selection quality;
The video transmission method according to claim 1 , further comprising: The seventeenth step includes
A twentieth step of selecting a tile ID of the corresponding small area from the tile ID list of the small area;
A twenty-first step of determining whether a transmission bit rate of the lowest quality of the tile ID of the small area is equal to or lower than an available band;
When it is determined that the transmission bit rate is equal to or lower than the usable bandwidth, the transmission bit rate of the lowest quality tile ID of the corresponding small area is subtracted from the usable band, and the tile ID of the corresponding small area is assigned a priority. A 22nd step of changing to the tile ID of the next small area and calculating the number of tile ID candidates of the small area after narrowing down;
A 23rd step of deleting a tile ID of a small area having a priority equal to or lower than a tile ID of the corresponding small area from the tile ID list of the small area when it is determined that the transmission bit rate is not equal to or less than an available bandwidth ;
The video transmission method according to claim 2 , further comprising: The eighteenth step includes
A 24th step of determining, as a tile ID of a small area to be improved, a tile ID having the lowest correction evaluation value in the tile ID list of the small area;
When the tile ID of the small area to be improved exists and the bit rate of the increase from the current selection quality to the improvement quality is equal to or less than the usable bandwidth, the bit rate of the increased amount is subtracted from the usable bandwidth. A 25th step of changing the selected quality to the improved quality and updating the correction evaluation value of the tile ID of the small area to be improved;
If the tile ID of the small area to be improved does not exist or the bit rate of the increase from the current selection quality to the improvement quality is not less than or equal to the usable bandwidth, the selection quality determination process ends. And the steps
The video transmission method according to claim 2 , further comprising: The first device includes:
A twenty-seventh step of storing each of the encoded streams of the plurality of small regions in a data storage unit;
A twenty-eighth step of requesting a stream of a small area corresponding to the gaze area after a predetermined time or more has elapsed since the user specified the gaze area based on the determined data ID;
A thirty-ninth step of extracting a stream of a small area corresponding to the gaze area corresponding to the determined data ID requested by the command from the encoded streams of the plurality of small areas; ,
A thirty-step of transmitting the extracted small area stream corresponding to the gaze area as a final video stream;
The video transmission method according to claim 1, further comprising: A video transmission system in which a first device encodes and multiplexes an input video and transmits it to a second device in a predetermined transmission band,
The first device includes:
A first means for converting the resolution of the input video and dividing each of the converted video into a plurality of small areas;
A second means for assigning a tile ID to each of the plurality of small areas;
A third means for encoding and multiplexing each stream of the plurality of small regions with a parameter specifying a plurality of encoding rates;
A fourth means for assigning a data ID to an encoding result for each stream of the plurality of small areas;
A fifth means for creating a correspondence table in which a tile ID assigned to each of the plurality of small areas is associated with a data ID assigned to an encoding result for each stream of the plurality of small areas;
The evaluation value for the data ID for each predetermined frame unit is encoded with each stream of the plurality of small areas before the third means encodes and multiplexes with each stream of the plurality of small areas encoded and multiplexed. A sixth means for calculating a quality value from the difference;
A seventh means for transmitting the correspondence table and the evaluation value;
The second device includes:
7-1 means for receiving the correspondence table and the evaluation value;
An eighth means for determining the target resolution from the gaze area designated by the user;
A ninth means for creating a tile ID list of a small area corresponding to the gaze area based on the correspondence table;
A data ID corresponding to the encoding result of each small area stream included in the tile ID list of the small area is extracted from the correspondence table, and an evaluation value for each data ID is averaged for each predetermined section based on the evaluation value. A tenth means for calculating as a corrected evaluation value, which is an average value of the fixed interval;
The encoding rate is identified from correspondence information between the correction evaluation value and the tile ID of the small area, and the data ID of the encoding result of the stream of the small area corresponding to the encoding rate is determined based on the correspondence table. An eleventh means for determining;
Eleventh means for transmitting a stream request including the data ID determined in the eleventh means;
The first device includes:
Eleventh means for receiving the stream request;
11th means for transmitting a stream of the data ID included in the stream request;
Including
The eighth means includes
A twelfth means for setting different threshold values for the case where the gaze area is enlarged and the case where the gaze area is reduced;
A thirteenth means for determining the target resolution based on the magnitude relationship between the vertical or horizontal width of the gaze area and the threshold ;
A video transmission system comprising:   The eleventh means is
  A fourteenth means for setting a transmission bit rate;
  Fifteenth means for calculating a usable bandwidth of the video by subtracting a bit rate based on data unrelated to the video from the transmission bit rate;
  A sixteenth means for determining a data ID for the encoding result of the stream in the small area based on the usable bandwidth of the video;
  Including
  The sixteenth means is
  Seventeenth means for narrowing down tile ID candidates having a transmission bit rate smaller than the usable bandwidth from the tile ID list of the small area based on the priority based on the correction evaluation value;
  Based on the evaluation value of the target resolution in the tile ID candidate, sorting is performed for each priority to generate correspondence information between the tile ID candidate and the evaluation value of the target resolution, and the tile of the generated correspondence information An eighteenth means for determining a selection quality of the tile ID of the small area from the evaluation value of the target resolution for each ID;
  Nineteenth means for determining a data ID for the encoding result of the stream of the small area based on the tile ID of the small area and the selection quality;
  The video transmission system according to claim 6, further comprising:   A twenty-seventh means for storing the encoded streams of the plurality of small regions in a data storage unit;
  28th means for requesting a command for a stream of a small area corresponding to the gaze area after a predetermined time or more has elapsed since the user specified the gaze area based on the determined data ID;
  29th means for extracting a stream of a small area corresponding to the gaze area corresponding to the determined data ID requested by the command from the encoded streams of the plurality of small areas. ,
  30th means for transmitting the extracted small area stream corresponding to the gaze area as a final video stream;
  The video transmission system according to claim 6, further comprising:

Images