JP2017123503A - Video distribution apparatus, video distribution method and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video distribution apparatus in which the probability, that a high resolution high quality video is interrupted in the area that a user wants to view, is reduced when switching the video to be viewed.SOLUTION: A video distribution apparatus includes: a division unit 102 for generating multiple divided videos by dividing each video, captured at multiple different view points, into multiple areas; and a distribution unit 104 for distributing the divided video of a video at other view point, corresponding to a divided video selected by a user viewing the video at a certain view point out of multiple different view points, out of respective divided videos of the video divided into multiple areas, with higher picture quality than that of other divided videos of a video at other view point. The distribution unit 104 acquires the information of the divided video of a video at other view point, corresponding to a divided video selected by the user on the basis of a relation map associating respective divided videos in each video divided into multiple areas, and distributes the divided video, indicated by the information thus acquired, with a higher picture quality than that of other divided videos of a video at other view point.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for distributing video shot at a plurality of points.

  Nowadays, viewing forms in which viewers select and view a part of video that has been widely photographed are expanding. For example, a service that allows the user to select and view the omnidirectional video shot using a camera that can shoot all directions at a point on a browser or a dedicated application on a browser or a dedicated application, Similarly, there is a form in which a panoramic image created by combining camera images is viewed by a user selecting a favorite position and angle of view (see, for example, Non-Patent Documents 1 and 2). Such a viewing mode can be said to be a video viewing with a high sense of presence because the user can select the position and angle of view he / she wants to see and feel as if he / she is standing on the spot.

  On the other hand, as an attempt to enjoy such a highly realistic video viewing experience at a remote place, research and development is also progressing on video streaming that viewers can enjoy without downloading video in advance (for example, Non-Patent Document 3 and 4). Due to their characteristics, these images shot over a wide range are often produced using multiple cameras and higher-definition cameras than usual, and have a resolution higher than that of commonly distributed images. There are often. However, in this research, the video was divided into tiled areas in advance, and only the tiled areas that the user was interested in were streamed at high resolution and high quality, so that even with limited network bandwidth, it had a large resolution. There is an advantage that video can be distributed. In addition, the video that covers the entire captured area other than the tiled area that the user is paying attention to is distributed simultaneously with low resolution and low quality, so that the video is also displayed when the part that the user pays attention to is changed. There is also an advantage that it is not lacking (falls in a state where there is no image).

“YouTube (registered trademark), 360-degree panorama video is supported. The Android app can change the field of view with the movement of the terminal”, [online], March 14, 2015, [Search December 15, 2015], Internet < URL: http://www.itmedia.co.jp/news/articles/1503/14/news016.html> Aditya Mavlankar and Bernd Girod, “Spatial-Random-Access-Enabled Video Coding for Interactive Virtual Pan / Tilt / Zoom Functionality”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 21, no. 5, pp. 577-588 , 2011. Daisuke Ochi, Shinnosuke Iwaki, Yutaka Kunita, Junichi Hirose, Kensaku Fujii, Akira Kojima, “HMD Viewing Spherical Video Streaming System”, ACM Multimedia 2014, 2014. Hideaki Kimata, Daisuke Ochi, Akio Kameda, Hajime Noto, Katsuhiko Fukazawa, Akira Kojima, “Mobile and Multi-device Interactive Panorama Video Distribution System”, IEEE GCCE 2012, 2012.

  However, in the above-described conventional research, when the user reselects the area that the user wants to view next, the user must wait until the high-resolution and high-quality video in that area is distributed to the viewing client. Furthermore, this problem is particularly noticeable when the user can freely switch and watch the video in an environment where video shooting is performed at multiple points. For example, when shooting the same subject from a different angle, when the user switches to a video shot at a different point, the subject that has been viewed with high resolution and high quality will continue to have the same quality. You will often want to see it. However, with the conventional technology, it is not always possible to view with high resolution and high quality, and even if the subject that has been viewed so far can be identified by the system, the subject is photographed after switching the viewing point. You have to wait until the high resolution and high quality video of the area is delivered. For this reason, there is a problem that when the user switches the video to be viewed, the high-resolution and high-quality video is interrupted.

  In view of the above circumstances, an object of the present invention is to provide a technique capable of reducing a probability that a high-resolution and high-quality video in a region that a user wants to watch is interrupted when the video to be viewed is switched.

  One aspect of the present invention includes a dividing unit that generates a plurality of divided videos by dividing each video shot at a plurality of different viewpoints into a plurality of regions, and each divided video of the video divided into the plurality of regions. Among them, the divided video of the video of another viewpoint corresponding to the divided video selected by the user who views the video from one of the different viewpoints is higher in image quality than the other divided video of the video of the different viewpoint. And the distribution unit selected by the user selected based on a related map in which each divided video is associated with each other in each of the videos divided into the plurality of regions. A video distribution device that acquires information on a divided video of a video of another viewpoint that has a corresponding relationship with the video, and distributes the divided video indicated by the acquired information with higher image quality than other divided video of the video of the different viewpoint is there.

  One aspect of the present invention is the video distribution device described above, wherein the distribution unit is a divided video that the user is viewing in the video based on a viewing log related to the video that the user has viewed in the past. On the other hand, a selection is made based on a related map in which a plurality of divided videos up to a predetermined order based on the number of times of viewing are divided among the divided videos viewed by the user in the video of another viewpoint related to the video. Information on the divided video of the video of the different viewpoint that is associated with the divided video selected by the user is obtained, and the divided video indicated by the acquired information is more than the other divided video of the video of the different viewpoint Deliver in high quality.

  One aspect of the present invention is the video distribution device described above, in which each video captured from a plurality of different viewpoints is divided into a plurality of regions, a division unit that generates a plurality of divided videos, and a plurality of regions Of the divided videos of the divided videos, the divided video of the video of another viewpoint corresponding to the divided video selected by the user who views the video at a certain viewpoint among the plurality of different viewpoints is displayed. A distribution unit that distributes the image with higher image quality than other divided videos, and the distribution unit includes a plurality of videos captured from different viewpoints based on the divided videos in the video that the user is viewing. Among the divided videos, a divided video whose corresponding point with the divided video is equal to or greater than a predetermined threshold is selected, and the selected number of divided videos are distributed with higher image quality than the other divided videos of the video of the different viewpoint.

  One aspect of the present invention is a division step of generating a plurality of divided videos by dividing each video shot at a plurality of different viewpoints into a plurality of regions, and each divided video of the video divided into the plurality of regions. Among them, the divided video of the video of another viewpoint corresponding to the divided video selected by the user who views the video from one of the different viewpoints is higher in image quality than the other divided video of the video of the different viewpoint. A distribution step of distributing to the plurality of regions, and in the distribution step, selected by the user selected based on a related map in which each divided video in each video divided into the plurality of areas is associated Acquires divided video information of a video of another viewpoint that has a corresponding relationship with the divided video, and distributes the divided video indicated by the acquired information with higher image quality than other divided videos of the video of the different viewpoint It is a video delivery method.

  A division step of generating a plurality of divided videos by dividing each video shot at a plurality of different viewpoints into a plurality of areas, and the plurality of different viewpoints among the divided videos of the videos divided into the plurality of areas A distribution step of distributing the divided video of the video of another viewpoint corresponding to the divided video selected by the user who views the video at a certain viewpoint from the other video with higher image quality than the other divided video of the video of the different viewpoint; And, in the distribution step, a corresponding point with the divided video among a plurality of divided videos of the video shot from a different viewpoint based on the divided video in the video viewed by the user is a predetermined threshold value. This is a video distribution method in which the above-mentioned divided video is selected, and a predetermined number of selected divided videos are distributed with higher image quality than other divided videos of the video of the different viewpoint.

  One aspect of the present invention is a computer program for causing a computer to execute as the video distribution apparatus.

  According to the present invention, it is possible to reduce the probability that a high-resolution and high-quality video in an area that a user wants to watch is interrupted when the video to be viewed is switched.

It is a figure which shows the system configuration | structure of the video delivery system 100 in 1st Embodiment. It is a flowchart which shows the flow of the process of the conventional video delivery. It is a flowchart which shows the flow of the process of the conventional video output. It is a figure for demonstrating the process based on the image | video image | photographed in several different imaging | photography spots (for example, two imaging | photography spots). It is a figure which shows an example in the case of performing manually as a creation method of a related map. It is a figure which shows the specific example of a related map. It is a flowchart which shows the flow of a process of the video delivery apparatus 10 in 1st Embodiment. 4 is a flowchart illustrating a processing flow of the viewing client 20. It is a figure which shows an example of matching of an image feature point.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 1 is a diagram illustrating a system configuration of a video distribution system 100 according to the first embodiment. The video distribution system 100 includes a video distribution device 10 and a viewing client 20. The video distribution device 10 and the viewing client 20 are connected so as to be able to communicate via a network. The network may be a network configured in any way. For example, the network may be configured using the Internet, may be configured using a LAN (Local Area Network), or may be configured using other networks. The video distribution device 10 and the viewing client 20 may be wired communication or wireless communication. In FIG. 1, the video distribution system 100 includes one viewing client 20, but the video distribution system 100 may include a plurality of viewing clients 20.

The video distribution device 10 inputs each video imaged by each imaging device installed at a plurality of points. Here, the input image may be a panoramic image obtained by connecting a plurality of imaging devices, an omnidirectional image captured by an omnidirectional camera, or a high-resolution image. It may be an image taken with a possible camera (for example, 4K camera, 8K camera, etc.), or an image taken at a wide angle with a fisheye lens attached to these cameras. The video distribution device 10 distributes the video desired by the user of the viewing client 20 to the viewing client 20 from the input video. For example, the video distribution apparatus 10 distributes video in which a specific area has high image quality and high resolution, and other areas have low image quality and low resolution. Here, the specific area represents an area in the video specified by the user.
The viewing client 20 inputs the video distributed from the video distribution device 10. The viewing client 20 presents the input video to the user. The viewing client 20 is, for example, a smart phone, a tablet terminal, a personal computer, a head mounted display, or the like.

  Next, specific configurations of the video distribution device 10 and the viewing client 20 will be described. First, the functional configuration of the video distribution device 10 will be described. The video distribution device 10 includes a CPU (Central Processing Unit), a memory, an auxiliary storage device, and the like connected by a bus, and executes a video distribution program. By executing the video distribution program, the video distribution device 10 functions as a device including a video storage unit 101, a division unit 102, a compression unit 103, a distribution unit 104, and a related map creation unit 105. All or some of the functions of the video distribution apparatus 10 may be realized using hardware such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA). . The video distribution program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. Further, the video distribution program may be transmitted / received via a telecommunication line.

The video storage unit 101 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The video storage unit 101 stores each video captured by each imaging device installed at a plurality of points, identification information for identifying the video, and a time stamp in association with each other.
The dividing unit 102 inputs a plurality of videos taken at different points stored in the video storage unit 101. The dividing unit 102 divides each input video into a plurality of areas. Specifically, the dividing unit 102 divides each input video into tiled areas of a (a ≧ 1) × b (b ≧ 1) (where either a or b is 2 or more). To divide. The dividing unit 102 divides each video into tile-shaped areas having the same size. Although there may be overlap in each region, in this description, it is assumed that there is no overlap for simplification. Hereinafter, the divided video is referred to as a divided video. The dividing unit 102 converts the resolution of each input video to a lower resolution than the original one. Hereinafter, a video converted to a resolution lower than that of the original is referred to as a low resolution video. The dividing unit 102 outputs each divided video of the plurality of videos and each low-resolution video to the compression unit 103.

  The compression unit 103 inputs the divided videos of the plurality of videos output from the video storage unit 101 and the plurality of low resolution videos. The compression unit 103 compresses each divided video and each low-resolution video of a plurality of inputted videos. An existing technique may be used as an encoder and encoding method used for compression. For example, H.264 or HEVC (High Efficiency Video Coding) generally used for compression can be used. The compression unit 103 outputs each divided video of the plurality of compressed videos and each low-resolution video to the distribution unit 104. Hereinafter, the compressed divided video and low-resolution video are referred to as post-compression divided video and compressed low-resolution video.

  The distribution unit 104 inputs the respective compressed divided videos of the plurality of videos output from the compression unit 103 and the plurality of post-compression low-resolution videos. The distribution unit 104 requests the related map creation unit 105 to create a related map using each compressed divided video of the plurality of input videos. At this time, the distribution unit 104 outputs a request for creating a related map and each compressed divided video of a plurality of videos used for creating the related map to the related map creating unit 105. The related map is a map in which information indicating a set of divided post-compression video in which the same or related subject is shown in each post-compression divided video of each video taken at different points at a certain time is recorded. It is. In the related map, as information indicating a set of compressed divided videos, the identification information of each video and the identification information of each compressed divided video are recorded in association with the time when the video was shot. .

  In addition, the distribution unit 104 inputs the related map created by the related map creating unit 105 in response to a related map creation request. The distribution unit 104 holds each compressed divided video of a plurality of videos, a plurality of compressed low-resolution videos, and a related map. In addition, the distribution unit 104 views the compressed divided video, the compressed low-resolution video, and the related map of the video requested by the viewing client 20 as the user viewing video (hereinafter referred to as “user viewing video”). Delivered to the client 20. Further, the distribution unit 104 distributes the compressed divided video of the video requested by the viewing client 20 as a separate stream (hereinafter referred to as “another stream video”) and the compressed low-resolution video to the viewing client 20.

  In addition, the storage format of the video held in the distribution unit 104 may be any storage format. For example, all the frame images constituting the video may be held one by one, or a video file for each of several predetermined frames may be held. In addition, a video file for each divided video after compression may be held. Furthermore, it may be held in a format such as HLS (HTTP Live Streaming) or MPEG-DASH, which has been generally used recently.

  The related map creation unit 105 inputs a request for creating a related map output from the distribution unit 104 and each compressed divided video of a plurality of videos used for creating the related map. The related map creating unit 105 creates a related map based on each compressed divided video of a plurality of inputted videos. Specifically, the related map creation unit 105 compares each compressed divided video of a plurality of input videos, selects a related compressed divided video set, and creates a related map indicating the selected set To do. The related map creation unit 105 outputs the created related map to the distribution unit 104. In the first embodiment, the related map creation unit 105 creates a related map based on a group manually selected by a user operating the video distribution device 10. A specific creation method will be described later.

  Next, the functional configuration of the viewing client 20 will be described. The viewing client 20 includes a CPU, a memory, an auxiliary storage device, and the like connected by a bus, and executes a video viewing program. By executing the video viewing program, the viewing client 20 receives the request unit 201, the designated area video receiving unit 202, the separate stream receiving unit 203, the shooting range cover stream receiving unit 204, the related map receiving unit 205, the received video storage unit 206, the verification It functions as an apparatus including the unit 207 and the presentation unit 208. All or some of the functions of the viewing client 20 may be realized using hardware such as an ASIC, PLD, or FPGA. The video viewing program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. Further, the video viewing program may be transmitted / received via a telecommunication line.

When the user performs an input operation for selecting a specific area to be viewed from the video, the request unit 201 requests the video distribution device 10 for the user viewing video. The input operation for selecting a specific area is, for example, a gyro that the user tilts or moves the smart phone, an operation that changes the value of the acceleration sensor, a flick or swipe operation, and the like. Further, the request unit 201 requests the video delivery apparatus 10 for another stream video. Examples of the input mechanism for the user to perform an input operation include a mouse, a keyboard, a touch panel, and a gyro / acceleration sensor.
The designated area video receiving unit 202 receives the compressed divided video of the user viewing video corresponding to the specific area selected by the user distributed from the video distribution device 10. The designated area video reception unit 202 outputs the compressed divided video of the received user viewing video to the received video storage unit 206.

The separate stream receiving unit 203 receives the compressed divided video of the separate stream video distributed from the video distribution device 10 and the compressed low-resolution video. The separate stream receiving unit 203 outputs the compressed divided video of the received separate stream video and the compressed low-resolution video to the received video storage unit 206.
The shooting range cover stream receiving unit 204 receives a compressed low-resolution video that covers the entire shooting range of the user viewing video distributed from the video distribution device 10. The shooting range cover stream receiving unit 204 outputs a compressed low-resolution video of the received user viewing video to the received video storage unit 206.

The related map receiving unit 205 receives the related map distributed from the video distribution device 10. The related map receiving unit 205 holds the received related map. Further, the related map receiving unit 205 refers to the related map in accordance with a request from the request unit 201 and responds with information described in the related map.
The received video storage unit 206 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The received video storage unit 206 includes a compressed divided video of the user viewing video output from the designated area video receiving unit 202, a compressed divided video and a compressed low resolution video of the different stream video output from the separate stream receiving unit 203. And the compressed low-resolution video of the user viewing video output from the shooting range cover stream receiving unit 204 is stored.

The collation unit 207 inputs the video stored in the received video storage unit 206. The collation unit 207 outputs the compressed divided video specified by the user of the input user viewing video and the compressed low-resolution video of the user viewing video to the presentation unit 208.
The presentation unit 208 presents a video based on the compressed divided video output from the matching unit 207 and the low-resolution video after compression of the user viewing video. Specifically, the presentation unit 208 first decodes both the compressed divided video and the compressed low-resolution video of the user viewing video. Thereby, the compressed divided video and the compressed low-resolution video are decoded. It should be noted that existing techniques may be used for decoding as well as encoding. Next, the presentation unit 208 generates a video to be presented to the user by superimposing the divided video on a position corresponding to the divided video on the decoded low-resolution video. Then, the presentation unit 208 presents the generated video to the user by displaying it. In the above description, audio is not particularly described, but generally audio formats that are muxed (multiplexed) into video (stereo, 5.1ch compressed with AAC (Advanced Audio Coding) or the like) are used. It is distributed along with the video and can be viewed with an audio decoder separately prepared on the viewing client 20 side.

  Next, the basic video distribution flow in the present invention will be described in detail with reference to FIGS. 2 and 3 show the flow of processing for video distribution and video output when there is only one conventional shooting point. Also, here, for convenience of explanation, the video input and stored in the video distribution device 10 is a omnidirectional video acquired by an omnidirectional camera (a rectangular video as an equirectangular projection). Suppose that

First, the flow of conventional video distribution processing will be described with reference to FIG.
The dividing unit 102 inputs the video stored in the video storage unit 101 (step S101). For example, the dividing unit 102 inputs each frame 30-1 to 30-t (t is an integer of 2 or more) of the video stored in the video storage unit 101. Next, the dividing unit 102 divides each frame of the input video. Here, as an example, the dividing unit 102 divides each frame 30-1 to 30-t of the input video into six divided videos 31-1 to 31-6. Further, the dividing unit 102 generates a low-resolution video by converting each of the frames 30-1 to 30-t to a resolution lower than the original resolution (Step S102). The dividing unit 102 outputs the divided videos 31-1 to 31-6 for each frame and the low-resolution video 32 for each frame to the compression unit 103.

  Note that the sizes of the divided video and the low-resolution video are not specified here, but the video size that can be encoded and decoded by a general encoder and a decoder installed in the viewing client 20 (for example, the divided video is a general VGA (640 × 468), WVGA (800 x 480), XGA (1024 x 768), HDTV (1280 x 720), full HD (1920 x 1080), and low resolution video at an original video area ratio of 1/4 or 1/3. Converted). A GOP (Group of Pictures) unit that is a set of pictures including at least one I-picture of video compression for each divided video and each low-resolution video (usually fn (1) to fn (15 for a divided video, for example). 15) In the case of a low-resolution video, it is assumed that the video files are saved for each f′n (1) to f′n (15)) (for example, Fn (1) and F′n (1), respectively).

  The compression unit 103 inputs the divided videos 31-1 to 31-6 of each frame output from the dividing unit 102 and the low-resolution video 32 of each frame. The compression unit 103 compresses the input divided videos 31-1 to 31-6 of each frame and the low-resolution video 32 of each frame (step S103). Here, it is assumed that 15 frames are collectively compressed into a video file. Thereby, the compression unit 103 generates a post-compression divided video and a post-compression low-resolution video. The compression unit 103 outputs the generated post-compression divided video and post-compression low-resolution video to the distribution unit 104. The distribution unit 104 inputs the compressed divided video and the compressed low-resolution video output from the compression unit 103. The distribution unit 104 distributes a video corresponding to the user of the viewing client 20 to the viewing client 20 based on the input compressed divided video and the compressed low-resolution video (step S104).

Next, the flow of conventional video output processing will be described with reference to FIG. In the process of FIG. 3, it is assumed that the user of the viewing client 20 is viewing a certain video.
The request unit 201 requests the video distribution apparatus 10 for a video corresponding to a specific area selected from videos at a certain time T (step S201). Thereafter, the designated area video receiving unit 202 receives the compressed divided video 33 corresponding to the specific area selected by the user from the video at time T in the process of step S201 from the video distribution apparatus 10 (step S202). . The designated area video receiving unit 202 outputs the received post-compression divided video 33 to the received video storage unit 206. Further, the shooting range cover stream receiving unit 204 receives the compressed low-resolution video 34 corresponding to the compressed divided video 33 at a certain time T (step S203). The shooting range cover stream receiving unit 204 outputs the received compressed low-resolution video 34 to the received video storage unit 206. The collation unit 207 collates whether or not the received video storage unit 206 has the same time-stamped video (compressed divided video 33 and compressed low-resolution video 34) (step S204). When videos having the same time stamp are prepared, the collation unit 207 outputs the compressed divided video 33 and the compressed low-resolution video 34 having the same time stamp to the presentation unit 208. On the other hand, when videos with the same time stamp are not prepared, the collation unit 207 waits until videos with the same time stamp are prepared. The presentation unit 208 inputs the post-compression divided video 33 and the post-compression low-resolution video 34 output from the collation unit 207. The presentation unit 208 decodes the input post-compression divided video 33 and post-compression low-resolution video 34 by decoding them. The presentation unit 208 generates a video by superimposing the divided video 36 on a corresponding location on the decoded low-resolution video 35. The presentation unit 208 presents the generated video to the user. In FIG. 3, a divided video 36 (in a thick frame) is an area selected by the user and is a high-resolution video. On the other hand, the outside of the divided image 36 is an area not selected by the user, and is a low-resolution image.

Next, a process based on videos shot at a plurality of different shooting points (for example, two shooting points) will be described with reference to FIG.
As shown in FIG. 4, the photographing devices 37-1 and 37-2 are installed at a plurality of different photographing points. Here, it is assumed that the imaging devices 37-1 and 37-2 are omnidirectional cameras. The imaging device 37-1 outputs the captured video 38-1 to the video distribution device 10. In addition, the imaging device 37-2 outputs the captured video 38-2 to the video distribution device 10. Thereby, the video for the two locations is stored in the video storage unit 101 of the video distribution device 10. In addition, when using a plurality of videos shot at a plurality of different shooting points, the user views the divided video 40 in which the subject 39-1 of the video 38-1 shot by the shooting device 37-1 is shown. After that, the video is switched to the video 38-2 captured by the imaging device 37-2 by some other means by the user (for example, a particular keyboard key is not particularly limited here). And

  When the video images of the imaging device 37-1 and the imaging device 37-2 are simply one-to-one and associated with the divided regions at the same position, the user is looking at the subject 39-1. After the video is switched, the subject is not shown in the divided video 41 of the photographing device 37-2 corresponding to the divided video 40 of the photographing device 37-1. Further, when the video of these two points is distributed to the viewing client 20 by the same method as the conventional method, the user switches the viewpoint (for example, from the video 38-1 of the imaging device 37-1 to the video 38-2 of the imaging device 37-2). Immediately after switching, a request for a partial video regarding the viewing area at the point after switching and a request for a low-resolution video are made to the video distribution apparatus 10. Therefore, the user cannot view a video of a new point (for example, video 38-2 of the imaging device 37-2) immediately after switching. This means that for a user who has previously viewed a high-resolution, high-quality segmented video 40 at a point of the photographing apparatus 37-1, the video is temporarily interrupted until a video at a new point is received. As a result, the quality of experience is reduced.

Therefore, in the present embodiment, in the conventional video distribution process, when the viewing point is changed, the location of the divided area that will be viewed next is associated and mapped (related map). The map is downloaded to the viewing client 20 at the start of video viewing, and thereafter received as a separate stream before the next viewing point changes based on the related map. By doing so, the user can continue to view the related subject with high resolution and high quality without interruption of the video even after the viewing point is changed.
In the above example, an example has been described in which a video shot with two omnidirectional cameras placed at two locations is distributed. However, a video shot with omnidirectional cameras placed at three or more locations is distributed. The same processing can also be applied to the case of doing so. Further, unless the user changes the viewing area, the viewing client 20 continuously receives partial videos related to the currently viewed area according to the related map.

A method for creating a related map used to solve the above problem will be specifically described with reference to FIG. Note that FIG. 5 illustrates a case where the map is manually performed as a related map creation method. FIG. 5 is a diagram illustrating an example in which the association map is manually created as a related map creation method.
The related map creation unit 105 displays the compressed divided videos 42-1 to 42-6 and 43-1 to 43-6 of the plurality of videos 38-1 and 38-2 on an output device (not shown). . Here, it is assumed that F ^A ₁ (T) to F ^A ₆ (T) are allocated to the post-compression divided videos 42-1 to 42-6 as identification information of the divided videos. Further, it is assumed that F ^B ₁ (T) to F ^B ₆ (T) are assigned to the divided video images 43-1 to 43-6 after compression as identification information of the divided video images. The user performs an association with the appearance based on each of the displayed post-compression divided videos 42-1 to 42-6 and 43-1 to 43-6. The result of the association is as shown in FIG. Then, the related map creating unit 105 creates a related map of the videos 38-1 and 38-2 based on the result associated with the user. A specific example of the created related map is shown in FIG.

FIG. 6 is a diagram illustrating a specific example of the relation map.
The association map has a plurality of records (hereinafter referred to as “combination records”) representing information on combinations. The combination record has values of time and set information. The time value represents the time when the video was shot. The value of the set information of a certain combination record represents a set of divided videos of a plurality of videos taken at a certain time. In FIG. 6, there are a video ID and a divided video ID as specific examples of the set information. The video ID represents information for identifying a video. The divided video ID represents information for identifying the compressed divided video.

In FIG. 6, the combination record recorded in the second row of the related map has a time value “T−1”, a video ID value “1”, and a divided video ID value “F ^A ₁ (T -1) ", the value of the video ID is" 2 ", and the value of the divided video ID is" F ^B ₁ (T-1) ". That is, the compressed divided video identified by the divided video ID “F ^A ₁ (T-1)” on the video identified by the video ID “1” shot at a different point at time T−1, and the video ID It shows that the divided video ID “F ^B ₁ (T−1)” on the video identified by “2” is associated with the compressed divided video.

FIG. 7 is a flowchart showing the flow of processing of the video distribution apparatus 10 in the first embodiment.
The dividing unit 102 inputs a plurality of videos stored in the video storage unit 101 (step S301). The dividing unit 102 divides each input video. More specifically, the dividing unit 102 divides each frame of each input video into tiles. By this processing, a plurality of divided videos of each frame of each video is generated. The dividing unit 102 outputs a plurality of divided videos of each frame of each generated video to the compression unit 103. Further, the dividing unit 102 converts each frame of each input video into a video having a resolution lower than the original resolution (step S302). By this process, a plurality of low-resolution videos of each frame of each video are generated. The dividing unit 102 outputs the generated plurality of divided videos and the plurality of low-resolution videos to the compression unit 103. The compression unit 103 inputs the plurality of divided videos output from the division unit 102 and the plurality of low resolution videos. The compression unit 103 compresses the plurality of input divided videos and the plurality of low resolution videos (step S303). By this processing, a post-compression divided video and a post-compression low-resolution video are generated. The compression unit 103 outputs the generated plurality of post-compression divided videos of each frame and the compressed low-resolution video of each frame to the distribution unit 104.

  The distribution unit 104 outputs a plurality of post-compression divided videos of each frame output from the compression unit 103 and a compressed low-resolution video of each frame to the related map generation unit 105, and transmits the related map to the related map generation unit 105. Request creation. The related map creating unit 105 creates a related map based on the plurality of post-compression divided videos of each frame and the compressed low-resolution video of each frame output from the distribution unit 104 (step S304). The related map creation unit 105 outputs the created related map to the distribution unit 104. The distribution unit 104 holds the related map output from the related map creation unit 105. In response to a video viewing request from the viewing client 20, the distribution unit 104 distributes the compressed divided video of the user viewing video, the compressed low-resolution video, and the related map to the viewing client 20 (step S305). Thereafter, the distribution unit 104 receives a distribution request for the compressed divided video of the different stream and the compressed low-resolution video from the viewing client 20 (step S306). In response to the received distribution request, the distribution unit 104 distributes the requested compressed divided video and the compressed low-resolution video to the viewing client 20 (step S307).

FIG. 8 is a flowchart showing the flow of processing of the viewing client 20.
The request unit 201 accepts selection of a specific area in the video being viewed from the user (step S401). When a specific area is selected, the request unit 201 determines the specific area selected by the user from the selected result. Then, the request unit 201 requests the video distribution apparatus 10 for a video corresponding to the specific area selected by the user and a low-resolution video at the same time (step S402). At this time, the request unit 201 refers to the related map stored in the related map receiving unit 205 and acquires information on the video area of another viewpoint that has a corresponding relationship with the area selected by the user. Then, the request unit 201 requests the viewing client 20 for a low-resolution video corresponding to the video of another viewpoint indicated by the acquired information and a divided video corresponding to the video area of the different viewpoint.

  The viewing client 20 receives the video distributed from the video distribution device 10 (step S403). Specifically, the designated area video reception unit 202 receives the post-compression divided video corresponding to the area selected by the user distributed from the video distribution apparatus 10. The designated area video receiving unit 202 outputs the received compressed divided video to the received video storage unit 206. The separate stream receiving unit 203 receives the low-resolution video after compression of another viewpoint and the divided video after compression of the video of another viewpoint distributed from the video distribution device 10. The separate stream receiving unit 203 outputs the received low-resolution video after compression of the different viewpoint and the compressed divided video of the video of the different viewpoint to the received video storage unit 206. The shooting range cover stream receiving unit 204 receives a compressed low-resolution video at the same shooting position as the shooting position selected by the user. The shooting range cover stream receiving unit 204 outputs the received low-resolution video to the received video storage unit 206. Thereafter, the collation unit 207 covers the shooting range at the same position as the compressed divided video of the time stamp of each video stored in the received video storage unit 206 and the shooting position selected by the user. The time stamp of the resolution video is collated (step S404).

  The collation unit 207 includes a compressed low-resolution video and a video with a time stamp of the compressed low-resolution video (compressed divided video at the same shooting position, compressed divided video at another viewpoint, and compressed low-resolution video at another viewpoint). ) Is output to the presentation unit 208. The presentation unit 208 decodes the video output from the collation unit 207 by decoding the video. Then, the presentation unit 208 presents a video to the user (step S405). Specifically, the presentation unit 208 presents a video generated by superimposing the divided video at the same shooting position on a corresponding location on the low-resolution video at the shooting position selected by the user. After that, it is assumed that the user has switched to another viewpoint video. In this case, the presentation unit 208 presents the video generated by superimposing the divided video of the different viewpoint on the corresponding location on the low-resolution video of the different viewpoint held (step S406).

According to the video distribution device 10 in the first embodiment configured as described above, the probability that the high-resolution and high-quality video in the area that the user wants to watch is interrupted when the video to be viewed is switched is reduced. It becomes possible. Hereinafter, this effect will be described in detail.
The video distribution device 10 creates a related map by associating each divided video of each video shot at a plurality of different points. The video distribution device 10 distributes the created related map to the viewing client 20 at a predetermined timing such as a timing when the user of the viewing client 20 views the video. Thereafter, when an input for selecting a specific area is made by the user of the viewing client 20, the video distribution apparatus 10 acquires information on the partial video corresponding to the specific area selected by the user from the viewing client 20. . Then, the video distribution device 10 distributes the compressed partial video indicated by the acquired information and the compressed low-resolution video related to the compressed partial video to the viewing client 20 in advance. Thereby, the viewing client 20 stores the video of another viewpoint and the divided video of the different viewpoint related to a specific area in the video currently being viewed. Therefore, even when the user switches from the video currently being viewed to the video of another viewpoint, the viewing client 20 can immediately present the video of the different viewpoint. Therefore, when the video to be viewed is switched, it is possible to reduce the probability that the high-resolution and high-quality video in the region that the user wants to view is interrupted.

  In addition, the video distribution device 10 distributes the divided video of a different viewpoint related to a specific area in the currently viewed video to the viewing client 20 with higher image quality than the other divided video. Therefore, even when the video to be viewed is switched, the subject that the user has focused on in the video that was viewed before switching can be viewed with high image quality even in the video after switching.

<Modification>
Each of the video storage unit 101, the division unit 102, the compression unit 103, the distribution unit 104, and the related map creation unit 105 may be provided with a part or all of the functional units in another device.
A CDN (Contents Delivery Network), various cache servers, and the like may be provided on the path between the video delivery apparatus 10 and the viewing client 20.

(Second Embodiment)
In the first embodiment, the case where the related map is manually created has been described as an example. However, in the second embodiment, the related map is automatically created. Specifically, the related map creation unit 105 automatically generates a related compressed divided video based on the number of correspondences of each image feature point that is one feature amount of the video with respect to each compressed divided video in the same frame. Associate. A specific creation method will be described below.
FIG. 9 shows an example of the association of image feature points. FIG. 9 is a diagram illustrating an example of association of image feature points. As in the first embodiment, FIG. 9 shows one frame at a certain time T of each video imaged by imaging devices (eg, omnidirectional cameras) installed at two locations (eg, location A and location B). (Divided into 6 regions). Further, in FIG. 9, at a point B corresponding to the feature point 45 detected in the area F ^A ₆ (T) (compressed divided video) among the plurality of feature points 45 detected in the frame 44-1. A feature point 46 in the frame 44-2 at the same time that the image was taken is indicated by a line segment. Note that the feature point detection method and the matching method are not limited here, and a general method may be used. For example, SURF (Speeded Up Robust Features), MSER (Maximally Stable Extermal Regions), corner detection, or the like may be used as a feature point detection method. For example, SIFT (Scale-Invariant Feature Transform) etc. may be used for the matching method.

Here, a process of associating a region of a video shot at another point corresponding to one region in a video shot at one point in order to create a related map is shown, and FIG. 9 shows F ^A ₆ ( Assume that an area in the frame 44-2 corresponding to T) is being searched. As shown in FIG. 9, there are four feature points detected in F ^A ₆ (T), three correspond to the feature points in F ^B ₄ (T), and the remaining one feature point is Corresponds to the feature points in F ^B ₅ (T). For example, when a rule is set that sets a region having the largest number of corresponding feature points here, the related map creation unit 105 selects F ^B ₄ (T) as a set of F ^A ₆ (T). Can do. The related map creation unit 105 automatically creates a related map by performing such processing for all regions. The flow of distribution after creating the related map is the same as in the first embodiment, and is therefore omitted.

According to the video delivery device 10 in the second embodiment configured as described above, the same effect as in the first embodiment can be obtained.
In addition, the video distribution apparatus 10 according to the second embodiment can automatically create a related map. Therefore, it is possible to reduce the time required for creating a related map by the user.

<Modification>
The second embodiment may be configured similarly to the first embodiment.
In the second embodiment, a configuration is shown in which a related post-compression divided video is identified based on corresponding points extracted from the video. However, the related map creation unit 105 performs block matching ( Block matching can be applied to the entire compressed divided video, or each divided divided video is further divided into several regions and the result of block matching (relates the compressed divided video with many corresponding blocks). And a related map may be created.

(Third embodiment)
In the third embodiment, when the user has already watched the target video for a predetermined number of times, the related map creation unit 105 specifies the past watched by the user after the point change operation from the video viewing log. A related map is created by estimating a region (divided video) and associating it with the specific region.
In the third embodiment, the video distribution device 10 further includes a user information storage unit. The user information storage unit is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The user information storage unit stores each user's viewing log for each user. The viewing log includes, for example, information such as the video ID of the video viewed by the user, the video ID of the video of another viewpoint related to the video viewed by the user, the number of times of viewing, and the specific area selected by the user after the viewpoint switching operation. Represents.

When the user has watched the target video a predetermined number of times, the related map creation unit 105 selects the target viewed after viewing the target video based on the viewing log stored in the user information storage unit. In a different viewpoint video and a different viewpoint video, for example, a specific area that was viewed most often is identified, a specific area of the identified different viewpoint video, and a specific video of the target video The association map is generated by associating with the area. In addition, the related map creation unit 105 is based on the viewing log, not only the area that the user watched most, but the area that the user watched next (second), and the next (third). A plurality of regions up to a predetermined order based on the number of times of viewing are identified, such as a region that was viewed at the same time, and each of the plurality of regions of the identified different viewpoint video is associated with a specific region of the target video A related map may be generated by In other words, based on the viewing log, the related map creation unit 105 sets a plurality of areas up to a predetermined rank based on the number of times of viewing among the areas of the different viewpoint video for one specific area of the target video. The association map may be generated by associating.
The distribution unit 104 inputs the related map generated by the related map creation unit 105. The distribution unit 104 acquires divided video information of a video of another viewpoint that has a corresponding relationship with the divided video selected by the user selected based on the input related map, and separates the divided video indicated by the acquired information. Deliver to the viewing client 20 with higher image quality than other divided videos of the viewpoint video.

  According to the video distribution apparatus 10 of the third embodiment configured as described above, a related map can be created from a user's past history. Then, the distribution unit 104 of the video distribution device 10 distributes the compressed divided video corresponding to the specific area of the video of the different viewpoint of the video being viewed by the user to the viewing client 20 based on the created related map. To do. Therefore, it is possible to distribute video that meets user needs.

(Fourth embodiment)
In the first to third embodiments, the configuration in which the related map is created in advance and distributed is shown. In the fourth embodiment, the video distribution apparatus 10 creates a related map in real time.
Specific processing will be described. The distribution unit 104 of the video distribution apparatus 10 grasps the video currently being viewed by the user. Further, since the distribution unit 104 knows the video that the user is currently viewing, the compressed divided video corresponding to the region with the most corresponding points among the video regions at different points in real time based on the information. Select. Then, the video distribution apparatus 10 distributes the related compressed divided video to the viewing client 20 in real time and stores it in the viewing client 20. When configured in this manner, the video distribution device 10 does not include the video storage unit 101. The distribution unit 104 may select a plurality of post-compression divided videos corresponding to areas in which the corresponding points are equal to or greater than a predetermined threshold among the video areas at different viewpoints. Then, the distribution unit 104 distributes a predetermined number of compressed divided videos among the plurality of selected compressed divided videos to the viewing client 20 in real time. Here, the predetermined number may be singular or plural.

  According to the video distribution apparatus 10 in the fourth embodiment configured as described above, it is possible to specify a related area in another viewpoint in real time. In this way, by identifying the relevant area in real time, even if it is an unknown video (video that has never been viewed so far), the user can immediately switch to high-resolution and high-quality after switching the viewpoint. Video can be viewed.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 ... Video delivery apparatus, 20 ... Viewing client, 101 ... Video storage part, 102 ... Division | segmentation part, 103 ... Compression part, 104 ... Delivery part, 105 ... Related map creation part, 106 ... User information storage part, 201 ... Request part , 202 ... Designated area video reception unit, 203 ... Separate stream reception unit, 204 ... Shooting range cover stream reception unit, 205 ... Related map reception unit, 206 ... Received video storage unit, 207 ... Verification unit, 208 ... Presentation unit

Claims (6)

A dividing unit that generates a plurality of divided videos by dividing each video shot from a plurality of different viewpoints into a plurality of regions;
Among the divided videos of the video divided into a plurality of regions, the divided video of the video of another viewpoint corresponding to the divided video selected by the user viewing the video at a certain viewpoint among the plurality of different viewpoints. A distribution unit that distributes images with higher image quality than other segmented images from different viewpoints;
With
The distribution unit is a video of another viewpoint corresponding to the divided video selected by the user selected based on a related map in which the divided videos in the videos divided into the plurality of areas are associated with each other. A video distribution device that acquires information on the divided video and distributes the divided video indicated by the acquired information with higher image quality than other divided video of the video of the different viewpoint.   The distribution unit, based on a viewing log related to the video that the user has viewed in the past, for the divided video that the user is viewing in the video, in the video of another viewpoint related to the video Selected from the related map in which a plurality of divided videos up to a predetermined rank based on the number of times of viewing are divided among the divided videos viewed by the user, the different video image corresponding to the divided video selected by the user The video distribution device according to claim 1, wherein information on a divided video of a video at a viewpoint is acquired, and the divided video indicated by the acquired information is distributed with higher image quality than the other divided video of the video at another viewpoint. A dividing unit that generates a plurality of divided videos by dividing each video shot from a plurality of different viewpoints into a plurality of regions;
Among the divided videos of the video divided into a plurality of regions, the divided video of the video of another viewpoint corresponding to the divided video selected by the user viewing the video at a certain viewpoint among the plurality of different viewpoints. A distribution unit that distributes images with higher image quality than other segmented images from different viewpoints;
With
The distribution unit is a divided video in which a corresponding point with the divided video is a predetermined threshold or more among a plurality of divided videos of a video shot from a different viewpoint based on the divided video in the video being viewed by the user. And distributing a predetermined number of selected divided videos with higher image quality than other divided videos of the video of the different viewpoint. A dividing step of generating a plurality of divided videos by dividing each video shot from a plurality of different viewpoints into a plurality of regions;
Among the divided videos of the video divided into a plurality of regions, the divided video of the video of another viewpoint corresponding to the divided video selected by the user viewing the video at a certain viewpoint among the plurality of different viewpoints. A delivery step of delivering a higher image quality than other divided videos of a different viewpoint;
Have
Another viewpoint video having a correspondence relationship with the divided video selected by the user selected based on the related map in which the divided videos are associated with each other in the videos divided into the plurality of areas in the distribution step The video distribution method of acquiring the information of the divided video and distributing the divided video indicated by the acquired information with higher image quality than the other divided video of the video of the different viewpoint. A dividing step of generating a plurality of divided videos by dividing each video shot from a plurality of different viewpoints into a plurality of regions;
Among the divided videos of the video divided into a plurality of regions, the divided video of the video of another viewpoint corresponding to the divided video selected by the user viewing the video at a certain viewpoint among the plurality of different viewpoints. A delivery step of delivering a higher image quality than other divided videos of a different viewpoint;
Have
In the distribution step, a divided video in which a corresponding point with the divided video is equal to or more than a predetermined threshold among a plurality of divided videos of the video shot from a different viewpoint based on the divided video in the video being viewed by the user And a predetermined number of selected divided videos are distributed with higher image quality than other divided videos of the video of the different viewpoint.   A computer program for causing a computer to execute as the video distribution apparatus according to claim 1.