JP2007189357A - Device and method for distributing arbitrary-viewpoint video - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform area destination capable of acquiring a desired video concerning a system for segmenting the video of the desired area from a panoramic video. <P>SOLUTION: An arbitrary-viewpoint video distribution device segments the video which is wider than the area indicated by a receiver from the panoramic video, and distributes it. The receiver receives the video and performs re-segmentation, based on the latest area indication, so as to eliminate a time till the reflection of an operation due to various kinds of delay. Consequently, a viewer designates the area to acquire the desired video. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  A free-viewpoint video that extracts (cuts out) and distributes a portion of video shot with a super-high resolution and a wide field of view (super wide angle) that greatly exceeds the resolution of the display, according to the field of view (region) requested by the viewer The present invention relates to a distribution apparatus and a free viewpoint video distribution method.

  Conventionally, by shooting multiple objects such as soccer games and pasting the images together like panoramic shots, you can create ultra-high resolution, wide-field images that cannot be shot with a single camera. Obtaining is performed (for example, refer patent document 1).

  The technique disclosed in Patent Document 1 generates a wide-field / high-resolution video using the method described above, and the corresponding range is determined according to viewpoint designation (area designation) performed for each viewer terminal. The video is extracted and distributed. As described above, by distributing according to the viewpoint designation performed for each viewer terminal, it is possible to distribute videos having different visual field ranges (regions) for each viewer desired by the viewer. Such a video distribution system is called a free viewpoint video distribution system.

In this free-viewpoint video distribution system, the viewer terminal designates the video viewpoint (cutout area designation) from the viewer terminal to the server, and the server side cuts out the video based on the designation and sends it to the viewer terminal. It is configured to send.
JP 2004-88483 A

  However, in the system having such a configuration, a delay in the network between the viewer terminal and the server after the viewer of the viewer terminal designates the viewpoint (area) until the corresponding video is displayed. There are various delays such as a delay in video generation processing (video cutout processing) on the server side, and a delay in video compression / decompression time. When providing such a free-viewpoint video for TV, it is desirable to specify the viewpoint (area) using the buttons on the remote control. The viewpoint (area) can be specified using the up / down / left / right buttons in the direction in which the button is pressed. The movement is performed continuously from the viewpoint. Compared to the case where the viewpoint (area) is specified discretely, when the viewpoint is continuously moved, these delays significantly reduce the operability.

  For example, when the viewer searches for a place to view while moving the viewpoint (area), the viewpoint (area) designation operation is reflected with a delay time due to these delays. That is, the video acquired by the viewer corresponds to a time later than the time point when the viewer designates the viewpoint (area). In other words, the video that can be acquired by the viewer may be different from the desired video that the viewer has specified the viewpoint (area). For example, even if the viewer designates the viewpoint (area) to see the video of the shoot scene before the goal, the shoot scene is over when the video is acquired and is different from the desired video of the viewer. is there.

  As described above, there is a problem that it is very difficult for a viewer to designate a viewpoint (region) so that a desired video can be acquired.

  The present invention has been made in view of this point, and an object of the present invention is to make it possible to specify an area in which a desired video can be acquired in a system that cuts out a video of a desired area from a panoramic video.

  In the present invention, the free viewpoint video distribution device cuts out and distributes a video wider than the region instructed from the receiving device from the panoramic video, whereas the receiving device receives this video and receives the latest region. By cutting out again based on the instruction, the time until the operation due to various delays is reflected is eliminated.

  As described above, by applying the present invention, it is possible to apparently eliminate the delay from when the viewer designates the area with the receiving apparatus until the video of the area is actually obtained. As a result, the viewer can specify an area so that a desired video can be acquired.

  A free-viewpoint video distribution apparatus according to the first aspect of the present invention, a video information database for storing panoramic video, a receiving unit for receiving designation of a video cutout area, and a predetermined area on the outer periphery of the designated video cutout area A viewpoint buffer region determination unit that determines a viewpoint buffer region to which the video buffer is attached, a video cutout unit that cuts out a video corresponding to the viewpoint buffer region from the panoramic video, a distribution unit that distributes the video cut out by the video cutout unit, The structure with is adopted.

  According to this, it is possible to apparently eliminate the delay from when the viewer designates the area with the receiving apparatus until the video of the area is actually obtained. Thereby, the viewer can specify an area (viewpoint movement) so that a desired video can be acquired.

  According to a second aspect of the present invention, in the free viewpoint video distribution device according to the first aspect, the viewpoint buffer area determination unit adds a fixed-size area to the outer periphery of the designated video cutout area.

  According to this, the viewpoint buffer area can be determined by simple processing.

  According to a third aspect of the present invention, in the free viewpoint video distribution device according to the first aspect, the viewpoint buffer area determination unit moves the specified video cutout area to the outer periphery of the specified video cutout area. An area of a size corresponding to the direction is added.

  The viewer is highly likely to specify the video clip area in the next moving direction of the specified video clip area. Therefore, a more effective viewpoint buffer area can be determined by widening the viewpoint buffer area in the direction in which the designated video cutout area moves.

  According to a fourth aspect of the present invention, in the free viewpoint video distribution device according to the first aspect, the viewpoint buffer area determination unit moves the specified video cutout area to the outer periphery of the specified video cutout area. An area of a size corresponding to the direction and moving speed is added.

  As the moving speed of the designated video cutout area increases, there is a high possibility that a large number of areas will be required in the moving direction next time. Thus, the viewpoint buffer area can be taken more effectively by this method.

  According to a fifth aspect of the present invention, in the free viewpoint video distribution device according to the third aspect or the fourth aspect, the viewpoint buffer area determination unit includes a previously specified clip area and a clip area specified this time. By comparing, the moving direction is detected.

  Thereby, the moving direction can be detected.

  According to a sixth aspect of the present invention, in the free viewpoint video distribution device according to the fourth aspect, the viewpoint buffer area determination unit includes the previously specified clip area and time, the clip area specified this time, and the current time. The moving direction and moving speed are detected by comparison.

  Thereby, a moving direction and a moving speed are detectable.

  A receiving apparatus according to a seventh aspect of the present invention includes an instruction unit that inputs a video cutout area and instructs another apparatus, and a viewpoint buffer area that adds a predetermined area to the outer periphery of the video cutout area that is designated from a panoramic video A video receiving unit that receives a video cut out of a video corresponding to the video, a cutout position correction unit that cuts out a video corresponding to the video cutout region currently specified from the received video, and the cutout position correction unit cut out the video And a playback unit that plays back the video.

  As a result, it is possible to apparently eliminate the delay from when the viewer designates the area with the receiving apparatus until the video of the area is actually obtained. As a result, the viewer can specify an area so that a desired video can be acquired.

  According to an eighth aspect of the present invention, in the receiving device according to the seventh aspect, the size of the viewpoint buffer area and the video corresponding to the designated viewpoint buffer area are reproduced and displayed after the video cutout area is designated. And a viewpoint moving speed control unit that restricts a specifiable range of the video cutout area based on the delay time until

  As a result, when moving the video cutout area continuously, it is possible to avoid the problem that the viewpoint buffer area is used up halfway and the movement temporarily stops in the middle, and always provides a smooth area movement. Can do.

  A receiving apparatus according to a ninth aspect of the present invention instructs a viewpoint buffer area determining unit that determines a viewpoint buffer area by adding an area to the outer periphery of a specified video cutout area, and instructs the other apparatus to determine the viewpoint buffer area. An instruction unit, a video receiving unit that receives a video clipped from the video corresponding to the viewpoint buffer area instructed from the panoramic video, and a cutout clipped from the received video corresponding to the video clip region currently specified A configuration including a position correction unit and a playback unit that plays back the video clipped by the cutout position correction unit is adopted.

  As a result, the receiving apparatus side can be provided with a function for determining the viewpoint buffer area.

  According to a tenth aspect of the present invention, there is provided a method of instructing a video cutout area from a receiving device that receives video to a free viewpoint video distribution server that stores the panoramic video, and cutting out and distributing the panoramic video based on the instruction from the receiving device. The free viewpoint video distribution server receives the designation of the video cutout area, determines a viewpoint buffer area to which a predetermined area is added to the outer periphery of the designated video cutout area, and determines the viewpoint buffer area from the panoramic video. The video corresponding to is cut out and distributed to the receiving device, and the receiving device receives the video corresponding to the viewpoint buffer area from the panoramic video, and corresponds to the video clipping area currently designated from the video. A free viewpoint video distribution method characterized in that video is cut out and played back.

(Embodiment 1)
Hereinafter, a free viewpoint video distribution system according to Embodiment 1 of the present invention will be described with reference to the accompanying drawings. First, the configuration of the free viewpoint video distribution system according to the first embodiment will be described with reference to FIG.

  The free viewpoint video distribution system 10 according to the first embodiment employs a configuration in which a free viewpoint video distribution server 100 that distributes video and a receiving device 101 that receives the distributed video are connected. In addition, a plurality of photographing devices 102 are connected to the free viewpoint video distribution server 100. Further, a viewpoint operation input device 106 such as a remote control and a display device 114 such as a monitor are connected to the receiving device 101.

  The plurality of photographing devices 102 perform panoramic photographing. In panoramic photography, photographing is performed while shifting the photographing direction as in a panoramic photograph, and the photographed images are bonded together to obtain a high-resolution image with a wide viewing angle. In the free-viewpoint video distribution, unlike panoramic photographs, in order to handle the video, a plurality of directions are simultaneously captured as moving image videos using a plurality of imaging devices 102 such as a plurality of video cameras.

  The free viewpoint video distribution server 100 is provided with a photographing device control unit 103. The photographing apparatus control unit 103 takes in a video (group) photographed by the photographing apparatus 102 and inputs it to the image processing unit 104 in a format (digital video data or the like) that can be image-processed.

  The image processing unit 104 combines a plurality of input videos (groups) for each frame, generates a panoramic video, and stores it in the video information database 105. As a detailed image processing algorithm or the like, a conventional method such as the method described in Patent Document 1 may be used, and thus detailed description thereof is omitted.

  The video information database 105 can store, for example, a soccer game as a panoramic video from the beginning to the end. Further, the video stored in the video information database 105 can be accessed in frame units to extract the video.

  Although it has been described that panoramic imaging using a plurality of imaging devices 102 and a panoramic image with a wide viewing angle and high resolution are obtained using the image processing unit 104, a method of acquiring a panoramic image may be other than this. Good. For example, if a super wide-angle lens and an imaging device capable of high-resolution imaging are used, it is possible to obtain a sufficiently wide viewing angle and high-resolution video with only one imaging device without panoramic imaging. In this case, a configuration in which video is directly input from the imaging device control unit 103 to the video information database 105 can be taken.

  The free viewpoint video distribution server 100 is provided with a cutout area determination unit 108. The cutout area determination unit 108 passes the coordinates of the viewpoint (area) designated from the receiving apparatus 101 to the viewpoint buffer area determination unit 110. In addition, the cutout region determination unit 108 inputs the viewpoint buffer region (details will be described later) determined by the viewpoint buffer region determination unit 110, and checks whether the viewpoint buffer region is in an appropriate range. Specifically, the cutout area determination unit 108 determines whether the viewpoint buffer area is within the coordinate range of the panoramic video stored in the video information database 105. If the designated area is outside the panoramic video coordinate range stored in the video information database 105, the cutout area determination unit 108 selects the area so that the viewpoint buffer area is within the panoramic video coordinate range. adjust. Then, the cutout region determination unit 108 passes the information on the viewpoint buffer region adjusted as necessary to the video cutout unit 109.

  The viewpoint buffer area determination unit 110 corrects the cutout area so as to take extra areas on the outside, top, bottom, left, and right of the designated area. This extra area is called a viewpoint buffer area. The viewpoint buffer area determination unit 110 determines the viewpoint buffer area, and returns the information of the corrected cutout area to the cutout area determination unit 108. The details of the viewpoint area buffer determination process of the viewpoint buffer area determination unit 110 will be described later.

  The video cutout unit 109 reads the panoramic video from the video information database 105, cuts out the video in the viewpoint buffer area designated by the cutout region determination unit 108 from the read panoramic video, and distributes the video to the receiving apparatus 101.

  In addition, since the free viewpoint video distribution server 100 is a video distribution system, unless otherwise specified, the video cutout unit 109 reads the video from the video information database 105 while sequentially advancing the time in units of frames, and specifies the specified area. The operation of cutting out and distributing is repeated.

  As described above, the free viewpoint video distribution server 100 is configured. Next, the configuration of the receiving apparatus 101 will be described.

  The receiving apparatus 101 is provided with a viewpoint instructing unit 107 for inputting a viewpoint direction of a video to be viewed, that is, a cutout area specified by a viewer using free viewpoint video distribution using a viewpoint operation input device 106 such as a remote controller. ing. The viewpoint instruction unit 107 calculates the requested free viewpoint video area based on the designated viewpoint direction, and gives an instruction to the cut-out area determination unit 108 of the free viewpoint video distribution server 100. This instruction is performed, for example, by designating the upper left coordinates (X1, Y1) and lower right coordinates (X2, Y2) of the cutout area.

  Here, the viewpoint designation (cutout area designation) method will be described in detail. In simple terms, what is called free viewpoint video distribution is to cut out a partial area from a panoramic video according to the instructions of the viewer and distribute it. Specifying the viewpoint is equivalent to specifying the area to be cut out.

  In the present invention, it is assumed that the viewer moves the cut-out region vertically and horizontally to obtain a desired viewpoint image. For example, when using a remote controller having up / down / left / right buttons as the viewpoint operation input device 106, the viewer presses a button in a desired direction using the up / down / left / right buttons while viewing the video.

  On the other hand, the viewpoint instruction unit 107 stores the coordinates of the current area as, for example, upper left coordinates (X1, Y1), lower right coordinates (X2, Y2), and when the button is pressed, Move in the direction from the coordinates of the area. The viewpoint instruction unit 107 repeats this movement while the button is pressed.

  In the first embodiment, the coordinate movement amount is not particularly limited. For example, it may be determined to move 50 pixels per second at a constant speed, or the amount of movement may be increased in an accelerated manner by continuously pressing a button. In this way, the viewer uses the up / down / left / right buttons of the remote controller 106 to designate a desired viewpoint.

  In addition, the receiving apparatus 101 is provided with a video receiving unit 111 that receives the video distributed from the video cutout unit 109 of the free viewpoint video distribution server 100. The video reception unit 111 sends the received video to the cutout position correction unit 112.

  The cutout position correction unit 112 obtains the latest cutout region information based on the viewer's viewpoint operation from the viewpoint instruction unit 107. Next, the cutout position correcting unit 112 cuts out the video from the free viewpoint video to which the viewpoint buffer region is received, which is received by the video receiving unit 111, according to the latest cutout region information. The cutout position correction unit 112 sends the video that has been cut out again to the video reproduction unit 113.

  In other words, the cutout position correction unit 112 receives the free viewpoint video that has been widely cut out and distributed with the viewpoint buffer area determined by the viewpoint buffer area determination unit 110 of the free viewpoint video distribution server 100 from the viewpoint instruction unit 107. Cut out again according to the latest cutout area obtained. Thereby, it can operate | move so that a viewer's viewpoint operation may be reflected in an image | video immediately.

  The video playback unit 113 converts the sent video into a format that can be played back, sends the video to the display device 114, and the display device 114 displays the video.

  In this way, the free viewpoint video distribution server 100 cuts out and distributes video wider than the display area specified by the receiving apparatus 101, and adjusts the cutting position based on the latest viewpoint position in the receiving apparatus 101.

  Next, details of the viewpoint area buffer determination processing of the viewpoint buffer area determination unit 110, which is a feature of the present invention, will be described. In the first embodiment, the viewpoint buffer area determination unit 110 adopts a method of “taking a fixed size”.

  The “fixed size” method is a method of taking a viewpoint buffer area 202 having a predetermined size at the top, bottom, left, and right of the coordinates of the area 201 specified by the viewpoint instruction unit 107 as shown in FIG. This is, for example, determined by fixed pixel values such as “100 pixels in the vertical direction” and “200 pixels in the horizontal direction” of the area 201 specified by the viewpoint instruction unit 107.

  Specifically, referring to FIG. 2, the upper size 203 and the lower size 204 are determined in advance as fixed values, such as 100 pixels each, and the left size 205 and right size 206 are each 200 pixels. .

  In addition, when there are a plurality of area sizes designated by the viewer, such as when accessed from a plurality of types of receiving apparatuses 101, instead of deciding the sizes 203 to 206 as constants, “10% is taken as the viewpoint buffer area 202” may be set as a ratio to the area size.

  This will be described with reference to FIG. 2. For example, if the size of the area 201 specified by the viewer is 1920 pixels wide and 1080 pixels high, the upper size 203 and the lower size 204 are 10% of the vertical 1080 pixels, respectively. 108 pixels, left size 205 and right size 206 are each 192 pixels, which is 10% of the horizontal 1920 pixels. In addition, when a receiving device 101 having a small screen size, such as a portable terminal, specifies an area of 320 pixels in width and 240 pixels in height, the upper size 203 and the lower size 204 are 32 pixels, the left size 205 and the right Each direction size 206 is 24 pixels.

  In this way, the viewpoint buffer area determination unit 110 determines an area larger than the area specified by the viewer as the viewpoint buffer area.

  Next, processing of the free viewpoint video distribution system 10 will be described using the sequence diagram of FIG.

  First, the plurality of imaging devices 102 capture a plurality of videos while shifting the imaging direction, and send the images to the imaging device control unit 103 (ST301). Next, the photographing apparatus control unit 103 takes in a plurality of videos (groups) and inputs them into the image processing unit 104 in a format that allows image processing, such as digital video data. Next, the image processing unit 104 combines a plurality of input videos (groups) for each frame to generate a panoramic video as shown by 400 in FIG. 4 (ST302). Then, the image processing unit 104 stores the generated panoramic video 400 in the video information database 105 (ST303).

  As described above, the free viewpoint video distribution server 100 generates and stores the panoramic video 400.

  On the other hand, when receiving apparatus 101 receives a viewpoint designation from viewpoint operation input apparatus 106 such as a remote controller, viewpoint instruction unit 107 inputs the viewpoint designation (ST304). The viewpoint instruction unit 107 calculates a free viewpoint video area from the input viewpoint instruction, and sends the calculated free viewpoint video area to the cut-out area determination unit 108 of the free viewpoint video distribution server 100 (ST305).

  Note that the viewpoint instruction unit 107 may send the size of the display device 114 to the video cutout unit 109.

  In response to this, the cutout region determination unit 108 passes the coordinates of the region specified by the viewpoint instruction unit 107 to the viewpoint buffer region determination unit 110. For example, when the designated area is indicated by 401 in FIG. 4, the viewpoint buffer area determination unit 110 determines the area indicated by 402 as the viewpoint buffer area by adding extra areas to the outside of the upper, lower, left, and right sides of the area 401. Then, it is sent to the cutout area determination unit 108 (ST306).

  The cut-out area determination unit 108 inputs the viewpoint buffer area determined by the viewpoint buffer area determination unit 110 and checks whether the viewpoint buffer area is in an appropriate range. Specifically, the cutout area determination unit 108 determines whether the viewpoint buffer area is within the coordinate range of the panoramic video stored in the video information database 105. If the designated area is outside the panoramic video coordinate range stored in the video information database 105, the cutout area determination unit 108 selects the area so that the viewpoint buffer area is within the panoramic video coordinate range. adjust. Then, the cutout region determination unit 108 passes the information on the viewpoint buffer region adjusted as necessary to the video cutout unit 109 (ST307).

  In the case of the example shown in FIG. 4, since the viewpoint buffer area 402 is within the panoramic video 400, the cutout area determination unit 108 passes information on the viewpoint buffer area 402 to the video cutout unit 109.

  Next, the video cutout unit 109 reads the panoramic video from the video information database 105, cuts out the video in the viewpoint buffer area designated by the cutout area determination unit 108 from the read panoramic video (ST308), The video in the viewpoint buffer area is distributed (ST309).

  In this way, the free viewpoint video distribution server 100 cuts out an area wider than the area specified by the receiving apparatus 101 from the panoramic video and sends it to the receiving apparatus 101. Thereby, the receiving apparatus 101 can further specify a region and cut out a video from the video sent from the free viewpoint video distribution server 100. That is, in the receiving apparatus 101, it is possible to specify an area to be cut out by viewing a real-time video.

  On the other hand, the receiving apparatus 101 receives the video in the viewpoint buffer area transmitted from the free viewpoint video distribution server 100 in the video receiving unit 111. Then, the video reception unit 111 sends the received video of the viewpoint buffer area to the cutout position correction unit 112.

  The cutout position correction unit 112 acquires information about the cutout region based on the viewer's current time point, that is, the latest viewpoint operation, from the viewpoint instruction unit 107 (ST310). Next, the cutout position correcting unit 112 cuts out the video in the viewpoint buffer area according to the latest cutout area information (ST311). Then, the cutout position correction unit 112 sends the video cut out according to the latest cutout area information to the video reproduction unit 113.

  For example, if the area indicated by 401 in FIG. 4 is the latest cutout area, the cutout position correcting unit 112 cuts out an image as indicated by 500 in FIG.

  Then, video playback section 113 plays back the cut video and displays it on display device 114 (ST312).

  Thus, since the video transmitted from the free viewpoint video distribution server 100 is wider than the designated area, the receiving apparatus 101 can further cut out the video by designating the area. That is, the receiving apparatus 101 can designate a video area to be cut out in real time from the video.

  In addition, since the process of cutting out the video of the designated area in real time is performed in the receiving apparatus 101, the delay in the network between the free viewpoint video distribution server 100 and the receiving apparatus 101, the video in the free viewpoint video distribution server 100 It is not affected by a delay in the generation process (video cutout process) or a delay in the video compression / decompression time. As a result, the video acquired by the viewer is the same as when the viewer designates the viewpoint. That is, the video that can be acquired by the viewer is the same as the desired video that the viewer has designated as a viewpoint. As a result, the viewer can specify the viewpoint so that the desired video can be acquired.

  For example, as shown in FIG. 4, when the receiving apparatus 101 designates the viewpoint in ST304, it is assumed that there is a ball player inside the area 401. In ST309, when the video is acquired, the player moves in the direction of arrow A. Consider a case in which the object moves outside the area 401 and inside the area 402. Even in this case, in the case of the present embodiment, the viewer can view the video in the area 402 and specify the viewpoint at the position where the player enters.

  As described above, according to the first embodiment, it is possible to apparently eliminate the delay from when the viewer designates the viewpoint with the receiving apparatus 101 until the video of the viewpoint is actually obtained. As a result, the viewer can specify the viewpoint so that the desired video can be acquired.

  Note that the receiving apparatus 101 may have a function for determining the viewpoint buffer area. This form will be described with reference to FIG.

  First, in the reception device 601, the viewpoint instruction unit 606 receives a viewpoint designation from the viewpoint operation input device 106, calculates a region to be cut out, and sends it to the cut-out region determination unit 604. The cutout area determination unit 604 sends information on the cutout area to the viewpoint buffer area determination unit 605.

  The viewpoint buffer area determination unit 605 determines a viewpoint buffer area by performing the same processing as the viewpoint buffer area determination unit 110 illustrated in FIG. 1 and sends the viewpoint buffer area to the cut-out area determination unit 604.

  The cutout area determination unit 604 determines whether the viewpoint buffer area is within the panoramic video range, and if so, sends the viewpoint buffer area to the video cutout unit 603 of the free viewpoint video distribution server 600.

  The video cutout unit 603 cuts out the video for the viewpoint buffer area received from the video information database 105 and sends the video to the video reception unit 111 of the reception device 601.

  The video receiver 111 sends the received video for the viewpoint buffer area to the cutout position corrector 112.

  The cutout position correction unit 112 acquires information about the cutout region based on the viewer's current time point, that is, the latest viewpoint operation, from the viewpoint instruction unit 606. Next, the cutout position correcting unit 112 cuts out the video in the viewpoint buffer area according to the latest cutout area information. Then, the cutout position correction unit 112 sends the video cut out according to the latest cutout area information to the video playback unit 113, and the playback unit 113 plays back the video.

  The operation of the free viewpoint video distribution server 100 and the receiving apparatus 101 according to the first embodiment may be a program and executed by a general-purpose computer.

(Embodiment 2)
A second embodiment will be described. The processing of the viewpoint buffer area determination unit 110 is different between the second embodiment and the first embodiment. Specifically, in the first embodiment, the viewpoint buffer area determination unit 110 fixes the size and position of the viewpoint buffer area, but in the second embodiment, the position of the viewpoint buffer area is determined for the specified viewpoint. Decided according to the moving direction. That is, the viewpoint buffer area determination unit 110 detects in which direction the area designated by the viewer is moving from the coordinate position of the area designated from the previous time, and there are many viewpoint buffer areas in the moving direction. It was made to become.

  Hereinafter, the viewpoint buffer area determination unit 110 according to the second embodiment will be described in detail.

  The viewpoint buffer area determination unit 110 records the coordinates of the area designated by the viewer at least once in the past, and the viewer can compare the coordinates of the area designated this time and the area designated last time so that the viewer can It is detected in which direction the viewpoint is moving.

  Specifically, the upper left coordinate of the area designated last time is (1000, 500), the lower right coordinates are (2920, 1580), the upper left coordinates of the area designated this time are (1010, 500), and the lower right coordinates are (2930). , 1580). In this case, the viewpoint buffer area determination unit 110 determines that the area is moving in the right direction.

  The viewpoint buffer area determination unit 110 determines the viewpoint buffer area after detecting the moving direction in this way. The viewpoint buffer area determination unit 110 determines in advance how to use the viewpoint buffer area as “size in the moving direction”, “size in the direction opposite to the moving direction”, and “size in the non-moving direction”. Can be determined.

  For example, the viewpoint buffer area determination unit 110 divides the movement direction into left and right and upper and lower components, and in the left and right direction, the size in the movement direction side is 300 pixels, and the size in the opposite direction is not moved to either 100 pixels. In this case, the left and right sides are determined to be 200 pixels, the vertical direction is determined to be 150 pixels on the moving direction side, the opposite direction size is set to 50 pixels, and the vertical direction is determined to be 100 pixels on both sides.

  A case where the viewpoint designation is moving in the right direction will be described with reference to FIG. In this case, the viewpoint buffer area determination unit 110 sets the left size 705 to 100 pixels, the right size 706 to 300 pixels, and has not moved up and down, so the up size 703 and the down size 704 are set to 100 pixels, respectively. And

  Note that the example in FIG. 7 is an example in which there is only movement in the left and right direction. For example, when the viewpoint is moving obliquely, a large number of viewpoint buffer areas are allocated on the movement direction side in each of the vertical and horizontal directions. Do it.

  Further, the allocation of the viewpoint buffer area of the area according to the moving direction may not be determined as a constant, but may be set at a ratio with respect to the size of the area designated by the viewer.

  As described above, according to the second embodiment, it is possible to detect in which direction the area designated by the viewer is moving, and to increase the viewpoint buffer area in the moving direction. The viewer is likely to designate the area in the next direction in the movement direction of the designated area. Therefore, a more effective viewpoint buffer area can be determined by widening the viewpoint buffer area in the direction in which the designated area moves. That is, it is possible to optimally assign the coverage of the viewpoint buffer area for the current operation.

(Embodiment 3)
A third embodiment will be described. In the third embodiment, a restriction on the movement speed specified by the viewpoint is provided.

  Thus, the delay until the viewer's viewpoint operation is reflected in the free viewpoint video can be absorbed by the viewpoint buffer area. That is, it is possible to prevent a situation in which the viewer's viewpoint operation is out of the range of the viewpoint buffer area.

  In this way, it is possible to prevent the viewer from instructing to move the viewpoint beyond the range that can be corrected by the viewpoint buffer area. As indicated by reference numeral 903 in FIG. 8, when the viewpoint designation range has moved to the end of the viewpoint buffer area 202, the viewpoint cannot be displayed beyond the viewpoint buffer area. It will stop.

  In the third embodiment, the viewpoint moving speed is devised. This is because if the restriction on the movement speed of the viewpoint is made strict so that only a low-speed movement can be performed, the viewpoint can move little by little. In other words, this is because the operability when the viewpoint is greatly moved is impaired.

  Therefore, in the third embodiment, the viewpoint moving speed is controlled according to the size of the viewpoint buffer area.

  Hereinafter, the free viewpoint video distribution system according to the third embodiment will be described with reference to FIG. In addition, the same code | symbol is provided to the part same as the part already demonstrated.

  The receiving device 801 is provided with a delay time measuring unit 807. The delay time measuring unit 807 measures the delay time. The delay time is the time from when the viewer designates the viewpoint until the free viewpoint video of that viewpoint is actually reproduced and displayed. The delay time measurement unit 807 uses the worst value of this time (the value when it takes the longest) as the delay time actually used for calculation.

  The delay time is mainly the packet round-trip time of the network between the free viewpoint video distribution server 800 and the receiving device 801, the processing time in the free viewpoint video distribution server 800 (in response to an instruction from the receiving device 801 in the free viewpoint video distribution server 800) To the time from when the video is transmitted) to the processing time at the receiving device 801 (the time from when the video is received by the receiving device 801 until it is displayed).

  The delay time can be measured as follows.

  First, the viewpoint instruction unit 803 adds a time stamp indicating the specified time to the information of the region to be specified to the cutout region determination unit 804. The cutout area determination unit 804 passes a time stamp to the video cutout unit 805 when instructing the cutout area. Then, the video cutout unit 805 adds a time stamp to the finally cut free viewpoint video and sends it to the receiving device 801. That is, the video cutout unit 805 multiplexes the time stamp when the viewer designates the area of the free viewpoint video with the data of the free viewpoint video.

  The receiving device 801 receives the sent video by the video receiving unit 805 and sends it to the cutout position correcting unit 112. The cutout position correction unit 112 cuts out the video again by the method described above, and then sends it to the video playback unit 810.

  The video reproduction unit 810 detects a time stamp from the transmitted video and sends it to the delay time measurement unit 807.

  The delay time measuring unit 807 obtains the delay time by comparing the sent time stamp with the time at that time (reproduction time stamp). As described above, the delay time measuring unit 807 accurately obtains the delay time. Then, the delay time measurement unit 807 passes the measured delay time to the viewpoint movement speed control unit 808.

  The method for measuring the delay time may be a method other than that described above. For example, the worst value (RTT: Round Trip Time) of the packet round trip time (RTT) between the free-viewpoint video distribution server 800 and the receiving device 801 measured using an ICMP message such as a PING command included in UNIX (registered trademark) or the like. The sum of the worst value of the processing time in the free viewpoint video distribution server 800 and the worst value of the processing time in the receiving device 801 measured in advance and the worst value of the processing time in the receiving device 801 may be used as the delay time. In this case, the delay time measuring unit 807 has a function of measuring the packet round-trip time and a function of setting the worst value of the processing time of the free viewpoint video distribution server 800 and the receiving device 801 from the outside.

  In addition, when it is possible to estimate in advance the worst value of the packet round-trip time of the network (the value when it takes the longest time), such as when distributing within a specific network such as CATV, a free viewpoint video It is also possible to define the delay time as a constant in advance together with the worst value of the processing time of the distribution server 800 and the receiving device 801. If measurement is not performed at the time of viewing as described above, there is no need to perform measurement processing in the receiving device 801. Therefore, the delay time measuring unit 807 is not provided in the receiving device 801, and the delay time is directly set in the viewpoint movement speed control unit 808. It may be set.

  In addition, the receiving device 801 is provided with a viewpoint movement speed control unit 808. The viewpoint movement speed control unit 808 determines the maximum value of the viewpoint movement speed within the range that can be corrected in the viewpoint buffer area from the delay time measured by the delay time measurement unit 807 and the size of the viewpoint buffer area determined by the viewpoint buffer area determination unit 811. calculate.

  Information on the size of the viewpoint buffer area is added to the free viewpoint video clipped by the video cutout section 805 from the viewpoint buffer area determination section 811 via the cutout area determination section 804 and distributed. The size information of the viewpoint buffer area is extracted by the video reproduction unit 810 via the video reception unit 805 and the cutout position correction unit 112 in the same manner as the time stamp information at the time of delay time measurement. Via 807, it is passed to the viewpoint movement speed control unit 808 together with the delay time information.

In this way, the viewpoint movement speed control unit 808 obtains information on the delay time and the size of the viewpoint buffer area. The viewpoint movement speed control unit 808 obtains the maximum value of the viewpoint movement speed by using the delay time and the size information of the viewpoint buffer area in the following equation.
(Maximum viewpoint movement speed) = (Size of viewpoint buffer area) ÷ (Delay time)

  Further, the viewpoint movement speed control unit 808 calculates the maximum value of the viewpoint movement speed for each of the upper, lower, left, and right because the viewpoint buffer areas exist above, below, left, and right of the area designated by the viewer.

  Also, the viewpoint movement speed control unit 808 periodically calculates the “maximum value of the viewpoint movement speed”, for example, for each frame of the free viewpoint video, and sends it to the viewpoint instruction unit 803.

  The viewpoint instruction unit 803 uses the “maximum value of viewpoint movement speed” sent from the viewpoint movement speed control unit 808 to limit the speed of viewpoint movement. Specifically, the viewpoint instruction unit 803 determines a coordinate range in which the viewpoint can be moved using the coordinate position of the area designated last time and the maximum value of the viewpoint movement speed.

  For example, the viewpoint instruction unit 803 has a maximum viewpoint moving speed of 50 pixels / second, the previously designated area is the upper left coordinates (1000, 500), the lower right coordinates (2920, 1580), and 0 from the previous time point. When it is desired to move to the right when 5 seconds have passed, control is performed so that the maximum of 25 pixels can be moved in the right direction. At that time, upper left coordinates (1025, 500), lower right coordinates (2945, 1580).

  Incidentally, the third embodiment is also devised in the processing of the viewpoint buffer area determination unit 811. Specifically, the viewpoint buffer area determination unit 811 according to the third embodiment makes the size of the viewpoint buffer area variable according to the moving direction and the moving speed.

  Since the maximum value of the viewpoint movement speed is determined by the size of the viewpoint buffer area, the larger the viewpoint buffer area, the larger the maximum value of the viewpoint movement speed, and the viewer can move the viewpoint faster. become.

  This is because the viewpoint buffer area determination unit 811 detects the viewpoint movement speed, and if the viewpoint movement speed increases, the viewpoint buffer area in the movement direction becomes larger, and conversely if the viewpoint movement speed decreases, this time. This is realized by determining the viewpoint buffer area in the moving direction to be smaller.

  The viewpoint buffer area determination unit 811 detects the movement speed in addition to the movement direction described in the section of the viewpoint buffer area according to the second embodiment. For example, the upper left coordinates of the area designated last time are (1000, 500), the lower right coordinates are (2920, 1580), the upper left coordinates of the area designated this time are (1010, 500), and the lower right coordinates are (2930, 1580). , It can be seen that the region has moved 10 pixels to the right. Then, it is assumed that the viewpoint buffer area determination unit 811 measures the elapsed time from the previous designation, and the time is, for example, 0.5 seconds. In this case, it can be seen that the viewpoint movement speed is 20 pixels / second. The viewpoint buffer area determination unit 811 increases or decreases the size of the viewpoint buffer area so that the following expression is established.

(Viewer buffer area size) ≥ (Viewpoint movement speed) x (Delay time)
Here, the viewpoint buffer area determination unit 811 needs to know the “delay time”, but this information may be notified together when the receiving apparatus 801 instructs the cut-out area.

  If the viewpoint moving speed is 20 pixels / second and the delay time is 2 seconds, the viewpoint buffer area needs to be at least 40 pixels. For example, an 80-pixel viewpoint buffer area that is twice the minimum of 40 pixels is provided. Like that. Then, a free viewpoint video in which a viewpoint buffer area of 80 pixels is set is delivered to the receiving device 801. This time, the maximum viewpoint movement speed is 40 pixels / perception based on the viewpoint buffer area size of 80 pixels and a delay time of 2 seconds. Seconds and maximum speed are increased, allowing you to move faster.

  As described above, in the third embodiment, the viewpoint moving speed control unit 808 is further added to the configuration of the first embodiment, so that the moving speed of the buffer can be limited, and the viewpoint is moved continuously. In such a case, it is possible to avoid the problem that the buffer is used up in the middle and the movement temporarily stops in the middle, and a smooth viewpoint movement can always be provided.

  Note that the viewpoint buffer area determination unit 811 may determine how to acquire the viewpoint buffer area in consideration of the moving speed of the viewpoint as well as the moving direction of the viewpoint by the user.

  The processing of the viewpoint buffer area determination unit 811 in this embodiment will be described with reference to FIG.

  First, when there is no movement of the viewpoint by the user, the viewpoint buffer area determination unit 811 adds an area equal to the specified area 201 in the vertical and horizontal directions, for example, an area of 100 pixels, as indicated by 1001 in the figure. Is the viewpoint buffer area 202.

  Next, assuming that the user moves the viewpoint in the right direction at a speed of 50 pixels / second, the viewpoint buffer area determination unit 811 displays an area of 50 pixels on the left side of the designated area 201 as indicated by 1002 in the figure. The viewpoint buffer area 202 is an area with a 150-pixel area added on the right side. In this state, since the viewpoint is not moved in the vertical direction, the viewpoint buffer area determination unit 811 adds an equal area in the vertical direction of the area 201.

  Next, assuming that the user moves the viewpoint in the right direction at a speed of 75 pixels / second, the viewpoint buffer area determination unit 811 displays an area of 25 pixels on the left side of the specified area 201 as indicated by 1003 in the figure. The viewpoint buffer area 202 is an area with a 175 pixel area added to the right side. In this state, since the viewpoint is not moved in the vertical direction, the viewpoint buffer area determination unit 811 adds an equal area in the vertical direction of the area 201.

  As described above, as the moving speed of the viewpoint is higher, a larger number of areas are assigned to the area 201. The faster the viewpoint movement is, the higher the possibility that more areas will be required in the moving direction next time, and this method makes it possible to take the viewpoint buffer area more effectively.

  Next, assuming that the user moves the viewpoint leftward at a speed of 12.5 pixels / second, the viewpoint buffer area determination unit 811 displays 25 pixels on the left side of the designated area 201 as indicated by 1004 in the figure. The viewpoint buffer area 202 is an area with a 175 pixel area added to the right side. That is, the viewpoint buffer area determination unit 811 allocates the same viewpoint buffer area as indicated by 1003.

  This is because the situation in which the direction of the viewpoint changes is likely due to fine adjustment of the viewpoint. In the case of fine adjustment of the viewpoint, there is a high possibility that the same viewpoint will be designated next, so it can be said that it is more effective not to change the way of taking the viewpoint buffer area.

  Next, assuming that the user moves the viewpoint leftward at a speed of 50 pixels / second, the viewpoint buffer area determination unit 811 displays an area of 100 pixels on the left side of the specified area 201, as indicated by 1005 in the figure. The viewpoint buffer area 202 is an area with a 100-pixel area added to the right side.

  As described above, when the viewpoint continuously moves in the same direction, it is highly likely that the viewpoint will move in the same direction next time. Therefore, a wider area is taken in the moving direction than in the previous step 1004.

  Then, assuming that the user moves the viewpoint leftward at a speed of 75 pixels / second, the viewpoint buffer area determination unit 811 displays an area of 175 pixels on the left side of the designated area 201, as indicated by 1006 in the figure. A region to which a 25-pixel region is added on the right side is a viewpoint buffer region 202.

  By doing so, the viewpoint buffer area can be determined more effectively.

  As described above, according to the third embodiment, in addition to the effects of the first embodiment, it is possible to limit the effective movement speed. The problem that the movement stops once in the middle can be avoided, and a smooth viewpoint movement can always be provided.

  According to the present invention, it is possible to specify a viewpoint so that a desired video can be acquired. The present invention can be applied regardless of a system using broadcasting or a system using the Internet as long as a part of a high-resolution wide-angle shot video such as panoramic shooting is cut out and distributed.

1 is a configuration diagram of a free viewpoint video distribution system according to a first embodiment of the present invention. The figure for demonstrating the viewpoint area buffer determination process of the viewpoint buffer area determination part concerning Embodiment 1. FIG. Processing Sequence Diagram of Free Viewpoint Video Distribution System According to First Embodiment Illustration for explaining panoramic video Illustration for explaining free viewpoint video Configuration diagram of a free viewpoint video distribution system according to another example of the first embodiment The figure for demonstrating the viewpoint area buffer determination process of the viewpoint buffer area determination part concerning Embodiment 2 of this invention. The figure for explaining the problem of the conventional free viewpoint distribution system Configuration diagram of a free viewpoint video distribution system according to a third embodiment of the present invention. The figure for demonstrating the viewpoint area buffer determination process of the viewpoint buffer area determination part concerning Embodiment 3 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Free viewpoint video delivery system 100, 600, 800 Free viewpoint video delivery server 101, 601, 801 Receiving device 105 Video information database 107, 606, 803 View instruction unit 108, 604, 804 Cut-out area determination unit 109, 805 Video cut-out unit 110, 605, 811 Viewpoint buffer area determination unit 112 Cutout position correction unit 810 Delay time measurement unit 808 Viewpoint movement speed control unit

Claims (10)

  A video information database for storing panoramic video; a receiving unit for receiving designation of a video clipping region; a viewpoint buffer region determining unit for determining a viewpoint buffer region to which a predetermined region is added to an outer periphery of the designated video clipping region; A free viewpoint video distribution apparatus comprising: a video cutout unit that cuts out video corresponding to the viewpoint buffer area from the panoramic video; and a distribution unit that distributes the video cut out by the video cutout unit.   The free viewpoint video distribution apparatus according to claim 1, wherein the viewpoint buffer area determination unit adds a fixed-size area to an outer periphery of the designated video cutout area.   The free viewpoint according to claim 1, wherein the viewpoint buffer area determination unit adds an area having a size according to a moving direction of the designated video clipping area to an outer periphery of the designated video clipping area. Video distribution device.   2. The viewpoint buffer area determination unit adds an area having a size according to a moving direction and a moving speed of the specified video cutout area to an outer periphery of the specified video cutout area. Free viewpoint video distribution device.   5. The free viewpoint according to claim 3, wherein the viewpoint buffer area determination unit detects the moving direction by comparing the clip area specified last time with the clip area specified this time. Video distribution device.   5. The viewpoint buffer area determination unit detects the moving direction and the moving speed by comparing the clip area and time specified last time with the clip area specified this time and the current time. Free viewpoint video distribution device.   An instruction unit for inputting a video cutout area and instructing another device, and video reception for receiving a video cut out of a video corresponding to a viewpoint buffer area provided with a predetermined area on the outer periphery of the video cutout area designated from a panoramic video A cutout position correction unit that cuts out a video corresponding to the video cutout region currently specified from the received video, and a playback unit that plays back the video cut out by the cutout position correction unit. A receiving device.   The range that can be specified for the video clipping area is limited by the size of the viewpoint buffer area and the delay time from when the video clipping area is designated until the video corresponding to the designated viewpoint buffer area is played back and displayed. The receiving apparatus according to claim 7, further comprising a viewpoint moving speed control unit.   A viewpoint buffer area determination unit that determines a viewpoint buffer area with an area added to the outer periphery of a specified video cutout area; an instruction unit that instructs the viewpoint buffer area to another device; and the viewpoint buffer area that is instructed from a panoramic video A video receiving unit that receives a video cut out of a video corresponding to the video, a cutout position correction unit that cuts out a video corresponding to the video cutout region currently specified from the received video, and the cutout position correction unit cut out the video And a playback unit that plays back the video. In a method of instructing a video cutout area from a receiving device that receives video to a free viewpoint video distribution server that stores panoramic video, and cutting out and delivering a panoramic video based on an instruction from the receiving device,
The free viewpoint video distribution server receives designation of a video cutout area, determines a viewpoint buffer area to which a predetermined area is added to an outer periphery of the designated video cutout area, and corresponds to the viewpoint buffer area from the panoramic video Cut out the video to be delivered and distribute it to the receiving device
The receiving device receives a video corresponding to the viewpoint buffer area from the panoramic video, cuts out a video corresponding to the video clipping area currently designated from the video, and plays back the video. Delivery method.

Images