JP7319228B2 - Image distribution device, image generation device and program - Google Patents

Description

The present invention relates to an image distribution device, an image generation device and a program, and more particularly to a device and a program capable of providing multi-viewpoint video in which the same scene can be viewed from different viewpoints.

2. Description of the Related Art In recent years, multi-view video technology has been provided that allows viewing the same scene from different viewpoints. For example, in addition to capturing images from a limited number of viewpoints using an image capturing device fixed at one predetermined point in a soccer stadium, multiple image capturing devices can be placed at various positions within the stadium, and these multiple image capturing devices can be viewed at desired angles. A free-viewpoint video is provided that allows you to view the video with More specifically, the viewer can freely set the viewpoint, such as reproducing the scene seen by the players on the pitch of the soccer stadium, or viewing the same scene from various angles. Furthermore, it is possible for the viewer to freely change the viewpoint while making the video slow motion, and to operate the viewpoint and the time as desired by the viewer.

As such multi-viewpoint video technology, an image distribution device and an image display device are provided, and the image distribution device generates a single integrated image in which a plurality of images of the same scene viewed from different viewpoints are arranged in one frame. A system has been proposed in which the integrated image is generated, the single integrated image is distributed to an image display device, and one of the plurality of images included in the single integrated image is displayed on the image display device (Patent Document 1 reference).

According to the system described in Patent Document 1, the same integrated image can be transmitted to a plurality of image display devices. Therefore, the system configuration can be simplified. In addition, by using a single image format as the integrated image, it is possible to reduce changes to the existing system configuration and reduce the amount of video data to be distributed by using existing image compression technology or the like.

International Publication No. 2018/155670 pamphlet

However, the system described in Patent Literature 1 arranges all of a plurality of images of the same scene viewed from different viewpoints in one frame to generate a single integrated image. Therefore, as the number of viewpoints that can be viewed by the user increases, the data amount of the integrated image becomes enormous. is a heavy load. As the load increases, congestion tends to occur in distribution from the image distribution device to the image display device. Congestion increases the time it takes for the delivered multi-view images to become available to viewers, reducing the quality of the viewing experience. Therefore, there is an even greater challenge to avoid increasing the amount of data transfer and to improve the quality of the viewing experience for multi-view video.

In order to prevent an increase in the amount of data transfer, a method of increasing the number of images included in the integrated image while keeping the data amount of the integrated image constant is conceivable. However, with this method, the amount of data for each individual image included in the integrated image must be suppressed. In general, an image with a larger amount of data has a higher image quality, and an image with a smaller amount of data has a lower image quality. Therefore, if the amount of data per individual image is suppressed, the image quality of each individual image will be low. In addition, if the image quality of the individual images is low, it will lead to a deterioration in the quality of the viewing experience for the viewer.

In addition, reducing delays when changing viewpoints is also important in improving the quality of viewing experience for multi-view video. If the delay in changing viewpoints is large, the viewer will have a longer time before the images from the changed viewpoint are available, and the quality of the viewing experience will be degraded.

From the above, in order to provide high-quality multi-view video, increasing the number of viewpoints, preventing an increase in the amount of data transfer, preventing deterioration in image quality, and reducing delay when changing viewpoints are mutually contradictory. It is required to meet both demands.

The present invention has been made in view of such circumstances. It is an object of the present invention to provide an image distribution device, an image generation device, and a program capable of reconciling conflicting demands with delay reduction when changing a viewpoint.

As a result of intensive studies to solve the above problems, the present inventors have found that, when a viewpoint designation operation is performed, a group of images obtained by viewing the same scene from different viewpoints is viewed from a predetermined viewpoint related to the designation operation. We have found that the above object can be achieved by externally distributing an integrated image in which a predetermined viewpoint image and peripheral viewpoint images seen from peripheral viewpoints located around the predetermined viewpoint are arranged in a single frame. I have completed my invention. Specifically, the present invention provides the following.

According to a first aspect of the invention, there is provided a receiving unit for externally receiving designation information indicating that an operation for designating a predetermined viewpoint has been performed from a plurality of different viewpoints for photographing the same scene, and a predetermined viewpoint viewed from the predetermined viewpoint. a distribution unit that distributes to the outside an integrated image in which a viewpoint image and one or a plurality of peripheral viewpoint images viewed from peripheral viewpoints positioned around the predetermined viewpoint among the plurality of viewpoints are integrated into one frame; To provide an image delivery device comprising:

According to the first aspect of the invention, the integrated image to be distributed to the outside is a predetermined viewpoint designated as a viewpoint desired to be viewed by the viewer's operation, among a plurality of different viewpoints for photographing the same scene. It includes a predetermined viewpoint image viewed from (desired viewing viewpoint). Therefore, an external device (for example, an image display device, etc.) to which the integrated image is delivered can reproduce the predetermined viewpoint image viewed from the viewpoint actually specified by the viewer.

Further, in addition to the predetermined viewpoint image, the integrated image includes one or a plurality of peripheral viewpoint images viewed from peripheral viewpoints positioned around the predetermined viewpoint, which are integrated in one frame. Distribute images externally. Therefore, when the external device performs switching of the viewpoint in the image being displayed (for example, switching from the viewpoint in the image being displayed to the viewpoint desired by the viewer), the image display at the viewpoint after switching is performed by the external device. It can be realized by internal processing of the device.

More specifically, after the viewer wishes to display an image viewed from a display viewpoint different from a predetermined viewpoint around the predetermined viewpoint, the viewer can use the integrated image corresponding to the predetermined viewpoint specified by the designation information. There is a waiting time before the video is delivered due to the time required for communication related to the delivery, the time required for decoding (decoding processing) of the delivered video, and the like. Since the integrated image to be distributed includes one or a plurality of peripheral viewpoint images seen from peripheral viewpoints positioned around the predetermined viewpoint, even during this waiting time, the viewer may not be able to see the predetermined viewpoints around the predetermined viewpoint. Images viewed from different display viewpoints can be used without delay. Therefore, according to the first aspect of the invention, it is possible to reduce the delay when changing viewpoints, which is a problem when providing multi-view video.

On the other hand, the images included in the integrated image are the predetermined viewpoint image and one or a plurality of peripheral viewpoint images, and are not all the images seen from all of the different viewpoints that capture the same scene. Therefore, the number of images included in the integrated image can be suppressed, and even when the resolution of the images included in the integrated image is relatively high, an increase in the amount of data transferred from the image distribution device to the outside can be prevented. Therefore, it is possible to simultaneously increase the number of viewpoints in multi-view video, prevent an increase in the amount of data transfer, and prevent deterioration in image quality, thereby improving the viewing experience of the viewer.

Therefore, according to the invention according to the first feature, the increase in the number of viewpoints in the multi-view video, the prevention of an increase in the amount of data transfer, the prevention of the deterioration of the image quality, and the reduction of the delay when changing the viewpoint are achieved. It is possible to provide an image delivery device that satisfies conflicting demands.

An invention according to a second feature is the invention according to the first feature, and provides an image delivery device in which the predetermined viewpoint image and the one or more peripheral viewpoint images included in the integrated image have the same resolution. do.

According to the second aspect of the invention, the predetermined viewpoint image and the peripheral viewpoint images included in the integrated image have the same resolution. Therefore, when an external device such as an image display device displays one of the images included in the integrated image, even if the display image on the external device is switched by an operation of designating a predetermined viewpoint, the external device can display images with the same resolution before and after switching without delay by internal processing.

Therefore, according to the second aspect of the invention, it is possible to provide an image delivery apparatus capable of further suppressing deterioration in the image quality of an image displayed by an external device, among the above-described conflicting demands.

An invention according to a third feature is the invention according to the first feature, wherein at least one of the one or more peripheral viewpoint images has a resolution higher than that of the predetermined viewpoint image. The one frame includes one or more first regions in which an image can be arranged and one or more second regions in which an image can be arranged and which has a smaller area than the first region. wherein at least the predetermined viewpoint image is arranged in the first area, and at least the peripheral viewpoint image having a resolution smaller than that of the predetermined viewpoint image is arranged in the second area. Provide equipment.

According to the third aspect of the invention, the integrated images are integrated within one frame, and the area of this one frame itself is constant. Also, the area of the second region is smaller than the area of the first region. Therefore, arranging both the first region and the second region in one frame allows more regions to be arranged in one frame than arranging only the first region in one frame. is placed. As a result, by arranging both the first region and the second region in one frame, compared to the case of arranging only the first region in one frame, peripheral viewpoints that are integrated into the integrated image can be obtained. You can increase the number of images. Therefore, according to the third aspect of the invention, it is possible to increase the number of peripheral viewpoint images that can be displayed without delay during the standby time without increasing the amount of data transfer.

Also, the image viewed by the viewer is a predetermined viewpoint image with relatively high resolution, and this predetermined viewpoint image is arranged in the first area having a relatively large area. Therefore, even if the second area is arranged within one frame that constitutes the integrated image, the resolution of the image itself viewed by the viewer does not decrease. Therefore, according to the third aspect of the invention, it is possible to increase the number of regions arranged in one frame forming the integrated image without lowering the resolution of the predetermined viewpoint image itself viewed by the viewer.

Therefore, according to the third aspect of the invention, it is possible to provide an image distribution apparatus that further meets the conflicting demands described above.

The invention according to a fourth feature is the invention according to the third feature, wherein in addition to the predetermined viewpoint image, the peripheral viewpoint images having the same resolution as the predetermined viewpoint image are arranged in the first region. , provides an image delivery device.

In addition to the predetermined viewpoint image, peripheral viewpoint images having the same resolution as the predetermined viewpoint image are also arranged in the first area having a relatively large area. Therefore, according to the invention according to the fourth feature, the peripheral viewpoint that is predicted to be highly likely to be specified later by the viewer is predicted to be selected from not only the predetermined viewpoint image that the viewer views, but also the plurality of peripheral viewpoint images. Images can be arranged in one frame that constitutes an integrated image without lowering the resolution.

Therefore, according to the fourth aspect of the invention, it is possible to provide an image delivery apparatus that further meets the conflicting demands described above.

The invention according to a fifth feature is the invention according to any one of the first to fourth features, wherein the predetermined viewpoint image and the peripheral viewpoint image are images taken at substantially the same time. An image delivery device is provided.

According to the fifth aspect of the invention, when the external device switches and displays the display image from the predetermined viewpoint image to the peripheral viewpoint image, the times before and after the switching are substantially the same. Therefore, according to the fifth aspect of the invention, when switching images in a multi-viewpoint display technique, a seek process that calls up a video at the same time as the predetermined viewpoint image from the integrated image or a video image at the same time as the predetermined viewpoint image is displayed. There is no need to perform the process of calling the video at the time from the image distribution device. As a result, the delay caused by seek processing or the like can be reduced.

Therefore, according to the fifth aspect of the invention, it is possible to provide an image distribution apparatus capable of further reducing the delay in changing the viewpoint among the conflicting demands described above.

The invention according to the sixth characteristic is different in category from the invention according to the first characteristic.

A seventh feature of the invention is a receiving unit for externally receiving designation information indicating that an operation for designating a predetermined viewpoint has been performed from a plurality of different viewpoints for photographing the same scene, and a predetermined viewpoint viewed from the predetermined viewpoint. a generation unit that generates an integrated image in which viewpoint images and one or more peripheral viewpoint images viewed from peripheral viewpoints positioned around the predetermined viewpoint among the plurality of viewpoints are integrated into one frame. A generator is provided.

According to the seventh aspect of the invention, it is possible to provide an image generating device capable of generating an integrated image suitable for the image delivery device of the first aspect of the invention.

The invention according to an eighth feature is a receiving step of causing an image display device to receive, from the outside, an integrated image in which a plurality of images taken from a plurality of viewpoints of the same scene are integrated into one frame; a display step of displaying one of the plurality of included images; and a viewpoint command step of transmitting designation information indicating that an operation for designating a predetermined viewpoint from the plurality of viewpoints has been performed to the outside. is executed, and when the viewpoint commanding step is executed, in the displaying step, a predetermined viewpoint image viewed from the predetermined viewpoint from among the plurality of images included in the integrated image is processed by the image display device. , and the integrated image received in the receiving step after the viewpoint command step is executed includes the predetermined viewpoint image viewed from the predetermined viewpoint and the surroundings of the predetermined viewpoint among the plurality of viewpoints. To provide a program that is an image in which one or a plurality of peripheral viewpoint images seen from peripheral viewpoints located in a single frame are integrated.

According to the eighth aspect of the invention, the image display device can display a predetermined viewpoint image viewed from a predetermined viewpoint designated from a plurality of viewpoints for photographing the same scene through transmission of designation information. Therefore, it is possible to increase the number of viewpoints of the multi-view video while preventing an increase in the amount of data transfer and deterioration of image quality.

When the predetermined viewpoint is changed to a viewpoint different from the predetermined viewpoints around the predetermined viewpoint, from the transmission of the designation information designating the changed predetermined viewpoint until the integrated image corresponding to the changed predetermined viewpoint can be displayed. , there is a waiting time caused by the time required for communication related to distribution, the time required for decoding (decoding processing) of the distributed video, and the like. Since the integrated image to be received includes one or a plurality of peripheral viewpoint images viewed from peripheral viewpoints positioned around the predetermined viewpoint, the image display device can display images around the predetermined viewpoint before change even during this standby time. An image viewed from a changed predetermined viewpoint different from the predetermined viewpoint can be displayed without delay. Therefore, in addition to increasing the number of viewpoints in a multi-view video, preventing an increase in the amount of data transfer, and preventing deterioration in image quality, it is also possible to reduce the delay when changing viewpoints at the same time. realizable. This can further improve the quality of the viewer's viewing experience.

Therefore, according to the invention according to the eighth feature, the increase in the number of viewpoints of the multi-view video, the prevention of an increase in the amount of data transfer, the prevention of the deterioration of the image quality of the image, and the reduction of the delay when changing the viewpoint can be mutually achieved. It is possible to provide a program that reconciles conflicting demands.

According to the present invention, the conflicting demands of increasing the number of viewpoints in a multi-view video, preventing an increase in the amount of data transfer, preventing deterioration in image quality, and reducing delay when changing viewpoints are achieved. An image distribution device, an image generation device, and a program can be provided.

FIG. 1 is a block diagram showing an example of the configuration of an image delivery system 1 of this embodiment. FIG. 2 is a main flowchart showing an example of the flow of image distribution processing using the image distribution device 2 of this embodiment. FIG. 3 is a conceptual diagram showing an example of a multi-view video. FIG. 4 is a conceptual diagram showing an example of an integrated image. FIG. 5 is an example of the integrated image table 221. As shown in FIG. FIG. 6 is a flow chart showing an example of the integrated image acquisition process flow using the image delivery device 2 of this embodiment. FIG. 7 is an example of the multi-view video table 222. As shown in FIG. FIG. 8 is a flowchart showing an example of the flow of image display processing using the image display device 3 of this embodiment. FIG. 9 is a conceptual diagram of viewpoint movement using an integrated image. FIG. 10 is a block diagram showing an example of the configuration of the image distribution system 1 of Modification 1. As shown in FIG.

An example of a preferred embodiment for carrying out the present invention will be described below with reference to the drawings. This is just an example, and the technical scope of the present invention is not limited to this.

<Image distribution system 1>
FIG. 1 is a block diagram showing an example of the configuration of an image delivery system 1 of this embodiment. An image delivery system 1 (hereinafter simply referred to as system 1) includes an image delivery device 2 (hereinafter also simply referred to as delivery device 2), an image display device 3 (hereinafter also simply referred to as display device 3), and a group of photographing devices. 4 and network 5 (first network 5A and second network 5B).

[Distribution device 2]
The distribution device 2 includes a control section 21 , a storage section 22 and a communication section 23 .

[Control unit 21]
The control unit 21 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like.

The control unit 21 reads a predetermined program, and cooperates with the storage unit 22 and/or the communication unit 23 as necessary to perform the specified information receiving unit 211 and the distribution unit, which are software configuration elements in the distribution device 2. 212, generation unit 213, and the like.

[Storage unit 22]
The storage unit 22 is a device for storing data and files, and has a data storage unit such as a hard disk, a semiconductor memory, a recording medium, or a memory card. The storage unit 22 has a mechanism that enables connection with a storage device or storage system such as NAS (Network Attached Storage), SAN (Storage Area Network), cloud storage, file server, distributed file system, etc. via a network. may

The storage unit 22 stores a control program executed by the microcomputer, an integrated image table 221, a multi-view video table 222, and the like.

(Integrated image table 221)
The integrated image table 221 stores data linking viewpoint data and integrated images. In this embodiment, viewpoint data refers to data relating to viewpoints for realizing multi-viewpoint video in which the same scene can be viewed from different viewpoints. The type of viewpoint data is not particularly limited. Data indicating the coordinates, data indicating the angle of view of the viewpoint, data indicating the label assigned to the viewpoint, or data obtained by combining two or more of these data regarding the viewpoint can be used.

In the present embodiment, an integrated image is an image obtained by integrating a plurality of viewpoint images selected according to a predetermined standard from among the plurality of viewpoint images corresponding to each of the plurality of viewpoints provided to realize a multi-view video. Say.

In this embodiment, the integrated image table 221 stores data linking viewpoint data and integrated images, so that the integrated image acquisition unit 212 can obtain a plurality of images provided for realizing multi-view video. Among the viewpoints, the commanded viewpoint image viewed from the commanded viewpoint according to the command from the viewer or the like who views the multi-view video through the image display device 3, and the images from one or more surrounding viewpoints around the commanded viewpoint It is possible to obtain an integrated image that is integrated with one or a plurality of surrounding viewpoint images.

(Multi-viewpoint video table 222)
The multi-viewpoint video table 222 stores data in which viewpoint data is associated with an image viewed from a viewpoint corresponding to the viewpoint data among a plurality of viewpoints provided for realizing a multi-viewpoint video. .

[Communication unit 23]
The communication unit 23 has a device for enabling the distribution device 2 to communicate with the display device 3 and other devices. The type of device is not particularly limited, and includes, for example, a network card compatible with the Ethernet standard, a Wi-Fi (Wireless Fidelity) compatible device conforming to IEEE802.11, and the like. In order to distribute images with a large amount of data, the communication section 23 preferably has a network card compatible with the Ethernet standard.

[Display device 3]
The display device 3 includes a control section 31 , a storage section 32 , a communication section 33 , a display section 34 and an input section 35 .

[Control unit 31]
The hardware configuration of the controller 31 is the same as that of the controller 21 . The control unit 31 reads a predetermined program, and cooperates with the storage unit 32, the communication unit 33, the display unit 34 and/or the input unit 35 as necessary, so that it is an element of the software configuration in the display device 3. It implements an integrated image receiving unit 311, a viewpoint commanding unit 312, an image display unit 313, and the like.

[Storage unit 32]
The hardware configuration of the storage section 32 is the same as that of the storage section 22 . The storage unit 32 stores control programs and the like executed by the microcomputer.

[Communication section 33]
The communication unit 33 has a device for enabling the display device 3 to communicate with the distribution device 2 and other devices. The hardware configuration of the communication section 33 is the same as that of the communication section 23 .

[Display unit 34]
The type of display unit 34 is not particularly limited. Examples of the display unit 34 include a monitor and a touch panel.

[Input unit 35]
The type of input unit 35 is not particularly limited. Examples of the input unit 35 include a keyboard, mouse, touch panel, and the like.

[Imaging device group 4]
The photographing device group 4 includes a plurality of photographing devices that photograph the same scene from different viewpoints.

The imaging devices included in the imaging device group 4 are preferably capable of shooting moving images. The ability of the imaging device to capture moving images can further improve the quality of viewing experience for viewers.

The imaging devices included in the imaging device group 4 are preferably connectable to the distribution device 2 and other devices. The imaging devices included in the imaging device group 4 may be connected to the distribution device 2 and other devices via a network 5, which will be described later, using a cable or the like conforming to the High-Definition Multimedia Interface (HDMI (registered trademark)). may be connected to the distribution device 2 or other devices. Since the photographing devices included in the photographing device group 4 can be connected to the distribution device 2 and other devices, the distribution device 2 can perform live distribution for distributing scenes photographed by the photographing device to the display device 3 in real time. .

It is preferable that the plurality of imaging devices included in the imaging device group 4 have the same upper limit of imaging resolution. Since the upper limit of the resolution that can be photographed by the photographing device is the same, the image quality when the viewpoint is moved can be maintained in the same state as before the movement, so that the quality of the viewing experience can be improved.

[Network 5]
The network 5 includes a first network 5A that connects the distribution device 2 and the display device 3 and a second network 5B that connects the distribution device 2 and the imaging device group 4 .

The types of the first network 5A and the second network 5B are not particularly limited, and are personal area networks, local area networks, intranets, extranets, the Internet, Wi-Fi networks, and bus networks conforming to the Universal Serial Bus (USB). , or a network obtained by combining a plurality of networks including these. The first network 5A and the second network 5B may be networks independent of each other, or may be connected to each other.

[Main flow chart for image distribution]
FIG. 2 is an example of a main flowchart showing the procedure of image distribution according to this embodiment. A preferred procedure of image distribution processing performed by the distribution device 2 will be described below with reference to FIG.

[Step S1: Determine whether or not the specified information has been received]
The control unit 21 cooperates with the storage unit 22 and the communication unit 23 to execute the designation information reception unit 211, and determines whether or not designation information has been received (step S1). If the designation information has been received, the control unit 21 shifts the process to step S2. If the designation information has not been received, the control unit 21 shifts the process to step S3.

The designation information indicates that an operation has been performed to designate a viewing desired viewpoint (predetermined viewpoint related to the designation operation; the same shall apply hereinafter) that the viewer desires to view from a plurality of different viewpoints for shooting the same scene. If there is, it is not particularly limited. The designation information includes information regarding an operation for designating a desired viewing viewpoint. The information related to the operation for specifying the desired viewing viewpoint included in the designation information is not particularly limited, and includes a label indicating the desired viewing viewpoint, a label indicating the imaging device that captures the image viewed from the desired viewing viewpoint, the coordinates of the desired viewing viewpoint, and the like. (coordinates and/or direction) and the like. By determining whether or not the control unit 21 has received the designation information, the control unit 21 can change the desired viewing viewpoint using the designation information when the control unit 21 receives the designation information.

[Step S2: Change Viewing Desired Viewpoint]
The control unit 21 cooperates with the storage unit 22 to execute the designation information reception unit 211, and uses the designation information to change the desired viewing viewpoint (step S2). The control unit 21 shifts the process to step S3. By the processing in step S2, the distribution device 2 can distribute the integrated image corresponding to the viewer's desired viewing viewpoint to the display device 3. You can play back images viewed from

[Step S3: Integrated Image Acquisition Processing]
The control unit 21 cooperates with the storage unit 22 to execute the distribution unit 212, executes integrated image acquisition processing which will be described later with reference to FIG. 6 (step S3), and shifts the processing to step S4. In the integrated image acquisition process, from the integrated image table 221, an integrated image including the desired viewing viewpoint image viewed from the desired viewing viewpoint and one or more peripheral viewpoint images viewed from a plurality of peripheral viewpoints around the desired viewing viewpoint is obtained. is the process of obtaining

The integrated image includes an image from the viewpoint desired to be viewed, and the distribution device 2 distributes the integrated image to the display device 3. Therefore, the display device 3 reproduces the image from the viewpoint desired to be viewed by the viewer. can. In addition to the desired viewing viewpoint image, the integrated image also includes peripheral viewpoint images viewed from one or more peripheral viewpoints around the desired viewing viewpoint. For example, when a user desires to switch from a viewpoint in an image being displayed to a viewpoint desired to be viewed, the display device 3 does not receive data distribution from the distribution device 2, and switches the viewpoint only by internal processing of the display device 3. can be realized.

To explain in more detail, after the viewer wishes to display an image viewed from a viewpoint different from the currently displayed viewpoint, the viewer displays an integrated image corresponding to the different viewpoint (new desired viewing viewpoint). There is a waiting time before it becomes available due to the time required for distribution, the time required for decoding (decoding processing) of the distributed video, and the like. Since the integrated image originally distributed from the distribution device 2 to the display device 3 includes not only the desired viewing viewpoint image, but also peripheral viewpoint images seen from peripheral viewpoints located around the desired viewing viewpoint, the waiting time is reduced. Even if there is, the viewer reads out the new desired viewing viewpoint image from the multiple peripheral viewpoint images that were originally distributed as the images included in the integrated image, and reads the desired viewing viewpoint image without delay (after the waiting time elapses). available without waiting). Therefore, it is possible to reduce the delay when changing viewpoints, which is a problem when providing multi-view video. This reduction in delay when changing viewpoints will be described in more detail later with reference to FIG.

On the other hand, the images included in the integrated image are the desired viewing viewpoint image and one or a plurality of peripheral viewpoint images, and are not all the images viewed from all of the different plural viewpoints of the same scene. Therefore, the number of images included in the integrated image can be suppressed, and even when the resolution of the images included in the integrated image is relatively high, an increase in the amount of data transferred from the distribution device 2 to the display device 3 can be prevented. can. Therefore, it is possible to simultaneously increase the number of viewpoints in multi-view video, prevent an increase in the amount of data transfer, and prevent deterioration in image quality, thereby improving the viewing experience of the viewer.

FIG. 3 is a conceptual diagram showing an example of a multi-view video.

The photographing devices 4A to 4X included in the photographing device group 4 photograph the photographing target T from different viewpoints. As a result, images 4a to 4x viewed from different viewpoints are obtained. The viewpoints may be on substantially the same plane, or may be on different planes. The image may be a still image or a moving image. Preferably, the images are moving images, as this can further enhance the quality of the viewing experience.

A viewpoint may be a virtual viewpoint set between two or more photographing devices selected from the photographing devices 4A to 4X. When the viewpoint is the virtual viewpoint, the image viewed from the virtual viewpoint may be an image captured by any one of these two or more imaging devices, or an image generated using the captured image. It's okay. It is preferable that the plurality of images 4a to 4x obtained by photographing by the plurality of photographing devices 4A to 4X have the same resolution.

FIG. 4 shows an image from the photographing device 4A when the viewing desired viewpoint is the viewpoint viewed from the photographing device 4A among a plurality of viewpoints viewed from the plurality of photographing devices 4A to 4X constituting the photographing device group 4. , and the images from the imaging devices provided around the imaging device 4A.

FIG. 4(a) shows an integrated image 101a when one frame is provided with nine areas (desired viewing viewpoint image 4a and eight peripheral viewpoint images 4u to 4x and 4b to 4e) having the same area. A configuration example is shown.

The integrated image 101a is divided into nine areas, and the images 4u to 4x and 4a to 4e are resized so as to fit in each divided area and arranged in each area of the integrated image 101a. . By configuring the integrated image 101a in this way, the delivery device 2 can deliver not only the view-desired viewpoint image 4a but also the eight peripheral viewpoint images 4u and the like to the display device 3, and the display device 3 can distribute the images included in the integrated image. The desired viewing viewpoint image 4a can be read out from the desired viewing viewpoint image 4a and peripheral viewpoint images 4u, etc., and displayed to the viewer.

When the viewer wishes to switch the desired viewing viewpoint, the display device 3 instructs the distribution device 2 to switch, and the distribution device 2 generates an integrated image including the new desired viewing viewpoint image after the switching. , distributes a new integrated image to the display device 3 . Therefore, when a viewer wishes to switch the desired viewing viewpoint, the time required for communication related to distribution and the decoding of the distributed video from when the switch is instructed to when the viewer can use a new integrated image will take time. A waiting time is required due to the time required for (decryption processing) and the like.

In the present embodiment, the integrated image 101a includes, in addition to the desired viewing viewpoint image 4a, viewpoints that are expected to be specified by the viewer with high probability, that is, images of peripheral viewpoints around the desired viewing viewpoint (peripheral viewpoint images 4u to 4u). 4x and 4b-4e). Therefore, even during the standby time described above, the display device 3 can read an image corresponding to a new viewing desired viewpoint from among the peripheral viewpoint images 4u to 4x and 4b to 4e included in the integrated image 101a, and display it to the viewer. . Therefore, according to this embodiment, it is possible to reduce the delay when changing viewpoints, which is a problem when providing multi-view video.

Desired viewing viewpoint image 4a and surrounding images 4u to 4x and 4b to 4e included in integrated image 101a preferably have the same resolution. If the resolutions are equal to each other, when the viewer wishes to switch the desired viewpoint for viewing, the resolution of the image newly displayed by the display device 3 after switching becomes the same as the resolution of the image before switching, which is given to the viewer. You can reduce stress even more.

FIG. 4(b) shows, in one frame, seven first regions (regions in which the viewing desired viewpoint image 4a and six peripheral viewpoint images 4v to 4x and 4b to 4d are arranged), and 8 shows a configuration example of the integrated image 101b when eight second regions (regions in which eight peripheral viewpoint images 4r to 4u and 4e to 4h are arranged) each having a smaller area are provided.

In the first area, a viewing desired viewpoint image 4a, a viewing desired viewpoint image 4a, and six peripheral viewpoint images 4v to 4x and 4b to 4d are arranged.

The six peripheral viewpoint images 4v to 4x and 4b to 4d are images captured by the imaging devices 4V to 4X and 4B to 4D. is equal to the resolution of Referring to FIG. 3, the photographing devices 4V to 4X and 4B to 4D are installed at positions relatively close to the photographing device 4A corresponding to the desired viewing viewpoint. Therefore, it is predicted that the viewpoints located in the photographing devices 4V to 4X and 4B to 4D are viewpoints with a relatively high possibility of being selected as viewpoints after switching when the viewer wishes to switch the desired viewing viewpoint. be done. According to the present embodiment, among a plurality of peripheral viewpoint images, a peripheral viewpoint image having a relatively high possibility of being selected as a viewpoint after switching is placed in the first region having a relatively large area. placed high on the

On the other hand, eight peripheral viewpoint images 4r to 4u and 4e to 4h are arranged in the second area.

The eight peripheral viewpoint images 4r to 4u and 4e to 4h are images captured by the imaging devices 4R to 4U and 4E to 4H. smaller than the resolution of Referring to FIG. 3, the photographing devices 4R to 4U and 4E to 4H are installed at positions relatively distant from the photographing device 4A corresponding to the desired viewing viewpoint. Therefore, it is predicted that the viewpoints located in the photographing devices 4R to 4U and 4E to 4H are viewpoints with a relatively low possibility of being selected as viewpoints after the switching when the viewer wishes to switch the desired viewing viewpoint. be done. According to the present embodiment, among the plurality of peripheral viewpoint images, a peripheral viewpoint image having a relatively low possibility of being selected as a viewpoint after switching is placed in the second region having a relatively small area. placed low on the

The integrated image 101b is integrated within one frame, and the area of this one frame itself is constant. Also, the area of the second region is smaller than the area of the first region. Therefore, arranging both the first region and the second region in one frame allows more regions to be arranged in one frame than arranging only the first region in one frame. is placed. As a result, by arranging both the first region and the second region in one frame, compared to the case of arranging only the first region in one frame, peripheral viewpoints that are integrated into the integrated image can be obtained. You can increase the number of images. Therefore, according to the present invention, the number of peripheral viewpoint images that can be displayed without delay during the waiting time can be increased without increasing the amount of data transfer.

The image viewed by the viewer is the viewing desired viewpoint image 4a having relatively high resolution, and the viewing desired viewpoint image 4a is arranged in the first area having a relatively large area. Therefore, even if the second area is arranged within one frame forming the integrated image 101b, the resolution of the image itself viewed by the viewer does not decrease. Therefore, according to the invention according to the present embodiment, the number of areas arranged in one frame forming the integrated image 101b can be increased without lowering the resolution of the viewing desired viewpoint image 4a itself viewed by the viewer. .

In addition, peripheral viewpoint images 4v to 4x and 4b to 4d having the same resolution as the viewing desired viewpoint image 4a are also arranged in the first area having a relatively large area, in addition to the viewing desired viewpoint image 4a. Therefore, according to the invention according to the present embodiment, not only the viewing desired viewpoint image 4a to be viewed by the viewer, but also the peripheral viewpoint images among the plurality of peripheral viewpoint images which are predicted to be highly likely to be designated by the viewer thereafter. The viewpoint images 4v to 4x and 4b to 4d can be arranged in one frame forming the integrated image without lowering the resolution.

As described above, according to the invention according to the present embodiment, in providing a high-quality multi-view video, it is possible to increase the number of viewpoints, prevent an increase in the amount of data transfer, prevent a decrease in image quality, and change the viewpoint when changing viewpoints. It is possible to provide the delivery device 2 capable of reconciling conflicting demands with the reduction of the delay.

4(a) and 4(b) show an example of arranging the view-desired viewpoint image 4a and the 8 or 14 peripheral viewpoint images 4u, etc. within one frame constituting the integrated images 101a and 101b. , but not limited to this. The arrangement of the images within one frame that constitutes the integrated image can be configured in an arbitrary manner according to the distribution situation and the like.

FIG. 4(c) shows five first regions (regions in which the viewing desired viewpoint image 4a and four peripheral viewpoint images 4w, 4x, 4b, and 4c are arranged) and 4 shows an example of the configuration of the integrated image 101b when 16 second regions (16 peripheral viewpoint images 4o to 4v and 4d to k) each having a smaller area are provided. For example, the arrangement of the images within one frame that constitutes the integrated image may be as shown in FIG. 4(c).

Also, in FIGS. 4A to 4C, the viewing desired viewpoint image 4a and the plurality of peripheral viewpoint images 4u, etc. are displayed from the upper left to the lower right of the integrated image by the photographing device according to the viewpoint shown in FIG. Although they are arranged along the line, they may be arranged according to the degree of possibility specified by the viewer.

The types of the viewing-desired viewpoint image 4a and the surrounding viewpoint images 4u are not particularly limited, and may be still images or moving images. When the multi-view video is a moving image, it is preferable that the times of the images arranged in one integrated image are substantially the same time. "Substantially the same time" is the time of any two images included in the multiple images arranged in the integrated image, where f is the frame rate of the moving image (the number of still images displayed per second). difference is at most (2/f) seconds. By arranging the images in this way, when the display device 3 switches and displays the display image from the viewing-desired viewpoint image to the peripheral viewpoint image, the time before and after switching becomes substantially the same time. Therefore, when performing image switching in the multi-viewpoint display technology, the display device 3 performs seek processing to call the video at the same time as the desired viewing viewpoint image from the integrated image, or retrieves the video at the same time as the desired viewing viewpoint image. There is no need to perform processing called from the distribution device 2 . As a result, the delay caused by seek processing or the like can be reduced.

FIG. 5 is an example of the integrated image table 221. As shown in FIG. The integrated image table 221 shown in FIG. 5 includes an ID for identifying an integrated image, viewpoint data related to a desired viewing viewpoint, a division number for storing the number of images included in the integrated image, and an integrated image viewed from the desired viewing viewpoint. , contains data associated with In the row of ID1 storing the corresponding integrated image when the viewing desired viewpoint is the viewpoint seen from the photographing device 4A, ID "1" and viewpoint data (photographing device, viewpoint X coordinate, Y coordinate, Z coordinate , the direction of the object to be photographed with respect to the viewpoint, the angle of elevation/depression when the viewpoint is to photograph the object to be photographed, the angle of view when the viewpoint is to photograph the object to be photographed), the number of divisions “9”, the desired viewing viewpoint image 4a, and peripheral viewpoints. Data associated with integrated images including images 4u to 4x and 4b to 4e are stored. The distribution unit 212 can acquire an integrated image by designating one or more of these ID, viewpoint data, and number of divisions. For example, the delivery unit 212 can acquire the integrated image stored in the row of ID1 by designating "an integrated image corresponding to the viewpoint corresponding to the imaging device 4A and including nine images".

As shown in the row of ID101 and the row of ID201, the integrated image table 221 shows an integrated image when a first area and a second area having a smaller area than the first area are provided in one constituting frame. Images may be stored. In addition, the integrated image table 221 stores a plurality of integrated images corresponding to the same viewing desired viewpoint and having different numbers of images included in the integrated image, as shown in rows of ID1, ID101, and ID201. good too. By configuring the integrated image table 221 in this way, the delivery unit 212 can select and deliver an appropriate integrated image according to the situation regarding delivery.

[Step S4: Distribute integrated image]
Return to FIG. The control unit 21 cooperates with the storage unit 22 and the communication unit 23 to execute the distribution unit 212 to distribute the integrated image acquired in step S3 to the display device 3 (step S4). The control unit 21 shifts the process to step S1 and repeats steps S1 to S4. Before distributing the integrated image to the display device 3, the distribution unit 212 preferably performs an encoding process of encoding the integrated image into a data format suitable for image distribution. By performing the encoding process, the distribution device 2 can prevent an increase in data transfer amount. Since the distribution unit 212 distributes the integrated image acquired in step S3 to the display device 3, the display device 3 that has received the distributed integrated image displays the view-desired viewpoint image viewed from the viewpoint actually specified by the viewer. can be displayed.

In step S<b>4 , the distribution unit 212 preferably distributes layout information indicating the layout of the plurality of images in the integrated image to the display device 3 . By distributing the arrangement information, even if the distribution device 2 selects the number of images arranged in the integrated image according to the distribution situation, the display device 3 receives the distributed arrangement information, and furthermore , the desired viewing viewpoint image viewed from the desired viewing viewpoint actually designated by the viewer can be selected and displayed by internal processing using the arrangement information.

In step S<b>4 , the distribution unit 212 preferably distributes viewpoint information indicating a plurality of viewpoints associated with each of the plurality of images in the integrated image to the display device 3 . By distributing the viewpoint information, even if the multi-view video is a multi-view video in which viewpoint coordinates, etc., change during distribution, the display device 3 receives the distributed viewpoint information, and furthermore, A viewing-desired viewpoint image viewed from a viewing-desired viewpoint actually designated by the viewer can be selected and displayed by internal processing using viewpoint information.

The distribution device 2 distributes an integrated image including an image viewed from the commanded viewpoint and two or more images viewed from a plurality of viewpoints around the commanded viewpoint in the procedure from steps S1 to S4. An image distribution device and a program that simultaneously increase the number of , prevent an increase in the amount of data transfer, prevent deterioration of image quality, and maintain image quality and reduce delay when moving the viewpoint. can provide

[Flowchart of Integrated Image Acquisition Processing]
FIG. 6 is an example of a flowchart showing the procedure of integrated image acquisition processing executed in step S3 of the processing of the distribution device 2 in this embodiment.

[Step S11: Acquisition of image viewed from desired view point]
The control unit 21 cooperates with the storage unit 22 to execute the generation unit 213, and acquires the desired viewing viewpoint image viewed from the desired viewing viewpoint from the multi-view video table 222 (step S11). The control unit 21 shifts the process to step S12. When the generation unit 213 acquires the viewpoint image desired to be viewed, the distribution device 2 can arrange the viewpoint image desired to be viewed in the integrated image.

FIG. 7 is an example of the multi-view video table 222. As shown in FIG. The multi-view video table 222 shown in FIG. 7 stores data in which IDs for identifying images, viewpoint data related to viewpoints, and integrated images viewed from the viewpoints are linked. In the row of ID1 storing the image 4a corresponding to the viewpoint seen from the photographing device 4A, ID "1" and viewpoint data related to the viewpoint seen from the photographing device 4A (photographing device, viewpoint X coordinate, Y coordinate, Z coordinates, direction of the object to be photographed with respect to the viewpoint, angle of elevation/depression when the viewpoint is to photograph the object to be photographed, angle of view when the object is to be photographed by the viewpoint), and data linking the image 4a is stored. The generation unit 213 can acquire an image by designating one or more of these IDs and viewpoint data. For example, the delivery unit 212 can acquire the image 4a stored in the row of ID1 by designating the "image seen from the viewpoint seen from the photographing device 4A".

[Step S12: Acquiring images seen from peripheral viewpoints located around the desired viewing viewpoint]
The control unit 21 cooperates with the storage unit 22 to execute the generation unit 213, and acquires a plurality of peripheral viewpoint images viewed from a plurality of peripheral viewpoints positioned around the desired viewing viewpoint from the multi-view video table 222 ( step S12). The control unit 21 shifts the process to step S13. When the generation unit 213 acquires one or more peripheral viewpoint images, the distribution device 2 can arrange the peripheral viewpoint images in the integrated image.

[Step S13: Generate Integrated Image]
Return to FIG. The control unit 21 cooperates with the storage unit 22 to execute the generation unit 213, and integrates the viewing desired viewpoint image acquired in step S11 and the peripheral viewpoint images acquired in step S12 into one frame. An integrated image is generated and stored in the integrated image table 221 (step S13). The control unit 21 shifts the process to step S14.

[Step S14: Obtain integrated image]
The control unit 21 cooperates with the storage unit 22 to execute the generation unit 213, and acquires the integrated image generated in step S13 from the integrated image table 221 (step S14). The control unit 21 ends the integrated image acquisition process and returns the process to step S3.

[Flowchart of Image Display Processing]
FIG. 8 is an example of a flowchart showing the procedure of image display processing performed by the display device 3 in this embodiment. A preferred procedure of image display processing performed by the display device 3 will be described below with reference to FIG.

[Step S21: Receive Integrated Image]
First, the control unit 31 cooperates with the storage unit 32 and the communication unit 33 to execute the integrated image receiving unit 311 and receive the integrated image from the outside (step S21). The control unit 31 shifts the process to step S22. By receiving the integrated image, the display device 3 can output multi-viewpoint video using the integrated image.

Integrated image receiving section 311 may further receive arrangement information indicating the arrangement of the plurality of images within the integrated image. By receiving the layout information, the display device 3 receives the distributed layout information and further , the desired viewing viewpoint image viewed from the desired viewing viewpoint actually designated by the viewer can be selected and displayed by internal processing using the arrangement information.

The integrated image receiving unit 311 may further receive viewpoint information indicating a plurality of viewpoints associated with each of the plurality of images in the integrated image. By receiving the viewpoint information, even if the multi-view video is a multi-view video in which viewpoint coordinates or the like changes during distribution, the display device 3 receives the distributed viewpoint information, and further, A viewing-desired viewpoint image viewed from a viewing-desired viewpoint actually designated by the viewer can be selected and displayed by internal processing using viewpoint information.

[Step S22: Determining whether or not an operation for designating a desired viewing viewpoint has been performed]
The control unit 31 cooperates with the storage unit 32 and the input unit 35 to execute the viewpoint instruction unit 312, and determines whether or not an operation to designate a desired viewing viewpoint (desired viewing viewpoint) has been performed via the input unit 35. Determine (step S22). If an operation to designate a desired viewing viewpoint (desired viewing viewpoint) has been performed, the control unit 31 shifts the process to step S23. If an operation to designate a desired viewing viewpoint (desired viewing viewpoint) has not been performed, the control unit 31 shifts the process to step S25.

The operation for specifying the desired viewing viewpoint via the input unit 35 is not particularly limited, and may be, for example, clicking an icon indicating the desired viewing viewpoint with a mouse, performing key input from a keyboard, or tapping an icon indicating the desired viewing viewpoint using a touch panel. A desired viewing viewpoint can be specified using the well-known technique of . The viewpoint commanding unit 312 determines whether or not an operation for designating a desired viewing viewpoint has been performed, so that the display device 3, when an operation for designating a desired viewing viewpoint has been performed, selects a desired viewing viewpoint for use in displaying an image. A series of processes for changing the viewpoint and transmitting the designation information to the distribution device 2 can be performed.

[Step S23: Change Viewing Desired Viewpoint]
The control unit 31 cooperates with the storage unit 32 to execute the viewpoint instruction unit 312, and changes the desired viewing viewpoint according to the operation performed in step S22 (step S23). The control unit 31 shifts the process to step S24. The viewpoint instruction unit 312 changes the desired viewing viewpoint according to the operation performed in step S22, so that the display device 3 can display an image viewed from the new desired viewing viewpoint according to the viewer's operation.

[Step S24: Send specified information]
The control unit 31 cooperates with the storage unit 32 to execute the viewpoint instruction unit 312, and transmits designation information indicating that an operation for designating a desired viewing viewpoint has been performed to the distribution device 2 (step S24). The control unit 31 shifts the process to step S25.

The designation information is not particularly limited as long as it indicates that an operation has been performed to designate a desired viewing viewpoint from a plurality of different viewpoints for shooting the same scene. The designation information includes information regarding an operation for designating a desired viewing viewpoint. The information related to the operation of specifying the desired viewing viewpoint included in the specification information is not particularly limited, and includes a label indicating the specified desired viewing viewpoint, a label indicating the imaging device that captures the image viewed from the specified desired viewing viewpoint, Coordinates and the like (coordinates and/or direction) of the designated viewing desired viewpoint and the like can be mentioned. Since the display device 3 transmits the designation information to the distribution device 2, the distribution device 2 changes the desired viewing viewpoint in response to the viewer performing an operation for designating the desired viewing viewpoint, and changes the desired viewing viewpoint to the specified desired viewing viewpoint. can deliver an integrated image equivalent to

[Step S25: Acquire an image viewed from a desired viewing viewpoint from an integrated image]
The control unit 31 cooperates with the storage unit 32 to execute the image display unit 313, and selects the desired viewing viewpoint image from the desired viewing viewpoint from among the plurality of images included in the integrated image received in step S21. Then, internal processing is performed to acquire the data (step S25). The control unit 31 shifts the process to step S26.

The internal processing of selecting and acquiring the desired viewing viewpoint image from the desired viewing viewpoint from among the multiple images included in the integrated image is not particularly limited, and viewing is performed from the integrated image based on a predetermined image arrangement. Internal processing for clipping a desired viewpoint image, and internal processing for clipping a desired viewpoint image from an integrated image using received arrangement information and/or viewpoint information. The image display unit 313 performs internal processing for selecting and acquiring a desired viewing viewpoint image viewed from a desired viewing viewpoint (desired viewing viewpoint), so that the display device 3 can display the desired viewing viewpoint image.

[Step S26: Display an image viewed from a desired view point]
The control unit 31 cooperates with the storage unit 32 and the display unit 34 to execute the image display unit 313, and displays the viewing desired viewpoint image acquired in step S25 on the display unit 34 (step S26). The control unit 31 returns the process to step S1 and repeats steps S21 to S26. By outputting the desired viewing viewpoint image acquired in step S25, the display device 3 can display the image viewed from the desired viewing viewpoint. In step S<b>26 , the image display unit 313 preferably changes the size of the desired viewing viewpoint image to a size suitable for the display unit 34 and displays the image on the display unit 34 . The image display unit 313 changes the size of the view-desired viewpoint image to a size suitable for the display unit 34 and displays it on the display unit 34 . can be kept constant to further improve the quality of the viewing experience.

In the above-described procedure from step S21 to step S26, a receiving step of causing the display device 3 to externally receive an integrated image in which a plurality of images taken from a plurality of viewpoints of the same scene are integrated into one frame; a display step of displaying one of the plurality of images included in the display, a viewpoint command step of transmitting designation information indicating that an operation has been performed to designate a desired viewing viewpoint from a plurality of viewpoints, In order to execute the above, it is possible to simultaneously increase the number of viewpoints in multi-view video, prevent an increase in the amount of data transfer, prevent deterioration of image quality, and reduce the delay when changing viewpoints. We can provide a program to

<Usage example of image distribution system>
Next, a usage example of the system 1 according to this embodiment will be described.

According to this embodiment, the distribution device 2 is configured to be connectable to the display device 3 via the first network 5A.

[Distribution of images]
A user, an administrator, or another program executed by the distribution device 2 commands the distribution device 2 to distribute the multi-view video to the display device 3 . The distribution device 2 executes the designation information receiving unit 211 and changes the desired viewing viewpoint if the designation information is received from the display device 3 . The distribution device 2 executes the distribution unit 212 to generate an integrated image in which the desired viewing viewpoint image of the same scene viewed from the desired viewing viewpoint and peripheral viewpoint images viewed from peripheral viewpoints around the desired viewing viewpoint are arranged in one frame. is acquired from the integrated image table 221 , and the acquired integrated image is distributed to the display device 3 .

[Image playback]
The display device 3 executes the integrated image receiving unit 311 and receives the integrated image from the distribution device 2 (receiving step). The display device 3 executes the viewpoint commanding unit 312, determines whether or not an operation for designating a desired viewing viewpoint has been performed, and changes the desired viewing viewpoint if an operation for designating a desired viewing viewpoint has been performed. , the designation information is transmitted to the distribution device 2 (viewpoint instruction step). The display device 3 executes the image display unit 313, acquires the desired viewing viewpoint image seen from the desired viewing viewpoint from the integrated image, and displays it on the display unit 34 (display step).

FIG. 9 is a conceptual diagram of viewpoint movement using an integrated image.

Photographing devices 4A to 4F and photographing devices 4R to 4X indicated by broken-line rectangles corresponding to a plurality of rows shown from top to bottom in FIG. Show the device. A desired viewing viewpoint V on the display device 3 indicated by a black circle indicates a desired viewing viewpoint on the display device 3 . It can be seen from the black circles and broken lines that the desired viewing viewpoint V moves as time progresses from left to right in FIG. Integrated images 101 to 110 indicated by solid-line rectangles corresponding to a plurality of columns shown from left to right in FIG. 1 shows viewpoints corresponding to a viewing desired viewpoint image and a plurality of peripheral viewpoint images included in . Hereinafter, the "viewpoint seen from the photographing device (code)" is also simply referred to as "viewpoint (code)". For example, "viewpoint 4A" indicates "viewpoint viewed from photographing device 4A".

Hereinafter, it will be described with reference to FIG. 9 how the system 1 of the present embodiment can reduce image display delay before and after the viewpoint is moved when the viewpoint is moved.

When the integrated image receiving unit 311 receives the integrated image 101 in step S21, the integrated image 101 overlaps rectangles corresponding to the imaging devices 4U to 4X and 4A to 4E, respectively. This indicates that integrated image 101 includes images 4u-4x and 4a-4e corresponding to imaging devices 4U-4X and 4A-4E. In addition, in the rectangle of the integrated image 101, the black circle of the desired viewing viewpoint V is inside the rectangle of the imaging device 4A. This indicates that the desired viewing viewpoint V is the viewpoint viewed from the photographing device 4A. As shown in FIG. 9, the integrated image 101 includes an image viewed from the image capturing device 4A and images viewed from the image capturing devices 4U to 4X and 4B to 4E positioned around the image capturing device 4A. Since the operation to designate the desired viewing viewpoint has not been performed, the image display unit 313 acquires the image 4a viewed from the viewpoint 4A in step S25, and displays the acquired image 4a in step S26.

When integrated image receiving unit 311 receives integrated image 102 in step S<b>21 , integrated image 102 includes an image viewed from the same viewpoint as integrated image 101 . Then, in step S22, an operation for designating the desired viewing viewpoint is performed, and in step S23, the desired viewing viewpoint V is changed to the viewpoint 4X. In step S<b>24 , the viewpoint commanding unit 312 transmits to the distribution device 2 designation information indicating that an operation for designating the viewpoint 4X has been performed. Since the desired viewing viewpoint V has been changed to the viewpoint 4X, the image display unit 313 acquires the image 4x viewed from the viewpoint 4X from the integrated image in step S25. Then, in step S26, the image display unit 313 displays the image 4x. Since integrated image 102 includes image 4a viewed from viewpoint 4A and images 4u to 4x and 4b to 4e viewed from viewpoints 4U to 4X and 4B to 4E positioned around viewpoint 4A, the desired viewing viewpoint is changed to viewpoint 4X. image 4x viewed from viewpoint 4X can be displayed. Since the image 4x viewed from the viewpoint 4X is included in the distributed integrated image 102, the image display unit 313 can display the image 4x viewed from the viewpoint 4X without delay.

In the example shown in FIG. 9, there is a time lag between the change in the desired viewing viewpoint V and the change in the range of viewpoints included in the integrated image. The desired viewing viewpoint V is changed for the first time at a location corresponding to the integrated image 102 , but the range of viewpoints included in the integrated image is changed for the first time at a location corresponding to the integrated image 104 . That is, there is a delay before the display device 3 can use the integrated image distributed in response to the change in the desired viewing viewpoint V. FIG. However, since each integrated image includes a desired viewing viewpoint image viewed from the desired viewing viewpoint and peripheral viewpoint images viewed from peripheral viewpoints around the desired viewing viewpoint, the desired viewing viewpoint V is the viewpoints 4A, 4X, 4W, and 4V. , 4W, 4X, 4A, and 4B, the images viewed from these viewpoints are included in the corresponding integrated images 101 to 110, and the display device 3 displays the viewpoints 4A, 4X, 4W, Images viewed from 4V, 4W, 4X, 4A, and 4B can be displayed without delay. That is, even if there is a delay before the viewer can use the integrated image based on the changed desired viewing viewpoint, the display device 3 can display the image viewed from the desired viewing viewpoint without delay.

Therefore, according to the system 1 according to the present embodiment, in providing high-quality multi-view video, the number of viewpoints can be increased, the amount of data transferred can be prevented from increasing, the image quality can be prevented from deteriorating, and the viewpoint can be changed. It is possible to provide an image distribution device, an image generation device, and a program that are capable of reconciling conflicting demands with delay reduction when performing.

<Modification>
Various modifications of the invention described in this embodiment will be exemplified below.

[Modification 1] (Data distribution for executing processing for generating an integrated image outside the image distribution device)
In the system 1 of the basic embodiment, the process of generating the integrated image is executed by the generation unit 213 included in the distribution device 2, but the process of generating the integrated image of the present invention is executed outside the distribution device 2. good too.

FIG. 10 is a block diagram showing an example of the configuration of the system 1 of this modified example. The system 1 of this modification includes a distribution device 2 that performs image distribution processing, etc., a display device 3, a group of photographing devices 4, a network 5, and an image generating device 6 that performs processing such as generating an integrated image (hereinafter referred to as Also simply referred to as generator 6).

[Distribution device 2]
The distribution device 2 of this modification includes a control section 21 , a storage section 22 and a communication section 23 .

[Control unit 21]
The hardware configuration of the controller 21 of this modified example is the same as that of the controller 21 of the basic embodiment. The control unit 21 reads a predetermined program, and cooperates with the storage unit 22 and/or the communication unit 23 as necessary to perform the specified information receiving unit 211 and the distribution unit, which are software configuration elements in the distribution device 2. 212 and the like.

[Storage unit 22]
The hardware configuration of the storage unit 22 of this modified example is the same as that of the storage unit 22 of the basic embodiment. The storage unit 22 stores a control program executed by a microcomputer, an integrated image table 221, and the like.

[Communication unit 23]
The hardware configuration of the communication unit 23 of this modified example is the same as that of the communication unit 23 of the basic embodiment.

[Display device 3]
The display device 3 of this modified example is the same as the display device 3 of the basic embodiment.

[Imaging device group 4]
The photographing device group 4 of this modified example is the same as the photographing device group 4 of the basic embodiment.

[Network 5]
The network 5 of this modification is similar to the network 5 of the basic embodiment. A network 5A connects the distribution device 2 and the display device 3 . A network 5B connects the distribution device 2, the group of photographing devices 4, and the generation device 6. FIG.

[Generation device 6]
The generation device 6 of this modification includes a control unit 61 , a storage unit 62 and a communication unit 63 .

[Control section 61]
The hardware configuration of the controller 61 of this modification is the same as that of the controller 21 . The control unit 61 reads a predetermined program and cooperates with the storage unit 62 and/or the communication unit 63 as necessary to implement the generation unit 611 and the like, which are software configuration elements in the generation device 6 .

[Storage unit 62]
The hardware configuration of the storage unit 62 of this modification is the same as that of the storage unit 22 . The storage unit 62 stores a control program executed by the microcomputer, an integrated image table 621, a multi-view video table 622, and the like.

[Communication unit 63]
The hardware configuration of the communication unit 63 of this modified example is the same as that of the communication unit 23 . The communication unit 63 has a device for enabling the generation device 6 to communicate with the distribution device 2 and other devices.

[Procedure of Image Distribution Processing]
The image distribution process performed by the distribution device 2 of this modification is such that the control unit 21 instructs the generation device 6 to generate an integrated image instead of performing the processing of steps S11 to S13 in step S3. It is the same as the delivery device 2 of the basic embodiment except that a series of processing for storing the integrated image obtained in the integrated image table 221 is performed.

[Procedure of Processing for Generating Integrated Image]
The generating device 6 of this modified example generates an integrated image in the same procedure as the flowchart shown in FIG. However, the processing performed by the control unit 21 by executing the generation unit 213 is performed by the control unit 61 by executing the generation unit 611 . The integrated image table 221 and the multi-view video table 222 stored in the storage unit 22 are processed using the integrated image table 621 and the multi-view video table 622 stored in the storage unit 62 . Processing related to the communication unit 23 is performed using the communication unit 63 . Instead of performing the process of step S14, a process of distributing the generated integrated image to the distribution device 2 is performed.

By executing the processing for generating the integrated image by the generation device 6 outside the distribution device 2, the amount of processing executed by the control unit 21 is reduced, and the control unit 21 can perform processing related to image distribution at a higher speed. can run to Therefore, the delay when moving the viewpoint is reduced, and the viewer's viewing experience is further improved.

The generation device 6 can also generate an integrated image by performing processing for generating an integrated image in advance. If the generation device 6 stores an integrated image generated in advance in the integrated image table 621, the generation device 6 transfers the integrated image stored in the integrated image table 621 to the integrated image table 221, so that the distribution device 2 can: Processing related to the acquisition of the integrated image can be executed much faster. This reduces the delay in moving the viewpoint, further improving the viewer's viewing experience. A method for transferring the integrated image stored in the integrated image table 622 to the integrated image table 221 is not particularly limited. As a method of transferring the integrated image stored in the integrated image table 621 to the integrated image table 221, for example, a method of distributing the integrated image stored in the integrated image table 621 to the distribution device 2 using the second network 5B, an integrated image A method of storing the integrated image stored in the table 621 in an easily removable storage medium such as a magnetic storage device or an optical medium, removing this storage medium from the generation device 6, attaching it to the distribution device 2, and transferring it to the integrated image table 221. , etc.

In the system 1 of this modified example, the generation device 6 outside the distribution device 2 executes the process of generating the integrated image. Processing related to image distribution can be executed at higher speed. Therefore, the delay when moving the viewpoint is reduced, and the viewer's viewing experience is further improved. In addition, since the generating device 6 of this modified example can also generate an integrated image by performing processing for generating an integrated image in advance, it is possible to execute processing related to acquisition of an integrated image at a higher speed. This reduces the delay in moving the viewpoint, further improving the viewer's viewing experience.

[Modification 2] (Image delivery system in which a group of photographing devices is separate)
Although the system 1 of the basic embodiment and modified example 1 includes the imaging device group 4 , the system 1 may be configured not to include the imaging device group 4 . The system 1 of this modified example has the same configuration as the system 1 of the basic embodiment or the system 1 of the modified example 1, but does not include the image capturing device group 4. An image generated by an external device using this technology is transferred to the multi-view video table 222 or the multi-view video table 622 to generate an integrated image. The means for transferring images to the multi-viewpoint video table 222 (622) is not particularly limited. Images may be transferred via various networks, or may be transferred via easily removable storage media such as magnetic storage devices and optical media.

Although the embodiments and various modifications of the present invention have been described above, the present invention is not limited to these embodiments and various modifications described above. Further, the effects described in the embodiments and various modifications of the present invention are merely enumerations of the most suitable effects resulting from the present invention, and the effects of the present invention are not limited to the embodiments and various modifications of the present invention. It is not limited to what is described.

Moreover, the above-described embodiments are described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the described configurations. Also, part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. .

1 image delivery system 2 image delivery device 21 control unit 211 designation information reception unit 212 delivery unit 213 generation unit 22 storage unit 221 integrated image table 222 multi-view video table 23 communication unit 3 image display device 31 control unit 311 integrated image reception unit 312 Viewpoint command unit 313 Image display unit 32 Storage unit 33 Communication unit 34 Display unit 35 Input unit 4 Photographing device group 4A to 4X Photographing devices 4a to 4x Image 5 Network 5A First network 5B Second network 6 Image generation device 61 Control unit 611 Generation unit 62 Storage unit 621 Integrated image table 622 Multi-view video table 63 Communication units 101 to 110, 101a, 101b, 101c Integrated image T Shooting target V Desired viewing viewpoint

Claims (6)

a receiving unit that externally receives designation information indicating that an operation has been performed to designate a predetermined viewpoint from a plurality of different viewpoints for shooting the same scene;
An integrated image in which a predetermined viewpoint image viewed from the predetermined viewpoint and one or a plurality of peripheral viewpoint images viewed from peripheral viewpoints positioned around the predetermined viewpoint among the plurality of viewpoints are integrated into one frame. and at least the peripheral viewpoint image arranged next to the predetermined viewpoint image among the one or more peripheral viewpoint images can be selected with an area equal to the predetermined viewpoint image after switching the predetermined viewpoint. a distribution unit that distributes the integrated image integrated within one frame to the outside;
with
resolution of any one or more of the one or more peripheral viewpoint images is lower than the resolution of the predetermined viewpoint image;
The one frame is provided with one or more first areas in which an image can be arranged and one or more second areas in which an image can be arranged and which are smaller in area than the first area. cage,
At least the predetermined viewpoint image is arranged in the first area,
The image delivery device , wherein at least the peripheral viewpoint images having a resolution smaller than that of the predetermined viewpoint image are arranged in the second area . 2. The image distribution apparatus according to claim 1, wherein in addition to the predetermined viewpoint image, the peripheral viewpoint images having the same resolution as the predetermined viewpoint image are arranged in the first area. 3. The image delivery device according to claim 1 , wherein said predetermined viewpoint image and said peripheral viewpoint images are images taken at substantially the same time . to the image distribution device,
a receiving step of externally receiving designation information indicating that an operation has been performed to designate a predetermined viewpoint from a plurality of different viewpoints for photographing the same scene;
An integrated image in which a predetermined viewpoint image viewed from the predetermined viewpoint and one or a plurality of peripheral viewpoint images viewed from peripheral viewpoints positioned around the predetermined viewpoint among the plurality of viewpoints are integrated into one frame. and at least the peripheral viewpoint image arranged next to the predetermined viewpoint image among the one or more peripheral viewpoint images can be selected with an area equal to the predetermined viewpoint image after switching the predetermined viewpoint. a delivery step of delivering the integrated image integrated into one frame to the outside;
and
resolution of any one or more of the one or more peripheral viewpoint images is lower than the resolution of the predetermined viewpoint image;
The one frame is provided with one or more first areas in which an image can be arranged and one or more second areas in which an image can be arranged and which are smaller in area than the first area. cage,
At least the predetermined viewpoint image is arranged in the first area,
A program according to claim 1, wherein at least the peripheral viewpoint images having a resolution smaller than that of the predetermined viewpoint image are arranged in the second area. a receiving unit that externally receives designation information indicating that an operation has been performed to designate a predetermined viewpoint from a plurality of different viewpoints for shooting the same scene;
An integrated image in which a predetermined viewpoint image viewed from the predetermined viewpoint and one or a plurality of peripheral viewpoint images viewed from peripheral viewpoints positioned around the predetermined viewpoint among the plurality of viewpoints are integrated into one frame. and at least the peripheral viewpoint image arranged next to the predetermined viewpoint image among the one or more peripheral viewpoint images can be selected with an area equal to the predetermined viewpoint image after switching the predetermined viewpoint. a generation unit that generates the integrated image integrated within one frame;
with
resolution of any one or more of the one or more peripheral viewpoint images is lower than the resolution of the predetermined viewpoint image;
The one frame is provided with one or more first areas in which an image can be arranged and one or more second areas in which an image can be arranged and which are smaller in area than the first area. cage,
At least the predetermined viewpoint image is arranged in the first area,
The image generating device , wherein at least the peripheral viewpoint images having a resolution smaller than that of the predetermined viewpoint image are arranged in the second region . to the image display device,
a receiving step of externally receiving an integrated image in which a plurality of images taken from a plurality of viewpoints of the same scene are integrated into one frame;
a display step of displaying one of the plurality of images included in the integrated image;
a viewpoint command step of transmitting designation information indicating that an operation to designate a predetermined viewpoint from the plurality of viewpoints has been performed to the outside;
and
When the viewpoint command step is executed, the display step selects and displays a predetermined viewpoint image viewed from the predetermined viewpoint from among the plurality of images included in the integrated image by internal processing of the image display device. let
The integrated image including the predetermined viewpoint image to be selected and displayed by the internal processing after the viewpoint command step is executed includes the predetermined viewpoint image viewed from the predetermined viewpoint and the predetermined viewpoint among the plurality of viewpoints. an image in which one or a plurality of peripheral viewpoint images seen from peripheral viewpoints positioned around the is integrated in the one frame, wherein at least the one or more peripheral viewpoint images are arranged next to the predetermined viewpoint image. an image obtained by integrating the peripheral viewpoint images to be selected in the one frame so that they can be selected after the viewpoint instruction step is executed with an area equal to the predetermined viewpoint image;
resolution of any one or more of the one or more peripheral viewpoint images is lower than the resolution of the predetermined viewpoint image;
The one frame is provided with one or more first areas in which an image can be arranged and one or more second areas in which an image can be arranged and which are smaller in area than the first area. cage,
At least the predetermined viewpoint image is arranged in the first area,
At least the peripheral viewpoint images having a resolution smaller than that of the predetermined viewpoint image are arranged in the second region .
program .