JP2017041707A - Reproduction device, information processing method, and communication system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent distribution of multiple pieces of moving image data obtained with a plurality of cameras from being delayed and thereby to allow a reproduction device to reproduce moving image data obtained by synthesizing the multiple pieces of moving image data.SOLUTION: A reproduction device obtains display range information indicating a specified display range of an integrated photographing range with a plurality of photographing ranges integrated, obtains positional relation information indicating a positional relation of the plurality of photographing ranges, determines one photographing range of the plurality of photographing ranges within the display range that is any one of the overlapping photographing ranges in an overlapping range where at least two photographing ranges overlap, as a photographing range to be used to display a moving image in the overlapping range, receives divided moving image data indicating a moving image in a divided range in the determined photographing range, as divided moving image data indicating the moving image in the division region overlapping with the overlapping range, synthesizes multiple pieces of the received division moving image data to generate synthesized moving image data, and reproduces the synthesized moving image data.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a technical field of a communication system in which moving image data is transmitted from a transmitting device to a reproducing device via a network, and the reproducing device reproduces the moving image data.

  Patent Document 1 discloses a technique in which a server divides moving image data captured by a camera, and a client terminal receives and reproduces the divided moving image data. Specifically, the encoding server generates low resolution video data, high resolution video data, etc. from the original video data from the camera, and divides the image frame area of the high resolution video data into a plurality of divided areas. A plurality of divided moving image data corresponding to the plurality of divided regions is generated. The client terminal receives the low-resolution moving image data from the distribution server and also receives the divided moving image data corresponding to the divided area overlapping the drawing area according to the user operation from the distribution server. Then, the client terminal displays a moving image based on the low-resolution moving image data and the high-resolution divided moving image data.

JP 2015-49641 A

  By the way, in order to increase the resolution of the moving image displayed by the playback device, shooting is performed with a plurality of cameras, and a plurality of moving images from a plurality of cameras are overlapped with a moving portion in a region where the shooting ranges of the plurality of cameras overlap. It is conceivable that the image is combined and displayed on the playback device. However, if the server device performs such a composition process, the processing load on the server device increases, and there is a risk that delivery of moving image data will be delayed. Further, for example, the server device generates a plurality of divided moving image data from the moving image data of each of the plurality of cameras, and transmits the divided moving image data corresponding to the divided region overlapping the display region specified by the user to the reproducing device. It is conceivable that the divided moving image data is synthesized and reproduced. In this case, the server device may perform basically the same processing as when shooting with one camera. However, if all of the divided moving image data in the region where the shooting ranges of a plurality of cameras overlap is transmitted to the playback device, the load on the network increases and the distribution of the moving image data may be delayed.

  The present invention has been made in view of the above points, and prevents a playback apparatus from playing back a plurality of synthesized video data by preventing a delay in delivery of a plurality of video data shot by a plurality of cameras. It is an object of the present invention to provide a reproducing apparatus, an information processing method, and a communication system that can be used.

  In order to solve the above problem, the invention according to claim 1 is characterized in that, for each of a plurality of moving image data shot by a plurality of cameras, a plurality of shooting ranges of the camera that shot the moving image data are divided. In a reproducing apparatus that receives divided moving image data transmitted via a network from a transmitting device that generates a plurality of divided moving image data indicating moving images of divided areas, an integrated shooting range obtained by integrating a plurality of shooting ranges of the plurality of cameras Among them, display range information acquisition means for acquiring display range information indicating the specified display range, positional relationship information acquisition means for acquiring positional relationship information indicating the positional relation of the plurality of imaging ranges, and the positional relationship information acquisition means Within the display range indicated by the display range information acquired by the display range information acquisition means based on the positional relationship information acquired by Then, a determination is made to determine any one shooting range among the overlapping shooting ranges in the overlapping range where at least two shooting ranges overlap among the plurality of shooting ranges as a shooting range used for displaying moving images in the overlapping range. And a division that overlaps the display range indicated by the display range information acquired by the display range information acquisition unit among the plurality of divided areas obtained by dividing the shooting range for each of the plurality of shooting ranges. Divided data receiving means for receiving divided moving picture data indicating a moving picture of a region from the transmitting device, wherein the determining means among the overlapping shooting ranges is used as divided moving picture data indicating a moving picture of a divided area overlapping the overlapping range. Divided data receiving means for receiving divided moving picture data indicating a moving picture of the determined divided area of the shooting range; and the divided data receiving means It characterized in that it comprises a synthesizing means for generating a composite video data by combining more received plurality of divided moving data, and a reproducing means for reproducing the synthesized moving image data generated by the synthesizing means.

  According to a second aspect of the present invention, in the playback device according to the first aspect, the determination unit is configured to determine a region that overlaps the display range acquired by the display range information acquisition unit in the overlapping shooting ranges. The maximum shooting range having the largest area is determined as the shooting range used for displaying the overlapping range of moving images.

  According to a third aspect of the present invention, in the playback device according to the second aspect, the determination unit includes the overlapping resolutions of the first moving image indicated by the moving image data corresponding to the maximum shooting range in the integrated shooting range. The display range of pixels constituting the first moving image that is lower than the resolution in the integrated shooting range of the second moving image indicated by the moving image data corresponding to the non-maximum shooting range different from the maximum shooting range in the shooting range. If the ratio of the number of pixels in the display range among the pixels constituting the second moving image is greater than or equal to a predetermined ratio with respect to the number of pixels in the second moving image, the non-maximum shooting range is set as the moving image in the overlapping range. The imaging range used for display is determined.

  The invention according to claim 5 is a plurality of moving images of a plurality of divided areas obtained by dividing a shooting range of the camera that has shot the moving image data for each of a plurality of moving image data shot by a plurality of cameras. In an information processing method executed by a playback device that receives divided moving image data transmitted from a transmission device that generates divided moving image data via a network, out of an integrated shooting range obtained by integrating a plurality of shooting ranges of the plurality of cameras A display range information acquisition step for acquiring display range information indicating a specified display range, a positional relationship information acquisition step for acquiring positional relationship information indicating a positional relationship among the plurality of imaging ranges, and the positional relationship information acquisition step Based on the acquired positional relationship information, the display range information acquired by the display range information acquisition step indicates Within the display range, any one of the shooting ranges that overlap in the overlapping range in which at least two shooting ranges of the plurality of shooting ranges overlap is used as a shooting range used for displaying the moving image of the overlapping range. And the display indicated by the display range information acquired by the display range information acquisition step among the plurality of divided regions obtained by dividing the shooting range for each of the plurality of shooting ranges. A divided data reception step of receiving, from the transmission device, divided moving image data indicating a moving image of a divided region overlapping the range, as divided moving image data indicating a moving image of the divided region overlapping the overlapping range, The divided moving image data indicating the moving image of the divided area of the shooting range determined by the determining step is received from the transmission device. A divided data receiving step, a combining step of combining a plurality of divided moving image data received by the divided data receiving step to generate combined moving image data, and a reproduction of reproducing the combined moving image data generated by the combining step And a step.

  The invention according to claim 6 is a communication system including a transmission device that transmits moving image data via a network and a playback device that receives the moving image data transmitted from the transmission device. Movie data acquisition means for acquiring a plurality of movie data captured by a camera, first control means for storing the plurality of movie data acquired by the movie data acquisition means in a storage means, and storage in the storage means Generating means for generating, for each of the plurality of moving image data, a plurality of divided moving image data indicating moving images of a plurality of divided areas obtained by dividing the shooting range of the camera that has captured the moving image data; A second control method for storing each generated divided moving image data in the storage means in association with identification information for identifying the divided moving image data. And identification information receiving means for receiving the identification information transmitted from the playback device, and divided video data transmission for transmitting the divided video data corresponding to the identification information received by the identification information receiving means to the playback device Display range information acquisition means for acquiring display range information indicating a specified display range among integrated shooting ranges obtained by integrating a plurality of shooting ranges of the plurality of cameras, and the plurality of the playback devices. Based on the positional relationship information acquired by the positional relationship information acquisition unit, the positional relationship information acquisition unit that acquires positional relationship information indicating the positional relationship of the imaging range of the display range information acquisition unit In the display range indicated by the display range information, a specific method for specifying an overlapping range in which at least two shooting ranges overlap among the plurality of shooting ranges. Determining means for determining any one of the at least two shooting ranges overlapping in the overlapping range specified by the specifying means as a shooting range used for displaying a moving image of the overlapping range; A moving image of a divided region that overlaps the display range indicated by the display range information acquired by the display range information acquisition unit among the plurality of divided regions obtained by dividing the shooting range for each of the plurality of shooting ranges. Identification information transmission means for transmitting identification information for identifying the divided moving image data indicating the divided moving image data indicating the divided moving image data indicating the moving image in the divided region overlapping the overlapping range, to the transmitting device, Identification information for identifying the divided moving image data indicating the moving image of the divided area of the shooting range determined by the determining means is sent out of the shooting range. Transmitting identification information transmitting means, divided data receiving means for receiving the divided moving picture data transmitted from the transmitting device, and combining the plurality of divided moving picture data received by the divided data receiving means It comprises: a synthesizing unit for generating; and a reproducing unit for reproducing the synthetic moving image data generated by the synthesizing unit.

  According to a seventh aspect of the present invention, in the communication system according to the sixth aspect, the transmission device extracts feature points from each of the plurality of moving images indicated by the plurality of divided moving image data stored in the storage unit. Among the feature points extracted by the extraction unit and the extraction unit, the plurality of shootings so that the positions in the integrated shooting range of the feature points having a correspondence relationship between at least two moving images of the plurality of moving images overlap. A positional relationship information determining unit that determines positional relationship information indicating a positional relationship of the range; and a positional relationship information transmitting unit that transmits the positional relationship information determined by the positional relationship information determining unit, the positional relationship The information acquisition means receives the positional relationship information transmitted from the transmission device.

  According to the first, fifth, or sixth aspect of the present invention, the playback apparatus uses, as the divided moving image data of the divided area overlapping the overlapping range, the divided area of any one of the shooting ranges overlapping in the overlapping range. The divided moving image data is acquired from the transmission device. Therefore, the transmission device may transmit the divided moving image data of the shooting range of any one camera as the divided moving image data of the overlapping range for the overlapping range, and the divided moving image data of the shooting range of the other camera for the overlapping range. There is no need to send. Further, since the playback device that has received the divided moving image data combines the divided moving image data, the transmitting device does not need to combine a plurality of moving image data acquired from a plurality of cameras. Therefore, it is possible to prevent a delay in delivery by a transmission device of a plurality of moving image data photographed by a plurality of cameras. Furthermore, the playback device only needs to perform the combining process when the overlapping range is included in the display range, and only the portion within the display range may be combined. Accordingly, since the minimum necessary synthesis processing is sufficient, the reproduction delay due to the synthesis processing can be reduced.

  According to the second aspect of the present invention, the playback apparatus converts the divided moving image data of the maximum shooting range having the largest area of the region overlapping the display range as the divided moving image data of the divided region overlapping the overlapping range of the combined moving image data. Used for generation. There is a high probability that the maximum shooting range occupies the center of the display range. Further, there is a probability that the maximum photographing range covers the entire display range. Therefore, in the synthesized moving image, the position of the seam of the video in the different shooting ranges can be shifted toward the end of the display range, or the seam can be prevented from being generated. Therefore, the visibility of the seam can be reduced.

  According to the third aspect of the present invention, the resolution in the display range is higher than the number of pixels in the display range among the pixels of the moving image of the camera having the largest area in the display range within the display range. When the ratio of the number of pixels in the display range out of the pixels of the moving image of the camera is equal to or greater than a predetermined ratio, the playback device uses a camera with a higher resolution in the display range as divided moving image data of the divided region overlapping the overlapping range The divided moving image data is used for generating the combined moving image data. If a moving image having a higher resolution is displayed in the overlapping range, the visibility of the overlapping range can be improved. In this case, when the shooting range having the larger area is determined to be the shooting range used for displaying the moving image in the overlapping range, the visibility of the moving image in the overlapping range is lowered although the visibility of the joint is lowered. On the other hand, when the shooting range with the higher resolution is determined to be the shooting range used for displaying moving images in the overlapping range, the resolution of moving images in the overlapping range is improved, but the joints are easily visible. According to the invention described in claim 3, by comparing and determining the ratio of the number of pixels and the predetermined ratio, it is possible to balance the reduction in the visibility of the joint and the improvement in the visibility of the moving image in the overlapping range.

  According to the fourth aspect of the present invention, the playback device generates the divided moving image data of the shooting range having the largest number of pixels in the display range as the divided moving image data of the divided region that overlaps the overlapping range. Used for. The larger the number of pixels in the region where the camera's shooting range and display range overlap among the pixels of the moving image shot by the camera, the larger the area of the region where the shooting range and display range overlap, or Increases resolution. Therefore, at least one of reducing the visibility of the seam and increasing the resolution in the overlapping range is possible.

  According to the seventh aspect of the present invention, it is possible to automatically specify the positional relationship of a plurality of shooting ranges using a range where the shooting ranges overlap.

It is a figure showing an example of outline composition of communications system SA of one embodiment. (A) is a figure which shows an example of the positional relationship of the imaging | photography range of three cameras, and the positional relationship of each imaging | photography range and a display range. (B) is a figure which shows an example of the overlap of each imaging | photography range and a display range. (C)-(E) is a figure which shows the example of determination of the division | segmentation moving image data received from the delivery server 2. FIG. (F) is a diagram showing an example of the result of combining images for one frame. (A) is a figure which shows an example of the positional relationship of the imaging | photography range of two cameras with which the imaging | photography range was fixed, and the positional relationship of each imaging | photography range and a display range. (B) is a diagram showing an example of the positional relationship between the shooting range of a camera with a fixed shooting range, the shooting range of the camera to which the shooting range moves, and the positional relationship between the shooting range and the display range. (A) is a flowchart showing an example of a moving image reproduction process in the client terminal 3. (B) is a flowchart showing an example of positional relationship information provision processing in the distribution server 2. 4 is a flowchart illustrating an example of moving image data acquisition processing in a client terminal 3. 4 is a flowchart illustrating an example of a composition process in the client terminal 3.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[1. Configuration of communication system]
FIG. 1 is a diagram illustrating a schematic configuration example of a communication system SA of the present embodiment. As shown in FIG. 1, the communication system SA includes a plurality of cameras 1-1 to 1-n (n is an integer satisfying n ≧ 1), a distribution server 2, and one or a plurality of client terminals 3. Is done. The distribution server 2 is an example of the transmission device of the present invention. The client terminal 3 is an example of a playback device of the present invention. The number of client terminals shown in FIG. 1 is an example, and the number is not limited to this number. The cameras 1-1 to 1-n are connected to the distribution server 2 via a communication cable, for example. The cameras 1-1 to 1-n may be connected to the distribution server 2 by radio. The distribution server 2 and the client terminal 3 are each connected to the network NW. The network NW is configured by, for example, the Internet.

  Each of the cameras 1-1 to 1-n is a video camera that shoots a moving image according to an operation of the photographer. Each of the cameras 1-1 to 1-n transmits the captured moving image data to the distribution server 2 via a communication cable. The moving image data transmitted from each of the cameras 1-1 to 1-n to the distribution server 2 is referred to as “photographed moving image data”. The shooting range of at least one of the cameras 1-1 to 1-n may be fixed. In addition, the shooting range of at least one of the cameras 1-1 to 1-n may change according to the operation of the photographer. The shooting range may be determined according to, for example, the position of the camera, the shooting direction, the angle of view, and the like. For example, the shooting range may be a virtual projection plane on which a subject video is projected as a video shot by a camera. In this case, the imaging range may be a flat surface or a curved surface, for example. In principle, the photographer installs or operates each camera so that at least a part of the shooting range of each camera overlaps at least a part of the shooting range of at least one other camera. Thereby, the photographing ranges of the cameras 1-1 to 1-n are integrated into one photographing range. The integrated shooting range is referred to as “integrated shooting range”. For example, the number of pixels in the vertical and horizontal directions of the moving images shot by the cameras 1-1 to 1-n may be the same. Alternatively, the resolution of at least one of the cameras 1-1 to 1-n may be different from the resolution of other cameras. For example, a 360-degree panoramic video may be shot by changing the shooting directions of the cameras 1-1 to 1-n. Alternatively, for example, an image in a predetermined direction viewed from a certain place may be taken by roughly matching the shooting directions of the cameras 1-1 to 1-n.

  The distribution server 2 is a server device that receives moving image data from, for example, the cameras 1-1 to 1-n and performs streaming distribution of the received moving image data to the client terminal 3. For example, the distribution server 2 may execute live distribution in which moving image data transmitted from the cameras 1-1 to 1-n is distributed in real time during shooting by the plurality of cameras 1-1 to 1-n. . Alternatively, for example, the distribution server 2 may store the moving image data transmitted from the cameras 1-1 to 1-n in advance and distribute the stored moving image data on demand. The distribution server 2 receives the captured moving image data transmitted from the cameras 1-1 to 1-n. The distribution server 2 generates a plurality of divided moving image data for each captured moving image data based on the captured moving image data. Specifically, the distribution server 2 divides the area of the image frame constituting the captured moving image data into a plurality of divided areas. The area of the image frame corresponds to the shooting range of the camera that shot the shot moving image data. The area of the image frame may be divided, for example, in a grid pattern. The number of divisions of the image frame area may be the same between the cameras 1-1 to 1-n, for example. Alternatively, the division number of the image frame region of at least one of the cameras 1-1 to 1-n may be different from the division number of the image frame region of another camera. The distribution server 2 generates a plurality of divided moving image data indicating moving images in a plurality of divided regions. The distribution server 2 transmits the divided moving image data requested from the client terminal 3 among the generated divided moving image data to the client terminal 3.

  The client terminal 3 is a terminal device that receives and reproduces moving image data that is stream-distributed from the distribution server 2. Examples of the client terminal 3 include a personal computer, a television, an STB, a mobile phone, a smartphone, and a tablet computer. The client terminal 3 receives, for example, designation of a display range by the user. The display range indicates a range displayed by the client terminal 3 in the integrated shooting range described above. For example, it is assumed that a moving image is projected on a virtual screen arranged in a three-dimensional virtual space, and there is a person who views the moving image in the three-dimensional virtual space. The client terminal 3 acquires display range information indicating the display range in accordance with a user operation. The display range information may include, for example, a viewpoint position, a line-of-sight direction, and a viewing angle of a person who views a moving image projected on the integrated shooting range. For example, the user can enlarge, reduce, or move the display range within the integrated shooting range. The client terminal 3 specifies, for each of the shooting ranges of the cameras 1-1 to 1-n, a divided region that overlaps the designated display range among a plurality of divided regions obtained by dividing the shooting range. And the client terminal 3 requests | requires the division | segmentation moving image data which show the moving image of the identified division | segmentation area | region to the delivery server 2. FIG. The client terminal 3 receives the divided moving image data transmitted from the distribution server 2 in response to this request. The client terminal 3 combines the received divided moving image data and generates combined moving image data indicating a moving image within the display range in the integrated shooting range. The client terminal 3 displays a moving image within the display range according to the generated combined moving image data. Thereby, the resolution of the displayed moving image can be improved as compared with the case where the moving image within the display range is displayed in the shooting range of one camera.

[2. Configuration of each device]
Next, the configuration of each device included in the communication system SA of the present embodiment will be described with reference to FIG. As illustrated in FIG. 1, the distribution server 2 includes a control unit 21, a storage unit 22, interface units 23 and 24, and the like. These components are connected to the bus 14. The interface unit 23 is connected to the network NW. The interface unit 24 is connected to the cameras 1-1 to 1-n via a communication cable. The control unit 21 includes a CPU (Central Processing Unit) as a computer, a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The storage unit 22 is configured by, for example, a hard disk drive. The storage unit 22 stores an OS, a server program, and the like. The server program is a program that causes the CPU to execute content distribution processing and the like. The storage unit 22 stores the captured moving image data received from the cameras 1-1 to 1-n and the divided moving image data generated from the captured moving image data. The control unit 21 acquires captured moving image data transmitted from each of the cameras 1-1 to 1-n, for example, in units of frames via the interface unit 23. The control unit 21 stores the acquired captured moving image data in the storage unit 22. Further, the control unit 21 generates a plurality of divided moving image data from each captured moving image data stored in the storage unit 22. For each image frame, the control unit 21 divides the image frame into a plurality of divided regions, and generates a plurality of divided moving image data corresponding to the plurality of divided regions. The control unit 21 stores each divided moving image data in the storage unit 22 in association with a divided moving image ID that identifies the divided moving image data. For example, the divided moving image ID is identification information for identifying a camera that has shot the captured moving image data that is the basis of the divided moving image data, and an identification that identifies the position of the divided area corresponding to the divided moving image data in the shooting range of the camera. Information.

  Next, as shown in FIG. 1, the client terminal 3 includes a control unit 31, a storage unit 32, a video RAM 33, a video control unit 34, an operation processing unit 35, an audio control unit 36, an interface unit 37, and the like. Is done. These components are connected to the bus 38. A display unit 34 a having a display is connected to the video control unit 34. The control unit 31 includes a CPU, ROM, RAM, and the like as a computer. An operation unit 35 a is connected to the operation processing unit 35. Examples of the operation unit 35a include a mouse, a keyboard, and a remote controller. A touch panel serving both as the display unit 34a and the operation unit 35a may be applied. The control unit 31 receives an operation instruction from the operation unit 35 a by the user via the operation processing unit 35. A speaker 36 a is connected to the sound control unit 36. The interface unit 37 is connected to the network NW. The storage unit 32 is configured by, for example, a hard disk drive or a flash memory. The storage unit 32 stores an OS, player software, and the like. The player software is a program that causes the CPU to execute moving image data reception and playback processing.

[3. Receiving and combining divided video data]
As described above, the client terminal 3 divides the divided moving image in the divided area that overlaps the designated display range among the plurality of divided areas obtained by dividing the shooting range for each of the shooting ranges of the cameras 1-1 to 1-n. Data is received from the distribution server 2. At this time, an area in which the shooting ranges of at least two cameras overlap in the display range is referred to as an “overlap range”. When the distribution server 2 transmits the divided moving image data of at least two cameras having overlapping shooting ranges to the client terminal 3 as the divided moving image data of the divided region overlapping the overlapping range, the load on the network increases. As a result, there is a possibility that delivery of moving image data is delayed. Therefore, the client terminal 3 uses the shooting range of any one of the shooting ranges of at least two cameras whose shooting ranges overlap based on a predetermined condition for displaying moving images in the overlapping range. Determine the shooting range. Then, the client terminal 3 receives, from the distribution server 2, only the divided moving image data in the determined shooting range among the shooting ranges overlapping in the overlapping range as the divided moving image data in the divided region overlapping with the overlapping range. Thereby, the client terminal 3 prevents an increase in the network load and the distribution load by the distribution server 2, and prevents a delay in the distribution of the moving image data.

  First, the client terminal 3 acquires display range information and positional relationship information indicating the positional relationship of the imaging ranges of the cameras 1-1 to 1-n. Based on the display range information and the positional relationship information, the client terminal 3 specifies whether or not the display range overlaps with the imaging ranges of the cameras 1-1 to 1-n and the overlapping range. Further, the client terminal 3 may determine a shooting range used for displaying a moving image of the overlapping range based on the display range information and the positional relationship information. The positional relationship information is, for example, the position of the camera 1-1, the shooting direction and the angle of view, and the relative position of each of the cameras 1-2 to 1-n with respect to the position of the camera 1-1, the shooting direction and the angle of view, The shooting direction and the angle of view may be included. In addition, for example, the positional relationship information may be obtained by photographing each projection plane of the cameras 1-2 to 1-n with respect to the position, direction, vertical length, and horizontal length of the photographing range as the projection plane of the camera 1-1. It may be the relative position, direction, vertical length and horizontal length of the range. In addition, the positional relationship information may further include the number of pixels in the vertical direction and the number of pixels in the horizontal direction of the moving images captured by the cameras 1-1 to 1-n.

  For example, the distribution server 2 may generate the positional relationship information, and the client terminal 3 may acquire the positional relationship information from the distribution server 2. Specifically, the distribution server 2 may acquire shooting parameters such as an angle of view from each of the cameras 1-1 to 1-n. Further, the distribution server 2 extracts feature points from image frames included in the captured moving image data received from the cameras 1-1 to 1-n. The distribution server 2 may extract feature points from the video by a known method, for example. An example of a feature point extraction method is SIFT (Scale-Invariant Feature Transform). The distribution server 2 identifies a feature point having a correspondence relationship between at least two cameras among the feature points extracted from the image frames having the same shooting times of the cameras 1-1 to 1-n. This feature point is called “corresponding point”. Corresponding points are feature points located in the same subject image among the respective images taken by different cameras. The distribution server 2 may specify the corresponding points using, for example, a known corresponding point search method. As described above, basically, at least a part of the shooting range of each camera overlaps at least a part of the shooting range of at least one other camera. Therefore, there is a probability that a plurality of moving images shot by different cameras include the same subject image. The distribution server 2 can determine the positional relationship between the shooting ranges of these cameras based on the positions of the corresponding points in the shooting ranges of at least two cameras whose corresponding points are specified. Specifically, the distribution server 2 determines the positional relationship of the shooting ranges so that the positions of corresponding points overlap in the integrated shooting range. For example, if the positional relationship between the shooting range of the camera 1-1 and the shooting range of the camera 1-2 is specified, and the positional relationship between the shooting range of the camera 1-2 and the shooting range of the camera 1-3 is specified, the camera 1 It is possible to specify the positional relationship of the imaging ranges of −1 to 1-3.

  Note that when the shooting ranges of all the cameras are fixed, for example, the positional relationship information may be set in advance by the administrator of the communication system SA. In this case, the client terminal 3 may acquire the set positional relationship information from the distribution server 2 or the like.

  Based on the positional relationship information and the display range information, the client terminal 3 determines a shooting range that is used for displaying moving images in the overlapping range among the shooting ranges of at least two cameras with overlapping shooting ranges. Specifically, the client terminal 3 determines each of the cameras 1-1 to 1-n from the positional relationship of the shooting ranges of the cameras 1-1 to 1-n indicated by the positional relationship information and the display range indicated by the display range information. The region that overlaps the display range is specified in the photographing range.

  For example, the client terminal 3 may calculate the area of an area where each shooting range overlaps the display range. Then, the client terminal 3 may determine the shooting range having the largest calculated area among the shooting ranges of at least two cameras that overlap the shooting ranges as the shooting range used for displaying the overlapping range of moving images. The shooting range in which the area of the area overlapping the display range is the largest is called “maximum shooting range”. The larger the area of the region that overlaps the display range, the higher the proportion of the display range. Therefore, there is a high probability that the maximum shooting range occupies the vicinity of the center of the display range. Therefore, when the images of the shooting ranges of at least two cameras are connected and combined, the joint of the images of the different shooting ranges can be moved closer to the end of the display area. For example, when the maximum shooting range overlaps the entire display range, the client terminal 3 receives only the divided moving image data of the maximum shooting range from the distribution server 2 over the entire display range, and receives the received divided moving image data. Synthesize. Therefore, in this case, it is possible to prevent a seam between images in different shooting ranges. In this way, the visibility of the seam can be reduced.

  For example, the client terminal 3 may calculate the number of pixels in an area where each shooting range overlaps the display range. In this case, the number of pixels is the number of pixels in a region overlapping the display range among the pixels of the moving image captured by the camera, and is not the number of pixels on the screen of the display unit 34a of the client terminal 3. The client terminal 3 may determine an area having the largest number of calculated pixels as an imaging range used for displaying an overlapping range of moving images, among the imaging ranges of at least two cameras that overlap the imaging ranges. The larger the number of pixels in the region that overlaps the display range, the larger the area of the region that overlaps the display range or the higher the resolution. Therefore, at least one of reducing the visibility of the seam as described above and increasing the resolution in the overlapping range is possible.

  For example, the client terminal 3 may calculate both the area of the area where each shooting range overlaps the display range and the number of pixels of the area where the shooting range overlaps the display range of the moving images shot by the camera for each camera. Good. Then, the client terminal 3 may calculate the resolution of each shooting range in the integrated shooting range using the area and the number of pixels. For example, the client terminal 3 integrates using the vertical length and horizontal length of each shooting range specified from the positional relationship information, and the vertical pixel count and horizontal pixel count of the moving image shot by each camera. The resolution of each shooting range in the shooting range can be calculated. The client terminal 3 may specify a shooting range with a resolution higher than the resolution of the maximum shooting range among the shooting ranges of at least two cameras whose shooting ranges overlap. When there is no shooting range with a resolution higher than the resolution of the maximum shooting range, the client terminal 3 may determine the maximum shooting range as a shooting range used for displaying moving images in the overlapping range. When there is a shooting range with a resolution higher than the resolution of the maximum shooting range, the client terminal 3 determines that the shooting range with a higher resolution overlaps the display range with respect to the number of pixels in the region where the maximum shooting range overlaps the display range. A ratio of the number of pixels in the region may be calculated. When the calculated ratio is equal to or higher than a predetermined setting ratio, the client terminal 3 may determine a shooting range with higher resolution as a shooting range used for displaying moving images in the overlapping range. When the calculated ratio is less than the set ratio, the maximum shooting range may be determined as a shooting range used for displaying moving images in the overlapping range. If a moving image having a higher resolution in the overlapping range is displayed, the visibility of the moving image in the overlapping range can be improved. In this case, when the maximum shooting range is determined to be a shooting range used for displaying moving images in the overlapping range, the visibility of the moving images in the overlapping range is reduced although the visibility of the seam is lowered. On the other hand, when the shooting range with the higher resolution is determined to be the shooting range used for displaying moving images in the overlapping range, the resolution of moving images in the overlapping range is improved, but the joints are easily visible. By comparing and determining the ratio of the number of pixels and the set ratio, the client terminal 3 can balance the reduction in the visibility of the seam and the improvement in the resolution of moving images in the overlapping range. For example, the administrator of the communication system SA may set the setting ratio in advance. The administrator can freely determine the setting ratio value.

  The client terminal 3 determines the shooting range having the smallest number of pixels in each divided region as the shooting range used for displaying the moving image of the overlapping range, from the shooting ranges of at least two cameras that overlap the shooting ranges. May be. When a part of the divided area overlaps the display range, the number of pixels in the part where the divided area does not overlap the display range tends to increase as the number of pixels in the divided area increases. When generating the combined moving image data, the client terminal 3 may perform the combining process using the pixel data within the display range, and may not use the pixel data outside the display range. As the number of pixels in a divided area increases, the amount of data that is not used for composition among the divided moving image data corresponding to the divided area tends to increase. The client terminal 3 reduces the amount of unnecessary data transmitted from the distribution server 2 by receiving from the distribution server 2 the divided moving image data of the divided area in the shooting range having the smallest number of pixels in one divided area. The distribution load by the distribution server 2 can be reduced. For example, the client terminal 3 includes only a plurality of shooting ranges in which the number of pixels in the region where the shooting range overlaps the display range is the same among the moving images shot by the camera. The shooting range with the smallest number of pixels in each divided area may be determined as the shooting range used for displaying moving images in the overlapping range.

  Note that the client terminal 3 may determine the shooting range used for displaying the overlapping range of moving images by a method different from the method described above, for example. For example, the client terminal 3 may randomly determine the shooting range used for displaying the overlapping range of moving images.

  FIG. 2A is a diagram illustrating an example of the positional relationship between the imaging ranges of the three cameras and the positional relationship between each imaging range and the display range. In this example, n = 3. CA1 to CA3 shown in FIG. 2A are imaging ranges. The shooting ranges CA1 to CA3 are the shooting ranges of the cameras 1-1 to 1-3, respectively. Each photographing area is divided into 16 divided areas. In FIG. 2A, Gi (x, y) indicates a divided area at coordinates (x, y) in the imaging range CAi. DA indicates a display range. A part of each of CA1 to CA3 overlaps the display range DA. For convenience of explanation, it is assumed that the number of pixels of the moving images photographed by the cameras 1-1 to 1-3 is the same. Further, in the positional relationship of the imaging ranges CA1 to CA3, the areas of the imaging ranges CA1 to CA3 are assumed to be the same. In the following, an example will be described in which the shooting range used for displaying moving images in the overlapping range is determined based on the number of pixels in the region where the shooting range overlaps the display range.

  FIG. 2B is a diagram illustrating an example of an overlap between each shooting range and the display range. FIGS. 2C to 2E are diagrams illustrating examples of determining the divided moving image data received from the distribution server 2. For example, it is assumed that the number of pixels in the area where the imaging range CA2 and the display area DA overlap is the largest among the imaging ranges CA1 to CA3. It is assumed that the number of pixels in the area where the imaging range CA3 and the display area DA overlap is the second largest.

  In this case, the client terminal 3 selects the imaging range CA2 having the largest number of pixels in the area overlapping the display area DA. Based on the positional relationship information and the display range information, the client terminal 3 identifies a divided area that overlaps the display area DA among the divided areas that constitute the imaging range CA2, as shown in FIG. 2C. . In the example of FIG. 2C, the client terminal 3 has divided areas G2 (0, 1), G2 (0, 2), G2 (0, 3), G2 (1, 1), G2 (1, 2). And G2 (1,3) are specified. Then, the client terminal 3 receives the divided moving image data corresponding to the specified divided area from the distribution server 2. The client terminal 3 excludes or deletes a portion that overlaps the imaging range CA1 from the display area DA. As a result, the display area DA is reduced as shown in FIG.

  Next, the client terminal 3 selects the imaging range CA3 having the second largest number of pixels in the area overlapping the display area DA. Then, as shown in FIG. 2D, the client terminal 3 identifies a divided area that overlaps the reduced display area DA among the divided areas constituting the imaging range CA3. In the example of FIG. 2D, the client terminal 3 identifies the divided areas G3 (0, 0) and G3 (0, 1). Then, the client terminal 3 receives the divided moving image data corresponding to the specified divided area from the distribution server 2. For example, the divided areas G3 (1, 0), G3 (1, 1), G3 (2, 0), and G3 (2, 1) overlap the display area DA shown in FIG. However, since the display area DA has become smaller, the divided moving image data in these divided areas is not received from the distribution server 2. The client terminal 3 excludes or deletes a portion that overlaps the imaging range CA3 from the display area DA. Thereby, the display area DA is further reduced as shown in FIG.

  Next, the client terminal 3 selects the imaging range CA1 in which the number of pixels in the area overlapping the display area DA is the third largest. Then, as shown in FIG. 2E, the client terminal 3 identifies a divided area that overlaps the display area DA that is further reduced, among the divided areas that form the imaging range CA1. In the example of FIG. 2E, the client terminal 3 identifies the divided areas G1 (2, 1), G1 (2, 2), G1 (3, 1), and G1 (3, 2). Then, the client terminal 3 receives the divided moving image data corresponding to the specified divided area from the distribution server 2. For example, the divided areas G1 (2, 3) and G1 (3, 3) overlap with the display area DA shown in FIG. However, since the display area DA has become smaller, the divided moving image data in these divided areas is not received from the distribution server 2.

  The client terminal 3 combines the divided moving image data received from the distribution server 2 to generate combined moving image data. First, the client terminal 3 has divided areas G1 (2,1), G1 (2,2), G1 (3,1), and G1 (3) of the imaging range CA1 having the third largest number of pixels in the area overlapping the display area DA. , 2) is drawn in the drawing area corresponding to the display range DA. At this time, the client terminal 3 does not have to draw an image of a portion that does not overlap the display range DA in each divided area.

  Next, the client terminal 3 displays the images of the frames of the divided areas G3 (0,0) and G3 (0,1) in the shooting range CA3 having the second largest number of pixels in the area overlapping the display area DA as the display range DA. Draw in the corresponding drawing area. At this time, the client terminal 3 may overwrite or fill the image of the shooting range CA1 with the image of the shooting range CA3 for the portion where the shooting range CA3 overlaps the shooting range CA1. At this time, the client terminal 3 may perform a process of making the joint between the image of the shooting range CA1 and the image of the shooting range CA3 inconspicuous. Alternatively, for example, the client terminal 3 may combine the image of the shooting range CA1 and the image of the shooting range CA3 for each pixel in the portion where the shooting range CA3 and the shooting range CA1 overlap. However, it is possible to reduce the load on the client terminal 3 required for the composition processing by performing overwriting or painting rather than performing composition for each pixel.

  Next, the client terminal 3 uses the divided areas G2 (0,1), G2 (0,2), G2 (0,3), G2 (1,1) of the shooting range CA2 having the largest number of pixels in the area overlapping the display area DA. 1) Draw the images of G2 (1,2) and G2 (1,3) frames in the drawing area corresponding to the display range DA. At this time, the client terminal 3 may overwrite or fill the image of the shooting range CA1 or CA3 with the image of the shooting range CA2 for a portion where the shooting range CA2 overlaps at least one of the shooting ranges CA1 and CA3. At this time, the client terminal 3 may perform a process of making the joint of the image inconspicuous. Alternatively, for example, the client terminal 3 may perform composition for each pixel.

  FIG. 2F is a diagram illustrating an example of a composition result of an image for one frame. Through the processing described above, the client terminal 3 generates an image as shown in FIG. 2F and displays it on the display unit 34a of the client terminal 3.

  As shown in FIG. 2B, the overlapping range is divided into overlapping ranges OA1 to OA4. The overlapping range OA1 is an area where the shooting range CA1 and the shooting range CA2 overlap. As a result of the processing described above, the client terminal 3 determines the shooting range CA2 of the shooting ranges CA1 and CA2 as a shooting range used for displaying moving images in the overlapping range OA1. The overlapping range OA2 is an area where the shooting range CA1 and the shooting range CA3 overlap. The client terminal 3 determines the shooting range CA3 out of the shooting ranges CA1 and CA3 as a shooting range used for displaying moving images in the overlapping range OA2. The overlapping range OA3 is an area where the shooting range CA2 and the shooting range CA3 overlap. The client terminal 3 determines the shooting range CA2 of the shooting ranges CA2 and AC3 as the shooting range used for displaying the moving image of the overlapping range OA3. The overlapping range OA4 is a region where the imaging ranges CA1 to C3 overlap. Among the shooting ranges CA1 to CA3, the shooting range CA2 is determined as a shooting range used for displaying moving images of the overlapping range OA2.

  FIG. 3A is a diagram illustrating an example of the positional relationship between the shooting ranges of two cameras with fixed shooting ranges and the positional relationship between each shooting range and the display range. In the following, an example will be described in which the shooting range used for displaying moving images in the overlapping range is determined using the area of the region where the shooting range overlaps the display range. In FIG. 3A, shooting ranges CA11 and CA12 are shooting ranges of the cameras 1-1 and 1-2, respectively. The shooting ranges CA11 and CA12 are fixed. The area of the shooting range CA11 and the area of the shooting range CA12 are the same. The right end area of the shooting range CA11 and the left end area of the shooting range CA12 overlap.

  As shown in FIG. 3A, the display range DA is first located on the place P11. At this time, a part of the shooting range CA11 overlaps the entire display range DA. On the other hand, a part of the shooting range CA12 overlaps a part of the display range DA. Therefore, the area of the region where the shooting range CA11 overlaps the display range DA is larger than the area of the region where the shooting range CA12 overlaps the display range DA. In the display range DA, the shooting range CA11 and the shooting range CA12 overlap in the overlapping range OA11. Therefore, the client terminal 3 determines the shooting range CA11 as the shooting range used for the moving image of the overlapping range OA11. Since the entire display range DA is covered with the shooting range CA11, the client terminal 3 receives only the divided moving image data of the shooting range CA11 from the distribution server 2 as the divided moving image data of the divided area overlapping the display range DA. become. Therefore, the client terminal 3 combines the videos of the divided areas of the shooting range CA11 to generate a video of the entire display range DA. Therefore, there is no seam between the video of the shooting range CA11 and the video of the shooting range CA12. If the shooting range CA12 is determined as the shooting range used for the moving image of the overlapping range OA11, the client terminal 3 receives the divided moving image data of the shooting range CA12 from the distribution server 2 as the divided moving image data of the overlapping range OA11. Then, the divided moving image data of the shooting range CA11 is received from the distribution server 2 as the divided moving image data of the range excluding the overlapping range OA11. Therefore, the client terminal 3 combines the video of the divided area of the shooting range CA11 and the video of the divided area of the shooting range CA12. Therefore, as shown in FIG. 3A, the joint between the video of the shooting range CA11 and the video of the shooting range CA12 appears in the approximate center of the display range DA.

  Thereafter, as shown in FIG. 3A, it is assumed that the display range DA has moved onto the place P12. At this time, a part of each of the photographing ranges CA11 and CA12 overlaps the entire display range DA. The area of the area where the imaging range CA12 overlaps the display range DA is larger than the area of the area where the imaging range CA11 overlaps the display range DA. In the display range DA, the shooting range CA11 and the shooting range CA12 overlap in the overlapping range OA12. Therefore, the client terminal 3 determines the shooting range CA12 as a shooting range used for moving images in the overlapping range OA12. In this case, the client terminal 3 combines the video of the divided area of the shooting range CA11 and the video of the divided area of the shooting range CA12. However, as shown in FIG. 3A, the joint between the video of the shooting range CA11 and the video of the shooting range CA12 appears at the left end of the display range DA. Therefore, the joint can be made inconspicuous or the visibility of the joint can be reduced. Assume that the shooting range CA11 is determined as the shooting range used for the moving image of the overlapping range OA12. In this case, as shown in FIG. 3A, the joint between the video of the shooting range CA11 and the video of the shooting range CA12 appears near the center of the display range DA.

  FIG. 3B is a diagram illustrating an example of a positional relationship between a shooting range of a camera with a fixed shooting range, a shooting range of a camera to which the shooting range moves, and a positional relationship between the shooting range and the display range. Below, the example which determines the imaging | photography range used for the display of the animation of an overlapping range using the ratio of the number of pixels is demonstrated. In FIG. 3B, shooting ranges CA21 and CA22 are the shooting ranges of the cameras 1-1 and 1-2, respectively. The shooting range CA21 is fixed, and the shooting range CA22 can be moved, enlarged, and reduced by the operation of the camera 1-2 by the photographer, for example. The area of the shooting range CA22 is smaller than the area of the shooting range CA21, and the shooting range CA22 moves within the shooting range CA21. The display range DA moves within the shooting range CA22. The number of pixels of the moving image shot by the camera 1-1 is the same as the number of pixels of the moving image shot by the camera 1-2. Accordingly, the resolution on the integrated shooting range of the moving image shot by the camera 1-2 is higher than the resolution on the integrated shooting range of the moving image shot by the camera 1-1. Further, the setting ratio is set to 2.

  As shown in FIG. 3B, the display range DA is first located on the place P21. At this time, a part of the shooting range CA21 overlaps the entire display range DA. Of the pixels of the moving image captured by the camera 1-1, the number of pixels in the area where the imaging range CA21 overlaps the display range DA is about 400,000. On the other hand, a part of the photographing range CA22 overlaps a part of the display range DA. Of the pixels of the moving image shot by the camera 1-2, the number of pixels in the area where the shooting range CA22 overlaps the display range DA is about 500,000. In the display range DA, the shooting range CA21 and the shooting range CA22 overlap in the overlapping range OA21. The ratio of the number of pixels in the area where the shooting range CA22 overlaps the display range DA to the number of pixels in the area where the shooting range CA21 overlaps the display range DA is 1.25. Since this ratio is smaller than the set ratio, the client terminal 3 determines the shooting range CA21 as the shooting range used for the moving image of the overlapping range OA21. Since the entire display range DA is covered with the shooting range CA21, the client terminal 3 receives only the divided moving image data of the shooting range CA21 from the distribution server 2 as the divided moving image data of the divided area overlapping the display range DA. become. Therefore, the client terminal 3 combines the video of the divided areas of the shooting range CA21 to generate a video of the entire display range DA. Therefore, there is no seam between the image of the shooting range CA21 and the image of the shooting range CA22. If the shooting range CA22 is determined to be a shooting range used for moving images in the overlapping range OA21, the resolution of moving images in the overlapping range DA is improved. However, as shown in FIG. 3B, a joint between the video of the shooting range CA21 and the video of the shooting range CA22 appears.

  Thereafter, as shown in FIG. 3B, it is assumed that the display range DA has moved onto the place P22. Also at this time, a part of the photographing range CA21 overlaps the entire display range DA. Of the pixels of the moving image shot by the camera 1-1, the number of pixels in the area where the shooting range CA21 overlaps the display range DA is about 250,000. On the other hand, the entire photographing range CA22 is included in the display range DA. Of the pixels of the moving image shot by the camera 1-2, the number of pixels in the area where the shooting range CA22 overlaps the display range DA is about 2 million. In the display range DA, the shooting range CA21 and the shooting range CA22 overlap in the overlapping range OA22. The ratio of the number of pixels in the area where the shooting range CA22 overlaps the display range DA to the number of pixels in the area where the shooting range CA21 overlaps the display range DA is 8. Since this ratio is larger than the set ratio, the client terminal 3 determines the shooting range CA22 as a shooting range used for moving images in the overlapping range OA22. Therefore, although a seam is generated at the edge of the overlapping range OA22, the resolution of the moving image in the overlapping range OA22 is improved as compared with the case where the photographing range CA21 is used.

[4. Operation of Communication System SA]
Next, the operation of the communication system SA of this embodiment will be described with reference to FIGS. In the following, an operation example in the case of determining the shooting range used for displaying the moving image in the overlapping range based on the number of pixels in the region where the shooting range overlaps the display range will be described. FIG. 4A is a flowchart illustrating an example of a moving image playback process in the client terminal 3. For example, when the user gives an instruction to play a moving image by operating the operation unit 35a, the client terminal 3 executes a moving image playing process.

  First, the control unit 31 of the client terminal 3 determines whether or not an operation for ending reproduction of a moving image has been performed by the user (step S1). At this time, if the control unit 31 determines that an operation for ending reproduction of the moving image has not been performed (step S1: NO), the control unit 31 proceeds to step S2. In step S2, the client terminal 3 acquires display range information. For example, the control unit 31 specifies whether or not each of the viewpoint position, the line-of-sight direction, and the viewing angle is changed, the amount of change, and the like based on the operation of the operation unit 35a by the user. Thereby, the control unit 31 determines the latest viewpoint position, line-of-sight direction, and viewing angle, and acquires display range information including the viewpoint position, line-of-sight direction, and field-of-view angle.

  Subsequently, the control part 31 acquires positional relationship information (step S3). For example, the control unit 31 may transmit a request for positional relationship information to the distribution server 2. And the control part 31 may acquire the positional relationship information transmitted from the delivery server 2 according to this request | requirement. Alternatively, the control unit 31 may receive, for example, positional relationship information transmitted from the distribution server 2 at predetermined time intervals and store it in the RAM. And the control part 31 may acquire the newest positional relationship information from RAM among the positional relationship information received by the predetermined time interval.

  Subsequently, the control part 31 performs a moving image data acquisition process (step S4). In the moving image data acquisition process, the control unit 31 receives the divided moving image data necessary for reproducing the moving image from the distribution server 2. Details of the moving image data acquisition processing will be described later.

  Subsequently, the control part 31 performs a synthetic | combination process (step S5). In the synthesis process, the control unit 31 synthesizes the received divided data and generates synthesized moving image data on the RAM. Details of the synthesis process will be described later.

  Next, the control unit 31 reproduces the combined moving image data (Step S6). Specifically, the control unit 31 writes an image frame included in the combined moving image data in the video RAM 33. The video control unit 34 outputs a video signal corresponding to the image frame written in the video RAM 33 to the display unit 34 a according to the control signal from the control unit 31. Thereby, the video for one image frame is displayed on the display unit 34a. The control unit 31 sequentially writes the image frames included in the combined moving image data in the video RAM 33 so that the combined moving image is displayed on the display unit 34a. Next, the control unit 31 proceeds to step S1.

  In step S1, when it is determined that the operation for ending the reproduction of the moving image has been performed (step S1: YES), the control unit 31 ends the moving image reproduction process.

  FIG. 4B is a flowchart illustrating an example of the positional relationship information providing process in the distribution server 2. For example, when the distribution server 2 receives a request for positional relationship information from the client terminal 3, the distribution server 2 may execute the positional relationship information providing process. Alternatively, the distribution server 2 may execute the positional relationship information providing process at predetermined time intervals, for example.

  The control unit 21 of the distribution server 2 acquires an image frame from each captured moving image data of the cameras 1-1 to 1-n stored in the storage unit 22 (step S11). For example, in the case of live distribution, the control unit 21 may acquire the latest image frame. For example, in the case of on-demand distribution, the control unit 21 may acquire an image frame corresponding to the current moving image playback time by the client terminal 3. In addition, the control unit 21 acquires shooting parameters of the cameras 1-1 to 1-n. And the control part 21 acquires an angle of view, the vertical pixel number of a moving image, the horizontal pixel number, etc. from an imaging parameter.

  Next, the control unit 21 extracts one or more feature points from each of the acquired image frames (step S12). Subsequently, the control part 21 produces | generates positional relationship information based on the extracted feature point (step S13). Specifically, the control unit 21 specifies feature points that have a correspondence relationship between at least two image frames. The control unit 21 calculates the positions and shooting directions of the cameras 1-1 to 1-n so that the positions of the feature points overlap in a plurality of image frames in which the feature points in the correspondence relationship are specified. The control unit 21 generates positional relationship information including the calculated position, shooting direction, angle of view, vertical pixel number, and horizontal pixel number. The control unit 21 transmits the generated positional relationship information to the client terminal 3 (step S14), and ends the positional relationship information providing process.

  FIG. 5 is a flowchart illustrating an example of the moving image data acquisition process in the client terminal 3. First, the control unit 31 substitutes 1 for the camera number i, and substitutes 0 for the number m of cameras whose shooting range overlaps the display range (step S21). Next, the control unit 31 determines whether or not the shooting range of the camera CA1-i overlaps the display range based on the display range information and the positional relationship information (step S22). Specifically, the control unit 31 calculates the position, direction, vertical length, and horizontal length of the shooting range of the camera CA1-i from the position, shooting direction, and angle of view of the camera CA1-i. The control unit 31 specifies the range of the visual field based on the viewpoint position, the visual field direction, and the visual field angle. And the control part 31 determines whether the imaging | photography range of camera CA1-i is included in a visual field based on the position of the imaging | photography range of camera CA1-i, a direction, and an area, and the range of visual field. When the shooting range is included in the field of view, the control unit 31 determines that the shooting range of the camera CA1-i overlaps the display range (step S22: YES), and proceeds to step S23. On the other hand, when the shooting range is not included in the field of view, the control unit 31 determines that the shooting range of the camera CA1-i does not overlap with the display range (step S22: NO), and proceeds to step S25.

  In step S <b> 23, the control unit 31 calculates the number of pixels in a region where the shooting range of the camera CA <b> 1-i overlaps the display range among the pixels of the moving image shot by the camera CA <b> 1-i. For example, the control unit 31 calculates the area of the region that overlaps the field of view within the imaging range of the camera CA1-i. Then, the control unit 31 determines the shooting range of the camera CA1-i based on the number of pixels of the moving image of the camera CA1-i and the ratio between the area of the shooting range of the camera CA1-i and the area of the calculated region. The number of pixels in the area overlapping the display range is calculated. Next, the control unit 31 adds 1 to the number m (step S24), and proceeds to step S25.

  In step S25, the control unit 31 determines whether the camera number i is less than the number of cameras. At this time, if the control unit 31 determines that the camera number i is less than the number of cameras (step S25: YES), the control unit 31 proceeds to step S26. In step S26, the control unit 31 adds 1 to the camera number i and proceeds to step S22. On the other hand, if the control unit 31 determines that the number i is not less than the number of cameras (step S25: NO), the control unit 31 proceeds to step S27.

  In step S27, the control unit 31 substitutes the number m for the rank number j. Next, the control unit 31 selects a camera having the calculated jth pixel number among the cameras having the calculated pixel number in step S23 (step S28). Next, the control unit 31 receives, from the distribution server 2, the divided moving image data of the divided area that overlaps the display area among the divided moving image data corresponding to the selected camera (step S29). For example, the control unit 31 specifies, as the selected camera display area, an area that overlaps the visual field specified from the display range information in the shooting range of the selected camera. In addition, the control unit 31 specifies the position and range of each divided region in the shooting range of the selected camera. The control unit 31 identifies one or a plurality of divided areas that overlap the selected camera display area among the divided areas in the shooting range of the selected camera. The control unit 31 determines a divided region ID corresponding to the identified divided region. The control unit 31 transmits the acquired divided region ID to the distribution server 2. The distribution server 2 acquires the divided moving image data corresponding to the divided region ID received from the client terminal 3 from the storage unit 22 and transmits it to the client terminal 3. The control unit 31 causes the storage unit 32 to store the divided moving image data received from the distribution server 2.

  Next, the control unit 31 updates the display range by excluding the shooting range of the selected camera from the display range (step S30). For example, the control unit 31 specifies the range of the line of sight that passes through the selected camera display area specified in step S29 from the viewpoint position. Then, the control unit 31 deletes the identified line-of-sight range from the current field-of-view range and reconstructs the field-of-view.

  Next, the control unit 31 determines whether or not the rank number j is greater than 1 (step S31). At this time, if the control unit 31 determines that the rank number j is greater than 1 (step S31: YES), the control unit 31 proceeds to step S32. In step S32, the control unit 31 subtracts 1 from the rank number j, and proceeds to step S28. In subsequent steps S29 and S30, processing is performed based on the updated visual field. On the other hand, when determining that the rank number j is not greater than 1 (step S31: NO), the control unit 31 ends the moving image data acquisition process.

  FIG. 6 is a flowchart illustrating an example of the composition process in the client terminal 3. First, the control unit 31 substitutes the number m for the rank number j (step S41). Next, the control unit 31 selects a camera having the calculated jth pixel number among the cameras having the calculated pixel number in step S23 of the moving image data acquisition process (step S42). Next, the control unit 31 acquires the divided moving image data of the divided area corresponding to the selected camera from the storage unit 32, and draws the image frame included in the acquired divided moving image data in the drawing area secured on the RAM. (Step S43). The drawing area is a storage area corresponding to the display range. At this time, the control unit 31 does not need to draw an image of a part of the divided region corresponding to the selected camera that is partly out of the display range. When the current rank number j is less than the number m, the image of the divided area of the camera having the smaller number of pixels is already drawn in the drawing area. In this case, the control unit 31 may overwrite the drawn image with an image of a divided region of the camera having the jth largest number of pixels.

  Next, the control unit 31 determines whether or not the rank number j is greater than 1 (step S44). At this time, if the control unit 31 determines that the rank number j is greater than 1 (step S44: YES), the control unit 31 proceeds to step S45. In step S45, the control unit 31 subtracts 1 from the rank number j, and proceeds to step S42. On the other hand, when it is determined that the rank number j is not greater than 1 (step S44: NO), the control unit 31 ends the synthesis process. Note that when the divided moving image data includes a plurality of image frames, the control unit 31 may alternately execute the synthesis processing and the reproduction in step S6 of the moving image reproduction processing for each image frame, for example.

  As described above, according to the present embodiment, the client terminal 3 acquires display range information and positional relationship information. In addition, the client terminal 3 determines any one of the shooting ranges that overlap in the overlapping range as a shooting range that is used for displaying a moving image in the overlapping range. In addition, the client terminal 3 receives, from the distribution server 2, divided moving image data indicating a moving image in a divided region that overlaps the display range among the plurality of divided regions obtained by dividing the shooting range for each of the plurality of shooting ranges. At this time, the client terminal 3 receives the divided moving image data indicating the moving image in the divided area of the determined shooting range as the divided moving image data indicating the moving image in the divided area overlapping the overlapping range. Therefore, it is possible to prevent a delay in delivery by a transmission device of a plurality of moving image data shot by a plurality of cameras.

  The distribution server 2 may generate a plurality of moving image data having different resolutions for each captured moving image data received from the cameras 1-1 to 1-n. For example, the distribution server 2 may generate high-resolution moving image data and low-resolution moving image data, or may generate high-resolution, medium-resolution, and low-resolution moving image data, respectively. The moving image data with the highest resolution may be, for example, the captured moving image data itself. The distribution server 2 may generate a plurality of divided moving image data for each moving image data having different resolutions. For example, the client terminal 3 may determine which resolution of the moving image data is received based on a predetermined condition. The reason is to improve the resolution of the moving image as much as possible while suppressing the delivery delay of the moving image data. For example, the client terminal 3 may determine based on the reception speed of the divided moving image data from the distribution server 2. For example, the client terminal 3 may receive the divided moving image data having a higher resolution from the distribution server 2 as the reception speed is higher. Further, for example, the client terminal 3 receives the divided moving image data with higher resolution from the distribution server 2 when receiving the divided moving image data in the shooting range as the shooting range having a larger ratio of the area overlapping the display range. May be. For example, the client terminal 3 may receive high-resolution divided video data for a shooting range in which the ratio of the area overlapping the display range is the largest, and may receive low-resolution divided video data for other shooting ranges. Good. Further, the distribution server 2 may always transmit all of the divided moving image data having the lowest resolution to the client terminal 3. Since the data amount of the low resolution divided moving image data is relatively small, even if the distribution server 2 always transmits the low resolution divided moving image data, a large distribution delay does not occur. For example, when the client terminal 3 attempts to receive high-resolution divided moving image data, it may take time to receive the high-resolution divided moving image data. In this case, the client terminal 3 first reproduces the moving image using the low-resolution divided moving image data, and then receives the high-resolution divided moving image data and then reproduces the moving image using the high-resolution divided moving image data. Also good. Further, as described above, the client terminal 3 receives the divided moving image data corresponding to the divided area overlapping the display range from the distribution server 2. At this time, when the display range is suddenly changed by the user's operation, an area in which the moving image cannot be displayed is generated in the display range after the change only with the divided video data acquired based on the display range before the change. There is. The client terminal 3 can display a moving image in such a region as low-resolution divided moving image data.

DESCRIPTION OF SYMBOLS 1 Video camera 2 Distribution server 3 Client terminal 21, 31 Control part 22, 32 Storage part 23, 37 Interface part 34a Display part 35a Operation part SA Communication system

Claims (7)

For each of a plurality of moving image data shot by a plurality of cameras, from a transmission device that generates a plurality of divided moving image data indicating moving images of a plurality of divided areas obtained by dividing the shooting range of the camera that has shot the moving image data In a playback apparatus that receives divided video data transmitted via a network,
Display range information acquisition means for acquiring display range information indicating a specified display range in an integrated shooting range obtained by integrating a plurality of shooting ranges of the plurality of cameras;
Positional relationship information acquisition means for acquiring positional relationship information indicating the positional relationship of the plurality of imaging ranges;
Based on the positional relationship information acquired by the positional relationship information acquisition unit, at least two of the plurality of imaging ranges in the display range indicated by the display range information acquired by the display range information acquisition unit Determining means for determining any one shooting range among the overlapping shooting ranges in the overlapping range where the shooting ranges overlap as a shooting range used for displaying the moving image of the overlapping range;
A moving image of a divided region that overlaps the display range indicated by the display range information acquired by the display range information acquisition unit among the plurality of divided regions obtained by dividing the shooting range for each of the plurality of shooting ranges. Divided data receiving means for receiving the divided moving image data indicating the moving image data indicating the moving image in the divided area overlapping the overlapping range, determined by the determining means in the overlapping shooting range Divided data receiving means for receiving divided moving image data indicating a moving image of a divided region of the shooting range;
Combining means for combining a plurality of divided moving image data received by the divided data receiving means to generate combined moving image data;
Playback means for playing back the synthesized video data generated by the synthesis means;
A playback apparatus comprising: The playback device according to claim 1,
The determination means uses the maximum photographing range having the largest area of the region that overlaps the display range acquired by the display range information acquisition means in the overlapping photographing ranges for displaying moving images in the overlapping range. A playback apparatus characterized by determining a shooting range. The playback device according to claim 2,
The determination means includes a moving image corresponding to a non-maximum shooting range in which the resolution in the integrated shooting range of the first moving image indicated by the moving image data corresponding to the maximum shooting range is different from the maximum shooting range in the overlapping shooting ranges. Of the pixels constituting the second moving image, the resolution is lower than the resolution in the integrated shooting range of the second moving image indicated by the data and the number of pixels in the display range among the pixels constituting the first moving image. When the ratio of the number of pixels in the display range is equal to or greater than a predetermined ratio, the non-maximum shooting range is determined as a shooting range used for displaying moving images in the overlapping range. The playback device according to claim 1,
The determining means is configured to display a shooting range having the largest number of pixels in the display range among the pixels constituting the moving image indicated by the corresponding moving image data in the overlapping shooting range. A reproduction apparatus characterized by determining a shooting range to be used. For each of a plurality of moving image data shot by a plurality of cameras, from a transmission device that generates a plurality of divided moving image data indicating moving images of a plurality of divided areas obtained by dividing the shooting range of the camera that has shot the moving image data In an information processing method executed by a playback device that receives divided video data transmitted via a network,
A display range information obtaining step for obtaining display range information indicating a designated display range in an integrated photographing range obtained by integrating a plurality of photographing ranges of the plurality of cameras;
A positional relationship information acquisition step of acquiring positional relationship information indicating the positional relationship of the plurality of imaging ranges;
Based on the positional relationship information acquired by the positional relationship information acquisition step, at least two of the plurality of imaging ranges in the display range indicated by the display range information acquired by the display range information acquisition step A determination step of determining any one shooting range among the overlapping shooting ranges in the overlapping range in which the shooting ranges overlap as a shooting range used for displaying a moving image of the overlapping range;
A moving image of a divided region that overlaps the display range indicated by the display range information acquired by the display range information acquisition step among the plurality of divided regions obtained by dividing the shooting range for each of the plurality of shooting ranges. Divided data reception step for receiving the divided moving image data indicating the moving image data indicating the moving image of the divided area overlapping the overlapping range, determined in the determining step in the overlapping shooting range A divided data reception step of receiving, from the transmission device, divided moving image data indicating a moving image of the divided region of the shooting range;
Combining a plurality of divided moving image data received by the divided data receiving step to generate combined moving image data;
A reproduction step of reproducing the synthesized moving image data generated by the synthesis step;
An information processing method comprising: In a communication system comprising a transmitting device that transmits moving image data via a network, and a playback device that receives moving image data transmitted from the transmitting device,
The transmitter is
Video data acquisition means for acquiring a plurality of video data shot by a plurality of cameras;
First control means for storing in the storage means the plurality of moving picture data acquired by the moving picture data acquisition means;
Generating means for generating, for each of the plurality of moving image data stored in the storage means, a plurality of divided moving image data indicating moving images in a plurality of divided areas obtained by dividing the shooting range of the camera that has captured the moving image data. When,
Second control means for storing each of the divided moving image data generated by the generating means in the storage means in association with identification information for identifying the divided moving image data;
Identification information receiving means for receiving identification information transmitted from the playback device;
Divided moving image data transmitting means for transmitting the divided moving image data corresponding to the identification information received by the identification information receiving means to the playback device;
With
The playback device
Display range information acquisition means for acquiring display range information indicating a specified display range in an integrated shooting range obtained by integrating a plurality of shooting ranges of the plurality of cameras;
Positional relationship information acquisition means for acquiring positional relationship information indicating the positional relationship of the plurality of imaging ranges;
Based on the positional relationship information acquired by the positional relationship information acquisition unit, at least two of the plurality of imaging ranges in the display range indicated by the display range information acquired by the display range information acquisition unit Determining means for determining any one shooting range among the overlapping shooting ranges in the overlapping range where the shooting ranges overlap as a shooting range used for displaying the moving image of the overlapping range;
A moving image of a divided region that overlaps the display range indicated by the display range information acquired by the display range information acquisition unit among the plurality of divided regions obtained by dividing the shooting range for each of the plurality of shooting ranges. Identification information transmission means for transmitting identification information for identifying the divided moving image data indicating the divided moving image data indicating the divided moving image data indicating the moving image in the divided region overlapping the overlapping range, to the transmitting device, Identification information transmitting means for transmitting identification information for identifying divided moving image data indicating a moving image of a divided region of the photographing range determined by the determining means in the photographing range;
Divided data receiving means for receiving the divided moving image data transmitted from the transmission device;
Combining means for combining a plurality of divided moving image data received by the divided data receiving means to generate combined moving image data;
Playback means for playing back the synthesized video data generated by the synthesis means;
A communication system comprising: The communication system according to claim 6,
The transmitter is
Extracting means for extracting feature points from each of a plurality of moving images indicated by the plurality of divided moving image data stored in the storage means;
Among the feature points extracted by the extracting means, the positional relationship of the plurality of shooting ranges so that the positions in the integrated shooting range of the feature points having a correspondence relationship between at least two videos of the plurality of movies overlap. Positional relationship information determining means for determining positional relationship information indicating
Positional relationship information transmitting means for transmitting the positional relationship information determined by the positional relationship information determining means;
Further comprising
The communication system characterized in that the positional relationship information acquisition means receives the positional relationship information transmitted from the transmission device.

Images