JP6948624B2 - Video distribution method and server - Google Patents

Description

The present invention relates to a video distribution method and the like for distributing video captured from a plurality of viewpoints.

As a video distribution method, for example, the technique described in Patent Document 1 is known. Further, a video distribution method for distributing video shot from a plurality of viewpoints is known (see, for example, Patent Document 2). In such a video distribution method, the user can specify and view an arbitrary video from a plurality of videos of a specific scene shot from different viewpoints.

Japanese Unexamined Patent Publication No. 2009-20626 Japanese Unexamined Patent Publication No. 2012-09949

In such a video distribution method, it is desired that an appropriate video can be distributed based on a plurality of videos.

Therefore, an object of the present invention is to provide a video distribution method or a video reception method capable of delivering an appropriate video based on a plurality of videos.

In order to achieve the above object, the video distribution method according to one aspect of the present invention is based on a plurality of videos obtained by shooting the same scene from a plurality of different viewpoints in a plurality of continuous periods. It is a video distribution method by a server that distributes one video signal to a terminal device, and is a first selection from the plurality of images for each of the plurality of periods based on evaluation values of the plurality of images by a plurality of users. The first selection step of selecting a video and the first video signal for continuously playing back only the plurality of first-selected videos selected for the plurality of periods among the plurality of videos are described above. It includes a first transmission step of transmitting to the terminal device.

Further, the video receiving method according to one aspect of the present invention receives a video signal based on a plurality of videos obtained by shooting the same scene from a plurality of different viewpoints in a plurality of continuous periods from a server. A method of receiving video by a terminal device for continuously playing back a plurality of selected videos selected from the plurality of videos based on evaluation values of the plurality of videos by a plurality of users for the plurality of periods. The present invention includes a reception step of receiving the video signal from the server and a reproduction step of continuously reproducing the plurality of selected images based on the video signal.

It should be noted that these general or specific embodiments may be realized in a recording medium such as a system, method, integrated circuit, computer program or computer-readable CD-ROM, and the system, method, integrated circuit, computer program. And any combination of recording media may be realized.

The present invention can provide a video distribution method or a video reception method capable of delivering an appropriate video based on a plurality of videos.

It is a figure which shows the structure of the video distribution system which concerns on Embodiment 1. FIG. It is a block diagram of the terminal apparatus which concerns on Embodiment 1. FIG. It is a flowchart which shows the process by the terminal apparatus which concerns on Embodiment 1. FIG. It is a flowchart which shows the viewpoint image switching process by the terminal apparatus which concerns on Embodiment 1. FIG. It is a flowchart which shows the evaluation addition process by the terminal apparatus which concerns on Embodiment 1. FIG. It is a flowchart which shows the information acquisition process by the terminal apparatus which concerns on Embodiment 1. It is a block diagram of the server which concerns on Embodiment 1. FIG. It is a flowchart which shows the process by the server which concerns on Embodiment 1. It is a flowchart which shows the viewpoint image switching process by the server which concerns on Embodiment 1. It is a flowchart which shows the evaluation addition process by the server which concerns on Embodiment 1. It is a flowchart which shows the information acquisition process by the server which concerns on Embodiment 1. It is a figure which shows the operation example of the favorite subject switching mode which concerns on Embodiment 1. FIG. It is a figure which shows the operation example of the recommended viewpoint image switching mode which concerns on Embodiment 1. FIG. It is a figure which shows the operation example of the recommended subject switching mode which concerns on Embodiment 1. FIG. It is a figure which shows the display screen example which concerns on Embodiment 1. FIG. It is a figure which shows the display screen example which concerns on Embodiment 1. FIG. It is a figure which shows the display screen example which concerns on Embodiment 1. FIG. It is a flowchart of the video distribution method by the server which concerns on Embodiment 1. FIG. It is a flowchart of the image receiving method by the terminal apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the structure of a video information processing system. It is a figure which shows an example of the notification screen displayed at the time of starting a camera. It is an overall block diagram of the content supply system which realizes a content distribution service. It is an overall block diagram of a system for digital broadcasting. It is a figure which shows an example of a smartphone. It is a block diagram which shows the configuration example of a smartphone.

The video distribution method according to one aspect of the present invention distributes a first video signal based on a plurality of images obtained by shooting the same scene from a plurality of different viewpoints in a plurality of continuous periods to a terminal device. A first selection step of selecting a first selection video from the plurality of videos for each of the plurality of periods based on evaluation values of the plurality of videos by a plurality of users. And a first transmission step of transmitting the first video signal for continuously reproducing the plurality of first-selected videos selected for the plurality of periods to the terminal device.

As a result, the video distribution method can distribute an appropriate video based on the evaluations of a plurality of users.

For example, the evaluation value is an evaluation value of each of the plurality of images, and in the first selection step, the evaluation value in the period is the highest from the plurality of images for each of the plurality of periods. An image higher than a predetermined value may be selected as the first selection image.

As a result, the video distribution method can distribute a video that continuously reproduces a highly evaluated viewpoint video based on the evaluations of a plurality of users.

For example, the evaluation value is an evaluation value of each of a plurality of subjects included in the plurality of images, and in the first selection step, the evaluation value is obtained from the plurality of images in each of the plurality of periods. An image showing a recommended subject higher than the highest value or a predetermined value may be selected as the first selection image.

As a result, the video distribution method can distribute a video that continuously reproduces a viewpoint video in which a highly evaluated subject is captured, based on the evaluations of a plurality of users.

For example, in the first selection step, in each of the plurality of periods, the image having the largest area in which the recommended subject is reflected may be selected as the first selection image from the plurality of images.

As a result, the video distribution method can distribute a video that continuously reproduces a viewpoint video in which a highly evaluated subject is greatly captured, based on the evaluations of a plurality of users.

For example, in the video distribution method, when the amount of data of the first video signal is larger than the upper limit value based on the reception band of the terminal device, each of the plurality of periods is based on the evaluation values of the plurality of videos. On the other hand, the second selection step of selecting the second selection video from the plurality of videos and the second selection step for continuously playing the second selection video selected for each of the plurality of periods. At least one of the plurality of second-selected videos selected for the plurality of periods, including a second transmission step of transmitting the video signal to the terminal device, was selected for the plurality of periods. Unlike the plurality of first-selected videos, the amount of data of the second video signal may be smaller than the upper limit value.

As a result, the video distribution method can distribute a video signal with a small amount of data when the reception band of the terminal device is narrow.

For example, when the plurality of images are transmitted from the plurality of cameras to the server in real time and the transmission of the image corresponding to the first selection image from the camera is interrupted, in the first selection step, the first selection step is performed. A second selected video different from the first selected video may be selected, and in the first transmission step, a second video signal for reproducing the second selected video may be transmitted to the terminal device.

As a result, the video distribution method can suppress the interruption of video distribution when the transmission of video from the camera is interrupted.

For example, the evaluation value is an evaluation value of each of the plurality of images, and in the first selection step, the evaluation value in the period is predetermined from the plurality of images for each of the plurality of periods. An image lower than the value may be selected as the first selection image.

Thereby, the video distribution method can reduce the occurrence of blind spots, for example, in surveillance applications.

For example, in the first transmission step, the video signal may be generated by joining the plurality of selected videos selected for the plurality of periods, and the video signal may be transmitted to the terminal device.

For example, in the transmission step, the plurality of videos and information for identifying the plurality of selected videos selected for the plurality of periods may be transmitted to the terminal device as the video signals. ..

Further, the video receiving method according to one aspect of the present invention receives a video signal based on a plurality of videos obtained by shooting the same scene from a plurality of different viewpoints in a plurality of continuous periods from a server. A method of receiving video by a terminal device for continuously playing back a plurality of selected videos selected from the plurality of videos based on evaluation values of the plurality of videos by a plurality of users for the plurality of periods. The present invention includes a reception step of receiving the video signal from the server and a reproduction step of continuously reproducing the plurality of selected images based on the video signal.

As a result, the video receiving method can reproduce an appropriate video based on the evaluations of a plurality of users.

For example, the evaluation value is an evaluation value of each of the plurality of images, and the selected image is selected from the plurality of images for each of the plurality of periods, and the evaluation value in the period is the most. The image may be higher or higher than a predetermined value.

As a result, the video receiving method can continuously reproduce the highly evaluated viewpoint video based on the evaluations of a plurality of users.

For example, the evaluation value is an evaluation value of each of a plurality of subjects included in the plurality of images, and the selected image is selected from the plurality of images in each of the plurality of periods. It may be an image showing a subject having the highest value or a value higher than a predetermined value.

As a result, the video receiving method can continuously reproduce the viewpoint video in which a highly evaluated subject is captured, based on the evaluations of a plurality of users.

For example, the video receiving method further includes a display step for displaying any of the plurality of videos, and in the display step, the evaluation value is the highest among the plurality of subjects included in the displayed video. Alternatively, a subject higher than a predetermined value may be highlighted.

As a result, the video receiving method can notify the user of a highly evaluated subject.

For example, the video receiving method further includes a first display step of displaying any of the plurality of videos and a case where any one of a plurality of subjects included in the displayed video is selected by the user. , A second display step of displaying a menu for evaluating the selected subject may be included.

As a result, the video receiving method can improve the convenience of the user.

For example, the video receiving method further includes a first display step of displaying any of the plurality of videos and a case where any one of a plurality of subjects included in the displayed video is selected by the user. The second display step of displaying a list of images of a plurality of periods in which the selected subject is included and the evaluation value is higher than a predetermined value among the plurality of images may be included.

As a result, the video receiving method can improve the convenience of the user.

For example, the video receiving method further includes a first display step of displaying any of the plurality of videos and a case where any one of a plurality of subjects included in the displayed video is selected by the user. A second display step of displaying a tracking image for tracking the selected subject, which is an image obtained by joining the plurality of images, may be included.

As a result, the video receiving method can improve the convenience of the user.

For example, the video receiving method further includes a first screen on which any of the plurality of videos is displayed, and a second screen smaller than the first screen and displaying the reproduced plurality of selected videos. It may include a first display step of displaying an output screen including the above, and a second display step of displaying the plurality of selected images on the first screen when the second screen is selected by the user.

As a result, the video receiving method can improve the convenience of the user.

For example, the evaluation value is an evaluation value of each of the plurality of images, and the selected image is selected from the plurality of images for each of the plurality of periods, and the evaluation value in the period is predetermined. The image may be lower than the specified value.

Further, the server according to one aspect of the present invention is a server that distributes video signals based on a plurality of images obtained by shooting the same scene from a plurality of different viewpoints in a plurality of continuous periods to a terminal device. The selection unit that selects the selected video from the plurality of videos for each of the plurality of periods based on the evaluation values of the plurality of videos by the plurality of users, and the selection unit for the plurality of periods. It includes a transmission unit that transmits the video signal for continuously reproducing the plurality of the first-selected videos to the terminal device.

As a result, the server can deliver an appropriate video based on the evaluations of a plurality of users.

Further, the terminal device according to one aspect of the present invention is a terminal that receives from a server a video signal based on a plurality of images obtained by shooting the same scene from a plurality of different viewpoints in a plurality of continuous periods. A device for continuously playing a plurality of selected images selected from the plurality of images based on evaluation values of the plurality of images by a plurality of users for the plurality of periods. It includes a receiving unit that receives from the server and a reproducing unit that continuously reproduces the plurality of selected images based on the video signal.

As a result, the terminal device can reproduce an appropriate image based on the evaluation of a plurality of users.

It should be noted that these general or specific embodiments may be realized in a recording medium such as a system, method, integrated circuit, computer program or computer-readable CD-ROM, and the system, method, integrated circuit, computer program. And any combination of recording media may be realized.

Hereinafter, embodiments will be specifically described with reference to the drawings. It should be noted that all of the embodiments described below show a specific example of the present invention. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, the order of steps, etc. shown in the following embodiments are examples, and are not intended to limit the present invention. Further, among the components in the following embodiments, the components not described in the independent claims indicating the highest level concept are described as arbitrary components.

(Embodiment 1)
In the video distribution system according to the present embodiment, a selected video is selected from a plurality of viewpoint videos based on evaluations by a plurality of users in each period, and video data obtained by connecting the selected plurality of selected videos is used as a terminal device. To deliver.

First, the configuration of the video distribution system 100 according to the present embodiment will be described. FIG. 1 is a block diagram showing an overall configuration of the video distribution system 100 according to the present embodiment. The video distribution system 100 includes a plurality of cameras 101, a terminal device 102, and a server 103, each of which can communicate via the network 104A or 104B.

The plurality of cameras 101 generate a plurality of viewpoint images by shooting the same scene from different viewpoints in the same time zone. Each camera 101 is carried by each of a plurality of users. For example, the plurality of cameras 101 are owned by a plurality of spectators in a place such as a sports stadium. The plurality of viewpoint images taken by the plurality of cameras 101 are transmitted to the server 103 via the network 104A. In addition, the camera 101 also acquires data such as position information used for calculating the degree of relevance between viewpoint images, and transmits the data to the server 103. The camera 101 may be a device having at least a photographing function, and is, for example, a digital still camera, a digital video camera, a smartphone, a mobile terminal, or the like. Further, the plurality of cameras 101 may include not only a camera owned by the audience but also a fixed camera or a camera for broadcasting.

Networks 104A and 104B are, for example, the Internet. Although networks 104A and 104B are shown individually in FIG. 1, a plurality of cameras 101, a terminal device 102, and a server 103 may be connected via a single network. In addition, part or all of the communication between the devices may be performed directly without going through the network 104A or 104B. Further, the connection between the devices may be either a wired connection or a wireless connection.

The terminal device 102 is a terminal used by the user 106, and is connected to a monitor 105 that displays an image. For example, the terminal device 102 is a smartphone, a mobile terminal, a personal computer, or the like. The terminal device 102 has the same function as the camera 101, and the user may be included in the audience, or the user may watch the video from a place other than the stadium.

The terminal device 102 receives the viewpoint image from the server 103 according to the instruction of the user 106, and outputs the viewpoint image to the monitor 105 by a method according to the instruction of the user 106. The monitor 105 may be any output device such as a PC monitor, a tablet terminal, a smartphone, a mobile phone, or a notebook PC monitor. Although the terminal device 102 and the monitor 105 are described separately here, the monitor 105 may be included in the terminal device 102.

Further, although the instruction from the user is given by, for example, a screen touch operation or a mouse, any input device may be used. The instructions of the user 106 are to select the viewpoint image to be viewed, leave a comment on the viewpoint image, evaluate the viewpoint image, leave a comment on the subject reflected in the viewpoint image, evaluate the subject, and examine the subject information. , And instructions such as ordering related goods related to the subject. A part of the user's instruction is transmitted to the server 103.

The server 103 temporarily holds a plurality of viewpoint images transmitted from the plurality of cameras 101. Further, the server 103 transmits a part or all of the plurality of viewpoint images to be held to the terminal device 102 in accordance with the request from the terminal device 102. Further, the server 103 performs processing such as switching of the viewpoint video according to the request from the terminal device 102, and transmits the video data obtained by the processing to the terminal device 102. Further, the server 103 receives the evaluation of the viewpoint image or the subject in the viewpoint image by the user from the terminal device 102, aggregates it as a comment and the evaluation for the viewpoint image or the subject, and stores it in the database.

Further, the server 103 transmits information to the terminal device 102 in accordance with a viewpoint image or an information presentation request for the subject from the terminal device 102. Further, the server 103 identifies a subject included in a plurality of viewpoint images captured by the camera 101 by using the information of the subject specified by the user transmitted from the terminal device 102. Specifically, the server 103 uses subject information (name, uniform number, etc.) specified by the user, or a subject in the viewpoint image specified by the user by touch operation, mouse operation, gesture operation, line-of-sight operation, or voice operation. The pixel information of the above is received, and the subject is specified by image processing such as feature amount matching between a plurality of viewpoint images captured by the camera 101 and the received image information. The server 103 may create a composite image by synthesizing a plurality of viewpoint images taken by a plurality of cameras 101. Further, this composite image may be included in a plurality of viewpoint images transmitted to the terminal device 102.

A plurality of viewpoint images are transmitted in real time from the plurality of cameras 101, and the user 106 may view the viewpoint images in real time using the terminal device 102, or at least one of transmission and viewing of the images is in real time. It does not have to be done in. That is, the user 106 may view the viewpoint video captured in the past at any timing. Further, the transmission and reception of the video signal (video) described below mainly means stream transmission and reception in which the video signal is continuously transmitted or received.

The configuration of each device will be described below. FIG. 2 is a block diagram showing the configuration of the terminal device 102. The terminal device 102 includes a receiving unit 111, a storage unit 112, an image processing unit 113, an output unit 114, a transmitting unit 115, and a control unit 116.

The receiving unit 111 receives the video data transmitted from the server 103. Specifically, this video data includes a viewpoint video instructed by the user. Further, this video data includes a video having a high degree of relevance to the viewpoint video as sub data. For example, this video is another viewpoint video whose shooting position is close to that of the viewpoint video being viewed, or another viewpoint video including a subject in the viewpoint video being viewed. In this way, the server 103 may transmit a video having a high degree of relevance to the viewpoint video being viewed to the terminal device 102 at the same time as the viewing video. In addition, the video data includes not only the viewpoint video specified by the user, but also comments, evaluations, and related information on the viewpoint video or the subject. In addition, the server 103 transmits information and the like of the viewpoint video with the largest number of viewers aggregated by the server 103.

The storage unit 112 temporarily holds the video data received by the reception unit 111. This enables seamless display when the viewpoint is switched.

The image processing unit 113 performs decoding processing when the viewpoint image is encoded. Further, the image processing unit 113 generates an output video to be presented to the user 106 by using the viewpoint video, the information about the viewpoint video received from the server 103, and the information about the subject. At this time, the image processing unit 113 separately acquires a viewpoint image having a large number of views from the server 103, statistical information of a subject having many comments or a high evaluation, and an output image that emphasizes the information. May be created. For example, the image processing unit 113 may emphasize these by illuminating the selection icon of the viewpoint image having a large number of viewers, or illuminating the outline of a highly evaluated subject or the like.

The output unit 114 outputs the output video generated by the image processing unit 113 to the monitor 105.

The control unit 116 receives a user instruction. Here, the user's instruction includes (1) a viewpoint image switching request, (2) an evaluation addition request, and (3) an information acquisition request.

(1) The viewpoint image switching request is a request for switching the viewpoint image to be viewed, and is (1-1) favorite viewpoint image switching mode, (1-2) favorite subject switching mode, and (1-3) recommended viewpoint image. It includes a switching mode and (1-4) recommended subject switching mode.

(1-1) The favorite viewpoint image switching mode is a mode in which the user 106 selects a favorite viewpoint image registered in advance. (1-2) The favorite subject switching mode is a mode in which the user 106 selects a viewpoint image including a favorite subject registered in advance. Further, in the favorite viewpoint image switching mode and the favorite subject switching mode, the user can select the viewpoint image or the subject at any time while viewing the image as well as the viewpoint image including the favorite viewpoint image or the subject registered in advance by the user.

(1-3) The recommended viewpoint image switching mode is a mode for selecting a viewpoint image that is highly evaluated by a plurality of users. (1-4) The recommended subject switching mode is a mode for selecting a viewpoint image including a subject highly evaluated by a plurality of users.

(2) The evaluation addition request is a request for evaluating the viewpoint image or the subject, and includes (2-1) the viewpoint image evaluation addition mode and (2-2) the subject evaluation addition mode. (2-1) Viewpoint image evaluation The additional mode is a mode for the user 106 to evaluate the viewpoint image. (2-2) Subject evaluation addition mode is a mode for the user 106 to evaluate a subject included in the viewpoint image.

(3) The information acquisition request is a request for acquiring information on the viewpoint image or the subject, and includes (3-1) the viewpoint image information acquisition mode and (3-2) the subject information acquisition mode. (3-1) The viewpoint video information acquisition mode is a mode for acquiring information on the viewpoint video. (3-2) The subject information acquisition mode is a mode for acquiring the subject information included in the viewpoint image.

When the user's instruction is (1) a viewpoint video switching request, the control unit 116 transmits the designated viewpoint video position information or the designated subject position information to the server 103 via the transmission unit 115.

Further, when the user's instruction is (2) an evaluation addition request, the control unit 116 provides a comment or evaluation information for the viewpoint image and time information at the time of adding the evaluation, or a comment or evaluation information for the designated subject and the evaluation information thereof. The time information and the like at the time of adding the evaluation are transmitted to the server 103 via the transmission unit 115.

Further, when the user's instruction is (3) an information acquisition request, the control unit 116 provides the server 103 with information on the position of the designated viewpoint image, information on the designated subject position, information on the information acquisition content, and the like. Is transmitted via the transmission unit 115 to. Here, the information acquisition content is information about the viewpoint image and the subject, for example, (1) a comment or evaluation for the viewpoint image or the subject, (2) a profile of the photographer or the subject of the viewpoint image, or (3) a viewpoint. Information on other contents taken by the photographer of the video, information on goods related to the subject, and the like.

When the subject is specified as in the case where the user's instruction is (1-2) favorite subject switching mode, (2-2) subject evaluation addition mode, or (3-2) subject information acquisition mode. In order to identify the subject specified by the user on the server 103 side, the subject information (name or uniform number, etc.) specified by the user, or the user by touch operation, mouse operation, gesture operation, line-of-sight operation, voice operation, etc. The pixel information of the subject in the specified viewpoint image is transmitted to the server 103.

The operation of the terminal device 102 will be described below. FIG. 3 is a flowchart of the entire process by the terminal device 102.

First, the terminal device 102 determines whether the user instruction is (1) a viewpoint image switching request, (2) an evaluation addition request, or (3) an information acquisition request (S101, S103, S105). ..

When the user instruction is a viewpoint image switching request (Yes in S101), the terminal device 102 performs the viewpoint image switching process (S102). When the user's instruction is an evaluation addition request (Yes in S103), the terminal device 102 performs the evaluation addition process (S104). When the user's instruction is an information acquisition request (Yes in S105), the terminal device 102 performs the information acquisition process (S106).

It should be noted that the terminal device 102 does not necessarily have to perform all of the above determinations, and may treat the above determinations as an option, for example, determining whether or not the user's instruction is a viewpoint video switching request. No.

Hereinafter, the details of the viewpoint image switching process (S102), the evaluation addition process (S104), and the information acquisition process (S106) will be described.

First, the viewpoint image switching process (S102) will be described. FIG. 4 is a flowchart of the viewpoint image switching process by the terminal device 102.

First, in the terminal device 102, whether the viewpoint image switching mode is (1-1) favorite viewpoint image switching mode, (1-2) favorite subject switching mode, or (1-3) recommended viewpoint image switching mode. (1-4) It is determined whether the mode is the recommended subject switching mode (S121, S123 and S126).

When the viewpoint video switching mode is the favorite viewpoint video switching mode (Yes in S121), the terminal device 102 requests the server 103 to deliver the viewpoint video at the viewpoint video position specified by the user 106 (S122).

When the viewpoint image switching mode is the favorite subject switching mode (Yes in S123), the terminal device 102 transmits the information of the subject specified by the user to the server 103 (S124). Here, the subject information is, for example, subject information such as the name or uniform number of the subject, or pixel information of the subject in the viewpoint image designated by the user by touch operation or mouse operation. The subject information (name, uniform number, etc.) is acquired by, for example, the user designating the subject information output on the screen by touch operation, mouse operation, voice, or the like. Further, the pixel information of the subject in the viewpoint image includes, for example, the pixel position or pixel information of the subject in the viewpoint image.

Further, the terminal device 102 requests the server to deliver the tracking image of the subject specified by the user (S125).

When the viewpoint video switching mode is the recommended viewpoint video switching mode (Yes in S126), the terminal device 102 requests the server 103 to distribute the recommended viewpoint video generated by the server 103 (S127).

When the viewpoint image switching mode is the recommended subject switching mode (No in S126), the terminal device 102 requests the server 103 to deliver the recommended subject tracking image generated by the server 103 (S128).

After steps S122, S125, S127 or S128, the terminal device 102 receives the video data transmitted from the server 103 and stores the received video data in the storage unit 112 (S129). The video data received from the server 103 may include, in addition to the viewpoint video, a video produced by processing or editing on the server 103 side.

Further, the terminal device 102 does not have to wait until all the reception of the video data received from the server 103 is completed, and the terminal device 102 may start the reproduction from the time when a certain amount of data is received. Further, the terminal device 102 may receive the video data from the server 103 and reproduce the previously reproduced video data until the reproduction can be started.

Next, the terminal device 102 extracts the video data to be displayed from the storage unit 112 and decodes the video data (S130). Next, the terminal device 102 generates a composite video (output video) by synthesizing the decoded video data and the display user interface (S131).

Next, the terminal device 102 outputs the composite video to a monitor 105 such as a display (S132). As a result, the composite image is displayed on the monitor 105.

Hereinafter, the evaluation addition process (S104) will be described. FIG. 5 is a flowchart of the evaluation addition process by the terminal device 102.

First, the terminal device 102 determines whether the evaluation addition mode is (2-1) viewpoint image evaluation addition mode or (2-2) subject evaluation addition mode (S141).

When the evaluation addition mode is the viewpoint video evaluation addition mode (Yes in S141), the terminal device 102 transmits the evaluation information for the viewpoint video being viewed by the user 106 and the time information at the time of evaluation addition to the server 103 (S142). .. Here, the evaluation information is a comment or evaluation for the viewpoint video. Further, the evaluation may be information indicating that the viewpoint image is good or bad, or may be information indicating a good degree or a bad degree.

When the evaluation addition mode is the subject evaluation addition mode (No in S141), the terminal device 102 transmits the information of the subject specified by the user to the server 103 (S143). The content of the subject information and the acquisition method thereof are the same as those in step S124 described above.

The subject information (name, uniform number, etc.) is acquired by, for example, the user designating the subject information output on the screen by touch operation, mouse operation, voice, or the like. Further, the pixel information of the subject in the viewpoint image includes, for example, the pixel position or pixel information of the subject in the viewpoint image.

Next, the terminal device 102 transmits the evaluation information for the subject designated by the user and the time information at the time of adding the evaluation to the server 103 (S144).

Hereinafter, the information acquisition process (S106) will be described. FIG. 6 is a flowchart of information acquisition processing by the terminal device 102.

First, the terminal device 102 determines whether the information acquisition mode is (3-1) the viewpoint video information acquisition mode or (3-2) the subject information acquisition mode (S161).

When the information acquisition mode is the viewpoint video information acquisition mode (Yes in S161), the terminal device 102 transmits a request for acquiring information regarding the viewpoint video being viewed by the user to the server 103 (S162). Here, the information regarding the viewpoint image is, for example, a comment or evaluation for the viewpoint image, a profile of the photographer of the viewpoint image, information on other contents photographed by the photographer of the viewpoint image, or the like.

When the information acquisition mode is the subject information acquisition mode (No in S161), the terminal device 102 transmits the information of the subject specified by the user to the server 103 (S163). The content of the subject information and the acquisition method thereof are the same as those in step S124 described above.

Next, the terminal device 102 transmits a request for acquiring information about the subject specified by the user to the server 103 (S164). Here, the subject information is, for example, a comment or evaluation for the subject, a profile of the subject, information on related goods of the subject, or the like.

After step S162 or S164, the terminal device 102 receives the viewpoint image or information about the subject transmitted from the server (S165). Next, the terminal device 102 adds information about the received viewpoint video or subject to the display user interface, and generates a composite video by synthesizing the interface with the video data being reproduced (S166). The terminal device 102 may superimpose and display the received viewpoint video or information on the subject on the video data being reproduced.

Next, the terminal device 102 outputs the generated composite video to the monitor 105 (S167). As a result, the composite image is displayed on the monitor 105.

The configuration of the server 103 will be described below. FIG. 7 is a block diagram showing the configuration of the server 103. The server 103 shown in FIG. 7 includes a receiving unit 121, a storage unit 122, an image processing unit 123, a transmitting unit 124, and a control unit 126.

The receiving unit 121 receives a plurality of viewpoint images obtained by the plurality of cameras 101 shooting the same scene from different viewpoints in the same time zone, and shooting camera information from the plurality of cameras 101. The shooting camera information includes, for example, position information of the camera 101, shooting time information, profile information of the photographer, and the like. Here, the shooting camera position information is, for example, information obtained by a sensor such as GPS built in the shooting camera. The shooting camera position information may be wireless information such as WiFi (registered trademark) or position information estimated from the feature amount of the shooting image. Further, the server 103 may estimate the position of the camera 101 from wireless information such as WiFi or a feature amount of the captured image.

The storage unit 122 stores a plurality of viewpoint images received by the reception unit 121 and shooting camera information. Further, the storage unit 122 stores the comment or evaluation added by the user to the viewpoint image or the subject together with the time information to which these are added. In addition, the storage unit 122 stores collected data such as a viewpoint image having a large number of comments or a high evaluation, or statistics on the subject. In addition, the storage unit 122 may store information about the viewpoint image and the subject, such as a profile of the photographer or the subject of the viewpoint image, or information on goods related to the subject.

The image processing unit 123 generates video data by processing or editing the viewpoint video in response to the viewpoint video switching request from the terminal device 102, and transmits the generated video data to the terminal device 102 via the transmission unit 124. Send. Further, the image processing unit 123 includes a subject indicated by image information by feature quantity matching or the like between the pixel information of the subject in the viewpoint image received from the terminal device 102 and the plurality of viewpoint images captured by the camera 101. The viewpoint image is specified, and the specified result is sent to the control unit 126.

The transmission unit 124 transmits the video data generated by the image processing unit 123.

The control unit 126 receives data from the terminal device 102 and the Internet (network 104A or 104B). Specifically, the control unit 126 receives the viewpoint image switching request, the evaluation addition request, and the information acquisition request from the terminal device 102. Further, the control unit 126 creates a search key for the subject, acquires the profile of the subject or information on related goods by an Internet search, stores the subject in the storage unit 122, or transmits the information to the terminal device 102. In addition, the control unit 126 sends and receives data to and from the web store for purchasing related goods.

The operation of the server 103 will be described below. FIG. 8 is a flowchart of the entire process by the server 103.

First, the server 103 receives a plurality of viewpoint images transmitted from the plurality of cameras 101, and stores the images together with the shooting camera information in the storage unit 122 (S201).

Next, the server 103 determines whether or not the viewpoint image switching request has been received from the terminal device 102 (S202). When the viewpoint video switching request is received (Yes in S202), the server 103 performs the viewpoint video switching process (S203).

Further, the server 103 determines whether or not the evaluation addition request has been received from the terminal device 102 (S204). When the evaluation addition request is received (Yes in S204), the server 103 performs the evaluation addition process (S205).

Further, the server 103 determines whether or not the information acquisition request has been received from the terminal device 102 (S206). When the information acquisition request is received (Yes in S206), the server 103 performs the information acquisition process (S207).

The processing order shown in FIG. 8 is an example, and the order of the determination processing in steps S202, S204, and S206 may be arbitrary.

The evaluation information added to the viewpoint image or the subject is not necessarily limited to the one directly acquired from the user 106, and may be acquired separately. For example, these evaluation information may be separately acquired from SNS or the like.

Hereinafter, the details of the viewpoint image switching process (S203), the evaluation addition process (S205), and the information acquisition process (S207) will be described.

First, the viewpoint image switching process (S203) will be described. FIG. 9 is a flowchart of the viewpoint image switching process by the server 103.

First, the server 103 determines whether the viewpoint image switching mode is (1-1) favorite viewpoint image switching mode, (1-2) favorite subject switching mode, or (1-3) recommended viewpoint image switching mode. It is determined whether there is (1-4) the recommended subject switching mode (S221, S223 and S226).

When the viewpoint video switching mode is the favorite viewpoint video switching mode (Yes in S221), the server 103 selects the viewpoint video at the received viewpoint video position from the storage unit 122 and sets it as the video data to be transmitted to the terminal device 102. (S222).

When the viewpoint image switching mode is the favorite subject switching mode (Yes in S223), the server 103 identifies the subject specified by the user using the subject information received from the terminal device 102 (S224). Specifically, the server 103 identifies the subject by feature amount matching or the like using subject information (name or uniform number, etc.), pixel information of the subject in the viewpoint image, and a plurality of viewpoint images of a group of shooting cameras. ..

Next, the server 103 selects a plurality of viewpoint images including the specified subject from the storage unit 122, and transmits the viewpoint image to the terminal device 102 by switching the viewpoint image every hour according to the ratio of the area occupied by the subject in the image. The video data to be generated is generated (S225). The server 103 may preferentially select not only the ratio of the area occupied by the subject in the image but also the viewpoint image in which the front face of the subject is always shown, or the subject is always reflected in the center. You may preferentially select the viewpoint image you are in. That is, the server 103 may select an image having a good image of the subject according to the shooting situation.

When the viewpoint image switching mode is the recommended viewpoint image switching mode (Yes in S226), the server 103 automatically switches the viewpoint image every hour according to the statistical data of the highly evaluated viewpoint image, so that the terminal device 102 Generate video data to be transmitted to (S227).

Here, the highly evaluated viewpoint image is a viewpoint image that has been highly evaluated by the user. For example, a menu for giving a high evaluation is prepared for each viewpoint image, and when the user selects the menu, the high evaluation of the viewpoint image is increased by 1. And the viewpoint image with high evaluation is the viewpoint image having the largest number of high evaluations or more than a predetermined value. Further, a menu for giving a low evaluation may be prepared. In this case, the evaluation of the viewpoint image may be determined based on the difference between the high evaluation and the low evaluation.

Further, the evaluation given by the user may be a stepwise numerical value such as 1 to 5. For example, 5 indicates the best and 1 indicates the worst. In such a case, for example, the evaluation of the viewpoint image may be determined according to the average value of the evaluation.

In addition, the number of evaluations given (both the number of high evaluations and the number of low evaluations) may be added to the above judgment. Further, it may be determined that the viewpoint video having a large number of playbacks (viewing times) is highly evaluated, or the viewpoint video having a long playback time (viewing time) may be highly evaluated.

In addition, a menu is prepared for the user to add a comment to the viewpoint video, and it may be judged that the higher the number of comments by the user, the higher the evaluation. Further, the content of the comment may be analyzed, and a high evaluation, a low evaluation, or a stepwise numerical value may be determined according to the content of the comment.

In addition, it may be determined that the larger the number of viewpoint images registered as favorites by the user, the higher the evaluation.

Further, two or more of the above determination methods may be combined.

Further, the statistical data may be statistical data of the entire shooting scene or statistical data of each period in which the shooting scene is divided. For example, in the case of a sports match, the statistical data may be statistical data for the entire period of the match, the first half and the second half, or statistical data for each predetermined period. Further, the statistical data may be statistical data for a long period of time beyond the shooting scene. For example, in the case of a sports match, the statistical data may be all statistical data of a plurality of games played within a week or a month, or statistical data during one or a plurality of seasons. good.

When the viewpoint image switching mode is the recommended subject switching mode (No in S226), the server 103 identifies the subject from the statistical data of the highly evaluated subject (S228). The subject with a high evaluation is determined by the same method as the above-mentioned viewpoint image with a high evaluation.

Next, the server 103 selects a plurality of viewpoint images including the specified subject from the storage unit 122, and switches the viewpoint image for each time according to the ratio of the area occupied by the subject in the image to the terminal device 102. Generate video data to be transmitted (S229). The server 103 may preferentially select not only the ratio of the area occupied by the subject in the image but also the viewpoint image in which the front face of the subject is always shown, or the subject is always reflected in the center. You may preferentially select the viewpoint image you are in. That is, the server 103 may select an image having a good image of the subject according to the shooting situation.

After steps S222, S225, S227 or S229, the server 103 transmits the generated video data to the terminal device 102 (S230).

In addition, when the server 103 does not show a moving subject in the viewpoint image in the favorite viewpoint image switching mode or the favorite subject switching mode, or when the shooting is interrupted due to the battery shortage of the camera shooting the viewpoint image, etc. When a specific event occurs, it may be detected and the image to be viewed may be automatically switched to another viewpoint image whose shooting position is close to the viewpoint image viewed by the user 106. Further, the server 103 may switch the switching mode to the recommended viewpoint image switching mode or the recommended subject switching mode. On the contrary, the server 103 may switch the switching mode to the favorite viewpoint image switching mode or the favorite subject switching mode when a specific event occurs in the recommended viewpoint image switching mode or the recommended subject switching mode. good.

Further, the number of comments and the evaluation value for the subject do not necessarily have to be limited to those added by the user, and the server 103 may acquire these separately. For example, the server 103 may set an initial value in advance based on the number of search results of the Web search by the name of the subject.

Hereinafter, the evaluation addition process (S205) will be described. FIG. 10 is a flowchart of the evaluation addition process by the server 103.

First, the server 103 determines whether the evaluation addition mode is (2-1) viewpoint image evaluation addition mode or (2-2) subject evaluation addition mode (S241).

When the evaluation addition mode is the viewpoint video evaluation addition mode (Yes in S241), the server 103 receives the evaluation information which is a comment or evaluation for the viewpoint video being viewed by the user and the time information at the time of evaluation addition from the terminal device 102. Then, the evaluation information of the received viewpoint image is stored in the storage unit 122 (S242).

When the evaluation addition mode is the subject evaluation addition mode (No in S241), the server 103 identifies the subject using the subject information received from the terminal device 102 (S243). The content of the subject information and the method of identifying the subject are the same as in step S224 described above.

Next, the server 103 receives the evaluation information for the subject designated by the user and the time information at the time of adding the evaluation from the terminal device 102, and stores the received evaluation information of the subject in the storage unit 122 (S244).

After step S242 or S244, the server 103 periodically updates the highly evaluated viewpoint image or subject statistical information by using the evaluation information for the viewpoint image or subject stored in the storage unit 122 and the time information at the time of adding the evaluation. Then, the updated statistical information is saved in the storage unit 122 (S245).

Hereinafter, the information acquisition process (S207) will be described. FIG. 11 is a flowchart of information acquisition processing by the server 103.

First, the server 103 determines whether the information acquisition mode is (3-1) the viewpoint video information acquisition mode or (3-2) the subject information acquisition mode (S261).

When the information acquisition mode is the viewpoint video information acquisition mode (Yes in S261), the server 103 reads the information about the viewpoint video being viewed by the user from the storage unit 122 and transmits the read information to the terminal device 102 (S262). ). Here, the information regarding the viewpoint image is, for example, a comment or evaluation for the viewpoint image, a profile of the photographer of the viewpoint image, information on other contents photographed by the photographer of the viewpoint image, or the like.

When the information acquisition mode is the subject information acquisition mode (No in S261), the server 103 identifies the subject designated by the user 106 using the subject information received from the terminal device 102 (S263). The content of the subject information and the acquisition method thereof are the same as those in step S224 described above. Further, the server 103 may narrow down the subjects to a few people by feature amount matching or the like, and let the user 106 select the target subject from among them.

Next, the server 103 reads the information about the subject designated by the user 106 from the storage unit 122, and transmits the read information to the terminal device 102 (S264). Here, the subject information is, for example, a comment or evaluation for the subject, a profile of the subject, information on related goods of the subject, or the like.

An operation example of the viewpoint image switching process will be described below. FIG. 12 is a diagram showing an operation example of the favorite subject switching mode.

In this example, the server 103 selects the viewpoint image A, the viewpoint image B, and the viewpoint image C including the specified subject from the storage unit 122, and the viewpoint is set for each time according to the ratio of the area occupied by the subject in the image. By switching the video, the video data to be transmitted to the terminal device 102 is generated. In the first period, the ratio of the area occupied by the subject specified by the user in the viewpoint image A is large. Therefore, in this period, the viewpoint image A is selected. Further, in the next period, the ratio of the area occupied by the subject designated by the user in the viewpoint image C is large. Therefore, in this period, the viewpoint image C is selected. In the final period, the ratio of the area occupied by the subject specified by the user in the viewpoint image B is large. Therefore, in this period, the viewpoint image B is selected. In this way, an image in which a plurality of viewpoint images are joined is generated according to the area of the subject.

If the continuous reproduction time of a certain viewpoint image does not exceed the threshold value, the server 103 may control so that the viewpoint image is not switched until the threshold value is exceeded. This makes it possible to prevent frequent switching of the viewpoint image. For example, even if the area of the subject is larger in the viewpoint image C than in the viewpoint image A, if the continuous playback time of the viewpoint image A does not exceed, for example, 5 seconds, the server 103 continuously continues the viewpoint image A. And play. Further, the server 103 switches to the viewpoint image C only when the subject is photographed on the viewpoint image C for, for example, 2 seconds or more.

FIG. 13 is a diagram showing an operation example of the recommended viewpoint image switching mode. As shown in FIG. 13, the server 103 generates video data to be transmitted to the terminal device 102 by switching the viewpoint video every hour according to the statistical data of the viewpoint video having a high evaluation. Note that the server 103 may control switching according to the continuous playback time, as in the case of the favorite subject switching mode.

FIG. 14 is a diagram showing an operation example of the recommended subject switching mode. As shown in FIG. 14, the server 103 selects a plurality of viewpoint images including a highly evaluated subject from the storage unit 122, and switches the viewpoint image every hour according to the ratio of the area occupied by the subject in the image. , Generates video data to be transmitted to the terminal device 102.

Note that the server 103 may control switching according to the continuous playback time, as in the case of the favorite subject switching mode.

Further, in the above description, in the case of performing the above switching, an example of generating a single video stream by switching on the server 103 and transmitting the video stream to the terminal device 102 has been described. A part of the processing may be performed by the terminal device 102. For example, the server 103 transmits a plurality of viewpoint images and information indicating the switching timing of the viewpoint images to the terminal device 102, and the terminal device 102 reproduces (displays) the plurality of viewpoint images based on the information. You may switch the viewpoint image to be used.

Hereinafter, an example of a display screen in the terminal device 102 will be described. FIG. 15 is a diagram showing an example of an output screen of the terminal device 102. Further, FIG. 15 shows a state in which a live image is displayed.

The output screen 300A shown in FIG. 15 includes a viewpoint image display screen 301, a viewpoint selection screen 302, operation buttons 303A to 303C, 304A and 304B, and information display screens 305 and 306.

The viewpoint video display screen 301 displays the viewpoint video selected by the user or the video data generated by the server 103 in the mode selected by the user.

When the user selects a player on the viewpoint image display screen 301 by touching or clicking the mouse, the circle-shaped operation menu 307C is displayed. The operation menu 307C includes a plurality of menus, and menus of "high evaluation", "low evaluation", "player information", and "favorite" are arranged clockwise from the top. When the user selects the "high evaluation" or "low evaluation" menu, an evaluation (high evaluation or low evaluation) is added to the selected player (subject evaluation addition mode).

The evaluation may be made by adding points such as 1 to 6 points, or may be added by comments such as "very good" and "good". Further, as the evaluation method, not only text or voice but also a method such as conveying a comment by a picture may be used.

Further, the value of points may be improved by setting an upper limit on the number of points that can be added by each user. In addition, the user may pay money to increase the upper limit of points that can be added.

In addition, when the user selects the "player information" menu, information about the player such as comments, evaluations, related goods information, or results for the player is displayed (subject information acquisition mode).

Further, when the user selects the "favorite" menu, the viewpoint video at or before and after the selected time may be registered as the favorite play of the selected player (subject evaluation addition mode).

Further, the server 103 or the terminal device 102 may access a website or the like related to related goods information, results, information about athletes, or the like, and the terminal device 102 may display the contents thereof. Further, the server 103 or the terminal device 102 may access a website or the like where related goods can be purchased, and the terminal device 102 may display the contents thereof. As a result, the viewer can easily purchase the goods while watching the game. Therefore, the service provider can connect the viewer and the goods sales company through this service, and can expect the effect of obtaining the margin from the goods sales company.

In addition, when the user selects the "favorite" menu, the selected player is registered as the user's favorite player. Then, the player registered in the favorite subject switching mode is used as the favorite subject.

Although only one viewpoint image display screen 301 exists here, a plurality of viewpoint image display screens may be displayed at the same time. For example, the output screen includes three viewpoint video display screens, the first viewpoint video display screen displays the viewpoint video in which the entire court is photographed, and the second viewpoint video display screen is designated by the user. The viewpoint image is displayed (favorite viewpoint image switching mode), and the viewpoint image of the subject specified by the user (favorite subject switching mode) is displayed on the third viewpoint image display screen. Further, the viewpoint image display screen 301 may display an image created by synthesizing a plurality of viewpoint images. In this case, for example, viewing from a multi-angle can be performed by changing the angle to the left, right, up and down while the user touches the screen.

In addition, a player with a high evaluation may be highlighted on the viewpoint video display screen 301. For example, the outline of a player with a high evaluation may be displayed shiningly.

The viewpoint selection screen 302 is a diagram showing the positions of a plurality of cameras 101 (viewpoint positions of viewpoint images). For example, in the example shown in FIG. 15, the positions of the plurality of cameras 101 are indicated by circles on the top view of the match venue.

Further, the positions of the circles with hatches indicate the viewpoint positions of the viewpoint image displayed on the current viewpoint image display screen 301. When the user selects the circle mark at each viewpoint position by touching or clicking the mouse, the displayed viewpoint image is switched to the viewpoint image at that position (favorite viewpoint image switching mode).

The viewpoint position of the highly evaluated viewpoint image may be highlighted on the viewpoint selection screen 302. Further, the server 103 may analyze voice data such as cheers measured at each viewpoint position, and highlight the viewpoint position where the cheers occur most at that time on the viewpoint selection screen 302. In addition, the viewpoint video in which the cheers occur most may be switched and displayed in order.

Further, when the user long-presses and selects the viewpoint position on the viewpoint selection screen 302 by touching or clicking the mouse, the circle-shaped operation menu 307A is displayed. The operation related to the operation menu 307A is the same as the operation related to the operation menu 307C described above, except that the target is changed from the player (subject) to the viewpoint image.

Specifically, the operation menu 307A includes a plurality of menus, and the menus of "high evaluation", "low evaluation", "viewpoint image information", and "favorite" are arranged clockwise from the top. When the user selects the "high evaluation" or "low evaluation" menu, an evaluation (high evaluation or low evaluation) is added to the selected viewpoint image (viewpoint image evaluation addition mode). The evaluation may be made by adding points such as 1 to 6 points, or may be added by comments such as "very good" and "good".

When the user selects the "viewpoint video information" menu, comments and evaluations for the viewpoint video, the profile of the photographer, other content information of the photographer, and the like are displayed (viewpoint video information acquisition mode).

Further, when the user selects the "favorite" menu, the viewpoint video at the time of selection or before and after the selection may be registered as a favorite time zone of the selected viewpoint video. (Viewpoint video evaluation addition mode). The registered favorite time zone is used when viewing the recording. For example, when viewing a recording, another viewpoint video in a favorite time zone is displayed.

When the user selects the "favorite" menu, the selected viewpoint image is registered as a favorite image. Then, the registered favorite video is used as the viewpoint video in the favorite viewpoint video switching mode.

The operation buttons 303A to 303C are buttons for selecting a favorite subject switching mode, a recommended subject switching mode, and a recommended viewpoint image switching mode, respectively.

The operation buttons 304A and 304B are buttons for switching between live video and recorded video. For example, when the user selects the operation button 304A, the live image is displayed on the viewpoint image display screen 301. When the user selects the operation button 304B, the recorded video is displayed on the viewpoint video display screen 301.

The information display screen 305 displays information such as the current game. For example, a circle with a number shown in FIG. 15 indicates a player's uniform number. Further, for example, the upper jersey number is the jersey number of the player who is currently participating in the game, and the lower jersey number is the jersey number of the substitute player.

When the user selects the player's uniform number by touching or clicking the mouse, the circle-shaped operation menu 307B is displayed. The operation related to the operation menu 307B is the same as the operation related to the operation menu 307C described above.

The information display screen 306 displays a total of evaluations and the like added by the user to the subject (player). Similar to the information display screen 305, the circle with the number indicates the player's uniform number. On the right side of each uniform number, the evaluations of a plurality of users for the player with the uniform number are shown. Specifically, the upper row shows the number of high evaluations, and the lower row shows the number of low evaluations. Further, in the example shown in FIG. 15, each of the high evaluation and the low evaluation is evaluated in 6 stages (1 to 6), and the number of evaluations given in each stage is described. In addition, the value aggregated for each predetermined period is displayed for this evaluation.

Further, at the left end of the information display screen 306, the time registered as a favorite scene with the vertical direction as the time axis is indicated by a star.

In addition, when the user clicks on the player's uniform number, the comment added to the player and the details of the evaluation are displayed.

Further, the aggregated value displayed on the information display screen 306 may be updated at regular time intervals. Alternatively, this aggregated value may be updated each time a rating is added by the user. Further, the points added to the subject may be fed back to the subject, such as being reflected in the annual salary of the subject.

FIG. 16 is a diagram showing an example of an output screen of the terminal device. Further, FIG. 16 shows a state in which the recorded video is displayed. For example, in the output screen 300A shown in FIG. 15, when the user selects the operation button 304B, the output screen 300B shown in FIG. 16 is displayed.

The output screen 300B shown in FIG. 16 includes an information display screen 308 instead of the information display screen 305 with respect to the output screen 300A shown in FIG.

The horizontal direction of the output screen 300B indicates the time axis, and the degree of excitement (for example, the volume of cheers) is displayed by the waveform. In addition, the star mark in the lower row indicates the time registered as a favorite, and the circle mark in the middle row indicates the time of the recommended scene. In addition, the circle in the lower row indicates the uniform number of the player who put the score of one team, and the numerical value above it indicates the score. Similarly, the circle in the upper row indicates the uniform number of the player who put the score of the other team, and the number below it indicates the score.

In addition, when the viewpoint image developed as a recommendation is viewed by more people, points may be returned to the photographer of the viewpoint image.

Further, at the time of displaying the recorded video, the viewing point may be displayed according to the viewpoint video or subject to which the user has added a favorite, the viewpoint video or subject with high evaluation, or the audio data such as cheers.

Further, when a certain subject is selected, a play collection according to the evaluation added to the subject may be displayed in a ranking format or the like, and this play collection may be reproduced. For example, when the user selects a player with a uniform number 5, a play collection in which high points (for example, 6 points) are added to the player with a uniform number 5 is displayed. Then, one play selected from them is reproduced.

Further, when the comment of each evaluation is displayed, a list of viewpoint images taken by the user who posted the comment may be displayed at the same time. In addition, the photographer may be registered as a favorite. In this case, there may be a mode in which only the viewpoint image taken by the photographer registered as a favorite can be viewed.

FIG. 17 is a diagram showing another example of the output screen according to the present embodiment. The output screen 300C shown in FIG. 17 includes a main screen 311, a first sub screen 312A, a second sub screen 312B, and a viewpoint selection screen 313.

The viewpoint image specified by the user is displayed on the main screen 311 (favorite viewpoint image switching mode). On the first sub screen 312A, the recommended viewpoint image generated in the above-mentioned recommended viewpoint image switching mode is displayed. On the second sub-screen 312B, the recommended subject image generated in the above-mentioned recommended subject switching mode is displayed.

In this state, when the user clicks the first sub screen 312A, the recommended viewpoint image is displayed on the main screen 311. Then, the next recommended viewpoint image is displayed on the first sub screen 312A.

Similarly, when the user clicks on the second sub screen 312B, the recommended subject image is displayed on the main screen 311. Then, the next recommended subject image is displayed on the second sub screen 312B.

Further, when the user specifies a subject in the viewpoint image displayed on the main screen 311 by clicking or the like, the mode shifts to the favorite subject switching mode, and the tracking image of the designated subject is displayed. Further, when the user clicks the main screen 311 while displaying the favorite subject image, the display is switched to the viewpoint image previously viewed.

Here, an example in which the recommended viewpoint image and the recommended subject image are displayed on the first sub screen 312A and the second sub screen 312B is shown, but the first recommended viewpoint image is displayed on the first sub screen 312A. It may be displayed and the second recommended viewpoint image may be displayed on the second sub screen 312B. Similarly, the first recommended subject image may be displayed on the first sub screen 312A, and the second recommended subject image may be displayed on the second sub screen 312B.

Further, the information display screens 305, 306 or 308 shown in FIG. 15 or 16 may be displayed by turning off the display of the first sub-screen 312A and the second sub-screen 312B.

Further, when the user double-clicks the main screen 311 or the like, only the main screen 311 may be displayed in full screen on the display.

The viewpoint selection screen 313 is the same as the viewpoint selection screen 302 shown in FIGS. 15 and 16.

Hereinafter, a method of distributing the video (tracking video) edited by the server 103 to the terminal device 102 will be described. First, the server 103 generates a first tracking image of the subject A and a second tracking image of the subject B. Here, for the sake of simplification of the description, an example in which two tracking images are generated will be described, but three or more tracking images may be generated. Further, the video transmitted from the server 103 is not limited to the tracking video, and may be one of the viewpoint videos, a favorite viewpoint video, a recommended viewpoint video, or the like.

The server 103 transmits a plurality of videos generated by multicast (broadcast) to the terminal device 102. Specifically, the server 103 distributes the first tracking video on 1ch and the second tracking video on 2ch. The terminal device 102 switches the displayed image by switching channels according to the user's request.

By using this method, the required band can be made constant. As a result, even when the number of users (the number of terminal devices 102) is large, it can be easily dealt with.

Although there is a problem that it is difficult to deal with a large number of users because the required bandwidth fluctuates, one video requested by the user is transmitted / received from the server 103 to the terminal device 102 or between the terminal devices. May be good. There is also an advantage that the required bandwidth can be reduced when the number of users is small.

The processing when a communication error occurs will be described below.

Ideally, the server 103 generates a recommended viewpoint image or a recommended subject image according to the evaluation result of the user. On the other hand, in the following cases, these images cannot be generated or transmitted.

First, the following problems may occur between the server 103 and the terminal device 102.

(1) When the amount of data of the transmitted video (recommended viewpoint video, recommended subject video, etc.) is larger than the receivable band of the terminal device 102. In such a case, any of the following methods can be used.

(1-1) The server 103 lowers the resolution or frame rate of the transmitted video, re-encodes it, and transmits the obtained video to the terminal device 102. In this way, the transmitted video with a reduced amount of data is used.

The server 103 may generate video signals having a plurality of bit rates by converting the resolution or frame rate of the viewpoint video transmitted from the camera 101, and store the plurality of video signals.

Further, when the bandwidth that can be used by the terminal device 102 fluctuates during viewing, the server 103 may switch the transmitted video according to the usable bandwidth.

Further, the server 103 may determine the number of images to be transmitted according to the communication band that can be used by the terminal device 102. Specifically, the server 103 increases the number of transmitted images as the communication band is wider.

(1-2) The server 103 substitutes another image close to the target image (for example, the shooting position is close, the image is close, etc.) with a small amount of data.

(2) When the transmitted video or the viewpoint video used to generate the transmitted video shows an image or area that should not be distributed. In such a case, any of the following methods can be used.

(2-1) The server 103 substitutes an image having no problem with another image close to the target image (for example, the shooting position is close, the image is close, etc.).

(2-2) The server 103 substitutes another highly evaluated image. For example, if the second highest rated video does not have the same problem, the server 103 substitutes the video.

In addition, the following problems may occur between the camera 101 (photographer) and the server 103.

(3-1) When the image from the camera 101 does not reach the server 103 due to packet loss or the like. (3-2) When the band of the camera 101 is unstable and the data transfer to the server 103 is delayed. (3-3) When the photographer suddenly interrupts shooting (battery shortage, heat problem, etc.). In such a case, any of the following methods can be used.

(4-1) The server 103 substitutes a highly evaluated viewpoint image or another image close to the subject image (for example, the shooting position is close, the image is close, etc.).

(4-2) The server 103 substitutes another highly evaluated image. For example, the server 103 substitutes for the second most highly rated video.

Hereinafter, a method of transmitting a video other than a video having a high user evaluation will be described.

Here, the unpopular viewpoint image is an image that is not seen by many people, that is, is not paid attention to. The unpopular viewpoint video is a video with a small number of playbacks, a video with a low evaluation, a video with a small number of evaluations or comments, and the like. However, such a viewpoint image may also be useful for some people, and the server 103 may connect and distribute the viewpoint images that have not received much attention. For example, the above system can be applied to a surveillance application (for example, a surveillance system including a plurality of surveillance cameras). In such a case, it is assumed that a plurality of people monitor the images of a plurality of surveillance cameras. In this case, the viewpoint video viewed by a plurality of people is sufficiently monitored, but the viewpoint video that is rarely viewed may become a blind spot. Therefore, the server 103 connects and distributes the viewpoint video (the viewpoint video in which the number of viewers is small) that has not received much attention. In addition, this image is displayed on a separate screen beside the viewpoint image that the observer is paying attention to. This makes it possible to reduce blind spots that the observer cannot monitor.

Further, when there is a viewpoint image that has not been noticed for a certain period of time, the observer may be notified or warned to that effect, or information indicating the viewpoint image that has not been noticed may be notified to the observer. Here, the viewpoint image that has not been noticed for a certain period of time is, for example, a viewpoint image that has not been reproduced for a certain period of time. When a plurality of viewpoint images are displayed at the same time, the viewpoint image that has not been noticed for a certain period of time is a viewpoint image that has not been selected as an image to be displayed on the main screen for a certain period of time, or an evaluation is performed for a certain period of time. It may be an unbroken viewpoint image. Further, the viewpoint image that has not been noticed for a certain period of time may be an image in which the number of reproductions, the number of selections, or the number of evaluations is equal to or less than a predetermined threshold value.

For example, on the viewpoint selection screen 302 or the like shown in FIG. 15, the viewpoint position of the viewpoint image that has not been noticed for a certain period of time may be highlighted. For example, the color of the viewpoint position may be changed to a color different from the color of another viewpoint position. As a result, it is possible to encourage the observer to monitor the viewpoint image that has not been noticed.

As described above, the server 103 according to the present embodiment performs the process shown in FIG.

The server 103 delivers to the terminal device 102 a first video signal based on a plurality of videos obtained by shooting the same scene from a plurality of different viewpoints in a plurality of consecutive periods.

First, the server 103 selects the first selection video from the plurality of videos for each of the plurality of periods based on the evaluation values of the plurality of videos by the plurality of users (S301). Specifically, the server 103 selects a video having a high evaluation value as the first selection video.

More specifically, as described in the recommended viewpoint video switching mode described above, the evaluation value is the evaluation value of each of the plurality of videos, and the server 103 uses the plurality of videos for each of the plurality of periods. The video having the highest evaluation value in the period or higher than the predetermined value is selected as the first selection video.

Alternatively, as described in the recommended subject switching mode described above, the evaluation value is the evaluation value of each of the plurality of subjects included in the plurality of images, and the server 103 uses the plurality of images in each of the plurality of periods. , The image showing the recommended subject having the highest evaluation value or higher than the predetermined value is selected as the first selection image. For example, the server 103 selects, as the first selection video, the video having the largest area of the recommended subject from the plurality of videos in each of the plurality of periods.

As described in the method of transmitting a video other than the video having a high user evaluation described above, the evaluation value is the evaluation value of each of the plurality of videos, and the server 103 uses the server 103 for each of the plurality of videos. Therefore, an image whose evaluation value in the period is lower than a predetermined value may be selected as the first selection image.

Next, the server 103 transmits a first video signal for continuously reproducing the plurality of first-selected videos selected for the plurality of periods to the terminal device 102 (S302). Specifically, the server 103 generates a video signal by joining a plurality of selected videos selected for a plurality of periods, and transmits the video signal to the terminal device 102. Alternatively, the server 103 transmits a plurality of videos and information for identifying the selected video selected for each of the plurality of periods to the terminal device 102 as video signals. In this case, the terminal device 102 uses the above information to connect a plurality of selected images selected for a plurality of periods to generate a video signal.

Further, as described in the process when a communication error occurs, when the data amount of the first video signal is larger than the upper limit value based on the reception band of the terminal device 102, the server 103 further evaluates a plurality of videos. A second for selecting a second selection video from a plurality of videos for each of the plurality of periods based on the value and continuously playing back the plurality of second selection videos selected for the plurality of periods. The video signal is transmitted to the terminal device 102. Here, at least one of the second-selected videos selected for each of the plurality of periods is different from the first-selected video selected for each of the plurality of periods, and the amount of data of the second video signal is large. It is smaller than the above upper limit.

Further, the plurality of images are transmitted from the plurality of cameras 101 to the server 103 in real time, and the server 103 is different from the first-selected image when the transmission of the image corresponding to the first-selected image from the camera 101 is interrupted. The second selected video is selected, and the second video signal for reproducing the second selected video is transmitted to the terminal device 102.

Further, the terminal device 102 according to the present embodiment performs the process shown in FIG.

The terminal device 102 receives from the server 103 a video signal based on a plurality of videos obtained by shooting the same scene from a plurality of different viewpoints in a plurality of continuous periods.

First, the terminal device 102 continuously reproduces a plurality of selected images selected from a plurality of images based on evaluation values of a plurality of images by a plurality of users for a plurality of periods. Received from (S311). Specifically, the selected video is a video having a high evaluation value.

More specifically, as described in the recommended viewpoint image switching mode described above, the evaluation value is the evaluation value of each of the plurality of images, and the selected image is from a plurality of images for each of the plurality of periods. It is a selected image having the highest evaluation value in the period or higher than a predetermined value.

Alternatively, as described in the recommended subject switching mode described above, the evaluation value is the evaluation value of each of the plurality of subjects included in the plurality of images, and the selected image is selected from a plurality of images in each of the plurality of periods. This is an image showing a subject having the highest evaluation value or a value higher than a predetermined value.

Alternatively, as described in the method of transmitting a video other than the video having a high user evaluation described above, the evaluation value is the evaluation value of each of the plurality of videos, and the selected video is a plurality of videos for each of the plurality of periods. This is an image selected from the above, in which the evaluation value in the period is lower than the predetermined value.

Next, the terminal device 102 continuously reproduces a plurality of selected images based on the received image signal (S312). Specifically, the terminal device 102 receives the video signal generated by joining the plurality of selected videos selected for each of the plurality of periods in the server 103, and reproduces the video signal. Alternatively, the terminal device 102 receives a plurality of images and information for identifying the selected image selected for each of the plurality of periods from the server 103, and uses the above information to set the plurality of periods. A video signal is generated by joining a plurality of selected videos selected on the other hand, and the video signal is reproduced.

Further, the terminal device 102 displays any of a plurality of images, and highlights the subject having the highest evaluation value or higher than a predetermined value among the plurality of subjects included in the displayed image. May be good.

Further, as shown in FIG. 15, the terminal device 102 displays any of a plurality of images, and when any one of the plurality of subjects included in the displayed images is selected by the user, the terminal device 102 is selected. The operation menu 307C for evaluating the subject may be displayed.

Further, the terminal device 102 displays any of a plurality of images, and when any one of the plurality of subjects included in the displayed images is selected by the user, the terminal device 102 is selected from the plurality of images. You may list the images of a plurality of periods in which the subject is included and the evaluation value is higher than a predetermined value.

Further, as shown in FIG. 17, the terminal device 102 displays any one of a plurality of images, and when any one of the plurality of subjects included in the displayed images is selected by the user, a plurality of images are displayed. It is possible to display a tracking image that tracks a selected subject, which is an image obtained by joining the images of the above.

Further, as shown in FIG. 17, the terminal device 102 displays a first screen (main screen 311) in which any of a plurality of images is displayed, and a plurality of selected images smaller than the first screen and reproduced. The output screen 300C including the second screen (first sub screen 312A or second sub screen 312B) is displayed, and when the second screen is selected by the user, a plurality of selected images are displayed on the first screen. You may.

A part of the processing performed by the server 103 may be performed by the terminal device 102, and a part of the processing performed by the terminal device 102 may be performed by the server 103. That is, the video distribution system 100 according to the present embodiment reproduces (displays) video signals based on a plurality of videos obtained by shooting the same scene from a plurality of different viewpoints in a plurality of continuous periods. do. The video distribution system 100 selects a selected video from a plurality of videos for each of the plurality of periods based on the evaluation values of the plurality of videos by the plurality of users. The video distribution system 100 generates a video signal by connecting a plurality of selected videos selected for a plurality of periods, and reproduces (displays) the video signal.

Although the video distribution method, the video reception method, and the video distribution system according to the embodiment have been described above, the present invention is not limited to this embodiment.

Further, each processing unit included in each device included in the video distribution system according to the above embodiment is typically realized as an LSI which is an integrated circuit. These may be individually integrated into one chip, or may be integrated into one chip so as to include a part or all of them.

Further, the integrated circuit is not limited to the LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connection and settings of circuit cells inside the LSI may be used.

In each of the above embodiments, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

In other words, each device included in the video distribution system includes a processing circuit and a storage device electrically connected to the processing circuit (accessible from the processing circuit). The processing circuit includes at least one of dedicated hardware and a program execution unit. Further, when the processing circuit includes a program execution unit, the storage device stores the software program executed by the program execution unit. The processing circuit uses the storage device to execute the video distribution method or the wave video reception method according to the above embodiment.

Further, the present invention may be the software program or a non-temporary computer-readable recording medium on which the program is recorded. Needless to say, the above program can be distributed via a transmission medium such as the Internet.

In addition, the numbers used above are all exemplified for the purpose of specifically explaining the present invention, and the present invention is not limited to the illustrated numbers.

Further, the order in which the steps included in the above-mentioned video distribution method or video reception method are executed is for exemplifying for the purpose of specifically explaining the present invention, and may be an order other than the above. Further, a part of the above steps may be executed at the same time (parallel) as other steps.

The video distribution method, the video reception method, the video distribution system, the server, and the terminal device according to one or more aspects of the present invention have been described above based on the embodiments. It is not limited. As long as it does not deviate from the gist of the present invention, one or more of the present embodiments may be modified by those skilled in the art, or may be constructed by combining components in different embodiments. It may be included within the scope of the embodiment.

(Embodiment 2)
Other application examples of the image processing method and the configuration of the apparatus shown in each of the above embodiments and a system using the same will be described. The system can be applied to video systems that are becoming more intelligent and the target space is becoming wider. For example, (1) surveillance systems implemented in store or factory security cameras, police in-vehicle cameras, etc., (2) individuals. Traffic information systems using owned cameras or in-vehicle cameras, or cameras installed on the road, (3) environmental survey or delivery systems using remote control or automatically controllable devices such as drones, and (4) entertainment It can be applied to a content transmission / reception system such as an installed camera in a facility or a stadium, a mobile camera such as a drone, or a video using a personally owned camera.

FIG. 20 is a diagram showing a configuration of the video information processing system ex100 according to the present embodiment. In the present embodiment, an example of preventing the occurrence of a blind spot and an example of prohibiting photography in a specific area will be described.

The video information processing system ex100 shown in FIG. 20 includes a video information processing device ex101, a plurality of cameras ex102, and a video receiving device ex103. The video receiving device ex103 does not necessarily have to be included in the video information processing system ex100.

The video information processing device ex101 includes a storage unit ex111 and an analysis unit ex112. Each of the N cameras ex102 has a function of capturing a video and a function of transmitting the captured video data to the video information processing device ex101. Further, the camera ex102 may have a function of displaying an image being photographed. The camera ex102 uses HEVC or H.I. It may be encoded using an encoding method such as 264 and then transmitted to the video information processing apparatus ex101, or unencoded video data may be transmitted to the video information processing apparatus ex101.

Here, each camera ex102 is a fixed camera such as a surveillance camera, a mobile camera mounted on an unmanned radio-controlled model, a car, or the like, or a user camera owned by the user.

The mobile camera receives an instruction signal transmitted from the video information processing apparatus ex101, and changes the position or shooting direction of the mobile camera itself according to the received instruction signal.

Further, the times of the plurality of cameras ex102 are calibrated using the time information of the server or the reference camera before the photographing disclosure. Further, the spatial positions of the plurality of cameras ex102 are calibrated based on the appearance of the object in the space to be photographed or the relative position from the reference camera.

The storage unit ex111 included in the information processing device ex101 stores the video data transmitted from the N cameras ex102.

The analysis unit ex112 detects a blind spot from the video data stored in the storage unit ex111, and transmits an instruction signal indicating an instruction to the moving camera to prevent the occurrence of the blind spot to the moving camera. The moving camera moves according to the instruction signal and continues shooting.

The analysis unit ex112 detects the blind spot using, for example, SfM (Structure from Motion). SfM is a method of restoring a three-dimensional shape of a subject from a plurality of images shot from different positions, and is widely known as a shape restoration technique for simultaneously estimating the subject shape and the camera position. For example, the analysis unit ex112 uses SfM to restore the three-dimensional shape in the facility or the stadium from the video data stored in the storage unit ex111, and detects a region that cannot be restored as a blind spot.

If the position and shooting direction of the camera ex102 are fixed and the information on the position and shooting direction is known, the analysis unit ex112 may perform SfM using these known information. If the position and shooting direction of the moving camera can be acquired by the GPS and the angle sensor provided in the moving camera, the moving camera transmits the information on the position and shooting direction of the moving camera to the analysis unit ex112, and the analysis unit. The ex112 may perform SfM using the transmitted position and shooting direction information.

The blind spot detection method is not limited to the method using SfM described above. For example, the analysis unit ex112 may grasp the spatial distance of the object to be imaged by using the information of the depth sensor such as the laser range finder. Further, the analysis unit ex112 detects information such as the camera position, shooting direction, zoom magnification, etc. from the image includes a preset marker or a specific object in the space, or if it is included, its size or the like. May be good. In this way, the analysis unit ex112 detects the blind spot by using an arbitrary method capable of detecting the photographing area of each camera. Further, the analysis unit ex112 acquires information such as the positional relationship between a plurality of shooting targets from video data or a proximity distance sensor, and identifies a region where a blind spot is likely to occur based on the acquired positional relationship. You may.

Here, the blind spot includes not only a portion where the image does not exist in the area to be photographed, but also a portion where the image quality is poor as compared with other portions, and a portion where a predetermined image quality is not obtained. The part to be detected may be appropriately set according to the configuration or purpose of the system. For example, the required image quality may be set high for a specific subject in the space to be photographed. On the contrary, the required image quality may be set low for a specific area in the shooting space, or may be set so as not to be determined as a blind spot even if the image is not shot.

The above-mentioned image quality includes various information related to the image such as the area occupied by the subject to be photographed in the image (for example, the number of pixels) or whether the subject to be photographed is in focus. It suffices to determine whether or not it is a blind spot based on the information of the above or a combination thereof.

In the above description, the detection of the area that is actually the blind spot has been described, but the area that needs to be detected in order to prevent the occurrence of the blind spot is not limited to the area that is actually the blind spot. For example, when there are a plurality of shooting targets and at least a part of them is moving, a new blind spot may be created by inserting another shooting target between one shooting target and the camera. On the other hand, the analysis unit ex112 detects the movements of a plurality of shooting targets from, for example, the shot video data, and based on the detected movements of the plurality of shooting targets and the position information of the camera ex102, a new blind spot is created. You may estimate the potential region. In this case, the video information processing apparatus ex101 may transmit an instruction signal to the moving camera so as to capture an area that may become a blind spot to prevent the occurrence of the blind spot.

When there are a plurality of mobile cameras, the video information processing device ex101 needs to select a mobile camera that transmits an instruction signal in order to capture a blind spot or a region that may be a blind spot. Further, when there are a plurality of mobile cameras and a plurality of areas that may become blind spots or blind spots, the video information processing apparatus ex101 has a blind spot or a potential area that may become a blind spot for each of the plurality of mobile cameras. It is necessary to decide whether to shoot. For example, the video information processing device ex101 selects the moving camera closest to the blind spot or the area that becomes the blind spot based on the area that may become the blind spot or the blind spot and the position of the area that each moving camera is shooting. do. Further, the video information processing device ex101 determines for each mobile camera whether or not a new blind spot occurs when the video data currently being shot by the mobile camera cannot be obtained, and the video data currently being shot is obtained. You may select a moving camera which is judged not to generate a blind spot even if it is not obtained.

With the above configuration, the video information processing apparatus ex101 can prevent the occurrence of the blind spot by detecting the blind spot and transmitting an instruction signal to the moving camera so as to prevent the blind spot.

(Modification example 1)
In the above description, an example in which an instruction signal for instructing movement is transmitted to the moving camera has been described, but the instruction signal may be a signal for instructing the user of the user camera to move. For example, the user camera displays an instruction image instructing the user to change the direction of the camera based on the instruction signal. The user camera may display an instruction image showing the movement route on the map as an instruction for the movement of the user. In addition, the user camera may display detailed shooting instructions such as the shooting direction, angle, angle of view, image quality, and movement of the shooting area in order to improve the quality of the acquired image, and further, a video information processing device. If the control is possible on the ex101 side, the video information processing device ex101 may automatically control the feature amount of the camera ex102 related to such shooting.

Here, the user camera is, for example, a smartphone, a tablet terminal, a wearable terminal, or an HMD (Head Mounted Display) owned by a spectator in the stadium or a security guard in the facility.

Further, the display terminal that displays the instruction image does not have to be the same as the user camera that captures the video data. For example, the user camera may transmit an instruction signal or an instruction image to a display terminal associated with the user camera in advance, and the display terminal may display the instruction image. Further, the information of the display terminal corresponding to the user camera may be registered in the video information processing apparatus ex101 in advance. In this case, the video information processing device ex101 may display the instruction image on the display terminal by directly transmitting the instruction signal to the display terminal corresponding to the user camera.

(Modification 2)
Even if the analysis unit ex112 generates a free viewpoint image (three-dimensional reconstruction data) by restoring the three-dimensional shape in the facility or the stadium from the image data stored in the storage unit ex111 using, for example, SfM. good. This free viewpoint image is stored in the storage unit ex111. The video information processing device ex101 reads out video data corresponding to the visual field information (and / or viewpoint information) transmitted from the video receiving device ex103 from the storage unit ex111 and transmits the video data to the video receiving device ex103. The video receiving device ex103 may be one of a plurality of cameras 111.

(Modification example 3)
The video information processing device ex101 may detect a shooting prohibited area. In this case, the analysis unit ex112 analyzes the captured image, and if the moving camera is photographing the shooting prohibited area, transmits a shooting prohibition signal to the moving camera. The mobile camera stops shooting while receiving the shooting prohibition signal.

The analysis unit ex112 determines, for example, whether the moving camera preset in the space is shooting the shooting prohibited area by matching the three-dimensional virtual space restored using SfM with the shot image. judge. Alternatively, the analysis unit ex112 determines whether the moving camera is shooting the shooting prohibited area by using a marker or a characteristic object arranged in the space as a trigger. The shooting prohibited area is, for example, a toilet in a facility or a stadium.

In addition, when the user camera is shooting a shooting prohibited area, the user camera may display a message on a display or the like connected wirelessly or by wire, or output sound or voice from a speaker or earphone. Then, the user may be notified that the current location is a shooting prohibited location.

For example, as the above message, it is displayed that shooting is prohibited in the direction in which the camera is currently pointed. Alternatively, the prohibited area for shooting and the current shooting area are shown on the displayed map. Further, the resumption of shooting is automatically performed, for example, when the shooting prohibition signal is no longer output. Alternatively, shooting may be resumed when the shooting prohibition signal is not output and the user performs an operation to restart shooting. Further, if the shooting is stopped and restarted a plurality of times in a short period of time, the calibration may be performed again. Alternatively, a notification may be given to the user to confirm the current position or prompt the user to move.

Further, in the case of special work such as police, a passcode or fingerprint authentication that turns off such a function for recording may be used. Further, even in such a case, if the image in the shooting prohibited area is displayed or saved to the outside, image processing such as mosaic may be automatically performed.

With the above configuration, the video information processing apparatus ex101 can set a certain area to prohibit shooting by determining that shooting is prohibited and notifying the user to stop shooting.

(Modification example 4)
Since it is necessary to collect images from a plurality of viewpoints in order to construct a three-dimensional virtual space from the images, the image information processing system ex100 sets an incentive for the user who has transferred the captured images. For example, the video information processing device ex101 provides video distribution to the user who has transferred the video at a free or discounted rate, or has a monetary value that can be used in an online or offline store or in-game, or a game or the like. Give points of non-monetary value such as social status in virtual space. Further, the video information processing device ex101 gives a particularly high point to a user who has transferred a shot video of a valuable field of view (and / or viewpoint) such as a large number of requests.

(Modification 5)
The video information processing device ex101 may transmit additional information to the user camera based on the analysis result of the analysis unit ex112. In this case, the user camera superimposes additional information on the captured image and displays it on the screen. The additional information is, for example, player information such as a player's name or height when a match at a stadium is filmed, and the player's name or face photo is associated with each player in the video. Is displayed. The video information processing device ex101 may extract additional information by searching via the Internet based on a part or all of the video data area. Further, the camera ex102 receives such additional information by short-range wireless communication such as Bluetooth (registered trademark) or visible light communication from lighting of a stadium or the like, and maps the received additional information to video data. May be good. Further, the camera ex102 is a table in which this mapping is held in a storage unit connected to the camera ex102 by wire or wirelessly, and is a table showing the correspondence between the information obtained by the visible light communication technology and the additional information. It may be performed based on a certain rule of, or it may be performed using the result of the most probable combination by searching the Internet.

Further, in the monitoring system, the accuracy of the monitoring system can be improved by superimposing, for example, information on the person to be watched on the user camera held by the guard in the facility.

(Modification 5)
The analysis unit ex112 may determine which area in the facility or the stadium the user camera is shooting by matching the free-viewpoint image with the image captured by the user camera. The method for determining the imaging region is not limited to this, and various imaging region determination methods or other imaging region determination methods described in the above-described embodiments may be used.

The video information processing device ex101 transmits the past video to the user camera based on the analysis result of the analysis unit ex112. The user camera superimposes the past image on the captured image or replaces the captured image with the past image and displays it on the screen.

For example, during half-time, the highlight scene of the first half is displayed as a past image. As a result, the user can enjoy the highlight scene of the first half as an image in the direction in which he / she is looking during the half-time. The past video is not limited to the highlight scene of the first half, but may be the highlight scene of the past game held at the stadium. Further, the timing at which the video information processing device ex101 delivers the past video is not limited to half-time, and may be, for example, after the end of the match or during the match. In particular, during a match, the video information processing device ex101 may deliver a scene considered to be important that the user has missed, based on the analysis result of the analysis unit ex112. Further, the video information processing device ex101 may deliver the past video only when there is a request from the user, or may deliver a delivery permission message before the past video is delivered.

(Modification 6)
The video information processing device ex101 may transmit advertisement information to the user camera based on the analysis result of the analysis unit ex112. The user camera superimposes the advertisement information on the captured image and displays it on the screen.

The advertisement information may be distributed, for example, during the half-time or immediately before the past video distribution after the end of the game, which is shown in the modified example 5. As a result, the distributor can obtain the advertisement fee from the advertiser, and can provide the user with the video distribution service at low cost or free of charge. Further, the video information processing device ex101 may deliver a message of permission to deliver the advertisement immediately before the delivery of the advertisement information, may provide a service free of charge only when the user views the advertisement, or may view the advertisement. The service may be provided at a lower cost than if not.

Also, when the user clicks "Order now" according to the advertisement, the staff who knows the user's position based on the system or some location information or the automatic delivery system at the venue will take the ordered drink to the seat. Will deliver it. The approval may be handed over to the staff, or may be made based on the credit card information set in advance in the application of the mobile terminal or the like. In addition, the advertisement may include a link to an e-commerce site and be ready for online shopping such as normal home delivery.

(Modification 7)
The video receiving device ex103 may be one of the cameras ex102 (user camera). In this case, the analysis unit ex112 determines which area in the facility or the stadium is being photographed by matching the free-viewpoint image with the image captured by the user camera. The method for determining the shooting area is not limited to this.

For example, when the user swipes in the direction of the arrow displayed on the screen, the user camera generates viewpoint information indicating that the viewpoint is moved in that direction. The video information processing device ex101 reads out the video data obtained by shooting the area moved by the viewpoint information from the shooting area of the user camera determined by the analysis unit ex112 from the storage unit ex111, and transmits the video data to the user camera. Start. Then, the user camera displays the video delivered from the video information processing device ex101 instead of the captured video.

As described above, the user in the facility or the stadium can watch the video from a favorite viewpoint by a simple operation such as swiping the screen. For example, a spectator watching a game on the 3rd base side of a baseball stadium can watch a video from the viewpoint of the 1st base side. In addition, in the monitoring system, the guards in the facility use a simple operation such as a screen swipe to watch the viewpoint that they want to check or the video that should be watched as an interrupt from the center while changing the viewpoint. Therefore, it is possible to improve the accuracy of the monitoring system.

Further, the distribution of the video to the user in the facility or the stadium is also effective when, for example, there is an obstacle between the user camera and the shooting target and there is an invisible area. In this case, the user camera may switch the video of a part of the shooting region of the user camera including the obstacle from the shot video to the distributed video from the video information processing device ex101 and display the video. The entire screen may be displayed by switching from the captured image to the distributed image. Further, the user camera may combine the captured image and the distributed image to display an image in which the viewing target can be seen through the obstacle. According to this configuration, even when the shooting target cannot be seen from the user's position due to the influence of obstacles, the video delivered from the video information processing device ex101 can be viewed, so that the influence of obstacles can be reduced. can.

Further, when displaying the distributed image as an image in an area invisible due to an obstacle, a display switching control different from the display switching control according to the input process by the user such as the screen swipe described above may be performed. .. For example, when it is determined that an obstacle is included in the shooting area based on the information on the movement and shooting direction of the user camera and the position information of the obstacle obtained in advance, the display from the shot video to the distributed video is displayed. Switching may be done automatically. Further, when it is determined by the analysis of the captured video data that an obstacle that is not the subject of shooting is displayed, the display may be automatically switched from the captured video to the distributed video. Further, when the area of the obstacle (for example, the number of pixels) included in the captured image exceeds a predetermined threshold value, or when the ratio of the area of the obstacle to the area of the object to be photographed exceeds a predetermined ratio, the captured image is used. The display may be automatically switched to the distribution video.

It should be noted that the display from the captured video to the distributed video may be switched and the display from the distributed video to the captured video may be switched according to the input process of the user.

(Modification 8)
The speed at which the video data is transferred to the video information processing device ex101 may be instructed based on the importance of the video data captured by each camera ex102.

In this case, the analysis unit ex112 determines the importance of the video data stored in the storage unit ex111 or the camera ex102 that captured the video data. The determination of importance here is performed based on, for example, information such as the number of people or moving objects included in the video, the image quality of the video data, or a combination thereof.

Further, the determination of the importance of the video data may be based on the position of the camera ex102 in which the video data is captured or the area in which the video data is captured. For example, when there are a plurality of other cameras ex102 being photographed near the target camera ex102, the importance of the video data captured by the target camera ex102 is reduced. Further, when there are a plurality of other cameras ex102 that are shooting the same area even if the position of the target camera ex102 is far from the other camera ex102, the importance of the video data taken by the target camera ex102 is determined. make low. Further, the determination of the importance of the video data may be performed based on the number of requests in the video distribution service. The method for determining the importance is not limited to the above-mentioned ones or a combination thereof, and may be any method according to the configuration or purpose of the monitoring system or the video distribution system.

Further, the determination of importance does not have to be based on the captured video data. For example, the importance of the camera ex102 that transmits video data to a terminal other than the video information processing device ex101 may be set high. On the contrary, the importance of the camera ex102 that transmits the video data to the terminal other than the video information processing device ex101 may be set low. As a result, for example, when a plurality of services that require transmission of video data share a communication band, the degree of freedom in controlling the communication band according to the purpose or characteristic of each service is increased. This makes it possible to prevent deterioration in the quality of each service due to the inability to obtain the required video data.

Further, the analysis unit ex112 may determine the importance of the video data by using the free viewpoint video and the video captured by the camera ex102.

The video information processing apparatus ex101 transmits a communication speed instruction signal to the camera ex102 based on the result of determining the importance performed by the analysis unit ex112. The video information processing device ex101, for example, instructs the camera ex102, which is capturing a video of high importance, with a high communication speed. Further, the video information processing apparatus ex101 may not only control the speed but also transmit a signal instructing a method of sending important information a plurality of times in order to reduce the demerit due to omission. As a result, communication within the facility or the entire stadium can be efficiently performed. The communication between the camera ex102 and the video information processing device ex101 may be wired communication or wireless communication. Further, the video information processing device ex101 may control only one of the wired communication and the wireless communication.

The camera ex102 transmits the captured video data to the video information processing device ex101 at a communication speed according to the communication speed instruction signal. If the camera ex102 fails to retransmit a predetermined number of times, the camera ex102 may stop retransmitting the captured video data and start transferring the next captured video data. As a result, communication within the facility or the entire stadium can be efficiently performed, and the processing speed of the analysis unit ex112 can be increased.

Further, when the communication speed assigned to each of the cameras ex102 is not a sufficient band for transferring the captured video data, the camera ex102 can transmit the captured video data at the assigned communication speed. The converted video data may be transmitted, or the transfer of the video data may be stopped.

Further, when the video data is used to prevent the occurrence of the blind spot as described above, it is possible that only a part of the shooting area included in the shot video data is required to fill the blind spot. There is sex. In this case, the camera ex102 generates extracted video data by extracting at least an area required for preventing the occurrence of blind spots from the video data, and the generated extracted video data is used as a video information processing device. It may be transmitted to ex101. According to this configuration, it is possible to suppress the occurrence of blind spots with a smaller communication band.

Further, for example, when the additional information is superimposed and displayed or the video is distributed, the camera ex102 needs to transmit the position information and the shooting direction information of the camera ex102 to the video information processing device ex101. In this case, the camera ex102, which has been allocated only a band insufficient for transferring the video data, may transmit only the position information and the shooting direction information detected by the camera ex102. Further, when the video information processing apparatus ex101 estimates the position information and the shooting direction information of the camera ex102, the camera ex102 converts the shot video data into the resolution required for estimating the position information and the shooting direction information. , The converted video data may be transmitted to the video information processing device ex101. According to this configuration, it is possible to provide a service of superimposing display of additional information or video distribution even for the camera ex102 to which a small communication band is allocated. Further, since the video information processing device ex101 can acquire information on the shooting area from more cameras ex102, even when the information on the shooting area is used for the purpose of detecting the area of interest, for example. It is valid.

The camera ex102 may switch the video data transfer process according to the allocated communication band described above based on the notified communication band, or the video information processing device ex101 operates each camera ex102. Each camera ex102 may be notified of a control signal that has been determined and indicates the determined operation. As a result, the processing can be appropriately divided according to the amount of calculation required for determining the operation switching, the processing capacity of the camera ex102, the required communication band, and the like.

(Modification 9)
The analysis unit ex112 may determine the importance of the video data based on the visual field information (and / or the viewpoint information) transmitted from the video receiving device ex103. For example, the analysis unit ex112 sets the importance of the captured video data including many regions indicated by the visual field information (and / or the viewpoint information) to be high. Further, the analysis unit ex112 may determine the importance of the video data in consideration of the number of people included in the video or the number of moving objects. The method of determining the importance is not limited to this.

The communication control method described in this embodiment does not necessarily have to be used in a system that reconstructs a three-dimensional shape from a plurality of video data. For example, in an environment where a plurality of cameras ex102 exist, when video data is selectively transmitted or transmitted by wired communication and / or wireless communication with different transmission speeds, the communication control method described in this embodiment is used. It is valid.

(Modification example 10)
In the video distribution system, the video information processing device ex101 may transmit an overview video showing the entire shooting scene to the video receiving device ex103.

Specifically, when the video information processing device ex101 receives the distribution request transmitted from the video receiving device ex103, the video information processing device ex101 reads out the overview video of the entire facility or the stadium from the storage unit ex111, and outputs the appearance video to the video receiving device. Send to ex103. This overview video may have a long update interval (a low frame rate may be used) and may have a low image quality. The viewer touches a part of the overview video displayed on the screen of the video receiving device ex103 that he / she wants to see. As a result, the video receiving device ex103 transmits the visual field information (and / or the viewpoint information) corresponding to the touched portion to the video information processing device ex101.

The video information processing device ex101 reads the video data corresponding to the visual field information (and / or the viewpoint information) from the storage unit ex111, and transmits the video data to the video receiving device ex103.

Further, the analysis unit ex112 generates a free viewpoint image by preferentially restoring the three-dimensional shape (three-dimensional reconstruction) of the region indicated by the visual field information (and / or the viewpoint information). The analysis unit ex112 restores the three-dimensional shape of the entire facility or stadium with an accuracy that gives an overview. As a result, the video information processing apparatus ex101 can efficiently restore the three-dimensional shape. As a result, it is possible to realize a high frame rate and high image quality of the free-viewpoint video in the area that the viewer wants to see.

(Modification 11)
The video information processing device ex101 may store, for example, three-dimensional shape restoration data of a facility or a stadium generated in advance from a design drawing or the like as a preliminary video. The pre-video is not limited to this, and may be virtual space data in which the unevenness of the space obtained from the depth sensor and the picture derived from the past or calibration image or video data are mapped for each object.

For example, when soccer is played in a stadium, the analysis unit ex112 restores the three-dimensional shape only to the players and the ball, and combines the obtained restoration data with the preliminary image to create a free-viewpoint image. May be generated. Alternatively, the analysis unit ex112 may preferentially restore the three-dimensional shape to the player and the ball. As a result, the video information processing apparatus ex101 can efficiently restore the three-dimensional shape. As a result, it is possible to realize a high frame rate and high image quality of the free-viewpoint video regarding the player and the ball that the viewer pays attention to. Further, in the monitoring system, the analysis unit ex112 may restore the three-dimensional shape by limiting it to only a person and a moving object, or giving priority to them.

(Modification 12)
The time of each device may be calibrated at the start of shooting based on the reference time of the server or the like. The analysis unit ex112 uses a plurality of video data shot at a time belonging to a preset time range according to the accuracy of the time setting among the plurality of shot video data shot by the plurality of cameras ex102. , Restore the three-dimensional shape. For the detection of this time, for example, the time when the captured video data is stored in the storage unit ex111 is used. The time detection method is not limited to this. As a result, the video information processing apparatus ex101 can efficiently restore the three-dimensional shape, so that it is possible to realize a high frame rate and high image quality of the free-viewpoint video.

Alternatively, the analysis unit ex112 may restore the three-dimensional shape by using only the high-quality data or preferentially using the high-quality data among the plurality of video data stored in the storage unit ex111. ..

(Modification 13)
The analysis unit ex112 may restore the three-dimensional shape by using the camera attribute information. In this case, the camera ex102 transmits the captured video data and the camera attribute information to the video information processing device ex101. The camera attribute information is, for example, a shooting position, a shooting angle, a shooting time, a zoom magnification, and the like.

As a result, the video information processing apparatus ex101 can efficiently restore the three-dimensional shape, so that it is possible to realize a high frame rate and high image quality of the free-viewpoint video.

Specifically, the camera ex102 defines three-dimensional coordinates in the facility or the stadium, and the camera ex102 captures information about which coordinates are taken from which angle, at what zoom, and at what time together with the video. It is transmitted to the video information processing device ex101 as camera attribute information. Further, when the camera ex102 is activated, the clock on the communication network in the facility or the stadium is synchronized with the clock in the camera, and time information is generated.

Further, the position and angle information of the camera ex102 is acquired by pointing the camera ex102 at a specific point in the facility or the stadium when the camera ex102 is activated or at an arbitrary timing. FIG. 21 is a diagram showing an example of a notification displayed on the screen of the camera ex102 when the camera ex102 is activated. According to this notification, when the user touches the display of the camera ex102 by aligning the "+" displayed in the center of the screen with the "+" in the center of the soccer ball in the advertisement on the north side of the stadium, the camera ex102 is released from the camera ex102. Acquire vector information up to the advertisement and specify the reference of camera position and angle. After that, the camera coordinates and angles at that time are specified from the motion information of the camera ex102. Of course, the display is not limited to this, and a display that indicates the coordinates, the angle, the moving speed of the shooting area, or the like may be used even during the shooting period by using an arrow or the like.

The coordinates of the camera ex102 may be specified using radio waves of GPS, WiFi (registered trademark), 3G, LTE (Long Term Evolution), and 5G (wireless LAN), or a beacon (Bluetooth (registered trademark)). , Ultrasonic wave) may be used. In addition, information on which base station in the facility or stadium the captured video data has arrived may be used.

(Modification 14)
The system may be provided as an application that operates on a mobile terminal such as a smartphone.

An account such as various SNS may be used for logging in to the above system. An account dedicated to the application or a guest account with limited functions may be used. By using the account in this way, it is possible to evaluate a favorite video, a favorite account, or the like. Further, by preferentially allocating a band to video data similar to the video data during shooting or viewing, video data from a viewpoint similar to the viewpoint of the video data being shot or viewed, the resolution of these video data. Can be enhanced. This makes it possible to more accurately restore the three-dimensional shape from these viewpoints.

In addition, the user can select a favorite image or video in the application and follow the other party to view the selected image with priority over other users, or by text chat etc. on condition that the other user approves. You can have a connection. In this way, it is possible to create a new community.

By connecting the users in the community in this way, the shooting itself and the sharing of the shot image become active, and it is possible to promote the restoration of the three-dimensional shape with higher accuracy.

In addition, depending on the setting of the connection within the community, the user can edit the image or video taken by another person, or collage the image of another person with his or her own image to create a new image or video. This makes it possible to share new video works, such as sharing new images or videos only with people in the community. Further, by inserting a CG character in this editing, the video work can be used for an augmented reality game or the like.

In addition, according to the system, three-dimensional model data can be sequentially output, so that a 3D printer or the like owned by the facility can output a three-dimensional object based on the three-dimensional model data in a characteristic scene such as a goal scene. .. As a result, after the match, objects based on the scene during the match can be sold as souvenirs such as key chains or distributed to participating users. Of course, it is also possible to print an image from the best viewpoint as a normal photograph.

(Modification 15)
Using the above system, for example, the rough state of the entire area can be managed at the center connected to the system from the images of the in-vehicle camera of the police and the wearable camera of the police officer.

During a general patrol, for example, still images are transmitted and received every few minutes. In addition, the center identifies areas with a high probability of crime based on a crime map based on the results of analysis using past crime data, etc., or areas related to the crime occurrence probability identified in this way. Holds data. In the identified areas with a high probability of crime, images may be sent and received more frequently, or images may be changed to moving images. Further, when an incident occurs, moving images or three-dimensional reconstruction data using SfM or the like may be used. In addition, the center or each terminal simultaneously corrects the image or virtual space by using the information of other sensors such as the depth sensor or the thermo sensor, so that the police officer can grasp the situation more accurately.

In addition, the center can feed back the information of the object to a plurality of terminals by using the three-dimensional reconstruction data. This allows an individual with each terminal to track the object.

Recently, for the purpose of investigating buildings or the environment, or taking pictures with a sense of reality such as sports, shooting from the air with a flightable device such as a quadcopter or a drone has been performed. Shooting with such an autonomous moving device tends to cause a problem that the image is blurred, but SfM can perform three-dimensionalization while correcting the blur depending on the position and inclination. As a result, it is possible to improve the image quality and the restoration accuracy of the space.

In some countries, it is mandatory to install an in-vehicle camera that takes pictures of the outside of the vehicle. Even in such an in-vehicle camera, by using three-dimensional data modeled from a plurality of images, it is possible to more accurately grasp the weather in the direction of the destination, the condition of the road surface, the degree of traffic congestion, and the like.

(Embodiment 3)
By recording a program for realizing the configuration of the image processing method shown in each of the above embodiments on a storage medium, the processing shown in each of the above embodiments can be easily carried out in an independent computer system. It becomes. The storage medium may be a magnetic disk, an optical disk, a magneto-optical disk, an IC card, a semiconductor memory, or the like, as long as it can record a program.

Further, here, an application example of the image processing method shown in each of the above embodiments and a system using the same will be described. The system is characterized by having an apparatus using an image processing method. Other configurations in the system can be changed appropriately as the case may be.

FIG. 22 is a diagram showing an overall configuration of a content supply system ex200 that realizes a content distribution service. The communication service providing area is divided into desired sizes, and base stations ex206, ex207, ex208, ex209, and ex210, which are fixed radio stations, are installed in each cell.

This content supply system ex200 includes a computer ex211, a PDA (Personal Digital Assistant) ex212, a camera ex213, a smartphone ex214, a game machine ex215, etc. via the Internet service provider ex202 and the communication network ex204, and the base station ex206 to ex210 on the Internet ex201. Each device is connected.

However, the content supply system ex200 is not limited to the configuration as shown in FIG. 22, and any element may be combined and connected. Further, each device may be directly connected to a communication network ex204 such as a telephone line, cable television, or optical communication without going through a fixed radio station base station ex206 to ex210. Further, the devices may be directly connected to each other via short-range radio or the like.

The camera ex213 is a device capable of shooting moving images such as a digital video camera, and the camera ex216 is a device capable of shooting still images and shooting moving images such as a digital camera. In addition, the smartphone ex214 is a GSM (registered trademark) (Global System for Mobile Communications) method, a CDMA (Code Division Multiple Access) method, a W-CDMA (Wideband-Code Division) method, a W-CDMA (Wideband-Code Division) method , HSPA (High Speed Packet Access), a smartphone device compatible with a communication method using a high frequency band, PHS (Personal Handyphone System), or the like.

In the content supply system ex200, the camera ex213 and the like are connected to the streaming server ex203 through the base station ex209 and the communication network ex204, so that live distribution and the like become possible. In live distribution, content (for example, live music video) captured by the user using the camera ex213 is encoded and transmitted to the streaming server ex203. On the other hand, the streaming server ex203 streams the content data transmitted to the requested client. Clients include a computer ex211, a PDAex212, a camera ex213, a smartphone ex214, a game machine ex215, and the like, which can decode the encoded data. Each device that receives the distributed data decodes the received data and reproduces it.

The captured data may be encoded by the camera ex213, by the streaming server ex203 that performs the data transmission processing, or may be shared with each other. Similarly, the decryption process of the delivered data may be performed by the client, the streaming server ex203, or shared with each other. Further, not limited to the camera ex213, the still image and / or the moving image data taken by the camera ex216 may be transmitted to the streaming server ex203 via the computer ex211. The coding process in this case may be performed by any of the camera ex216, the computer ex211 and the streaming server ex203, or may be shared with each other. Further, regarding the display of the decoded image, a plurality of devices connected to the system may work together to display the same image, or the entire image may be displayed by a device having a large display unit, and the image may be displayed on a smartphone ex214 or the like. A part of the area may be enlarged and displayed.

Further, these coding / decoding processes are generally performed by the computer ex211 or the LSI ex500 of each device. The LSIex500 may be one-chip or may be composed of a plurality of chips. The moving image coding / decoding software is incorporated into some recording medium (CD-ROM, flexible disk, hard disk, etc.) that can be read by a computer ex211 or the like, and the coding / decoding processing is performed using the software. You may. Further, when the smartphone ex214 is equipped with a camera, the moving image data acquired by the camera may be transmitted. The moving image data at this time is the data encoded by the LSI ex500 possessed by the smartphone ex214.

Further, the streaming server ex203 may be a plurality of servers or a plurality of computers, and may disperse data for processing, recording, and distribution.

As described above, in the content supply system ex200, the client can receive and reproduce the encoded data. In this way, in the content supply system ex200, the client can receive, decode, and reproduce the information transmitted by the user in real time, and even a user who does not have special rights or equipment can realize personal broadcasting.

Not limited to the example of the content supply system ex200, as shown in FIG. 23, each of the above embodiments may be applied to the digital broadcasting system ex300. Specifically, at the broadcasting station ex301, the multiplexed data in which music data or the like is multiplexed with the video data is transmitted via radio waves to the communication or the satellite ex302. This video data is data encoded by the moving image coding method described in each of the above embodiments. In response to this, the broadcasting satellite ex302 transmits radio waves for broadcasting, and the radio waves are received by the home antenna ex304 capable of receiving satellite broadcasting. A device such as a television (receiver) ex400 or a set-top box (STB) ex317 decodes and reproduces the received multiplexed data.

Further, the multiplexed data recorded in the recording medium ex315 such as DVD or BD or the memory ex316 such as SD is read and decoded, or the video signal is encoded in the recording medium ex315 or the memory ex316, and in some cases, the music signal is used. It is possible to mount the moving image decoding device or the moving image coding device shown in each of the above embodiments on the reader / recorder ex318 for multiplexing and writing. In this case, the reproduced video signal is displayed on the monitor ex319, and the video signal can be reproduced in another device or system by the recording medium ex315 or the memory ex316 in which the multiplexed data is recorded. Further, a moving image decoding device may be mounted in a set-top box ex317 connected to a cable ex303 for cable television or an antenna ex304 for satellite / terrestrial broadcasting, and this may be displayed on a television monitor ex319. At this time, the moving image decoding device may be incorporated in the television instead of the set-top box.

FIG. 24 is a diagram showing a smartphone ex214. Further, FIG. 25 is a diagram showing a configuration example of the smartphone ex214. The smartphone ex214 has an antenna ex450 for transmitting and receiving radio waves to and from the base station ex210, a camera unit ex465 capable of taking images and still images, an image captured by the camera unit ex465, an image received by the antenna ex450, and the like. A display unit ex458 such as a liquid crystal display for displaying the decoded data is provided. The smartphone ex214 further includes an operation unit ex466 such as a touch panel, an audio output unit ex457 such as a speaker for outputting audio, an audio input unit ex456 such as a microphone for inputting audio, a captured video, and a still image. , Recorded audio, or memory unit ex467 that can store encoded or decoded data such as received video, still image, mail, etc., or memory ex316 illustrated in FIG. 23, or a user is specified. The slot unit ex464, which is an interface unit with SIMex468 for authenticating access to various data including the network, is provided.

In the smartphone ex214, the power supply circuit unit ex461, the operation input control unit ex462, the video signal processing unit ex455, the camera interface unit ex463, and the LCD ( Liquid Crystal Display) control unit ex459, modulation / demodulation unit ex452, multiplexing / separation unit ex453, audio signal processing unit ex454, slot unit ex464, and memory unit ex467 are connected to each other via a bus ex470.

The power supply circuit unit ex461 activates the smartphone ex214 in an operable state by supplying electric power to each unit from the battery pack when the call ends and the power key is turned on by the user's operation.

Based on the control of the main control unit ex460 having a CPU, ROM, RAM, etc., the smartphone ex214 converts the voice signal picked up by the voice input unit ex456 in the voice call mode into a digital voice signal by the voice signal processing unit ex454. This is subjected to spectrum diffusion processing by the modulation / demodulation unit ex452, digital-analog conversion processing and frequency conversion processing by the transmission / reception unit ex451, and then transmitted via the antenna ex450. Further, the smartphone ex214 amplifies the received data received via the antenna ex450 in the voice call mode, performs frequency conversion processing and analog-digital conversion processing, and performs spectrum reverse diffusion processing by the modulation / demodulation unit ex452, and the voice signal processing unit ex454. After converting to an analog audio signal with, this is output from the audio output unit ex457.

Further, when the e-mail is transmitted in the data communication mode, the text data of the e-mail input by the operation of the operation unit ex466 of the main body unit is sent to the main control unit ex460 via the operation input control unit ex462. The main control unit ex460 performs spread spectrum processing on the modulation / demodulation unit ex452, digital-to-analog conversion processing and frequency conversion processing on the transmission / reception unit ex451, and then transmits the text data to the base station ex210 via the antenna ex450. .. When receiving an e-mail, the received data is processed in the reverse manner and output to the display unit ex458.

When transmitting video, still images, or video and audio in the data communication mode, the video signal processing unit ex455 compresses the video signal supplied from the camera unit ex465 by the moving image coding method shown in each of the above embodiments. The encoded video data is transmitted to the multiplexing / separating unit ex453. Further, the audio signal processing unit ex454 encodes the audio signal picked up by the audio input unit ex456 while the camera unit ex465 is capturing the video, still image, etc., and sends the encoded audio data to the multiplexing / separation unit ex453. do.

The multiplexing / separating unit ex453 multiplexes the encoded video data supplied from the video signal processing unit ex455 and the encoded audio data supplied from the audio signal processing unit ex454 by a predetermined method, and is obtained as a result. The multiplexed data is subjected to spectrum diffusion processing by the modulation / demodulation unit (modulation / demodulation circuit unit) ex452, digital-analog conversion processing and frequency conversion processing by the transmission / reception unit ex451, and then transmitted via the antenna ex450.

When receiving video file data linked to a homepage, etc. in data communication mode, or when receiving e-mail with video and / or audio attached, the multiplexed data received via the antenna ex450 is decoded. In order to do so, the multiplexing / separating unit ex453 separates the multiplexed data into a bit stream of video data and a bit stream of audio data, and processes the video data encoded via the synchronization bus ex470 as a video signal. In addition to supplying the coded audio data to the unit ex455, the encoded audio data is supplied to the audio signal processing unit ex454. The video signal processing unit ex455 decodes the video signal by decoding by the video decoding method corresponding to the video coding method shown in each of the above embodiments, and the display unit ex458 via the LCD control unit ex459. Then, for example, the video and still images included in the moving image file linked to the homepage are displayed. Further, the audio signal processing unit ex454 decodes the audio signal, and the audio output unit ex457 outputs the audio.

Further, the terminals such as the smartphone ex214 are a transmission / reception terminal having both an encoder and a decoder, a transmitter terminal having only an encoder, and a receiving terminal having only a decoder, as in the case of the television ex400. The implementation format is conceivable. Further, the digital broadcasting system ex300 has been described as receiving and transmitting multiplexed data in which music data and the like are multiplexed with video data, but data in which character data and the like related to video are multiplexed in addition to audio data. It may be the video data itself instead of the multiplexed data.

Further, the present invention is not limited to the above-described embodiment, and various modifications or modifications can be made without departing from the scope of the present invention.

The present invention can be applied to a video distribution system that distributes video shot by a plurality of cameras.

100 Video distribution system 101 Camera 102 Terminal device 103 Server 104A, 104B Network 105 Monitor 106 User 111, 121 Reception unit 112, 122 Storage unit 113, 123 Image processing unit 114 Output unit 115, 124 Transmission unit 116, 126 Control unit 300A, 300B, 300C Output screen 301 Viewpoint image display screen 302,313 Viewpoint selection screen 303A, 303B, 303C, 304A, 304B Operation buttons 305, 306, 308 Information display screen 307A, 307B, 307C Operation menu 311 Main screen 312A 1st sub Screen 312B 2nd sub screen

Claims (4)

It is a video distribution method by a server that distributes a first video signal based on a plurality of images obtained by shooting the same scene from a plurality of different viewpoints in a plurality of continuous periods to a terminal device.
A first selection step of selecting a first selection video from the plurality of videos for each of the plurality of periods based on the evaluation of the plurality of videos by a plurality of users.
With the first transmission step of transmitting the first video signal for continuously reproducing only the plurality of first-selected videos selected for the plurality of periods among the plurality of videos to the terminal device. Including
The evaluation is a video distribution method that indicates the quality of the plurality of videos. The video distribution method according to claim 1, wherein the evaluation includes a degree of badness with respect to the plurality of videos. The evaluation is the video distribution method according to claim 1 or 2, which is input by a user in the terminal device as a comment. A server that distributes a first video signal based on a plurality of images obtained by shooting the same scene from a plurality of different viewpoints in a plurality of consecutive periods to a terminal device.
A selection unit that selects a first selection video from the plurality of videos for each of the plurality of periods based on the evaluation of the plurality of videos by a plurality of users.
A transmission unit that transmits the first video signal for continuously reproducing only the plurality of first-selected videos selected for the plurality of periods among the plurality of videos is provided. ,
The evaluation is a server that indicates the quality of the plurality of images.

Images