JP2021150863A - Display control device, operation method of display control device, and operation program of display control device - Google Patents

Abstract

To provide a display control device, an operation method of the display control device, and an operation program of the display control device which enables a user to understand at what viewpoint and how a specific target object existing in a scene looks like.SOLUTION: An instruction reception unit of a terminal device accepts a switching start instruction to start switching a viewpoint during the reproduction of an image at one of viewpoints obtained by imaging the same scene at the same timing by multiple imaging devices arranged at different positions. When an image reproduction processing unit receives the switching start instruction, the image reproduction processing unit performs emphasis processing for emphasizing a specific target object existing in the scene with respect to the frame image of a candidate image which is an image of the viewpoint being the switching candidate. The image reproduction processing unit controls to present the viewpoint as a switching candidate by displaying the frame image of the candidate image after the emphasis processing.SELECTED DRAWING: Figure 11

Description

The technique of the present disclosure relates to a display control device, an operation method of the display control device, and an operation program of the display control device.

Patent Document 1 describes a user's image of one viewpoint selected by the user from among the images of a plurality of viewpoints obtained by imaging the same scene at the same timing by a plurality of imaging devices arranged at different positions. The technology for distributing and displaying the information on the terminal device is described. In Patent Document 1, the viewpoint of an image to be displayed on the terminal device is switched in response to a user's instruction to confirm switching of the viewpoint such as a swipe operation.

Japanese Unexamined Patent Publication No. 2016-01145

In the technique described in Patent Document 1, it is difficult for the user to grasp what kind of viewpoint the specific object such as a person existing in the scene looks like.

The technology of the present disclosure includes a display control device, an operation method of the display control device, and an operation of the display control device, which makes it easy for the user to understand what kind of viewpoint the specific target object existing in the scene looks like. The purpose is to provide a program.

The display control device of the present disclosure includes at least one processor, and the processor is one of a plurality of viewpoint images obtained by capturing the same scene at the same timing by a plurality of image pickup devices arranged at different positions. When a switching start instruction to start switching viewpoints is received during playback of the viewpoint image, and a switching start instruction is received, a specific target existing in the scene with respect to the candidate image which is the viewpoint image as a switching candidate. By performing an emphasis process for emphasizing an object and displaying a candidate image after the emphasis process, control is performed to present a viewpoint as a switching candidate.

As the enhancement process, the processor preferably performs a process of hiding the background other than the specific target object in the frame image of the candidate video.

There are a plurality of candidate images, and it is preferable that the processor controls to switch and display the plurality of candidate images after the emphasis processing at predetermined time intervals.

When the video is being reproduced while the video from multiple viewpoints is being captured by the plurality of imaging devices, the processor selects the video currently being played from the viewpoints presented as switching candidates. When the switching confirmation instruction for determining the viewpoint of the image to be reproduced next is received, it is preferable to start the reproduction of the image of the viewpoint determined by the switching confirmation instruction in real time in accordance with the timing of imaging.

Further, when the image is reproduced after the images of the plurality of viewpoints are captured by the plurality of imaging devices, the processor is next to the image currently being reproduced from the viewpoints presented as switching candidates. When the switching confirmation instruction for confirming the viewpoint of the image to be reproduced is received, it is preferable to start the reproduction of the image of the viewpoint confirmed by the switching confirmation instruction by time-shifting to the time when the switching start instruction is received. ..

It is preferable that the processor receives a switching start instruction by a plurality of types of gesture operations of the user for the touch panel display displaying the image, and changes the viewpoint as a switching candidate according to the type of the gesture operation.

The method of operating the display control device of the present disclosure is during reproduction of an image of one viewpoint among the images of a plurality of viewpoints obtained by imaging the same scene at the same timing by a plurality of image pickup devices arranged at different positions. , The instruction reception step that accepts the switching start instruction to start switching the viewpoint, and when the switching start instruction is received, the specific target object existing in the scene is emphasized with respect to the candidate image that is the image of the viewpoint that is the switching candidate. It is provided with a processing step for performing the emphasis processing and a display control step for controlling to present a viewpoint as a switching candidate by displaying the candidate image after the emphasis processing.

The operation program of the display control device of the present disclosure is during playback of an image of one viewpoint among the images of a plurality of viewpoints obtained by imaging the same scene at the same timing by a plurality of image pickup devices arranged at different positions. , The instruction reception unit that accepts the switching start instruction to start switching the viewpoint, and when the switching start instruction is received, the specific target object existing in the scene is emphasized with respect to the candidate image that is the image of the viewpoint that is the switching candidate. The computer functions as a processing unit that performs emphasis processing and a display control unit that controls to present a viewpoint that is a switching candidate by displaying a candidate image after the emphasis processing.

The method of operating the display control device of the present disclosure is a first image of one viewpoint among the images of a plurality of viewpoints obtained by imaging the same scene at the same timing by a plurality of image pickup devices arranged at different positions. The first playback step for playing back the video of A viewpoint that is a candidate for switching is presented by displaying a processing step that emphasizes a specific target object existing in the scene and a candidate image after the emphasis processing on the candidate image that is the image of the viewpoint. A display control step for controlling, a second instruction receiving step for receiving a switching confirmation instruction for determining the viewpoint of the second image to be reproduced next to the first image from the viewpoints presented as switching candidates, and switching. It includes a second reproduction step of reproducing the second image of the viewpoint confirmed by the confirmation instruction.

According to the technology of the present disclosure, the display control device, the operation method of the display control device, and the display control device make it easy for the user to understand what kind of viewpoint the specific target object existing in the scene looks like. Operation program can be provided.

It is a figure which shows the video distribution system. It is a figure which shows the image information. It is a block diagram which shows the computer which constitutes a video distribution server and a terminal apparatus. It is a block diagram of a video distribution server. It is a block diagram of a terminal device. It is a figure which shows the initial screen. It is a figure which shows the video reproduction screen. It is a table which shows the relationship between the type of a gesture operation, and the image pickup apparatus of a candidate target. It is a figure which shows the relationship between the operation amount of a gesture operation and the image pickup apparatus of a candidate target, FIG. 9A shows the case where the operation amount is relatively small, and FIG. 9B shows the case where the operation amount is relatively large. It is a figure which shows the detail of the highlighting process and the switching display process performed in a video reproduction processing section. It is a figure which shows the example of the emphasis processing. It is a figure which shows the example of the highlighting process and the switching display process. It is a figure which shows the state of switching the viewpoint of an image when a switching confirmation instruction is given. It is a figure which shows the transition of the image which is reproduced on the image reproduction screen. It is a flowchart which shows the processing procedure of a terminal apparatus and a video distribution server. It is a flowchart which shows the processing procedure of a terminal apparatus and a video distribution server. It is a flowchart which shows the processing procedure of a terminal apparatus and a video distribution server. It is a figure which shows another example of the transition of the image reproduced on the image reproduction screen. It is a figure which shows another example of the emphasis processing. It is a figure which shows another aspect of a video distribution server and a terminal apparatus. It is a figure which shows another aspect of a video distribution server and a terminal apparatus. It is a figure which shows the video reproduction screen provided with the comment display column.

In FIG. 1, the video distribution system 2 includes a video distribution server 10, an imaging device 11, and a terminal device 12. The video distribution server 10, the imaging device 11, and the terminal device 12 are connected to each other via a network 13 so as to be able to communicate with each other. The network 13 is, for example, a WAN (Wide Area Network) such as the Internet or a public communication network.

There are a plurality of image pickup devices 11 and they are arranged at different positions. More specifically, the plurality of image pickup devices 11 are possessed by a plurality of spectators 23 who are watching a soccer game being played in the soccer field 21 of the stadium 20 in the spectator seats 22. The spectator 23 uses the image pickup device 11 to image the image 28 of the soccer game (see FIG. 2 and the like) at the same timing. Then, the video information 24 including the captured video 28 is transmitted to the video distribution server 10 through the network 13. Here, the soccer game is an example of a "scene" related to the technique of the present disclosure.

The terminal device 12 is possessed by a user 25, which is one of a plurality of spectators 23. The terminal device 12 has at least a function of receiving the video information 24 from the video distribution server 10 through the network 13 and a function of reproducing and displaying the video 28 included in the received video information 24. Specifically, the function of reproducing and displaying the image 28 is realized by the touch panel display 26. The touch panel display 26 has a function of accepting a gesture operation of the user 25 in addition to the function of reproducing and displaying the image 28. The touch panel display 26 reproduces and displays the image 28 of one viewpoint selected by the user 25 among the images 28 of the plurality of viewpoints obtained by the plurality of imaging devices 11. Further, the viewpoint of the image 28 to be reproduced and displayed on the touch panel display 26 can be switched by a gesture operation. The terminal device 12 is an example of a "display control device" according to the technique of the present disclosure. The number of terminal devices 12 may be one or more, and similarly, the number of users 25 may be one or more. Further, the user 25 does not have to be one of the plurality of spectators 23, and may be outside the stadium 20.

The image pickup device 11 and the terminal device 12 have different names and symbols for convenience of explanation, but the image pickup device 11 also has the functions of the terminal device 12, and the terminal device 12 also has the functions of the image pickup device 11. That is, the image pickup device 11 and the terminal device 12 are actually the same type of device. Therefore, the image pickup device 11 may be the terminal device 12, or the terminal device 12 may be the image pickup device 11. Therefore, the spectator 23 can reproduce and display the video 28 from a different viewpoint while capturing the video 28 with the image pickup device 11. Similarly, the user 25 can capture the image 28 while reproducing and displaying the images 28 from different viewpoints on the terminal device 12. Specifically, the image pickup device 11 and the terminal device 12 are a digital camera, a smartphone, a tablet terminal, a notebook-type personal computer, and the like.

The video distribution server 10 is, for example, a server computer, a workstation, or the like. The video distribution server 10 receives the video information 24 from the image pickup device 11 through the network 13. Further, the video distribution server 10 distributes the video information 24 to the terminal device 12 through the network 13. In the following, an example in which the video information 24 is transmitted from a plurality of imaging devices 11 to the video distribution server 10 in real time and the user 25 views the video 28 on the terminal device 12 in real time during a soccer match. That is, an example will be described in which the video 28 is being reproduced while the video 28 from a plurality of viewpoints is being imaged by the plurality of image pickup devices 11.

As shown in FIG. 2, the video information 24 includes a video 28, a device ID (Identification Data) of the image pickup device 11, and a position where the image pickup device 11 is arranged. The video 28 includes audio, and when the video 28 is reproduced and displayed on the touch panel display 26 of the terminal device 12, the audio is also output from the speaker (not shown) of the terminal device 12. The device ID is a symbol, a number, or the like for uniquely identifying the image pickup device 11. The position where the image pickup device 11 is arranged is, for example, the latitude and longitude and the altitude detected by the GPS (Global Positioning System) function built in the image pickup device 11.

In FIG. 3, the computers constituting the video distribution server 10 and the terminal device 12 basically have the same configuration, and the storage device 30, the memory 31, the CPU (Central Processing Unit) 32, the communication unit 33, and the UI (User). An Interface) system device 34 is provided. These are interconnected via a bus line 35.

The storage device 30 is a hard disk drive built in the computers constituting the video distribution server 10 and the terminal device 12, or connected via a cable or a network. Alternatively, the storage device 30 is a disk array in which a plurality of hard disk drives are connected in series. The storage device 30 stores a control program such as an operating system, various application programs (hereinafter, abbreviated as AP (Application Program)), various data associated with these programs, and the like. A solid state drive may be used instead of the hard disk drive.

The memory 31 is a work memory for the CPU 32 to execute a process. The CPU 32 comprehensively controls each part of the computer by loading the program stored in the storage device 30 into the memory 31 and executing the processing according to the program.

The communication unit 33 is a network interface that controls transmission of various information via the network 13. The UI device 34 displays various information to the user 25 and receives input of various operation instructions from the user 25. In the video distribution server 10, the UI device 34 is, for example, a display, a keyboard, and a mouse. In the terminal device 12, the UI device 34 is the touch panel display 26 described above.

In the following description, the subscript "A" is attached to each part of the computer constituting the video distribution server 10, and the subscript "B" is attached to each part of the computer constituting the terminal device 12. To distinguish.

In FIG. 4, the server program 40 is stored in the storage device 30A of the video distribution server 10. When the server program 40 is started, the CPU 32A of the video distribution server 10 cooperates with the memory 31 and the like to read / write (hereinafter abbreviated as RW (Read Write)) control unit 50, request reception unit 51, and distribution. It functions as a control unit 52.

The RW control unit 50 controls the storage of various information in the storage device 30A and the reading of various information in the storage device 30A. The RW control unit 50 stores the video information 24 from the image pickup device 11 in the storage device 30A. As a result, the storage device 30A stores a plurality of video information 24 as the video information group 55.

The RW control unit 50 has at least one frame image 28CF of a viewpoint video (hereinafter referred to as a candidate video) 28C (see FIG. 14) as a switching candidate among the plurality of video information 24 constituting the video information group 55. Read from storage device 30A. Although not shown, the RW control unit 50 reads out the device ID of the image pickup device 11 registered in the video information 24 including the candidate video 28C together with the frame image 28CF. The RW control unit 50 outputs the frame image 28CF of the read candidate image 28C and the device ID of the image pickup device 11 to the distribution control unit 52.

Further, the RW control unit 50 reads out one video information 24 including the video 28 to be reproduced by the terminal device 12 from the storage device 30A among the plurality of video information 24 constituting the video information group 55. The RW control unit 50 outputs the read video information 24 to the distribution control unit 52.

The request receiving unit 51 receives the distribution request 60 from the terminal device 12. The distribution request 60 requests the distribution of the video information 24. The delivery request 60 includes a device ID for uniquely identifying the terminal device 12.

Further, the request receiving unit 51 receives the candidate video request 61 from the terminal device 12. The candidate video request 61 requests the distribution of the frame image 28CF of the candidate video 28C. The candidate video request 61 includes information on the device ID of the terminal device 12, the position where the terminal device 12 is arranged, and the type and amount of the user 25's gesture operation on the touch panel display 26. The position where the terminal device 12 is arranged is, for example, the latitude and longitude and the altitude detected by the GPS function built in the terminal device 12, as in the case of the image pickup device 11.

Further, the request receiving unit 51 receives the switching request 62 from the terminal device 12. The switching request 62 requests the distribution of the video information 24 including the video 28 in order to switch the viewpoint from the viewpoint of the video 28P currently being played back to one video 28. The switching request 62 includes the device ID of the terminal device 12 and the device ID of the image pickup device 11 of the video information 24 requesting distribution. The request receiving unit 51 outputs the distribution request 60, the candidate video request 61, and the switching request 62 to the RW control unit 50 and the distribution control unit 52. The video 28P currently being played is an example of the "first video" according to the technique of the present disclosure.

The distribution control unit 52 controls the distribution of the video information 24 to the terminal device 12. Further, the distribution control unit 52 controls distribution of the frame image 28CF and the device ID of the image pickup device 11 to the terminal device 12. The distribution control unit 52 determines the distribution destination terminal device of the video information 24, the frame image 28CF, and the device ID of the image pickup device 11 according to the device ID of the terminal device 12 included in the distribution request 60, the candidate video request 61, and the switching request 62. Identify twelve. Of the video information 24, the device ID of the image pickup device 11 and the position where the image pickup device 11 is arranged are information unnecessary for the terminal device 12. Therefore, the video information 24 including only the video 28, excluding the device ID of the image pickup device 11 and the position where the image pickup device 11 is arranged, may be delivered to the terminal device 12.

In FIG. 5, the video sharing AP 70 is stored in the storage device 30B of the terminal device 12. The video sharing AP70 is a program for operating the computer constituting the terminal device 12 as a "display control device" according to the technology of the present disclosure. That is, the video sharing AP70 is an example of the “display control device operation program” according to the technology of the present disclosure.

When the video sharing AP 70 is activated, the CPU 32B of the terminal device 12 functions as an instruction receiving unit 75, a request transmitting unit 76, a receiving unit 77, and a video reproduction processing unit 78 in cooperation with the memory 31 and the like. The CPU 32B is an example of a "processor" according to the technique of the present disclosure.

The instruction receiving unit 75 receives various operation instructions 79 input to the touch panel display 26 by the user 25. The operation instruction 79 includes a distribution instruction for the video information 24, a switching start instruction for starting the switching of the viewpoint, a switching confirmation instruction for determining the viewpoint of the video to be reproduced next to the video 28P currently being reproduced, and the like. The instruction receiving unit 75 outputs the information of the operation instruction 79 to the request transmitting unit 76.

The request transmission unit 76 transmits a request corresponding to the information of the operation instruction 79 from the instruction reception unit 75 to the video distribution server 10. For example, the request transmission unit 76 outputs the distribution request 60 to the video distribution server 10 in response to the distribution instruction of the video information 24. Further, the request transmission unit 76 outputs the candidate video request 61 to the video distribution server 10 in response to the switching start instruction. Further, the request transmission unit 76 outputs the switching request 62 to the video distribution server 10 in response to the switching confirmation instruction.

The receiving unit 77 receives the video information 24 from the video distribution server 10. The receiving unit 77 outputs the video information 24 to the video reproduction processing unit 78. Further, the receiving unit 77 receives the frame image 28CF from the video distribution server 10 and the device ID of the imaging device 11. The receiving unit 77 outputs the frame image 28CF to the video reproduction processing unit 78, and outputs the device ID of the imaging device 11 to the request transmitting unit 76, respectively.

The video reproduction processing unit 78 performs a process of reproducing the video 28 included in the video information 24 from the reception unit 77. Specifically, the video reproduction processing unit 78 generates a video reproduction screen 80 (see also FIG. 7) for reproducing the image 28. The video reproduction processing unit 78 outputs the generated video reproduction screen 80 to the touch panel display 26.

The video reproduction processing unit 78 performs emphasis processing on the frame image 28CF of the candidate video 28C to emphasize a specific target object existing in the scene (see FIG. 10 and the like). Further, the video reproduction processing unit 78 controls to present the viewpoint as a switching candidate by displaying the frame image 28CF after the enhancement processing on the touch panel display 26. At this time, when there are a plurality of frame image 28CFs, the video reproduction processing unit 78 performs a switching display process of switching and displaying the plurality of frame images 28CFs at predetermined time intervals (see FIG. 10 and the like). That is, the video reproduction processing unit 78 is an example of the “processing unit” and the “display control unit” according to the technology of the present disclosure.

FIG. 6 is an initial screen 85 displayed on the touch panel display 26 when the video sharing AP 70 is activated in the terminal device 12. The initial screen 85 is generated by the video reproduction processing unit 78 and output to the touch panel display 26. Information 86 of the video sharing group in which the user 25 can participate is displayed on the initial screen 85. The information 86 of the video sharing group is derived based on the position where the terminal device 12 is arranged. In FIG. 6, the position of the terminal device 12 is in the "Toyama Prefectural Stadium", and the information 86 of the video sharing group of the soccer game of "Toyama Turgies VS Shizuoka Wolves" held at the "Toyama Prefectural Stadium" is displayed. The example is shown. The video sharing group may be set in the video distribution server 10 by the operation manager of the video distribution server 10 and the information may be transmitted from the video distribution server 10 to the terminal device 12. Alternatively, the audience 23 or the user 25 may set the video distribution server 10 and transmit the information from the video distribution server 10 to the terminal device 12.

When the user 25 intends to join the video sharing group indicated by the information 86, the user 25 selects the join button 87. The selection of the participation button 87 corresponds to the distribution instruction of the video information 24, and the request transmission unit 76 outputs the distribution request 60 when the participation button 87 is selected.

FIG. 7 shows an example of the video reproduction screen 80 displayed on the touch panel display 26 of the terminal device 12. The video 28 is reproduced and displayed on the video playback screen 80.

For example, the RW control unit 50 uses one video information 24 that is randomly selected and read from the video information group 55 as the video information 24 that is first distributed in response to the distribution request 60. Alternatively, the RW control unit 50 sets the video information 24 including the video 28 captured by the image pickup device 11 located at the farthest position from the terminal device 12 as the video information 24 to be delivered first. Further, the RW control unit 50 first delivers the video information 24 including the video 28 that the user 25 currently viewing the most, or the video information 24 including the video 28 having the highest cumulative number of views in the past. Information 24 may be used. Further, the video information 24 including the video 28 captured by the image pickup device 11 selected by the user 25 may be the video information 24 to be distributed first. In this case, the touch panel display 26 may display a screen having a bird's-eye view image of the stadium 20 provided with an icon indicating the position where the image pickup device 11 is arranged, and the user 25 may select the desired image pickup device 11 icon. good.

As shown in Table 90 of FIG. 8, there are four types of gesture operations of the switching start instruction: a left swipe operation, a right swipe operation, a pinch-in operation, and a pinch-out operation. The left swipe operation is, for example, an operation of swiping the index finger from right to left on the touch panel display 26. The right swipe operation is, for example, an operation of swiping the index finger from left to right on the touch panel display 26. The pinch-in operation is, for example, an operation in which the thumb and the index finger are brought into contact with the touch panel display 26 with the thumb and the index finger separated, and then one or both of the thumb and the index finger are slid on the touch panel display 26 to move the thumb and the index finger closer to each other. Is. In the pinch-out operation, for example, after touching the touch panel display 26 with the thumb and index finger close to each other, one or both of the thumb and index finger are slid on the touch panel display 26 to move the thumb and index finger apart. It is an operation.

An imaging device 11 as a candidate target according to the type of each gesture operation is addressed to each gesture operation. The image pickup device 11 as a candidate target is one or more image pickup devices 11 among the plurality of image pickup devices 11. The image 28 captured by the image pickup device 11 of the candidate target is, that is, the candidate image 28C. The image reproduction processing unit 78 treats the frame image of the image 28 captured by the image pickup device 11 of the candidate image as the frame image 28CF of the candidate image 28C.

For the left swipe operation, the image pickup device 11 located to the right of the reference position RP is addressed, and for the right swipe operation, the image pickup device 11 located to the left of the reference position RP is addressed as the candidate image pickup device 11. It has been. Further, for the pinch-in operation, the image pickup device 11 located behind the reference position RP is addressed, and for the pinch-out operation, the image pickup device 11 located in front of the reference position RP is addressed as the candidate image pickup device 11. ing. The reference position RP is, for example, the position where the image pickup device 11 that is capturing the image 28P currently being reproduced is arranged. In this way, the instruction receiving unit 75 changes the viewpoint as a switching candidate according to the type of gesture operation. The reference position RP when the video information 24 is first distributed in response to the distribution request 60 may be the position where the terminal device 12 is arranged.

The image pickup device 11 of the candidate target also changes depending on the operation amount of the gesture operation. That is, in FIG. 9 exemplifying the right swipe operation, when the operation amount is relatively small as shown in FIG. 9A, for example, one image pickup device 11 at a position close to the reference position RP (in FIG. 9A, the left side of the reference position RP). One image pickup device 11) located adjacent to the image pickup device 11) is set as the candidate image pickup device 11. In this case, only one frame image 28CF is distributed from the video distribution server 10 to the terminal device 12. On the other hand, when the amount of operation is relatively large as shown in FIG. 9B, for example, a total of five image pickup devices 11 counting to the left from the reference position RP are designated as candidate image pickup devices 11. In this case, the number of frame images 28CF distributed from the video distribution server 10 to the terminal device 12 is five.

The gesture operation of the switching confirmation instruction is, for example, a single tap operation in which the desired frame image 28CF is displayed on the touch panel display 26, and an arbitrary portion of the touch panel display 26 is tapped once with an index finger or the like.

As shown in FIG. 10, the enhancement process is a process of emphasizing a specific target object existing in the scene with respect to the frame image 28CF of the candidate image 28C. More specifically, the enhancement process uses a well-known image recognition technique to extract a specific target object from the frame image 28CF (step ST100), and hides the background other than the specific target object in the frame image 28CF by painting it in black. (Step ST110). The operation manager of the video distribution server 10 may set the specific target object in the video distribution server 10 and transmit the information from the video distribution server 10 to the terminal device 12, or the user 25 sets the specific target object in the terminal device 12. You may.

The switching display process is performed when there are a plurality of image pickup devices 11 as candidate targets and there are a plurality of frame image 28CFs. The switching display process is a process of switching and displaying a plurality of frame images 28CF after the highlighting process at predetermined time intervals (hereinafter, set time intervals) (step ST200). The set time interval is several seconds, for example, 2 seconds. The switching display process is terminated when there is a switching confirmation instruction or an instruction to end the switching display process (YES in step ST210). The end instruction is, for example, a double tap operation in which an arbitrary portion of the touch panel display 26 is tapped twice in succession with the index finger or the like.

In FIG. 11 showing a specific example of the emphasis processing, the video reproduction processing unit 78 extracts the player 100 and the soccer ball 101 shown in the frame image 28CF. Next, the video reproduction processing unit 78 hides the soccer field 21 other than the player 100 and the soccer ball 101 by painting them in black as shown by hatching, and obtains a frame image 28CFX. In this case, the player 100 and the soccer ball 101 are examples of the "specific object" according to the technique of the present disclosure. Further, the soccer field 21 other than the player 100 and the soccer ball 101 is an example of the "background" according to the technique of the present disclosure.

FIG. 12 is an example when there are a plurality of frame images 28CF. In this case as well, as in the case of FIG. 11, the video reproduction processing unit 78 extracts the player 100 and the soccer ball 101 reflected in the frame images 28CF_1 and 28CF_2. Then, the soccer field 21 other than the player 100 and the soccer ball 101 is painted black to hide the frame images 28CFX_1 and 28CFX_2.

Subsequently, the video reproduction processing unit 78 performs switching display processing, and switches and displays the frame images 28CFX_1 and 28CFX_2 after the emphasis processing on the video reproduction screen 80 at set time intervals.

FIG. 13 shows a case where a switching confirmation instruction is given to the frame image 28CFX_1 in the example of FIG. In this case, the video reproduction processing unit 78 reproduces and displays the original candidate image 28C_1 of the frame image 28CFX_1 on the image reproduction screen 80 instead of the image 28P that was reproduced before the switching start instruction. The original candidate image 28C_1 of the frame image 28CFX_1 is an example of the “second image” according to the technique of the present disclosure.

As shown in FIG. 14, before the switching start instruction is received by the instruction receiving unit 75, the video 28 reproduced on the video reproduction screen 80 is, of course, the video 28P currently being reproduced. When the instruction receiving unit 75 receives the switching start instruction, the video reproduction screen 80 displays a frame image 28CFX in which the background other than the specific target object is hidden by the emphasis processing, whereby the switching candidate is displayed. The viewpoint is presented. The time when the frame image 28CFX is displayed is the time when the playback of the video 28P currently being played is stopped.

FIG. 14 illustrates a case where there is one frame image 28CF, but when there are a plurality of frame images 28CF, the switching display process is performed and the plurality of frame image 28CFX after the highlighting process are set at the set time interval. It is switched and displayed with. After that, when the instruction receiving unit 75 receives the switching confirmation instruction, the image 28 reproduced on the image reproduction screen 80 is switched to the candidate image 28C to which the switching confirmation instruction is given. The candidate image 28C is captured at the time when the instruction receiving unit 75 receives the switching confirmation instruction. In this way, the video reproduction processing unit 78 starts the reproduction of the candidate image 28C of the viewpoint determined by the switching confirmation instruction in real time in accordance with the timing of imaging of the candidate image 28C. Therefore, the candidate image 28C_1 shown in FIG. 13 is advanced by the time from the time when the frame image 28CF_1 is captured to the time from the switching start instruction to the switching confirmation instruction.

Next, the operation of the above configuration will be described with reference to the flowcharts of FIGS. 15 to 17. First, when the server program 40 is started, as shown in FIG. 4, the CPU 32A of the video distribution server 10 functions as the RW control unit 50, the request reception unit 51, and the distribution control unit 52. Further, when the video sharing AP70 is activated, as shown in FIG. 5, the CPU 32B of the terminal device 12 functions as an instruction receiving unit 75, a request transmitting unit 76, a receiving unit 77, and a video reproduction processing unit 78. ..

The initial screen 85 shown in FIG. 6 is displayed on the touch panel display 26 of the terminal device 12. When the participation button 87 is selected by the user 25, the distribution instruction of the video information 24 is received by the instruction receiving unit 75 (step ST500). Then, the request transmission unit 76 transmits the distribution request 60 to the video distribution server 10 (step ST510).

In the video distribution server 10, the request receiving unit 51 receives the distribution request 60 (step ST600). The distribution request 60 is output from the request receiving unit 51 to the RW control unit 50 and the distribution control unit 52.

The RW control unit 50 reads out the video information 24 including the video 28 that is first played back on the video playback screen 80 from the video information group 55 of the storage device 30A (step ST610). The read video information 24 is output from the RW control unit 50 to the distribution control unit 52. Then, the distribution control unit 52 distributes the video information 24 to the terminal device 12 that is the source of the distribution request 60 (step ST620).

In the terminal device 12, the video information 24 is received by the receiving unit 77 (step ST520). The video information 24 is output from the receiving unit 77 to the video reproduction processing unit 78.

The video reproduction processing unit 78 generates a video reproduction screen 80 based on the image information 24, and the generated image reproduction screen 80 is displayed on the touch panel display 26 (step ST530). As a result, the video 28 is provided for viewing by the user 25. Step ST530 is an example of the "first regeneration step" according to the technique of the present disclosure.

When the user 25 wants to switch the viewpoint of the video 28, he / she performs one of the gesture operations of the switching start instruction shown in FIG. 8 on the touch panel display 26 on which the video playback screen 80 is displayed. As a result, as shown in FIG. 16, the switching start instruction is received by the instruction receiving unit 75 (step ST700). Then, the request transmission unit 76 transmits the candidate video request 61 to the video distribution server 10 (step ST710). The step ST700 is an example of the "instruction receiving step" and the "first instruction receiving step" according to the technique of the present disclosure.

In the video distribution server 10, the request receiving unit 51 receives the candidate video request 61 (step ST800). The candidate video request 61 is output from the request receiving unit 51 to the RW control unit 50 and the distribution control unit 52.

The RW control unit 50 reads out the frame image 28CF of the candidate image 28C corresponding to the candidate image request 61 from the storage device 30A (step ST810). The read frame image 28CF is output from the RW control unit 50 to the distribution control unit 52. Then, the distribution control unit 52 distributes the frame image 28CF to the terminal device 12 of the transmission source of the candidate video request 61 (step ST820).

In the terminal device 12, the frame image 28CF is received by the receiving unit 77 (step ST720). The frame image 28CF is output from the receiving unit 77 to the video reproduction processing unit 78.

As shown in FIGS. 10 to 12, the video reproduction processing unit 78 performs enhancement processing on the frame image 28CF to obtain the frame image 28CFX (step ST730). Then, under the control of the video reproduction processing unit 78, the frame image 28CFX is displayed on the video reproduction screen 80. When there are a plurality of frame images 28CFX, the plurality of frame images 28CFX are switched and displayed at set time intervals (step ST740). As a result, the viewpoint as a switching candidate is presented to the user 25. Step ST730 is an example of a "processing step" according to the technique of the present disclosure. Further, step ST740 is an example of a "display control step" according to the technique of the present disclosure.

The user 25 performs a single tap operation on an arbitrary portion of the touch panel display 26 in a state where the frame image 28CFX of a desired viewpoint is displayed on the video reproduction screen 80. As a result, as shown in FIG. 17, the switching confirmation instruction is received by the instruction receiving unit 75 (step ST900). Then, the request transmission unit 76 transmits the switching request 62 to the video distribution server 10 (step ST910). Step ST900 is an example of the "second instruction receiving step" according to the technique of the present disclosure.

In the video distribution server 10, the request receiving unit 51 receives the switching request 62 (step ST1000). The switching request 62 is output from the request receiving unit 51 to the RW control unit 50 and the distribution control unit 52.

The RW control unit 50 reads out the video information 24 including the candidate video 28C in response to the switching request 62 from the video information group 55 of the storage device 30A (step ST1010). The read video information 24 is output from the RW control unit 50 to the distribution control unit 52. Then, the distribution control unit 52 distributes the video information 24 to the terminal device 12 that is the source of the switching request 62 (step ST1020).

In the terminal device 12, the video information 24 is received by the receiving unit 77 (step ST920). The video information 24 is output from the receiving unit 77 to the video reproduction processing unit 78.

As shown in FIG. 13, the image reproduction processing unit 78 switches the display from the image 28P currently being reproduced to the candidate image 28C for which the switching confirmation instruction is given (step ST930). Step ST930 is an example of a "second regeneration step" according to the technique of the present disclosure.

On the video reproduction screen 80, the display is switched from the image 28P currently being reproduced to the frame image 28CFX of the candidate image 28C in which the background other than the specific target object is hidden in response to the switching start instruction. Then, the display is switched from the frame image 28CFX to the candidate image 28C in response to the switching confirmation instruction. According to the frame image 28CFX in which the emphasizing process for emphasizing the specific target object is performed, the user 25 sees what kind of viewpoint the specific object object looks like as compared with the case where the emphasizing process is not performed. Easy to grasp.

The enhancement process is a process of hiding the background other than the specific target object in the frame image 28CF. According to the process having such a visual effect, the user 25 can more easily grasp what kind of viewpoint the specific target object looks like.

When there are a plurality of frame images 28CFX, the video reproduction processing unit 78 controls to switch and display the plurality of frame images 28CFX after the enhancement process at set time intervals. Therefore, the user 25 can compare the plurality of frame images 28CFX and can easily select the candidate image 28C of the viewpoint that he / she prefers.

The image reproduction processing unit 78 starts reproduction of the candidate image 28C of the viewpoint determined by the switching confirmation instruction in real time in accordance with the timing of imaging of the candidate image 28C. Therefore, there is no time difference between the image 28 displayed on the image reproduction screen 80 and the actual scene for the time from the switching start instruction to the switching confirmation instruction. Therefore, it is possible to prevent the user 25 from feeling uncomfortable due to the time difference between the video 28 displayed on the video playback screen 80 and the actual scene.

The instruction receiving unit 75 receives the designation of the touch panel display 26 of the terminal device 12 by a plurality of types of gesture operations of the user 25 as shown in FIG. Then, the video reproduction processing unit 78 changes the viewpoint as a switching candidate according to the type of gesture operation. Therefore, it is possible to easily confirm the appearance of the image 28 of each viewpoint by a simple gesture operation.

In the above embodiment, during a soccer match, the video information 24 is transmitted from the plurality of imaging devices 11 to the video distribution server 10 in real time, and the user 25 views the video 28 on the terminal device 12 in real time. However, it is not limited to this. At least one of the transmission of the video information 24 from the image pickup apparatus 11 to the video distribution server 10 and the viewing of the video 28 may not be performed in real time during the soccer game. Specifically, the image 28 may be transmitted from the image pickup apparatus 11 to the image distribution server 10 after the soccer match is completed. Further, after the soccer match is over, the user 25 may watch the video 28 on the terminal device 12. That is, the video 28 may be reproduced after the video 28 from a plurality of viewpoints has been captured by the plurality of imaging devices 11.

When the user 25 views the video 28 on the terminal device 12 after the fact, the following aspects may be adopted.

In the above embodiment, as shown in FIG. 14, the reproduction of the candidate image 28C of the viewpoint determined by the switching confirmation instruction cut is started in real time at the timing of imaging of the candidate image 28C. On the other hand, when the user 25 subsequently views the video 28 on the terminal device 12, as shown in FIG. 18, the switching start instruction is received for the reproduction of the viewpoint candidate video 28C confirmed by the switching confirmation instruction. Time-shift to a point in time and start. In the case of non-real-time viewing, the user 25 does not compare the video 28 displayed on the video playback screen 80 with the actual scene. Therefore, even if the mode shown in FIG. 18 is adopted, there is a concern that the user 25 may feel uncomfortable. No. Since the video 28P currently being played back and the candidate video 28C are switched without a time lag, the display is rather natural for the user 25 who views the video 28 after the fact.

In the above embodiment, the emphasis processing is performed only on the candidate image 28C, but the present invention is not limited to this. For example, as shown in FIG. 19, the frame image 28PF of the video 28P currently being reproduced may be enhanced to obtain the frame image 28PFX. In this case, the image 28 reproduced on the image reproduction screen 80 changes in this order in the order of the frame image 28PF of the image 28P currently being reproduced, the frame image 28PFX after the enhancement processing, and the frame image 28CFX of the candidate image 28C after the enhancement processing. .. The switching display process may be performed including the frame image 28PFX. In this way, the user 25 can give a switching confirmation instruction after comparing the image condition of the specific target object of the video 28P currently being reproduced with the image condition of the specific target object of the candidate image 28C.

When starting the reproduction of the candidate video 28C for which the switching confirmation instruction has been given, the candidate video 28C may be paused and displayed for a predetermined period, for example, 2 seconds. By doing so, it is possible to impress the user 25 that the viewpoint has certainly been switched to the candidate image 28C.

The highlighting process is not limited to the mode of hiding the background other than the specific target object illustrated in the above embodiment. For example, the outline of a specific target object may be traced with a thick line of a specific color such as red. Alternatively, an animation in which light is emitted from a specific target object may be synthesized.

When the image 28 is imaged by the image pickup device 11 and the image 28 from a different viewpoint is reproduced and displayed, and when the image 28 is reproduced and displayed by the terminal device 12, the image 28 is imaged. Either one of the video 28 and the video 28 being reproduced may be displayed on the master screen and the other on the slave screen.

The gesture operation is not limited to the example left swipe operation, right swipe operation, pinch-in operation, and pinch-out operation. For example, the pinch-in operation may be replaced with a down swipe operation, and the pinch-out operation may be replaced with an up swipe operation.

The switching start instruction is not limited to the illustrated gesture operation. A switching start instruction may be input by a GUI (Graphical User Interface) such as a cross button provided on the video reproduction screen 80. Alternatively, a button is provided on the video reproduction screen 80, and when the button is operated, the video 28 of all the imaging devices 11 is treated as the candidate video 28C, and the frame image of the video 28 of all the imaging devices 11 is emphasized. The frame image may be switched and displayed at set time intervals. In this way, the image 28 to be reproduced can be selected from the images 28 of all the image pickup devices 11.

The display of the plurality of frame image CFX after the enhancement process may be manually switched by a swipe operation or the like. In this case, if the frame image CFX after the enhancement process is continuously displayed without being switched for a predetermined period, the original candidate image 28C of the frame image CFX is displayed without waiting for the switching confirmation instruction. You may switch.

In the above embodiment, a soccer game is illustrated as a scene, but the present invention is not limited to this. It may be a match of other sports such as athletics, baseball, basketball, volleyball, badminton, table tennis, rugby, American football, etc. It may also be a racetrack, a concert, a recital, an athletic meet, a wedding ceremony, or the like. Alternatively, it may be a street imaged by a plurality of surveillance cameras. Therefore, the specific target object is not limited to the illustrated player 100 and the soccer ball 101. The person shown in all the images 28 may be extracted, and the extracted person may be used as a specific target object.

The hardware configuration of the computers constituting the video distribution server 10 and the terminal device 12 can be appropriately changed according to the required performance such as processing capacity, safety, and reliability. Furthermore, not only hardware but also APs such as the server program 40 can be duplicated or distributed and stored in a plurality of storage devices for the purpose of ensuring safety and reliability. ..

In the above embodiment, one video information 24 including the video 28 to be reproduced and displayed on the video playback screen 80 is distributed from the video distribution server 10 to the terminal device 12, but the present invention is not limited to this. As shown in FIG. 20, the video information group 55 may be distributed from the video distribution server 10 to the terminal device 12 in response to the distribution request 60.

In this case, the receiving unit 77 of the terminal device 12 receives the video information group 55. The CPU 32B of the terminal device 12 is provided with an RW control unit having the same function as the RW control unit 50, in addition to the above-mentioned processing units 75 to 78. The RW control unit stores the video information group 55 received by the reception unit 77 in the storage device 30B. Further, the RW control unit reads out one video information 24 including the video 28 to be reproduced and displayed on the video reproduction screen 80 from the storage device 30B among the plurality of video information 24 constituting the video information group 55. Then, the read video information 24 is output to the video reproduction processing unit 78. In this way, if the video information group 55 is distributed from the video distribution server 10 to the terminal device 12, it is not necessary to transmit the candidate video request 61 and the switching request 62 from the terminal device 12 to the video distribution server 10.

The video distribution server 10 may take part or all of the functions of the "display control device" according to the technique of the present disclosure. When the video distribution server 10 is responsible for the function of the video reproduction processing unit 78, the video distribution server 10 responds to the distribution request 60, the candidate video request 61, and the switching request 62 from the terminal device 12, as shown in FIG. The video reproduction screen 80 is distributed to the terminal device 12. Information about the emphasis processing is attached to the video reproduction screen 80 distributed in response to the candidate video request 61. The information regarding the enhancement process includes, for example, information indicating a black-painted non-display area as a background in the frame image 28CF of the candidate image 28C. The video distribution server 10 outputs the video playback screen 80 in the form of screen data created by a markup language such as XML (Extension Markup Language). In addition, instead of XML, another data description language such as JSON (Javascript (registered trademark) Object Notification) may be used.

As in the video reproduction screen 105 shown in FIG. 22, a comment display field 106 input by the spectator 23 who possesses the image pickup device 11 that captures the image 28P currently being reproduced may be provided. The number of display columns 106 is the same as the number of images 28. The display of the display field 106 is switched according to the switching of the video 28.

In the above embodiment, for example, a process of executing various processes such as a RW control unit 50, a request reception unit 51, a distribution control unit 52, an instruction reception unit 75, a request transmission unit 76, a reception unit 77, and a video reproduction processing unit 78. As the hardware structure of the processing unit, various processors (processors) shown below can be used. For various processors, in addition to CPUs 32A and 32B, which are general-purpose processors that execute software (server program 40, video sharing AP70) and function as various processing units, manufacture of FPGA (Field Programmable Gate Array) and the like. It has a circuit configuration specially designed to perform a specific process such as a programmable logic device (PLD), which is a processor whose circuit configuration can be changed later, and / or an ASIC (Application Special Integrated Circuit). A dedicated electric circuit or the like that is a processor is included.

One processor may be composed of one of these various processors, or a combination of two or more processors of the same type or different types (for example, a combination of a plurality of FPGAs and / or a CPU and a CPU). It may be configured in combination with FPGA). Further, a plurality of processing units may be configured by one processor.

As an example of configuring a plurality of processing units with one processor, first, one processor is configured by a combination of one or more CPUs and software, as represented by a computer such as a client and a server. There is a form in which a processor functions as a plurality of processing units. Secondly, as typified by System on Chip (SoC), there is a form in which a processor that realizes the functions of the entire system including a plurality of processing units with one IC (Integrated Circuit) chip is used. be. As described above, the various processing units are configured by using one or more of the above-mentioned various processors as a hardware structure.

Further, as the hardware structure of these various processors, more specifically, an electric circuit (Circuitry) in which circuit elements such as semiconductor elements are combined can be used.

The techniques of the present disclosure can also be appropriately combined with the various embodiments described above and / or various variations. Further, not limited to the above embodiment, it goes without saying that various configurations can be adopted as long as they do not deviate from the gist. Further, the technique of the present disclosure extends to a storage medium for storing the program non-temporarily in addition to the program.

The contents described and illustrated above are detailed explanations of the parts related to the technique of the present disclosure, and are merely examples of the technique of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of a portion of the art of the present disclosure. Therefore, unnecessary parts may be deleted, new elements may be added, or replacements may be made to the described contents and illustrated contents shown above within a range that does not deviate from the gist of the technique of the present disclosure. Needless to say. In addition, in order to avoid complications and facilitate understanding of the parts related to the technology of the present disclosure, the above-mentioned description and illustrations require special explanation in order to enable the implementation of the technology of the present disclosure. The explanation about the common general technical knowledge is omitted.

As used herein, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, B alone, or a combination of A and B. Further, in the present specification, when three or more matters are connected and expressed by "and / or", the same concept as "A and / or B" is applied.

All documents, patent applications and technical standards described herein are to the same extent as if the individual documents, patent applications and technical standards were specifically and individually stated to be incorporated by reference. Incorporated by reference in the book.

2 Video distribution system 10 Video distribution server 11 Imaging device 12 Terminal device (display control device)
13 Network 20 Stadium 21 Soccer field 22 Audience seat 23 Audience 24 Video information 25 User 26 Touch panel display 28 Video 28C Candidate video (second video)
28CF, 28CFX Candidate video frame image 28P Currently playing video (first video)
28PF, 28PFX Frame image of the video currently being played 30, 30A, 30B Storage device 31 Memory 32, 32A CPU
32B CPU (processor)
33 Communication unit 34 UI device 35 Bus line 40 Server program 50 Read / write control unit (RW control unit)
51 Request receiving unit 52 Distribution control unit 55 Video information group 60 Distribution request 61 Candidate video request 62 Switching request 70 Video sharing application program (video sharing AP, operation program of display control device)
75 Instruction reception unit 76 Request transmission unit 77 Reception unit 78 Video playback processing unit (processing unit, display control unit)
79 Operation instructions 80, 105 Video playback screen 85 Initial screen 86 Video sharing group information 87 Participation button 90 Table 100 Player (specific target object)
101 Soccer ball (specific target object)
106 Display column RP Reference position ST100, ST110, ST200, ST210, ST500, ST510, ST520, ST600, ST610, ST620, ST710, ST720, ST800, ST810, ST820, ST910, ST920, ST1000, ST1010, ST1020 Step ST530 Step (No. 1 playback step)
ST700 step (instruction reception step, first instruction reception step)
ST730 step (processing step)
ST740 step (display control step)
ST900 step (second instruction reception step)
ST930 step (second reproduction step)

Claims (9)

With at least one processor
The processor
Switching start instruction to start switching of viewpoints during playback of the video of one viewpoint among the images of multiple viewpoints obtained by imaging the same scene at the same timing by a plurality of image pickup devices arranged at different positions. Accept,
When the switching start instruction is received, the candidate image, which is the image of the viewpoint that is the switching candidate, is subjected to the emphasis processing for emphasizing the specific target object existing in the scene.
By displaying the candidate image after the enhancement process, control is performed to present the viewpoint as the switching candidate.
Display control device. The processor
The display control device according to claim 1, wherein as the enhancement process, a process of hiding a background other than the specific target object in the frame image of the candidate image is performed. There are multiple candidate videos
The processor
The display control device according to claim 1 or 2, wherein the plurality of candidate images after the emphasis processing are controlled to be switched and displayed at predetermined time intervals. In the case where the video is being reproduced while the video of the plurality of viewpoints is being captured by the plurality of image pickup devices.
The processor
When a switching confirmation instruction for determining the viewpoint of the video to be played next to the currently playing video is received from the viewpoints presented as the switching candidates,
The display control device according to any one of claims 1 to 3, wherein the reproduction of the image of the viewpoint confirmed by the switching confirmation instruction is started in real time in accordance with the timing of the imaging. In the case where the video is reproduced after the video of the plurality of viewpoints is captured by the plurality of image pickup devices.
The processor
When a switching confirmation instruction for determining the viewpoint of the video to be played next to the currently playing video is received from the viewpoints presented as the switching candidates,
The display control device according to any one of claims 1 to 4, wherein the reproduction of the image of the viewpoint confirmed by the switching confirmation instruction is started by time-shifting at the time when the switching start instruction is received. The processor
The touch panel display that displays the image receives the switching start instruction by the user's multiple types of gesture operations.
The display control device according to any one of claims 1 to 5, wherein the viewpoint that is a switching candidate is changed according to the type of the gesture operation. During playback of the video of one of the viewpoints obtained by imaging the same scene at the same timing by a plurality of image pickup devices arranged at different positions, a switching start instruction for starting the switching of the viewpoint is issued. Instructions to accept Accepting steps and
When the switching start instruction is received, the processing step of emphasizing the specific target object existing in the scene with respect to the candidate image which is the image of the viewpoint which is the switching candidate, and the processing step.
A display control step that controls to present a viewpoint that is a switching candidate by displaying the candidate image after the enhancement process.
A method of operating a display control device comprising. Switching start instruction to start switching of viewpoints during playback of one of the images of multiple viewpoints obtained by imaging the same scene at the same timing by a plurality of imaging devices arranged at different positions. The instruction reception department that accepts
When the switching start instruction is received, the processing unit that emphasizes the specific target object existing in the scene with respect to the candidate image that is the image of the viewpoint that is the switching candidate, and the processing unit.
As a display control unit that controls to present a viewpoint as a switching candidate by displaying the candidate image after the enhancement process.
An operation program for the display control device that makes the computer work. A first reproduction step of reproducing a first image which is an image of one viewpoint among the images of a plurality of viewpoints obtained by imaging the same scene at the same timing by a plurality of image pickup devices arranged at different positions.
During the playback of the first video, the first instruction receiving step for receiving the switching start instruction for starting the viewpoint switching, and the first instruction receiving step.
When the switching start instruction is received, the processing step of emphasizing the specific target object existing in the scene with respect to the candidate image which is the image of the viewpoint which is the switching candidate, and the processing step.
A display control step that controls to present a viewpoint that is a switching candidate by displaying the candidate image after the enhancement process.
From the viewpoints presented as the switching candidates, a second instruction receiving step for receiving a switching confirmation instruction for determining the viewpoint of the second image to be reproduced next to the first image, and
A second reproduction step of reproducing the second image of the viewpoint determined by the switching confirmation instruction, and
A method of operating a display control device comprising.

Images