JP2019115035A - Display control device and display control method - Google Patents

Abstract

To provide a technology for allowing a user to select an appropriate virtual viewpoint.SOLUTION: A display control device acquires a plurality of virtual viewpoint images according to the positions and directions of a plurality of virtual viewpoints, and displays the plurality of acquired virtual viewpoint images on a screen used for operations related to the positions and directions of the virtual viewpoints. The virtual viewpoint images according to the positions and directions of the virtual viewpoints determined on the basis of the operations are created on the basis of a plurality of photographic images obtained by a plurality of cameras photographing an area to be photographed, and transmitted to another device.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a display technology of a virtual viewpoint image based on a captured image.

  In recent years, a plurality of cameras are installed at different positions, synchronized shooting with multiple viewpoints, and using a plurality of images (multi-viewpoint images) obtained by the shooting, not only images from the camera installation position but also arbitrary viewpoints A technique for generating an image (virtual viewpoint image) from the (virtual viewpoint) has attracted attention.

  Generation and viewing of a virtual viewpoint image based on a plurality of viewpoint images can be realized by the following process. That is, images taken by a plurality of cameras are collected in an image processing unit such as a server, and the image processing unit performs processing such as rendering based on a virtual viewpoint to generate a virtual viewpoint image, and the virtual viewpoint image is used as a user It can be realized by displaying on the terminal.

  In a service using virtual viewpoint images, for example, it is possible for a video producer to produce powerful viewpoint content from a video of a soccer or basketball game. In addition, the user watching the content can freely watch the game while freely moving the viewpoint, and can give the user a high sense of presence. Patent Document 1 discloses a technique in which a user selects one virtual viewpoint from a plurality of virtual viewpoints.

JP, 2014-215828, A

  In the invention described in Patent Document 1, the user can select one virtual viewpoint from among a plurality of prepared virtual viewpoints, and can display virtual viewpoint images in the selected virtual viewpoint. In addition, since the approximate position of the virtual viewpoint is displayed as a character string, it is possible to distinguish between virtual viewpoints. However, for example, when an operator of a virtual viewpoint uses in live sports broadcasting, it is not possible to know what appears in the virtual viewpoint image after switching before switching the virtual viewpoint. Therefore, there were cases where it was not possible to make an appropriate selection of a virtual viewpoint according to play. The present invention provides a technique for causing a user to select an appropriate virtual viewpoint.

  According to one aspect of the present invention, there is provided an acquisition unit for acquiring a plurality of virtual viewpoint images according to positions and orientations of a plurality of virtual viewpoints, and the plurality of virtual viewpoint images acquired by the acquisition unit And display control means for displaying on the screen used for the operation related to the orientation, and the virtual viewpoint image according to the position and the orientation of the virtual viewpoint determined based on the operation is an area to be photographed by a plurality of cameras Are generated on the basis of a plurality of photographed images obtained by photographing and are transmitted to another device.

  The configuration of the present invention allows the user to select an appropriate virtual viewpoint.

FIG. 1 is a block diagram showing an example of the configuration of a system. FIG. 2 is a block diagram showing an example of the functional configuration of a display control apparatus 100. The flowchart of the processing which a system performs. FIG. 7 is a view showing an example of the configuration of a display screen and an example of arrangement of virtual viewpoints. The figure which shows the structural example of "the information which concerns on several virtual viewpoints." FIG. 2 is a block diagram showing an example of the functional configuration of a display control apparatus 100. The flowchart of the processing which a system performs. The figure which shows an example of an input device. The flowchart of the processing which a system performs. FIG. 2 is a block diagram showing an example of the functional configuration of a display control apparatus 100. The flowchart of the processing which a system performs. The figure explaining a 4th embodiment. The figure explaining a 4th embodiment. The figure which shows an example of a virtual viewpoint set, and the structural example of table information.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific embodiments of the configuration described in the claims.

First Embodiment
First, a configuration example of a system according to the present embodiment will be described using the block diagram of FIG. As shown in FIG. 1, the system according to the present embodiment includes a display control device 100, an image generation device 110, and imaging devices 109-1 to 109-n (n is an integer of 2 or more).

  First, the imaging devices 109-1 to 109-n will be described. The imaging devices 109-1 to 109-n are provided to capture images of a subject from a plurality of different directions, and each of the imaging devices 109-1 to 109-n constitutes a captured image. The image (captured image) of each frame is output to the image generation device 110. The connection form between the imaging devices 109-1 to 109-n and the image generation device 110 is not limited to a specific connection form. That is, the connection form between the imaging devices 109-1 to 109-n and the image generation device 110 may be a daisy chain or may be a star network configuration. Also, this connection form may be wireless, wired, or a combination thereof.

  Next, the display control device 100 will be described. The display control device 100 requests the image generation device 110 to generate a virtual viewpoint image based on a virtual viewpoint, and performs display control of the virtual viewpoint image generated by the image generation device 110 according to the request. The display control device 100 is configured by a computer device such as a PC (personal computer), a smartphone, a tablet terminal device, and the like.

  A central processing unit (CPU) 101 executes processing using computer programs and data stored in a read-only memory (ROM) 102 and a random access memory (RAM) 103. Thus, the CPU 101 performs operation control of the entire display control apparatus 100, and executes or controls each process described later on the assumption that the display control apparatus 100 performs.

  The ROM 102 stores computer programs and data that do not require rewriting, such as start programs and setting data.

  The RAM 103 has an area for storing computer programs and data loaded from the ROM 102 or the HDD (hard disk drive) 104. Furthermore, the RAM 103 has an area for storing data (for example, a virtual viewpoint image) received from the image generation device 110 via the communication I / F (interface) 106. The RAM 103 also has a work area used when the CPU 101 executes various processes. Thus, the RAM 103 can appropriately provide various areas.

  The HDD 104 stores an OS (Operating System), and computer programs and data for causing the CPU 101 to execute or control each process described later on the assumption that the display control apparatus 100 performs the process. The data stored in the HDD 104 includes what is described as known information in the following description. The computer program and data stored in the HDD 104 are appropriately loaded to the RAM 103 under the control of the CPU 101, and are to be processed by the CPU 101.

  The display unit 105 is configured of a CRT, a liquid crystal screen, or the like, and can display the processing result of the CPU 101 as an image, characters, or the like. The display unit 105 may be a projection device that projects the processing result of the CPU 101 as an image or a character. The display unit 105 may be a touch panel screen, a head mounted display such as an HMD (head mounted display), a handheld display, a smartphone, or a tablet terminal.

  The communication I / F 106 functions as an interface for performing data communication with the image generation apparatus 110. The connection form between the display control device 100 and the image generation device 110 is not limited to a specific connection form. For example, the network between the display control device 100 and the image generation device 110 may be a wireless network, a wired network, or a combination thereof.

  The operation unit 107 includes a user interface such as a keyboard, a mouse, a joystick, a game pad, and a touch panel, and can input various instructions to the CPU 101 by a user operation.

  The CPU 101, the ROM 102, the RAM 103, the HDD 104, the display unit 105, the communication I / F 106, and the operation unit 107 are all connected to the bus 108. Note that the configuration of the display control device 100 illustrated in FIG. 1 is an example, and any configuration may be adopted as long as the processing to be described later can be realized as the processing performed by the display control device 100 below. For example, a configuration may be added to the display control apparatus 100 to read and write information on an information recording medium such as a flexible disk (FD), a CD-ROM, a DVD, a USB memory, an MO, and a flash memory. In this case, information described as being held by the HDD 104 may be stored in the information recording medium. Further, as a medium for recording information, a server device connected to the display control device 100 via a network may be used. As described above, the device that holds information such as computer programs and data handled by the display control device 100 is not limited to a specific device.

  Next, the image generation device 110 will be described. The image generation device 110 generates a virtual viewpoint image based on a virtual viewpoint according to a request from the display control device 100 based on an image captured by one or more of the imaging devices 109-1 to 109-n. Generate. Then, the image generation device 110 transmits the generated virtual viewpoint image to the display control device 100. The image generation device 110 is configured by a computer device such as a PC (personal computer), a smartphone, a tablet terminal device, or the like.

  Next, a functional configuration example of the display control device 100 will be described using the block diagram of FIG. The management unit 201 manages information related to a plurality of virtual viewpoints held in the HDD 104 and the RAM 103. A configuration example of “information related to a plurality of virtual viewpoints” managed by the management unit 201 will be described with reference to FIG. The table information of FIG. 5 is an example of “information related to a plurality of virtual viewpoints”. In the table information of FIG. 5, viewpoint information (viewpoint type, viewpoint ID, position, direction, angle of view) of each of six virtual viewpoints is registered. In FIG. 5, the viewpoint information includes the viewpoint type, viewpoint ID, position, direction, and angle of view, but the information included in the viewpoint information is not limited thereto, and, for example, the viewpoint type and viewpoint ID In addition to the position, the direction, and the angle of view, other information such as a focal length may be included. Further, the contents of the information defining each virtual viewpoint and the method of managing information defining each viewpoint are not limited to a specific form.

  As shown in FIG. 5, only one virtual viewpoint among the six virtual viewpoints is the selected virtual viewpoint (viewpoint type), and the remaining five virtual viewpoints are the switching virtual viewpoints (viewpoint type). The “selected virtual viewpoint” is a virtual viewpoint to be operated by the user operating the operation unit 107, and the “switching virtual viewpoint” is selected as the selected virtual viewpoint by the user operating the operation unit 107. It is a virtual viewpoint that is a candidate for In the case of FIG. 5, the virtual viewpoint corresponding to the viewpoint ID = 408 is the selected virtual viewpoint, and the virtual viewpoint corresponding to the viewpoint ID = 409 to 413 is the switching virtual viewpoint.

  For example, it is assumed that the user operates the operation unit 107 to select (designate) a virtual viewpoint of viewpoint ID = 409 as a selected virtual viewpoint. At this time, the management unit 201 changes the viewpoint type corresponding to the viewpoint ID = 409 to “selected virtual viewpoint”, and changes the viewpoint type corresponding to the viewpoint ID = 408 to “switching virtual viewpoint”. Thus, by operating the operation unit 107, the user can select any one virtual viewpoint from among the six virtual viewpoints registered in the table information of FIG. 5 as a selected virtual viewpoint. In the following, for detailed description, it is assumed that table information as shown in FIG. 5 is registered in the HDD 104 and the RAM 103, and the management unit 201 manages the table information. The initial value of the switching virtual view may be arbitrarily set in advance by the user, or may be any value determined by the system.

  Then, the management unit 201 reads out the viewpoint information of each of the switching virtual viewpoints registered in the table information and outputs the information to the output unit 202. Further, the management unit 201 reads out the viewpoint information of the selected virtual viewpoint registered in the table information. Then, the management unit 201 transmits a second position obtained by copying the position contained in the read out viewpoint information, a second direction obtained by copying the direction contained in the viewpoint information, and an angle of view contained in the viewpoint information. To create a second angle of view. Then, the management unit 201 selects selected viewpoint information including the viewpoint type and viewpoint ID included in the viewpoint information of the selected virtual viewpoint registered in the table information, the second position, the second direction, and the second angle of view. And outputs the selected viewpoint information to the output unit 202.

  The output unit 202 corresponds to the selected virtual viewpoint and each switching virtual viewpoint by outputting the viewpoint information and the selected viewpoint information output from the management unit 201 to the image generation apparatus 110 via the communication I / F 106. The image generation apparatus 110 is requested to generate a virtual viewpoint image.

  The receiving unit 203 communicates a virtual viewpoint image (a virtual viewpoint image corresponding to a selected virtual viewpoint, a virtual viewpoint image corresponding to a switching virtual viewpoint) generated by the image generation apparatus 110 in response to the request from the output unit 202. It is received from the image generation apparatus 110 via the I / F 106. In addition, it is assumed that the virtual viewpoint image output from the image generation device 110 is attached with the viewpoint ID of the virtual viewpoint corresponding to the virtual viewpoint image.

  The control unit 204 arranges and displays the virtual viewpoint images received by the receiving unit 203 on the display screen of the display unit 105 without overlapping each other. For example, as shown in FIG. 4A, in the display area 402 in the display area 401 of the display unit 105, a virtual viewpoint attached with a viewpoint ID corresponding to a selected virtual viewpoint among virtual viewpoint images received by the receiving unit 203. The image is displayed. In the display areas 403 to 406 in the display area 401, virtual viewpoint images to which the viewpoint ID of the switching virtual viewpoint is attached among the virtual viewpoint images received by the reception unit 203 are displayed. In the case of FIG. 5, the virtual viewpoint image generated by the image generation apparatus 110 for the selected virtual viewpoint of viewpoint ID = 408 is displayed in the display area 402. In addition, virtual viewpoint images generated by the image generation device 110 for the switching virtual viewpoints of the viewpoint IDs = 409 to 412 are displayed in the display areas 403 to 406, respectively.

  The display area 401 may be the entire area of the display screen of the display unit 105 or a partial area of the display screen of the display unit 105 (for example, an area in a window displayed on the display unit 105). It is good.

  The input unit 205 receives at least one of a second position, a second direction, and a second angle of view included in the selected viewpoint information according to the operation input (user operation) input by the user operating the operation unit 107. By changing one, the user can operate the selected virtual viewpoint. That is, even if the user operates the operation unit 107 to change the position, direction, or angle of view of the selected virtual viewpoint, this change is not reflected in the above table information, and the virtual viewpoint image corresponding to the selected virtual viewpoint is It will be reflected in the generation.

  Furthermore, when the user operates the operation unit 107 to select the selected virtual viewpoint, the input unit 205 instructs the management unit 201 to change the selected virtual viewpoint. The change processing of the selected virtual viewpoint by the management unit 201 is as described above.

  In addition, below, each function part (The management part 201, the output part 202, the receiving part 203, the control part 204, the input part 205) of FIG. 2 is demonstrated as a main body of a process. However, in practice, the CPU 101 executes a computer program for causing the CPU 101 to execute the functions of these functional units, whereby the functions of the functional units are realized. The same applies to FIGS. 6 and 10 described later. Note that part or all of the functional configuration shown in FIG. 2 may be realized by dedicated hardware. Dedicated hardware is, for example, an ASIC, an FPGA, a GPU, and the like. Further, part of the functional configuration shown in FIG. 2 may be included in one or more other devices different from the display control device 100.

  Further, although the image generating apparatus 110 and the display control apparatus 100 are separate apparatuses in FIG. 1, the image generating apparatus 110 and the display control apparatus 100 may be integrated. In that case, the output unit 202 and the receiving unit 203 can be omitted.

  Next, the process performed by the system according to the present embodiment will be described according to the flowchart of FIG. The processing according to the flowchart of FIG. 3 starts from the state where the application for setting the virtual view is activated, and is targeted until the application for setting the virtual view ends. The processing in steps S302 to S310 in FIG. 3 is looped once per frame of the virtual viewpoint image displayed on the setting screen of the virtual viewpoint (for example, the display area 401 in FIG. 4A). For example, when generating a virtual viewpoint image at 60 fps (frames / second), this loop will be repeated 60 times a second.

  In step S301, the management unit 201 reads out the viewpoint information of each switching virtual viewpoint registered in the above-mentioned table information, and selects the selected viewpoint information from the viewpoint information of the selected virtual viewpoint registered in the above-mentioned table information. Generate Then, the management unit 201 outputs each of the viewpoint information of each switching virtual viewpoint and the selected viewpoint information of the selected virtual viewpoint to the output unit 202.

  In step S302, the output unit 202 determines whether the viewpoint information of all the switching virtual viewpoints output from the management unit 201 has been transmitted to the image generation apparatus 110. As a result of the determination, if there is any viewpoint information that has not been transmitted to the image generation device 110 among the viewpoint information of each switching virtual viewpoint output from the management unit 201, the process proceeds to step S303. . On the other hand, as a result of the determination, if the viewpoint information of all the switching virtual viewpoints output from the management unit 201 has been transmitted to the image generation apparatus 110, the process proceeds to step S305.

  In step S303, the output unit 202 outputs untransmitted viewpoint information among the viewpoint information of the switching virtual viewpoints output from the management unit 201 to the image generation apparatus 110 via the communication I / F 106. The image generation device 110 includes a position, a direction, and an angle of view included in the viewpoint information output from the output unit 202, and an image captured by one or more of the imaging devices 109-1 to 109-n, To generate a virtual viewpoint image based on Then, the image generation device 110 attaches a viewpoint ID included in the viewpoint information to the generated virtual viewpoint image, and transmits the virtual viewpoint image to the display control device 100.

  In step S304, the reception unit 203 receives the virtual viewpoint image transmitted from the image generation device 110 via the communication I / F 106. Then, the control unit 204 displays the received virtual viewpoint image in a display area (any of the display areas 403 to 406) corresponding to the viewpoint ID attached to the virtual viewpoint image.

  In step S305, when the user operates the operation unit 107 and performs an operation input for changing the position / direction / angle of view of the selected virtual viewpoint, the second position / second information included in the selected viewpoint information is input. A change according to the operation input is made for the two directions / the second angle of view. For example, when the user performs an operation input for changing the position of the selected virtual viewpoint by operating the operation unit 107, the input unit 205 responds to the operation input for the second position included in the selected viewpoint information. To change. When the user performs an operation input for changing the direction of the selected virtual viewpoint by operating the operation unit 107, the input unit 205 responds to the operation input in the second direction included in the selected viewpoint information. To change. When the user performs an operation input for changing the angle of view of the selected virtual viewpoint by operating the operation unit 107, the input unit 205 performs an operation input of the second angle of view included in the selected viewpoint information. Change according to.

  In step S306, the output unit 202 transmits the selected viewpoint information to the image generation apparatus 110 via the communication I / F 106, and generates a virtual viewpoint image corresponding to the selected virtual viewpoint based on the selected viewpoint information. Are requested to the image generation apparatus 110. In response to the request, the image generation device 110 generates a virtual viewpoint image corresponding to the selected virtual viewpoint in the same manner as step S303 above, and a viewpoint included in the selected viewpoint information in the generated virtual viewpoint image. An ID is attached and transmitted to the display control apparatus 100.

  In step S307, the reception unit 203 receives the virtual viewpoint image transmitted from the image generation apparatus 110 in step S306 via the communication I / F 106. Then, the control unit 204 displays the received virtual viewpoint image in the display area (display area 402) corresponding to the viewpoint ID attached to the virtual viewpoint image.

  In step S308, the input unit 205 determines whether the user has input a selection instruction of a selected virtual viewpoint by operating the operation unit 107. For example, it is assumed that the user operates the operation unit 107 to specify the display area 405 among the display areas 403 to 406 in FIG. 4A. At this time, the input unit 205 determines that there is an input of an operation instruction to select a virtual viewpoint corresponding to the virtual viewpoint image displayed in the display area 405 as a selected virtual viewpoint. Note that the selection operation method of the selection virtual viewpoint is not limited to a specific selection operation method.

  As a result of this determination, when the selection instruction of the selected virtual viewpoint is input, the process proceeds to step S309, and when the selection instruction of the selected virtual viewpoint is not input, the process proceeds to step S310.

  In step S309, the management unit 201 selects “the selected virtual view type managed in the table information about the switching virtual view corresponding to the virtual view virtual selected on the display area 401 by the user operating the operation unit 107. Change to "viewpoint". Then, the management unit 201 changes the viewpoint type managed in the table information for the virtual viewpoint image displayed in the display area 402 to the “switching virtual viewpoint”. Then, the management unit 201 generates a second position obtained by copying the position managed in the table information for the virtual viewpoint whose viewpoint type is changed to “selected virtual viewpoint”. Further, the management unit 201 generates a second direction obtained by copying the direction managed in the table information with respect to the virtual viewpoint whose viewpoint type has been changed to “selected virtual viewpoint”. Further, the management unit 201 generates a second angle of view obtained by copying the angle of view managed in the table information for the virtual viewpoint whose viewpoint type has been changed to “selected virtual viewpoint”. That is, each of the second position, the second direction, and the second angle of view always represents the latest position, direction, and angle of view of the currently selected virtual viewpoint. Then, the management unit 201 determines the viewpoint type “selected viewpoint information”, the second position, the second direction, and the second angle of view updated in step S309, and the virtual viewpoint whose viewpoint type is changed to “selected virtual viewpoint”. The selected viewpoint information including the viewpoint ID managed by the table information is generated. The management unit 201 may set the position / direction / angle of view corresponding to the viewpoint ID set as the selected virtual viewpoint to be the same as the value of the switching virtual viewpoint specified by the switching instruction.

  In this manner, the user (the operator of the virtual viewpoint) operates the operation unit 107 to obtain a desired virtual viewpoint image while viewing the virtual viewpoint image displayed in the display area 402. The angle of view can be manipulated. Also, the user can search for a desired viewpoint to obtain a desired virtual viewpoint image by referring to the display areas 403 to 406. Then, when a virtual viewpoint image of a desired viewpoint is found, the virtual viewpoint image is displayed in the display area 402 by selecting and designating the virtual viewpoint image, and the virtual viewpoint can be manipulated. However, as shown in FIG. 4A, it is desirable that the display areas 402 to 406 be displayed without overlapping each other, and it is desirable that they can be viewed at one time.

  In step S310, the input unit 205 determines whether the end instruction of the virtual viewpoint setting application has been input by the user operating the operation unit 107. As a result of this determination, when an instruction to end the virtual view setting application is input, the processing according to the flowchart in FIG. 3 is ended, and the instruction to end the virtual view setting application is not input. The process proceeds to step S302.

  An arrangement example of virtual viewpoints is shown in FIG. As shown in FIG. 4B, virtual viewpoints 408 to 413 are set so that the field 407 can be observed from various directions at various positions in the field 407 of the soccer, and the positions and directions of the respective virtual viewpoints are set. The angle of view is managed as shown in FIG. For example, the position of the virtual viewpoint is a three-dimensional position in the coordinate system defined for the field 407, the direction of the virtual viewpoint is pan, tilt, and roll values, and the angle of view of the virtual viewpoint is the horizontal of the virtual viewpoint. Let it be the value of the angle of view. The display control apparatus 100 may display the screen shown in FIG. 4 (b) together with the screen shown in FIG. 4 (a). The display control apparatus 100 displays an icon indicating the position of each virtual viewpoint (an icon indicating a camera in the example shown in FIG. 4B) in the overhead view of the field 407 which is an imaging target. Further, the display control apparatus 100 may distinguishably display the position of the selected virtual viewpoint and the position of the switching virtual viewpoint in a distinguishable manner. For example, the display control apparatus 100 may display the position of the selected virtual viewpoint and the position of the switching virtual viewpoint in different colors or icons. In addition, the display control apparatus 100 may associate and display the viewpoint ID with the icon indicating the position of the virtual viewpoint. Thus, by displaying the arrangement of the virtual viewpoints shown in FIG. 4B, the positional relationship between the selected virtual viewpoint and the switching virtual viewpoint can be easily grasped, and a desired virtual viewpoint image can be obtained. You can look for the desired viewpoint.

  In the table information in FIG. 5, the viewpoint type corresponding to the viewpoint ID = 413 is “switching virtual viewpoint (non-display)”, which indicates that it is not a display target. The control unit 204 does not display the virtual viewpoint image corresponding to this viewpoint type. In addition, a virtual viewpoint image corresponding to this viewpoint type may not be generated.

  As described above, according to the present embodiment, the virtual viewpoint image corresponding to each switching virtual viewpoint is displayed as reference information to be referred to in order to select the next selected virtual viewpoint. Therefore, for example, it is possible to intuitively grasp which virtual viewpoint should be selected next, rather than displaying the captured image by each imaging device as reference information.

  Note that the virtual viewpoint image (virtual viewpoint image corresponding to the selected virtual viewpoint) displayed in the display area 402 may be stored in the HDD 104 of the display control apparatus 100, for example, via the communication I / F 106. It may be sent to an external device. For example, in the latter case, the virtual viewpoint image (virtual viewpoint image corresponding to the selected virtual viewpoint) displayed in the display area 402 may be broadcasted for broadcast via the communication I / F 106.

<Modification>
The operator of the virtual viewpoint only needs to be able to grasp the condition of the target being imaged by the imaging device, and thus may generate a simple virtual viewpoint image as a virtual viewpoint image, for example. 3DCG (three-dimensional computer graphic) is used for the simple virtual viewpoint image. First, the reception unit 203 acquires, from the image generation device 110, the position of the foreground (for example, the player and the ball) in the captured image. In the process of generating a virtual viewpoint image, processing is usually performed to separate the foreground and the background. In the process, it is common to find the position of the foreground. Therefore, the display control device 100 can easily acquire the position of the foreground from the image generation device 110. Next, the display control device 100 arranges the three-dimensional CG model of the player and the ball prepared in advance at the position acquired from the image generation device 110. In addition, three-dimensional CG models of the stadium field and the audience seat as background shall be prepared separately. When these three-dimensional models are input to the 3DCG rendering engine of the display control apparatus 100, it is possible to generate a simple virtual viewpoint image seen from a virtual viewpoint. Although it can not be denied that it is inferior to a normal virtual viewpoint image in terms of expression ability, the operator has the merits of being able to grasp, for example, the playing situation and reducing computer resources. Also, for example, the virtual viewpoint images displayed in the display area 402 and the virtual viewpoint images corresponding to the switching virtual viewpoints displayed in the display areas 403 to 406 may be displayed at different frame rates.

  As described above, in the present embodiment, a virtual viewpoint image corresponding to the selected virtual viewpoint selected by the user as an operation target among a plurality of virtual viewpoints and a virtual viewpoint image corresponding to a virtual viewpoint other than the selected virtual viewpoint among the plurality of virtual viewpoints And displays the screen arranged without overlapping each other. As a result, since it is possible to know what appears in the virtual viewpoint image after switching before the operator switches to a virtual viewpoint, it is possible to appropriately select the virtual viewpoint.

Second Embodiment
In each of the following embodiments and modifications, differences from the first embodiment will be described, and it is assumed that the second embodiment is the same as the first embodiment unless otherwise specified. In the first embodiment, by specifying any one of the displayed virtual viewpoint images, a virtual viewpoint corresponding to the specified one virtual viewpoint image is selected as a selected virtual viewpoint. However, the selection method of the selection virtual viewpoint is not limited to the selection method described in the first embodiment. In this embodiment, a selection method different from the first embodiment will be described as the selection method of the selection virtual viewpoint.

  An example of the functional configuration of the display control apparatus 100 according to the present embodiment will be described using the block diagram of FIG. The same reference numerals as in FIG. 2 denote the same functional units in FIG. 6, and a description thereof will be omitted. The processing unit 601 assigns a switching virtual viewpoint (a switching virtual viewpoint corresponding to each of the display areas 403 to 406 in the case of FIG. 4A) to any of the plurality of selection buttons of the operation unit 107.

  Then, the input unit 205 determines that the switching virtual viewpoint corresponding to the selection button pressed by the user is selected as the selected virtual viewpoint. The management unit 201 changes the currently selected virtual viewpoint in the table information to the switching virtual viewpoint, and changes the switching virtual viewpoint corresponding to the selection button pressed by the user to the selected virtual viewpoint.

  Next, the process performed by the system according to the present embodiment will be described according to the flowchart of FIG. The processing according to the flowchart of FIG. 7 starts from the state where the application for setting the virtual view is activated, and is targeted until the application for setting the virtual view is ended. In FIG. 7, the same processing steps as the processing steps shown in FIG. 3 are assigned the same step numbers, and the explanation of the processing steps is omitted.

  In step S701, the processing unit 601 selects any of the plurality of selection buttons of the operation unit 107 that the switching virtual viewpoint (switching virtual viewpoints corresponding to the display areas 403 to 406 in the case of FIG. 4A). Assign to

  An example of an input device applicable as the operation unit 107 is shown in FIG. Selection buttons 803 to 806 are provided in the input device 801 shown in FIG. The processing unit 601 assigns, for example, display regions 403 to 406 (that is, switching virtual viewpoints corresponding to the display regions 403 to 406) to the selection buttons 803 to 806, respectively. Thus, when the user presses the selection button 803, the input unit 205 determines that the virtual viewpoint image in the display area 403 is selected (a virtual viewpoint corresponding to the virtual viewpoint image is selected). Can. In addition, when the user presses the selection button 804, the input unit 205 may determine that the virtual viewpoint image in the display area 404 is selected (a virtual viewpoint corresponding to the virtual viewpoint image is selected). it can. When the user presses the selection button 805, the input unit 205 can determine that the virtual viewpoint image in the display area 405 is selected (a virtual viewpoint corresponding to the virtual viewpoint image is selected). When the user presses the selection button 806, the input unit 205 can determine that the virtual viewpoint image of the display area 406 is selected (a virtual viewpoint corresponding to the virtual viewpoint image is selected).

  Thus, when the display area and the selection button are both arranged side by side, the left display area is assigned to the left selection button, the second display area from the left is assigned to the second selection button from the left, and so on. Assign display areas to the selection buttons according to the order of arrangement. Also, when the display area and the selection button are both arranged vertically, the upper display area is assigned to the upper selection button, the second display area from the upper end is assigned to the second selection button from the upper end, and so on. Assign display areas to the selection buttons according to the order of arrangement. Note that which display area is assigned to which selection button is not limited to adopting a specific assignment method.

  When the number of switching virtual views is larger than the number of selection buttons on the input device, for example, switching virtual views may be allocated as long as they are allocated from the left. As a method of managing which switching button is assigned to which selection button, for example, in the table information, the viewpoint information of each switching virtual viewpoint includes identification information of the selection button assigned to the switching virtual viewpoint. You may do so. However, the management method is not limited to a specific management method.

  However, in step S308, the input unit 205 determines that the switching virtual viewpoint corresponding to the selection button pressed by the user is selected as the selected virtual viewpoint (a selection instruction is input).

  Further, the input device 801 is provided with a joystick 802 operated by the user in order to move the selected virtual viewpoint. The user can move the position, the direction, and the like of the viewpoint by operating the joystick 802. By the operation of the joystick 802 by the user, the component corresponding to the operation input among the second position, the second direction, and the second angle of view included in the selected viewpoint information is changed.

  As described above, according to the present embodiment, by assigning the switching virtual viewpoint to the selection button on the input device, the operator of the virtual viewpoint operates the virtual viewpoint without holding the mouse or the screen on the screen. A selection of virtual viewpoints can also be made on the input device. This makes it possible to quickly select a virtual viewpoint.

Third Embodiment
In this embodiment, the editing operation on the table information is enabled. In the present embodiment, the configuration of a system according to the second embodiment is used. The process performed by the system according to the present embodiment will be described according to the flowchart of FIG. The processing according to the flowchart of FIG. 9 starts from the state where the application for setting the virtual view is activated, and is targeted until the application for setting the virtual view is ended. In FIG. 9, the same step numbers are assigned to the same processing steps as the processing steps shown in FIGS. 3 and 7, and the description of the processing steps is omitted.

  In step S901, the input unit 205 determines whether or not the user operates the operation unit 107 to input an instruction (registration instruction) to register the position, direction, and angle of view of the current selected virtual viewpoint in the table information. to decide. The input method of the registration instruction is not limited to a specific input method. For example, the user may input a registration instruction by instructing the registration button displayed on the display screen of the display unit 105 by operating the operation unit 107, or the user may press a specific button on the operation unit 107. You may input a registration instruction by doing. If it is determined that the registration instruction is input, the process proceeds to step S902. If the registration instruction is not input, the process proceeds to step S903.

  In step S902, the management unit 201 issues a new viewpoint ID, and the issued viewpoint ID, the viewpoint type “switching virtual viewpoint”, the second position included in the current selected viewpoint information, and the second New viewpoint information including the direction and the second angle of view is generated and registered in the table information. Further, the management unit 201 registers in the table information as to where in the display areas 403 to 406 the virtual viewpoint image corresponding to the registered virtual viewpoint is to be displayed or not displayed, or is separate from the table information. Register as information.

  In step S 903, when the user operates operation unit 107, an instruction (sorting instruction) to change the arrangement order of virtual viewpoint images displayed in each of display regions 403 to 406 is input by the user operating operation unit 107. Decide whether or not If it is determined that the reordering instruction is input, the process proceeds to step S904. If the reordering instruction is not input, the process proceeds to step S905.

  In step S904, the control unit 204 changes the arrangement order of the virtual viewpoint images displayed in each of the display areas 403 to 406 according to the operation input input by the user operating the operation unit 107. The operation performed by the user for this rearrangement is not limited to a specific operation. For example, when the user operates the operation unit 107 and instructs the change button displayed on the display screen of the display unit 105, the input unit 205 instructs the virtual viewpoint image displayed in the display areas 403 to 406 to be operated. Accept When the user presses a specific button on the operation unit 107, the input unit 205 may receive an operation instruction for the virtual viewpoint image displayed in the display areas 403 to 406.

  Then, for example, it is assumed that the user operates the operation unit 107 and drags the virtual viewpoint image in the display area 404 into the display area 406. At this time, the control unit 204 moves the virtual viewpoint images displayed in the display areas 405 and 406 to the display areas 404 and 405, respectively. Also, the control unit 204 moves the virtual viewpoint image (virtual viewpoint image displayed in the display area 404) dragged into the display area 406 into the display area 406. At this time, the management unit 201 changes the association between the display area and the switching virtual viewpoint.

  Further, for example, it is assumed that the user operates the operation unit 107 to drag the virtual viewpoint image in the display area 404 and move it outside the display area 401. At this time, the control unit 204 moves the virtual viewpoint images displayed in the display areas 405 and 406 to the display areas 404 and 405, respectively. The management unit 201 hides the viewpoint type of the virtual viewpoint image dragged and moved out of the display area 401, and instead, the viewpoint of the switching virtual viewpoint of the viewpoint ID = 413 where the viewpoint type is not displayed. Remove hide from type. Then, a virtual viewpoint image corresponding to the switching virtual viewpoint of viewpoint ID = 413 is generated as described above and displayed in the display area 406. Note that there are various operations for the rearrangement operation of the virtual viewpoint images corresponding to the switching virtual viewpoints, and it is not limited to a specific operation.

  In step S905, when the user operates the operation unit 107, the input unit 205 receives an instruction (deletion instruction) to delete any of the virtual viewpoint images displayed in the display areas 403 to 406. Determine if it was. As a result of this determination, when the deletion instruction is input, the process proceeds to step S906, and when the deletion instruction is not input, the process proceeds to step S907.

  In step S906, the management unit 201 deletes, from the table information, the viewpoint information of the virtual viewpoint for which the deletion instruction has been made, and the control unit 204 displays a virtual viewpoint image corresponding to the virtual viewpoint for which the deletion instruction is made. Delete from the display screen of the unit 105. The operation performed by the user for this deletion is not limited to a specific operation. For example, when the user operates the operation unit 107 and instructs the delete button displayed on the display screen of the display unit 105, the input unit 205 instructs the virtual viewpoint image displayed in the display areas 403 to 406 to be operated. Accept. When the user presses a specific button on the operation unit 107, the input unit 205 may receive an operation instruction for the virtual viewpoint image displayed in the display areas 403 to 406.

  Then, for example, when the user operates the operation unit 107 and designates the display area 404, the management unit 201 makes table information of the viewpoint information of the switching virtual viewpoint corresponding to the virtual viewpoint image displayed in the display region 404 from the table information. Remove from Then, the control unit 204 deletes the virtual viewpoint image from the display area 404.

  In step S 907, the input unit 205 instructs the user to operate the operation unit 107 to switch the viewpoint information of the switching virtual viewpoint in the table information to the viewpoint information of another switching virtual viewpoint (viewpoint switching instruction). It is determined whether or not is input. For example, when the viewpoint set switching button displayed on the display screen of the display unit 105 is designated or the specific button in the operation unit 107 is pressed, the input unit 205 receives a viewpoint set switching instruction. Judge. If it is determined that the viewpoint set switching instruction is input, the process proceeds to step S908. If the viewpoint set switching instruction is not input, the process proceeds to step S310.

  In step S 908, the management unit 201 replaces the viewpoint information having the viewpoint type “switching virtual viewpoint” in the table information with any of the virtual viewpoint sets created in advance and registered in the HDD 104. The virtual viewpoint set is viewpoint information of one or more virtual viewpoints, as described later. The number of virtual viewpoint sets registered in the HDD 104 may be one or more.

  When one virtual viewpoint set is registered in the HDD 104, the management unit 201 replaces the viewpoint information of the “virtual viewpoint for switching” of the viewpoint type in the table information with the one virtual viewpoint set.

  When a plurality of virtual viewpoint sets are registered in the HDD 104, the management unit 201 sets the viewpoint information of the viewpoint type “switching virtual viewpoint” in the table information to any one of the plurality of virtual viewpoint sets. replace. The method for selecting one virtual viewpoint set to be used for the above replacement from a plurality of virtual viewpoint sets is not limited to a specific method. For example, the control unit 204 displays a list in which the contents and file names of the plurality of virtual viewpoint sets are described on the display screen of the display unit 105. When the user operates the operation unit 107 to select one of the plurality of virtual viewpoint sets by looking at the display screen, the management unit 201 selects, in the table information, viewpoint information of “switching virtual viewpoint” as the viewpoint type. , Replace with the selected virtual viewpoint set.

  The configuration in which a plurality of virtual viewpoint sets are provided and one of them is selected and used is, for example, assuming that the operator of the virtual viewpoints uses properly depending on the playing situation. For example, it is useful to prepare in advance a virtual viewpoint set for corner kick, a virtual viewpoint set for kickoff, a virtual viewpoint set for penalty kick, and the like. Furthermore, in the case of a virtual viewpoint set for a penalty kick, a virtual viewpoint for viewing the direction of the penalty spot from the goal side, a virtual viewpoint for the opposite direction, a virtual viewpoint for viewing the goal side over the penalty spot, and a virtual viewpoint for viewing the bench It is convenient to prepare as a set. When the table information is changed in step S 908, the management unit 201 reads out the viewpoint information of each switching virtual viewpoint registered in the changed table information, and outputs the viewpoint information of each switching virtual viewpoint. Output to the unit 202.

  An example of a virtual viewpoint set is shown in FIG. FIG. 14A shows a configuration example of each of the virtual viewpoint set 1401, the virtual viewpoint set 1402, and the virtual viewpoint set 1403. In the virtual viewpoint set 1401, viewpoint information of four switching virtual viewpoints is registered. In the virtual viewpoint set 1402, viewpoint information of three switching virtual viewpoints is registered. In the virtual viewpoint set 1403, viewpoint information of five switching virtual viewpoints is registered.

  Here, when the virtual viewpoint set 1401 is selected in the above step S 908, table information after replacing the viewpoint information of the viewpoint type “switching virtual viewpoint” in the table information of FIG. 5 with the virtual viewpoint set 1401. An example of the configuration of is shown in FIG. In the table information of FIG. 14B, the number of switching virtual viewpoints is reduced to four from the table information of FIG. 5, and the position, direction, and angle of view are replaced with the values of the virtual viewpoint set 1401, and the viewpoint ID is newly added. Are allocated (413-416). In addition, since the selection virtual viewpoint does not change by this operation, the same value as FIG. 5 is maintained.

Fourth Embodiment
In the first to third embodiments, the frame rate of the virtual viewpoint image displayed in the display area 402 and the frame rate of the virtual viewpoint image displayed in the display areas 403 to 406 are the same. However, the frame rates of virtual viewpoint images of all virtual viewpoints need not be the same. In order for the image generation device 110 to generate a virtual viewpoint image, reading of a large amount of data and data processing are required. Therefore, as described in the first to third embodiments, generating virtual viewpoint images corresponding to all of the selected virtual viewpoint and the switching virtual viewpoint at 60 fps (frames / second) requires a huge amount of computer resources, The cost will be high. Thus, in the present embodiment, frame rate control in consideration of the image generation capability of the image generation device 110 will be described.

  An example of the functional configuration of the display control apparatus 100 according to the present embodiment will be described using the block diagram of FIG. The same reference numerals as in FIGS. 2 and 6 denote the same functional units in FIG. 10, and a description of the functional units will be omitted.

  The processing unit 1001 sets the frame rate of the virtual viewpoint image corresponding to the selected virtual viewpoint and the frame rate of the virtual viewpoint image corresponding to the switching virtual viewpoint in accordance with the image generation capability of the image generation apparatus 110.

  The process performed by the system according to the present embodiment will be described according to the flowchart of FIG. The processing according to the flowchart in FIG. 11 starts from the state where the application for setting the virtual view is activated, and is targeted until the application for setting the virtual view ends. In FIG. 11, the same step numbers are assigned to the same processing steps as the processing steps shown in FIGS. 3, 7 and 9, and the description of the processing steps is omitted.

  In step S1101, the processing unit 1001 sets, for each virtual viewpoint registered in the table information, a frame rate of a virtual viewpoint image corresponding to the virtual viewpoint. Note that in step S303 according to the present embodiment, the output unit 202 communicates, among the viewpoint information of the switching virtual viewpoints output from the management unit 201, the viewpoint information corresponding to the virtual viewpoint image to be generated in the current frame. The image is output to the image generation apparatus 110 via the / F 106. In step S306 according to the present embodiment, if the current frame is a frame that generates a virtual viewpoint image corresponding to the selected virtual viewpoint, the output unit 202 sends the selected viewpoint information to the image generation device 110 via the communication I / F 106. Send to.

  The process in step S1101 described above will be described using a specific example. Here, it is assumed that the image generation capability of the image generation apparatus 110 is 120 fps (an image of 120 frames can be generated per second), and as shown in FIG. 5, one selected virtual viewpoint and four switching virtual viewpoints exist. Think about the case.

  The virtual viewpoint image corresponding to the selected virtual viewpoint is an image indispensable to the operation of the selected virtual viewpoint, and the result of specifying the selected virtual viewpoint needs to be displayed as a virtual viewpoint image without delay. It has been found that the operation of the selected virtual viewpoint becomes difficult as the delay time from the specification of the selected virtual viewpoint to the display of the virtual viewpoint image increases. Therefore, 60 fps is allocated to the selected virtual viewpoint, and the remaining 60 fps are divided into four switching virtual viewpoints. That is, as shown in FIG. 12A, virtual viewpoint images corresponding to the selected virtual viewpoint are generated at 60 fps and displayed in the display area 402, and virtual viewpoint images corresponding to each switching virtual viewpoint are 15 fps. It is generated and displayed in the display areas 403 to 406. Although the display of the virtual viewpoint image corresponding to the switching virtual viewpoint is not smooth somewhat, the frame rate is sufficient to grasp what is reflected in the virtual viewpoint. In the present embodiment, when the frame rate for each virtual viewpoint is determined in step S1101, the frame rate for each virtual viewpoint is registered in the table information managed by the management unit 201, as shown in FIG.

  At this time, the virtual viewpoint image of each frame generated by the image generation apparatus 110 in step S303 and step S306 will be described using FIG. Since the image generation capability of the image generation apparatus 110 is 120 fps and the frame rate of the virtual viewpoint image corresponding to the selected virtual viewpoint is 60 fps, the virtual viewpoint image corresponding to the selected virtual viewpoint is once every two frames. It is generated and displayed. In FIG. 13, since the virtual viewpoint image corresponding to the selected virtual viewpoint is generated and displayed in the first frame, it is generated and displayed in the (2 × f + 1) (f is an integer of 1 or more) frame thereafter become.

  Further, since the image generation capability of the image generation apparatus 110 is 120 fps and the frame rate of the virtual viewpoint image corresponding to each of the four switching virtual viewpoints is 15 fps, the virtual viewpoint image corresponding to each switching virtual viewpoint is 8 It is generated and displayed at a rate of once per frame. For example, in FIG. 13, since the virtual viewpoint image corresponding to the switching virtual viewpoint 1 is generated and displayed in the second frame, it is generated and displayed in the (8 × f + 2) frame thereafter. Similarly, the switching virtual viewpoint 2, the switching virtual viewpoint 3, and the switching virtual viewpoint 4 are displayed in the fourth, sixth, and eighth frames, respectively, and thereafter, the (8 × f + 4) th frame, the (8 × f + 6) frame It is generated and displayed in the eye, (8 × f + 8) frame.

  Therefore, the output unit 202 outputs viewpoint information of a virtual viewpoint corresponding to a virtual viewpoint image to be generated and displayed in the current frame to the image generation apparatus 110, and the image generation apparatus 110 receives the viewpoint received from the output unit 202. Generate a virtual viewpoint image corresponding to the information. By evenly distributing the timing of generating virtual viewpoint images, smoother image display becomes possible.

  The numerical values listed here are all examples used to explain the specific example, and the above operation is not limited to these numerical values. That is, they should be set appropriately in accordance with the capabilities of the system and the requirements of a broadcast system that broadcasts virtual viewpoint images.

  As described above, according to the present embodiment, by appropriately distributing the frame rate at which the virtual viewpoint image is generated between the selected virtual viewpoint and the switching virtual viewpoint, it is possible to operate and play virtual viewpoints without increasing computer resources. It is possible to simultaneously realize the selection of virtual viewpoints according to the situation.

Fifth Embodiment
In the first to fourth embodiments, the virtual viewpoint image corresponding to the selected virtual viewpoint and the virtual viewpoint image corresponding to the switching virtual viewpoint are displayed on one display screen, but the present invention is not limited to this. Absent. For example, the virtual viewpoint image corresponding to the selected virtual viewpoint and the virtual viewpoint image corresponding to the switching virtual viewpoint may be displayed on separate display devices.

  In the third and fourth embodiments, in the processing loop of generation and display of virtual viewpoint images, processing of registration, rearrangement, deletion, switching of viewpoint sets, etc. of virtual viewpoints is performed. It is not limited to For example, processing of registration, rearrangement, deletion, switching of viewpoint set, and the like of virtual viewpoints may be performed by a separate thread from the processing loop of generation and display of virtual viewpoint images.

  In addition, you may use combining suitably one part or all of each embodiment and each modification which were demonstrated above. In addition, part or all of each embodiment and each modification described above may be selectively used.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

201: Management unit 202: Output unit 203: Reception unit 204: Control unit 205: Input unit

Claims (16)

Acquisition means for acquiring a plurality of virtual viewpoint images according to the positions and orientations of the plurality of virtual viewpoints;
A display control unit configured to display the plurality of virtual viewpoint images acquired by the acquisition unit on a screen used for an operation related to the position and orientation of the virtual viewpoint;
A virtual viewpoint image corresponding to the position and the orientation of the virtual viewpoint determined based on the operation is generated based on a plurality of captured images obtained by capturing a plurality of cameras for capturing a shooting target area, A display control apparatus characterized in that it is transmitted.   The display control device according to claim 1, wherein the virtual viewpoint image transmitted to the other device is transmitted by broadcast. It has registration operation reception means for receiving a user operation for newly registering a virtual viewpoint,
The display control device according to claim 1, wherein the display control unit adds a virtual viewpoint image to be displayed on the screen in response to reception of the user operation by the registration operation reception unit. A switching operation receiving unit configured to receive a user operation for switching a virtual viewpoint image to be displayed on the screen;
The display control unit is configured to display a first plurality of virtual viewpoint images corresponding to a first plurality of virtual viewpoints on the screen in response to reception of the user operation by the switching operation reception unit. The display control apparatus according to any one of claims 1 to 3, wherein a second plurality of virtual viewpoint images corresponding to a plurality of two virtual viewpoints are switched to the state of being displayed on the screen. . It has deletion operation reception means for receiving a user operation for deleting a virtual viewpoint,
5. The display control means according to any one of claims 1 to 4, wherein the number of virtual viewpoint images to be displayed on the screen is reduced according to the reception of the user operation by the deletion operation reception means. The display control apparatus as described in a term. And a rearranging operation receiving unit configured to receive a user operation for rearranging the virtual viewpoint images displayed on the screen;
The display control means rearranges the virtual viewpoint image displayed on the screen according to the reception of the user operation by the rearranging operation reception means. The display control device according to item 1.   The display control means displays a virtual viewpoint image to be transmitted to another device among the plurality of virtual viewpoint images more largely than the other virtual viewpoint images. The display control device according to any one of the above.   The display control means is characterized in that, among the plurality of virtual viewpoint images, a virtual viewpoint image transmitted to another device is displayed at a frame rate higher than that of the other virtual viewpoint images. The display control device according to any one of the above. Viewpoint operation receiving means for receiving a user operation related to the change of the position and the orientation of the virtual viewpoint;
Changing means for changing at least one of the position and the orientation of the virtual viewpoint corresponding to the virtual viewpoint image transmitted to the other device in accordance with the reception of the user operation by the viewpoint operation reception unit The display control device according to any one of claims 1 to 8, characterized in that: Viewpoint operation receiving means for receiving a user operation related to the change of the position and the orientation of the virtual viewpoint;
Transmission means for transmitting information indicating the position and orientation of the virtual viewpoint after the change to the generation device for generating the virtual viewpoint image according to the reception of the user operation by the viewpoint operation reception means; And
The display control device according to any one of claims 1 to 9, wherein the virtual viewpoint image generated by the generation device is transmitted to the other device. Selection operation receiving means for receiving a user operation for selecting a virtual viewpoint image from among the plurality of virtual viewpoint images;
Switching means for executing switching processing for switching a virtual viewpoint corresponding to the virtual viewpoint image transmitted to the other device in response to reception of the user operation by the selection operation receiving device. The display control apparatus according to any one of claims 1 to 10.   The display control means displays the virtual viewpoint image selected by the user operation accepted by the selection operation acceptance means at the display position of the virtual viewpoint image which has been selected until the user operation acceptance by the selection operation acceptance means The display control device according to claim 11, wherein the display content on the screen is changed so as to cause the display control. A display control method performed by the display control device;
An acquisition step of acquiring a plurality of virtual viewpoint images according to positions and orientations of the plurality of virtual viewpoints;
A display control step of displaying the plurality of virtual viewpoint images acquired in the acquisition step on a screen used for an operation related to the position and orientation of the virtual viewpoint;
A virtual viewpoint image corresponding to the position and the orientation of the virtual viewpoint determined based on the operation is generated based on a plurality of captured images obtained by capturing a plurality of cameras for capturing a shooting target area, A display control method characterized in that it is transmitted.   A changing step of changing at least one of the position and the orientation of the virtual viewpoint corresponding to the virtual viewpoint image to be transmitted to the other device according to reception of a predetermined user operation; The display control method according to 13.   A switching process for switching a virtual viewpoint corresponding to the virtual viewpoint image transmitted to the other device is executed in response to reception of a user operation for selecting a virtual viewpoint image from among the plurality of virtual viewpoint images. The display control method according to claim 13 or 14, further comprising a switching step.   The program for operating a computer as each means of the display control apparatus in any one of Claims 1-12.

Images