JP2019057291A - Control device, control method, and program - Google Patents

Abstract

To provide a control device capable of easily setting a virtual viewpoint of a three-dimensional image by means of a virtual camera.SOLUTION: A display control device 100 has a touch panel which integrally includes a display part 206 and an operation part 201. A viewpoint control part 204 is configured to perform the steps of: detecting an operation such as multi-touch, slide, pinch-in/pinch-out made on a display surface for displaying virtual viewpoint images by a user; controlling at least either one of the position and the direction of a virtual viewpoint relevant to the generation of a virtual viewpoint image responding to the user's operation; and changing the magnification of the image.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a virtual viewpoint control method.

  A virtual viewpoint video generation technique for generating a video of an arbitrary viewpoint from a plurality of shot videos obtained by shooting with a plurality of cameras having different viewpoints is known. As a virtual viewpoint video generation method (rendering method), a method of generating a virtual viewpoint video based on a predetermined virtual viewpoint movement path, a position and posture of a virtual viewpoint specified by a viewer, etc. A method for generating a virtual viewpoint video is known.

  According to the virtual viewpoint video generation technology, it is possible to view a highly interactive video. On the other hand, it is difficult for a device such as a tablet or a smartphone having a touch panel as a main interface to operate the viewpoint as desired. In Patent Document 1, a user selects one of each of a plurality of viewpoint coordinate data and a plurality of rotation start point data, and then sets a viewpoint by inputting a rotation angle and a movement amount of the viewpoint. Is described.

Japanese Patent Laying-Open No. 2015-187797

  The method of Patent Document 1 has many operation procedures for setting a virtual viewpoint. An object of the present invention is to make it easier to set a virtual viewpoint.

  In order to solve the above problems, a control device according to the present invention has the following configuration. That is, a control device for controlling the position and orientation of a virtual viewpoint related to a virtual viewpoint video generated using image data obtained by photographing a predetermined photographing target region from different directions, When a touch operation that separates the first touch position and the second touch position is performed on the display surface that displays the virtual viewpoint video, the direction of the virtual viewpoint and the attention coordinate based on the virtual viewpoint change. If the position of the virtual viewpoint changes without any change, and a touch operation is performed to move the first touch position and the second touch position in parallel on the display surface, the direction of the virtual viewpoint changes. The virtual viewpoint based on the change of the first and second touch positions on the display surface so that the position of the virtual viewpoint and the target coordinate changes without Determining means for determining a change amount of parameters, in accordance with the changed amount determined by the determining means, and a control means for controlling the virtual viewpoint.

  According to the present invention, setting of a virtual viewpoint can be performed more easily.

FIG. 2 is a block diagram showing a hardware configuration of a control device 100. FIG. 2 is a block diagram showing a functional configuration of a control device 100. FIG. 6 is a diagram illustrating an example of virtual viewpoint control in a one-finger slide operation according to the first embodiment. FIG. 6 is a diagram illustrating an example of virtual viewpoint control in a three-finger slide operation according to the first embodiment. FIG. 5 is a diagram illustrating an example of virtual viewpoint control in a two-finger slide operation according to the first embodiment. FIG. 6 is a diagram illustrating an example of virtual viewpoint control in a two-finger pinch-out operation according to the first embodiment. 3 is a flowchart showing a flow of processing in the control device 100 according to the first embodiment. The flowchart which shows the flow of the virtual viewpoint control according to user operation.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and all the combinations of features described in the present embodiment are not necessarily essential to the solution means of the present invention. In addition, about the same structure, the same code | symbol is attached | subjected and demonstrated.

<Embodiment 1>
Embodiment 1 demonstrates the example which controls the position and direction, etc. of a virtual viewpoint by a user operating a display screen (touch panel), and produces | generates the virtual viewpoint image | video according to it. In the present embodiment, “change the position of the virtual camera” and “change the position of the virtual viewpoint” are used as the same meaning. Further, “change the orientation of the virtual camera” and “change the orientation of the virtual viewpoint” are used as the same meaning.

  In this embodiment, a virtual viewpoint video is a video generated based on a plurality of captured images obtained by a plurality of cameras that capture a field (photographing target region) from different directions, and a virtual viewpoint (virtual camera) ), And the like. Further, the virtual viewpoint video of this embodiment may be video data in which each image frame is compressed by a predetermined moving image compression method, or a video in which each image frame is compressed by a predetermined still image compression method. It may be data or uncompressed video data.

  A system configuration example of the control device 100 in the present embodiment will be described with reference to FIG. In the figure, a CPU 101 executes a program stored in a ROM 103 and / or a hard disk drive (HDD) 105 using a RAM 102 as a work memory, and controls each component to be described later via a system bus 112. Thereby, various processes described later are executed. The HDD interface (I / F) 104 is an interface such as serial ATA (SATA) that connects the control device 100 to a secondary storage device such as the HDD 105 or an optical disk drive. The CPU 101 can read data from the HDD 105 and write data to the HDD 105 via the HDD interface (I / F) 104. Further, the CPU 101 expands the data stored in the HDD 105 in the RAM 102. Further, the CPU 101 can store various data on the RAM 102 obtained by executing the program in the HDD 105. The input interface (I / F) 106 connects an input device 107 such as a touch panel, a keyboard, a mouse, a digital camera, and a scanner for inputting one or a plurality of coordinates and the control device 100. The input interface (I / F) 106 is a serial bus interface such as USB or IEEE1394. The CPU 101 can read data from the input device 107 via the input I / F 106. An output interface (I / F) 108 is an image output interface such as DVI or HDMI (registered trademark) that connects an output device 109 such as a display and the control apparatus 100. The CPU 101 can display the virtual viewpoint video by sending data related to the virtual viewpoint video to the output device 109 via the output I / F 108. The network interface (I / F) 110 is a network card such as a LAN card that connects the control device 100 and the external server 111. The CPU 101 can read data from the external server 111 via the network I / F 110.

  In this embodiment, an example in which the input device 107 is the touch panel of the control apparatus 100 will be mainly described. That is, the control device 100 may be a smartphone or a tablet terminal. In this case, the input device 107 (touch panel) and the output device 109 (display screen) are integrated with the control device 100. Further, not all of the configurations shown in FIG. 1 are indispensable configurations. For example, when reproducing a virtual viewpoint video stored in the HDD 105, the external server 111 is not necessary. Conversely, when generating a virtual viewpoint video acquired from the external server 111, the HDD 105 is unnecessary. In addition, the control device 100 may include a plurality of CPUs 101. Further, one or a plurality of dedicated hardware or GPU (Graphics Processing Unit) different from the CPU 101 may be provided, and at least a part of the processing by the CPU 101 may be performed by the GPU or the dedicated hardware. Examples of dedicated hardware include ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), and DSP (Digital Signal Processor).

  In this embodiment, a method for controlling a virtual camera in accordance with a user's intention by a user operation on the touch panel will be described. The user operation in this embodiment includes at least a tap operation, a slide operation with one to three fingers, and a pinch-in / pinch-out operation.

  A user operation with one finger is easy to perform even for beginners, but it is difficult to input complicated information, and a user operation with three fingers is difficult to perform. Therefore, a simple virtual camera control process is assigned to a user operation (slide operation) with one finger and three fingers. Then, a complicated virtual camera control process is assigned to the operation of two fingers.

  A specific user operation and the operation of the virtual camera will be described with reference to FIGS. FIG. 3 shows a state in which the position and posture of the virtual camera change due to a single finger slide operation. When a slide operation is performed to the left with one finger on the image 301 before the user operation, the position and orientation of the virtual camera are changed, and the image 302 is displayed. At this time, the position and orientation of the virtual camera changes from the virtual viewpoint 305 to the virtual viewpoint 306 that wraps around to the right with the point 304 in the three-dimensional space drawn at the center of the overhead image 303 as the center. That is, when the number of coordinates on the display screen (on the display surface) specified simultaneously by the user operation is the first number (one), the viewpoint control unit 204 determines whether or not the virtual camera corresponds to the movement of the coordinates. Is turned around a predetermined attention coordinate. In other words, when the number of coordinates on the display surface simultaneously designated by the user operation is the first number (one), the viewpoint control unit 204 determines whether the virtual camera is moved according to the movement of the coordinates. Control is performed so as to move while paying attention to a predetermined attention coordinate.

  As shown in FIG. 3, when a single-finger slide operation is detected, the moving range of the virtual camera is limited to a circle 307 and the moving direction is limited to the horizontal direction. As a result, for example, even in a user who is unfamiliar with the touch operation or in a usage scene where the trajectory of the touch is likely to fluctuate, the virtual camera close to the user's intention can be moved. Therefore, it is possible to easily generate a virtual viewpoint video such as a bullet time without blurring.

  FIG. 4 shows a state in which the position of the virtual camera is changed by a three-finger slide operation. When a slide operation is performed to the right with three fingers on the image 401 before the user operation, the position of the virtual camera is changed and the image 402 is displayed. At this time, the position of the virtual camera moves from the virtual viewpoint 404 to the virtual viewpoint 405 as shown in the overhead image 403. That is, when the number of coordinates on the display screen simultaneously designated by the user operation is the third number (three), the viewpoint control unit 204 determines that the virtual camera is in the three-dimensional space according to the change in the coordinates. The position of the virtual camera is controlled so as to move in parallel.

  Note that “three coordinates are specified simultaneously” is not limited to the case where three fingers touch the display screen at the same time. For example, when the third finger touches the display screen while holding the state where two fingers are touching the display screen, it is determined that the three fingers are simultaneously touching the display screen. . As described above, even when the touch start timing of the display screen is different for each finger, a plurality of coordinates may be designated at the same time.

  In addition, the viewpoint control unit 204 of the present embodiment makes the movement amount of the finger on the display screen equal to the movement amount of the drawing position in the virtual viewpoint video of the three-dimensional point 406 specified according to the user operation. Move the virtual camera. In this way, by matching the amount of movement of the finger with the amount of movement of the three-dimensional point 406, the user can more intuitively control the virtual camera. However, the movement amount of the finger may be controlled to be different from the movement amount of the drawing position of the three-dimensional point 406 in the virtual viewpoint video. In the user operation with three fingers, the finger movement amount may be an average value of the movement amounts of the three fingers, a median value, or a representative value. May be used, or other values may be used. In the present embodiment, when a three-finger slide operation is performed, the position of the virtual camera is changed, but the attitude of the virtual camera is not changed. That is, when the number of coordinates on the display screen simultaneously designated by the user operation is the third number (three), the viewpoint control unit 204 changes the position of the virtual viewpoint according to the change in the coordinates. The orientation of the virtual viewpoint is not changed.

  FIG. 5 shows how the position and orientation of the virtual camera change due to a two-finger slide operation. When a slide operation is performed in the lower left direction with two fingers on the image 501 before the user operation, the position and orientation of the virtual camera are changed, and the image 502 is displayed. At this time, the position and orientation of the virtual camera change from the virtual viewpoint 505 to the virtual viewpoint 506 around the three-dimensional point 504 determined based on the position of the user's finger, as shown in the overhead view image 503. The moving range of the virtual camera is limited to the spherical surface 507. In other words, when a one-finger slide operation is performed, the position in the height direction of the virtual camera is not changed, but the horizontal position is changed, whereas when a two-finger slide operation is performed, Change both the height and horizontal position of the virtual camera. That is, when the number of coordinates on the display screen simultaneously designated by the user operation is the second number (two), the viewpoint control unit 204 changes the position of the virtual camera to the first according to the movement of the coordinates. Change to direction and second direction. As described above, the control device 100 according to the present embodiment performs more complex control of the virtual camera when the slide operation is performed with two fingers than when the slide operation is performed with one finger. To do. By performing a two-finger slide operation, a virtual viewpoint video in which an arbitrary object is viewed from an arbitrary direction can be generated.

  FIG. 6 shows how the position of the virtual camera changes due to a two-finger pinch-out operation. When a pinch-out operation is performed on the image 601 before the user operation, the position of the virtual camera is changed and the image 602 is displayed. At this time, the position of the virtual camera moves from the virtual viewpoint 604 to the virtual viewpoint 605 as shown in the overhead image 603. When a two-finger pinch-in operation is performed, the virtual viewpoint 605 moves in the direction of the virtual viewpoint 604. That is, when the number of coordinates on the display screen simultaneously designated by the user operation is the second number (two), the viewpoint control unit 204 responds to the visual line direction of the virtual camera according to the change in the coordinates. Move the virtual camera in the desired direction.

  By the pinch-in operation and the pinch-out operation, the position of the virtual camera moves back and forth as indicated by a dotted line 606. The subject (for example, an object such as a player) is displayed larger as the finger is spread, and the subject can be displayed smaller as the finger is narrowed, enabling intuitive operation. In this way, the virtual camera can be controlled with a high degree of freedom by allowing the virtual camera to move back and forth and rotate in response to a two-finger user operation. Note that, instead of changing the position of the virtual camera according to the pinch-in operation and the pinch-out operation, a parameter related to the zoom value of the virtual camera may be changed.

  Further, in the present embodiment, an example in which the first number is “1”, the second number is “2”, and the third number is “3” will be mainly described, but the present invention is not limited thereto. For example, the first number may be “3”, the second number may be “2”, and the third number may be “1”, or the first number may be “1” and the second number may be “1”. 2 ”and the third number may be“ 4 ”.

  The flow of processing performed by the control device 100 of this embodiment will be described with reference to FIGS. 2 and 7. FIG. 2 is a block diagram illustrating a functional configuration of the control device 100 according to the present embodiment. The CPU 101 reads out a program stored in the ROM 103 and / or the HDD 104 and executes the RAM 102 as a work area, thereby playing a role of each functional block in the control device 100 shown in FIG. The operation unit 201 and the display unit 206 in FIG. 2 correspond to the input device 107 and the output device 109 in FIG. Further, the CPU 101 does not have to play the role of all the functional blocks in the control device 100, and a dedicated processing circuit corresponding to each functional block may be provided.

  FIG. 7 is a flowchart showing the flow of processing performed by the control device 100 of the present embodiment. Each process described with reference to FIG. 7 is realized by the CPU 101 included in the control device 100 reading a program stored in the ROM 103 and / or the HDD 104 and executing the RAM 102 as a work area.

  In step S <b> 701, the acquisition data control unit 202 stands by until a tap operation on the operation unit 201 is detected. The tap operation is an operation of touching the display screen with a finger for a short time. An arbitrary value can be set as the threshold of the contact time used for the determination of the tap operation. When a tap operation is detected, if the playback state of the current virtual viewpoint video on the display unit 206 is paused, the playback state is changed. On the other hand, if the virtual viewpoint video is in the playback state when the tap operation is detected, the acquisition data control unit 202 changes the virtual viewpoint video to the paused state. Thus, the acquisition data control unit 202 changes the playback state of the virtual viewpoint video in response to a tap operation on the display screen, so that the user can switch the playback state by an intuitive operation. However, S701 is not an essential process.

  In step S <b> 702, the viewpoint control unit 204 outputs camera parameters related to the position and orientation of the virtual camera to the drawing unit 205 based on the detection result of the user operation on the operation unit 201. Details of S702 will be described later with reference to FIG.

  In step S <b> 703, the data acquisition unit 203 acquires data (polygon data and texture data) necessary for rendering from the HDD 105 or the external server 111 and outputs the data to the drawing unit 205. If the virtual viewpoint video is being reproduced as a moving image, the data acquisition unit 203 acquires data necessary for rendering the next image frame. On the other hand, if the reproduction of the virtual viewpoint video is in a paused state, data necessary for rendering the image frame that is already being reproduced is acquired. In the case of the temporary stop state, it is not necessary to acquire data again.

  In step S <b> 704, the drawing unit 205 generates a virtual viewpoint video based on the data acquired from the data acquisition unit 203 and the camera parameters acquired from the viewpoint control unit 204, and outputs the generated virtual viewpoint video to the display unit 206. To do. The display unit 206 displays the virtual viewpoint video acquired from the drawing unit 205. Since the existing technology can be used for rendering, it will not be described in detail here. Camera parameters can be classified into external parameters and internal parameters of the virtual camera. The external parameters of the virtual camera are parameters that represent the position and orientation of the virtual camera. The internal parameters of the virtual camera are parameters that represent the optical characteristics of the virtual camera. The external parameters and internal parameters will be described more specifically. If the vector representing the position of the virtual camera is t and the matrix representing the rotation is R, the external parameters of the virtual camera can be expressed as follows.

  Here, the coordinate system is described as a left-handed coordinate system, and in the virtual viewpoint, the right is the + x direction, the top is the + y direction, and the front is the + z direction.

Further, when the principal point position of the virtual viewpoint video is (c _x , c _y ) and the focal length of the virtual camera is f, the internal parameter K of the virtual camera can be expressed as follows.

  The camera parameter expression method may be an expression other than a matrix. For example, the position of the virtual camera may be represented by three-dimensional coordinates, and the posture of the virtual camera may be represented by an enumeration of yaw, roll, and pitch values. Further, the external parameters and the internal parameters are not limited to those described above. For example, information indicating the zoom value of the virtual camera may be acquired as an internal parameter of the virtual camera. As described above, there are various variations in the parameters of the virtual camera used for generating the virtual viewpoint video. The above is the flow of processing performed by the control device 100 of the present embodiment.

<Control of virtual camera according to user operation>
Details of the processing of S702 in FIG. 7 will be described with reference to FIG. In step S <b> 702, the viewpoint control unit 204 acquires the detection result of the user operation on the operation unit 201, and outputs the camera parameters of the virtual camera used for drawing the virtual viewpoint video to the drawing unit 205.

In step S <b> 801, the viewpoint control unit 204 acquires a detection result of a user operation on the operation unit 201. The detection result of the user operation includes the number n of touched points on the display screen, the two-dimensional screen coordinates x- _i (i = 1 to n) of the touched point, and the representative points of the touched point. It is assumed that a two-dimensional screen coordinate x ′ is included. Further, the detection result of the user operation includes a two-dimensional vector d = (d _x , d _y ) representing a movement amount from the representative point in the previous image frame, and a three-dimensional point specified based on the representative point. It is assumed that a three-dimensional vector T representing the position of is included. However, it is not always necessary to acquire all the above information as detection results. For example, when one finger is operated, the two-dimensional screen coordinates xi and the two-dimensional screen coordinates x ′ of the representative point are the same, and either one can be omitted.

In the coordinate system of the two-dimensional screen, the upper left is the origin, the right is + x, and the lower is + y. The representative point is a coordinate located at the center of gravity of the two-dimensional screen coordinates x _i of a plurality of touched points. However, the representative point is not limited to the center of gravity, may be the coordinates of the position to the average of the two-dimensional screen coordinates x _i, as one point of a plurality of 2-dimensional screen coordinates x _i are selected randomly Alternatively, the point touched for the longest time may be selected.

  The three-dimensional point is a point at which a ray collides with a subject by virtually flying a ray (ray cast) in the shooting direction of the virtual camera, starting from the three-dimensional coordinates corresponding to the position of the virtual camera. This three-dimensional point is used as a rotation base point or a movement reference point in the operation of the virtual camera. The three-dimensional point is determined only when the number of touches has changed from the previous image frame. When the number of touches has not changed, the three-dimensional vector T determined in the processing of the previous image frame is used as it is. In the present embodiment, an example in which a three-dimensional point is represented by a three-dimensional vector T will be described.

  In step S <b> 802, the viewpoint control unit 204 determines whether to reset the viewpoint according to a user operation on the operation unit 201. In this embodiment, when a specific area on the display screen (for example, an area where the viewpoint reset button is displayed) is tapped, it is determined that the viewpoint reset is performed.

  In step S803, the viewpoint control unit 204 resets the position and orientation of the virtual camera. That is, the viewpoint control unit 204 changes the position and orientation of the virtual viewpoint to a predetermined position and orientation in response to a user operation being detected at a predetermined position on the display screen. Then, the viewpoint control unit 204 outputs the camera parameters of the virtual camera at the reset time to the drawing unit 205. In this embodiment, it is assumed that the position of the virtual camera at the time of reset is [0 0 0] and the attitude of the virtual camera is a unit matrix. However, the viewpoint information at the time of reset is not limited to the above. For example, values set in advance by the user may be used, or recommended viewpoint information embedded in the image data may be read and used.

  In step S <b> 804, the viewpoint control unit 204 determines a virtual camera control method based on the touched score n. By controlling the virtual camera according to the number of touched fingers, a variety of control can be realized. When the number of touch points is 0, the process proceeds to S805, and the current position and orientation of the virtual camera are output to the drawing unit 205.

  When the number of touch points is one (that is, when a single finger operation is detected), the process advances to step S806, and the viewpoint control unit 204 rotates the virtual camera around the above-described three-dimensional point as a rotation center. The three-dimensional point is a point at which a ray collides with a subject by virtually emitting a ray (raycast) in the shooting direction of the virtual camera, starting from a three-dimensional coordinate corresponding to the position of the virtual camera (for example, the center coordinate of the virtual camera). . In other words, the three-dimensional point is a coordinate on the three-dimensional space corresponding to the center position of the virtual viewpoint video displayed at the time of touching. However, the three-dimensional point is not limited to this example. Details of S806 to S808 will be described later.

  When the number of touch points is two (that is, when a two-finger operation is detected), the process advances to step S809, and the viewpoint control unit 204 performs both the height direction and the horizontal direction of the virtual camera according to the slide operation by the user. Control to change the position of. In step S811, the viewpoint control unit 204 performs control to move the position of the virtual camera in the front-rear direction in accordance with a pinch-in operation and / or a pinch-out operation. Details of S809 to S812 will be described later.

  When the number of touch points is three (that is, when a three-finger operation is detected), the process advances to step S813, and the viewpoint control unit 204 performs control to move the virtual camera in parallel according to the slide operation. Details of S813 and S814 will be described later.

  In step S806, the viewpoint control unit 204 determines the coordinates of the three-dimensional point that is the rotation center when the virtual camera is rotated. In response to the detection of the touch of one finger, the viewpoint control unit 204, for example, virtually emits light rays into the three-dimensional space with the center of the virtual camera as the viewpoint, and sets the point colliding with the subject as the three-dimensional point. The three-dimensional point is represented as a three-dimensional vector A and is used as the center of rotation. Note that once the three-dimensional point is determined, it may not be determined again while the touch state continues.

In step S807, the viewpoint control unit 204 acquires the movement amount d of the representative point, and determines the movement amount of the virtual camera. In the case of a single-finger user operation, the moving direction of the virtual camera is only the rotation direction around the three-dimensional point. In the case of a single finger user operation, the virtual camera does not move in the vertical direction but moves only in the horizontal direction. In this way, the virtual camera can be moved smoothly without being affected by camera shake in the slide operation. Note that the viewpoint control unit 204 of this embodiment determines the movement amount (horizontal rotation amount θ) of the virtual camera by multiplying the movement amount d _x of the representative point by the scale factor s. Assuming that the resolution of the display screen is width w pixels and the rotation amount when the slide operation is performed from end to end of the display screen is 360 degrees, the scale s for determining the rotation amount θ [degree] from the movement amount d _x is as follows. It can be expressed by an expression.

  Using this scale factor, the rotation amount of the virtual camera can be expressed by the following equation.

  In the present embodiment, the explanation is focused on an example in which the moving direction of the virtual camera based on the user operation with one finger is only the horizontal direction, but the moving direction may be only the vertical direction. Further, depending on the contents of the user operation, it may be determined whether to move only in the horizontal direction or only in the vertical direction. Specifically, the movement direction may be determined according to the direction of the slide operation of a predetermined number of image frames after the touch is detected. For example, the amount of movement in the x direction and the y direction by a slide operation in a predetermined number of image frames is compared. If the amount of movement in the x direction is large, only the horizontal direction is moved. It may be moved. The method for determining the scale factor s is not limited to the above. For example, the user may specify an arbitrary value, or the user may arbitrarily select from a plurality of options.

In step S <b> 808, the viewpoint control unit 204 determines the position and orientation of the virtual camera according to the user's slide operation, and outputs the result to the drawing unit 205. The position R _n-1 and the orientation t _n-1 of the virtual camera, the position R _n and orientation t _n of the virtual camera when rotated by θ in the horizontal direction about the coordinate A can be expressed by the following equation.

  However, R (θ, φ) is a rotation matrix that rotates by θ in the horizontal direction and φ in the vertical direction. The formula for obtaining the position and orientation of the current virtual camera that has been rotated is not limited to this.

In step S809, the viewpoint control unit 204 acquires the movement amount d of the representative point and determines the movement amount of the virtual camera. In the case of a two-finger user operation, in order to realize control with a high degree of freedom, unlike S807, the virtual camera can be rotated in both the horizontal and vertical directions around a three-dimensional point. The horizontal rotation amount θ and the scale s are obtained in the same manner as in S807. The amount of rotation φ in the vertical direction can be expressed by the following equation.
φ = s × d _y
In step S <b> 810, the viewpoint control unit 204 determines the position and orientation of the virtual camera according to the user's slide operation, and outputs the result to the drawing unit 205. The position R _n-1 and the orientation t _n-1 of the virtual camera, rotated by θ in the horizontal direction about the three-dimensional point T, the position R _n of the virtual camera when rotated by φ in the vertical _direction, attitude t ' _n can be represented by the following formula.

  However, the formula for obtaining the position and orientation of the virtual camera when rotating around the three-dimensional point T is not limited to this. For example, by using a predetermined coefficient or the like, it is possible to increase the movement amount of the virtual camera relative to the movement amount of the finger, and conversely, it is possible to reduce the movement amount of the virtual camera relative to the movement amount of the finger. It is.

In step S811, the viewpoint control unit 204 determines the amount of movement of the virtual camera in the front-rear direction according to the user's pinch-in operation and pinch-out operation. If the distance between the two fingers when the current image frame is displayed is _dn, and the distance between the two fingers when the previous image frame is displayed is _dn-1 , the amount of change Δd is [Delta] d ₌ a d _{n -d n-1.} The virtual camera is moved back and forth in proportion to the amount of change. Assuming that the sensitivity of movement is m, the movement amount Δz of the virtual camera can be expressed as Δz = m × Δd. That is, the movement amount of the virtual camera is determined according to the movement amount of the finger per unit time by the pinch-in operation and the pinch-out operation. The method for determining the movement amount is not limited to the method described above. For example, the movement amount in the three-dimensional space may be determined based on the distance from the virtual camera to the three-dimensional point T so as to match the movement amount of the finger on the display screen.

  In step S <b> 812, the viewpoint control unit 204 determines the position of the virtual camera according to the user's pinch-in operation and pinch-out operation, and outputs the result to the drawing unit 205. The position of the virtual camera moved by Δz in the front-rear direction is expressed by the following expression.

  In step S813, the viewpoint control unit 204 determines the amount of movement of the virtual camera in the vertical and horizontal directions in accordance with the user's slide operation. In the present embodiment, the movement amount is determined so that the three-dimensional point moves on the display screen by a distance equal to the movement amount of the finger on the display screen. In other words, when the display position of an object (for example, a soccer player) is touched with three fingers and the three fingers are slid on the display screen, the positional relationship between the display position of the object and the three fingers does not change. As described above, the position of the virtual camera changes. The movement amounts Δx and Δy can be expressed by the following equations, where r is the distance from the virtual camera to the touched three-dimensional point.

  In step S <b> 814, the viewpoint control unit 204 determines the position and orientation of the virtual camera according to the user's slide operation, and outputs the result to the drawing unit 205. The position and orientation of the virtual camera when it is moved by Δx in the left-right direction and Δy in the up-down direction are expressed by the following equations.

  Note that the correspondence between the number of fingers and the content of processing is not limited to the example described above. For example, the control of the position and posture of the virtual camera by the sliding operation with one finger and the sliding operation with three fingers can be interchanged. That is, the virtual camera may be translated vertically and horizontally based on a sliding operation with one finger, and the virtual camera may be rotated around a three-dimensional point based on a sliding operation with three fingers. . Further, the relationship between the number of fingers and the control method may be arbitrarily set by the user. In this way, it is possible to provide an operation function that matches the skill of the user, the display environment of the virtual viewpoint video, and the like. In this embodiment, the process when the number of touch points is 0 (S805), the process when 1 point (S806 to S808), the process when 2 points (S809 to S812), and the process when 3 points are used. The description has been mainly focused on the double example of performing all (S813, S814). However, it is not limited to this example. For example, only the process when the number of touch points is 2 (S809 to S812) and the process when the number of touch points is 3 (S813, S814) may be executed, or the process when the number of touch points is 1 ( Only S806 to S808) may be executed. Also, which process is to be enabled may be switched according to the prior user setting.

  As described above, the control device 100 according to the present embodiment controls at least one of the position and the orientation of the virtual viewpoint according to a user operation on the display surface (display screen) for displaying the virtual viewpoint video. . According to such a configuration, the user can control the virtual viewpoint more easily than in the past. In addition, the control device 100 according to the present embodiment switches the virtual viewpoint control method according to the user operation according to the number of fingers detected from the display surface (the number of coordinates simultaneously specified by the user operation). By adopting such a configuration, the user can control the virtual viewpoint according to the intention by a more intuitive operation. In other words, according to the control device 100 of the present embodiment, there is an effect that it is possible to generate a virtual viewpoint video more in line with the user's intention.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 100 Control apparatus 201 Operation part 202 Acquisition data control part 203 Data acquisition part 204 View point control part 205 Drawing part 206 Display part

Claims (27)

A control device for controlling the position and orientation of a virtual viewpoint related to a virtual viewpoint video generated using image data obtained by photographing a predetermined photographing target region from different directions by a plurality of cameras,
When a touch operation that separates the first touch position and the second touch position is performed on the display surface that displays the virtual viewpoint video, the direction of the virtual viewpoint and the attention coordinate based on the virtual viewpoint change. If the position of the virtual viewpoint changes without being performed and a touch operation is performed in which the first touch position and the second touch position are moved in parallel on the display surface, the orientation of the virtual viewpoint is changed. Determination to determine a change amount of a parameter related to the virtual viewpoint based on the change of the first and second touch positions on the display surface so that the position of the virtual viewpoint and the coordinate of interest changes without changing. Means,
And a control unit that controls the virtual viewpoint according to the amount of change determined by the determination unit.   When a touch operation that brings the first touch position and the second touch position closer to each other on the display surface that displays the virtual viewpoint video is performed, the determining unit determines the direction of the virtual viewpoint and the virtual position. The control device according to claim 1, wherein a change amount of a parameter related to the virtual viewpoint is determined so that the position of the virtual viewpoint changes without changing the attention coordinate based on the viewpoint. A control device for controlling the position and orientation of a virtual viewpoint related to a virtual viewpoint video generated using image data obtained by photographing a predetermined photographing target region from different directions by a plurality of cameras,
When a touch operation that separates the first touch position and the second touch position is performed on the display surface that displays the virtual viewpoint video, the position and orientation of the virtual viewpoint and the coordinates of interest based on the virtual viewpoint When the zoom parameter related to the virtual viewpoint is changed without changing, and a touch operation is performed in which the first touch position and the second touch position are moved in parallel on the display surface, Change of parameters related to the virtual viewpoint based on the change of the first and second touch positions on the display surface so that the position of the virtual viewpoint and the coordinate of interest changes without changing the direction of the virtual viewpoint. A determining means for determining the amount;
And a control unit that controls the virtual viewpoint according to the amount of change determined by the determination unit.   When a touch operation that brings the first touch position and the second touch position closer to each other on the display surface that displays the virtual viewpoint video is performed, the determination unit has a zoom parameter related to the virtual viewpoint. The control device according to claim 3, wherein a change amount of a parameter related to the virtual viewpoint is determined so as to change.   5. The method according to claim 1, wherein the determining unit determines a parameter change amount related to the virtual viewpoint to be larger as a change amount of the position of the first and second touch positions is larger. The control device according to item.   When one position on the display surface is touched and it is detected that the position has changed, the determination unit determines that the virtual viewpoint is centered on the attention coordinate in accordance with the change in the position. 6. The control device according to claim 1, wherein the amount of change in the position and orientation of the virtual viewpoint is determined so as to turn in a straight line.   When one position on the display surface is touched and it is detected that the position has changed, the determination unit pays attention to the attention coordinate according to the change in the position. 6. The control device according to claim 1, wherein an amount of change in the position and orientation of the virtual viewpoint is determined such that the position and orientation of the virtual viewpoint changes without change.   8. The attention coordinate is a coordinate determined based on a center position of a virtual viewpoint image displayed on the display surface when a touch position on the display surface is detected. The control apparatus of any one of these.   When a touch operation is performed in which the first touch position and the second touch position are moved in parallel on the display surface, the determination unit determines that the position of the virtual viewpoint is 3 according to the touch operation. The control device according to claim 1, wherein a change amount of a parameter related to the virtual viewpoint is determined so as to move linearly in a dimensional space.   When a touch operation is performed in which the first touch position and the second touch position are moved in parallel on the display surface, the determination unit determines that the position of the virtual viewpoint is in a direction corresponding to the touch operation. The control device according to claim 1, wherein a change amount of a parameter related to the virtual viewpoint is determined so as to move linearly.   When a predetermined touch operation on the display surface is detected while the playback of the virtual viewpoint video on the display surface is in a paused state, the control unit changes the paused state to the playback state. The control device according to claim 1, wherein the control device is one of the features described above.   The control means changes the position and orientation of the virtual viewpoint to a predetermined position and orientation in response to a touch operation detected at a predetermined position on the display surface. The control device according to any one of 1 to 11.   The control device according to any one of claims 1 to 12, wherein one or more objects exist in the predetermined photographing target region.   The control device according to claim 13, wherein at least one of the objects is a sports player. A control device for controlling the position and orientation of a virtual viewpoint related to a virtual viewpoint video generated using image data obtained by photographing a predetermined photographing target region from different directions by a plurality of cameras,
When a pinch operation is performed on the display surface that displays the virtual viewpoint video, the virtual viewpoint moves in a direction corresponding to the orientation of the virtual viewpoint, and the first touch position and the first touch position on the display surface. When an operation for moving the two touch positions in parallel is performed, the virtual viewpoint moves in a direction corresponding to the moving direction of the first and second touch positions according to the touch operation on the display screen. Determining means for determining a control amount for movement of the virtual viewpoint;
And a control unit that controls the virtual viewpoint according to the control amount determined by the determination unit.   When a pinch-in operation is performed, the determination unit determines a control amount for moving the virtual viewpoint so that the virtual viewpoint moves backward, and when a pinch-out operation is performed, the determination unit The control device according to claim 15, wherein a control amount for moving the virtual viewpoint is determined so that the virtual viewpoint moves forward. A control device for controlling the position and orientation of a virtual viewpoint related to a virtual viewpoint video generated using image data obtained by photographing a predetermined photographing target region from different directions by a plurality of cameras,
When a pinch operation is performed on a display surface that displays the virtual viewpoint video, the zoom parameter related to the virtual viewpoint changes without moving the virtual viewpoint, and the first touch position on the display surface And the second touch position are performed on the display screen so that the virtual viewpoint moves in a direction corresponding to the moving direction of the first and second touch positions. Determining means for determining a control amount related to the virtual viewpoint according to the touch operation performed;
And a control unit that controls the virtual viewpoint according to the control amount determined by the determination unit.   When a pinch-in operation is performed, the determination unit determines a control amount so that the zoom magnification related to the virtual viewpoint is reduced, and when a pinch-out operation is performed, the determination unit performs zooming related to the virtual viewpoint. The control device according to claim 17, wherein the control amount is determined so as to increase the magnification. A control method performed by a control device that controls the position and orientation of a virtual viewpoint related to a virtual viewpoint video generated using image data obtained by shooting a predetermined shooting target region from different directions. ,
When a touch operation that separates the first touch position and the second touch position is performed on the display surface that displays the virtual viewpoint video, the direction of the virtual viewpoint and the attention coordinate based on the virtual viewpoint change. If the position of the virtual viewpoint changes without being performed and a touch operation is performed in which the first touch position and the second touch position are moved in parallel on the display surface, the orientation of the virtual viewpoint is changed. Determination to determine a change amount of a parameter related to the virtual viewpoint based on the change of the first and second touch positions on the display surface so that the position of the virtual viewpoint and the coordinate of interest changes without changing. Process,
And a control step of controlling the virtual viewpoint according to the change amount determined by the determination step.   When a touch operation that brings the first touch position and the second touch position closer to each other on the display surface that displays the virtual viewpoint video is performed, in the determination step, the orientation of the virtual viewpoint and the virtual The control method according to claim 19, wherein the change amount of the parameter relating to the virtual viewpoint is determined so that the position of the virtual viewpoint changes without changing the attention coordinate based on the viewpoint. A control method performed by a control device that controls the position and orientation of a virtual viewpoint related to a virtual viewpoint video generated using image data obtained by shooting a predetermined shooting target region from different directions. ,
When a touch operation that separates the first touch position and the second touch position is performed on the display surface that displays the virtual viewpoint video, the position and orientation of the virtual viewpoint and the coordinates of interest based on the virtual viewpoint When the zoom parameter related to the virtual viewpoint is changed without changing, and a touch operation is performed in which the first touch position and the second touch position are moved in parallel on the display surface, Change of parameters related to the virtual viewpoint based on the change of the first and second touch positions on the display surface so that the position of the virtual viewpoint and the coordinate of interest changes without changing the direction of the virtual viewpoint. A decision step to determine the amount;
And a control step of controlling the virtual viewpoint according to the change amount determined by the determination step.   When a touch operation that brings the first touch position and the second touch position closer to each other on the display surface that displays the virtual viewpoint video is performed, the zoom parameter related to the virtual viewpoint is determined in the determination step. The control method according to claim 21, wherein a change amount of a parameter related to the virtual viewpoint is determined so as to change. A control method performed by a control device that controls the position and orientation of a virtual viewpoint related to a virtual viewpoint video generated using image data obtained by shooting a predetermined shooting target region from different directions,
When a pinch operation is performed on the display surface that displays the virtual viewpoint video, the virtual viewpoint moves in a direction corresponding to the orientation of the virtual viewpoint, and the first touch position and the first touch position on the display surface. When the operation of moving the two touch positions in parallel is performed, the touch performed on the display screen so that the virtual viewpoint moves in a direction corresponding to the moving direction of the first and second touch positions. A determining step for determining a control amount for moving the virtual viewpoint according to an operation;
And a control step of controlling the virtual viewpoint according to the control amount determined in the determination step.   When a pinch-in operation is performed, in the determination step, a control amount for moving the virtual viewpoint is determined so that the virtual viewpoint moves backward, and when a pinch-out operation is performed, in the determination step 24. The control method according to claim 23, wherein a control amount for moving the virtual viewpoint is determined so that the virtual viewpoint moves forward. A control method performed by a control device that controls the position and orientation of a virtual viewpoint related to a virtual viewpoint video generated using image data obtained by shooting a predetermined shooting target region from different directions,
When a pinch operation is performed on a display surface that displays the virtual viewpoint video, the zoom parameter related to the virtual viewpoint changes without moving the virtual viewpoint, and the first touch position on the display surface And the second touch position are performed on the display screen so that the virtual viewpoint moves in a direction corresponding to the moving direction of the first and second touch positions. A determination step of determining a control amount related to the virtual viewpoint according to the touch operation performed;
And a control step of controlling the virtual viewpoint according to the control amount determined in the determination step.   When a pinch-in operation is performed, in the determination step, a control amount is determined so that a zoom magnification according to the virtual viewpoint is decreased, and when a pinch-out operation is performed, in the determination step, a zoom according to the virtual viewpoint is performed. 26. The control method according to claim 25, wherein the control amount is determined so as to increase the magnification.   A program for causing a computer to operate as each unit of the control device according to any one of claims 1 to 18.

Images