JP2020053088A - Control device, control method, and program - Google Patents

Abstract

To provide a control device capable of easily setting a virtual viewpoint.SOLUTION: A viewpoint control part 204 is configured to perform the steps of: detecting a user's operation made on a display surface for displaying virtual viewpoint images (S801); and controlling at least either one of a position and a direction of a virtual viewpoint relevant to generation of a virtual viewpoint image responding to the user's operation (S803, S808, S812, and S814).SELECTED DRAWING: Figure 2

Description

  The present invention relates to a virtual viewpoint control method.

  2. Description of the Related Art A virtual viewpoint video generation technology for generating a video of an arbitrary viewpoint from a plurality of captured videos obtained by capturing with a plurality of cameras having different viewpoints is known. As a virtual viewpoint video generation method (rendering method), according to a method of generating a virtual viewpoint video based on a predetermined moving path of the virtual viewpoint, a position and a posture of the virtual viewpoint specified by a viewer or the like, and the like. A method for generating a virtual viewpoint video is known.

  According to the virtual viewpoint video generation technology, it is possible to view a video with high interactivity. On the other hand, with devices such as tablets and smartphones that use a touch panel as a main interface, it is difficult to operate the viewpoint as desired. Patent Literature 1 discloses that a user selects one of each of a plurality of viewpoint coordinate data and a plurality of rotation start data, and then sets a viewpoint by inputting a rotation angle and a movement amount of the viewpoint. Is described.

JP-A-2005-187797

  The method of Patent Document 1 has many operation procedures for setting a virtual viewpoint. An object of the present invention is to make it possible to set a virtual viewpoint more easily.

  In order to solve the above problems, a control device according to the present invention has the following configuration. That is, control for controlling the position of the virtual viewpoint and the direction of the line of sight from the virtual viewpoint with respect to the virtual viewpoint image generated using image data obtained by photographing a predetermined imaging target region from different directions by a plurality of imaging devices. The apparatus, wherein when an operation of moving a touch position while touching one or more positions on a display surface displaying the virtual viewpoint image is performed, without changing a line-of-sight direction from the virtual viewpoint. Either move the position of the virtual viewpoint in a direction according to the line of sight, or rotate and move the position of the virtual viewpoint on a predetermined plane with reference to a rotation base point in the line of sight from the virtual viewpoint, or On the basis of the rotation base point in the line of sight from the virtual viewpoint, to rotate the position of the virtual viewpoint in a direction orthogonal to the direction of the rotational movement on the predetermined plane, Determining means for determining a parameter relating to the position of the virtual viewpoint and a line-of-sight direction from the virtual viewpoint based on a change in the touch position on the display surface; and controlling the virtual viewpoint in accordance with the parameters determined by the determining means. Control means for performing the operation.

  According to the present invention, the setting of the virtual viewpoint can be performed more easily.

FIG. 2 is a block diagram showing a hardware configuration of the control device 100. FIG. 2 is a block diagram showing a functional configuration of the control device 100. FIG. 4 is a diagram illustrating an example of virtual viewpoint control in a one-finger slide operation according to the first embodiment. FIG. 4 is a diagram illustrating an example of virtual viewpoint control in a three-finger slide operation according to the first embodiment. FIG. 4 is a diagram illustrating an example of virtual viewpoint control in a two-finger slide operation according to the first embodiment. FIG. 4 is a diagram illustrating an example of virtual viewpoint control in a two-finger pinch-out operation according to the first embodiment. 5 is a flowchart illustrating a flow of processing in the control device 100 according to the first embodiment. 9 is a flowchart illustrating a flow of virtual viewpoint control according to a user operation.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and all combinations of features described in the present embodiments are not necessarily essential to the solution of the present invention. The same components will be described with the same reference numerals.

<First embodiment>
In the first embodiment, an example will be described in which a user operates a display screen (touch panel) to control the position and orientation of a virtual viewpoint and generate a virtual viewpoint video according to the control. In the present embodiment, “change the position of the virtual camera” and “change the position of the virtual viewpoint” have the same meaning. Further, "change the posture of the virtual camera" and "change the direction of the virtual viewpoint" have the same meaning.

  Further, in the present embodiment, the virtual viewpoint video is a video generated based on a plurality of captured videos obtained by a plurality of cameras capturing a field (an imaging target region) from different directions, and includes a virtual viewpoint (virtual camera). ) Is an image generated according to the position and orientation. Further, the virtual viewpoint video of the present embodiment may be video data in which each image frame is compressed by a predetermined moving image compression method, or video in which each image frame is compressed by a predetermined still image compression method. It may be data or uncompressed video data.

  An example of a system configuration of the control device 100 according to the present embodiment will be described with reference to FIG. In FIG. 1, a CPU 101 executes a program stored in a ROM 103 and / or a hard disk drive (HDD) 105 using a RAM 102 as a work memory, and controls each component described later via a system bus 112. Thereby, various processes described later are executed. The HDD interface (I / F) 104 is an interface such as a serial ATA (SATA) for connecting the control device 100 to a secondary storage device such as the HDD 105 or an optical disk drive. The CPU 101 can read data from the HDD 105 and write data to the HDD 105 via the HDD interface (I / F) 104. Further, the CPU 101 loads the data stored in the HDD 105 into the RAM 102. Further, the CPU 101 can save various data on the RAM 102 obtained by executing the program in the HDD 105. An input interface (I / F) 106 connects an input device 107 such as a touch panel, a keyboard, a mouse, a digital camera, or a scanner for inputting one or a plurality of coordinates to the control device 100. The input interface (I / F) 106 is, for example, a serial bus interface such as USB or IEEE1394. The CPU 101 can read data from the input device 107 via the input I / F 106. The output interface (I / F) 108 is a video output interface, such as DVI or HDMI (registered trademark), for connecting the output device 109 such as a display and the control device 100. The CPU 101 can execute the display of the virtual viewpoint video by sending data relating to the virtual viewpoint video to the output device 109 via the output I / F 108. The network interface (I / F) 110 is a network card such as a LAN card that connects the control device 100 and the external server 111. The CPU 101 can read data from the external server 111 via the network I / F 110.

  In the present embodiment, an example in which the input device 107 is the touch panel of the control device 100 will be mainly described. That is, the control device 100 may be a smartphone, a tablet terminal, or the like. In this case, the input device 107 (touch panel) and the output device 109 (display screen) are integrated with the control device 100. Further, not all of the configurations shown in FIG. 1 are necessarily required. For example, when reproducing the virtual viewpoint video stored in the HDD 105, the external server 111 is unnecessary. Conversely, when generating a virtual viewpoint video acquired from the external server 111, the HDD 105 is unnecessary. Further, the control device 100 may include a plurality of CPUs 101. The CPU 101 may include one or more dedicated hardware different from the CPU 101 or a GPU (Graphics Processing Unit), and at least a part of the processing by the CPU 101 may be performed by the GPU or dedicated hardware. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors).

  In the present embodiment, a method of controlling a virtual camera according to a user's intention by a user operation on a touch panel will be described. It is assumed that the user operation in the present embodiment includes at least a tap operation, a slide operation of one to three fingers, and a pinch-in / pinch-out operation.

  Even a beginner can easily perform a user operation with one finger, but it is difficult to input complicated information, and a user operation with three fingers is difficult to perform a detailed operation. Therefore, a simple virtual camera control process is assigned to the user operation (slide operation) of one finger and three fingers. Then, a complicated virtual camera control process is assigned to the operation of two fingers.

  The specific relationship between the user operation and the operation of the virtual camera will be described with reference to FIGS. FIG. 3 shows how the position and the posture of the virtual camera are changed by the sliding operation of one finger. When a slide operation is performed to the left with one finger on the image 301 before the user operation, the position and orientation of the virtual camera are changed, and the image 302 is displayed. At this time, the position and orientation of the virtual camera change from a virtual viewpoint 305 to a virtual viewpoint 306 that moves to the right around a point 304 in the three-dimensional space drawn at the center of the overhead image 303. That is, when the number of coordinates on the display screen (on the display surface) specified simultaneously by the user operation is the first number (one), the viewpoint control unit 204 changes the virtual camera in accordance with the movement of the coordinates. Is turned around predetermined attention coordinates. In other words, if the number of coordinates on the display surface specified simultaneously by the user operation is the first number (one), the viewpoint control unit 204 changes the virtual camera in accordance with the movement of the coordinates. Control is performed so as to move while paying attention to predetermined attention coordinates.

  As shown in FIG. 3, when a one-finger slide operation is detected, the movement range of the virtual camera is limited to the circle 307, and the movement direction is limited to the horizontal direction. Accordingly, for example, even in a user who is unfamiliar with the touch operation or in a use scene in which the trajectory of the touch is likely to be blurred, the virtual camera can be moved closer to the intention of the user. Therefore, it is possible to easily generate a virtual viewpoint image such as a bullet time without blurring.

  FIG. 4 shows a state in which the position of the virtual camera is changed by a three-finger sliding operation. When a slide operation is performed to the right with three fingers on the image 401 before the user operation, the position of the virtual camera is changed, and the image 402 is displayed. At this time, the position of the virtual camera moves from the virtual viewpoint 404 to the virtual viewpoint 405 as shown in the overhead image 403. That is, when the number of coordinates on the display screen specified simultaneously by the user operation is the third number (three), the viewpoint control unit 204 sets the virtual camera in the three-dimensional space according to the change in the coordinates. The position of the virtual camera is controlled so as to move in parallel.

  Note that “three coordinates are specified at the same time” is not limited to the case where three fingers simultaneously touch the display screen. For example, when the third finger touches the display screen while holding the state where the two fingers are touching the display screen, it is determined that the three fingers are simultaneously touching the display screen. . As described above, even when the timing to start touching the display screen differs for each finger, a plurality of coordinates may be specified at the same time.

  In addition, the viewpoint control unit 204 of the present embodiment makes the moving amount of the finger on the display screen equal to the moving amount of the drawing position of the three-dimensional point 406 specified according to the user operation in the virtual viewpoint image. Move the virtual camera. As described above, by matching the moving amount of the finger with the moving amount of the three-dimensional point 406, the user can more intuitively control the virtual camera. However, control may be performed such that the moving amount of the finger and the moving amount of the drawing position of the three-dimensional point 406 in the virtual viewpoint video are different. Further, in the user operation with three fingers, the moving amount of the finger may use an average value of the moving amounts of the three fingers, a median value, or a representative value. May be used, or another value may be used. In the present embodiment, when a three-finger slide operation is performed, the position of the virtual camera is changed, but the posture of the virtual camera is not changed. That is, when the number of coordinates on the display screen specified simultaneously by the user operation is the third number (three), the viewpoint control unit 204 changes the position of the virtual viewpoint according to the change in the coordinates. The direction of the virtual viewpoint is not changed.

  FIG. 5 shows how the position and the direction of the virtual camera are changed by the slide operation of two fingers. When a slide operation is performed on the image 501 before the user operation with two fingers in the lower left direction, the position and orientation of the virtual camera are changed, and the image 502 is displayed. At this time, the position and orientation of the virtual camera change from the virtual viewpoint 505 to the virtual viewpoint 506 around a three-dimensional point 504 determined based on the position of the user's finger, as shown in the overhead image 503. The moving range of the virtual camera is limited to the spherical surface 507. In other words, when a one-finger slide operation is performed, the position of the virtual camera in the height direction is not changed, and the position in the horizontal direction is changed. Change both the height and horizontal position of the virtual camera. That is, when the number of coordinates on the display screen specified simultaneously by the user operation is the second number (two), the viewpoint control unit 204 sets the position of the virtual camera to the first position in accordance with the movement of the coordinates. The direction and the second direction are changed. As described above, the control device 100 of the present embodiment executes more complicated virtual camera control when the slide operation is performed with two fingers than when the slide operation is performed with one finger. I do. By performing a two-finger sliding operation, a virtual viewpoint video in which an arbitrary object is viewed from an arbitrary direction can be generated.

  FIG. 6 illustrates a state where the position of the virtual camera is changed by the pinch-out operation of two fingers. When the pinch-out operation is performed on the image 601 before the user operation, the position of the virtual camera is changed, and the image 602 is displayed. At this time, the position of the virtual camera moves from the virtual viewpoint 604 to the virtual viewpoint 605 as shown in the overhead view image 603. When a two-finger pinch-in operation is performed, the virtual viewpoint moves from the virtual viewpoint 605 to the virtual viewpoint 604. That is, when the number of coordinates on the display screen specified at the same time by the user operation is the second number (two), the viewpoint control unit 204 responds to the change in the coordinates and the direction of the line of sight of the virtual camera. Move the virtual camera in the direction

  By the pinch-in operation and the pinch-out operation, the position of the virtual camera moves back and forth as indicated by a dotted line 606. The subject (for example, an object such as a player) is displayed large as the finger is widened, and the subject is displayed small as the finger is narrowed, thereby enabling intuitive operation. As described above, by moving the virtual camera back and forth and rotating in accordance with the user operation of the two fingers, it is possible to realize the control of the virtual camera with a high degree of freedom. Instead of changing the position of the virtual camera according to the pinch-in operation and the pinch-out operation, a parameter related to the zoom value of the virtual camera may be changed.

  In the present embodiment, an example in which the first number is “1”, the second number is “2”, and the third number is “3” will be mainly described, but the present invention is not limited to this. For example, the first number may be “3”, the second number may be “2”, the third number may be “1”, or the first number may be “1”, and the second number may be “1”. 2 "and the third number may be" 4 ".

  The flow of processing performed by the control device 100 of the present embodiment will be described with reference to FIGS. FIG. 2 is a block diagram illustrating a functional configuration of the control device 100 according to the present embodiment. The CPU 101 reads out a program stored in the ROM 103 and / or the HDD 104 and executes the program using the RAM 102 as a work area, thereby playing a role of each functional block inside the control device 100 shown in FIG. Note that the operation unit 201 and the display unit 206 in FIG. 2 correspond to the input device 107 and the output device 109 in FIG. 1, respectively. Further, the CPU 101 does not need to play the role of all the function blocks in the control device 100, and a dedicated processing circuit corresponding to each function block may be provided.

  FIG. 7 is a flowchart illustrating a flow of a process performed by the control device 100 according to the present embodiment. Each process described with reference to FIG. 7 is realized by CPU 101 included in control device 100 reading out a program stored in ROM 103 and / or HDD 104 and executing RAM 102 as a work area.

  In step S701, the process waits until the acquired data control unit 202 detects a tap operation on the operation unit 201. The tap operation is an operation of touching the display screen with a finger for a short time. An arbitrary value can be set as the threshold value of the contact time used for the determination of the tap operation. When the tap operation is detected, if the current reproduction state of the virtual viewpoint video on the display unit 206 is the pause state, the state is changed to the reproduction state. On the other hand, if the virtual viewpoint video is being reproduced when the tap operation is detected, the acquired data control unit 202 changes the virtual viewpoint video to the paused state. As described above, the acquired data control unit 202 changes the reproduction state of the virtual viewpoint video according to the tap operation on the display screen, so that the user can switch the reproduction state by an intuitive operation. However, S701 is not an essential process.

  In step S702, the viewpoint control unit 204 outputs camera parameters related to the position and orientation of the virtual camera to the drawing unit 205 based on the detection result of the user operation on the operation unit 201. Details of S702 will be described later with reference to FIG.

  In step S703, the data acquisition unit 203 acquires data (polygon data or texture data) necessary for rendering from the HDD 105 or the external server 111, and outputs the data to the rendering unit 205. If the virtual viewpoint video is being reproduced as a moving image, the data acquisition unit 203 acquires data necessary for rendering the next image frame. On the other hand, if the reproduction of the virtual viewpoint video is in a pause state, data necessary for rendering the image frame that is already being reproduced is acquired. In the case of the pause state, it is not necessary to acquire data again.

  In step S704, the drawing unit 205 generates a virtual viewpoint video based on the data acquired from the data acquisition unit 203 and the camera parameters acquired from the viewpoint control unit 204, and outputs the generated virtual viewpoint video to the display unit 206. I do. The display unit 206 displays the virtual viewpoint video acquired from the drawing unit 205. The rendering is not described in detail here because existing techniques can be used. The camera parameters can be classified into external parameters and internal parameters of the virtual camera. The external parameters of the virtual camera are parameters representing the position and orientation of the virtual camera. In addition, the internal parameters of the virtual camera are parameters representing optical characteristics of the virtual camera. The external parameters and the internal parameters will be described more specifically. Assuming that a vector representing the position of the virtual camera is t and a matrix representing the rotation is R, the external parameters of the virtual camera can be represented as follows.

  Here, the coordinate system is described as a left-handed coordinate system, and in the virtual viewpoint, the right is the + x direction, the upper is the + y direction, and the front is the + z direction.

If the principal point position of the virtual viewpoint video is (c _x , _cy ) and the focal length of the virtual camera is f, the internal parameter K of the virtual camera can be expressed as follows.

  The method of expressing the camera parameters may be an expression other than the matrix. For example, the position of the virtual camera may be represented by three-dimensional coordinates, and the posture of the virtual camera may be represented by a list of values of yaw, roll, and pitch. Further, the external parameters and the internal parameters are not limited to those described above. For example, information indicating the zoom value of the virtual camera may be obtained as an internal parameter of the virtual camera. As described above, there are various variations in the parameters of the virtual camera used for generating the virtual viewpoint video. The above is the flow of the processing performed by the control device 100 of the present embodiment.

<Control of virtual camera according to user operation>
Details of the processing in S702 of FIG. 7 will be described with reference to FIG. In step S702, the viewpoint control unit 204 acquires a detection result of a user operation performed on the operation unit 201, and outputs camera parameters of a virtual camera used for drawing a virtual viewpoint video to the drawing unit 205.

In step S801, the viewpoint control unit 204 acquires a detection result of a user operation performed on the operation unit 201. The detection result of the user operation includes the number n of touched points on the display screen, the two-dimensional screen coordinates x-- _i (i = 1 to n) of the touched point, and the representative point of the touched point. It is assumed that two-dimensional screen coordinates x 'are included. In addition, the detection result of the user operation includes a two-dimensional vector d = (d _x , _dy ) representing an amount of movement from the representative point in the previous image frame, and a three-dimensional point specified based on the representative point. Is assumed to be included. However, it is not always necessary to acquire all the above information as a detection result. For example, when an operation with one finger is performed, one of the two-dimensional screen coordinates xi and the two-dimensional screen coordinates x ′ of the representative point can be omitted because they are the same.

The coordinate system of the two-dimensional screen has the origin at the upper left, + x at the right, and + y at the lower. Representative point is the coordinates located at the center of gravity of the two-dimensional screen coordinates x _i of the plurality of points are touched. However, the representative point is not limited to the center of gravity, may be the coordinates of the position to the average of the two-dimensional screen coordinates x _i, as one point of a plurality of 2-dimensional screen coordinates x _i are selected randomly Alternatively, the point touched for the longest time may be selected.

  In addition, the three-dimensional point is a point where light rays are virtually skipped (ray cast) in the shooting direction of the virtual camera starting from the three-dimensional coordinates corresponding to the position of the virtual camera, and collide with the subject. The three-dimensional point is used as a rotation base point or a movement reference point in the operation of the virtual camera. The three-dimensional point is determined only when the number of touches has changed from the previous image frame, and when the number of touches has not changed, the three-dimensional vector T determined in the processing of the previous image frame is used as it is. In the present embodiment, an example in which a three-dimensional point is represented by a three-dimensional vector T will be described. However, the three-dimensional point need not always be represented in a vector format.

  In step S802, the viewpoint control unit 204 determines whether to perform a viewpoint reset in accordance with a user operation on the operation unit 201. In the present embodiment, when a specific area on the display screen (for example, an area where the viewpoint reset button is displayed) is tapped, it is determined that the viewpoint is reset.

  In step S803, the viewpoint control unit 204 resets the position and orientation of the virtual camera. That is, the viewpoint control unit 204 changes the position and orientation of the virtual viewpoint to a predetermined position and orientation in response to detection of a user operation at a predetermined position on the display screen. Then, the viewpoint control unit 204 outputs the camera parameters of the virtual camera at the time of reset to the drawing unit 205. In the present embodiment, the position of the virtual camera at the time of reset is [0 0 0], and the posture of the virtual camera is a unit matrix. However, the viewpoint information at the time of reset is not limited to the above. For example, a value set in advance by a user may be used, or recommended viewpoint information embedded in image data may be read and used.

  In step S804, the viewpoint control unit 204 determines a control method of the virtual camera based on the touched score n. By making the control method of the virtual camera different according to the number of touched fingers, more various controls can be realized. If the number of touch points is 0, the process advances to step S805 to output the current position and orientation of the virtual camera to the drawing unit 205.

  When the number of touch points is one (that is, when the operation of one finger is detected), the process proceeds to S806, and the viewpoint control unit 204 rotates the virtual camera around the above three-dimensional point. The three-dimensional point is a point at which light rays are virtually skipped (ray cast) in the shooting direction of the virtual camera starting from three-dimensional coordinates (for example, the center coordinates of the virtual camera) corresponding to the position of the virtual camera, and collide with the subject. . In other words, the three-dimensional point is a coordinate in the three-dimensional space corresponding to the center position of the virtual viewpoint image displayed at the time of touch. However, the three-dimensional point is not limited to this example. Details of S806 to S808 will be described later.

  If the number of touch points is two (that is, if an operation with two fingers is detected), the process proceeds to step S809, and the viewpoint control unit 204 performs both the height direction and the horizontal direction of the virtual camera in accordance with the slide operation by the user. Control to change the position of. In step S811, the viewpoint control unit 204 performs control to move the position of the virtual camera in the front-rear direction according to the pinch-in operation and / or the pinch-out operation. Details of S809 to S812 will be described later.

  When the number of touch points is three (that is, when the operation of three fingers is detected), the process proceeds to S813, and the viewpoint control unit 204 performs control to move the virtual camera in parallel according to the slide operation. Details of S813 and S814 will be described later.

  In step S806, the viewpoint control unit 204 determines the coordinates of a three-dimensional point that is the center of rotation when rotating the virtual camera. In response to the detection of one finger touch, for example, the viewpoint control unit 204 virtually emits a light ray to a three-dimensional space with the center of the virtual camera as a viewpoint, and sets a point that has collided with the subject as a three-dimensional point. The three-dimensional point is represented as a three-dimensional vector A, and is used as a rotation center. Note that once the three-dimensional point is determined, it is not necessary to determine it again while the touch state continues.

In step S807, the viewpoint control unit 204 acquires the moving amount d of the representative point, and determines the moving amount of the virtual camera. In the case of one-finger user operation, the moving direction of the virtual camera is only the rotation direction about the three-dimensional point. In the case of a one-finger user operation, the virtual camera does not move in the vertical direction, but moves only in the horizontal direction. With this configuration, the virtual camera can be smoothly moved without being easily affected by camera shake in the slide operation. Note that the viewpoint control unit 204 of the present embodiment determines the movement amount (the horizontal rotation amount θ) of the virtual camera by multiplying the movement amount d _x of the representative point by the scale factor s. Assuming that the resolution of the display screen is width w pixels and the rotation amount when the slide operation is performed from end to end of the display screen is 360 degrees, a scale s for determining the rotation amount θ [degree] from the movement amount d _x is as follows. It can be expressed by an equation.

  Using this scale factor, the rotation amount of the virtual camera can be expressed by the following equation.

  In this embodiment, an example in which the moving direction of the virtual camera based on a user operation of one finger is limited to only the horizontal direction will be mainly described, but the moving direction may be limited to the vertical direction. Further, it may be determined whether to perform the movement only in the horizontal direction or the movement only in the vertical direction according to the content of the user operation. Specifically, the moving direction may be determined according to the direction of the sliding operation of a predetermined number of image frames after the touch is detected. For example, the amount of movement in the x and y directions by a slide operation in a predetermined number of image frames is compared. If the amount of movement in the x direction is large, the movement is performed only in the horizontal direction. It may be moving. Further, the method of determining the scale coefficient s is not limited to the method described above. For example, the user may specify an arbitrary value, or the user may arbitrarily select from a plurality of options.

In step S808, the viewpoint control unit 204 determines the position and orientation of the virtual camera according to the user's slide operation, and outputs the result to the drawing unit 205. The position R _n-1 and the orientation t _n-1 of the virtual camera, the position R _n and orientation t _n of the virtual camera when rotated by θ in the horizontal direction about the coordinate A can be expressed by the following equation.

  Here, R (θ, φ) is a rotation matrix that rotates by θ in the horizontal direction and φ in the vertical direction. Further, the formula for calculating the current position and orientation of the rotated virtual camera is not limited to this.

In step S809, the viewpoint control unit 204 acquires the movement amount d of the representative point, and determines the movement amount of the virtual camera. Note that in the case of a two-finger user operation, unlike S807, the virtual camera can be rotated in both the horizontal and vertical directions around a three-dimensional point, in order to realize control with a high degree of freedom. The horizontal rotation amount θ and the scale s are obtained in the same manner as in S807. The amount of rotation φ in the vertical direction can be represented by the following equation.
φ = s × d _y
In step S810, the viewpoint control unit 204 determines the position and orientation of the virtual camera according to the user's slide operation, and outputs the result to the drawing unit 205. The position R _n-1 and the orientation t _n-1 of the virtual camera, rotated by θ in the horizontal direction about the three-dimensional point T, the position R _n of the virtual camera when rotated by φ in the vertical _direction, attitude t ' _n can be represented by the following equation.

  However, the expression for obtaining the position and orientation of the virtual camera when the virtual camera is rotated about the three-dimensional point T is not limited to this. For example, by using a predetermined coefficient or the like, it is possible to increase the movement amount of the virtual camera with respect to the movement amount of the finger, and conversely, it is also possible to reduce the movement amount of the virtual camera with respect to the movement amount of the finger It is.

In step S811, the viewpoint control unit 204 determines the amount of movement of the virtual camera in the front-back direction according to the pinch-in operation and the pinch-out operation of the user. Assuming that the distance between the two fingers when the current image frame is displayed is _dn and the distance between the two fingers when the immediately preceding image frame is displayed is _dn-1 , the change amount Δd is [Delta] d ₌ a d _{n -d n-1.} The virtual camera is moved back and forth in proportion to the amount of change. Assuming that the sensitivity of the movement is m, the movement amount Δz of the virtual camera can be represented by Δz = m × Δd. That is, the moving amount of the virtual camera is determined according to the moving amount of the finger per unit time due to the pinch-in operation and the pinch-out operation. Note that the method of determining the moving amount is not limited to the method described above. For example, the moving amount in the three-dimensional space may be determined based on the distance from the virtual camera to the three-dimensional point T so as to match the moving amount of the finger on the display screen.

  In step S <b> 812, the viewpoint control unit 204 determines the position of the virtual camera according to the user's pinch-in operation and pinch-out operation, and outputs the result to the drawing unit 205. The position of the virtual camera moved by Δz in the front-rear direction is represented by the following equation.

  In step S813, the viewpoint control unit 204 determines the amount of movement of the virtual camera in the up, down, left, and right directions according to the slide operation of the user. In the present embodiment, the moving amount is determined such that the three-dimensional point moves on the display screen by a distance equal to the moving amount of the finger on the display screen. That is, when the display position of an object (for example, a soccer player) is touched with three fingers and the three fingers are slid on the display screen, the positional relationship between the display position of the object and the three fingers does not change. Thus, the position of the virtual camera changes. The movement amounts Δx and Δy can be expressed by the following expressions, where r is the distance from the virtual camera to the touched three-dimensional point.

  In step S814, the viewpoint control unit 204 determines the position and orientation of the virtual camera according to the user's slide operation, and outputs the result to the drawing unit 205. The position and orientation of the virtual camera when moving by Δx in the horizontal direction and Δy in the vertical direction are represented by the following equations.

  Note that the correspondence between the number of fingers and the content of the processing is not limited to the example described above. For example, the control of the position and orientation of the virtual camera by the sliding operation of one finger and the sliding operation of three fingers can be exchanged. That is, the virtual camera may be moved up, down, left, and right in parallel based on the sliding operation of one finger, and the virtual camera may be rotated around the three-dimensional point based on the sliding operation of three fingers. . Further, the relationship between the number of fingers and the control method may be arbitrarily set by the user. This makes it possible to provide an operation function that matches the user's skill, the display environment of the virtual viewpoint video, and the like. Further, in the present embodiment, processing when the number of touch points is 0 (S805), processing when the number of touch points is 1 (S806 to S808), processing when the number of touch points is 2 (S809 to S812), processing when the number of touch points is 3 The description has been made centering on an example in which (S813, S814) are all performed twice. However, it is not limited to this example. For example, only the processing when the number of touch points is two (S809 to S812) and the processing when the number of touch points is three (S813 and S814) may be performed, or the processing when the number of touch points is one ( Only steps S806 to S808) may be executed. Further, which process is enabled may be switched according to a user setting in advance.

  As described above, the control device 100 of the present embodiment controls at least one of the position and the orientation of the virtual viewpoint according to the user operation on the display surface (display screen) for displaying the virtual viewpoint video. . According to such a configuration, the user can control the virtual viewpoint more easily than before. In addition, the control device 100 according to the present embodiment switches the control method of the virtual viewpoint according to the user operation according to the number of fingers detected from the display surface (the number of coordinates specified simultaneously by the user operation). By adopting such a configuration, the user can control the virtual viewpoint according to the intention by a more intuitive operation. In other words, according to the control device 100 of the present embodiment, there is an effect that it is possible to generate a virtual viewpoint video more in line with the user's intention.

<Other embodiments>
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

REFERENCE SIGNS LIST 100 control device 201 operation unit 202 acquired data control unit 203 data acquisition unit 204 viewpoint control unit 205 drawing unit 206 display unit

Claims (21)

A control device that controls a position of a virtual viewpoint and a gaze direction from the virtual viewpoint with respect to a virtual viewpoint image generated using image data obtained by photographing a predetermined imaging target region from different directions by a plurality of imaging devices. So,
When an operation of moving the touch position while touching one or more positions on the display surface displaying the virtual viewpoint image is performed, the position of the virtual viewpoint is changed without changing the line of sight from the virtual viewpoint. Move in the direction according to the line of sight, or rotate and move the position of the virtual viewpoint on a predetermined plane with reference to the rotation base point in the line of sight from the virtual viewpoint, or look from the virtual viewpoint On the basis of the rotation base point in the direction, based on a change in the touch position on the display surface, so as to rotate and move the position of the virtual viewpoint in a direction orthogonal to the direction of the rotational movement on the predetermined plane. Determining means for determining a parameter relating to the position of the virtual viewpoint and the viewing direction from the virtual viewpoint,
Control means for controlling the virtual viewpoint in accordance with the parameters determined by the determining means.   A case where a pinch-out operation of separating the first touch position and the second touch position from each other while touching the first touch position and the second touch position on the display surface is performed; and When a pinch-in operation is performed to bring the first touch position and the second touch position closer to each other, the determining unit may change the position of the virtual viewpoint without changing a line-of-sight direction from the virtual viewpoint. The control device according to claim 1, wherein the parameter is determined such that the parameter is changed in a direction corresponding to the viewing direction. A control device that controls a position of a virtual viewpoint and a gaze direction from the virtual viewpoint with respect to a virtual viewpoint image generated using image data obtained by photographing a predetermined imaging target region from different directions by a plurality of imaging devices. So,
When an operation of moving a touch position is performed while touching one or more positions on a display surface displaying the virtual viewpoint image, a zoom related to the virtual viewpoint without changing a line of sight from the virtual viewpoint. Change the parameter, rotate the position of the virtual viewpoint on a predetermined plane with reference to the rotation base point in the line of sight from the virtual viewpoint, or change the rotation base point in the line of sight from the virtual viewpoint. As a reference, based on a change in a touch position on the display surface, the position of the virtual viewpoint and the position of the virtual viewpoint are rotated and moved in a direction orthogonal to the direction of the rotational movement on the predetermined plane. Deciding means for deciding a parameter related to a line-of-sight direction from the virtual viewpoint,
Control means for controlling the virtual viewpoint in accordance with the parameters determined by the determining means.   A case where a pinch-out operation of separating the first touch position and the second touch position from each other while touching the first touch position and the second touch position on the display surface is performed; and When a pinch-in operation is performed to bring the first touch position and the second touch position closer to each other, the determining unit may perform zooming on the virtual viewpoint without changing a line of sight from the virtual viewpoint. The control device according to claim 3, wherein the parameter is determined so as to change the parameter.   When a slide operation of moving the touch position while touching the position 1 on the display surface is performed, the determining unit determines the position on the predetermined plane on the basis of the rotation base point in the line of sight from the virtual viewpoint. The control device according to any one of claims 1 to 4, wherein the parameter is determined so that the position of the virtual viewpoint is rotated.   The control device according to any one of claims 1 to 5, wherein the predetermined plane is a horizontal plane.   When a slide operation of moving the predetermined number of touch positions in parallel while touching two or more predetermined numbers of positions on the display surface is performed, the determining unit determines a position in a line-of-sight direction from the virtual viewpoint. 7. The parameter according to claim 1, wherein the parameter is determined such that the position of the virtual viewpoint is rotationally moved in a direction orthogonal to a direction of the rotational movement on the predetermined plane with reference to a rotational base point. The control device according to claim 1.   The control device according to claim 7, wherein the predetermined number is two.   9. The parameter according to claim 1, wherein the determining unit determines the parameter such that a position of the virtual viewpoint is moved while a line of sight from the virtual viewpoint is directed to a predetermined rotation base point. 2. The control device according to claim 1. Having a detecting means for detecting a specific user operation on the display surface,
The control unit is configured to respond to the detection of the specific user operation by the detection unit so that a position of the virtual viewpoint and a line-of-sight direction from the virtual viewpoint become a predetermined position and direction. The control device according to claim 1, wherein a viewpoint is controlled.   The control device according to claim 10, wherein the specific user operation on the display surface is a touch operation on a predetermined position on the display surface.   When a predetermined touch operation on the display surface is detected while the reproduction of the virtual viewpoint image displayed on the display surface is in a pause state, the control unit displays the virtual viewpoint image on the display surface. 12. The control device according to claim 1, wherein a reproduction state of the virtual viewpoint image is changed from the pause state to a reproduction state.   The control device according to claim 1, wherein one or more objects exist in the predetermined imaging target area. The at least one of the objects is a sports player.
The control device according to claim 1. A control device that controls a position of a virtual viewpoint and a gaze direction from the virtual viewpoint related to a virtual viewpoint image generated using image data obtained by photographing a predetermined imaging target region from different directions by a plurality of imaging devices. Control method to be performed,
When an operation of moving the touch position while touching one or more positions on the display surface displaying the virtual viewpoint image is performed, the position of the virtual viewpoint is changed without changing the line of sight from the virtual viewpoint. Move in the direction according to the line of sight, or rotate and move the position of the virtual viewpoint on a predetermined plane with reference to the rotation base point in the line of sight from the virtual viewpoint, or look from the virtual viewpoint On the basis of the rotation base point in the direction, based on a change in the touch position on the display surface, so as to rotate and move the position of the virtual viewpoint in a direction orthogonal to the direction of the rotational movement on the predetermined plane. A determining step of determining a parameter relating to a position of the virtual viewpoint and a viewing direction from the virtual viewpoint,
Controlling the virtual viewpoint in accordance with the parameters determined in the determining step.   A case where a pinch-out operation of separating the first touch position and the second touch position from each other while touching the first touch position and the second touch position on the display surface is performed; and When a pinch-in operation is performed to bring the first touch position and the second touch position closer to each other, the determining step includes changing a position of the virtual viewpoint without changing a line-of-sight direction from the virtual viewpoint. 16. The control method according to claim 15, wherein the parameter is determined so as to change in a direction corresponding to the line-of-sight direction. A control device that controls a position of a virtual viewpoint and a gaze direction from the virtual viewpoint related to a virtual viewpoint image generated using image data obtained by photographing a predetermined imaging target region from different directions by a plurality of imaging devices. Control method to be performed,
When an operation of moving a touch position is performed while touching one or more positions on a display surface displaying the virtual viewpoint image, a zoom related to the virtual viewpoint without changing a line of sight from the virtual viewpoint. Change the parameter, rotate the position of the virtual viewpoint on a predetermined plane with reference to the rotation base point in the line of sight from the virtual viewpoint, or change the rotation base point in the line of sight from the virtual viewpoint. As a reference, based on a change in a touch position on the display surface, the position of the virtual viewpoint and the position of the virtual viewpoint are rotated and moved in a direction orthogonal to the direction of the rotational movement on the predetermined plane. A determining step of determining a parameter related to a line-of-sight direction from the virtual viewpoint;
Controlling the virtual viewpoint in accordance with the parameters determined in the determining step.   A case where a pinch-out operation of separating the first touch position and the second touch position from each other while touching the first touch position and the second touch position on the display surface is performed; and When a pinch-in operation for bringing the first touch position and the second touch position closer to each other is performed, the determining step includes the step of zooming the virtual viewpoint without changing a line-of-sight direction from the virtual viewpoint. The control method according to claim 17, wherein the parameter is determined so as to change the parameter.   When a slide operation for moving the touch position while touching the position 1 on the display surface is performed, the determining step is performed on the predetermined plane on the basis of the rotation base point in the line of sight from the virtual viewpoint. 19. The control method according to claim 15, wherein the parameter is determined such that the position of the virtual viewpoint is rotated. A detecting step of detecting a touch operation on a predetermined position on the display surface,
The control step is such that, in response to the detection of the specific user operation in the detection step, the virtual viewpoint position and the line-of-sight direction from the virtual viewpoint become the predetermined position and direction. The control method according to claim 15, wherein the viewpoint is controlled.   A program for causing a computer to operate as each unit of the control device according to claim 1.

Images