JP7561533B2 - Photographing device, image processing device, control method for photographing device, control method for image processing device, and photographing system - Google Patents

Description

The present invention relates to technology for displaying a subject.

In recent years, virtual viewpoint image generation technology has been attracting attention. This technology uses a multi-viewpoint shooting system in which multiple cameras are arranged to surround a subject or the space in which the subject exists, to generate images seen from a viewpoint where no camera actually exists. Virtual viewpoint image generation technology estimates the position and three-dimensional shape model of the subject from images captured by the multi-viewpoint shooting system, and reconstructs images seen from a virtual viewpoint.

Conventionally, when a subject of interest is far away, it is common to use so-called telephoto photography, in which a lens with a long focal length is attached to a mirrorless or single-lens reflex camera to take the image. However, in telephoto photography, it is difficult to keep the subject of interest within the angle of view, and it is also difficult to find a subject of interest that is outside the angle of view. For this reason, techniques are known that notify the position of a subject of interest that is outside the angle of view. For example, in Patent Document 1, position information when the subject of interest goes outside the angle of view is displayed on the screen, making it easier to find the subject. In Patent Document 2, a position detection sensor such as a GPS is attached to the subject, and the detected position information is displayed on the screen to prevent the subject from being lost.

JP 2005-341449 A JP 2009-224900 A

However, in Patent Document 1, it is not possible to predict the movement of the subject after it has gone out of the angle of view, so it becomes difficult to find the subject of interest if the subject moves quickly or if it has been out of the angle of view for a long time. In addition, in Patent Document 2, it is difficult to determine the relationship between the current camera attitude and the position of the subject, so it is not clear how to move the camera to capture the subject within the angle of view.

This invention provides a technology for notifying a user of information necessary for operations to bring a subject within the angle of view of a photographing device when the subject is located outside the angle of view of the photographing device.

One aspect of the present invention is an imaging device, comprising:
an acquisition means for determining a two-dimensional position of the subject in a coordinate system based on the images captured by the photographing device, based on the three-dimensional position and three-dimensional orientation of the photographing device and the three-dimensional position of the subject estimated from the images captured by a plurality of cameras arranged to surround the subject;
If the two-dimensional position is outside the range of the angle of view of the imaging device, information required for operations to bring the two-dimensional position within the range of the angle of view of the imaging device is obtained, and a display control means is provided to perform a display based on the obtained information.

According to the configuration of the present invention, when a subject is located outside the range of the angle of view of a photographing device, the user can be notified of information necessary for operations to bring the subject within the range of the angle of view of the photographing device.

FIG. 1 is a diagram showing an example of the configuration of an imaging system. FIG. 2 is a block diagram showing an example of the hardware configuration of the imaging system. FIG. 2 is a block diagram showing an example of the functional configuration of an image capturing device 110 and an image processing device 109. 4 is a flowchart of a process performed by the photographing device 110 and the image processing device 109. FIG. 2 is a diagram showing the relationship between the world coordinate system, the camera coordinate system, and the image coordinate system. FIG. 2 is a diagram showing an example of a two-dimensional position of a subject 111 in an image coordinate system. 7A is a diagram showing an example of how notification information is displayed, and FIG. 7B is a diagram showing an example of a combination of the size and color of an icon 703 according to the length (amount of movement) of a vector. FIG. 2 is a block diagram showing an example of the functional configuration of an image capturing device 110 and an image processing device 109. 4 is a flowchart of a process performed by the photographing device 110 and the image processing device 109. 4 is a flowchart of a process performed by the photographing device 110 and the image processing device 109. FIG. 4 is a diagram showing an example of a movement vector of a subject 111. 1A is a diagram showing an example of displaying notification information, and FIG. 1B is a diagram showing an example of a table configuration.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

[First embodiment]
The imaging system according to the present embodiment includes a multi-view imaging system in which a plurality of cameras are arranged to surround a space in which a subject exists, and an imaging device for imaging the subject. An example of the configuration of the imaging system according to the present embodiment will be described with reference to FIG.

In the multi-viewpoint shooting system according to this embodiment, the cameras 101 to 108 are arranged to surround a soccer court on which a subject 111 is located, and the cameras 101 to 108 are connected to the image processing device 109 via a wired and/or wireless network. The cameras 101 to 108 start shooting based on instructions from the image processing device 109, and the captured images (captured images) are transmitted to the image processing device 109 via the network. The image processing device 109 estimates the position and three-dimensional shape model of the subject 111 from the captured images received from the cameras 101 to 108 based on conventional virtual viewpoint video generation technology, and generates an image seen from a virtual viewpoint (virtual viewpoint video). The image processing device 109 may display the generated video on its own display screen, or may transmit it to an external device via a network, and the output destination of the generated video is not limited to a specific output destination. The image processing device 109 is a computer device that performs such operations, and is a computer device such as a PC (personal computer), a smartphone, or a tablet terminal device.

The image capturing device 110 is a camera such as a single-lens reflex or mirrorless camera, and is connected to the image processing device 109 via a wired and/or wireless network. The image capturing device 110 transmits captured images (captured images) to the image processing device 109 via the network. The image processing device 109 generates information required for operations to fit the subject 111 within the angle of view of the image capturing device 110 based on the captured image received from the image capturing device 110 and the three-dimensional position and orientation of the subject 111 obtained by the above-mentioned virtual viewpoint video generation technology, and transmits the information to the image capturing device 110. The image capturing device 110 can notify the user of information required for operations to fit the subject 111 within the angle of view of the image capturing device 110 by performing a display based on the information transmitted from the image processing device 109.

Note that FIG. 1 shows the main components of the imaging system that will be referred to in the following description, and the imaging system may include further devices in addition to the components shown in FIG. 1. For example, the imaging system may include devices such as a device that relays data communication, and a server device for storing the images captured by the cameras 101-108 and the generated virtual viewpoint video.

Next, an example of the hardware configuration of the imaging system according to this embodiment will be described using the block diagram in FIG. 2. In FIG. 2, the cameras 101 to 108, the imaging device 110, and the image processing device 109 are all connected to a LAN, but the network connecting the devices can take a variety of forms and is not limited to a specific form.

First, an example of the hardware configuration of the image capture device 110 will be described. The CPU 201 executes various processes using computer programs and data stored in the main memory 202. As a result, the CPU 201 controls the operation of the entire image capture device 110, and executes or controls various processes that will be described as being performed by the image capture device 110.

The main memory 202 has areas for storing computer programs and data loaded from the storage unit 203, data received from the outside via the communication unit 207, images captured by the image capture unit 206, etc. Furthermore, the main memory 202 has a work area used when the CPU 201 executes various processes. In this way, the main memory 202 can provide various areas as appropriate.

The storage unit 203 stores an OS (operating system), computer programs and data for causing the CPU 201 to execute or control various processes described as being performed by the imaging device 110, and the like. The computer programs and data stored in the storage unit 203 are loaded into the main memory 202 as appropriate under the control of the CPU 201, and are processed by the CPU 201. For example, a non-volatile memory such as a silicon disk can be used for the storage unit 203.

The operation unit 204 is a user interface that includes buttons, a mode dial, switches, levers, a touch panel screen, etc., and can be operated by the user to input various instructions to the CPU 201.

The display unit 205 has a screen such as an LCD screen or a touch panel screen, and displays the results of processing by the CPU 201 as images, text, etc. If the display unit 205 has a touch panel screen, the operation input entered by the user by operating the touch panel screen is notified to the CPU 201.

The image capturing unit 206 has a configuration capable of capturing still images and moving images, and includes, for example, an optical system, a shutter, an image sensor, etc. The image capturing unit 206 controls the optical system and the image sensor, such as adjusting the focus, opening/closing the shutter, and adjusting the aperture, in response to instructions from the CPU 201.

The communication unit 207 is a device that complies with communication standards such as Ethernet and IEEE 802.11, and the image capture device 110 can perform data communication with external devices via the communication unit 207. The CPU 201, main memory 202, storage unit 203, operation unit 204, display unit 205, image capture unit 206, and communication unit 207 are all connected to a bus 208.

Next, an example of the hardware configuration of the image processing device 109 will be described. The CPU 209 executes various processes using computer programs and data stored in the main memory 210. As a result, the CPU 209 controls the operation of the entire image processing device 109, and executes or controls various processes that will be described as being performed by the image processing device 109.

The main memory 210 has areas for storing computer programs and data loaded from the storage unit 211, data received from the outside via the communication unit 212, etc. Furthermore, the main memory 210 has a work area used when the CPU 209 executes various processes. In this way, the main memory 210 can provide various areas as appropriate.

The storage unit 211 stores an OS, computer programs and data for causing the CPU 209 to execute or control various processes described as being performed by the image processing device 109, and the like. The computer programs and data stored in the storage unit 211 are loaded into the main memory 210 as appropriate under the control of the CPU 209, and become the subject of processing by the CPU 209.

The communication unit 212 is a device for performing data communication with external devices, and the image processing device 109 can perform data communication with external devices via the communication unit 212. For example, the image processing device 109 can transmit instructions to the cameras 101 to 108, such as to start or stop shooting, or to change the shutter speed or aperture, via the communication unit 212, and can receive captured images from the cameras 101 to 108. Also, for example, the image processing device 109 can perform data communication with the image capture device 110 via the communication unit 212.

The image processing device 109 (CPU 209) executes a computer program for generating a virtual viewpoint image. As a result, the image processing device 109 estimates the three-dimensional position and three-dimensional posture of the subject 111 and the three-dimensional shape model of the subject 111 from the images captured by the cameras 101 to 108 at the same time. The image processing device 109 then uses these estimation results to render and generate an image of a space viewed from a virtual viewpoint where no cameras actually exist. The three-dimensional shape model of the subject 111 can be estimated from the coordinate values of feature points that match between images based on the positions and postures of the cameras that have been calibrated in advance. Known three-dimensional shape model estimation methods include MVS (Multi View Stereo) and VH (Visual Hull). The CPU 209, main memory 210, storage unit 211, and communication unit 212 are all connected to the bus 213.

The block diagram of FIG. 3 shows an example of the functional configuration of each of the imaging device 110 and the image processing device 109. The processing performed by the imaging device 110 and the image processing device 109 having the example of the functional configuration shown in FIG. 3 will be described with reference to the flowchart of FIG. 4. Note that the following description will be made with the functional units shown in FIG. 3 as the subject of the processing. However, in reality, the functions of the functional units of the imaging device 110 are realized by the CPU 201 executing a computer program for causing the CPU 201 to realize the functions of the functional units of the imaging device 110. Also, the functions of the functional units of the image processing device 109 are realized by the CPU 209 executing a computer program for causing the CPU 209 to realize the functions of the functional units of the image processing device 109. Note that the functional units shown in FIG. 3 may be implemented in hardware.

In step S401, the acquisition unit 301 of the imaging device 110 acquires a captured image captured by the imaging unit 206. The captured image may be a still image or an image of each frame in a moving image.

In step S402, the transmission unit 302 of the imaging device 110 transmits the captured image acquired by the acquisition unit 301 in step S401 to the image processing device 109. In step S403, the reception unit 303 of the image processing device 109 receives the captured image transmitted by the transmission unit 302 of the imaging device 110 in step S402.

In step S404, the estimation unit 304 of the image processing device 109 estimates the position and orientation (three-dimensional position and three-dimensional orientation) of the image capturing device 110 in the world coordinate system based on the captured image received by the receiving unit 303 of the image processing device 109 in step S403. The world coordinate system is an orthogonal coordinate system with one point in real space as the origin and three axes perpendicular to each other at the origin as the x-axis, y-axis, and z-axis. In this embodiment, as an example, the world coordinate system is an orthogonal coordinate system with the center position of the center circle of the soccer court shown in FIG. 1 as the origin, the axis along the long side of the soccer court as the y-axis, the axis along the short side of the soccer court as the x-axis, and the axis along the direction perpendicular to the surface of the soccer court as the z-axis. In the following, the Cartesian coordinate system (based on the position and orientation of the camera 110) in which the three-dimensional position of the camera 110 is the origin and the axes corresponding to the three components in the three-dimensional orientation of the camera 110 are the x-axis, y-axis, and z-axis, respectively, is referred to as the camera coordinate system. In addition, the Cartesian coordinate system in which the center position of the image captured by the camera 110 is the origin, the horizontal direction of the image is the x-axis, and the vertical direction of the image is the y-axis is referred to as the image coordinate system.

The relationship between the world coordinate system, camera coordinate system, and image coordinate system is shown in Figure 5. Ow is the origin of the world coordinate system, and Xw, Yw, and Zw respectively indicate the x-axis, y-axis, and z-axis in the world coordinate system. Oc is the origin of the camera coordinate system, and Xc, Yc, and Zc respectively indicate the x-axis, y-axis, and z-axis in the camera coordinate system. Oi is the origin of the image coordinate system, and Xi and Yi respectively indicate the x-axis and y-axis in the image coordinate system. Here, the equation for converting a three-dimensional position Pw = (Xw, Yw, Zw) in the world coordinate system to a three-dimensional position Pc = (Xc, Yc, Zc) in the camera coordinate system is expressed as follows.

R is the composition of the rotation matrices around each axis, and T is the translation matrix. In addition, the equation for converting a three-dimensional position Pc in the camera coordinate system to a two-dimensional position Pi = (Xi, Yi) in the image coordinate system according to the pinhole camera model is expressed as follows.

Here, λ is the scale factor of the image, f is the focal length, (s _x , s _y ) are the pixel size in the x-axis direction and the pixel size in the y-axis direction, respectively, (x _co , y _co ) is the image principal point position, and _ks is the shear coefficient that represents the inclination of y with respect to the x-axis in the image plane.

The matrix [R T] is called an external parameter that represents the three-dimensional position and three-dimensional orientation of the image capture device 110 in the world coordinate system, and the matrix A is called an internal parameter that represents the projection characteristics of the image capture device 110. Estimating the position and orientation of the image capture device 110 involves estimating the matrix [R T].

The position and orientation of the image capture device 110 in the world coordinate system can be estimated by solving the so-called PnP (Perspective n-Point) problem. The PnP problem is a problem of estimating the external parameters of the image capture device 110, assuming that the correspondence between multiple 3D points and 2D points and the internal parameters of the image capture device 110 are known. Mathematically, the external parameters can be found if there are six or more pairs of known corresponding points. Known position and orientation estimation methods include RANSAC (RANdom SAMple Consensus).

In this manner, in this embodiment, the position and orientation of the image capturing device 110 in the world coordinate system is estimated based on the two-dimensional positions of feature points in the captured image, the known three-dimensional positions of the feature points, and the above-mentioned internal parameters. Such technology for estimating the position and orientation of the image capturing device 110 from an image captured by the image capturing device 110 is a well-known technology, and therefore a detailed description thereof will be omitted. Note that the method for acquiring the position and orientation of the image capturing device 110 is not limited to a specific method. For example, the position and orientation of the image capturing device 110 may be acquired using a sensor such as a gyro sensor, a GPS sensor, an optical sensor, or an ultrasonic sensor, or the position and orientation of the image capturing device 110 may be acquired using a sensor and a captured image in combination.

In step S405, the acquisition unit 305 of the image processing device 109 acquires the three-dimensional position of the subject 111 in the world coordinate system. For example, the image capturing device 110 displays the captured image captured by itself on the display unit 205, and allows the user to select a target subject to be processed from among the subjects in the captured image. The user touches the target subject on the screen of the display unit 205 or operates the operation unit 204 to select the target subject, and performs a selection operation of the target subject. The image capturing device 110 then transmits information for identifying the selected target subject to the image processing device 109. In addition, for example, user information may be registered in the multi-view shooting system so that the image capturing device 110 and the target subject (player) can be managed. The user may log in to the multi-view shooting system from a mobile terminal or the like, and be able to specify the target subject at any time. In addition, there may be multiple target subjects. Below, a case in which the subject 111 is specified as the target subject will be described.

As described above, the image processing device 109 executes a computer program for generating a virtual viewpoint image, thereby estimating the three-dimensional position and three-dimensional posture of the subject 111 in the world coordinate system from the images captured by the cameras 101 to 108 at the same time. Therefore, in step S405, the acquisition unit 305 acquires the estimated three-dimensional position of the subject 111 in the world coordinate system.

In step S406, the calculation unit 306 of the image processing device 109 uses the above formulas (1) and (2) to calculate the two-dimensional position of the subject 111 in the image coordinate system from the three-dimensional position of the subject 111 in the world coordinate system acquired in step S405. If the two-dimensional position of the subject 111 in the image coordinate system is outside the range of the angle of view of the image capture device 110, the calculation unit 306 generates information necessary for an operation to bring the two-dimensional position within the range of the angle of view of the image capture device 110.

An example of the two-dimensional position of the subject 111 in the image coordinate system is shown in FIG. 6. Reference numeral 601 indicates the two-dimensional position of the subject 111 calculated in step S406, and reference numeral 602 indicates the range of the angle of view of the image capture device 110. The range of the angle of view of the image capture device 110 can be calculated from the above internal parameters.

As shown in FIG. 6, the acquired two-dimensional position 601 does not fall within the range 602 of the angle of view of the image capture device 110, and therefore the subject 111 is not included in the captured image captured with the angle of view of this range 602. In this embodiment, in such a case, the user is notified of the direction and extent to which the image capture device 110 should be oriented in order to include the subject 111 within the range of the angle of view. To this end, the calculation unit 306 obtains an intersection Ep between a straight line from the origin of the image coordinate system toward the two-dimensional position 601 and the end of the range 602 of the angle of view, and obtains a vector 603 from the intersection Ep toward the two-dimensional position 601. The direction of this vector 603 is the movement direction of the image capture device 110 required to place the two-dimensional position 601 within the range 602 of the angle of view of the image capture device 110. The length of this vector 603 is the amount of movement of the image capture device 110 required to place the two-dimensional position 601 within the range 602 of the angle of view of the image capture device 110.

In step S407, the transmission unit 307 of the image processing device 109 transmits the vector information (vector information) calculated by the calculation unit 306 of the image processing device 109 in step S406 to the imaging device 110. In step S408, the reception unit 308 of the imaging device 110 receives the vector information transmitted by the transmission unit 307 of the image processing device 109 in step S407.

In step S409, the display control unit 309 of the image capturing device 110 causes the display unit 205 to display notification information for informing the user of the amount and direction of movement of the image capturing device 110 required to bring the two-dimensional position of the subject 111 determined in step S406 within the range of the angle of view of the image capturing device 110, based on the vector information received by the receiving unit 308 of the image capturing device 110 in step S408.

7A shows an example of the display of notification information on the display unit 205. A captured image 700 is displayed on the display screen 701 of the display unit 205, and a surrounding area 702 surrounding the display area of the captured image 701 is an area for displaying an icon 703 for notifying the user of the amount and direction of movement of the imaging device 110 required to bring the two-dimensional position of the subject 111 within the range of the angle of view of the imaging device 110 when the two-dimensional position of the subject 111 exists outside the captured image 700 (i.e., outside the range of the angle of view of the imaging device 110). The icon 703 is located on a straight line extending from the center position of the captured image 700 in the direction of the above vector and is located within the area 702. In FIG. 7A, the icon 703 notifies the user that the two-dimensional position of the subject 111 exists on the extension of a straight line from the center position of the captured image 700 toward the position of the icon 703. In other words, a user who sees this icon 703 can understand that the direction from the center position of the captured image 700 toward the position of the icon 703 is "the direction of movement of the image capturing device 110 required to bring the two-dimensional position of the subject 111 within the range of the angle of view of the image capturing device 110."

In addition, the display form of the icon 703 notifies the user of "the amount of movement of the image capturing device 110 required to bring the two-dimensional position of the subject 111 within the range of the angle of view of the image capturing device 110." For example, the size of the icon 703 may be increased as the length of the above vector is smaller (the amount of movement is smaller), and the size of the icon 703 may be decreased as the length of the above vector is larger (the amount of movement is larger). In addition, the brightness or saturation of the icon 703 may be increased as the length of the above vector is smaller (the amount of movement is smaller), and the brightness or saturation of the icon 703 may be decreased as the length of the above vector is larger (the amount of movement is larger). In addition, two or more of the size, brightness, and saturation of the icon 703 may be changed according to the length (amount of movement) of the above vector. An example of the size and color combination of the icon 703 according to the length (amount of movement) of the above vector is shown in FIG. 7(b).

The user may also change the icon for each subject. For example, when the user selects a target subject, the user may select a corresponding icon. This allows the user to grasp the direction and amount of movement of the image capture device 110 for each subject, even if the two-dimensional positions of multiple subjects are outside the range of the angle of view of the image capture device 110.

The notification information is not limited to the icon as described above. For example, the icon 703 in FIG. 7(a) may be tilted to face the direction of movement. The shape of the icon is also not limited to the shape of the icon 703 shown in FIG. 7(a). The direction and amount of movement of the imaging device 110, identification information of the subject, and the like may be displayed as text information on the display unit 205.

As described above, according to this embodiment, it is possible to obtain the highly accurate two-dimensional position of a subject located outside the range of the angle of view of the image capture device 110, and to notify the user of the amount and direction of movement of the image capture device 110 required to bring the two-dimensional position within the range of the angle of view of the image capture device 110. As a result, even if the subject moves outside the range of the angle of view of the image capture device 110, the subject can be tracked with high accuracy, and the subject can be captured within the angle of view without losing sight of it.

Second Embodiment
In the following embodiments including this embodiment, the difference from the first embodiment will be described, and it will be assumed that the embodiments are the same as the first embodiment unless otherwise specified below. In the first embodiment, the image processing device 109 is configured to obtain the moving direction and moving amount of the image capturing device 110 and notify the image capturing device 110. With this, even if the calculation performance of the image capturing device 110 is low, the above processing can be performed. However, when multiple image capturing devices 110 are connected to the image processing device 109, the calculation load of the image processing device 109 becomes high. Therefore, in this embodiment, a configuration for the image capturing device 110 to obtain its own moving direction and moving amount will be described.

The block diagram of FIG. 8 shows an example of the functional configuration of each of the imaging device 110 and the image processing device 109. The processing performed by the imaging device 110 and the image processing device 109 having the example of the functional configuration shown in FIG. 8 will be described with reference to the flowchart of FIG. 9. Note that the following description will be made with the functional units shown in FIG. 8 as the subject of the processing. However, in reality, the functions of the functional units of the imaging device 110 are realized by the CPU 201 executing a computer program for causing the CPU 201 to realize the functions of the functional units of the imaging device 110. Also, the functions of the functional units of the image processing device 109 are realized by the CPU 209 executing a computer program for causing the CPU 209 to realize the functions of the functional units of the image processing device 109. Note that the functional units shown in FIG. 8 may be implemented in hardware.

In step S401, the acquisition unit 301 of the image capture device 110 acquires an image captured by the image capture unit 206. In step S901, the transmission unit 801 of the image processing device 109 transmits, as reference point information, information about an object in real space that is used as a reference when estimating the position and orientation of the image capture device 110 from the image captured by the image capture device 110 to the image capture device 110. An "object in real space that is used as a reference when estimating the position and orientation of the image capture device 110" is an object whose position does not change in real space (for example, structures such as goals and flags, or lines on a field). The reference point information includes the color and three-dimensional position of such an object.

In step S902, the receiving unit 802 of the image capturing device 110 receives the reference point information transmitted by the transmitting unit 801 of the image processing device 109 in step S901. In step S903, the estimation unit 803 of the image capturing device 110 estimates the position and orientation (three-dimensional position and three-dimensional orientation) of the image capturing device 110 in the world coordinate system based on the captured image acquired in step S401 and the reference point information received in step S902. The method of estimating the position and orientation in this step is the same as the estimation method in step S404 above.

In step S405, the acquisition unit 305 of the image processing device 109 acquires the three-dimensional position of the subject 111 in the world coordinate system. In step S904, the transmission unit 804 of the image processing device 109 transmits the three-dimensional position of the subject 111 in the world coordinate system acquired in step S405 to the imaging device 110.

In step S905, the receiving unit 805 of the image capturing device 110 receives the three-dimensional position of the subject 111 in the world coordinate system transmitted by the transmitting unit 804 of the image processing device 109 in step S904.

In step S906, the calculation unit 806 determines the two-dimensional position of the subject 111 in the image coordinate system in the same manner as in step S406 above. If the two-dimensional position of the subject 111 in the image coordinate system is outside the range of the angle of view of the image capture device 110, the calculation unit 806 generates information necessary for an operation to bring the two-dimensional position within the range of the angle of view of the image capture device 110.

In step S409, the display control unit 309 causes the display unit 205 to display notification information for informing the user of the amount and direction of movement of the image capturing device 110 required to bring the two-dimensional position of the subject 111 within the range of the angle of view of the image capturing device 110, as in the first embodiment.

In this way, according to this embodiment, even if multiple image capture devices 110 are connected to the image processing device 109, the load on the image processing device 109 can be reduced, and processing can be performed by many image capture devices 110.

[Third embodiment]
In the first and second embodiments, the user is notified of the amount and direction of movement of the image capturing device 110 required to bring the two-dimensional position of the subject 111 within the range of the angle of view of the image capturing device 110. This allows the user to grasp where the subject 111 is located outside the angle of view. However, if the subject 111 moves quickly, even if the position or attitude of the image capturing device 110 is changed to bring the subject 111 within the angle of view, the subject 111 is moving during that time, making it difficult to capture the subject 111 within the angle of view.

Therefore, in this embodiment, in addition to the amount and direction of movement of the image capture device 110 required to bring the two-dimensional position of the subject 111 within the range of the angle of view of the image capture device 110, the moving speed and direction of the subject 111 are notified to the user. The processing performed by the image capture device 110 and the image processing device 109 will be described with reference to the flowchart in FIG. 10.

In step S1001, the acquisition unit 305 acquires the three-dimensional position of the subject 111 in the world coordinate system a predetermined time before the present. The three-dimensional position acquired in step S1001 is the three-dimensional position of the subject 111 in the world coordinate system estimated a predetermined time before by execution of a computer program for generating a virtual viewpoint video. Note that this "predetermined time" may be set to the time required to calculate the movement speed of the subject 111. For example, if the subject 111 is a person, a period of about 2 to 5 seconds may be set. Alternatively, the predetermined time may be set shorter, and the calculated movement speed may be calculated by combining it with past results.

In step S1002, the calculation unit 306 performs the same processing as in step S406 above using the above formulas (1) and (2), thereby determining the two-dimensional position of the subject 111 in the image coordinate system from the three-dimensional position of the subject 111 in the world coordinate system acquired in step S1001. The calculation unit 306 then determines a vector having the two-dimensional position of the subject 111 a predetermined time ago determined in step S1002 as its start point and the latest two-dimensional position of the subject 111 determined in step S406 as its end point, as the "movement vector of the subject 111 (two-dimensional movement vector)." The direction of the movement vector represents the movement direction of the subject 111, and the length of the movement vector represents the movement speed of the subject 111.

Figure 11 shows an example of the movement vector of subject 111. In Figure 11, vector 1102, which has its start point at the two-dimensional position 1101 of subject 111 a predetermined time ago calculated in step S1002, and its end point at the latest two-dimensional position 601 of subject 111 calculated in step S406, is defined as the "movement vector of subject 111."

In step S1003, the transmission unit 307 transmits to the image capture device 110 information on the vector (vector information) calculated in step S406, as well as information on the movement vector of the subject 111 calculated in step S1002 (movement vector information). In step S1004, the reception unit 308 receives the vector information and movement vector information transmitted by the transmission unit 307 in step S1003.

In step S1005, the display control unit 309 causes the display unit 205 to display notification information for notifying the user of the amount and direction of movement of the image capturing device 110 required to bring the two-dimensional position of the subject 111 determined in step S406 within the range of the angle of view of the image capturing device 110, based on the vector information received in step S1004. Furthermore, the display control unit 309 causes the display unit 205 to display notification information for notifying the user of information related to the movement of the subject 111, based on the movement vector information received in step S1004.

An example of the display of notification information on the display unit 205 is shown in FIG. 12(a). In this embodiment, when the two-dimensional position of the subject 111 is outside the captured image 700 (outside the range of the angle of view of the image capturing device 110), in addition to the icon 703 described above, a movement information icon 1201 corresponding to the direction and length of the movement vector described above is displayed as notification information in the area 702.

In this embodiment, the display control unit 309 refers to the table shown in FIG. 12(b) and selects and displays a movement information icon corresponding to the direction and length of the movement vector. More specifically, the display control unit 309 selects and displays a movement information icon corresponding to a combination of the movement direction closest to the direction of the movement vector among eight directions (up, upper right, right, lower right, down, lower left, left, upper left) and the movement speed closest to the movement speed corresponding to the length of the movement vector among three types of movement speed (walking, running, full speed).

In the example of FIG. 12(b), for example, if the length of the movement vector is less than the first value, i.e., the subject is stationary, no arrow is displayed. If the length of the movement vector is equal to or greater than the first value and less than the second value, i.e., walking speed, a movement information icon with one arrow tip is displayed. If the length of the movement vector is equal to or greater than the second value and less than the third value, i.e., running speed, a movement information icon with two arrow tips is displayed. If the length of the movement vector is equal to or greater than the third value, i.e., full sprinting speed, a movement information icon with three arrow tips is displayed.

When displaying information about a movement vector, other information may be displayed in addition to or instead of the movement information icon. For example, the direction and length of the movement vector may be displayed as text information in addition to or instead of the movement information icon.

In this way, according to this embodiment, the user can predict the destination of the subject and change the position and orientation of the imaging device 110, allowing the user to follow and capture the subject.

The numerical values, processing timing, processing order, display screen configuration, information to be displayed, display format, processing subject, etc. used in the above explanation are given as examples to provide a concrete explanation, and are not intended to be limiting.

Furthermore, any or all of the embodiments described above may be used in appropriate combination.Furthermore, any or all of the embodiments described above may be used selectively.

Other Embodiments
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

109: Image processing device 110: Image capture device 301: Acquisition unit 302: Transmission unit 303: Reception unit 304: Estimation unit 305: Acquisition unit 306: Calculation unit 307: Transmission unit 308: Reception unit 309: Display control unit

Claims (14)

An imaging device,
an acquisition means for determining a two-dimensional position of the subject in a coordinate system based on the images captured by the photographing device, based on the three-dimensional position and three-dimensional orientation of the photographing device and the three-dimensional position of the subject estimated from the images captured by a plurality of cameras arranged to surround the subject;
and if the two-dimensional position is outside the range of the angle of view of the imaging device, obtaining information necessary for an operation to bring the two-dimensional position within the range of the angle of view of the imaging device, and performing a display based on the obtained information. The imaging device according to claim 1, characterized in that, if the two-dimensional position is outside the range of the angle of view of the imaging device, the display control means determines the amount of movement and direction of movement of the imaging device to bring the two-dimensional position within the range of the angle of view of the imaging device, and performs display based on the determined amount of movement and direction of movement. The imaging device according to claim 2, characterized in that the display control means determines an intersection between a straight line passing through the origin of the coordinate system and the two-dimensional position and an end of the range of the angle of view of the imaging device, determines a vector from the intersection point toward the two-dimensional position, and determines the length and direction of the determined vector as the amount of movement and the direction of movement, respectively. The imaging device according to claim 2 or 3, characterized in that the display control means performs display based on the amount of movement and the direction of movement in a surrounding area surrounding a display area in which the captured image is displayed. 5. The photographing device according to claim 1, wherein the acquisition means acquires the three-dimensional position of the subject from an external device that estimates the three-dimensional position of the subject based on images photographed by the multiple cameras. an acquisition means for determining a two-dimensional position of the subject in a coordinate system based on the images captured by the imaging device, based on the three-dimensional position and three-dimensional orientation of the imaging device and the three-dimensional position of the subject estimated from images captured by a plurality of cameras arranged to surround the subject;
and a transmission means for, if the two-dimensional position is outside the range of the angle of view of the image capturing device, obtaining information necessary for an operation to bring the two-dimensional position within the range of the angle of view of the image capturing device, and transmitting the obtained information to the image capturing device. The image processing device according to claim 6, characterized in that the acquisition means generates a three-dimensional shape model of the subject based on the images captured by the multiple cameras, and generates an image of the space from a virtual viewpoint using the three-dimensional shape model. Furthermore,
a means for calculating a two-dimensional movement vector of the object based on the two-dimensional position of the object calculated by the calculation means,
8. The image processing apparatus according to claim 6, wherein said transmission means transmits information on said motion vector to said image capturing apparatus in addition to said information. A method for controlling an imaging device, comprising:
an acquisition step in which an acquisition means of the image capturing device acquires a two-dimensional position of the subject in a coordinate system based on the images captured by the image capturing device, based on the three-dimensional position and three-dimensional attitude of the image capturing device and the three-dimensional position of the subject estimated from the images captured by a plurality of cameras arranged to surround the subject;
and a display control step in which, if the two-dimensional position is outside the range of the angle of view of the photographing device, a display control means of the photographing device obtains information necessary for an operation to bring the two-dimensional position within the range of the angle of view of the photographing device, and performs a display based on the obtained information. A control method for an image processing device, comprising:
an acquisition step in which an acquisition means of the image processing device determines a two-dimensional position of the subject in a coordinate system based on the images captured by the imaging device, based on the three-dimensional position and three-dimensional attitude of the imaging device and the three-dimensional position of the subject estimated from the images captured by a plurality of cameras arranged to surround the subject;
a transmission step in which, if the two-dimensional position is outside the range of the angle of view of the photographing device, a transmission means of the image processing device obtains information necessary for an operation to bring the two-dimensional position within the range of the angle of view of the photographing device, and transmits the obtained information to the photographing device. A photographing system having a photographing device and an image processing device,
The image processing device includes:
an acquisition means for determining a two-dimensional position of the subject in a coordinate system based on the images captured by the imaging device, based on the three-dimensional position and three-dimensional orientation of the imaging device and the three-dimensional position of the subject estimated from images captured by a plurality of cameras arranged to surround the subject;
a transmitting means for, if the two-dimensional position is outside the range of the angle of view of the image capture device, obtaining information required for an operation to bring the two-dimensional position within the range of the angle of view of the image capture device, and transmitting the obtained information to the image capture device;
The imaging device is
a receiving means for receiving the information transmitted by the transmitting means;
and a display control means for performing a display based on the information received by the receiving means. A photographing system having a photographing device and an image processing device,
The imaging device is
an acquisition means for acquiring a two-dimensional position of the subject in a coordinate system based on an image captured by the photographing device, based on the three-dimensional position and three-dimensional attitude of the photographing device and the three-dimensional position of the subject;
a display control means for, if the two-dimensional position is outside the range of the angle of view of the image capture device, obtaining information necessary for an operation to bring the two-dimensional position into the range of the angle of view of the image capture device, and performing a display based on the obtained information;
The image processing device includes:
An estimation means for estimating a three-dimensional position of a subject based on images captured by a plurality of cameras;
and a means for transmitting the three-dimensional position of the subject estimated by the estimation means to the imaging device. A computer program for causing a computer to function as each of the means of the imaging device according to any one of claims 1 to 5. A computer program for causing a computer to function as each of the means of the image processing device according to any one of claims 6 to 8 .

Images