JP2019067419A - Image processing device, image processing method, and program - Google Patents

Abstract

To provide an image processing device capable of obtaining well-balanced image quality in a subject existing in various positions in a photographic space, when generating a reconstructed image from a virtual viewpoint.SOLUTION: An imaging system 100 acquires a plurality of image data obtained on the basis of imaging by a plurality of imaging apparatuses 102 installed so as to image a first position 101 and a plurality of image data obtained on the basis of imaging by a plurality of imaging apparatuses 104 installed so as to image a second position 103. The three-dimensional shape data of the subject used for the generation of a virtual viewpoint image is generated on the basis of the plurality of image data of at least one of images obtained by imaging the first position and the second position. At least one of the plurality of imaging apparatuses for imaging the first position is installed closer to the second position than the first position. At least one of the plurality of imaging apparatuses for imaging the second position is installed closer to the first position than the second position.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  There is known a technique for reconstructing an image obtained when an object is observed from an arbitrary virtual viewpoint from images obtained by imaging the object with a plurality of imaging devices. For example, Patent Document 1 discloses the following method. First, a three-dimensional model of the subject is created using captured images of the subject captured by a plurality of cameras. Next, a texture image of each position on the three-dimensional model is generated by blending a plurality of captured images. Based on the three-dimensional model and the texture image thus generated, it is possible to reconstruct an image from a virtual viewpoint in which no camera is arranged.

  Patent Document 1 shows an example in which 40 cameras are installed toward a subject so as to surround the subject. On the other hand, Patent Document 2 proposes a method of further using a zenith camera installed on a subject toward the subject in order to improve the accuracy of the reconstructed image.

Unexamined-Japanese-Patent No. 2010-020487 JP, 2010-039501, A

  The methods of Patent Documents 1 and 2 are suitable for generating a reconstructed image of an object present at a predetermined position. However, the methods of Patent Documents 1 and 2 have problems when applied to a case where a subject is scattered and moving in a wide space (for example, a competition field), such as sports photography. For example, when each camera is installed toward the center of the field according to Patent Document 2, a high quality reconstructed image can be generated for an object located at the center of the field. On the other hand, with regard to the subject at the end of the field, the number of cameras that capture this subject is limited, and a reconstructed image may not be obtained or the quality may be degraded.

  An object of the present invention is to provide an image processing apparatus capable of obtaining balanced image quality with respect to objects present at various positions in a photographing space when generating a reconstructed image from a virtual viewpoint. I assume.

In order to achieve an object of the present invention, for example, an image processing apparatus of the present invention has the following configuration. That is,
A plurality of image data obtained based on imaging by a plurality of imaging devices installed to image a first position, and a plurality of images installed to image a second position different from the first position Acquisition means for acquiring a plurality of image data obtained based on imaging by the imaging device of
Based on imaging by a plurality of image data obtained based on imaging by a plurality of imaging devices installed to image the first position and imaging by a plurality of imaging devices installed to image the second position And shape generation means for generating three-dimensional shape data of a subject used for generating a virtual viewpoint image based on at least one of a plurality of image data among a plurality of image data obtained.
At least one imaging device among a plurality of imaging devices installed to image the first position is installed closer to the second position than the first position,
At least one imaging device among a plurality of imaging devices installed to image the second position is installed closer to the first position than the second position. .

  It is possible to provide an image processing apparatus in which balanced image quality can be obtained for objects present at various positions in the imaging space when a reconstructed image from a virtual viewpoint is generated.

FIG. 1 is a view showing an example of the arrangement of an imaging system according to the first embodiment; FIG. 7 is a view showing an example of the arrangement of an imaging system according to the second embodiment; FIG. 7 is a view showing an example of the arrangement of an imaging system according to the third embodiment; FIG. 7 is a view showing an example of the arrangement of an imaging system according to the third embodiment; FIG. 14 is a diagram showing an example of the hardware configuration of an image processing apparatus according to a fourth embodiment. FIG. 13 is a block diagram showing an example of the functional arrangement of an image processing apparatus according to the fourth embodiment; 10 is a flowchart illustrating an example of processing in an image processing method according to a fourth embodiment.

  Hereinafter, embodiments of the present invention will be described based on the drawings. However, the scope of the present invention is not limited to the following embodiments. For example, in order to make it possible to generate a reconstructed image for an object present anywhere in the imaging space, a camera with a short focal length (wide angle of view) may be installed so as to surround the imaging space it can. On the other hand, if the focal length of the camera is shortened, the quality of the reconstructed image may be reduced because the subject on the captured image becomes smaller.

  In the following embodiments, the camera is installed in consideration of reducing the imaging area where the quality deterioration occurs due to the small number of cameras imaging the subject (hereinafter referred to as improvement of the degree of freedom of the viewpoint). . Further, in the following embodiments, the camera is disposed in consideration of suppression of quality deterioration (hereinafter, referred to as improvement in image quality) caused by the small object image on the captured image.

Embodiment 1
An imaging system 100 according to the first embodiment will be described with reference to FIG. The imaging system 100 includes a plurality of imaging devices 102 and 104, and acquires a captured image using each of the plurality of imaging devices 102 and 104 in order to generate a free viewpoint image. A free viewpoint image refers to a reconstructed image from an arbitrarily set virtual viewpoint. The imaging devices 102 and 104 are disposed toward the imaging space so as to surround the imaging space in which the subject is present. For example, the imaging devices 102, 104 can be placed to surround the imaging field 108 (e.g., a competition ground).

  The plurality of imaging devices 102 and 104 included in the imaging system 100 includes a first imaging device group 102 including one or more imaging devices, and a second imaging device group 104 including one or more imaging devices. including. The first imaging device group 102 is installed toward the first gaze point 101. Further, the second imaging device group 104 is installed toward the second gaze point 103 different from the first gaze point 101. The gaze point is an arbitrary point set on the imaging space. In the present embodiment, the subject is present on the imaging field 108, and the gaze point is also set on the imaging field 108. Specifically, the gaze point is located at the intersection of the optical axis of the imaging apparatus and the imaging field 108. However, the fixation point may be set in the air.

  In one embodiment, the first imaging device group 102 is installed toward the first gaze point 101 so that the first fixation point 101 is in focus, and the second imaging device group 104 is a second imaging device group. It is installed toward the second fixation point 103 so that the fixation point 103 is in focus. However, for example, when the depth of field of the imaging device is deep, it is not necessary to accurately bring the imaging device into focus.

  The first imaging device group 102 is installed so that the first area 105 on the imaging field is in the field of view. In other words, the first area 105 is a common visual field area of the first imaging device group 102. In addition, the second imaging device group 104 is installed so that the second area 106 on the imaging field falls within the field of view. In other words, the second area 106 is a common visual field area of the second imaging device group 104. According to such a configuration, the first imaging device group 102 can capture an object on at least the first area 105. In addition, the second imaging device group 104 can capture an object on at least the second area 106.

  The first imaging device group 102 and the second imaging device group 104 are installed so as to surround the imaging field 108, respectively. Also, the first imaging device group 102 is installed so as to surround the first fixation point 101. For example, in FIG. 1, at least one imaging device 102 a of the first imaging device group 102 is closer to the second region 106 than the first region 105. The imaging device 102 a is closer to the second gaze point 103 than the first gaze point 101. According to such a configuration, the first imaging device group 102 can capture an object on the first area 105 from various directions. On the other hand, the second imaging device group 104 is installed so as to surround the second fixation point 101. For example, in FIG. 1, at least one imaging device 104 a of the second imaging device group 104 is closer to the first region 105 than the second region 106. Further, the imaging device 104 a is closer to the first fixation point 101 than the second fixation point 103. According to such a configuration, the second imaging device group 104 can capture an object on the second area 106 from various directions.

  In the present embodiment, the first area 105 and the second area 106 cover substantially the entire imaging field 108. For example, the ratio of the sum of the first area 105 and the second area 106 to the imaging field 108 is 80% or more in one embodiment, 90% or more in another embodiment, and another embodiment. And 95% or higher, and in a further embodiment 100%. With such a configuration, at least the first imaging device group 102 or the second imaging device group 104 can capture a captured image from various directions with respect to subjects present at various positions of the imaging field 108. it can. Therefore, it is possible to generate reconstructed images of subjects present at various positions of the imaging field 108, that is, to improve the degree of freedom of the viewpoint. In another embodiment, it is also possible to provide an imaging device group installed so that a further area on the imaging field 108 falls within the field of view. In this case, the imaging system can be configured such that the common viewing area of each imaging device group covers substantially the entire imaging field 108.

  Further, in the present embodiment, the common area 107 between the first area 105 and the second area 106 is small. The ratio of the common area 107 to the sum of the first area 105 and the second area 106 is 30% or less in one embodiment, 20% or less in another embodiment, and 10 in another embodiment. % Or less, and in a further embodiment 5% or less. Furthermore, in the present embodiment, the center of gravity of the first area 105 is not included in the second area 106. Also, the center of gravity of the second area 106 is not included in the first area 105. Thus, by narrowing the first area 105 and the second area 106, it is possible to narrow the angle of view of the first imaging device group 102 and the second imaging device group 104, that is, the image quality can be reduced. An improvement can be realized.

  In one embodiment, the number of imaging devices included in the first imaging device group 102 may be determined according to the size of the first area 105. That is, as the first region 105 is larger, the number of imaging devices can be increased. The same applies to the second region 106.

  According to the prior art, the fixation point of each imaging device is set at the center of the imaging field 108. For this reason, the subject existing in the peripheral portion of the shooting field 108 can be imaged only by a small number of imaging devices, and a reconstructed image of the subject may not be generated. On the other hand, according to the present embodiment, in a wider area of the imaging field 108, the subject can be imaged by a predetermined number or more of imaging devices, and it becomes possible to generate a reconstructed image of the subject. That is, the degree of freedom of the viewpoint can be improved. According to the present embodiment, the number of imaging devices capable of imaging an object present at the center of the imaging field 108 may be reduced compared to the prior art, but still generates reconstructed images of sufficient quality for this object. It is thought that can be done. That is, the image quality can be maintained.

Second Embodiment
In the first embodiment, the plurality of imaging devices 102 and 104 are installed to face either the first or second gaze point 101 or 103. The installation method of the imaging device is not limited to this. For example, by installing a plurality of imaging devices in various directions in the imaging space, a subject can be imaged by a predetermined number or more of imaging devices in a wider area in the imaging space, that is, the degree of freedom of the viewpoint Can be improved. Moreover, according to such a configuration, it is not necessary to widen the angle of view of the imaging device so as to cover the entire imaging space, that is, improvement in image quality can be realized. Thus, the first imaging device group including one or more imaging devices installed toward the first fixation point, and the second imaging point different from the first fixation point are installed By using one or more imaging devices, it is possible to achieve both the freedom of the viewpoint and the image quality. In the second embodiment, an example of such a configuration will be described.

  An imaging system 200 according to the second embodiment will be described with reference to FIG. The imaging system 200 includes a plurality of imaging devices 202, 204, and 206 as in the first embodiment, and acquires a plurality of captured images for generating a free viewpoint image using each of the imaging devices. The differences from the first embodiment will be described below.

  The imaging system 200 includes a first imaging device group 202 including one or more imaging devices installed toward the first fixation point 201, and a second fixation point 203 different from the first fixation point 201. And a second imaging device group 204 installed toward the The imaging system 200 further includes a third imaging device group 206 installed toward a third fixation point 205 different from the first and second fixation points 201 and 203. The first, second, and third gaze points 201, 203, and 205 exist on a line segment 210 provided on the imaging space. In the example of FIG. 2, the first, second, and third gaze points 201, 203, 205 exist on a line segment 210 provided on the imaging field 108.

  According to such a configuration, as compared with the case where the fixation point of each imaging device is at the center of the imaging field 108 as in the prior art, the area where the object can be imaged by a predetermined number or more of imaging devices Can spread. Therefore, the degree of freedom of the viewpoint can be improved. In particular, when the subject tends to move in a certain direction, by setting the line segment in accordance with this direction, the degree of freedom of the viewpoint can be maintained even if the subject moves.

  The first, second, and third imaging device groups 202, 204, and 206 can be arranged such that an object at each point on the line segment 210 can be viewed by at least two or more cameras. In FIG. 2, the gaze point 201 exists on a line connecting the gaze points 203 and 205. In this case, each imaging device can be installed such that the imaging device 202 faces the gaze point 201, the imaging device 204 faces the gaze point 203, and the imaging device 206 faces the gaze point 205. Here, the imaging device 204 near the fixation point 205 which is one end of the line segment is installed toward the fixation point 203 which is the other end of the line segment. Further, the imaging device 206 near the fixation point 203 which is one end of the line segment is installed toward the fixation point 205 which is the other end of the line segment. As described above, by setting the imaging device to a distant gaze point, it is possible to capture an object by a plurality of imaging devices in a wider area, and the degree of freedom of the viewpoint can be improved.

  Although the case where one line segment 210 is provided is shown in FIG. 2, two or more line segments can be set in the imaging space. For example, by providing a new line segment orthogonal to the line segment 210 on the shooting field 108 and installing an imaging device toward each gaze point on the new line segment, a wider range on the shooting field 108 It is possible to generate a reconstructed image of the subject.

  Further, in FIG. 1, the fixation point of the imaging devices constituting the first imaging device group 102 may be shifted. For example, in the first imaging device group 102, one or more imaging devices installed toward the first gaze point 101, and one or more installed toward one end of the line segment including the first gaze point 101. And one or more imaging devices installed toward the other end of the line segment. The same applies to the second imaging device group 104.

Modification of Embodiments 1 and 2
In the first and second embodiments, the focal length (field angle) of each imaging device may be the same. On the other hand, by changing the focal length (field angle) of each imaging device, it is possible to maintain the image quality of the reconstructed image regardless of the position of the virtual viewpoint.

  For example, in the example of FIG. 1, the case of generating a reconstructed image of a subject present at the gaze point 101 will be considered. When the virtual viewpoint is on the left side (the imaging device 102b side), an image captured by the imaging device 102b (and an imaging device located on the left side of the subject) largely contributes to the reconstructed image. On the other hand, when the virtual viewpoint is on the right side (the imaging device 102a side), an image captured by the imaging device 102a (and an imaging device located on the right side of the subject) largely contributes to the reconstructed image. Here, the distance between the imaging device 102a and the subject is shorter than the distance between the imaging device 102b and the subject. Therefore, when the angle of view of the imaging devices 102a and 102b is the same, the subject becomes smaller in the captured image by the imaging device 102a as compared with the image captured by the imaging device 102b. Therefore, the image quality of the subject may vary depending on the position of the virtual viewpoint.

  In one embodiment, the focal length (field angle) of the imaging device can be changed according to the distance between the imaging device and the gaze point. For example, as the distance between the imaging device and the gaze point is longer, the focal length of the imaging device can be made longer (the angle of view can be narrowed). According to the example of FIG. 1, the first imaging device group 102 includes a fourth imaging device 102a and a fifth imaging device 102b having different angles of view. The fifth imaging device 102 b has a wide angle of view and is close to the first gaze point 101 as compared with the fourth imaging device 102 a.

Although the setting method of a specific focal distance (field angle) is not particularly limited, for example, it is set so that the size of the subject present in the vicinity of the fixation point becomes approximately the same on the captured image obtained by each imaging device It can be performed. For example, the focal length of the imaging device can be determined according to the following equation.
(Focal length of imaging device A) = (focal length of imaging device B) × (distance between imaging device A and its gaze point) / (distance between imaging device B and its gaze point)

  This method is also applicable to the example of FIG. That is, the focal distances (angles of view) of the imaging devices 204 and 206 are determined according to the distance between the imaging device 204 and its gaze point 203 and the distance between the imaging device 206 and its gaze point 205. Can.

Third Embodiment
In the first and second embodiments, the configuration in which the fixation points of the imaging devices are dispersed has been described. In the third embodiment, a configuration for changing the angle of view of each imaging device will be described. The imaging system 300 according to the third embodiment includes a plurality of imaging devices 302 and 304 as in the first embodiment, and acquires a plurality of captured images for generating a free viewpoint image using each of the imaging devices 302 and 304. The differences from the first embodiment will be described below.

  The imaging system 300 according to the third embodiment includes a first imaging device group 302 including one or more imaging devices, and one or more imaging devices having a wider angle of view than the first imaging device group. And a second imaging device group 304. According to such a configuration, the image quality of the area covered by the first imaging device group 302 can be improved. In addition, since a wide area in the imaging space is covered by the second imaging device group 304, the degree of freedom of the viewpoint can also be improved.

  An example of such a configuration will be described with reference to FIG. The first imaging device group 302 is installed such that the first area 301 on the imaging field 108 falls within the field of view. In other words, the first area 301 is a common visual field area of the first imaging device group 302. In addition, the second imaging device group 304 is installed so that the second area 305 on the imaging field 108 falls within the field of view. In other words, the second area 305 is a common visual field area of the second imaging device group 304. Here, the first area 301 is included in the second area 305. FIG. 3 shows an image 306 captured by the imaging device 302 a included in the first imaging device group 302 and an image 307 captured by the imaging device 304 a included in the first imaging device group 304. .

  According to such a configuration, the first imaging device group 302 can capture an object on the first area 301 from various directions. In addition, it is possible to narrow the angle of view of the first imaging device group 302, that is, to improve the image quality. On the other hand, the image quality may be limited because the second imaging device group 304 has a wide angle of view, but it is possible to image the subject on the second area 305 from various directions. Can be secured. As described above, according to the configuration of FIG. 3, it is possible to obtain the degree of freedom and the image quality of the viewpoint that is well balanced as a whole.

  Another configuration example will be described with reference to FIG. The first imaging device group 402 is installed so that the first area 401 provided on the imaging field 108 falls within the field of view. In other words, the first area 401 is a common visual field area of the first imaging device group 402. The second imaging device group is configured by imaging devices having a wider angle of view than the first imaging device group 402. Although FIG. 4 shows a second imaging device group configured by one imaging device 404a, the second imaging device group may include two or more imaging devices.

  The second imaging device group can be installed toward a gaze point provided outside the imaging field 108. For example, in FIG. 4, the second imaging device group is installed toward the audience seat. In FIG. 4, as an example, an image 403 captured by one imaging device 402 a of the first imaging device group 402 and an image captured by one imaging device 404 a of the second imaging device group And 405 are shown. The second imaging device group can be installed so as to be able to image a subject area other than the first area 401. For example, the second imaging device group can be installed and its angle of view can be adjusted so that the entire scene (entire audience seat) is covered. According to such a configuration, it is possible to obtain a reconstructed image of an object outside the imaging field 108 such as a spectator seat.

  Thus, even with the configuration of FIG. 4, by combining the imaging device group having a narrower angle of view and the imaging device group having a wider angle of view, it is possible to obtain the degree of freedom and image quality of the viewpoint balanced as a whole. be able to.

Fourth Embodiment
In Embodiments 1 to 3, an example of an imaging system for generating a free viewpoint image has been described. In the present embodiment, an image processing apparatus that generates a free viewpoint image and a processing method thereof will be described using a captured image obtained by an imaging system. In the present embodiment, mainly, a method of generating a reconstructed image in which one moving subject existing in the imaging space is captured will be described. However, a similar method can also be used to generate a reconstructed image in which two or more subjects appear. Further, the subject may be a moving object such as a person, or may be a non-moving object such as a floor or a background.

  The image processing apparatus according to the present embodiment may be, for example, a computer including a processor and a memory. FIG. 5 shows an example of the hardware configuration of the image processing apparatus 500 according to the present embodiment. The image processing apparatus 500 includes a CPU 501, a main memory 502, a storage unit 503, an input unit 504, a display unit 505, an external I / F unit 506, and a bus 507.

  The CPU 501 executes arithmetic processing and various programs. The main memory 502 provides the CPU 501 with programs, data, work areas and the like necessary for processing. The storage unit 503 is, for example, a hard disk or a silicon disk, and stores various programs or data. The input unit 504 is, for example, a keyboard, a mouse, an electronic pen, or a touch panel, and receives an operation input from a user. The display unit 505 is, for example, a display, and displays a GUI or the like according to the control of the CPU 501. An external I / F unit 506 is an interface with an external device. In the present embodiment, the external I / F unit 506 is connected to the imaging device group via the LAN 508, and transmits and receives video data, control signal data, and the like to and from the imaging device group. A bus 507 connects the above-described units to transfer data.

  For example, the operation of each part described below as shown in FIG. 6 can be realized as follows. That is, a program corresponding to the operation of each unit stored in a computer readable storage medium such as the storage unit 503 is expanded in the main memory 502. The operation of each unit described below can be realized by the CPU 501 operating according to this program. However, the operation of all or part of the units described later may be realized by dedicated hardware such as an ASIC.

  The image processing apparatus 500 is connected to the first imaging device group 509 and the second imaging device group 510, and constitutes an image processing system 511. The imaging device groups 509 and 510 start and stop shooting, change settings (such as shutter speed or aperture), and transfer shooting data in accordance with control signals from the image processing device 500. The imaging device groups 509 and 510 can be arranged around the imaging space according to, for example, the first to third embodiments.

  FIG. 6 shows a functional configuration of the image processing apparatus 500. As shown in FIG. 6, the image processing apparatus 500 includes an image acquisition unit 610, a viewpoint setting unit 620, a shape estimation unit 630, and an image generation unit 640.

  The image acquisition unit 610 acquires a plurality of captured images obtained by imaging a subject by each of the imaging device groups 509 and 510 configured by a plurality of imaging devices. The image acquisition unit 610 controls the plurality of imaging devices 509 and 510 via the LAN 508 to cause the imaging device groups 509 and 510 to simultaneously capture an object, and to transmit the obtained captured image to the image acquisition unit 610. Can. A moving image can also be acquired by the imaging device group 509, 510 performing imaging continuously. In this case, the image processing apparatus 500 can generate a moving image composed of reconstructed images by performing processing using the captured image of each frame.

  The viewpoint setting unit 620 sets a virtual viewpoint of the reconstructed image. Specifically, the viewpoint setting unit 620 can set the three-dimensional position of the virtual viewpoint and its attitude (or the optical axis direction). The viewpoint setting unit 620 can also set the angle of view, resolution, and the like of the virtual viewpoint. The viewpoint setting unit 620 can perform setting of the virtual viewpoint based on a user instruction input via the input unit 504.

  The shape estimation unit 630 estimates the shape of the subject based on the captured image acquired by the image acquisition unit 610. More specifically, the shape estimation unit 630 cuts out a desired subject region from each captured image, and estimates the three-dimensional position and the shape of the subject based on the obtained image. In the present embodiment, the positional relationship between the imaging device groups 509 and 510 is known, and is recorded in advance in the storage unit 503 or the like. A method for estimating the three-dimensional position and shape of the subject based on the captured image of the subject obtained by the imaging device groups 509 and 510 is known, and any method can be adopted. For example, a three-dimensional model of an object can be generated using the visual volume tolerance method or stereo matching method described in Patent Document 1.

  The shape estimation unit 630 can estimate the shape of the subject using all the captured images obtained by the plurality of imaging devices 509 and 510. On the other hand, the shape estimation unit 630 can also estimate the shape of the subject using a captured image captured by the selected imaging device group selected from the plurality of imaging devices 509 and 510. In this case, the shape estimation unit 630 selects a selected imaging device group used for estimating the shape of the subject from the plurality of imaging devices 509 and 510. Then, the shape estimation unit 630 estimates the shape of the subject using the captured image obtained by the selected imaging device group. In the present embodiment, the captured image obtained by the non-selected imaging device group which is not selected is not used for estimation of the shape of the subject. As can be understood from the fact that the image generation unit 640 described later estimates the color of the subject using the captured images obtained by the non-selected imaging device group, the captured image including the subject has the shape of the subject. It may not be used for estimation.

  In the present embodiment, the shape estimation unit 630 selects the selected imaging device group according to the arrangement of the virtual viewpoints. For example, the shape estimation unit 630 can select the selected imaging device group based on the relationship between the gaze point of the virtual viewpoint and the gaze point of the imaging device. Here, the shape estimation unit 630 can select an imaging device group having a gaze point closer to the fixation point of the virtual viewpoint as a selected imaging device group. As an example, according to FIG. 1, there are a first imaging device group 102 installed toward the first fixation point 101 and a second imaging device group 104 installed toward the second fixation point 103. The case will be described. If the gaze point of the virtual viewpoint is closer to the second gaze point 103 than the first gaze point 101, the shape estimation unit 630 selects the second imaging device group 104 as the selected imaging device group. When the gaze point of the virtual viewpoint is closer to the first gaze point 101 than the second gaze point 103, the shape estimation unit 630 selects the first imaging device group 102 as the selected imaging device group. As described above, the shape estimation unit 630 is provided for the first imaging device group 102 installed toward the first gaze point 101 as the selected imaging device group and the second installed for the second gaze point 103. It is possible to switch which of the imaging device group 104 is selected. This switching is performed according to which of the second gaze point 103 and the first gaze point 101 the fixation point of the virtual viewpoint is closer.

  Further, according to FIG. 2, a case where there are imaging devices installed toward different gazing points will be described. In this case, the shape estimation unit 630 can select an imaging device group having a gaze point within a predetermined range from the gaze point of the virtual viewpoint as a selected imaging device group. The shape estimation unit 630 can also select, as a selected imaging device group, a predetermined number of imaging device groups having a fixation point closer to the fixation point of the virtual viewpoint.

  As another example, the shape estimation unit 630 can select the selected imaging device group based on the relationship between the gaze point of the virtual viewpoint and the common viewing area of the imaging device group. For example, when the gaze point of the virtual viewpoint is included in the common view area of the imaging device group, the shape estimation unit 630 can select the imaging device group as a selected imaging device group. As an example, according to FIG. 3, the case where there is a first imaging device group 302 having a common visual field region 301 and a second imaging device group 304 having a common visual field region 305 will be described. When the gaze point of the virtual viewpoint is included in the common visual field area 301, the shape estimation unit 630 selects the first imaging device group 302 as the selected imaging device group. When the fixation point of the virtual viewpoint is not included in the common visual field area 301 but is included in the common visual field area 305, the shape estimation unit 630 selects the second imaging device group 304 as a selected imaging device group. Here, priority can be set to the imaging device group, and in this example, the first imaging device group 302 is selected with priority over the second imaging device group 304. Here, the shape estimation unit 630 preferentially selects the first imaging device group 302 with a narrower angle of view. As described above, according to whether or not the gaze point of the virtual viewpoint is included in the common visual field area 301, the shape estimation unit 630 sets the first imaging device group installed so that the common visual field area 301 falls within the visual field. It is possible to switch whether or not to select 302 as the selected imaging device group.

  As described above, by estimating the shape of the subject using the captured images captured by the selected imaging device group, the number of captured images used for the estimation process can be reduced. Therefore, the load of estimation processing can be reduced. On the other hand, since it is highly possible that the subject included in the reconstructed image appears in the captured image obtained by the selected imaging device group selected according to the arrangement of the virtual viewpoint, the three-dimensional shape of this subject is highly accurate. And maintain the image quality. Here, although the method of selecting the imaging device group based on the fixation point of the virtual viewpoint has been described, the present invention is not limited to this method. For example, the shape estimation unit 630 may select an imaging device group in which a gaze point exists in the visual field region of the virtual viewpoint, or selects an imaging device group having a common visual field region overlapping the visual field region of the virtual viewpoint. It is also good.

  Further, although the method of selecting the imaging device group based on the arrangement of virtual viewpoints has been described here, the present invention is not limited to this method. For example, the imaging device group can also be selected based on the relationship between the fixation point or the common visual field area of the imaging device group and the position of the subject. As an example, for a subject close to the first gaze point 101 or the common visual field area 105, the shape can be estimated using a captured image captured by the first imaging device group 102. The shape of the subject close to the second gaze point 103 or the common visual field area 106 can be estimated using the captured image captured by the second imaging device group 104 or 304.

  The image generation unit 640 estimates the color of the subject using both the captured image obtained by the selected imaging device group and the captured image obtained by the non-selected imaging device group not selected as the selected imaging device group Do. Then, the image generation unit 640 generates a reconstructed image from a virtual viewpoint based on the estimated shape and color of the subject. In the example of FIG. 1, when the gaze point of the virtual viewpoint is present in the vicinity of the gaze point 101, the shape estimation unit 630 selects the first imaging device group 102 and does not select the second imaging device group 104. On the other hand, since the imaging device 104a is close to the fixation point of the virtual viewpoint, in the captured image obtained by the imaging device 104a, there is a high possibility that the subject appearing in the reconstructed image is large. For this reason, in the present embodiment, when estimating the color of the subject, the quality of the reconstructed image is improved by using also the captured image obtained by the non-selected imaging device group.

  A method of estimating the color of the subject from the captured image and the shape of the subject is known, and the image generation unit 640 can generate a reconstructed image using any method. For example, the three-dimensional position and shape of the subject are obtained by the shape estimation unit 630, and by using the position, posture, angle of view, and resolution information of each imaging device group 509, 510, It is possible to identify pixels on the corresponding captured image. The color of each position on the subject can be estimated based on the color information of the pixel thus identified.

  As an example, a method of generating a reconstructed image using a distance map will be briefly described. First, the image generation unit 640 generates a distance map of the subject viewed from the virtual viewpoint, using the position and shape information of the subject, and the position and orientation information of the virtual viewpoint. Next, with respect to each pixel of the reconstructed image, the image generation unit 640 specifies the position of the subject shown in this pixel by referring to the distance map, and further refers to the picked-up image, the color information of the subject shown in this pixel get. Thus, the image generation unit 640 can generate a reconstructed image.

  As shown in FIG. 4, when an image outside the shooting field 108 such as a spectator seat is captured using the imaging device group 404a, the following processing can be performed. That is, the image generation unit 640 can determine the color of the subject in the imaging field 108 using the captured image obtained by the imaging device group 402. On the other hand, the image generation unit 640 can determine the color of the subject outside the shooting field 108 using the captured image obtained by the imaging device group 404a. As another method, the image generation unit 640 can also determine the color of the subject by weighted combination. In this case, the subject in the imaging field 108 is given a larger weight to the captured image obtained by the imaging device group 402, and the subject outside the imaging field 108 is given a larger weight to the captured image obtained by the imaging device group 404a. It can be granted.

  Next, the flow of processing of the image processing method according to the present embodiment will be described according to FIG. In step S710, the plurality of imaging devices 509 and 510 perform imaging of a subject, and the image acquisition unit 610 acquires the acquired captured image. In step S720, the viewpoint setting unit 620 sets a virtual viewpoint of the reconstructed image. In step S730, the shape estimation unit 630 selects an imaging device used to estimate the shape of the subject. In step S 740, the shape estimation unit 630 estimates the shape of the subject using the captured image obtained by the selected imaging device. As described above, step S730 may not be performed, and in step S740, the shape of the subject may be estimated using captured images obtained by all the imaging devices. In step S750, the image generation unit 640 generates a reconstructed image based on the obtained shape of the subject and the captured images obtained by the plurality of imaging devices 509 and 510.

Fifth Embodiment
In the first to third embodiments, the case where the positions and orientations of the plurality of imaging devices are fixed has been described. In the present embodiment, the attitude or angle of view of at least one imaging device is controlled. The configuration of the imaging system according to the present embodiment is the same as in the first to third embodiments, but the attitude or angle of view of at least one imaging device is configured to be controllable, and the attitude or angle of view of at least one imaging device And a controller for controlling the As a control device, for example, an image processing device 500 shown in FIG. 5 can be used. In the present embodiment, at least one imaging device is mounted on the motor-driven pan head, and the CPU 501 controls the attitude of the motor-driven pan head via the LAN 508 to change the point of gaze of this imaging device. Also, the CPU 501 can control the angle of view of the imaging device via the LAN 508. Thus, the control device can control the posture or the angle of view of the imaging device so that the imaging system described in the first to third embodiments is configured.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

100: imaging system 101, 103: gaze point 102, 104: imaging device 500: image processing device 610: image acquisition unit 620: viewpoint setting unit 630: shape estimation unit 640: image generation unit

Claims (23)

A plurality of image data obtained based on imaging by a plurality of imaging devices installed to image a first position, and a plurality of images installed to image a second position different from the first position Acquisition means for acquiring a plurality of image data obtained based on imaging by the imaging device of
Based on imaging by a plurality of image data obtained based on imaging by a plurality of imaging devices installed to image the first position and imaging by a plurality of imaging devices installed to image the second position And shape generation means for generating three-dimensional shape data of a subject used for generating a virtual viewpoint image based on at least one of a plurality of image data among a plurality of image data obtained.
At least one imaging device among a plurality of imaging devices installed to image the first position is installed closer to the second position than the first position,
At least one imaging device among a plurality of imaging devices installed to image the second position is installed closer to the first position than the second position. Image processing device.   At least one imaging device of the plurality of imaging devices installed to image the first position is at least one imaging device of the plurality of imaging devices installed to image the second position The image processing apparatus according to claim 1, wherein the image processing apparatus is installed near the second position.   At least one imaging device of the plurality of imaging devices installed to image the first position is two imaging devices of the plurality of imaging devices installed to image the second position The image processing apparatus according to claim 1, wherein the image processing apparatus is installed between the two.   The image processing apparatus according to any one of claims 1 to 3, wherein the first position and the second position are included in the same imaging field. Each of the plurality of imaging devices installed to image the first position is installed such that a first region including the first position on the imaging field falls within a field of view;
5. The apparatus according to claim 4, wherein at least two imaging devices among the plurality of imaging devices installed to image the first position are installed to sandwich the first region. Image processing device.   The plurality of imaging devices installed to image the first position and the plurality of imaging devices installed to image the second position are installed at the same height from the imaging field. The image processing apparatus according to claim 4 or 5, characterized in that:   The plurality of imaging devices installed to image the first position include a first imaging device and a second imaging device having different angle of view from each other. An image processing apparatus according to item 1.   The image processing according to claim 7, wherein the second imaging device is installed at a position having a wide angle of view and close to the first position as compared with the first imaging device. apparatus. The plurality of imaging devices installed to image the first position have in-focus positions set based on the first position,
The plurality of imaging devices installed to image the second position have their in-focus positions set based on the second position. The image processing apparatus according to claim 1.   The shape generation unit is configured to capture a plurality of image data obtained based on imaging by a plurality of imaging devices installed to capture the first position, and a plurality of images configured to capture the second position. The image processing apparatus according to any one of claims 1 to 9, wherein three-dimensional shape data of the subject is generated based on a plurality of image data obtained based on imaging by the imaging device of .   The acquisition unit further acquires a plurality of image data obtained based on imaging by a plurality of imaging devices installed to capture a third position different from the first position and the second position. The image processing apparatus according to any one of claims 1 to 10, wherein   The image processing apparatus according to claim 11, wherein the first position, the second position, and the third position are on a straight line on an imaging field. A plurality of image data obtained based on imaging by a plurality of imaging devices installed to image a first area and a second area different at least in part from the first area Acquisition means for acquiring a plurality of image data obtained based on imaging by a plurality of installed imaging devices;
Based on imaging by a plurality of image data obtained based on imaging by a plurality of imaging devices installed to image the first area and imaging by a plurality of imaging devices installed to image the second area And shape generation means for generating three-dimensional shape data of a subject used for generating a virtual viewpoint image based on at least one of a plurality of image data among a plurality of image data obtained.
At least one imaging device among a plurality of imaging devices installed to image the first area is installed closer to the second area than the first area,
At least one imaging device among a plurality of imaging devices installed to image the second area is installed closer to the first area than the second area. Image processing device. The at least one imaging device installed to image the first area and installed closer to the second area than the first area is more than the center point of the first area Located near the center point of the second region,
The at least one imaging device installed to image the second area and installed closer to the first area than the second area is more than the center point of the second area The image processing apparatus according to claim 13, wherein the image processing apparatus is installed near the center point of the first area. The acquisition unit further acquires information on the position and the direction of the virtual viewpoint.
The image generation unit further includes an image generation unit that generates the virtual viewpoint image based on the information on the position and the direction of the virtual viewpoint acquired by the acquisition unit and the three-dimensional shape data of the subject generated by the shape generation unit. The image processing apparatus according to any one of claims 1 to 14, characterized in that:   The image processing apparatus according to any one of claims 1 to 15, wherein the subject includes a moving object.   The image processing apparatus according to any one of claims 1 to 16, wherein the subject includes a person. A plurality of image data obtained based on imaging by a plurality of imaging devices installed to image a first position, and a plurality of images installed to image a second position different from the first position Acquiring a plurality of image data obtained based on imaging by the imaging device of
Based on imaging by a plurality of image data obtained based on imaging by a plurality of imaging devices installed to image the first position and imaging by a plurality of imaging devices installed to image the second position Generating three-dimensional shape data of a subject used for generating a virtual viewpoint image based on at least one of a plurality of image data among a plurality of image data obtained by
At least one imaging device among a plurality of imaging devices installed to image the first position is installed closer to the second position than the first position,
At least one imaging device among a plurality of imaging devices installed to image the second position is installed closer to the first position than the second position. Image processing method.   At least one imaging device of the plurality of imaging devices installed to image the first position is at least one imaging device of the plurality of imaging devices installed to image the second position The image processing method according to claim 18, wherein the image processing method is installed near the second position.   At least one of the plurality of imaging devices installed to image the first position is at least one of two imaging devices of the plurality of imaging devices installed to image the second position. The image processing method according to claim 18, wherein the image processing method is installed between the two.   In the generation step, a plurality of image data obtained based on imaging by a plurality of imaging devices installed to image the first position, and a plurality of images installed to image the second position The image processing method according to any one of claims 18 to 20, wherein three-dimensional shape data of the subject is generated based on a plurality of image data obtained based on imaging by an imaging device. . A plurality of image data obtained based on imaging by a plurality of imaging devices installed to image a first area and a second area different at least in part from the first area Acquiring a plurality of image data obtained based on imaging by a plurality of installed imaging devices;
A plurality of image data obtained based on imaging by a plurality of imaging devices installed to image the first area and a plurality of images installed to image a second position different from the first area A generation step of generating three-dimensional shape data of a subject used for generation of a virtual viewpoint image based on at least one of a plurality of image data among a plurality of image data obtained based on imaging by the imaging device of And
At least one imaging device among a plurality of imaging devices installed to image the first area is installed closer to the second area than the first area,
At least one imaging device among a plurality of imaging devices installed to image the second area is installed closer to the first area than the second area. Image processing method.   The program for operating a computer as each means of the image processing apparatus in any one of Claims 1-17.

Images