JP2024041435A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the accuracy of a three-dimensional model that represents the shape of a subject estimated based on a picked-up image.

SOLUTION: An image processing apparatus acquires information indicating one or more foreground areas extracted from a picked-up image. The image processing apparatus selects, from the one or more foreground areas, at least partial foreground area based at least on whether the foreground area overlaps the boundary of a predetermined area in the image. The image processing apparatus outputs the selected at least partial foreground area in order to generate information indicating a three-dimensional shape of a subject.

SELECTED DRAWING: Figure 4

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present disclosure relates to an image processing device, an image processing method, and a program, and particularly relates to a technique for extracting a foreground region from an image.

A technique for generating an image of a subject from an arbitrary virtual viewpoint (virtual viewpoint image) is known. In such technology, synchronized photographing is performed using a plurality of cameras installed at different positions. Furthermore, a virtual viewpoint image is generated using images captured from a plurality of viewpoints obtained through such photography. Specifically, a foreground silhouette image showing the subject area is generated from the captured image using a background subtraction method, a machine learning method, or the like. Then, a three-dimensional model of the subject is generated using the foreground silhouette images obtained from the plurality of captured images using a visual volume intersection method or the like. Then, a virtual viewpoint image is generated by combining an image of the three-dimensional model from an arbitrary viewpoint and a background image. In this way, it becomes possible to view the subject in the three-dimensional background space from any desired viewpoint.

Here, if the foreground silhouette image includes an unnecessary foreground silhouette, the accuracy of the obtained three-dimensional model will deteriorate. Therefore, it is desirable to remove unnecessary foreground silhouettes from the foreground silhouette image. For example, Patent Document 1 proposes a technique for correcting a foreground silhouette image (mask image). According to the method disclosed in Patent Document 1, the foreground silhouette image is corrected by filling in defects in the foreground silhouette image based on projection of a three-dimensional model (voxel data) and deleting foreground silhouettes having a predetermined number of pixels or less.

Japanese Patent Application Publication No. 2011-043879 Japanese Patent Application Publication No. 2020-141375

The inventors of the present application have found that when foreground silhouette correction is performed using the method of Patent Document 1, the accuracy of the 3D model is degraded, or parts of the 3D model are likely to be lost, especially when the subject is at the edge of the camera's field of view.

The technology of the present disclosure aims to improve the accuracy of a three-dimensional model representing the shape of a subject estimated based on a captured image.

An image processing device according to an embodiment has the following configuration. That is,
acquisition means for acquiring information indicating one or more foreground regions extracted from the captured image;
selection means for selecting at least some foreground regions from the one or more foreground regions based at least on whether the foreground region overlaps a boundary of a predetermined region in the image;
output means for outputting the selected at least part of the foreground region to a shape generation means for generating information indicating a three-dimensional shape of the subject;
Equipped with.

The accuracy of the three-dimensional model representing the shape of the subject estimated based on the captured image can be improved.

FIG. 3 is a diagram illustrating a photographing method for generating a virtual viewpoint video. FIG. 1 is a block diagram showing an example of a functional configuration of an image processing device according to an embodiment. 1 is a flowchart of an image processing method according to an embodiment. The figure which shows the example of removal of the foreground silhouette. The figure which shows the example of removal of the foreground silhouette. FIG. 1 is a block diagram illustrating an example of a hardware configuration of an image processing device according to an embodiment. The figure which shows the example of distortion correction.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claims. Although a plurality of features are described in the embodiment, not all of these features are essential, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

According to the method described in Patent Document 1, foreground silhouettes having a predetermined number of pixels or less are removed. For this reason, when a subject is located at the edge of the image, a small foreground silhouette showing a part of the subject may be removed. As a result, in the three-dimensional model obtained, the corresponding part of the subject may be missing. On the other hand, according to an embodiment of the present disclosure, the foreground silhouette at the edge of the image is protected, so it is possible to suppress a phenomenon in which a part of the subject is missing in the three-dimensional model.

FIG. 1 shows a usage scene of an image processing apparatus according to an embodiment. Imaging devices 110, 120, 130, and 140 are installed around the imaging area 100. These imaging devices 110 to 140 synchronously photograph subjects 101 and 102 located in the photographing area 100. The image processing device according to one embodiment acquires captured images obtained by synchronous shooting from the imaging devices 110 to 140. As described later, the image processing device generates foreground silhouette images 111, 121, 131, and 141 based on these captured images. Further, the image processing device generates three-dimensional models of the subjects 101 and 102 from these foreground silhouette images based on, for example, a visual volume intersection method. Further, the image processing device generates a virtual viewpoint image by combining a three-dimensional shape from an arbitrary viewpoint and a background image. In this way, the image processing device can generate virtual viewpoint images of the subjects 101 and 102.

An image processing device according to an embodiment can be realized by a computer including a processor and a memory. Examples of the computer include a general-purpose desktop computer, laptop computer, tablet PC, or smartphone. An example of the hardware configuration of the image processing apparatus 200 according to one embodiment will be described below with reference to FIG. 6. The image processing device 200 includes a CPU 611 , a ROM 612 , a RAM 613 , an auxiliary storage device 614 , a display section 615 , an operation section 616 , a communication I/F 617 , and a bus 618 .

The CPU 611 controls the entire image processing apparatus 200 using computer programs and data stored in the ROM 612 or RAM 613. In this way, the CPU 611 realizes each function of the image processing apparatus 200 shown in FIG. 2. The ROM 612 stores programs that do not require modification. The RAM 613 temporarily stores programs or data supplied from the auxiliary storage device 614, or data supplied from the outside via the communication I/F 617. Auxiliary storage device 614 stores various data such as image data or audio data. The auxiliary storage device 614 is, for example, a hard disk drive.

The display unit 615 can display information such as a GUI (Graphical User Interface) for a user to operate the image processing apparatus 200. The display unit 615 is, for example, a liquid crystal display or an LED. The operation unit 616 inputs various instructions to the CPU 611 based on user operations. The operation unit 616 is, for example, a keyboard, a mouse, a joystick, a touch panel, or the like. The CPU 611 operates as a display control unit that controls the display unit 615 and an operation control unit that controls the operation unit 616. In the example of FIG. 6, the display unit 615 and the operation unit 616 are located inside the image processing device 200. However, at least one of the display unit 615 and the operation unit 616 may be included in another device that exists outside the image processing device 200.

The communication I/F 617 is used for communication with an external device of the image processing device 200. For example, when the image processing device 200 is connected to an external device by wire, a communication cable is connected to the communication I/F 617. When the image processing device 200 has a function of wirelessly communicating with an external device, the communication I/F 617 includes an antenna. A bus 618 connects each part of the image processing device 200 and transmits information.

In this way, a processor such as the CPU 611 executes a program stored in a memory such as the ROM 612, the RAM 613, or the auxiliary storage device 614, thereby realizing the functions of each part shown in FIG. 2, etc., which will be described later. I can do it. Note that the image processing device 200 may include one or more dedicated hardware different from the CPU 611. This dedicated hardware can execute at least part of the processing by the CPU 611. Examples of specialized hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). Further, the image processing device according to one embodiment may be configured by, for example, a plurality of information processing devices connected via a network.

FIG. 2 is a block diagram illustrating an example of the configuration of an image processing apparatus 200 according to an embodiment. The image processing device 200 includes an image acquisition section 201, an extraction section 202, a protection determination section 203, a removal section 204, a shape generation section 205, and an image generation section 206.

The image acquisition unit 201 acquires an image. For example, the image acquisition unit 201 can acquire an image including a subject. In this embodiment, the image acquisition unit 201 can acquire images captured by the imaging devices 110 to 140 (hereinafter referred to as "input images"). The image acquisition unit 201 may acquire an image from an imaging device such as a camera via a cable such as an SDI cable. Further, the image acquisition unit 201 may acquire image data via an interface such as USB or PCIe.

The extraction unit 202 acquires information indicating one or more foreground regions extracted from the image. In this embodiment, the extraction unit 202 extracts one or more foreground regions from the input image. The extraction unit 202 can extract a subject area to be treated as the foreground from the input image. A background subtraction method can be used as a foreground extraction method. In this case, the extraction unit 202 generates a foreground silhouette image representing the foreground region based on the input image acquired by the image acquisition unit 201 and the background image. A foreground silhouette image can show multiple foreground regions. Hereinafter, each foreground region shown by a foreground silhouette image will be referred to as a foreground silhouette.

The extraction unit 202 can generate a background image from the input image acquired by the image acquisition unit 201. As a method of generating a background image, for example, a method of determining a background area that does not change among a plurality of input images can be cited. The image acquisition unit 201 can generate a background image using the background region in the input image determined in this way. Note that the foreground extraction method used by the extraction unit 202 is not limited to the background subtraction method. The extraction unit 202 may use a method based on machine learning or chromakey processing.

The protection determination unit 203 protects the foreground area extracted by the extraction unit 202. Here, the protection determining unit 203 can select the foreground area to be protected based at least on whether the foreground area overlaps the boundary of a predetermined area in the input image. The protected foreground region is protected from removal by the remover 204.

In one embodiment, this predetermined region is the entire input image. In the following example, the protection determining unit 203 separates the foreground silhouette included in the foreground silhouette image. Specifically, the protection determination unit 203 determines the circumscribed rectangular area of each foreground silhouette. Then, the protection determining unit 203 determines whether each foreground silhouette overlaps the image edge of the input image based on the determined circumscribed rectangle. The protection determination unit 203 can protect the foreground silhouette extending over the edge of the image. A specific example will be described later. Note that in this specification, the edge of an image or region means a boundary line or edge between the image and the outside of the image, or between the region and the outside of the region.

The removal unit 204 selects at least a portion of the foreground areas from among the one or more foreground areas extracted by the extraction unit 202. The foreground region selected by the removal unit 204 can be used by the shape generation unit 205 to generate shape data. Here, the removal unit 204 removes the foreground area based at least on whether the foreground area is protected, that is, based on at least whether the foreground area overlaps the boundary of a predetermined area in the input image. You can choose. In this embodiment, the foreground region not selected by the removal unit 204 is not used by the shape generation unit 205 to generate shape data. In other words, the removal unit 204 can remove some of the foreground areas from among the one or more foreground areas extracted by the extraction unit 202, based at least on whether the foreground areas are protected.

Further, the removal unit 204 can select at least some foreground regions from one or more foreground regions further based on the area of the foreground regions. In one embodiment, the removing unit 204 removes unnecessary foreground silhouettes other than the foreground silhouettes determined to be protected by the protection determining unit 203 from the foreground silhouette image. The removal unit 204 can determine whether the foreground silhouette is unnecessary according to predetermined conditions. In this embodiment, the removal unit 204 makes such a determination according to the area of the foreground silhouette. As an example, the removal unit 204 can remove a foreground silhouette if the area of the foreground silhouette is less than a threshold. Such a threshold value may be set in advance, or may be designated by the user via a control UI (not shown). On the other hand, the foreground silhouette determined to be protected by the protection determining unit 203 is not removed by the removing unit 204. In other words, the removal unit 204 can remove foreground silhouettes that meet predetermined conditions and are not protected. On the other hand, the removing unit 204 can select protected foreground silhouettes and foreground silhouettes that are not protected but do not satisfy a predetermined condition. In this case, foreground silhouettes that meet the predetermined conditions but are protected are not removed.

Thus, in one embodiment, the foreground region protected by the protection determiner 203 is not removed by the remover 204. In other words, if there is a foreground area protected by the protection determining unit 203, the removing unit 204 can select the foreground area so that at least the foreground area protected by the protection determining unit 203 is selected. .

On the other hand, the method by which the removal unit 204 selects a foreground area depending on whether the foreground area is protected by the protection determination unit 203 is not limited to this method. For example, the removal unit 204 can determine whether to select an unprotected foreground silhouette according to a first condition. On the other hand, the removal unit 204 can determine whether to select a protected foreground silhouette according to a second condition different from the first condition. As an example, the first condition may be that the size of the foreground silhouette is less than a first threshold. In this case, the second condition may be that the size of the foreground silhouette is less than a second threshold, which is smaller than the first threshold. Such a method also makes it possible to suppress the deletion of foreground silhouettes at the edges of the image.

The shape generation unit 205 generates information indicating the three-dimensional shape of the subject based on information indicating the foreground area selected by the removal unit 204 from among the foreground areas extracted by the extraction unit 202. For example, the shape generation unit 205 can generate shape data representing the three-dimensional shape of the object representing the subject based on the plurality of foreground silhouettes that were not removed by the removal unit 204. As a method for generating such three-dimensional shape data, a visual-hull method using object silhouette information, a multi-view stereo method using triangulation, or the like can be used. Such three-dimensional shape data can be, for example, polygon data or voxel data.

The image generation unit 206 generates a virtual viewpoint image of the subject based on information indicating the three-dimensional shape of the subject. For example, the image generation unit 206 can use the three-dimensional shape data and texture information obtained from the shape generation unit 205 to generate a virtual viewpoint image corresponding to the parameters of the virtual camera specified by the user. Note that texture information regarding a specific position of the object indicated by the three-dimensional shape data can be generated based on an input image obtained by photographing the corresponding position of the subject, which is obtained by the image acquisition unit 201. . As a specific method for generating a virtual viewpoint image, for example, the method described in Patent Document 2 can be used. Further, the image generation unit 206 can send the generated virtual viewpoint image for display to a display device 210 such as a display.

Note that the process of generating the foreground silhouette image may be performed by another device. For example, in one embodiment, the imaging devices 110-140 may generate foreground silhouette images based on captured images. In this case, the imaging device may have the functions of an image acquisition unit 201, an extraction unit 202, a protection determination unit 203, and a removal unit 204. In such a configuration, the imaging device can extract and select a foreground region from an image. Further, the removal unit 204 of the imaging device can output information indicating the selected foreground region to the image processing device 200. In this case, the shape generation unit 205 of the image processing device 200 can generate information indicating the three-dimensional shape of the subject based on information indicating the foreground region acquired from the imaging device. Note that an imaging device that performs such processing can be realized using, for example, the hardware and program shown in FIG.

Furthermore, in another embodiment, the imaging device may have the functions of an image acquisition unit 201 and an extraction unit 202. In such a configuration, the imaging device can extract the foreground region from the image. Further, the imaging device can output information indicating the extracted foreground region to the image processing device 200. In this case, the protection determining unit 203 and the removing unit 204 of the image processing device 200 can select the foreground region acquired from the imaging device. The removal unit 204 can then output information indicating the selected foreground region. Further, the shape generation unit 205 included in the image processing device 200 or another image processing device can acquire information indicating the selected foreground region output by the removal unit 204. The shape generation unit 205 can then generate information indicating the three-dimensional shape of the subject based on the acquired information indicating the foreground region.

Below, a method for removing unnecessary foreground silhouettes while protecting the foreground silhouettes that extend to the edges of the image will be described with reference to FIGS. 4(A) to 4(B). In the example of FIGS. 4A and 4B, foreground silhouettes that extend to the edges of the image are protected, while unprotected foreground silhouettes that have an area less than the threshold are removed.

The foreground silhouette image 400 generated by the extraction unit 202 includes foreground silhouettes 401 to 405, as shown in FIG. 4(A). Although the circumscribed rectangle of the foreground silhouette is shown by a broken line in FIG. 4A, such a rectangle is not included in the actual foreground silhouette image. Foreground silhouettes 401 to 403 are foreground silhouettes extracted due to salt-and-pepper noise during photographing. Foreground silhouettes 402 and 403 span the edges of the image. Foreground silhouette 404 corresponds to subject 101. Further, a foreground silhouette 405 corresponds to the subject 102. The foreground silhouette 405 hangs over the edges of the image.

FIG. 4B shows a foreground silhouette image 410 after the removing unit 204 has performed a process of removing unnecessary foreground silhouettes. As shown in FIG. 4(B), foreground silhouette image 410 includes foreground silhouettes 412 to 415. The foreground silhouette 401 is not protected because it does not cover the edge of the image. Since the foreground silhouette 401 has an area less than the threshold value, it is removed by the removal unit 204. Foreground silhouettes 402, 403, and 405 have areas less than the threshold area, but span the edges of the image. Therefore, the foreground silhouettes 402, 403, and 405 were protected by the protection determination unit 203 and were not removed by the removal unit 204. Foreground silhouette 404 is not protected because it does not span the edges of the image. However, since the foreground silhouette 404 has an area larger than the threshold value, it was not removed by the removal unit 204.

FIG. 3 is a flowchart showing the flow of the image processing method according to this embodiment. Note that the symbol "S" means a step. In S301, the image acquisition unit 201 acquires an input image as described above. In S302, the extraction unit 202 extracts one or more foreground silhouettes from the input image. As mentioned above, the extraction unit 202 can generate the foreground silhouette image 400 based on the input image and the background image.

In S303, the protection determination unit 203 separates the foreground silhouettes from each other. For example, the protection determination unit 203 separates the foreground silhouette included in the foreground silhouette image acquired in S302. For example, the protection determination unit 203 can divide the foreground silhouette into each circumscribed rectangular area, as indicated by broken lines in the foreground silhouette image 400. As another method, the protection determining unit 203 may separate a group of pixels indicating a continuous foreground silhouette from other foreground silhouettes as one foreground silhouette.

The processes of S304 to S306 are performed for each of the foreground silhouettes separated from each other in S303. In S304, the protection determining unit 203 acquires position information of the foreground silhouette. The position information may include, for example, the coordinates (x, y) of the upper left vertex of the area, the width (w) of the area, and the height (h) of the area.

In S305, the protection determining unit 203 determines whether the foreground silhouette covers the edge of the image. For example, the protection determining unit 203 can determine whether the foreground silhouette covers the edge of the image based on the position information of the circumscribed rectangle of the foreground silhouette obtained in S304. Whether or not the foreground silhouette extends over the edge of the image can be determined, for example, according to equation (1). edge_flag is a flag indicating whether or not the foreground silhouette covers the edge of the image. When this flag is 1, it indicates that the foreground silhouette covers the edge of the image. Further, when this flag is 0, it indicates that the foreground silhouette does not cover the edge of the image. x and y represent the x and y coordinates of the upper left vertex of the circumscribed rectangle of the foreground silhouette, respectively. Furthermore, w and h represent the width and height of the circumscribed rectangle of the foreground silhouette. WIDTH and HEIGHT represent the width and height of the foreground silhouette image (corresponding to the width and height of the input image in this example). If it is determined that the foreground silhouette covers the edge of the image, the process advances to S306. Further, if it is determined that the foreground silhouette does not overlap the edge of the image, the processes of S304 to S306 are performed for the next foreground silhouette. Note that the above-mentioned determination method is only an example, and it may be determined whether the foreground silhouette overlaps the edge of the image using other methods using information that can be obtained from the foreground silhouette.

In S306, the protection determining unit 203 protects the foreground silhouette determined to extend over the edge of the image in S305. For example, the protection determination unit 203 can set a protection flag indicating that the foreground silhouette is protected. By referring to this protection flag, it can be determined in subsequent processing whether or not the foreground silhouette is protected.

The processes of S307 to S309 are also performed for each of the foreground silhouettes separated from each other in S303. In S307, the removal unit 204 obtains the area of the foreground silhouette. The area of the foreground silhouette can be determined according to the number of pixels of the foreground silhouette. On the other hand, the area of the foreground silhouette may be determined according to the width and height of the rectangular area described above.

In S308, the removal unit 204 determines whether the foreground silhouette acquired in S307 has an area less than a threshold value. If the area of the foreground silhouette is less than the threshold, the process advances to S309. Further, if it is determined that the foreground silhouette has an area equal to or larger than the threshold value, the processes of S307 to S309 are performed for the next foreground silhouette.

In S309, the removal unit 204 removes the unprotected foreground silhouette that was determined to have an area less than the threshold in S307. However, the removal unit 204 does not delete the foreground silhouette protected in S305.

According to the embodiments described above, deletion of the foreground silhouette at the edge of the image is prevented or suppressed. For this reason, when a part of the subject is captured at the edge of the image, it is suppressed that part of the subject is missing in the three-dimensional shape. Note that according to this embodiment, unnecessary foreground silhouettes caused by noise or the like may remain at the edges of the image. However, the possibility that a foreground silhouette with a small area will overlap the edge of the image is not very high. Furthermore, even if the removal unit 104 cannot remove the unnecessary foreground silhouette with a small area, the influence of such unnecessary foreground silhouettes on the estimation of the three-dimensional shape of the subject is small. On the other hand, there are many cases in which a part of the subject covers the edge of the image, such as when the subject enters the imaging range from outside the imaging range. Furthermore, removing the foreground silhouette corresponding to such an object by the removing unit 104 only on the condition that the object has a certain area has a large influence on the estimation of the three-dimensional shape of the object. For these reasons, by adopting the configuration of this embodiment, it is possible to improve the overall quality of data representing a three-dimensional shape.

<Modified example>
As described above, the protection determination unit 203 can select the foreground area to be protected based on whether the foreground area exists at the edge of a predetermined area. In the process shown in FIG. 3, the protection determination unit 203 protected the foreground area located at the edge of the input image. In this way, the protection determining unit 203 can protect the foreground area located at the edge of the entire image area of the input image. On the other hand, the protection determining unit 203 may protect a foreground area located at the edge of another area. For example, the predetermined area may be a foreground extraction target area (hereinafter referred to as a foreground extraction area) in the image. In this case, for example, the protection determination unit 203 can protect the foreground area located at the edge of the designated foreground extraction area.

The input image may include regions that are not relevant to three-dimensional shape generation. For example, a captured image obtained by an imaging device installed at a specific location may include an area where a moving object other than the subject exists, such as an audience seat, in addition to an area where the subject may be captured. do not have. When a foreground region is extracted from such a region, a three-dimensional shape other than the target object may be generated. Furthermore, the captured image may include an area where only the background is visible. Therefore, in one embodiment, such a region is treated as a region unrelated to the generation of a three-dimensional shape. That is, the extraction unit 202 does not need to extract foreground regions from regions that are not related to three-dimensional shape generation. In one embodiment, the extraction unit 202 extracts one or more foreground regions from the foreground region extraction target region in the image. Such an extraction target area can be determined in advance. It is expected that such processing will suppress the extraction of unnecessary foreground regions and improve the accuracy of information indicating three-dimensional shapes.

In order to realize such processing, the foreground extraction area and the non-foreground extraction area can be specified using, for example, a mask image (hereinafter referred to as a foreground extraction area mask). However, when deleting a foreground silhouette with a small area, if part of the subject falls on the boundary between the foreground extraction area and the non-foreground extraction area, part of the subject will be Foreground silhouettes shown were sometimes removed. Therefore, in this modification, the protection determination unit 203 determines whether the foreground silhouette extends over the edge of the input image, as well as the boundary between the foreground extraction area and the non-foreground extraction area.

Such a modification will be explained with reference to FIGS. 5(A) to 5(C). In the examples shown in FIGS. 5A to 5C, the foreground silhouette at the edge of the image or at the boundary between the foreground extraction area and the non-foreground extraction area is protected. In this example, the extraction unit 202 generates a foreground silhouette image 500 from the input image using a foreground extraction region mask 520 shown in FIG. 5C. The foreground extraction region mask 520 shows a non-foreground extraction region 521 and a foreground extraction region 522. The extraction unit 202 extracts a foreground silhouette from the area indicated by the foreground extraction area 522 in the input image. In FIG. 5A, a foreground silhouette image 500 includes foreground silhouettes 403 to 406. Foreground silhouettes 403 to 405 are similar to those in FIG. 4(A). Further, a foreground silhouette 406 is a foreground silhouette extracted due to salt-and-pepper noise during photographing. Further, a foreground silhouette 407 is a silhouette corresponding to the subject.

FIG. 5B shows a foreground silhouette image 510 after the removal unit 204 has performed a process of removing unnecessary foreground silhouettes. The foreground silhouette 406 does not extend to the edge of the image or the boundary of the foreground extraction region 522 and has an area less than the threshold, so it was removed by the removal unit 204. Foreground silhouette 407 has an area less than the threshold area, but spans the boundary with foreground extraction region 522. Therefore, the foreground silhouette 407 was protected by the protection determination unit 203 and was not removed by the removal unit 204. As a result, foreground silhouette image 510 includes foreground silhouette 417.

For example, when the circumscribed rectangular area of the foreground silhouette includes both the non-foreground extraction area 521 and the foreground extraction area 522, the protection determination unit 203 determines whether the foreground silhouette is the non-foreground extraction area 521 and the foreground extraction area 522. It can be determined that it is within the boundary of . This determination method is merely an example, and other methods may be used to determine whether the foreground silhouette extends over the edge of the foreground extraction area using information that can be obtained from the foreground silhouette.

According to such a configuration, deletion of the foreground silhouette on the boundary between the foreground extraction area and the non-foreground extraction area is prevented or suppressed. For this reason, when a part of the subject is captured at the boundary of the foreground extraction area, it is suppressed that part of the subject is missing in the three-dimensional shape.

As another example, the predetermined area may be an effective area where an image obtained by imaging is captured. In this case, the protection determining unit 203 can protect the foreground area located at the edge of the effective area in the input image. For example, the input image may be obtained by performing image correction on a captured image captured by an imaging device. Examples of image correction include deformation processing such as lens distortion correction processing that takes into account lens distortion of an imaging device. When such transformation processing is performed, the position of the image edge changes. Therefore, a portion of the corrected input image that corresponds to the edge of the captured image may be located inside the edge of the corrected input image. In this case, the input image includes a transformed captured image. The area occupied by the deformed captured image in the input image can be called an effective area. This effective area is an area inside the portion corresponding to the edge of the captured image. In such a case, the protection determining unit 203 can protect the foreground silhouette based on whether the foreground silhouette exists at the edge of the captured image before correction.

Such distortion correction processing may be performed by a distortion correction unit (not shown) included in the image processing device 200, or may be performed by the imaging device. For example, the distortion correction unit applies parameters set in advance to the image acquired by the image acquisition unit 201 from the imaging device, or parameters set by the user via a control UI (not shown). Then, distortion correction processing can be performed.

Such a modification will be explained with reference to FIGS. 7(A) to (E). A captured image 800 shown in FIG. 7A includes a subject image 801 and a subject image 802. FIG. 7B shows an input image 810 obtained by applying distortion correction processing to the captured image 800. In the input image 810, the respective subjects appear as images 811 and 812. As shown in FIG. 7(B), the position of the image edge has changed due to the distortion correction process. That is, the central area of input image 810 is an effective area corresponding to captured image 800. On the other hand, the peripheral area of the input image 810 is an invalid area that does not correspond to the captured image 800. Here, the boundary of the effective area is determined to coincide with the edge of the captured image 800 after being transformed by the lens distortion correction process. In this embodiment, a distortion correction unit (not shown) or an imaging device that performs distortion correction processing generates a valid area mask 820 that indicates a valid area 821 and an invalid area 822, as shown in FIG. 7(C).

FIG. 7(D) shows a foreground silhouette image 830 generated by the extraction unit 202. Foreground silhouette image 830 includes foreground silhouettes 831-834. Foreground silhouettes 831 and 832 are foreground silhouettes corresponding to different subjects, respectively. Also, the foreground silhouette 832 spans the edge of the effective area. Foreground silhouettes 833 to 834 are foreground silhouettes extracted due to salt-and-pepper noise during photographing. Among these foreground silhouettes, foreground silhouette 834 spans the edge of the effective area. In FIG. 7D, the circumscribed rectangle of the foreground silhouette is shown by a broken line, and the boundary between the valid area and the invalid area is shown by a solid line, but these lines are not included in the actual foreground silhouette image 830. .

Here, the protection determination unit 203 can refer to the effective area mask 820 and determine whether the foreground silhouette exists at the edge of the effective area. Then, the protection determining unit 203 can protect the foreground silhouette existing at the edge of the effective area. As a specific example, in S305, the protection determination unit 203 determines whether the foreground silhouette overlaps the edge of the image or the boundary between the valid area and the invalid area, based on the position of the circumscribed rectangular area of the foreground silhouette acquired in S304. can be determined. Then, in S306, the protection determining unit 203 can protect the foreground silhouette at the edge of the image or the edge of the effective area. The determination of whether the foreground silhouette crosses the boundary between the valid region and the invalid region is the same as the determination of whether the foreground silhouette crosses the boundary between the foreground extraction region and the non-foreground extraction region, which has already been described. It can be carried out. Note that such a determination method is only an example, and other methods can be used to determine whether the foreground silhouette extends over the edge of the effective region, using information that can be obtained from the foreground silhouette and information indicating a valid region or an invalid region. may be determined.

Even in such an example, the removal unit 204 can remove unprotected foreground silhouettes that have an area less than the threshold. For example, as shown in FIG. 7E, the foreground silhouette 833 is removed by the removal unit 204 because it does not extend over the boundary of the effective area and has an area less than the threshold value. On the other hand, foreground silhouettes 832 and 834 have areas less than the threshold area, but span the edges of the effective area. Therefore, the foreground silhouettes 832 and 834 were protected by the protection determination unit 203 and were not removed by the removal unit 204. As a result, the foreground silhouette image 840 after unnecessary foreground silhouettes are deleted by the removal unit 204 includes foreground silhouettes 841, 842, and 844.

According to such a configuration, deletion of the foreground silhouette on the boundary of the effective area is prevented or suppressed. Therefore, even if a part of the subject overlaps the edge of the image before the deformation process, part of the subject is prevented from being missing in the three-dimensional shape.

Up to this point, we have protected the foreground silhouette based on whether it spans the edge of the input image, the boundary of the foreground extraction area, or the boundary of the effective area. The method was explained. However, these determinations may be combined. For example, if the foreground silhouette overlaps any one of the boundary of the input image, the boundary of the foreground extraction area, or the boundary of the effective area, the protection determining unit 203 may protect the foreground silhouette.

In the above example, the removal unit 204 selected the foreground region based on the area of the foreground region. However, the method by which the removal unit 204 selects the foreground region is not limited to this method. For example, the removing unit 204 may select at least some foreground regions from one or more foreground regions based on the feature amount of the foreground regions. As a specific example, the removal unit 204 may extract a feature amount from the foreground silhouette or a portion corresponding to the foreground silhouette in the input image, and select the foreground silhouette based on the analysis result of the feature amount. Further, the removal unit 204 may select the foreground region based on the type of subject represented by the foreground region. The removal unit 204 can determine the subject represented by the foreground region based on the analysis results of the feature amounts as described above. For example, the remover 204 may select foreground silhouettes that represent a particular type of object, such as a person, a ball, a soccer goal, or a rugby goal post. On the other hand, the removal unit 204 can remove other types of foreground silhouettes. Regardless of which method is used, the removal unit 204 can prevent or suppress removal of the protected foreground silhouette. That is, in addition to determining whether the foreground region overlaps the boundary of a predetermined region, the removal unit 204 can select a foreground silhouette based on these determinations. According to such a configuration, even if the foreground region of the subject that spans the boundary of the region is small and the feature amount cannot be analyzed correctly, it is possible to prevent or suppress the removal of this foreground region. can.

(Other examples)
The above technology supplies a program that implements one or more functions of the above embodiments to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device reads the program. This can also be achieved by executing a process. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The disclosure of this specification includes the following imaging device, imaging system, method, and program.

(Item 1)
acquisition means for acquiring information indicating one or more foreground regions extracted from the captured image;
selection means for selecting at least some foreground regions from the one or more foreground regions based at least on whether the foreground region overlaps a boundary of a predetermined region in the image;
output means for outputting the selected at least part of the foreground region to a shape generation means for generating information indicating a three-dimensional shape of the subject;
An image processing device comprising:

(Item 2)
The image processing device according to item 1, wherein the acquisition means extracts the one or more foreground regions from the image.

(Item 3)
The image is an image including the subject,
shape generating means for generating information indicating a three-dimensional shape of the subject based on information indicating the foreground region selected by the selecting means;
image generation means for generating a virtual viewpoint image of the subject based on information indicating a three-dimensional shape of the subject;
The image processing device according to item 1 or 2, further comprising:

(Item 4)
The image processing device according to any one of items 1 to 3, wherein the predetermined area is the entire image.

(Item 5)
The image processing device according to any one of items 1 to 3, wherein the predetermined area is an extraction target area of a foreground area in the image.

(Item 6)
The acquisition means extracts the one or more foreground regions from a foreground region extraction target region in the image,
The image processing device according to item 2, wherein the predetermined area is an extraction target area of a foreground area in the image.

(Item 7)
The image processing device according to any one of items 1 to 3, wherein the predetermined area is an effective area in which an image obtained by imaging is captured.

(Item 8)
Any one of items 1 to 3, wherein the image includes a transformed captured image, and the predetermined area is an area occupied by the transformed captured image in the image. Image processing device described in the item.

(Item 9)
The image has been subjected to lens distortion correction processing,
The image according to any one of items 1 to 3, wherein the boundary of the predetermined area is determined to coincide with an edge of the captured image after being transformed by the lens distortion correction process. Processing equipment.

(Item 10)
According to any one of items 1 to 9, the selection means selects at least some of the foreground regions from the one or more foreground regions further based on the area of the foreground region. The image processing device described.

(Item 11)
The selection means selects a foreground region having an area equal to or larger than a threshold value and a foreground region that lies on the boundary of the predetermined region, and selects a foreground region that has an area less than the threshold value but does not cross the boundary of the predetermined region. The image processing device according to any one of items 1 to 10, characterized in that the foreground region is not selected.

(Item 12)
The selection means selects from a foreground area that does not fall on the boundary of the predetermined area according to a first condition, and selects from a foreground area that falls on the boundary of the predetermined area according to a second condition. The image processing device according to any one of items 1 to 11, characterized in that:

(Item 13)
Further comprising a protection means for protecting a foreground region on a boundary of the predetermined region from removal among the one or more foreground regions extracted from the captured image,
The selection means may remove at least some of the foreground regions from among the one or more foreground regions extracted from the captured image based at least on whether the foreground region is protected. The image processing device according to any one of items 1 to 12.

(Item 14)
Item 13, wherein the selection means selects a foreground area having an area equal to or greater than a threshold value from among unprotected foreground areas, and removes a foreground area having an area less than the threshold value without selecting it. image processing device.

(Item 15)
Any one of items 1 to 14, wherein the selection means selects at least some of the foreground regions from the one or more foreground regions further based on the feature amount of the foreground region. The image processing device described in .

(Item 16)
Any one of items 1 to 15, wherein the selection means selects at least some of the foreground regions from the one or more foreground regions, further based on the type of subject represented by the foreground regions. The image processing device described in item 1.

(Item 17)
An image processing method performed by an image processing device, the method comprising:
acquiring information indicating one or more foreground regions extracted from the captured image;
selecting at least some foreground regions from the one or more foreground regions based at least on whether the foreground region overlaps a boundary of a predetermined region in the image;
outputting the at least some of the selected foreground regions to generate information indicating a three-dimensional shape of the subject;
An image processing method characterized by comprising:

(Item 18)
A program for causing a computer to function as the image processing device according to any one of items 1 to 16.

The technology according to the present application is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope thereof. Accordingly, the claims are appended hereto to disclose their scope.

200: Image processing device, 201: Image acquisition unit, 202: Extraction unit, 203: Protection determination unit, 204: Removal unit, 205: Shape generation unit, 206: Image generation unit

Claims (18)

acquisition means for acquiring information indicating one or more foreground regions extracted from the captured image;
selection means for selecting at least some foreground regions from the one or more foreground regions based at least on whether the foreground region overlaps a boundary of a predetermined region in the image;
output means for outputting the selected at least part of the foreground region to a shape generation means for generating information indicating a three-dimensional shape of the subject;
An image processing device comprising: The image processing apparatus according to claim 1, wherein the acquisition means extracts the one or more foreground regions from the image. The image is an image including the subject,
shape generating means for generating information indicating a three-dimensional shape of the subject based on information indicating the foreground region selected by the selecting means;
image generation means for generating a virtual viewpoint image of the subject based on information indicating a three-dimensional shape of the subject;
The image processing device according to claim 1, further comprising: The image processing device according to claim 1, wherein the predetermined area is the entire image. The image processing apparatus according to claim 1, wherein the predetermined area is an extraction target area of a foreground area in the image. The acquisition means extracts the one or more foreground regions from a foreground region extraction target region in the image,
The image processing apparatus according to claim 2, wherein the predetermined area is an extraction target area of a foreground area in the image. The image processing apparatus according to claim 1, wherein the predetermined area is an effective area in which an image obtained by imaging is captured. The image processing according to claim 1, wherein the image includes a transformed captured image, and the predetermined area is an area occupied by the transformed captured image in the image. Device. The image has been subjected to lens distortion correction processing,
The image processing device according to claim 1, wherein the boundary of the predetermined area is determined to coincide with an edge of the captured image after being transformed by the lens distortion correction process. The image processing apparatus according to claim 1, wherein the selection means selects the at least some of the foreground regions from the one or more foreground regions further based on the area of the foreground region. The selection means selects a foreground region having an area equal to or larger than a threshold value and a foreground region that lies on the boundary of the predetermined region, and selects a foreground region that has an area less than the threshold value but does not cross the boundary of the predetermined region. The image processing device according to claim 1, characterized in that no foreground region is selected. The selection means selects from a foreground area that does not fall on the boundary of the predetermined area according to a first condition, and selects from a foreground area that falls on the boundary of the predetermined area according to a second condition. The image processing device according to claim 1, characterized in that: Further comprising a protection means for protecting a foreground region on a boundary of the predetermined region from removal among the one or more foreground regions extracted from the captured image,
The selection means removes at least some of the foreground regions from among the one or more foreground regions extracted from the captured image based at least on whether the foreground region is protected. The image processing device according to claim 1. 14. The selecting means selects a foreground region having an area equal to or more than a threshold value from unprotected foreground regions, and removes the foreground region having an area less than the threshold value without selecting it. The image processing device described. The image processing device according to claim 1, wherein the selection means selects the at least some of the foreground regions from the one or more foreground regions further based on the feature amount of the foreground region. . The image according to claim 1, wherein the selection means selects the at least some of the foreground regions from the one or more foreground regions further based on the type of subject represented by the foreground region. Processing equipment. An image processing method performed by an image processing device, the method comprising:
acquiring information indicating one or more foreground regions extracted from the captured image;
selecting at least some foreground regions from the one or more foreground regions based at least on whether the foreground region overlaps a boundary of a predetermined region in the image;
outputting the at least some of the selected foreground regions to generate information indicating a three-dimensional shape of the subject;
An image processing method characterized by comprising: A program for causing a computer to function as the image processing device according to any one of claims 1 to 16.

Images