JP7159384B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to a technique for extracting a foreground subject area from a captured image.

2. Description of the Related Art Conventionally, there is a background subtraction method as a method for extracting an area of a foreground subject from a captured image obtained by imaging a subject (including a foreground subject and a background subject). In the background subtraction method, the foreground object is calculated based on the pixel-by-pixel difference between the pixel values of the captured image showing the foreground and background objects and the pixel value of the background image showing only the background object. Create a foreground image by extracting the region by . At this time, if an image in which only the background is captured in advance under specific conditions is used as the background image, and the background changes due to changes in sunlight over time, the area of the subject in the foreground is extracted. There is a problem that the accuracy is lowered.

In order to solve the above problem, Patent Literature 1 discloses a technique for extracting a foreground subject area regardless of changes in the background by using a background image created based on a plurality of images captured at different times. do.

Further, Patent Document 2 discloses a technique for extracting an area of a foreground subject regardless of changes in the subject over time using a background image created based on a plurality of images captured from different viewpoints at the same time. Disclose.

JP 2012-104053 A Japanese Patent Application Laid-Open No. 2014-230180

However, in Patent Document 1, when the foreground subject is stationary without moving, the area of the foreground subject is erroneously determined to be the area of the background subject, so the background image cannot be created with high accuracy. Therefore, there is a problem that the accuracy of extracting the area of the subject in the foreground is lowered.

In Patent Document 2, a background image is created by supplementing information on a background subject that cannot be seen from a single viewpoint with information from another viewpoint. Background images cannot be created with high accuracy in areas such as areas where multiple subjects overlap. Therefore, there is a problem that the accuracy of extracting the area of the subject in the foreground is lowered.

Therefore, in view of the above-mentioned problems, the present invention is capable of accurately identifying an area of a foreground subject regardless of whether or not the subject changes (moves, etc.) with the passage of time and the state of the subject such as the density of the foreground subject. The purpose is to extract.

The present invention provides a first acquisition unit that acquires an image of interest including a subject that is captured from a viewpoint of interest and acquires a plurality of reference images that are captured from a plurality of viewpoints different from the viewpoint of interest. a generation means for generating a plurality of transformed images viewed from the viewpoint of interest by transforming the plurality of reference images acquired by the second acquisition means; and the first acquisition means Based on an index relating to a difference between a pixel value of a pixel of interest in the obtained image of interest and a pixel value of a pixel corresponding to the pixel of interest in each of the plurality of converted images generated by the generating means, the determining means for determining an image area of a subject , wherein the determining means specifies pixels to be corrected in the image of interest based on the index, and corrects the specified pixels to be corrected to obtain a corrected image of interest. and determines an image area of the subject in the image of interest based on the image of interest acquired by the first acquisition means and the corrected image of interest .

According to the present invention, it is possible to extract the area of the foreground object with high accuracy regardless of whether or not the object changes (moves, etc.) with the passage of time and the condition of the object such as the density of the foreground object.

FIG. 2 is a block diagram showing the hardware configuration of the image processing apparatus according to Embodiments 1 to 3; FIG. 2 is a block diagram showing the functional configuration of the image processing apparatus according to the first embodiment; 4 is a flow chart showing the flow of processing for extracting a foreground region in the first embodiment; FIG. 4 is a diagram for explaining an outline of processing for extracting a foreground region according to the first embodiment; 4A and 4B are diagrams for explaining image conversion in the first embodiment; A diagram for explaining the effects of the first embodiment. FIG. 11 is a block diagram showing the functional configuration of the image processing apparatus according to the second embodiment; FIG. 10 is a flow chart showing the flow of processing for extracting a foreground region in the second embodiment; FIG. FIG. 5 is a diagram for explaining a continuity calculation method in Example 2; A diagram for explaining the effect of the second embodiment. FIG. 11 is a block diagram showing the functional configuration of the image processing apparatus according to the third embodiment; 10 is a flow chart showing the flow of processing for extracting a foreground region in the third embodiment; A diagram for explaining the effect of the third embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the following embodiments do not limit the present invention, and not all combinations of features described in the following embodiments are essential for the solution of the present invention. In addition, the same components will be described with the same reference numerals.

[Example 1]
In the first embodiment, a foreground subject at a viewpoint of interest is detected based on a multi-viewpoint image, specifically, an image of a background subject that partially includes images of the foreground subject at a plurality of different viewpoints (hereinafter referred to as an incomplete background image). Create an image of the background subject that does not include the image of the background (hereinafter referred to as the complete background image). The complete background image is then used to extract the foreground subject area from the image to be processed.

<Outline of processing for extracting the foreground area>
An overview of the processing for extracting the foreground region in this embodiment will be described below with reference to FIG. In this embodiment, first, background image data 401 at a plurality of different viewpoints is obtained. Background image data is an image of a background subject, a so-called background image. The background image data acquired here does not need to be a complete background image that does not include the image of the subject in the foreground (hereinafter referred to as the foreground image). It is desirable that the image is taken at It is assumed that the plurality of background image data 401 to be acquired includes an image at the same viewpoint as the viewpoint 402 from which the image from which the foreground region is to be extracted is captured. Hereinafter, an image from which a foreground region is to be extracted is called a target image (data), and a viewpoint from which the target image (data) is captured is called a viewpoint of interest.

Next, background image data 403 at the viewpoint of interest is created by converting the acquired background image data 401 for each viewpoint into an image when viewed from the viewpoint of interest 402 with the ground surface as a reference. The number of background image data 403 created here is the same as the number of background image data 401 . Background image data 403 obtained by converting the background image data 401 is hereinafter referred to as converted background image data 403 .

Here, the subject in the foreground means a subject existing in a position close to the imaging device among the subjects included in the captured image. For example, if the target image data is data of a competition scene such as a sport, the foreground subjects are people such as athletes and referees, and equipment such as goals and balls. In a plurality of images captured continuously, images that generally continue to move are included. On the other hand, the background subject means a subject behind the foreground subject because it exists at a position far from the imaging device among the subjects included in the captured image. For example, if the target image data is the data of a competition scene such as a sport, the background object may be a ground made of grass or soil, or the floor of a gymnasium, and the background object is continuous along the time series. Many of the images captured in .

Such foreground objects have height above the ground plane, while background objects do not. Therefore, using a plurality of converted background image data 403, an image (foreground image) of a subject having a height above the ground surface, that is, a subject in the foreground is detected, and the detected foreground image is removed from the incomplete background image. Thus, a complete background image at the viewpoint 402 of interest is created. Specifically, for a plurality of pieces of converted background image data 403 including an image at a viewpoint 402 of interest, the degree of matching between pixels of interest is calculated for each pixel, and pixels with a low degree of matching are pixels of the image area of the subject in the foreground. Detect as As described above, the converted background image data 403 is an image obtained by converting the background image data 401 into an image viewed from the viewpoint 402 with the ground surface as a reference plane. Therefore, the coordinates of the pixels in the areas 405 to 407 in the background image data 401 corresponding to the object 404 existing on the ground and having no height are commonly present in the same position in all the transformed background image data 403. Transformed into coordinates of pixels in region 408 . On the other hand, the coordinates of the pixels in areas 410 to 412 in the background image data 401 corresponding to the object 409 having height are converted into the coordinates of the pixels in areas 413 to 415 whose positions differ depending on the viewpoint. Therefore, in a plurality of pieces of converted background image data 403, pixels with a high degree of matching between pixels of interest are regarded as pixels in an image area of a background object that does not have a height, and pixels with a low degree of matching are regarded as having a height. The pixels in the image area of the foreground subject with This creates a complete background image. Finally, the foreground region is extracted by comparing the complete background image and the target image data at the created viewpoint 402 of interest.

The above is the outline of the processing performed in the present embodiment. Note that the target image data to be used is not limited to the above example, and various image data such as data captured by a surveillance camera can be used. Also, here, the case where the background image data 401 includes the image at the viewpoint of interest has been described, but the present embodiment can also be applied to the case where the background image data does not include the image at the viewpoint of interest. , a specific processing method will be described later.

<Hardware Configuration of Image Processing Apparatus>
The hardware configuration of the image processing apparatus of this embodiment will be described below. FIG. 1 is a block diagram showing an example of the hardware configuration of the image processing apparatus of this embodiment. The image processing apparatus 100 of this embodiment comprises a CPU 101 , a RAM 102 , a ROM 103 , a secondary storage device 104 , an input interface 105 and an output interface 106 , and these components are interconnected by a system bus 107 . The image processing apparatus 100 is also connected to an external storage device 108 via an input interface 105 , and is connected to an external storage device 108 and a display device 109 via an output interface 106 .

The CPU 101 executes programs stored in the ROM 103 using the RAM 102 as a work memory, and controls each component of the image processing apparatus 100 through the system bus 107 . As a result, various processes to be described later are executed.

The secondary storage device 104 is a storage device that stores various data handled by the image processing apparatus 100, and an HDD is used in this embodiment. The CPU 101 can write data to the secondary storage device 104 and read data stored in the secondary storage device 104 via the system bus 107 . In addition to the HDD, various storage devices such as an optical disk drive and a flash memory can be used as the secondary storage device 104 .

The input interface 105 is, for example, a serial bus interface such as USB or IEEE1394. The image processing apparatus 100 acquires data from an external storage device 108 (for example, a storage medium such as a hard disk, memory card, CF card, SD card, USB memory, etc.) via the input interface 105 . An input device (not shown) for user input, such as a mouse and a keyboard, can also be connected to the input interface 105 . The output interface 106 includes a serial bus interface such as USB and IEEE1394 similar to the input interface 105, as well as video output terminals such as DVI and HDMI (registered trademark). Data is output from the image processing apparatus 100 to an external device via the output interface 106 . The image processing apparatus 100 displays images by outputting processed images to a display device 109 (various image display devices such as a liquid crystal display) via an output interface 106 . The image processing apparatus 100 also has components other than those described above.

<Regarding the process of extracting the foreground area>
Processing for extracting a foreground region performed by the image processing apparatus 100 according to the present embodiment will be described below with reference to FIGS. 2 and 3. FIG. FIG. 2 is a block diagram showing the functional configuration of the image processing apparatus 100, and FIG. 3 is a flow chart showing the flow of processing for extracting the foreground region. The CPU 101 of the image processing apparatus 100 uses the RAM 102 as a work memory and executes the program stored in the ROM 103 to function as each component shown in FIG. 2 and execute a series of processes shown in FIG. It should be noted that the CPU 101 does not need to perform all of the processing described below, and the image processing apparatus 100 may be configured such that a part or all of the processing is performed by one or a plurality of processing circuits other than the CPU 101. good.

The flow of processing performed by each component will be described below. In step S<b>301 , the target image data acquisition unit 201 acquires target image data from the external storage device 108 or the secondary storage device 104 via the input interface 105 . As described above, the target image data is an image from which the foreground area is to be extracted. Further, the target image data acquisition unit 201 defines the viewpoint of the camera that captured the target image data as the viewpoint of interest. Although the case where the target image data is a single image is described here, the present embodiment can also be applied to the case where the target image data is a plurality of images. Furthermore, the target image data acquisition unit 201 acquires parameters of the camera that captured the target image data (hereinafter referred to as camera parameters) together with the target image data. Here, the camera parameter is a parameter that enables calculation of projecting a point in a three-dimensional space onto an image captured by the camera, and is an external parameter representing the position and orientation of the camera, a focal length, and an optical center. and an internal parameter representing Measured values and design values stored in memory in advance may be used as camera parameters. The target image data acquisition unit 201 outputs the target image data to the foreground extraction unit 207 and outputs the camera parameters to the image conversion unit 203 .

In step S<b>302 , the background image data acquisition unit 202 acquires a plurality of background image data at a plurality of different viewpoints from the external storage device 108 or the secondary storage device 104 via the input interface 105 . Here, the background image data is an image of an object in the background in an environment (weather, time of day, etc.) that is substantially the same as the environment when the target image data was captured. Note that, as described above, the background image data acquired in this step does not need to be a background image that does not include any foreground image (a complete background image).

In this embodiment, a plurality of images corresponding to a plurality of different times acquired by continuously capturing a scene from the same viewpoint along the time series are filtered using a median filter. , create background image data for each viewpoint. However, the method of creating background image data is not limited to this method. For example, the background image data may be created using another filter such as a mean filter, or the background image data may be created by performing clustering processing on a plurality of images. Alternatively, background image data obtained by capturing an image in advance without a foreground subject may be used for each viewpoint.

The background image data acquisition unit 202 also acquires camera parameters corresponding to each background image data together with the background image data. Furthermore, the background image data acquisition unit 202 stores each background image data in association with a number for distinguishing the viewpoint of the camera (hereinafter referred to as camera viewpoint number) in order to distinguish each of the plurality of background image data. The background image data acquisition unit 202 outputs the background image data and camera parameters to the image conversion unit 203 and outputs only the background image data to the correction unit 206 .

In step S303, the image conversion unit 203 uses the camera parameters acquired from the target image data acquisition unit 201 and the background image data acquisition unit 202 to transform the background image data acquired from the background image data acquisition unit 202 from the viewpoint of interest. Convert it into an image as you see it. Specifically, each piece of background image data is subjected to projective transformation with the ground surface as a reference, thereby obtaining an image as seen from the viewpoint of interest. The background image (data) obtained by image conversion in this step is called a converted background image (data). Thus, the image conversion unit 203 functions as a converted background image data creation unit. Here, the method of image conversion in this step will be described with reference to FIG.

As shown in FIG. 5, when a point 501 in a three-dimensional space is projected onto an image of a camera 502, a point 504 formed by the intersection of a straight line connecting the point 501 and the camera 502 and an image plane 503 is , is a projection image of a point 501 in the three-dimensional space onto an image plane 503. FIG. Similarly, with a camera 505 (a camera with a different viewpoint) that exists at a position different from that of the camera 502 , a point 507 formed by intersecting a straight line connecting the point 501 and the camera 505 with an image plane 506 is the image plane of the point 501 . 506 is a projected image. Here, consider the case where all the points in the three-dimensional space projected onto the image planes 503 and 506, including the point 501, are on the same plane, which is the ground plane. In this case, the coordinates ( _{u 0} , v ₀ ) are transformed to coordinates (u ₁ , v ₁ ) on the image plane 506 .

In step S303, the camera of the viewpoint corresponding to the background image data acquired from the background image data acquisition unit 202 is set to the camera 502 described above, and the camera of the viewpoint of interest determined by the target image data acquisition unit 201 is set to the camera 505. , are executed for each background image data. Therefore, the number of converted background image data acquired in this step is the same as the number of background image data acquired by the background image data acquisition unit 202 . Also, each piece of converted background image data is stored in association with the viewpoint number of each piece of background image data acquired by the background image data acquisition unit 202 . The image conversion unit 203 outputs the converted background image data to the degree-of-match calculation unit 204 and the correction unit 206 .

In step S304, the image conversion unit 203 selects, from the background image data acquired from the background image data acquisition unit 202, the image corresponding to the viewpoint closest to the camera position (viewpoint) that captured the target image data as a reference. It is defined as a background image (hereinafter referred to as a reference background image). Specifically, the distance between the coordinates (Xo, Yo, Zo) of the viewpoint of interest and the coordinates (Xi, Yi, Zi) of the viewpoint corresponding to the background image data acquired from the background image data acquisition unit 202 is calculated for each viewpoint. calculate. Here, i represents a viewpoint number, and 1≦i<number of viewpoints+1. Then, the viewpoint (reference viewpoint) that minimizes the calculated distance is detected, and the background image (data) corresponding to the reference viewpoint is set as the reference background image (data). The image conversion unit 203 outputs the viewpoint number corresponding to the reference background image to the degree-of-match calculation unit 204 and the correction unit 206 . In this embodiment, the viewpoint number corresponding to the reference background image is called the reference viewpoint number.

In step S305, the degree-of-match calculation unit 204 determines a pixel of interest in the converted background image data, which is a target for determining whether pixels match in a plurality of pieces of converted background image data. In this embodiment, first, the upper left pixel of the converted background image data is selected as the pixel of interest, and then unprocessed pixels are sequentially selected as the pixel of interest. It should be noted that the pixel of interest may be determined in any order as long as it is determined whether or not the pixels in a plurality of pieces of converted background image data match for all the pixels of the converted background image data.

In step S306, the degree-of-match calculation unit 204 uses a plurality of pieces of converted background image data acquired from the image conversion unit 203 to determine the difference between the converted background image data corresponding to the reference viewpoint number and other converted background image data. A degree of matching is calculated for the pixel of interest. A method for calculating the degree of matching will be specifically described below.

First, the matching degree calculation unit 204 acquires the pixel value B _j (u ₂ , v ₂ ) of the converted background image data at the determined coordinates (u ₂ , v ₂ ) of the pixel of interest. Here, j represents a subscript that distinguishes each of a plurality of pieces of converted background image data, and the degree-of-match calculation unit 204 acquires pixel values for the number of pieces of converted background image data. Next, the degree-of-match calculation unit 204 calculates an intermediate value of all the acquired pixel values. This intermediate value is used as the reference value M when calculating the degree of matching. Note that the reference value is not limited to this, and any value that reflects the statistical properties of a plurality of pixel values, such as an average value, may be used as the reference value.

Next, the degree-of-match calculation unit 204 calculates the degree of coincidence of the pixel of interest as the pixel value B ₀ (u ₂ , v ₂ ) of the pixel of interest in the converted background image data corresponding to the reference viewpoint number, and the calculated reference value M(u ₂ , v ₂ ) are used to calculate by equation (2).

Here, k represents a subscript for identifying RGB3 channels. The degree of matching D calculated by Equation (2) becomes smaller as the variations in pixel values in the plurality of pieces of converted background image data are smaller. Note that the matching degree to be used is not limited to this, and any value indicating the difference between pixels may be used. For example, the sum of the differences between the pixel value B ₀ (u ₂ , v ₂ ) of the pixel of interest in the converted background image data corresponding to the reference viewpoint number and the pixel value of the pixel of interest in the other converted background image data is defined as the degree of matching. You can use it as

In step S307, the degree-of-match calculation unit 204 determines whether the processing of steps S305 and S306 has been performed for all pixels of the converted background image data. If the result of determination in step S307 is true, the matching degree calculation unit 204 outputs the calculated matching degree of all pixels to the correction determination unit 205 and the calculated reference value to the correction unit 206, and proceeds to step S308. On the other hand, if the result of determination in step S307 is false, the process returns to step S305.

In step S308, the correction determination unit 205 initializes the flag map, that is, sets the pixel values of all pixels of the flag map to zero. The flag map initialized in this step is used to determine pixels to be corrected when the pixels of the converted background image data corresponding to the reference viewpoint number are corrected in step S311. In this flag map, 1 is assigned to the pixel value corresponding to the pixel to be corrected, and 0 is assigned to the pixel value corresponding to the pixel not to be corrected. Due to the initialization in this step, all pixels of the converted background image data corresponding to the reference viewpoint number are not subject to correction processing.

In step S<b>309 , the correction determination unit 205 updates the flag map based on the matching degree acquired from the matching degree calculation unit 204 . Specifically, the correction determination unit 205 determines the pixel value of the flag map corresponding to the pixel that is highly likely to be the pixel in the image area of the subject in the foreground in the converted background image data corresponding to the reference viewpoint number. to 1. In this embodiment, if the calculated degree of matching D is equal to or greater than a predetermined threshold value, the degree of matching between the pixels of the converted background image data corresponding to the reference viewpoint number and the pixels of other converted background image data is low. , it is determined that there is a high possibility that the pixel of interest is a pixel in the image area of the subject in the foreground. On the other hand, if the degree of matching D is less than the threshold, the degree of matching between the pixels of the converted background image data corresponding to the reference viewpoint number and the pixels of other converted background image data is high. Suppose that there is a high possibility that it is a pixel in a region. Note that the threshold used in this step is determined based on the maximum value of the pixel value, etc., and any value smaller than 20% of the maximum value, for example, within the range of 1% to 5% of the maximum value is used. Determine the threshold. That is, if an arbitrary value is a, the threshold is a×a×3 because the sum of squared differences is used as the degree of matching in Equation (2). Note that if the sum of differences is used as the degree of matching, the threshold is a×3. Also, whether or not the pixel is in the image area of the subject in the foreground is determined for each pixel. Correction determination section 205 outputs the updated flag map to correction section 206 .

In step S310, the correction unit 206 determines a pixel of interest in the converted background image data corresponding to the reference viewpoint number. In this embodiment, first, the upper left pixel of the converted background image data corresponding to the reference viewpoint number is selected as the pixel of interest, and then unprocessed pixels are sequentially selected as the pixel of interest. As long as the pixel values of all pixels of the converted background image data corresponding to the reference viewpoint number are updated based on the flag map (step S311), the pixels of interest may be determined in any order.

In step S<b>311 , the correction unit 206 corrects the pixel of interest in the converted background image corresponding to the reference viewpoint number based on the flag map acquired from the correction determination unit 205 . In this embodiment, when the pixel value of the flag map corresponding to the pixel of interest in the converted background image corresponding to the reference viewpoint number is 1, the pixel value of the pixel of interest is calculated using the reference value calculated by the degree-of-match calculation unit 204. replace. On the other hand, if the flag map pixel value corresponding to the pixel of interest in the converted background image corresponding to the reference viewpoint number is 0, the pixel value of the pixel of interest is not changed. Note that the method of correcting pixel values is not limited to this, and other methods such as replacing with pixel values of a background image corresponding to a viewpoint adjacent to the reference viewpoint may be used.

In step S312, the correction unit 206 determines whether the processing of steps S310 to S311 has been performed for all pixels of the converted background image data corresponding to the reference viewpoint number. If the determination result in step S312 is true, the correction unit 206 outputs the converted background image data corresponding to the corrected reference viewpoint number to the foreground extraction unit 207, and proceeds to step S313. If the result is false, return to step S310.

In step _S313 , the foreground extraction unit 207 extracts target image data (I and the foreground subject area is extracted from the foreground subject. Specifically, as shown in Equation (3), the sum of squared differences is calculated for each pixel between the complete background image I _b and the target image data I, and the pixels whose sum of squared differences is equal to or greater than a threshold are By regarding pixels in the image area of the subject, an image If in which the area of the subject in the _foreground is extracted is created. The image _If is a binary image in which 1 is assigned to the pixel values corresponding to the pixels in the foreground subject image area, and 0 is assigned to the pixel values corresponding to the pixels in the background subject image area.

Here, Th represents a threshold, and k represents a subscript for identifying RGB3 channels. Note that the threshold value used here is determined based on the maximum value of the pixel value, etc., and a value smaller than 20% of the maximum value of the pixel value, for example, any value within the range of 1% to 5% of the maximum value. may be used to determine the threshold. The method of obtaining this threshold is the same as in the case of equation (2). Thus, the foreground extraction unit 207 functions as a foreground image data creation unit. The foreground extraction unit 207 outputs the created image _If to the secondary storage device 104, the external storage device 108, or the display device 109, and a series of processing is completed. The above is the process for extracting the foreground region, which is executed by the image processing apparatus 100 according to the present embodiment.

<Effects of this embodiment>
The effects of this embodiment will be described below with reference to FIG. In FIG. 6, image data 601 is background image data at a viewpoint 602 created based on a plurality of images captured continuously in time series according to the conventional method. Background image data 601 includes foreground subjects such as a foreground subject 603 (goalkeeper) and a foreground subject (goal) 604 . The reason for this is that the foreground subjects 603 and 604 are in the same position and do not move when the continuous images for creating the background image data are captured. , 604 was mistakenly considered as a background object. When the foreground area is extracted from the target image data 605 using the background image data 601, the foreground image data 606 is obtained. In the foreground image data 606, an area of the foreground subject that is generally moving, other than the subjects 603 and 604, can be extracted. However, the area of the stationary foreground objects 603 and 604 cannot be extracted.

Image data 607 is background image data at a viewpoint 602 created based on a plurality of images captured from a plurality of different viewpoints at the same time as the target image data 605 was captured according to the conventional method. Although the background image data 607 does not include foreground subjects such as the foreground subject 603 (goalkeeper) and the foreground subject (goal) 604, part of the background subject is missing. The reason for this is that the foreground subjects are densely packed in the scene imaged to create the background image data, and some of the foreground subjects cannot be seen from a plurality of viewpoints. When the foreground area is extracted from the target image data 605 using the background image data 607, the foreground image data 608 is obtained. In the foreground image data 608, the area of the foreground subject having a height above the ground surface can be roughly extracted. However, since the foreground subjects are densely packed, an area 609 of the foreground subjects that cannot be seen from multiple viewpoints cannot be extracted.

In contrast, in this embodiment, background image data 610, which is a complete background image, is created using incomplete background images (for example, background image data 601) at a plurality of different viewpoints. When the foreground area is extracted from the target image data 605 using the background image data 610, foreground image data 611 is obtained. In the foreground image data 611, the areas of the stationary foreground subjects 603 and 604 and the areas of the foreground subjects that cannot be seen from a plurality of viewpoints can be extracted with high accuracy. As described above, according to the present embodiment, regardless of the presence or absence of change (movement, etc.) of the subject with the passage of time and the state of the subject such as the density of the foreground subject, the area of the foreground subject can be extracted with high accuracy. can do.

[Example 2]
In the first embodiment, when a complete background image is created based on a plurality of incomplete background images, the degree of matching, which indicates the degree of matching of the pixel of interest of the transformed background image that differs depending on the viewpoint, is used. On the other hand, in this embodiment, when creating a complete background image based on a plurality of incomplete background images, in addition to the degree of matching, the degree of smoothness of change in the pixel value of the pixel of interest in the converted background image, which varies depending on the viewpoint, is calculated. , so-called continuity is used. In addition, the same reference numerals as in the first embodiment are attached to the same configuration and the same processing as in the first embodiment, and the description thereof is omitted.

<Outline of processing for extracting the foreground area>
An overview of the processing for extracting the foreground region in this embodiment will be described below. In this embodiment, the continuity of pixel values between viewpoints is calculated using transformed background image data obtained by transforming background image data at a plurality of different viewpoints into an image viewed from the viewpoint of interest. The continuity of pixel values is the degree of smoothness of change in pixel values between the converted background image data at a viewpoint of interest and the converted background image data at a viewpoint adjacent to the viewpoint of interest.

Specifically, the pixel value of the pixel of interest in the converted background image data corresponding to the reference viewpoint number is compared with the pixel value of the pixel of interest in the converted background image data at a viewpoint adjacent to the reference viewpoint, and the difference between the pixel values is calculated. The sum of is calculated as continuity. Subsequently, using the degree of matching described in the first embodiment and the continuity calculated in the present embodiment, pixels with a low degree of matching and uneven pixel value changes are extracted from the image area of the foreground subject. , and is detected as a pixel to be corrected. Then, by updating the pixel values of the detected correction target pixels and correcting the converted background image data, a complete background image is created. Finally, the created complete background image is compared with the target image data to extract the foreground region.

In Example 1, it is determined whether the pixel of interest is a pixel in the image area of the subject in the foreground using only the matching degree of the pixel values calculated based on the background image data at all viewpoints. Therefore, it is highly likely that pixels in the image area of the background subject that have different pixel values due to changes in color appearance depending on the viewpoint are also pixels in the image area of the foreground subject. detected. As a result, since pixels that do not need to be corrected are also corrected, errors occur in the converted background image data after correction, and a complete background image that does not include the image of the subject in the foreground cannot be created with high accuracy. As a background subject whose appearance changes depending on the viewpoint, grass that is mowed with directionality, which is present in an image of a competition scene such as a sport, is exemplified. In grass that is mowed with directionality, the appearance of the color of the grass differs depending on the viewing direction, and as a result, even if the grass is at the same position, the pixel value changes depending on the viewpoint. When the first embodiment is applied to such a scene in which grass is the background subject, the degree of matching between pixels in a plurality of pieces of converted background image data is low. It is erroneously determined to be a pixel in the image area of the subject in the foreground. In order to prevent such an erroneous determination, in this embodiment, continuity is used in addition to the degree of matching to determine whether the pixel of interest is a pixel in the image area of the subject in the foreground. In general, for a subject whose color appearance changes depending on the viewpoint, there may be a noticeable difference in color appearance between distant viewpoints, but there is no change in color appearance between close viewpoints. Moderate. Therefore, in this embodiment, pixels in the image area of the background subject whose pixel values change due to the difference in color appearance, and an image area of the foreground subject whose pixel values change due to the height above the ground surface. to distinguish between pixels of As a result, the converted background image can be corrected with high accuracy to create a complete background image, so the foreground object area can be extracted with high accuracy from the target image data. It should be noted that the subject whose color appearance changes depending on the viewpoint is not limited to the above example of the lawn, and there are various other subjects such as the floor of a gymnasium.

<Regarding the process of extracting the foreground area>
Processing for extracting a foreground region performed by the image processing apparatus 100 according to this embodiment will be described below with reference to FIGS. 7 and 8. FIG. FIG. 7 is a block diagram showing the functional configuration of the image processing apparatus 100 in this embodiment, and FIG. 8 is a flow chart showing the flow of processing for extracting a foreground region in this embodiment. The CPU 101 of the image processing apparatus 100 uses the RAM 102 as a work memory and executes the program stored in the ROM 103 to function as each component shown in FIG. 7 and execute a series of processes shown in FIG. It should be noted that the CPU 101 does not need to perform all of the processing described below, and the image processing apparatus 100 may be configured such that a part or all of the processing is performed by one or a plurality of processing circuits other than the CPU 101. good.

In step S801, the continuity calculation unit 701 determines a pixel of interest in the converted background image data for which continuity is to be calculated. In this embodiment, first, the upper left pixel of the converted background image is selected as the pixel of interest, and then unprocessed pixels are sequentially selected as the pixel of interest. Note that the pixels of interest may be determined in any order as long as continuity calculation is executed for all pixels of the converted background image data.

In step S802, the continuity calculation unit 701 uses the converted background image data acquired from the image conversion unit 203 to calculate the continuity of the pixel values of the pixel of interest in the converted background images corresponding to the reference viewpoint and its surrounding viewpoints. Calculate Here, a continuity calculation method in this step will be described with reference to FIG.

First, the cameras 902 and 903 adjacent to the camera 901 corresponding to the reference viewpoint number determined by the image conversion unit 203 are detected, and the viewpoint numbers corresponding to these cameras are acquired. The acquired viewpoint number is hereinafter referred to as an adjacent viewpoint number. Here, the camera adjacent to the camera 901 corresponding to the reference viewpoint number is determined based on the distance to the camera 901 calculated from the coordinates of the camera in the three-dimensional space. In this embodiment, a camera 902, which has the shortest distance to the camera 901 among the cameras existing on the left side of the camera 901, and a camera 903, which has the shortest distance to the camera 901 among the cameras existing on the right side of the camera 901. is detected as a camera adjacent to camera 901 .

Next, pixel values of pixels 907, 908, and 909 at the coordinates (u ₂ , v ₂ ) of the target pixel are calculated from the converted background image 904 corresponding to the reference viewpoint number and the converted background images 905 and 906 corresponding to the adjacent viewpoint numbers. is obtained, and using the obtained pixel values, the continuity is calculated by Equation (4).

where B ₉₀₁ (u ₂ , v ₂ ), B ₉₀₂ (u ₂ , v ₂ ), B ₉₀₃ (u ₂ , v ₂ ) are transformed background images 904, 905 corresponding to cameras 901, 902, 903, respectively. , 906 represent pixel values of target pixels 907 , 908 , and 909 . Also, k represents a subscript for identifying RGB3 channels. The value of C calculated by Equation (4) becomes smaller as the change in pixel value between viewpoints becomes smoother. Note that the continuity to be used is not limited to C calculated by Equation (4), and any value that indicates the continuity of pixel values between viewpoints, such as a second-order differential from a discrete value, may be used. In this embodiment, the camera 901 corresponding to the reference viewpoint number and the cameras 902 and 903 adjacent to each other are used. However, the cameras to be used are not limited to these. You can use it. For example, on the left side of the camera 901 corresponding to the reference viewpoint number, instead of the camera 902, a camera next to the camera 902 and closest to the camera 901 may be used. The same is true for the camera used on the right side of camera 901 .

In step S803, the continuity calculation unit 701 determines whether the processing of steps S801 and S802 has been performed for all pixels of the converted background image data. If the determination result in step S803 is true, the continuity calculation unit 701 outputs the calculated continuity of all pixels to the correction determination unit 702, and proceeds to step S308. Return to S801.

In step S<b>804 , the correction determination unit 702 updates the flag map based on the matching degree obtained from the matching degree calculation unit 204 and the continuity obtained from the continuity calculation unit 701 . Specifically, the correction determination unit 702 determines the pixels of the flag map corresponding to the pixels that are highly likely to be the pixels in the image area of the subject in the foreground in the converted background image data corresponding to the reference viewpoint number. Change the value to 1. In this embodiment, if the calculated degree of matching D is equal to or greater than a predetermined threshold and the calculated continuity C is equal to or greater than a predetermined threshold, the converted background image corresponding to the reference viewpoint number and the other converted background are Assume that the degree of matching and the degree of smoothness of change in the pixel of interest with the image are low. That is, it is determined that there is a high possibility that the pixel of interest is a pixel in the image area of the subject in the foreground. On the other hand, if these conditions are not satisfied, it is determined that there is a high possibility that the pixel of interest is a pixel in the image area of the background subject. Note that the threshold used in this step is determined based on the maximum value of the pixel value, etc., and any value smaller than 20% of the maximum value, for example, within the range of 1% to 5% of the maximum value is used. You can find the threshold. The method of obtaining this threshold value is the same as in the first embodiment. Further, whether or not the pixel is in the image area of the subject in the foreground is determined for each pixel. The correction determination unit 702 outputs the updated flag map to the correction unit 206 . The above is the process for extracting the foreground region, which is executed by the image processing apparatus 100 according to the present embodiment.

<Effects of this embodiment>
The effects of this embodiment will be described below with reference to FIG. The image data 1002 is converted background image data obtained by converting the background image data for each viewpoint into an image viewed from the viewpoint 1001 with reference to the ground surface. Here, it is assumed that the viewpoint 1001 is the viewpoint of interest and the reference viewpoint. A subject 1003 is a background subject (for example, lawn) whose color changes depending on the viewpoint, and a subject 1005 is a foreground subject.

When the first embodiment is applied to the scene shown in FIG. 10 and a complete background image is created based on the multi-viewpoint incomplete background image, background image data 1004 is acquired. In the background image data 1004, the image of the subject 1005 in the foreground is removed, but the subject 1003 in the background is not captured correctly. The reason for this is that when a complete background image is created, whether or not the pixel of interest is a pixel in the image area of the subject in the foreground is determined based only on the degree of matching. This is because the pixels are determined to be pixels in the image area of the subject. As a result, background image data 1004 in which the pixels in the image area of the background subject 1003 are corrected is created. Even if an attempt is made to extract the foreground area from the target image data using the background image data 1004, the foreground area cannot be extracted with high accuracy.

On the other hand, in this embodiment, when creating a complete background image based on an incomplete multi-viewpoint background image, whether or not the pixel of interest is a pixel in the image area of the subject in the foreground is determined by matching degree and continuity. Based on As a result, background image data 1006 in which pixels in the image area of the background subject 1003 are not corrected is generated without determining pixels in the image area of the background subject 1003 as pixels in the image area of the foreground subject. In the background image data 1006, the image of the subject 1005 in the foreground is removed while the subject 1003 in the background is captured correctly. By extracting the foreground area from the target image data using the background image data 1006, the foreground area can be extracted with high accuracy. As described above, according to the present embodiment, even when the background subject is a subject whose color appearance changes depending on the viewpoint, the area of the foreground subject can be extracted with high accuracy.

[Example 3]
In Example 1 and Example 2, a complete background image is created based on incomplete background images at a plurality of different viewpoints, and the created complete background image and target image data are compared to determine the Extract regions. On the other hand, in the present embodiment, incomplete foreground images at a plurality of different viewpoints are used to extract the foreground region so as not to include regions due to shadows. Here, the imperfect foreground image means an image in which a foreground subject area and a shadow area associated with the foreground subject are extracted as the foreground area.

In the present embodiment, a plurality of transformed foreground image data are acquired by transforming an incomplete foreground image for each viewpoint into an image viewed from the viewpoint of interest with reference to the ground surface. A degree of matching between pixels is calculated in the converted foreground image data. As described in the first embodiment, the foreground subject has a height above the ground, but the shadow attached to the foreground subject does not have a height above the ground. Therefore, in this embodiment, pixels with a high degree of matching between pixels are detected in a plurality of pieces of converted foreground image data, and the detected pixels are corrected as highly likely to be pixels in a shadow area having no height. . As a result, it is possible to create a foreground image in which only the area of the subject in the foreground having height is extracted as the foreground area without extracting the shadow area. Hereinafter, an image in which only a foreground object area having a height is extracted as a foreground area without extracting a shadow area is referred to as a complete foreground image. Note that the same reference numerals as in the above-described embodiment are attached to the same configuration and the same processing as those in the above-described embodiment, and the description thereof is omitted.

<Regarding the process of extracting the foreground area>
Processing for extracting a foreground region performed by the image processing apparatus 100 according to this embodiment will be described below with reference to FIGS. 11 and 12. FIG. FIG. 11 is a block diagram showing the functional configuration of the image processing apparatus 100 in this embodiment, and FIG. 12 is a flow chart showing the flow of processing for extracting the foreground region in this embodiment. The CPU 101 of the image processing apparatus 100 uses the RAM 102 as a work memory to execute the program stored in the ROM 103, thereby functioning as each component shown in FIG. 11 and executing a series of processes shown in FIG. It should be noted that the CPU 101 does not need to perform all of the processing described below, and the image processing apparatus 100 may be configured such that a part or all of the processing is performed by one or a plurality of processing circuits other than the CPU 101. good.

In step S<b>1201 , the camera parameter acquisition unit 1101 acquires camera parameters of the camera that captured the target image data from the external storage device 108 or the secondary storage device 104 via the input interface 105 . Also, the camera parameter acquisition unit 1101 determines the viewpoint of the camera that captured the target image data as the viewpoint of interest. The camera parameters acquired in this step are the same as the camera parameters described in the first embodiment. The camera parameter acquisition unit 1101 outputs camera parameters to the image conversion unit 1103 .

In step S<b>1202 , the foreground image data acquisition unit 1102 acquires a plurality of foreground image data at a plurality of different viewpoints from the external storage device 108 or the secondary storage device 104 via the input interface 105 . The foreground image data acquired in this step is an image obtained by extracting a foreground object area, and the extracted area includes a shadow area. In this embodiment, the foreground image data is created based on the captured image captured in advance and the background image. A method for creating the foreground image data will be specifically described below. The captured image used here is an image in which the foreground subject and the background subject in the target image data are captured in substantially the same environment as the environment in which the target image data was captured. Also, the background image is an image of the subject in the background of the target image data captured in substantially the same environment as the environment in which the target image data was captured. In this embodiment, for each viewpoint, the pixel value of the captured image and the pixel value of the background image are compared pixel by pixel. is set to 1, a binary image is created for each viewpoint. This binary image is the foreground image data. Note that the method for creating foreground image data is not limited to this, and the foreground image data to be created is not limited to a binary image, and may be a multivalued image. Also, the foreground image data acquisition unit 1102 acquires camera parameters corresponding to each foreground image data together with the foreground image data. Furthermore, the foreground image data acquisition unit 1102 stores each foreground image data in association with the viewpoint number of the camera in order to distinguish each of the plurality of foreground image data. The foreground image data acquisition unit 1102 outputs the foreground image data and camera parameters to the image conversion unit 1103 .

In step S1203, the image conversion unit 1103 uses the camera parameters obtained from the camera parameter acquisition unit 1101 and the foreground image data acquisition unit 1102 to convert the foreground image data obtained from the foreground image data acquisition unit 1102 from the viewpoint of interest. Convert to an image of the case. The transformation in this step is similar to that in step S303 of the first embodiment, and for each viewpoint, the foreground image data is subject to projective transformation with the ground plane as a reference, thereby obtaining an image viewed from the viewpoint of interest. The foreground image (data) obtained by image conversion in this step is called a converted foreground image (data). Thus, the image conversion unit 1103 functions as a converted foreground image data creating unit. The image conversion unit 1103 outputs the converted foreground image data to the degree-of-match calculation unit 1104 .

In step S1204, the image conversion unit 1103 converts an image corresponding to a viewpoint closest to the camera position (viewpoint of interest) that captured the target image data from the foreground image data acquired from the foreground image data acquisition unit 1102 to a reference foreground. It is defined as an image (hereinafter referred to as a reference foreground image). Specifically, the distance between the coordinates of the viewpoint of interest and the coordinates of the viewpoint corresponding to the foreground image data is calculated for each viewpoint. Then, the foreground image (data) corresponding to the viewpoint (reference viewpoint) with the smallest calculated distance is set as the reference foreground image (data). The image conversion unit 1103 outputs the viewpoint number corresponding to the reference foreground image to the correction unit 1105 . In this embodiment, the viewpoint number corresponding to the reference foreground image is called the reference viewpoint number.

In step S1205, the degree-of-match calculation unit 1104 determines a pixel of interest in the converted foreground image data, which is a target for determining whether pixels match in a plurality of pieces of converted foreground image data. In this embodiment, first, the upper left pixel of the converted foreground image data is selected as the pixel of interest, and then unprocessed pixels are sequentially selected as the pixel of interest. It should be noted that the pixel of interest may be determined in any order as long as it is determined whether pixels in a plurality of pieces of converted foreground image data match for all the pixels of the converted foreground image data.

In step S1206, the degree-of-match calculation unit 1104 uses a plurality of pieces of converted foreground image data acquired from the image conversion unit 1103 to determine the difference between the converted foreground image data corresponding to the reference viewpoint number and other converted foreground image data A degree of matching is calculated for the pixel of interest. A method for calculating the degree of matching will be specifically described below.

First, the matching degree calculation unit 1104 acquires the pixel value F _l (u ₂ , v ₂ ) of the transformed foreground image data at the determined coordinates (u ₂ , v ₂ ) of the pixel of interest. Here, l represents a subscript that distinguishes each of a plurality of pieces of converted foreground image data, and the degree-of-match calculation unit 1104 acquires pixel values for the number of pieces of converted foreground image data. Next, the degree-of-match calculation unit 1104 calculates the average value of all the acquired pixel values. In this embodiment, this average value is used as the matching degree. Also, the degree of matching is not limited to this, and a value that reflects the statistical properties of a plurality of pixel values may be used as the degree of matching.

In step S1207, the degree-of-match calculation unit 1104 determines whether the processing of steps 1205 and S1206 has been performed for all pixels of the converted foreground image data. If the result of the determination in step S1207 is true, the matching degree calculation unit 1104 outputs the calculated matching degrees of all pixels to the correcting unit 1105, and proceeds to step S1208. On the other hand, if the result of determination in step S1207 is false, the process returns to step S1205.

In step S1208, the correction unit 1105 determines a pixel of interest in the converted foreground image data corresponding to the reference viewpoint number. In this embodiment, first, the upper left pixel of the converted foreground image data corresponding to the reference viewpoint number is selected as the pixel of interest, and unprocessed pixels are sequentially selected as the pixel of interest. As long as pixel values are updated (step S1209) based on the degree of matching for all pixels of the converted foreground image data corresponding to the reference viewpoint number, the pixels of interest may be determined in any order.

In step S<b>1209 , the correction unit 1105 detects pixels that are highly likely to be pixels in a shadow area in the transformed foreground image corresponding to the reference viewpoint number, based on the degree of matching acquired from the degree of matching calculation unit 1104 . Then, the correction unit 1105 changes the pixel value of the detected pixel to 0, thereby removing the region due to the shadow from the incomplete foreground image. In this embodiment, if the calculated degree of matching is equal to or greater than a predetermined threshold value, the degree of matching between pixels of interest from all viewpoints is high. judged to be of high quality. Then, the pixel value of the pixel of interest in the converted foreground image data corresponding to the reference viewpoint number is changed to zero. On the other hand, if the calculated degree of matching is less than the threshold, it is determined that the degree of matching between pixels of interest in all viewpoints is low, and that the pixel of interest is likely to be a pixel in a region of a tall foreground subject. In this case, the pixel value of the pixel of interest in the converted foreground image corresponding to the reference viewpoint number is not changed. Although 0.8 is used as the threshold value in this embodiment, the value of the threshold value is not limited to this.

In step S1210, the correction unit 1105 determines whether the processing of steps S1208 and S1209 has been performed for all pixels of the converted foreground image data corresponding to the reference viewpoint number. If the result of determination in step S1210 is true, the correction unit 1105 outputs the converted foreground image data corresponding to the corrected reference viewpoint number to the secondary storage device 104, the external storage device 108, or the display device 109. , the series of processing is completed. On the other hand, if the determination result in step S1210 is false, the process returns to step S1208. The above is the process for extracting the foreground region, which is executed by the image processing apparatus 100 according to the present embodiment.

<Effects of this embodiment>
The effects of this embodiment will be described below with reference to FIG. A subject 1301 is a foreground subject whose shadow 1302 is present on the ground surface 1303 . The image data 1304 is a foreground image obtained by extracting the area of the object 1301 and the shadow 1302 accompanying the subject 1301 at a plurality of different viewpoints as the foreground area. In this embodiment, the height from the ground surface is determined based on the degree of matching between pixels of interest in a plurality of converted foreground image data obtained by converting the image data 1304 into an image when viewed from the viewpoint 1305 of interest. Detect pixels in areas with no shadows. A foreground image 1306 is created by correcting the detected pixels. In the foreground image 1306, the area of the shadow 1302 attached to the foreground object 1301 having height is removed, and only the area of the foreground object 1301 can be extracted. As described above, according to the present embodiment, even if there is a shadow attached to a tall foreground subject, only the area due to the foreground subject is extracted with high accuracy without extracting the area due to the shadow. can be extracted to

In this embodiment, as an incomplete foreground image, a foreground image created based on an image captured in advance and a background image is used. good too. In this case, the area of the foreground subject can be extracted with higher accuracy than when the first or second embodiment and the third embodiment are executed independently.

[Other Examples]
Embodiments of the present invention are not limited to the examples described above, but can take various forms. For example, in the above embodiment, the size of the background image data, which is an incomplete background image, and the size of the target image data are the same, but these sizes may not be the same. . In that case, the viewpoint of the ground surface from above is used as a reference viewpoint, and the background image is converted into an image when viewed from the reference viewpoint. Then, the background image is corrected using the converted image, and the corrected background image is converted into an image when viewed from the viewpoint of interest, thereby creating background image data corresponding to the target image data.

In addition, in the above-described embodiment, the pixel values in the RGB space are used in the calculation of the degree of matching and the extraction of the foreground, but the information to be used is not limited to this. For example, pixel values in different color spaces such as HSV and Lab may be used to calculate the degree of matching and extract the foreground.

Furthermore, in the above-described embodiment, only one plane of the ground surface is used as a reference when projectively transforming an image, but a plurality of planes parallel to the ground surface may be used as a reference. For example, a plurality of planes are set by dividing the height from the ground surface from 0 to 1 cm at equal intervals, and all transformed images obtained by projective transformation based on each of the set planes are used. The degree of matching may be calculated using the By doing so, the robustness against camera parameter errors is improved.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

201... Target image data acquisition unit 202... Background image data acquisition unit 203... Image conversion unit 204... Match degree calculation unit 206... Correction unit 207... Foreground extraction unit

Claims (18)

a first acquisition means for acquiring an image of interest including a subject, which is captured and acquired from a viewpoint of interest;
a second obtaining means for obtaining a plurality of reference images captured from a plurality of viewpoints different from the viewpoint of interest;
generating means for transforming the plurality of reference images acquired by the second acquisition means to generate a plurality of transformed images viewed from the viewpoint of interest;
Based on an index relating to the difference between the pixel value of the pixel of interest in the image of interest acquired by the first acquisition means and the pixel value of the pixel corresponding to the pixel of interest in each of the plurality of converted images generated by the generation means and determining means for determining an image area of the subject in the image of interest,
The determination means identifies pixels to be corrected in the image of interest based on the index, corrects the identified pixels to be corrected, thereby generating a corrected image of interest, and the image of interest acquired by the first acquisition means. and determining an image area of the subject in the image of interest based on the corrected image of interest.
An image processing apparatus characterized by: 2. The image processing apparatus according to claim 1, wherein said determining means specifies pixels for which said index is equal to or greater than a threshold value as pixels to be corrected. The determining means determines the image area of the subject in the target image based on another index related to the degree of smoothness of change in pixel values of pixels corresponding to the target pixel in each of the plurality of converted images. 3. The image processing apparatus according to claim 1, wherein: 4. The image processing apparatus according to claim 3, wherein said another index is an index based on a converted image at a viewpoint closest to said viewpoint of interest and a converted image at a viewpoint adjacent to said viewpoint. 5. The image processing apparatus according to claim 3, wherein said determining means specifies pixels for which said index is equal to or greater than a threshold and said other index is equal to or greater than said threshold as pixels to be corrected. The index is the difference between the pixel value of the pixel of interest in the image of interest acquired by the first acquisition means and the pixel value of the pixel corresponding to the pixel of interest in each of the plurality of converted images generated by the generation means. 6. The image processing apparatus according to any one of claims 1 to 5, characterized in that it is an index related to summation. a first acquisition means for acquiring an image of interest including a subject, which is captured and acquired from a viewpoint of interest;
a second obtaining means for obtaining a plurality of reference images captured from a plurality of viewpoints different from the viewpoint of interest;
generating means for transforming the plurality of reference images acquired by the second acquisition means to generate a plurality of transformed images viewed from the viewpoint of interest;
An index relating to a difference between a pixel value of a pixel of interest in the image of interest acquired by the first acquisition means and a statistical value of pixel values of pixels corresponding to the pixel of interest in each of the plurality of converted images generated by the generation means. determining means for determining the image area of the subject in the image of interest based on;
An image processing device having 8. The image processing apparatus according to claim 7 , wherein the statistical value is an intermediate value or average value of pixel values of pixels corresponding to the target pixel in each of the plurality of converted images. 9. The method according to any one of claims 1 to 8 , wherein the generation means performs projective transformation on each of the plurality of reference images into an image viewed from the viewpoint of interest with reference to the ground surface. Image processing device. wherein said generating means projectively transforms each of said plurality of reference images into an image viewed from said viewpoint of interest with reference to said ground surface and a plurality of planes parallel to said ground surface. 10. The image processing device according to Item 9 . The determining means is
identifying a pixel to be corrected in the image of interest based on the index;
generating a corrected image of interest by correcting the identified correction target pixels;
11. The image area of the object in the image of interest is determined based on the image of interest acquired by the first acquisition unit and the corrected image of interest . The described image processing device. 12. The image processing apparatus according to claim 11 , wherein said determining means specifies pixels for which said index is equal to or greater than a threshold value as pixels to be corrected. The determining means determines the image area of the subject in the target image based on another index related to the degree of smoothness of change in pixel values of pixels corresponding to the target pixel in each of the plurality of converted images. 13. The image processing apparatus according to claim 11 , characterized by: 14. The image processing apparatus according to claim 13 , wherein said another index is an index based on a transformed image at a viewpoint closest to said viewpoint of interest and a transformed image at a viewpoint adjacent to said viewpoint. 15. The image processing apparatus according to claim 13 , wherein the determination unit specifies pixels for which the index is equal to or greater than a threshold and the other index is equal to or greater than a threshold as pixels to be corrected. a first acquisition step of acquiring an image of interest including the subject, which is captured and acquired from the viewpoint of interest;
a second obtaining step of obtaining a plurality of reference images captured and obtained from a plurality of viewpoints different from the viewpoint of interest;
a generation step of transforming the plurality of reference images acquired in the second acquisition step to generate a plurality of transformed images viewed from the viewpoint of interest;
Based on an index relating to the difference between the pixel value of the pixel of interest in the image of interest acquired by the first acquisition step and the pixel value of the pixel corresponding to the pixel of interest in each of the plurality of converted images generated by the generation step and a determining step of determining an image area of the subject in the image of interest,
In the determining step, based on the index, pixels to be corrected in the image of interest are identified, the identified pixels to be corrected are corrected to generate a corrected image of interest, and the image of interest obtained by the first obtaining step is generated. and determining an image area of the subject in the image of interest based on the corrected image of interest.
An image processing method comprising: a first acquisition step of acquiring an image of interest including the subject, which is captured and acquired from the viewpoint of interest;
a second obtaining step of obtaining a plurality of reference images captured and obtained from a plurality of viewpoints different from the viewpoint of interest;
a generation step of transforming the plurality of reference images acquired in the second acquisition step to generate a plurality of transformed images viewed from the viewpoint of interest;
An index relating to the difference between the pixel value of the pixel of interest in the image of interest acquired by the first acquisition step and the statistical value of the pixel values of the pixels corresponding to the pixel of interest in each of the plurality of converted images generated by the generation step a determination step of determining the image area of the subject in the image of interest based on;
An image processing method comprising: A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 15 .

Images