JP2021108193A - Image processing device, image processing method, and program - Google Patents

Abstract

To extract an area due to a subject in foreground with high accuracy, irrespective of a subject state such as presence or absence of a change of subject (movement, etc.), or dense or sparse arrangement of the subject in the foreground over time.SOLUTION: An image processing device according to the present invention is characterized to comprise: means that acquires target image data being an image by imaging a subject in foreground and a subject in background from an interest viewpoint; means that acquires background image data being the image of the subject in the background in a plurality of different viewpoints; means that generates a plurality of pieces of converted background image data by converting the acquired plurality of pieces of background image data; calculation means that calculates a coincidence degree between interest pixels in the plurality of pieces of converted background image data; means that corrects the converted background image data in a reference viewpoint based on the coincidence degree; and means that generates the foreground image data being an image, in which an area due to the subject in the foreground is extracted, based on the target image data and the corrected converted background image data.SELECTED DRAWING: Figure 3

Description

The present invention relates to a technique for extracting a region of a foreground subject from a captured image.

Conventionally, there is a background subtraction method as a method of extracting a region by a subject in the foreground from an captured image acquired by imaging a subject (including a subject in the foreground and a subject in the background). In the background subtraction method, the subject in the foreground is based on the pixel value of the captured image in which the subject in the foreground and the subject in the background are shown and the pixel value in the background image in which only the subject in the background is shown. Create a foreground image by extracting the area by. At this time, when an image showing only the background imaged in advance under specific conditions is used as the background image, if the background changes due to a change in sunshine over time, the area of the subject in the foreground is extracted. There was a problem that the accuracy was lowered.

In order to solve the above problem, Patent Document 1 discloses a technique for extracting a region due to a subject in the foreground regardless of a change in the background by using a background image created based on a plurality of images having different imaging times. do.

Further, Patent Document 2 discloses a technique of extracting a region of a subject in the foreground by using a background image created based on a plurality of images taken from different viewpoints at the same time regardless of the change of the subject with the passage of time. Disclose.

Japanese Unexamined Patent Publication No. 2012-104053 Japanese Unexamined Patent Publication No. 2014-230180

However, in Patent Document 1, when the subject in the foreground is stationary and stopped, the area due to the subject in the foreground is erroneously determined as the area due to the subject in the background, so that the background image cannot be created accurately. Therefore, there is a problem that the accuracy of extracting the region by the subject in the foreground is lowered.

Further, in Patent Document 2, a background image is created by supplementing information on a background subject that cannot be seen from a single viewpoint with information from another viewpoint, but foreground subjects existing in the scene are densely packed in the foreground. The background image cannot be created accurately in the area where the subjects overlap. Therefore, there is a problem that the accuracy of extracting the region by the subject in the foreground is lowered.

Therefore, in view of the above problems, the present invention has high accuracy in determining the area of the foreground subject regardless of the subject state such as the presence or absence of changes (movement, etc.) of the subject with the passage of time and the sparseness of the foreground subject. The purpose is to extract.

The present invention comprises a target image data acquisition means for acquiring target image data which is an image of a foreground subject and a background subject from a viewpoint of interest, and a background image which is an image of the background subject from a plurality of different viewpoints. Converted background image data that creates a plurality of converted background image data by converting the background image data acquisition means for acquiring data and the plurality of acquired background image data into images when viewed from the viewpoint of interest. The creation means, the calculation means for calculating the degree of matching indicating the degree of matching between the pixels of interest in the plurality of converted background image data, and the converted background image data in the viewpoint determined according to the distance from the viewpoint of interest. Foreground image data that creates foreground image data that is an image in which a region of the foreground subject is extracted based on the correction means that corrects based on the degree of coincidence, the target image data, and the corrected conversion background image data. It is an image processing apparatus characterized by having a creating means.

According to the present invention, it is possible to extract a region of a subject in the foreground with high accuracy regardless of the state of the subject such as the presence or absence of a change (movement or the like) of the subject with the passage of time and the density of the subject in the foreground.

A block diagram showing a hardware configuration of the image processing apparatus according to the first to third embodiments. The block diagram which shows the functional structure of the image processing apparatus in Example 1. Flow chart showing the flow of the process of extracting the foreground region in the first embodiment The figure explaining the outline of the process of extracting the foreground region in Example 1. The figure explaining the image conversion in Example 1. The figure explaining the effect of Example 1. The block diagram which shows the functional structure of the image processing apparatus in Example 2. Flow chart showing the flow of the process of extracting the foreground region in the second embodiment The figure explaining the continuity calculation method in Example 2. The figure explaining the effect of Example 2. The block diagram which shows the functional structure of the image processing apparatus in Example 3. Flow chart showing the flow of the process of extracting the foreground region in the third embodiment The figure explaining the effect of Example 3.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the following embodiments do not limit the present invention, and not all combinations of features described in the following embodiments are essential to the means of solving the present invention. The same components will be described with the same reference numerals.

[Example 1]
In the first embodiment, the subject in the foreground at the viewpoint of interest is based on a multi-viewpoint image, specifically, an image of a background subject including a part of an image of the subject in the foreground at a plurality of different viewpoints (hereinafter, an incomplete background image). Create an image of the subject in the background that does not include the image of (hereinafter referred to as the complete background image). Then, using the complete background image, the area of the subject in the foreground is extracted from the image to be processed.

<Overview of the process of extracting the foreground area>
Hereinafter, the outline of the process of extracting the foreground region in this embodiment will be described with reference to FIG. In this embodiment, first, background image data 401 from a plurality of different viewpoints is acquired. The background image data is an image of a subject in the background, a so-called background image. The background image data acquired here does not have to be a complete background image that does not include the image of the subject in the foreground (hereinafter referred to as the foreground image) at all, but is a time close to the time when the image of the target for extracting the foreground region was captured. It is desirable that the image is taken in. It is assumed that the plurality of background image data 401 to be acquired include an image at the same viewpoint as the viewpoint 402 that captured the image of the target for which the foreground region is to be extracted. Hereinafter, the target image from which the foreground region is extracted is referred to as a target image (data), and the viewpoint obtained by capturing the target image (data) is referred to as a viewpoint of interest.

Next, the background image data 401 at the viewpoint of interest is created by converting the acquired background image data 401 into an image when viewed from the viewpoint 402 of interest with respect to the ground surface as a reference. The number of background image data 403 created here is the same as the number of background image data 401. Hereinafter, the background image data 403 obtained by converting the background image data 401 will be referred to as a converted background image data 403.

Here, the subject in the foreground means a subject existing in a position close to the image pickup apparatus among the subjects included in the captured image. For example, when the target image data is data obtained by capturing a competition scene such as sports, a person such as a player or a referee or an instrument such as a goal or a ball is a subject in the foreground, and the subject in the foreground is in chronological order. This includes those that generally continue to move in a plurality of images taken continuously. On the other hand, the background subject means a subject behind the subject in the foreground because it exists at a position far from the imaging device among the subjects included in the captured image. For example, when the target image data is data obtained by capturing a competition scene such as sports, the background subject is a ground composed of turf or soil, the floor of a gymnasium, etc., and the background subject is continuous in chronological order. Most of the multiple images taken in 1 are almost stopped.

Such a foreground subject has a height above the ground surface, while a background subject does not have a height above the ground surface. Therefore, using the plurality of converted background image data 403, an image (foreground image) of a subject having a height from the ground surface, that is, a subject in the foreground is detected, and the detected foreground image is removed from the incomplete background image. By doing so, a complete background image at the viewpoint 402 of interest is created. Specifically, for a plurality of converted background image data 403 including the image at the viewpoint 402 of interest, the degree of matching between the pixels of interest is calculated for each pixel, and the pixels with a low degree of matching are the pixels of the image region of the subject in the foreground. Detect as. As described above, the converted background image data 403 is an image obtained by converting the background image data 401 into an image when viewed from the viewpoint 402 of interest with the ground surface as a reference plane. Therefore, the coordinates of the pixels of the regions 405 to 407 in the background image data 401 corresponding to the subject 404 existing on the ground surface and having no height are commonly present at the same position in all the converted background image data 403. It is converted to the coordinates of the pixels of the area 408. On the other hand, the coordinates of the pixels of the regions 410 to 412 in the background image data 401 corresponding to the subject 409 having the height are converted into the coordinates of the pixels of the regions 413 to 415 whose positions differ depending on the viewpoint. Therefore, in the plurality of converted background image data 403, pixels having a high degree of matching between the pixels of interest are regarded as pixels in the image region of the subject of the background having no height, and pixels having a low degree of matching are defined as heights. It is regarded as a pixel in the image area of the subject in the foreground. This creates a complete background image. Finally, the foreground region is extracted by comparing the complete background image at the created viewpoint 402 of interest with the target image data.

The above is the outline of the processing performed in this embodiment. The target image data to be used is not limited to the above example, and various image data such as data captured by a surveillance camera can be used. Further, although the case where the background image data 401 includes the image at the viewpoint of interest is described here, the present embodiment can be applied even when the background image data does not include the image at the viewpoint of interest. , The specific processing method will be described later.

<About the hardware configuration of the image processing device>
Hereinafter, the hardware configuration of the image processing apparatus of this embodiment will be described. FIG. 1 is a block diagram showing an example of the hardware configuration of the image processing device of this embodiment. The image processing device 100 of this embodiment includes a CPU 101, a RAM 102, a ROM 103, a secondary storage device 104, an input interface 105, and an output interface 106, and these components are connected to each other by a system bus 107. Further, the image processing device 100 is connected to the external storage device 108 via the input interface 105, and is connected to the external storage device 108 and the display device 109 via the output interface 106.

The CPU 101 uses the RAM 102 as a work memory to execute a program stored in the ROM 103, and comprehensively controls each component of the image processing device 100 via the system bus 107. As a result, various processes described later are executed.

The secondary storage device 104 is a storage device that stores various data handled by the image processing device 100, and an HDD is used in this embodiment. The CPU 101 can write data to the secondary storage device 104 and read data stored in the secondary storage device 104 via the system bus 107. In addition to the HDD, various storage devices such as an optical disk drive and a flash memory can be used as the secondary storage device 104.

The input interface 105 is, for example, a serial bus interface such as USB or IEEE 1394, and data, commands, and the like are input from an external device to the image processing device 100 via the input interface 105. The image processing device 100 acquires data from an external storage device 108 (for example, a storage medium such as a hard disk, a memory card, a CF card, an SD card, or a USB memory) via an input interface 105. An input device (not shown) for the user to input, such as a mouse or a keyboard, can also be connected to the input interface 105. The output interface 106 includes, for example, a video output terminal such as DVI or HDMI (registered trademark), in addition to a serial bus interface such as USB or IEEE1394 similar to the input interface 105. Data is output from the image processing device 100 to the external device via the output interface 106. The image processing device 100 displays an image by outputting the processed image or the like to the display device 109 (various image display devices such as a liquid crystal display) via the output interface 106. Although there are components of the image processing apparatus 100 other than those described above, the description thereof will be omitted because they are not the main focus of the present invention.

<About the process of extracting the foreground area>
Hereinafter, the process of extracting the foreground region executed by the image processing apparatus 100 in this embodiment will be described with reference to FIGS. 2 and 3. FIG. 2 is a block diagram showing a functional configuration of the image processing device 100, and FIG. 3 is a flowchart showing a flow of processing for extracting a foreground region. The CPU 101 of the image processing device 100 functions as each component shown in FIG. 2 by executing a program stored in the ROM 103 using the RAM 102 as a work memory, and executes a series of processes shown in FIG. It should be noted that it is not necessary that all of the processes shown below are executed by the CPU 101, and even if the image processing device 100 is configured so that a part or all of the processes are performed by one or a plurality of processing circuits other than the CPU 101. good.

Hereinafter, the flow of processing performed by each component will be described. In step S301, the target image data acquisition unit 201 acquires the target image data from the external storage device 108 or from the secondary storage device 104 via the input interface 105. As described above, the target image data is an image to be extracted from the foreground region. Further, the target image data acquisition unit 201 defines the viewpoint of the camera that has captured the target image data as the viewpoint of interest. Although the case where the target image data is one image is described here, the present embodiment can be applied to the case where the target image data is a plurality of images. Further, the target image data acquisition unit 201 acquires the parameters of the camera that captured the target image data (hereinafter, camera parameters) together with the target image data. Here, the camera parameter is a parameter that enables calculation to project a point in three-dimensional space onto an image captured by the camera, and is an external parameter representing the position and orientation of the camera, a focal length, and an optical center. Includes internal parameters that represent. Measured values and design values stored in advance in the memory may be used as camera parameters. The target image data acquisition unit 201 outputs the target image data to the foreground extraction unit 207 and the camera parameters to the image conversion unit 203.

In step S302, the background image data acquisition unit 202 acquires a plurality of background image data from a plurality of different viewpoints from the external storage device 108 or from the secondary storage device 104 via the input interface 105. Here, the background image data is an image of a background subject in an environment (weather, time zone, etc.) that is substantially the same as the environment in which the target image data was captured. As described above, the background image data acquired in this step does not have to be a background image (complete background image) that does not include a foreground image at all.

In this embodiment, a plurality of images corresponding to a plurality of different times acquired by continuously imaging a scene from the same viewpoint along a time series are filtered by using an intermediate value filter. , Create background image data for each viewpoint. However, the method of creating background image data is not limited to this method. For example, the background image data may be created by using another filter such as an average value filter, or the background image data may be created by performing clustering processing on a plurality of images. Further, the background image data acquired by taking an image in advance in a state where the subject in the foreground does not exist may be used for each viewpoint.

Further, the background image data acquisition unit 202 acquires the camera parameters corresponding to each background image data together with the background image data. Further, the background image data acquisition unit 202 stores each background image data in association with a number for distinguishing the viewpoint of the camera (hereinafter, a viewpoint number of the camera) in order to distinguish each of the plurality of background image data. The background image data acquisition unit 202 outputs the background image data and the camera parameters to the image conversion unit 203, and outputs only the background image data to the correction unit 206.

In step S303, the image conversion unit 203 uses the camera parameters acquired from the target image data acquisition unit 201 and the background image data acquisition unit 202 to obtain the background image data acquired from the background image data acquisition unit 202 from the viewpoint of interest. Converts to an image when viewed. Specifically, each background image data is projected and transformed with the ground surface as a reference to obtain an image when viewed from the viewpoint of interest. The background image (data) obtained by the image conversion in this step is called a converted background image (data). In this way, the image conversion unit 203 functions as a conversion background image data creation means. Here, the image conversion method in this step will be described with reference to FIG.

As shown in FIG. 5, when a certain point 501 in the three-dimensional space is projected on the image of the camera 502, the point 504 formed by the intersection of the straight line connecting the point 501 and the camera 502 and the image surface 503 is formed. It is a projected image of the point 501 in the three-dimensional space on the image plane 503. Similarly, in the camera (camera of another viewpoint) 505 existing at a position different from the camera 502, the point 507 formed by the intersection of the straight line connecting the point 501 and the camera 505 and the image surface 506 is the image surface of the point 501. It becomes a projection image to 506. Here, a case where all the points in the three-dimensional space projected on the image surface 503 and the image surface 506 including the point 501 exist on the same plane which is the ground surface will be examined. _{In this case, using the 3 × 3 homography matrix H 01} calculated by the camera parameters of the camera 502 and the camera 505, _{the coordinates (u 0} , v) of any pixel on the image plane 503 are calculated by the equation (1). ₀ ) is converted to _{the coordinates (u 1} , v ₁ ) on the image plane 506.

In step S303, projection conversion is performed in which the camera of the viewpoint corresponding to the background image data acquired from the background image data acquisition unit 202 is the camera 502 described above, and the camera of the viewpoint of interest defined by the target image data acquisition unit 201 is the camera 505. , Execute for each background image data. Therefore, the number of converted background image data acquired in this step is the same as the number of background image data acquired by the background image data acquisition unit 202. Further, the converted background image data is stored in association with the viewpoint number of each background image data acquired by the background image data acquisition unit 202. The image conversion unit 203 outputs the conversion background image data to the matching degree calculation unit 204 and the correction unit 206.

In step S304, the image conversion unit 203 uses the image corresponding to the viewpoint closest to the camera position (viewpoint of interest) in which the target image data is captured from the background image data acquired from the background image data acquisition unit 202 as a reference. It is defined as a background image (hereinafter referred to as a reference background image). Specifically, the distance between the coordinates of the viewpoint of interest (Xo, Yo, Zo) and the coordinates of the viewpoint corresponding to the background image data acquired from the background image data acquisition unit 202 (Xi, Yi, Zi) is determined for each viewpoint. calculate. Here, i represents a viewpoint number, and 1 ≦ i <number of viewpoints + 1. Then, the viewpoint (reference viewpoint) that minimizes the calculated distance is detected, and the background image (data) corresponding to the reference viewpoint is used as the reference background image (data). The image conversion unit 203 outputs the viewpoint number corresponding to the reference background image to the matching degree calculation unit 204 and the correction unit 206. In this embodiment, the viewpoint number corresponding to the reference background image is referred to as a reference viewpoint number.

In step S305, the matching degree calculation unit 204 determines the pixel of interest in the converted background image data, which is the target for determining whether the pixels match in the plurality of converted background image data. In this embodiment, first, the upper left pixel of the converted background image data is selected as the pixel of interest, and then the unprocessed pixel is sequentially selected as the pixel of interest. For all the pixels of the converted background image data, the pixels of interest may be determined in any order as long as the determination of whether the pixels match in the plurality of converted background image data is executed.

In step S306, the matching degree calculation unit 204 uses the plurality of conversion background image data acquired from the image conversion unit 203 to obtain between the conversion background image data corresponding to the reference viewpoint number and the other conversion background image data. The degree of coincidence in the pixel of interest is calculated. Hereinafter, the method for calculating the degree of agreement will be specifically described.

First, the matching degree calculation unit 204 acquires _{the pixel value B j} (u ₂ , v ₂ ) of the converted background image data at the _{determined coordinates (u 2} , v _{2) of the pixel of interest.} Here, j represents a subscript that distinguishes each of the plurality of converted background image data, and the matching degree calculation unit 204 acquires pixel values corresponding to the number of the converted background image data. Next, the matching degree calculation unit 204 calculates an intermediate value of all the acquired pixel values. This intermediate value is used as a reference value M when calculating the degree of agreement. The reference value is not limited to this, and any value that reflects the statistical properties of a plurality of pixel values, such as an average value, may be used as the reference value.

Next, the matching degree calculation unit 204 calculates the matching degree in the pixel of interest as the pixel value B ₀ (u ₂ , v ₂ ) of the pixel of interest in the converted background image data corresponding to the reference viewpoint number, and the reference value M (u). Calculated by Eq. (2) using ₂ , v _2).

Here, k represents a subscript for identifying an RGB3 channel. The degree of coincidence D calculated by the equation (2) becomes smaller as the variation in the pixel values in the plurality of converted background image data is smaller. The degree of matching used is not limited to this, and any value indicating the difference between pixels may be used. _{For example, the sum of the differences between the pixel values B 0} (u ₂ , v ₂ ) of the pixel of interest in the converted background image data corresponding to the reference viewpoint number and the pixel values of the pixels of interest in the other converted background image data is the degree of coincidence. It may be used as.

In step S307, the matching degree calculation unit 204 determines whether the processing of steps S305 to S306 has been performed on all the pixels of the converted background image data. If the result of the determination in step S307 is true, the matching degree calculation unit 204 outputs the calculated matching degree of all pixels to the correction determination unit 205 and the calculated reference value to the correction unit 206, and proceeds to step S308. On the other hand, if the result of the determination in step S307 is false, the process returns to step S305.

In step S308, the correction determination unit 205 initializes the flag map, that is, sets the pixel values of all the pixels of the flag map to 0. The flag map initialized in this step is used to determine the pixel to be corrected when correcting the pixel of the converted background image data corresponding to the reference viewpoint number in step S311. In this flag map, 1 is assigned to the pixel value corresponding to the pixel to be corrected, and 0 is assigned to the pixel value corresponding to the pixel not to be corrected. By the initialization in this step, all the pixels of the converted background image data corresponding to the reference viewpoint number are not subject to the correction processing.

In step S309, the correction determination unit 205 updates the flag map based on the degree of agreement acquired from the degree of agreement calculation unit 204. Specifically, the correction determination unit 205 is a pixel value of a flag map corresponding to a pixel in the converted background image data corresponding to the reference viewpoint number, which is considered to be a pixel in the image region of the subject in the foreground. Is changed to 1. In this embodiment, if the calculated degree of coincidence D is equal to or greater than a predetermined threshold value, the degree of matching between the pixels of the converted background image data corresponding to the reference viewpoint number and the pixels of the other converted background image data is low. , It is determined that the pixel of interest is likely to be a pixel in the image region of the subject in the foreground. On the other hand, if the degree of coincidence D is less than the threshold value, the degree of matching between the pixels of the converted background image data corresponding to the reference viewpoint number and the pixels of the other converted background image data is high, so that the pixel of interest is the image of the subject in the background. It is highly likely that it is a pixel in the area. The threshold value used in this step is determined based on the maximum value of the pixel value, etc., and a value smaller than 20% of the maximum value, for example, an arbitrary value within the range of 1% to 5% of the maximum value is used. Determine the threshold. That is, assuming that an arbitrary value is a, the threshold value is a × a × 3 because the sum of squared differences is used as the degree of coincidence in the equation (2). If the sum of the differences is used as the degree of coincidence, the threshold value is a × 3. Further, it is determined for each pixel whether or not it is a pixel in the image area of the subject in the foreground. The correction determination unit 205 outputs the updated flag map to the correction unit 206.

In step S310, the correction unit 206 determines the pixel of interest in the converted background image data corresponding to the reference viewpoint number. In this embodiment, first, the upper left pixel of the converted background image data corresponding to the reference viewpoint number is selected as the pixel of interest, and then the unprocessed pixel is sequentially selected as the pixel of interest. If the pixel values are updated (step S311) based on the flag map for all the pixels of the converted background image data corresponding to the reference viewpoint number, the pixels of interest may be determined in any order.

In step S311, the correction unit 206 corrects the pixel of the pixel of interest in the converted background image corresponding to the reference viewpoint number based on the flag map acquired from the correction determination unit 205. In this embodiment, when the pixel value of the flag map corresponding to the pixel of interest in the converted background image corresponding to the reference viewpoint number is 1, the pixel value of the pixel of interest is the reference value calculated by the matching degree calculation unit 204. replace. On the other hand, when the pixel value of the flag map corresponding to the pixel of interest in the converted background image corresponding to the reference viewpoint number is 0, the pixel value of the pixel of interest is not changed. The method for correcting the pixel value is not limited to this, and another method such as replacing with the pixel value of the background image corresponding to the viewpoint adjacent to the reference viewpoint may be used.

In step S312, the correction unit 206 determines whether the processing of steps S310 to S311 has been performed on all the pixels of the converted background image data corresponding to the reference viewpoint number. If the result of the determination in step S312 is true, the correction unit 206 outputs the converted background image data corresponding to the reference viewpoint number for which the correction has been completed to the foreground extraction unit 207, and proceeds to step S313 while proceeding to the determination. If the result is false, the process returns to step S310.

In step S313, the foreground extraction unit 207 uses the converted background image data (referred to as a complete background image I _b ) corresponding to the corrected reference viewpoint number acquired from the correction unit 206 to be the target image data (I and). Extract the area of the subject in the foreground from). Specifically, as shown in the equation (3), the _{difference squared sum is calculated for each pixel between the complete background image I b} and the target image data I, and the pixel in which the difference squared sum is equal to or greater than the threshold value is in the foreground. By regarding it as a pixel in the image region of the subject, an image _If is created by extracting the region of the subject in the foreground. The image _If is a binary image, 1 is assigned to the pixel value corresponding to the pixel in the image area of the subject in the foreground, and 0 is assigned to the pixel value corresponding to the pixel in the image area of the subject in the background.

Here, Th represents a threshold value, and k represents a subscript for identifying an RGB3 channel. The threshold value used here is determined based on the maximum value of the pixel value or the like, and a value smaller than 20% of the maximum value of the pixel value, for example, an arbitrary value within the range of 1% to 5% of the maximum value can be used. It may be used to obtain the threshold value. The method of obtaining this threshold value is the same as in the case of the equation (2). In this way, the foreground extraction unit 207 functions as a foreground image data creation means. The foreground extraction unit 207 outputs the created image _If to the secondary storage device 104, the external storage device 108, and the display device 109, and completes a series of processes. The above is the process of extracting the foreground region, which is executed by the image processing apparatus 100 in this embodiment.

<About the effect of this example>
Hereinafter, the effects of this embodiment will be described with reference to FIG. In FIG. 6, the image data 601 is background image data at the viewpoint 602 created based on a plurality of images continuously captured in chronological order according to a conventional method. The background image data 601 shows a foreground subject such as a foreground subject 603 (goalkeeper) and a foreground subject (goal) 604. The reason for this is that when the continuous images for creating the background image data are captured, the subjects 603 and 604 in the foreground exist at the same position and do not move. As a result, the subject 603 in the foreground is created when the background image data is created. , 604 was mistakenly regarded as a background subject. When the foreground region is extracted from the target image data 605 using the background image data 601, the foreground image data 606 is acquired. In the foreground image data 606, a region other than the subjects 603 and 604, which is a substantially moving foreground subject, can be extracted. However, the area of the stopped foreground subjects 603 and 604 cannot be extracted.

Further, the image data 607 is background image data at the viewpoint 602 created based on a plurality of images captured from a plurality of different viewpoints at the same time as the time when the target image data 605 was imaged according to the conventional method. Although the background image data 607 does not show the foreground subject such as the foreground subject 603 (goalkeeper) or the foreground subject (goal) 604, a part of the background subject is missing. The reason for this is that the subjects in the foreground are densely packed in the scene captured to create the background image data, and a part of the subjects in the foreground cannot be seen from a plurality of viewpoints. When the foreground region is extracted from the target image data 605 using the background image data 607, the foreground image data 608 is acquired. In the foreground image data 608, the area of the subject in the foreground having a height from the ground surface can be roughly extracted. However, because the subjects in the foreground are densely packed, the region 609 due to the subjects in the foreground that cannot be seen from a plurality of viewpoints cannot be extracted.

On the other hand, in this embodiment, background image data 610, which is a complete background image, is created by using incomplete background images (for example, background image data 601) from a plurality of different viewpoints. When the foreground region is extracted from the target image data 605 using the background image data 610, the foreground image data 611 is acquired. In the foreground image data 611, the area of the foreground subject 603 and 604 that is stopped and the area of the foreground subject that cannot be seen from a plurality of viewpoints can be extracted with high accuracy. As described above, according to the present embodiment, the area due to the subject in the foreground is extracted with high accuracy regardless of the state of the subject such as the presence or absence of the change (movement, etc.) of the subject with the passage of time and the density of the subject in the foreground. can do.

[Example 2]
In the first embodiment, when a complete background image is created based on a plurality of incomplete background images, a degree of coincidence indicating the degree of matching of the converted background images, which differs depending on the viewpoint, at the pixel of interest is used. On the other hand, in this embodiment, when creating a complete background image based on a plurality of incomplete background images, in addition to the degree of matching, the degree of smoothness of change in the pixel value of the converted background image that differs depending on the viewpoint in the pixel of interest. , So-called continuity is used. The same configuration and the same processing as in the first embodiment are designated by the same reference numerals as those in the first embodiment, and the description thereof will be omitted.

<Overview of the process of extracting the foreground area>
Hereinafter, the outline of the process for extracting the foreground region in this embodiment will be described. In this embodiment, the continuity of pixel values between viewpoints is calculated using the converted background image data obtained by converting the background image data from a plurality of different viewpoints into an image when viewed from the viewpoint of interest. The continuity of pixel values is the degree of smoothness of change in pixel values between the converted background image data at the viewpoint of interest and the converted background image data at a viewpoint adjacent to the viewpoint of interest.

Specifically, the pixel value of the pixel of interest in the converted background image data corresponding to the reference viewpoint number is compared with the pixel value of the pixel of interest in the converted background image data in the viewpoint adjacent to the reference viewpoint, and the difference between the pixel values is compared. Is calculated as continuity. Subsequently, using the degree of agreement described in Example 1 and the continuity calculated in this example, pixels having a low degree of agreement and a non-smooth change in pixel value are obtained in the image region of the subject in the foreground. It is considered that there is a high possibility that it is a pixel of, and it is detected as a pixel to be corrected. Then, the pixel value of the detected pixel to be corrected is updated and the converted background image data is corrected to create a complete background image. Finally, the foreground area is extracted by comparing the created complete background image with the target image data.

In the first embodiment, it was determined whether or not the pixel of interest is a pixel in the image region of the subject in the foreground by using only the degree of coincidence of the pixel values calculated based on the background image data in all viewpoints. Therefore, it is highly likely that the pixels in the image area of the subject in the background, which have different pixel values due to the change in the appearance of colors depending on the viewpoint, are also the pixels in the image area of the subject in the foreground, and are used as the pixels to be corrected. Detected. As a result, pixels that do not need to be corrected are also corrected, so that an error occurs in the converted background image data after correction, and it is not possible to accurately create a complete background image that does not include the image of the subject in the foreground. As a background subject whose appearance changes depending on the viewpoint, there is a turf that is mowed in a direction, which is present in an image of a competition scene such as sports. The turf that is mowed with directionality has a different appearance of turf color depending on the viewing direction, and as a result, the pixel value changes depending on the viewpoint even if the turf is at the same position. When Example 1 is applied to such a scene in which the turf is the background subject, the degree of matching between the pixels in the plurality of converted background image data is low, so that the pixels in the turf image area which is the background subject are It is erroneously determined to be a pixel in the image area of the subject in the foreground. In order to prevent such erroneous determination, in this embodiment, it is determined whether or not the pixel of interest is a pixel in the image region of the subject in the foreground by using continuity in addition to the degree of coincidence. In general, for a subject whose color appearance changes depending on the viewpoint, there may be a noticeable difference in the color appearance between distant viewpoints, but the change in color appearance between adjacent viewpoints is It is gradual. Therefore, in this embodiment, the pixels in the image area of the background subject whose pixel values have changed due to the difference in color appearance and the image area of the foreground subject whose pixel values have changed due to the height from the ground surface. Distinguish from the pixels of. As a result, the converted background image can be corrected with high accuracy to create a complete background image, so that the region of the subject in the foreground can be extracted with high accuracy from the target image data. It should be noted that the subject whose color appearance changes depending on the viewpoint is not limited to the above-mentioned example of the lawn, and there are various subjects such as the floor of the gymnasium.

<About the process of extracting the foreground area>
Hereinafter, the process of extracting the foreground region executed by the image processing apparatus 100 in this embodiment will be described with reference to FIGS. 7 and 8. FIG. 7 is a block diagram showing a functional configuration of the image processing apparatus 100 in this embodiment, and FIG. 8 is a flowchart showing a flow of processing for extracting a foreground region in this embodiment. The CPU 101 of the image processing device 100 functions as each component shown in FIG. 7 by executing a program stored in the ROM 103 using the RAM 102 as a work memory, and executes a series of processes shown in FIG. It should be noted that it is not necessary that all of the processes shown below are executed by the CPU 101, and even if the image processing device 100 is configured so that a part or all of the processes are performed by one or a plurality of processing circuits other than the CPU 101. good.

In step S801, the continuity calculation unit 701 determines the pixel of interest in the converted background image data to be calculated for continuity. In this embodiment, first, the upper left pixel of the converted background image is selected as the pixel of interest, and then the unprocessed pixel is sequentially selected as the pixel of interest. As long as the continuity is calculated for all the pixels of the converted background image data, the pixels of interest may be determined in any order.

In step S802, the continuity calculation unit 701 uses the conversion background image data acquired from the image conversion unit 203 to maintain the continuity of the pixel values of the pixels of interest in the conversion background image corresponding to the reference viewpoint and the viewpoints around the reference viewpoint. Is calculated. Here, the method of calculating the continuity in this step will be described with reference to FIG.

First, the cameras 902 and 903 adjacent to the cameras 901 corresponding to the reference viewpoint numbers determined by the image conversion unit 203 are detected, and the viewpoint numbers corresponding to these cameras are acquired. Hereinafter, the acquired viewpoint number will be referred to as an adjacent viewpoint number. Here, the camera adjacent to the camera 901 corresponding to the reference viewpoint number is determined based on the distance to the camera 901 calculated from the coordinates in the three-dimensional space of the camera. In this embodiment, the camera 902 having the shortest distance to the camera 901 among the cameras on the left side of the camera 901 and the camera 903 having the shortest distance to the camera 901 among the cameras on the right side of the camera 901. Is detected as a camera adjacent to the camera 901.

Next, from the converted background image 904 corresponding to the reference viewpoint number and the converted background images 905 and 906 corresponding to the adjacent viewpoint number, the pixel values of the pixels 907, 908, _{and 909 of the coordinates (u 2} , v _{2) of the pixel of interest.} Is acquired, and the continuity is calculated by the equation (4) using the acquired pixel values.

Here, B ₉₀₁ (u ₂ , v ₂ ), B ₉₀₂ (u ₂ , v ₂ ), and B ₉₀₃ (u ₂ , v ₂ ) are converted background images 904 and 905 corresponding to the cameras 901, 902, and 903, respectively. , 906 represents the pixel values of the pixels of interest 907, 908, 909. Further, k represents a subscript for identifying the RGB3 channel. The value of C calculated by the equation (4) becomes smaller as the change of the pixel value between the viewpoints becomes smoother. The continuity used is not limited to C calculated by the equation (4), and any value indicating the continuity of the pixel values between viewpoints, such as the second derivative from the discrete value, may be used. Further, in this embodiment, the case where the cameras 901 corresponding to the reference viewpoint number and the adjacent cameras 902 and 903 are used is described, but the cameras used are not limited to these, and other cameras may be used depending on how the subject is viewed. You may use it. For example, on the left side of the camera 901 corresponding to the reference viewpoint number, instead of the camera 902, a camera having a short distance to the camera 901 next to the camera 902 may be used. The same applies to the camera used on the right side of the camera 901.

In step S803, the continuity calculation unit 701 determines whether the processes of steps S801 to S802 have been performed on all the pixels of the converted background image data. If the result of the determination in step S803 is true, the continuity calculation unit 701 outputs the calculated continuity of all pixels to the correction determination unit 702 and proceeds to step S308, while if the result of the determination is false, the step Return to S801.

In step S804, the correction determination unit 702 updates the flag map based on the degree of agreement acquired from the degree of agreement calculation unit 204 and the continuity acquired from the continuity calculation unit 701. Specifically, the correction determination unit 702 is a pixel of the flag map corresponding to a pixel in the converted background image data corresponding to the reference viewpoint number, which is considered to be a pixel in the image region of the subject in the foreground. Change the value to 1. In this embodiment, if the calculated degree of coincidence D is equal to or higher than the predetermined threshold value and the calculated continuity C is equal to or higher than the predetermined threshold value, the conversion background image corresponding to the reference viewpoint number and other conversion backgrounds are used. It is assumed that the degree of matching with the image and the degree of smoothness of change in the pixel of interest are low. That is, it is determined that there is a high possibility that the pixel of interest is a pixel in the image region of the subject in the foreground. On the other hand, if these conditions are not satisfied, it is determined that the pixel of interest is likely to be a pixel in the image region of the subject in the background. The threshold value used in this step is determined based on the maximum value of the pixel value, etc., and a value smaller than 20% of the maximum value, for example, an arbitrary value within the range of 1% to 5% of the maximum value is used. You may find the threshold. The method of obtaining this threshold value is the same as that in the first embodiment. Further, it is determined for each pixel whether or not it is a pixel in the image area of the subject in the foreground. The correction determination unit 702 outputs the updated flag map to the correction unit 206. The above is the process of extracting the foreground region, which is executed by the image processing apparatus 100 in this embodiment.

<About the effect of this example>
Hereinafter, the effects of this embodiment will be described with reference to FIG. The image data 1002 is converted background image data acquired by converting the background image data into an image when viewed from the viewpoint 1001 with the ground surface as a reference for each viewpoint. Here, the viewpoint 1001 is assumed to be a viewpoint of interest and a reference viewpoint. Further, the subject 1003 is a background subject (for example, a lawn) whose color appearance changes depending on the viewpoint, and the subject 1005 is a foreground subject.

When Example 1 is applied to the scene shown in FIG. 10 and a complete background image is created based on an incomplete background image from multiple viewpoints, background image data 1004 is acquired. In the background image data 1004, although the image of the subject 1005 in the foreground can be removed, the subject 1003 in the background is not correctly captured. The reason for this is that when creating a complete background image, it is determined whether the pixel of interest is a pixel in the image area of the subject in the foreground based only on the degree of coincidence, so that the pixel in the image area of the subject 1003 in the background is the foreground. This is because it is determined to be a pixel in the image area of the subject. As a result, the background image data 1004 in which the pixels in the image region of the background subject 1003 are corrected is created. Even if an attempt is made to extract the foreground region from the target image data using the background image data 1004, the foreground region cannot be extracted with high accuracy.

On the other hand, in this embodiment, when creating a complete background image based on an incomplete background image from multiple viewpoints, whether the pixel of interest is a pixel in the image region of the subject in the foreground is determined by the degree of coincidence and continuity. Judgment is based on. As a result, the pixels in the image area of the background subject 1003 are not determined to be the pixels in the image area of the subject in the foreground, and the background image data 1006 in which the pixels in the image area of the background subject 1003 are not corrected is created. In the background image data 1006, the background subject 1003 is correctly captured while removing the image of the foreground subject 1005. By extracting the foreground region from the target image data using the background image data 1006, the foreground region can be extracted with high accuracy. As described above, according to the present embodiment, even when the background subject is a subject whose color appearance changes depending on the viewpoint, the region of the foreground subject can be extracted with high accuracy.

[Example 3]
In the first and second embodiments, a complete background image is created based on incomplete background images from a plurality of different viewpoints, and the created complete background image is compared with the target image data to obtain a subject in the foreground. Extract the area. On the other hand, in this embodiment, the foreground region is extracted so as not to include the region due to the shadow by using the incomplete foreground image from a plurality of different viewpoints. Here, the incomplete foreground image means an image in which the region of the subject in the foreground and the region of the shadow accompanying the subject in the foreground are extracted as the foreground region.

In this embodiment, by converting an incomplete foreground image for each viewpoint into an image when viewed from the viewpoint of interest with the ground surface as a reference, a plurality of converted foreground image data are acquired, and a plurality of acquired foreground image data are acquired. The degree of coincidence between pixels is calculated in the converted foreground image data. As described in the first embodiment, the subject in the foreground has a height from the ground surface, but the shadow accompanying the subject in the foreground does not have a height from the ground surface. Therefore, in this embodiment, pixels having a high degree of matching between pixels are detected in a plurality of converted foreground image data, and the detected pixels are corrected as being likely to be pixels in a shadow region having no height. .. As a result, it is possible to create a foreground image in which only the area due to the subject in the foreground having a height is extracted as the foreground area without extracting the area due to the shadow. Hereinafter, an image in which only the area due to the subject in the foreground having a height without extracting the area due to the shadow is extracted as the foreground area is referred to as a complete foreground image. The same configuration and the same processing as those in the above-described embodiment are designated by the same reference numerals as those in the above-mentioned embodiment, and the description thereof will be omitted.

<About the process of extracting the foreground area>
Hereinafter, the process of extracting the foreground region executed by the image processing apparatus 100 in this embodiment will be described with reference to FIGS. 11 and 12. FIG. 11 is a block diagram showing a functional configuration of the image processing apparatus 100 in this embodiment, and FIG. 12 is a flowchart showing a flow of processing for extracting a foreground region in this embodiment. The CPU 101 of the image processing device 100 functions as each component shown in FIG. 11 by executing a program stored in the ROM 103 using the RAM 102 as a work memory, and executes a series of processes shown in FIG. It should be noted that it is not necessary that all of the processes shown below are executed by the CPU 101, and even if the image processing device 100 is configured so that a part or all of the processes are performed by one or a plurality of processing circuits other than the CPU 101. good.

In step S1201, the camera parameter acquisition unit 1101 acquires the camera parameters of the camera that captured the target image data from the external storage device 108 or the secondary storage device 104 via the input interface 105. Further, the camera parameter acquisition unit 1101 defines the viewpoint of the camera that has captured the target image data as the viewpoint of interest. The camera parameters acquired in this step are the same as the camera parameters described in the first embodiment. The camera parameter acquisition unit 1101 outputs the camera parameters to the image conversion unit 1103.

In step S1202, the foreground image data acquisition unit 1102 acquires a plurality of foreground image data at a plurality of different viewpoints from the external storage device 108 or from the secondary storage device 104 via the input interface 105. The foreground image data acquired in this step is an image obtained by extracting a region due to the subject in the foreground, and it is assumed that the extracted region includes a region due to a shadow. In this embodiment, the foreground image data is created based on the captured image and the background image captured in advance. Hereinafter, the method for creating the foreground image data will be specifically described. The captured image used here is an image obtained by capturing a foreground subject and a background subject in the target image data in an environment substantially the same as the environment in which the target image data was captured. The background image is an image obtained by capturing a background subject in the target image data in an environment substantially the same as the environment in which the target image data was captured. In this embodiment, the pixel value of the captured image and the pixel value of the background image are compared for each pixel for each viewpoint, the pixel value of the pixel having the same pixel value is 0, and the pixel value of the other pixel is 0. By setting to 1, a binary image for each viewpoint is created. This binary image is the foreground image data. The method for creating the foreground image data is not limited to this, and the foreground image data to be created is not limited to the binary image and may be a multivalued image. Further, the foreground image data acquisition unit 1102 acquires the camera parameters corresponding to each foreground image data together with the foreground image data. Further, the foreground image data acquisition unit 1102 stores each foreground image data in association with the viewpoint number of the camera in order to distinguish each of the plurality of foreground image data. The foreground image data acquisition unit 1102 outputs the foreground image data and camera parameters to the image conversion unit 1103.

In step S1203, the image conversion unit 1103 used the camera parameters obtained from the camera parameter acquisition unit 1101 and the foreground image data acquisition unit 1102 to view the foreground image data obtained from the foreground image data acquisition unit 1102 from the viewpoint of interest. Convert to a case image. The conversion in this step is the same conversion as in step S303 of the first embodiment, and the foreground image data is projected and transformed with respect to the ground surface for each viewpoint to obtain an image when viewed from the viewpoint of interest. The foreground image (data) obtained by the image conversion in this step is called a converted foreground image (data). In this way, the image conversion unit 1103 functions as a conversion foreground image data creation means. The image conversion unit 1103 outputs the conversion foreground image data to the matching degree calculation unit 1104.

In step S1204, the image conversion unit 1103 uses the image corresponding to the viewpoint closest to the camera position (viewpoint of interest) that captured the target image data as the reference foreground from the foreground image data acquired from the foreground image data acquisition unit 1102. It is defined as an image (hereinafter referred to as a reference foreground image). Specifically, the distance between the coordinates of the viewpoint of interest and the coordinates of the viewpoint corresponding to the foreground image data is calculated for each viewpoint. Then, the foreground image (data) corresponding to the viewpoint (reference viewpoint) that minimizes the calculated distance is used as the reference foreground image (data). The image conversion unit 1103 outputs the viewpoint number corresponding to the reference foreground image to the correction unit 1105. In this embodiment, the viewpoint number corresponding to the reference foreground image is referred to as a reference viewpoint number.

In step S1205, the matching degree calculation unit 1104 determines the pixel of interest in the converted foreground image data, which is the target for determining whether the pixels match in the plurality of converted foreground image data. In this embodiment, first, the upper left pixel of the converted foreground image data is selected as the pixel of interest, and then the unprocessed pixel is sequentially selected as the pixel of interest. For all the pixels of the converted foreground image data, the pixels of interest may be determined in any order as long as the determination of whether the pixels match in the plurality of converted foreground image data is executed.

In step S1206, the matching degree calculation unit 1104 uses the plurality of converted foreground image data acquired from the image conversion unit 1103 to obtain between the converted foreground image data corresponding to the reference viewpoint number and the other converted foreground image data. The degree of coincidence in the pixel of interest is calculated. Hereinafter, the method for calculating the degree of agreement will be specifically described.

First, the matching degree calculation unit 1104 acquires _{the pixel value F l} (u ₂ , v ₂ ) of the converted foreground image data at the _{determined coordinates (u 2} , v _{2) of the pixel of interest.} Here, l represents a subscript that distinguishes each of the plurality of converted foreground image data, and the matching degree calculation unit 1104 acquires pixel values corresponding to the number of converted foreground image data. Next, the matching degree calculation unit 1104 calculates the average value of all the acquired pixel values. In this embodiment, this average value is used as the degree of agreement. The degree of coincidence is not limited to this, and a value that reflects the statistical properties of a plurality of pixel values may be used as the degree of coincidence.

In step S1207, the matching degree calculation unit 1104 determines whether the processing of steps 1205 to S1206 has been performed on all the pixels of the converted foreground image data. If the result of the determination in step S1207 is true, the matching degree calculation unit 1104 outputs the calculated matching degree of all the pixels to the correction unit 1105, and proceeds to step S1208. On the other hand, if the result of the determination in step S1207 is false, the process returns to step S1205.

In step S1208, the correction unit 1105 determines the pixel of interest in the converted foreground image data corresponding to the reference viewpoint number. In this embodiment, first, the upper left pixel of the converted foreground image data corresponding to the reference viewpoint number is selected as the pixel of interest, and the unprocessed pixel is sequentially selected as the pixel of interest. If the pixel values are updated based on the degree of coincidence (step S1209) for all the pixels of the converted foreground image data corresponding to the reference viewpoint number, the pixels of interest may be determined in any order.

In step S1209, the correction unit 1105 detects pixels that are likely to be pixels in the shadow region in the converted foreground image corresponding to the reference viewpoint number, based on the degree of coincidence acquired from the match degree calculation unit 1104. Then, the correction unit 1105 removes the region due to the shadow from the incomplete foreground image by changing the pixel value of the detected pixel to 0. In this embodiment, if the calculated degree of coincidence is equal to or greater than a predetermined threshold value, the degree of coincidence between the pixels of interest at all viewpoints is high, so that the pixels of interest may be pixels in a shadow region having no height. It is judged that the sex is high. Then, the pixel value of the pixel of interest in the converted foreground image data corresponding to the reference viewpoint number is changed to 0. On the other hand, if the calculated degree of coincidence is less than the threshold value, it is determined that the degree of coincidence between the pixels of interest in all viewpoints is low, and it is highly possible that the pixels of interest are pixels in the region of the subject in the foreground having a height. In this case, the pixel value of the pixel of interest in the converted foreground image corresponding to the reference viewpoint number is not changed. In this embodiment, 0.8 is used as the threshold value, but the value of the threshold value is not limited to this.

In step S1210, the correction unit 1105 determines whether the processes of steps S1208 to S1209 have been performed on all the pixels of the converted foreground image data corresponding to the reference viewpoint number. When the result of the determination in step S1210 is true, the correction unit 1105 outputs the converted foreground image data corresponding to the reference viewpoint number for which the correction has been completed to the secondary storage device 104, the external storage device 108, or the display device 109. , A series of processing is completed. On the other hand, if the result of the determination in step S1210 is false, the process returns to step S1208. The above is the process of extracting the foreground region, which is executed by the image processing apparatus 100 in this embodiment.

<About the effect of this example>
Hereinafter, the effects of this embodiment will be described with reference to FIG. The subject 1301 is a foreground subject in which the shadow 1302 of the subject itself exists on the ground surface 1303. The image data 1304 is a foreground image obtained by extracting a region formed by a subject 1301 and a shadow 1302 associated therewith from a plurality of different viewpoints as a foreground region. In this embodiment, the height from the ground surface is obtained based on the degree of matching between the pixels of interest in the plurality of converted foreground image data obtained by converting the image data 1304 into an image when viewed from the viewpoint of interest 1305. Detects pixels in the area with no shadows. Then, the foreground image 1306 is created by correcting the detected pixels. In the foreground image 1306, the region due to the shadow 1302 accompanying the subject 1301 in the foreground having a height is removed, and only the region due to the subject 1301 in the foreground can be extracted. As described above, according to the present embodiment, even if there is a shadow accompanying the subject in the foreground having a height, only the region due to the subject in the foreground is highly accurate without extracting the region due to the shadow. Can be extracted to.

In this embodiment, the foreground image created based on the captured image and the background image captured in advance is used as the incomplete foreground image, but the foreground image created in Examples 1 and 2 is used. May be good. In that case, the region due to the subject in the foreground can be extracted with higher accuracy than when the first and second embodiments and the third embodiment are executed independently.

[Other Examples]
The embodiment of the present invention is not limited to the above-described embodiment, and various embodiments can be taken. For example, in the above-described embodiment, the case where the size of the background image data which is an incomplete background image and the size of the target image data are the same is described, but these sizes do not have to be the same. .. In that case, the viewpoint viewed from above on the ground surface is used as the reference viewpoint, and the background image is converted into the image viewed from the reference viewpoint. Then, the background image is corrected using the converted image, and the corrected background image is converted into an image when viewed from the viewpoint of interest to create background image data corresponding to the target image data.

Further, in the above-described embodiment, the pixel value in the RGB space is used in the calculation of the degree of coincidence and the extraction of the foreground, but the information used is not limited to this. For example, pixel values in different color spaces such as HSV and Lab may be used to calculate the degree of coincidence and extract the foreground.

Further, in the above-described embodiment, when the image is projected and transformed, only one plane on the ground surface is used as a reference, but a plurality of planes parallel to the ground surface may be used as a reference. For example, a plurality of planes are set by carving the height from the ground surface from 0 to 1 cm at equal intervals, and all the converted images obtained by the projective transformation based on each of the set planes are used. The degree of coincidence may be calculated. By doing so, the robustness against the error of the camera parameter is improved.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

201 ... Target image data acquisition unit 202 ... Background image data acquisition unit 203 ... Image conversion unit 204 ... Matching degree calculation unit 206 ... Correction unit 207 ... Foreground extraction unit

The present invention obtains a first acquisition means for acquiring a focus image including a subject, which is captured and acquired from a viewpoint of interest, and a plurality of reference images captured and acquired from a plurality of viewpoints different from the viewpoint of interest. The second acquisition means, the generation means that converts the plurality of reference images acquired by the second acquisition means to generate a plurality of converted images when viewed from the viewpoint of interest, and the first acquisition means. The said in the image of interest, based on an index relating to the difference between the pixel value of the pixel of interest in the acquired image of interest and the pixel value of the pixel of interest and the corresponding pixel in each of the plurality of converted images generated by the generation means. It is an image processing apparatus having a determination means for determining an image area of a subject.

Claims (15)

A target image data acquisition means for acquiring target image data, which is an image obtained by capturing an image of a foreground subject and a background subject from a viewpoint of interest.
A background image data acquisition means for acquiring background image data which is an image of the subject in the background from a plurality of different viewpoints, and
A conversion background image data creation means for creating a plurality of conversion background image data by converting each of the acquired plurality of background image data into an image when viewed from the viewpoint of interest.
A calculation means for calculating the degree of coincidence indicating the degree of coincidence between the pixels of interest in the plurality of converted background image data, and
A correction means for correcting the converted background image data in the viewpoint determined according to the distance from the viewpoint of interest based on the degree of coincidence.
An image process characterized by having a foreground image data creating means for creating foreground image data which is an image in which a region of the foreground subject is extracted based on the target image data and the corrected converted background image data. Device. The conversion background image data creation means according to claim 1, wherein the acquired background image data is projected and converted into an image when viewed from the viewpoint of interest with the ground surface as a reference. Image processing device. The conversion background image data creating means projects the acquired plurality of background image data into an image when viewed from the viewpoint of interest with reference to a plurality of planes parallel to the ground surface in addition to the ground surface. The image processing apparatus according to claim 2, wherein the image processing apparatus is used. The image processing apparatus according to any one of claims 1 to 3, further comprising a detection means for detecting a pixel to be corrected by the correction means based on the degree of coincidence. When the calculated degree of coincidence indicates that the degree of coincidence between the pixels of interest in the plurality of converted background image data is low, the detection means is characterized in that the pixel of interest is detected as the object to be corrected. The image processing apparatus according to claim 4. A continuity calculation means for calculating the continuity indicating the degree of smoothness of the change in the pixel value between the pixels of interest in the plurality of converted background image data,
The image processing apparatus according to any one of claims 1 to 3, further comprising a detection means for detecting a pixel to be corrected by the correction means based on the degree of coincidence and the continuity. .. 6. The continuity calculation means is characterized in that the continuity is calculated based on the converted background image data at the viewpoint closest to the viewpoint of interest and the converted background image data at the viewpoint adjacent to the viewpoint. The image processing apparatus according to. The calculated degree of matching indicates that the degree of matching between the pixels of interest in the plurality of converted background image data is low, and the calculated continuity is the pixels between the pixels of interest in the plurality of converted background image data. The image processing apparatus according to claim 6 or 7, wherein when the smoothness of the change in the value is shown to be low, the detection means detects the pixel of interest as the correction target. The correction means is characterized in that the pixel value of the pixel of interest in the converted background image data at the viewpoint closest to the viewpoint of interest is replaced with an intermediate value of the pixel value of the pixel of interest in the plurality of converted background images. The image processing apparatus according to any one of claims 1 to 8. The image processing apparatus according to any one of claims 1 to 9, wherein the background image data includes an image of the subject in the foreground. The background image data is an image created based on a plurality of images corresponding to a plurality of different times taken continuously in a time series at each of the plurality of different viewpoints, or at each of the plurality of different viewpoints. The image processing apparatus according to any one of claims 1 to 10, wherein the image is captured in a state where the subject in the foreground does not exist and only the subject in the background exists. A target image data acquisition means for acquiring target image data, which is an image obtained by capturing an image of a foreground subject and a background subject from a viewpoint of interest.
Foreground image data acquisition means for acquiring foreground image data in which a region formed by a subject in the foreground and a shadow accompanying the subject is extracted from a plurality of different viewpoints.
A conversion foreground image data creation means for creating a plurality of conversion foreground image data by converting each of the acquired plurality of foreground image data into an image when viewed from the viewpoint of interest.
A calculation means for calculating the degree of coincidence indicating the degree of coincidence between the pixels of interest in the plurality of converted foreground image data, and
A correction means for correcting the converted foreground image data in the viewpoint determined according to the distance from the viewpoint of interest based on the degree of coincidence.
An image processing apparatus comprising: foreground image data creating means for creating foreground image data in which a region of the foreground subject is extracted based on the target image data and the corrected converted foreground image data. The step of acquiring the target image data, which is an image of the foreground subject and the background subject from the viewpoint of interest,
A step of acquiring background image data which is an image of the subject in the background from a plurality of different viewpoints,
A step of creating a plurality of converted background image data by converting each of the acquired plurality of background image data into an image when viewed from the viewpoint of interest.
A step of calculating the degree of coincidence indicating the degree of coincidence between the pixels of interest in the plurality of converted background image data, and
A step of correcting the converted background image data in the viewpoint determined according to the distance from the viewpoint of interest based on the degree of coincidence, and
An image processing method comprising a step of creating foreground image data, which is an image in which a region of the foreground subject is extracted, based on the target image data and the corrected conversion background image data. The step of acquiring the target image data, which is an image of the foreground subject and the background subject from the viewpoint of interest,
A step of acquiring foreground image data in which a region formed by a subject in the foreground and a shadow accompanying the subject is extracted from a plurality of different viewpoints.
A step of creating a plurality of converted foreground image data by converting each of the acquired plurality of foreground image data into an image when viewed from the viewpoint of interest.
The step of calculating the degree of coincidence indicating the degree of coincidence between the pixels of interest in the plurality of converted foreground image data, and
A step of correcting the converted foreground image data in the viewpoint determined according to the distance from the viewpoint of interest based on the degree of coincidence, and
An image processing method comprising a step of creating foreground image data in which a region due to a subject in the foreground is extracted based on the target image data and the corrected foreground image data. A program for operating a computer as the image processing device according to any one of claims 1 to 12.

Images