JP7271115B2 - Image processing device, background image generation method and program - Google Patents

Description

The present invention relates to a technique for generating a background image from an image captured by an imaging device.

As a method of generating a background image from an image captured by an imaging device, there is a method of generating a background image from an area excluding the foreground area of the captured image. In Japanese Patent Laid-Open No. 2004-100001, among the pixels of an image obtained by imaging a predetermined area, pixels whose pixel value changes are less than or equal to a predetermined threshold for a certain period of time are treated as pixels representing the background, and are regarded as background images. A technique for generating is described.

JP-A-2001-43458

An image captured by an imaging device includes an area of pixel values having a small difference from the pixel value of the background even though it is not the background. For example, among the shadows of an object (subject) present in an image captured by an imaging device, the pixel values of the light shadows caused by sunlight or lighting conditions differ from the pixel values of the original background existing in the light shadow area. It is a pixel value with a small difference. For this reason, if the difference is obtained by comparing the pixel values of a captured image obtained by capturing a predetermined area and a background image generated from an image captured by capturing the same area, the light shadow area present in the captured image and the background The pixel value difference from the image may not exceed a predetermined threshold. Since an area where the difference is equal to or less than the threshold is processed as the background, there is a risk that the background image will include a light shadow area that should be processed as the foreground area.

The image processing apparatus according to the present disclosure includes acquisition means for acquiring a captured image captured by an imaging device, a pixel value of each pixel of the captured image acquired by the acquisition means, and before the captured image is captured, detection means for detecting, as a difference area, an area composed of pixels having a difference greater than a predetermined value from a pixel value of each pixel of a comparison target image obtained based on the image captured by the imaging device; setting means for setting a first area wider than the difference area including at least an area corresponding to a shadow of an object in the captured image based on the difference area detected by the detection means; and generating means for generating a background image using pixels forming a second area that is different from the first area set by the setting means.

According to the present invention, a suitable background image can be generated.

1 illustrates a virtual viewpoint system; FIG. 2 is a block diagram showing the hardware configuration of the image processing apparatus; FIG. 2 is a block diagram showing the functional configuration of the image processing device; FIG. It is a figure which shows the process which updates a background image from a captured image. It is a figure explaining expansion processing. 6 is a flowchart showing dilation processing; It is a figure explaining the calculation method of the expansion pixel number. 6 is a flowchart showing dilation processing;

The present invention will now be described in detail on the basis of some exemplary embodiments with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

<Embodiment 1>
The present embodiment is directed to an image processing apparatus that performs processing such that a foreground having a small pixel value difference from the background is not treated as a background in processing for generating a background image based on an image captured by an imaging device such as a camera. It is a related form.

FIG. 1 shows a schematic configuration of the system of this embodiment. A plurality of cameras 102 are arranged side by side around the stadium 101 so that the field 104 of the stadium can be imaged from a plurality of directions. Since each camera 102 has a fixed attitude and position in principle, each camera images the same imaging area in the time direction. Each camera 102 has input/output hardware (not shown) for data transmission. The cameras 102 are connected to each other in a ring configuration using a network such as a network cable, and image data representing images (moving images or still images) are sequentially transmitted to adjacent cameras via the network. . One of the cameras 102 is connected to an image processing device 200 (see FIG. 2), and image data of images captured by each camera 102 is transmitted to the image processing device 200 .

In this embodiment, an image processing apparatus 200 relating to a system in which a plurality of cameras are arranged around a stadium as shown in FIG. 1 to generate a virtual viewpoint image will be described below as an example. Here, a virtual viewpoint image is an image generated by an end user or a designated operator or the like by freely manipulating the position and orientation of a camera (virtual camera) corresponding to a virtual viewpoint. A virtual viewpoint image may be a moving image or a still image.

It is assumed that a game such as soccer is being played on the field 104 of the stadium 101 and that a person 103 exists as an object in the field 104 . Image data of images captured by each camera 102 is transmitted to the image processing apparatus 200 via the network. An object is a specific person, such as a player, coach, or referee. Alternatively, the object may be an object with a predetermined image pattern, such as a ball or goal. Also, the object may be a moving body or a stationary body.

FIG. 2 is a diagram showing an example of the hardware configuration of the image processing apparatus 200. As shown in FIG. The image processing apparatus 200 is composed of a CPU 201 , a RAM 202 , a ROM 203 , an HDD 204 and an input/output I/F 205 . Each unit constituting the image processing apparatus 200 is interconnected by a system bus 206 . Also, the image processing apparatus 200 is connected to the camera 102 and the display operation unit 210 via the input/output I/F 205 .

The CPU 201 executes a program stored in the ROM 203 using the RAM 202 as a work memory, and controls each unit of the image processing apparatus 200 through the system bus 206 . As a result, various processes to be described later are realized. The HDD 204 is a large-capacity storage device that stores various data handled by the image processing apparatus 200, and may be an SSD, for example. The CPU 201 can write data to the HDD 204 and read data stored in the HDD 204 via the system bus 206 .

The input/output I/F 205 is, for example, a serial bus I/F such as USB or IEEE1394. done through The display operation unit 210 is configured by, for example, a liquid crystal display having a touch panel function, and displays necessary information to the user or acquires instructions from the user via the UI screen. Although there may be components of the image processing apparatus 200 other than those described above, the description thereof is omitted.

FIG. 3 is a block diagram showing the functional configuration of the image processing apparatus 200. As shown in FIG. The image processing apparatus 200 includes an image acquisition unit 301, a difference area detection unit 302, a background image storage unit 303, a difference area setting unit 304, a background image generation unit 305, a background difference processing unit 306, a foreground texture generation unit 307, and a virtual viewpoint image. It has a generator 308 . Note that the difference area is composed of a plurality of pixels, but the plurality of pixels need not be connected and may be separated into two or more. Also, the plurality of pixels forming the difference area may be separated from each other. Also, the present embodiment can be applied even when the number of pixels whose difference is less than the predetermined threshold is one.

The image acquisition unit 301 acquires image data of an image captured by the camera 102 . A captured image is an image such as a still image or a frame image of a moving image. The difference area detection unit 302 detects a difference area, which is an area in which the difference in pixel values between the captured image and the comparison target image is larger than a predetermined value. In this embodiment, the background image stored in the background image storage unit 303 is used as the comparison target image. Details will be described later. A background image storage unit 303 stores a background image generated before the captured image is captured. Background image storage unit 303 is realized by ROM 203 or HDD 204 of image processing apparatus 200 .

Although the background image storage unit 303 is included in the configuration of the image processing apparatus 200 in this embodiment, it may be implemented by a ROM, HDD, or the like in a device different from the image processing apparatus 200 . In that case, the image processing apparatus 200 acquires the background image by being connected via a network or the like to an apparatus having a background image storage unit.

A difference area setting unit 304 expands the difference area based on the difference area detected by the difference area detection unit 302 to set an area wider than the difference area. In this embodiment, dilation processing is performed as processing for widening the difference region. A description of the expansion process will be given later.
The background image generation unit 305 generates a background image based on the captured image and the image representing the difference area expanded by the difference area setting unit 304 . In this embodiment, as will be described later, the background image stored in the background image storage unit 303 is updated based on the captured image and the image showing the difference area expanded by the difference area setting unit 304. This is the form in which the background image is generated. Alternatively, the background image creation unit 305 may generate an image of an area other than the difference area expanded by the difference area setting unit 304 in the captured image as the background image.

A background difference processing unit 306 compares the updated background image and the captured image, and generates a foreground mask indicating a foreground area from the area where the difference exists. A foreground texture generation unit 307 generates a foreground texture by cutting out the foreground texture based on the captured image and the foreground mask. A virtual viewpoint image generator 308 generates a virtual viewpoint image from the foreground mask, the foreground texture, and the background image.

Note that the background difference processing unit 306, the foreground texture generation unit 307, and the virtual viewpoint image generation unit 308 may be implemented as a configuration of a device separate from the image processing device 200. FIG. In that case, the device having the background difference processing unit 306 acquires the captured image and the updated background image from the image processing device 200 via a network or the like.

The functions of each module in FIG. 3 are implemented by the CPU 201 developing program codes stored in the ROM 203 in the RAM 202 and executing them. In addition, for example, the image processing apparatus 200 may have a form having an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array). In that case, each module shown in FIG. 3 is mounted inside an ASIC or FPGA as hardware.

[Description of background image update process]
FIG. 4 is a diagram illustrating, as a comparative example, update processing of a background image in a case where an image captured by an imaging device such as a camera is not subjected to expansion processing. FIG. 4A is an example of a captured image captured by the camera 102, in which a background field 401 and a person appearing as an object 402 inside the field. A shadow of an object (a shadow of a person) 403 appears inside the field. A captured image is an image such as a still image or a frame image of a moving image. FIG. 4(b) is a background image stored in the background image storage unit 303 as the background of the same imaging area as the imaged image of FIG. 4(a).

FIG. 4(c) is a diagram showing an area where the difference in pixel value between the area of the captured image and the corresponding area of the background image exceeds a predetermined value. An image showing a "difference area", which is an area where the difference in pixel value between the captured image and the background image exceeds a predetermined value, is an image in which each pixel is divided into a black area and a white area, as shown in FIG. It is an image represented by a binary image. In FIG. 4C, a black area indicates a difference area 404 in which the difference exceeds a predetermined value, and indicates that the area is determined to be the foreground. On the other hand, white areas indicate areas where the difference is equal to or less than a predetermined value. A region other than the difference region is called a non-difference region 405 . A non-difference area 405 indicates an area determined to be the background.

FIG. 4(d) is an example of the background image after the stored background image is updated based on the captured image of FIG. 4(a). When the image showing the non-difference region 405 in FIG. 4C is superimposed on the captured image in FIG. The background image is updated by overwriting the pixels corresponding to the non-difference area 405 of the background image. At this time, the difference between the pixel values of the shadow 403 of the object and the blurred area 407 (area where the pixel values are mixed at the boundary between the object and the background) generated in the outline of the object is small from the pixel value of the background image. Therefore, the shadow 403 of the object and the blurred area 407 may not be included in the difference area 404, which is an area where the difference is larger than a predetermined value. In this case, the shadow 403 of the object and the blurred area 407 are included in the non-difference area 405 of FIG. 4(c). Since the non-difference area 405 is processed as the background as described above, the shadow 403 of the object and the blurred area 407 may be captured as the background and the background image may be updated as in the image of FIG. 4(d).

If the shadow area or the blurred area is captured as part of the background image, it may cause inconvenience in the processing performed based on the updated background image. For example, the background difference processing unit 306 compares the updated background image with a captured image of the same area, and separates the area where the difference exists to generate a foreground mask indicating the foreground area that is the object. There is At this time, if the updated background image includes the shadow of the object as shown in FIG. 4D, the shadow of the object in the captured image is not determined to be the difference from the background image. Therefore, the shadow of the object may not be processed as the foreground area.

FIG. 5 is a diagram illustrating processing for updating a background image from a captured image captured by an imaging device such as a camera using the technology according to the present embodiment. FIGS. 5(a), 5(b) and 5(c) are the same views as FIGS. 4(a), 4(b) and 4(c) respectively. FIG. 5D is a diagram after dilation processing is performed on the difference area 404 (black area) by the difference area setting unit 304 .

Here, the "expansion process" means that when dilating the difference area 404 in FIG. It is to process to spread. For example, each pixel in the difference area is set as a target pixel, and the dilation process is performed for each unit area in which the target pixel is surrounded by one pixel. Specifically, if the pixel of interest is black, all the pixels (eight pixels) in the unit area are processed to have the same black color as the pixel of interest. The number of pixels (one pixel in the above example) for dilation processing at that time is called the "dilation pixel number". Note that the expansion processing method is not limited. For example, when a target pixel has black pixels around the target pixel, the pixel value of the target pixel may be changed to a black pixel value.

An example of the result of dilation processing is shown in FIG. 5(d). In this way, the difference area 404 in FIG. 5(c) is expanded, and the shadow 403 of the object and the blurred area 407 generated in the outline portion of the object are included in the difference area 501 after the expansion process. become.

The image of FIG. 5(e) is an image showing the background image after being updated based on the image of FIG. 5(a) in which the saved background image is the captured image. The difference region setting unit 304 superimposes the image of FIG. 5(d) including the dilated difference region 501 and the captured image of FIG. 5(a). At that time, the pixels of the captured image corresponding to the non-difference region 502 (white region), which is the region other than the dilated difference region, are superimposed on the pixels of the same region of the saved background image of FIG. background image is updated. The difference area 501 is expanded to include the shadow 403 of the object and the blurred area 407 occurring at the outline of the object. Therefore, the non-difference area 502 indicating the background does not include the shadow 403 of the object and the blurred area 407 generated in the outline portion of the object. Therefore, unlike the background image of FIG. 4(d), the background image of FIG. 5(d) updated according to the present embodiment does not include the shadow 403 and the blurred area 407. FIG.

Therefore, for example, the background difference processing unit 306 can perform difference processing by comparing a background image in which the shadow 403 of the object is not captured and the captured image. Therefore, since the background difference processing unit 306 can recognize the shadow 403 of the object as a difference, the shadow of the object can also be processed as a foreground area when generating the foreground mask.

[flowchart]
FIG. 6 is a flow chart showing a series of processes when the image processing apparatus 200 in this embodiment updates the background image. A series of processes shown in the flowchart of FIG. 6 are performed by the CPU 201 expanding the program code stored in the ROM 203 into the RAM 202 and executing the program code. Alternatively, some or all of the functions of the steps in FIG. 6 may be realized by hardware such as ASIC, FPGA, or electronic circuits. Note that the symbol "S" in the description of each process means a step in the flowchart.

In S601, the image acquisition unit 301 acquires image data representing a captured image sent from each camera 102 via the network.

In step S<b>602 , the image acquisition unit 301 performs preprocessing such as image vibration correction, image distortion correction such as lens distortion, and color and gamma adjustment on the image data representing the captured image.

In S603, the difference area detection unit 302 acquires image data representing the preprocessed captured image. Further, the difference area detection unit 302 acquires image data representing the background image stored in the background image storage unit 303 as a comparison target image. The difference area detection unit 302 calculates the difference between the pixel values of each pixel in the acquired area of the background image and the corresponding area of the captured image, and determines whether the difference is greater than a predetermined value. If the difference is greater than a predetermined value, the difference area detection unit 302 determines that the pixel is a pixel in the difference area.

Here, the predetermined value is a value determined so that the background area of the captured image is not determined to be the difference area even when the pixel value fluctuates due to minute noise included in the background area of the captured image. is. Thus, the predetermined value is determined based on noise variations. The predetermined value is, for example, a predetermined value such as a value that is several percent of the pixel value of the background image. Alternatively, the image processing apparatus 200 may be provided with a function of monitoring the noise level of the captured image, and the difference area detection unit 302 may change the predetermined value as needed according to the noise level of the captured image. The user gives an instruction to change a predetermined value via the display operation unit 210 of the image processing apparatus 200, the difference area detection unit 302 acquires the instruction, and determines the difference area based on the predetermined value determined by the user. It can be in any form.

In S<b>604 , the difference area setting unit 304 calculates a feature amount that serves as a basis for determining the number of dilated pixels. In this embodiment, the number of pixels (area) of the difference area for each object is used as the feature amount. The difference area setting unit 304 counts the pixels in the difference area for each object to calculate the number of pixels in the difference area.

In S605, the difference area setting unit 304 multiplies the number of pixels in the difference area for each object by a predetermined coefficient to determine the number of dilation pixels for dilating the difference area.

In S<b>606 , the difference area setting unit 304 performs expansion processing on the difference area for each object based on the expansion pixel number determined by the processing in S<b>605 .

In step S<b>607 , the background image generation unit 305 generates the imaged image acquired by the image acquisition unit 301 , the background image stored in the background image storage unit 303 , and the difference area and non-difference area expanded by the difference area setting unit 304 . Get an image showing The background image creation unit 305 converts each pixel of the captured image corresponding to the non-differential region 502 after expansion processing to each pixel corresponding to the non-differential region of the background image stored in the background image storage unit 303, A background image is generated by overwriting so as to replace each pixel of the captured image. That is, the background image is updated such that the pixel value of each pixel of the overwritten background image is replaced with the pixel value of each pixel of the captured image to be overwritten. The updated background image is stored in the background image storage unit 303 .

The image acquisition unit 301 transmits image data representing the acquired captured image to the difference area detection unit 302 at regular intervals, and each time, the difference area detection unit 302 acquires image data representing the captured image. Therefore, although the description is omitted, the processing from S601 to S607 is repeatedly performed. The background image storage unit 303 stores the background image updated immediately before the captured image is captured. Therefore, when the process of S601 is performed next, the process of determining the difference area of S603 is performed by comparing the updated background image and the new captured image. In this manner, the background image can be updated at any time.

With the above processing, the processing for updating the background image from the captured image ends. As the processing for generating the virtual viewpoint image, the processing for generating the foreground image is performed after this.

When generating a virtual viewpoint image, processing is performed in which the foreground, which is the main subject (object), is modeled separately from the background portion, and then rendered. When modeling the foreground, foreground mask information corresponding to the silhouette of the foreground viewed from a plurality of cameras and foreground texture information (for example, R, G, and B color information for each pixel of the foreground) and Is required.

The process of separating the foreground from the background portion is called foreground-background separation. The foreground/background separation process is a process for estimating a foreground region, and is generally performed by a background subtraction method. In the background subtraction method, a difference between a background image and an input image including the foreground is obtained, and an area that is a group of pixels determined to have a difference value equal to or greater than a predetermined threshold value is regarded as the foreground area. In other words, a background image is required to obtain the foreground.

The background difference processing unit 306 acquires the updated background image from the background image creation unit 305 . Also, the background difference processing unit 306 acquires the captured image from the image acquisition unit 301 . A background subtraction processing unit 306 performs a background subtraction method that compares the captured image with the updated background image, and creates a foreground mask indicating the foreground area from the area with the difference.

The background image used in creating the foreground mask is the background image updated by the background image creating unit 305. For example, as shown in FIG. . Therefore, according to the present embodiment, the background difference processing unit 306 can perform difference processing by comparing the background image in which the shadow of the object is not captured and the captured image. Therefore, since the background difference processing unit 306 can recognize the shadow area of the object as a difference, the shadow of the object can also be determined as the foreground area when generating the foreground mask.

Next, the foreground texture generation unit 307 acquires the captured image from the image acquisition unit 301 and acquires the foreground mask from the background difference processing unit 306 . A foreground texture generation unit 307 generates a foreground texture by cutting out the foreground texture based on the captured image and the foreground mask.

Next, processing for generating a virtual viewpoint image using the background image, foreground mask, and foreground texture will be described. A virtual viewpoint image generation unit 308 acquires a background image, a foreground mask, and a foreground texture generated by multi-viewpoint images captured by other multi-viewpoint cameras. The virtual viewpoint image generation unit 308 uses the acquired foreground masks and foreground textures of multiple viewpoints to perform a three-dimensional shape estimation process for each object present in the captured scene. As an estimation method, a known method such as a visual-hull method using contour information of an object or a multi-view stereo method using triangulation may be applied. As a result, data representing the three-dimensional shape of the object (for example, polygon data or voxel data) is generated. Then, the image quality level of the generated virtual viewpoint image is set.

Parameters such as the position/orientation of the virtual camera (virtual camera path) and the gaze point (virtual gaze point path) in the target time frame of the virtual viewpoint image are set based on user input via the display operation unit 210 . A virtual viewpoint image generation unit 308 generates a virtual viewpoint image according to the parameters of the virtual camera described above. A virtual viewpoint image can be generated by using computer graphics technology, using 3D shape data of an object obtained by shape estimation processing, and an image viewed from a set virtual camera. A well-known technique may be appropriately applied to this generation process, and the explanation is omitted because it is not the main focus of the present embodiment.

As described above, according to the present embodiment, a background image can be generated without processing a foreground area having a small difference in pixel value from the background as the background.

[Modification]
In the description of the present embodiment, an embodiment using dilation processing has been described as one of the setting processing for widening the difference region. Setting processing for widening the difference region is not limited to dilation processing. For example, the user may instruct the area to expand the difference area via the display operation unit 210, and the difference area setting unit 304 may expand the difference area based on the instruction.

In the present embodiment, the number of pixels (area) is used as a feature quantity that is the basis for determining the number of dilated pixels. As a modification, information on the height of the difference area or the width of the difference area, which is a measure of the shape of the object, may be used as a feature amount. Alternatively, the amount of movement of the object compared with the past frame image, which serves as a measure of the degree of outline blur due to the movement of the object, may be used as the feature amount.

As a method for determining the number of dilated pixels from the feature amount, the feature amount is multiplied by a constant coefficient in this embodiment. Alternatively, the image processing apparatus 200 may manage the number of expanded pixels corresponding to the feature amount in a table format, and the number of expanded pixels may be determined according to the table.

In this embodiment, the number of dilated pixels is determined according to the feature amount, but the number of dilated pixels may be determined as a constant value in order to reduce the processing. Alternatively, the user instructs the expansion pixel number via the display operation unit 210, the difference area setting unit 304 acquires the expansion pixel number instructed by the user, and the difference area setting unit 304 acquires the expansion pixel number according to the user instruction. may be changed.

Further, the difference area setting unit 304 may further expand the lower part of the outline of the difference area so that the shadow of the object is included in the difference area. First, the difference area setting unit 304 expands the difference area so that the blurred area of the object is included. Next, the difference area setting unit 304 expands the area below the difference area so that the shadow area of the object is also included in the difference area. In this manner, the difference area setting unit 304 may perform the expansion process a plurality of times with different values for the number of pixels to be expanded for including the blurred area and the shadow area of the object in the difference area. Alternatively, the number of expansion pixels may be set so as to further expand the area below the difference area, and the area below the outline of the difference area may be further expanded in one expansion process. Areas of shadow extend more from the outline of the object than areas of blur. For this reason, it is possible to reduce the area to be subjected to the expansion process of the difference area as a whole, rather than uniformly performing the expansion process on the difference area in order to include the blur area and the shadow area in the difference area. Therefore, the number of dilated pixels that differs depending on the location of the difference area is used, so that the area of the background to be updated can be widened.

<Embodiment 2>
In this embodiment, two thresholds are provided, and the number of dilated pixels is determined by estimating a shadow area or a blurred area of an object based on the relationship between the positions of pixels that reach the respective thresholds. By estimating the shadow area or the blur area, the area in which the difference area is expanded by expansion processing can be made an appropriate area. As a result, for example, it is possible to reduce the area to be expanded by the dilation process, and to widen the area of the background to be updated.

This embodiment will be described with a focus on differences from the first embodiment. Parts not specified are the same in configuration and processing as in the first embodiment. Since the configuration of the image processing apparatus 200 is the same as that of the first embodiment, description thereof is omitted.

[Description of how to determine the number of dilated pixels]
FIG. 7 is a diagram for explaining processing for determining the number of dilated pixels from two thresholds in this embodiment. The horizontal axis of the graph in FIG. 7(b) represents the position on the line in the horizontal direction (x direction) from a to a' of the captured image in FIG. 7(a). The vertical axis of the graph in FIG. 7B represents the difference in pixel value between the pixels on the line from a to a' on the captured image and the pixels at the same position on the comparison target image. The curve on the graph in FIG. 7(b) connects points obtained by plotting the coordinates indicating the position on the line from a to a′ and the difference in pixel value between the captured image and the comparison target image at the position. It is a thing. Note that the present embodiment will also be described assuming that the stored background image is used as the image to be compared.

The first threshold is a predetermined value (see Embodiment 1) that indicates the boundary between the difference area and the non-difference area of the captured image, and is indicated by th1 in the graph of FIG. 7B. That is, the area of the horizontal axis x in which the difference in the graph in FIG. 7B is larger than th1 is the difference area. The second threshold is set as a value smaller than the first threshold, and is shown as th2 in the graph of FIG. 7(b). The second threshold is set, for example, as half the value of the first threshold.

Let x1 be the value of the horizontal axis x when the curve on the graph reaches the first threshold value (th1) in the difference, which is the vertical axis of the graph. Similarly, let x2 be the value of the horizontal axis x when the curve on the graph reaches the second threshold value (th2) in the difference on the vertical axis. At this time, let (x0, 0) be the coordinates where the extension of the straight line connecting the coordinates (x1, th1) and (x2, th2) intersects the horizontal axis x. From FIG. 7B, x0 is calculated by the following equation.
x0=(th1×x2−th2×x1)/(th1−th2) (1)

The number of expansion pixels is determined by estimating this x0 to be the boundary between the shadow area of the object or the blurred area and the background on the aa' line. For example, as described in the first embodiment, the difference area detection unit 302 determines an area having a difference larger than a predetermined value as a difference area. The predetermined value in this embodiment is the first threshold. Therefore, the difference area detection unit 302 does not determine that an area in which the horizontal axis x of the graph is smaller than x1 is a difference area. On the other hand, the difference area setting unit 304 dilates the difference area so that the shadow area or blur area of the object is included in the difference area.
x1-x0 (2)

When the number of dilated pixels is obtained from equation (2), since the number of pixels is a natural number, the value obtained by rounding up the decimal point of x1-x0 obtained by equation (2) is used as the number of dilated pixels. In this way, the difference area setting unit 304 determines the number of dilated pixels in which the shadow area or the blurred area is included in the difference area by dilation processing.

[flowchart]
FIG. 8 is a flowchart showing dilation processing by determining the number of dilated pixels from two thresholds. S801 to S802 are the same processing as S601 to S602 in FIG. 6, so description thereof will be omitted.

In S803, the difference area detection unit 302 provides a specific line on the object of the acquired captured image, calculates the pixel value difference for each pixel on the line between the captured image and the background image, and calculates the difference. It is determined whether it is larger than the first threshold. The difference area detection unit 302 stores the coordinate on the line when the difference reaches the first threshold (th1) as x1.

In S804, similarly, the difference area detection unit 302 determines whether the difference in pixel value of each pixel between the captured image and the background image on the line is greater than the second threshold. The difference area detection unit 302 stores the coordinate on the line when the difference reaches the second threshold (th2) as x2.

In S805, the difference region setting unit 304 obtains the coordinates (x0, 0) based on the coordinate values (x1, th1) and (x2, th2) of the boundary obtained by the two thresholds. The difference area setting unit 304 calculates x1-x0, and acquires a value obtained by rounding up the small end point of the calculated value as the number of dilated pixels.

In S806, the difference area setting unit 304 performs expansion processing on the difference area using the expansion pixel number calculated in S805.

Since the processing of S807 is the same as that of S607 in FIG. 6, the description thereof is omitted. Further, since the subsequent processing is also the same as that of the first embodiment, the explanation is omitted.

As described above, in this embodiment, two thresholds are provided to estimate a shadow area and a blurred area and determine the number of dilated pixels. By estimating the shadow area or the blur area, the area for widening the difference area can be set as an appropriate area, and as a result, for example, the background area to be updated can be widened.

[Modification]
In the above embodiment, the number of dilated pixels to be applied to the entire difference area is determined from a specific line, but the number of dilated pixels may be determined from a plurality of lines. For example, the difference area setting unit 304 sets a plurality of lines such as aa' in FIG. 7A so that the entire object is covered. Next, the difference area setting unit 304 obtains x1 and x0 of each line, calculates the number of dilated pixels for each line individually, applies a different number of dilated pixels to each line, and performs dilation processing for each line. You may In that case, in the description of the second embodiment, x0 is determined for the area on the left side of the object. Determine the number of dilated pixels for each of the border of and the right border.

Lines such as aa' in FIG. 7(a) can be drawn not only in the horizontal direction but also in the vertical direction, and the difference area setting unit 304 can determine the number of expanded pixels based on the vertical line. good. Since the shadow of an object usually appears vertically downward, the difference area setting unit 304 can determine the number of dilated pixels according to the shadow area by providing lines in the vertical direction as well. Alternatively, the difference area setting unit 304 may provide lines in the vertical direction and the horizontal direction, and determine the number of expanded pixels from each line.

Alternatively, an average value of the numbers of expanded pixels calculated from a plurality of lines may be calculated, and the average value may be used as the number of expanded pixels of the entire difference area.

<Other embodiments>
In the above-described embodiment, a system that generates a virtual viewpoint image by arranging a plurality of cameras around a stadium has been described as an example. However, any mode may be used as long as the background image is updated from image data captured by a single camera, and is not bound by the relative geometrical installation conditions between a plurality of cameras. Therefore, for example, a system in which the image processing apparatus 200 acquires image data from a monitoring camera installed in a premises, outdoors, or at a remote location and updates the background image can also be implemented.

In the above-described embodiment, the comparison target image is the background image stored in the background image storage unit 303, and the background image is compared with the captured image, which is the comparison image, to determine the difference area. The image to be compared may also be an image in which the same area as the captured image of the comparison image is captured, and which is captured before the captured image is captured. The number of images to be compared is not limited to one, and may be plural. In this case, an area in which there is a difference between the captured image of the comparison image and any of the images to be compared is determined as a difference area.

In the above-described embodiment, one image processing apparatus 200 acquires image data from a plurality of cameras and generates a background image, a foreground mask, and a foreground texture for each camera. The hardware (not shown) of each camera may have a function other than the function of the virtual viewpoint image generation unit of the image processing device 200, without being limited to the above-described embodiment. For example, in the hardware of each camera, a background image, a foreground mask and a foreground texture are generated from the images captured by each camera, and the background image, the foreground mask and the foreground texture generated in another device that generates a virtual viewpoint image. may be sent.

Embodiments such as systems, devices, methods, programs, or recording media (storage media) are possible. Specifically, it may be applied to a system composed of a plurality of devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to a device composed of a single device. good.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

301 image acquisition unit 302 difference area detection unit 303 background image storage unit 304 difference area setting unit 305 background image creation unit

Claims (22)

Acquisition means for acquiring a captured image captured by an imaging device;
A pixel value of each pixel of the captured image acquired by the acquisition means, and a pixel value of each pixel of the comparison target image obtained based on the image captured by the imaging device before the captured image is captured. , detecting means for detecting, as a difference area, an area composed of pixels having a difference greater than a predetermined value;
setting means for setting, based on the difference area detected by the detection means, a first area wider than the difference area including at least an area corresponding to a shadow of an object in the captured image ;
generating means for generating a background image using pixels constituting a second area, which is different from the first area set by the setting means, in the captured image;
An image processing device comprising: The setting means sets a region obtained by adding a region corresponding to the shadow of the object in the captured image to a region wider than the difference region including the region corresponding to the outline blur due to the motion of the object in the captured image. 2. The image processing apparatus according to claim 1, wherein the area is set as the first area. The generating means updates each pixel constituting an area corresponding to the second area in the previously generated background image with each pixel constituting the second area in the captured image. 3. The image processing apparatus according to claim 1 , wherein the background image is generated by: The generating means converts the pixel values of the pixels forming the region corresponding to the second region in the previously generated background image to the pixel values of the pixels forming the second region in the captured image. 4. The image processing apparatus according to claim 3 , wherein a previously generated background image is updated by substituting pixel values. 5. The image processing apparatus according to claim 3, wherein the image to be compared is a previously generated background image. The image processing apparatus according to any one of claims 1 to 4 , wherein the comparison target image is an image captured by the imaging device before the captured image is captured. 7. The image processing apparatus according to claim 1, wherein said setting means performs said setting based on a predetermined number of dilated pixels. 8. The image processing apparatus according to claim 7 , wherein said setting means sets a value corresponding to the area of said difference area as said number of dilated pixels. 8. The image processing apparatus according to claim 7 , wherein said setting means obtains a value specified by a user, and performs said setting using said value specified by said user as said number of expanded pixels. 8. The image processing apparatus according to claim 7 , wherein said setting means determines an amount of movement of an object, and performs said setting using a value corresponding to said amount of movement as said number of expanded pixels. Acquisition means for acquiring a captured image captured by the imaging device;
A pixel value of each pixel of the captured image acquired by the acquisition means, and a pixel value of each pixel of the comparison target image obtained based on the image captured by the imaging device before the captured image is captured. , detecting means for detecting, as a difference area, an area composed of pixels having a difference greater than a predetermined value;
setting means for setting a first area, which is a wider area than the difference area detected by the detection means, based on a predetermined number of dilated pixels;
generating means for generating a background image using pixels constituting a second area, which is different from the first area set by the setting means, in the captured image;
has
The detection means determines a difference between a pixel value of each pixel on a line of the captured image and a pixel value of each pixel on a line corresponding to the line of the comparison target image, and the difference is the predetermined value. determining a first position that is a value of and a second position that is a second value where the difference is less than the predetermined value;
The image processing apparatus, wherein the setting means determines the predetermined number of dilated pixels based on the relationship between the first position and the second position. The setting means extends a straight line connecting the coordinates indicating the first position and the coordinates indicating the second position in the graph indicating the difference corresponding to each position on the line of the captured image. 12. The method according to claim 11 , wherein a third position is determined from the coordinates on the line at which the difference is 0, and the number of dilated pixels is determined based on the first position and the third position. image processing device. The detection means determines the first position and the second position for each of the plurality of lines on the captured image,
13. The image processing apparatus according to claim 11 , wherein the setting unit determines the number of expanded pixels for each line. 14. The image processing apparatus according to any one of claims 1 to 13 , wherein the setting means performs the setting so that a specific area in the difference area is wider than other areas. 15. The image processing apparatus according to claim 1, wherein said detection means updates said predetermined value according to noise contained in said captured image. The captured image is an image obtained by capturing a predetermined area,
The image processing apparatus according to any one of claims 1 to 15, wherein the comparison target image is also an image corresponding to the predetermined area. 17. The image processing apparatus according to any one of claims 1 to 16 , wherein a foreground image is generated based on the background image generated by the generation means and the captured image. 18. The image processing device according to any one of claims 1 to 17 , wherein the image processing device generates the virtual viewpoint image using the multi-viewpoint images captured by a plurality of cameras and the background image generated by the generating means. 1. The image processing apparatus according to claim 1. 19. The apparatus according to any one of claims 1 to 18 , further comprising separating means for separating a foreground and a background in said captured image obtained by said obtaining means based on said background image generated by said generating means. 1. The image processing device according to item 1. The separating means determines, as the foreground area, a pixel area in which a difference between the background image generated by the generating means and the captured image obtained by the obtaining means is equal to or greater than a predetermined threshold. The image processing apparatus according to claim 19 . an acquisition step of acquiring a captured image captured by an imaging device;
A pixel value of each pixel of the captured image acquired in the acquiring step, and a pixel value of each pixel of a comparison target image obtained based on the image captured by the imaging device before the captured image is captured. a detection step of detecting, as a difference area, an area composed of pixels whose difference between , is greater than a predetermined value;
a setting step of setting a first area wider than the difference area including at least an area corresponding to a shadow of an object in the captured image based on the difference area detected by the detection step;
a generation step of generating a background image using pixels constituting a second region, which is a region different from the first region set in the setting step, in the captured image;
A method for generating a background image, comprising: A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 20 .

Images