JP2018195190A - Image processing apparatus and image processing method - Google Patents

Abstract

To improve specific accuracy in a foreground region inside an input image.SOLUTION: By using difference on the basis of a first feature amount between an input image and a background image, a first region in the input image is specified. By using difference on the basis of a second feature amount between the input image and the background image, a second region in the input image is specified. A foreground region corresponding to a predetermined object in the input image is determined on the basis of the first region and the second region.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a technique for specifying a foreground region in an input image.

  In recent years, attention has been paid to a technique of installing a plurality of cameras at different positions, performing synchronous shooting from multiple viewpoints, and generating an arbitrary virtual viewpoint content using a plurality of viewpoint images obtained by the shooting. In this way, according to the technology for generating an arbitrary virtual viewpoint content from a plurality of viewpoint images, for example, a soccer or basketball highlight scene can be viewed and viewed from various angles. High sense of presence can be given to the user.

  On the other hand, the generation and browsing of arbitrary virtual viewpoint content based on a plurality of viewpoint images is performed by consolidating images taken by a plurality of cameras into an image processing unit such as a server, and performing processing such as rendering in the image processing unit. This can be realized by transmitting to the user terminal. Various techniques have been developed for technologies for generating virtual viewpoint images. For example, there is a technique in which a foreground such as a player is separated from a plurality of camera images, three-dimensional coordinates are calculated, and reprojected onto a plane to generate a virtual viewpoint image.

  Here, as a method for separating the foreground, for example, a so-called background subtraction method is generally known. First, background information is generated based on image information for a certain period of time in the past, and a difference from the current image is detected and separated as a foreground. As for foreground separation, other methods such as a feature amount and machine learning are known. Patent Document 1 discloses a method for improving the performance of foreground separation by combining estimation of a point of interest based on spatial frequency and foreground shape estimation processing based on color information.

JP 2014-232477 A

  However, foreground shape estimation using one feature has a limit in performance. For example, in the method based on color information, the performance may deteriorate when the foreground and background colors are close.

  The present invention has been made in view of such problems, and provides a technique for improving the accuracy of specifying a foreground region in an input image.

  According to an aspect of the present invention, a first specifying unit that specifies a first region in an input image using a difference based on a first feature amount between the input image and the background image, the input image, Using a difference based on a second feature amount with respect to a background image, a second specifying means for specifying a second area in the input image, the first area specified by the first specifying means, and And determining means for determining a foreground area corresponding to a predetermined subject in the input image based on the second area specified by the second specifying means.

  According to the configuration of the present invention, the specific accuracy of the foreground area in the input image can be improved.

It is a figure which shows the structural example of a system. 3 is a block diagram illustrating a functional configuration example of an image processing apparatus 103. FIG. 3 is a flowchart of processing performed by the image processing apparatus 103. 3 is a flowchart of processing performed by the image processing apparatus 103. It is a figure which shows an example of a foreground area | region. 3 is a block diagram illustrating a functional configuration example of an image processing apparatus 103. FIG. 5 is a flowchart of processing performed by a foreground separation unit 606. It is a block diagram which shows the hardware structural example of a computer apparatus.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
A configuration example of a system according to the present embodiment will be described with reference to FIG. As shown in FIG. 1, the system according to the present embodiment is based on a plurality of cameras 102 connected in a ring shape via a transmission path and images captured by at least some of the plurality of cameras 102. And an image processing device 103 that generates a virtual viewpoint image.

  The plurality of cameras 102 are installed around the stadium 101, and each camera 102 captures a moving image of the stadium 101 from different directions. One of the plurality of cameras 102 is connected to the image processing apparatus 103, and an image captured by each camera 102 (an image of each frame constituting a moving image) is transmitted via a transmission path, another camera 102, or the like. The image is transferred to the image processing apparatus 103. As a result, images captured by the plurality of cameras 102 are input to the image processing apparatus 103 in units of frames.

  The image processing device 103 is based on a group of captured images necessary for generating an image (virtual viewpoint image) of the stadium 101 viewed from a virtual viewpoint designated by the user, among the captured images of the respective cameras 102. Then, the virtual viewpoint image is generated. It is assumed that a competition such as soccer is being performed in the stadium 101, and a person 104 as a subject is present in the stadium 101. However, the subject to be photographed is not limited to the competition in the stadium 101, but may be, for example, performance on the stage. Further, the viewpoint is not limited to that specified by the user, but may be determined automatically.

  Note that the configuration of the system is not limited to the configuration illustrated in FIG. 1 as long as the image processing apparatus 103 can acquire images captured by the plurality of cameras 102 arranged around the stadium 101. For example, each camera 102 and the image processing apparatus 103 may be directly connected. Further, for example, each camera 102 is provided with a removable storage device for storing the captured image. The storage device is detached from the camera 102 and connected to the image processing device 103, whereby the captured image is displayed on the image processing device 103. May be acquired.

  Next, a functional configuration example of the image processing apparatus 103 will be described with reference to the block diagram of FIG. In the present embodiment, the image processing in the image processing apparatus 103 is processed by hardware such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) incorporated in the image processing apparatus 103. However, in the present embodiment, each functional unit (each module) illustrated in FIG. 2 is mounted as hardware inside an ASIC or FPGA.

  The data receiving unit 201 acquires captured images captured by the respective cameras 102 and stores the acquired captured images in the recording unit 202. It is assumed that identification information unique to the camera 102 that captured the captured image is attached to the captured image. The recording unit 202 is configured by, for example, a hard disk, an SSD (solid state drive), or a combination thereof. Then, the data reception unit 201 captures images necessary for the generation processing of a virtual viewpoint image to be generated from the group of captured images stored in the recording unit 202 according to the virtual viewpoint position input by user designation or the like. An image group is read from the recording unit 202. Then, the data receiving unit 201 outputs the captured image group read from the recording unit 202 to the background generation unit 203, the foreground separation unit 205, the foreground separation unit 206, and the generation unit 209.

  In the background buffer 204, for each camera 102, identification information unique to the camera 102 is associated with a background image (hereinafter referred to as a registered background image) generated from a captured image captured by the camera 102 in the past. Stored. In the present embodiment, the background image is an image obtained by removing an image (foreground image) of a predetermined subject such as a person from the captured image. For example, when a soccer game on the stadium 101 is photographed by the camera 102, an image of a player or a ball becomes a foreground image, and an image of the ground of the stadium 101 that does not include a player or a ball becomes a background image.

  The background generation unit 203 reads from the background buffer 204 the registered background image stored in the background buffer 204 in association with the same identification information as the identification information attached to the captured image input from the data reception unit 201. The background generation unit 203 generates a background image representing the background in the captured image based on the captured image corresponding to the same identification information and the registered background image, and is stored in the background buffer 204 in association with the identification information. The registered background image is updated using the generated background image. Various methods can be applied to the method for generating the background image representing the background of the captured image based on the captured image and the registered background image. For example, the background generation unit 203 uses a difference image between the captured image and the registered background image (an image obtained by calculating a difference in pixel values at the same pixel position between the captured image and the registered background image), and uses the background in the captured image. A background image may be generated. A background image generation method using a mixed Gaussian model may be used. The mixed Gaussian model is a well-known method and will not be described in detail. Then, the background generation unit 203 outputs the generated background image to the foreground separation unit 205 and the foreground separation unit 206.

  The foreground separation unit 205 determines the difference between the first feature amount for each pixel of the captured image input from the data reception unit 201 and the background image generated by the background generation unit 203 as a background image representing the background in the captured image. Based on this, the foreground area in the captured image is estimated. In the present embodiment, the foreground area is an area where a predetermined subject such as a person in the captured image is located. Note that the foreground area is not limited to an area corresponding to a person, and may be an area corresponding to a moving object or an area corresponding to a subject specified in advance. Processing performed by the foreground separation unit 205 will be described with reference to FIG. The foreground separation unit 205 uses color information as the first feature amount. In the present embodiment, the first feature amount for a certain pixel is determined according to the pixel value of the pixel.

  In step S301, the foreground separation unit 205 generates a difference image based on color information between the captured image (input image) and the background image. In the difference image generated in step S301, the pixel value at the pixel position (x, y) in the difference image has the color information at the pixel position (x, y) in the captured image and the pixel position (x in the background image). , Y) is an image representing a difference from the color information. For example, the pixel values of R, G, B at the pixel position (x, y) in the captured image are PR, PG, PB, and R, G, B at the pixel position (x, y) in the background image The pixel values are PR ′, PG ′, and PB ′. At this time, the pixel value Δ at the pixel position (x, y) in the difference image is determined by calculating Δ = | PR−PR ′ | + | PG−PG ′ | + | PB−PB ′ |. . Of course, the method of obtaining the difference image for the color information between the captured image and the background image is not limited to such a method. For example, the color information may be RGB, may be each color component in the YUV color space or other color space, and is not limited to a specific color component.

  In step S302, the foreground separation unit 205 performs pixel value smoothing processing on the difference image. By performing the smoothing process, pixel loss and noise are removed. For smoothing, there is a simple average process as the simplest one, but in order to improve the performance, for example, a Gaussian filter, a bilateral filter, etc. are known, and any filter may be used.

  In step S303, the foreground separation unit 205 determines whether or not the pixel value of each pixel in the difference image subjected to the smoothing process in step S302 is greater than a threshold value. As a result of this determination, the foreground separation unit 205 determines, for a pixel whose pixel value is larger than the threshold value in the difference image, a bit value representing “a pixel constituting the foreground region (foreground pixel)” for the pixel in step S304. Assign “1”. On the other hand, the foreground separation unit 205 assigns a bit value “0” representing “a pixel constituting the background region (background pixel)” to the pixel having a pixel value equal to or smaller than the threshold in the difference image in step S305. . By such processing, a binary image having a bit value corresponding to each pixel of the difference image as a pixel value can be generated. When the pixel value at the pixel position (x, y) in the binary image is “1”, this indicates that the pixel at the pixel position (x, y) in the captured image is a pixel estimated as the foreground pixel. On the other hand, when the pixel value at the pixel position (x, y) in the binary image is “0”, the pixel at the pixel position (x, y) in the captured image is a pixel estimated as the background pixel. Show. As described above, the foreground separation unit 205 generates a binary image based on the difference in color information between the captured image and the background image as information representing the estimation result of the foreground region on the captured image.

  Note that in the binary image generated in steps S303 to S305, in actuality, a small pixel dropout occurs in the area corresponding to the foreground, or in fact, there is fine noise in the area corresponding to the background. Occur. For this reason, in the next step S306, the foreground separation unit 205 performs the smoothing process as described above on the binary image, thereby removing missing pixels and noise. The smoothing process may be performed on the entire binary image, or may be performed on a pixel region having a pixel value “1” and its vicinity.

  Then, the foreground separation unit 205 sends the binary image (first binary image) generated by performing the processing according to the flowchart of FIG. 3A to the subsequent foreground region adjustment unit 207. The foreground separation unit 205 performs processing according to the flowchart of FIG. 3A every time a captured image is input from the data reception unit 201.

  The foreground separation unit 206 is a second feature amount difference for each pixel between the captured image input from the data reception unit 201 and the background image generated by the background generation unit 203 as a background image representing the background area in the captured image. Based on the above, the foreground region in the captured image is estimated. Processing performed by the foreground separation unit 206 will be described with reference to FIG. 3B showing a flowchart of the processing. The foreground separation unit 206 uses texture information as the second feature amount. That is, in the present embodiment, the second feature amount for a certain pixel is determined according to the pixel value of the pixel and the pixel values of pixels around the pixel.

  In step S307, the foreground separation unit 206 generates a difference image based on texture information between the captured image (input image) and the background image. The pixel value at the pixel position (x, y) in the difference image generated in step S307 is obtained as follows, for example. An image area centered on the pixel position (x, y) in the captured image (for example, an image area of 3 × 3 pixels) and an image area centered on the pixel position (x, y) in the background image (for example, 3 × A difference (absolute value) of luminance values between pixels at the same pixel position between the three pixel image areas) and the total value S of the obtained differences at the pixel position (x, y) in the difference image The pixel value. The luminance value of the pixel is an L component value in the Lab color space. If the pixel value of the pixel is in the RGB color space, the L component value can be obtained by a known method from the R component pixel value, the G component pixel value, and the B component pixel value. The luminance value of the pixel is not limited to the L component, and may be a Y component value such as YUV or YCbCr.

  In addition, for example, HOG (Histograms of Oriented Gradients) feature quantity, LBP (Local Binary Pattern) feature quantity, or the like may be used as the feature quantity based on texture information.

  In step S308, the foreground separation unit 206 performs pixel value smoothing processing on the difference image in the same manner as in step S302. In step S309, the foreground separation unit 206 determines whether or not the pixel value of the pixel in the difference image subjected to the smoothing process in step S308 is greater than a threshold value. As a result of this determination, the foreground separation unit 206 determines, for the pixel whose pixel value is larger than the threshold value in the difference image, a bit value representing “a pixel constituting the foreground region (foreground pixel)” for the pixel in step S310. Assign “1”. On the other hand, the foreground separation unit 206 assigns a bit value “0” representing “a pixel forming the background region (background pixel)” to the pixel having a pixel value equal to or smaller than the threshold in the difference image in step S311. . By such processing, a binary image having a bit value corresponding to each pixel of the difference image as a pixel value can be generated in the same manner as in steps S304 and S305 described above. As described above, the foreground separation unit 206 generates a binary image based on the difference in texture information between the captured image and the background image as information representing the estimation result of the foreground region on the captured image.

  In step S312, the foreground separation unit 206 performs the above-described smoothing process on the binary image in the same manner as in step S306, thereby removing pixel omissions and noise.

  The foreground separation unit 206 transmits the binary image (second binary image) generated by performing the processing according to the flowchart of FIG. 3B to the foreground region adjustment unit 207 in the subsequent stage. Note that the foreground separation unit 206 performs processing according to the flowchart of FIG. 3B every time a captured image is input from the data reception unit 201.

  An example of a binary image generated according to the flowchart of FIG. 3A is shown in FIG. As shown in FIG. 5A, the foreground (person 104) is detected as a human shape on the binary image, but here, as an example, a part of the detected foreground area has a large omission. And When the foreground area is estimated based on the difference in color information, such omission may occur particularly when the foreground and background colors are close.

  An example of a binary image generated according to the flowchart of FIG. 3B is shown in FIG. As shown in FIG. 5B, the boundary portion of the detected foreground (person 104) region is not smooth on the binary image, and a small omission is seen inside the foreground region. When the foreground region is estimated based on the difference in texture information, the boundary between the foreground and the background tends to be rough as described above.

  The foreground area adjustment unit 207 adjusts the foreground area in the second binary image generated by the foreground separation unit 206. An example of the foreground region adjustment process in the second binary image will be described with reference to the flowchart of FIG.

  In step S401, the foreground area adjustment unit 207 performs a labeling process on each of the first binary image and the second binary image, thereby each of the first binary image and the second binary image. A label is assigned to the foreground region (a pixel region having a pixel value “1”). In the labeling process, a group of connected pixels having the same pixel value “1” is detected as one foreground area (area corresponding to the same subject), and a label (number) is added to each foreground area.

  For the sake of simplicity, the same label number is assigned to the foreground region of interest in the first binary image and the foreground region closest to the foreground region of interest among the foreground region group in the second binary image. It shall be added. However, if the correspondence between the foreground region of interest in the first binary image and the foreground region closest to the foreground region of interest in the foreground region group in the second binary image can be derived, the respective binary values It is not limited to adding the same label number to the foreground region adjacent in the image.

  Then, the processing in step S403 is performed for each label number in the first binary image (second binary image) (repeated by the number corresponding to the number of labels), so that the foreground area corresponding to each label number is processed. The process of step S403 is performed.

  In step S403, the foreground area adjustment unit 207 selects one unselected label number as the selected label number, and performs contraction processing on the foreground area to which the selected label number is added in the second binary image. Thus, the foreground area is adjusted. For the shrinking process, a process called morphological operation is used. Morphological operations are generally well known and will not be described. Further, the contraction amount at this time is such that when the first binary image and the second binary image are overlapped, the foreground area in the second binary image is substantially inside the foreground area in the first binary image. It is determined in advance as a constant value. FIG. 5C shows an example of the result of the contraction process for the foreground region in FIG.

  Another example of the foreground area adjustment process in the second binary image will be described with reference to the flowchart in FIG. In step S405, the foreground area adjustment unit 207 performs a labeling process on each of the first binary image and the second binary image, similarly to step S401 described above.

  By performing the processing in steps S407 to S411 for each label number in the first binary image (second binary image) (repeating the number of times according to the number of labels), the foreground region corresponding to each label number The processes of steps S407 to S411 are performed.

  In step S407, the foreground area adjustment unit 207 selects one unselected label number as the selected label number, and out of the foreground area group in the first binary image, the area S1 of the foreground area corresponding to the selected label number. (For example, the number of pixels included in the foreground area) is specified.

  In step S408, the foreground area adjustment unit 207 selects the area S2 of the foreground area corresponding to the selected label number in the foreground area group in the second binary image (for example, the number of pixels included in the foreground area). Is identified.

  In step S409, the foreground area adjustment unit 207 obtains the ratio of the area S2 to the area S1 (S2 / S1). In step S410, the foreground area adjustment unit 207 determines whether or not a condition of S2 / S1 <θ is satisfied. Here, θ is a predetermined threshold value. As a result of this determination, if S2 / S1 <θ, the process returns to step S407. On the other hand, if S2 / S1 ≧ θ, the process proceeds to step S411.

  In step S411, the foreground area adjustment unit 207 performs the contraction process on the foreground area to which the selection label number is added in the second binary image in the same manner as in step S403 described above, thereby reducing the foreground area. adjust. The adjustment amount (shrinkage amount) may be determined in advance, or may be determined based on, for example, the value of S1 and the value of S2.

  With the above processing, the area of the foreground region in the second binary image is equal to or less than a certain ratio with respect to the area of the foreground region in the first binary image. Therefore, as a result, when the first binary image and the second binary image are overlaid, the boundary of the foreground region in the second binary image is substantially inside the foreground region in the first binary image. Can be expected.

  Furthermore, as another example of the foreground area adjustment method in the foreground area adjustment unit 207, there is a method of comparing the lengths of the boundary portions of the foreground area. In step S407, the length of the boundary portion of the foreground region in the first binary image is obtained as S11, and in step S408, the length of the boundary portion of the foreground region in the second binary image is obtained as S12. In step S409 and subsequent steps, processing is performed using S11 and S12, respectively, instead of S1 and S2.

  The foreground area determination unit 208 includes a foreground area (first area) in the first binary image and a foreground area (second area) in the second binary image that has been adjusted (shrinked) by the foreground area adjustment unit 207. Based on the above, the foreground region in the captured image is determined. The foreground area determination processing by the foreground area determination unit 208 will be described with reference to the flowchart of FIG.

  In step S313, the foreground region determination unit 208 performs an OR operation on the pixel values for each pixel at the same position in the first binary image and the second binary image. When the pixel value at the pixel position (x, y) in the binary image (third binary image) obtained by performing such an OR operation is “1”, the pixel position (x , Y) indicates that the pixel estimated as the foreground pixel. On the other hand, when the pixel value at the pixel position (x, y) in the third binary image is “0”, the pixel at the pixel position (x, y) in the captured image is a pixel estimated as the background pixel. Indicates that there is. That is, the region having a pixel value “1” in the third binary image is the sum of the foreground region in the first binary image and the foreground region in the second binary image. In other words, the foreground area determined by the foreground area determination unit 208 is configured by pixels included in at least one of the foreground area specified by the foreground separation unit 205 and the foreground area specified by the foreground separation unit 206. . Note that the OR operation is not limited to the entire region of the first binary image and the second binary image, and for example, the OR operation may be performed for each foreground region having the same label number.

  FIG. 5D shows a foreground area obtained by the foreground area determination unit 208 using the foreground area in FIG. 5A and the foreground area in FIG. As shown in FIG. 5D, compared to the foreground area of FIG. 5A, the missing foreground area is filled and the quality of the foreground shape is improved. Compared with the foreground area in FIG. 5B, the boundary portion of the foreground area in FIG. 5B is adjusted to be inside the foreground area in FIG. And the quality of the foreground shape is improved.

  The generation unit 209 uses the captured image input from the data reception unit 201, the background image stored in the background buffer 204, and the third binary image generated by the foreground region determination unit 208. Generate a viewpoint image. As a method for generating a virtual viewpoint image, for example, a method such as VisualHull is known. For example, the generation unit 209 reads the background image necessary for generating the virtual viewpoint image corresponding to the viewpoint designated by the user from the background buffer 204, and projects the read background image on the two-dimensional plane, thereby A background at a virtual viewpoint is generated on a plane. In addition, the generation unit 209 has a pixel value in the third binary image generated by the foreground region determination unit 208 for the captured image input from the data reception unit 201 on the captured image corresponding to the pixel group having the pixel value “1”. The area of the pixel group is specified as the foreground area. Then, the generation unit 209 extracts the foreground area specified for the captured image from each captured image input from the data reception unit 201, and based on the extracted foreground area, a three-dimensional model of the foreground (for example, the person 104). Is generated. Then, the generation unit 209 generates a virtual viewpoint image on the two-dimensional plane by projecting the generated three-dimensional model onto the two-dimensional plane according to the viewpoint designated by the user. Since VisualHull is a well-known technique, a detailed description thereof will be omitted.

  The generation unit 209 outputs the generated virtual viewpoint image, but the output destination is not limited to a specific output destination. For example, the image may be displayed on a display device connected to the image processing apparatus 103 or may be recorded in a storage device. Moreover, you may transmit to an external terminal via a network.

  As described above, according to the present embodiment, the quality of the foreground shape is improved by performing the foreground separation using a plurality of foreground separation methods having different characteristics and performing the adjustment after adjusting the result. Can do. That is, the accuracy of specifying the foreground area in the input image can be improved. As a result, the image quality of the generated virtual viewpoint image can be improved.

[Second Embodiment]
In the first embodiment, the texture information of the image is used as the second feature amount, but the luminance information of the image is used in the present embodiment. That is, in the present embodiment, the second feature amount for a certain pixel is determined according to the pixel value of the pixel. In the following, differences from the first embodiment will be described mainly, and unless otherwise noted, the same as the first embodiment.

  A functional configuration example of the image processing apparatus 103 will be described with reference to the block diagram of FIG. In FIG. 6, the same functional units as those shown in FIG. 2 are denoted by the same reference numerals, and description thereof will be omitted. Also in this embodiment, as in the first embodiment, the image processing in the image processing apparatus 103 is processed by hardware such as an ASIC or FPGA built in the image processing apparatus 103. However, in the present embodiment, each functional unit (each module) illustrated in FIG. 6 is mounted as hardware inside an ASIC or FPGA.

  The foreground separation unit 606 performs the imaging based on the difference in luminance information between the captured image input from the data reception unit 201 and the background image generated by the background generation unit 203 as a background image representing the background area in the captured image. Estimate the foreground region in the image. Processing performed by the foreground separation unit 606 will be described with reference to FIG. 7 showing a flowchart of the processing. In FIG. 7, the same processing steps as those in FIG. 3B are denoted by the same step numbers, and description of the processing steps is omitted.

  In step S707, the foreground separation unit 606 generates a difference image based on luminance information between the captured image (input image) and the background image. In the difference image generated in step S707, the pixel value at the pixel position (x, y) in the difference image has the luminance value at the pixel position (x, y) in the captured image and the pixel position (x in the background image). , Y) is an image representing a difference between the luminance value and y).

  In step S709, the foreground separation unit 606 determines whether the pixel value of the pixel in the difference image subjected to the smoothing process in step S308 is greater than a threshold value. As a result of this determination, the foreground separation unit 606, for a pixel whose pixel value is larger than the threshold value in the difference image, for the pixel in step S710, represents a bit value representing “a pixel constituting the foreground region (foreground pixel)”. Assign “1”. On the other hand, the foreground separation unit 606 assigns a bit value “0” representing “a pixel constituting the background region (background pixel)” to the pixel having a pixel value equal to or smaller than the threshold in the difference image in step S711. . By such processing, a binary image having a bit value corresponding to each pixel of the difference image as a pixel value can be generated.

  According to this embodiment, the area of the foreground region in the second binary image is equal to or less than a certain ratio with respect to the area of the foreground region in the first binary image. Therefore, as a result, when the first binary image and the second binary image are overlapped, it can be expected that the boundary of the foreground region in the second binary image is almost inside.

<Modification>
In each of the above embodiments, the foreground area in the second binary image is contracted in view of the size of the foreground area in the first binary image. However, depending on the situation, the foreground in the second binary image is reduced. In view of the size of the area, the foreground area in the first binary image may be contracted or expanded.

  In the first and second embodiments, foreground separation is performed to generate a virtual viewpoint image, but the foreground separation described in the first and second embodiments is performed in image processing in other applications. You may do it.

  In each of the above embodiments, the case where the same image processing apparatus 103 executes the foreground separation process and the virtual viewpoint image generation process has been described, but the present invention is not limited thereto. For example, there may be a plurality of image processing apparatuses 103 corresponding to the plurality of cameras 102 in the system, and each image processing apparatus may perform foreground separation processing on the captured image of the corresponding camera 102. Then, each of the plurality of image processing apparatuses 103 may output the result of the foreground separation process performed by the own apparatus to a generation apparatus that performs a virtual viewpoint image generation process.

  Moreover, you may combine suitably a part or all of said embodiment containing this modification, and you may selectively use a part or all of said embodiment containing this modification.

[Other Embodiments]
In the first and second embodiments, the functional units illustrated in FIGS. 2 and 6 have been described as being configured by hardware. However, functional units other than the recording unit 202, the background buffer 204, and the data receiving unit 201 may be implemented by software (computer program). In such a case, a computer device having a memory functioning as the recording unit 202 and the background buffer 204, an interface (I / F) functioning as the data receiving unit 201, and capable of executing the software is an image processing device 103. Can be applied to.

  A hardware configuration example of a computer apparatus applicable to the image processing apparatus 103 will be described with reference to the block diagram of FIG. Note that the hardware configuration shown in FIG. 8 is merely an example of the hardware configuration of a computer device applicable to the image processing apparatus 103.

  The CPU 801 executes processing using computer programs and data stored in the RAM 802 and the ROM 803. As a result, the CPU 801 controls the operation of the entire computer apparatus and executes or controls each process described above as being performed by the image processing apparatus 103.

  The RAM 802 has an area for storing computer programs and data loaded from the ROM 803 and the external storage device 806 and data received from the outside via the I / F 807 (for example, a captured image group). Further, the RAM 802 has a work area used when the CPU 801 executes various processes. As described above, the RAM 802 can appropriately provide various areas. The ROM 803 stores computer programs and data that do not require rewriting.

  The operation unit 804 is configured by a user interface such as a keyboard and a mouse, and can input various instructions to the CPU 801 by a user operation. For example, the user can designate the position of the virtual viewpoint by operating the operation unit 804.

  The display device 805 is configured by a CRT, a liquid crystal screen, or the like, and can display a processing result by the CPU 801 using an image, text, or the like. For example, the display device 805 can display a GUI (graphical user interface) for designating the position of the virtual viewpoint or the virtual viewpoint image generated by the generation unit 209. Note that the operation unit 804 and the display device 805 may be integrated to form a touch panel screen.

  The external storage device 806 is a mass information storage device represented by a hard disk drive device. The external storage device 806 includes an OS (operating system) and computer programs and data for causing the CPU 801 to implement the functions of the functional units other than the recording unit 202, the background buffer 204, and the data receiving unit 201 in FIGS. Is saved. The data stored in the external storage device 806 includes data handled as known information in the above description, such as the above-described threshold value, a lookup table for converting RGB into Lab or YUV, and the like. include. Computer programs and data stored in the external storage device 806 are appropriately loaded into the RAM 802 under the control of the CPU 801 and are processed by the CPU 801. Note that the recording unit 202 and the background buffer 204 described above can be mounted using a RAM 802 or an external storage device 806.

  The I / F 807 functions as the data receiving unit 201 described above, and functions as an interface for receiving images captured by the respective cameras 102. In addition, a control signal for each camera 102 is transmitted from the computer device to each camera 102 via the I / F 807. The CPU 801, RAM 802, ROM 803, operation unit 804, display device 805, external storage device 806, and I / F 807 are all connected to the bus 808.

  In the present invention, a program that realizes one or more functions of the above-described embodiments is supplied to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read the program. It can also be realized by processing to be executed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  201: Data reception unit 202: Recording unit 203: Background generation unit 204: Background buffer 205: Foreground separation unit 206: Foreground separation unit 207: Foreground region adjustment unit 208: Foreground region determination unit 209: Generation unit

Claims (12)

First specifying means for specifying a first region in the input image using a difference based on a first feature amount between the input image and the background image;
Using a difference based on a second feature amount between the input image and the background image, second specifying means for specifying a second region in the input image;
Determination means for determining a foreground area corresponding to a predetermined subject in the input image based on the first area specified by the first specifying means and the second area specified by the second specifying means. An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the first specifying unit specifies, as the first region, a region where a difference based on color information between the input image and the background image exceeds a threshold value.   The image processing apparatus according to claim 1, wherein the second specifying unit specifies, as the second region, a region where a difference based on texture information between the input image and the background image exceeds a threshold value.   The image processing apparatus according to claim 1, wherein the second specifying unit specifies, as the second region, a region in which a difference based on luminance information between the input image and the background image exceeds a threshold value.   The determining means is at least one of the first area specified by the first specifying means and the contracted second area obtained by contracting the second area specified by the second specifying means. 5. The image processing apparatus according to claim 1, wherein a region included in the input image is determined as a foreground region in the input image.   The second region that has been contracted is specified by the second specifying unit and the area of the first region specified by the first specifying unit and the second specifying unit specified by the second specifying unit. The image processing apparatus according to claim 5, wherein the image processing device is a region contracted based on a ratio with the area of the second region.   The contracted second area is the second area specified by the second specifying means, the length of the boundary portion of the first area specified by the first specifying means, and the second specifying. The image processing apparatus according to claim 5, wherein the image processing device is a region contracted based on a ratio with a length of a boundary portion of the second region specified by the means.   The image processing apparatus according to claim 1, further comprising an acquisition unit configured to acquire a captured image obtained by each of a plurality of imaging apparatuses connected in a ring shape as the input image.   The image processing apparatus according to claim 1, further comprising a generation unit configured to generate a virtual viewpoint image based on the foreground region determined by the determination unit. A plurality of imaging devices connected in a ring shape;
A system comprising: the image processing apparatus according to claim 8. An image processing method performed by an image processing apparatus,
A first specifying step in which a first specifying unit of the image processing device specifies a first region in the input image using a difference based on a first feature amount between the input image and a background image;
A second specifying step in which a second specifying unit of the image processing apparatus specifies a second region in the input image using a difference based on a second feature amount between the input image and the background image; When,
A determination unit of the image processing device applies a predetermined subject in the input image based on the first region specified in the first specifying step and the second region specified in the second specifying step. And a determining step for determining a corresponding foreground region.   The computer program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 9.