JP2021056936A - Image processing device, image processing method, and program - Google Patents

Abstract

To allow for obtaining a high-precision foreground image even when it is difficult to accurately extract edge information because a boundary between a foreground object and a background is ambiguous.SOLUTION: In a provisional foreground image representing a foreground area in a captured image, a contour area that is likely to contain errors is detected. Pixels constituting the detected contour area are then corrected based on whether colors of pixels of the captured image corresponding to these pixels are similar to colors of a background or not.SELECTED DRAWING: Figure 4

Description

The present invention relates to a technique for generating a foreground image showing the shape of an object from a captured image.

Conventionally, there is a background subtraction method as a method of extracting a foreground region corresponding to an object (subject) from a captured image. The background subtraction method corresponds to an object based on the difference between each pixel value of the captured image in which the object as the foreground is shown and each pixel value of the background-only image (background image) in which the object is not shown. Extract the foreground area. Here, in the vicinity of the boundary between the object and the background, the difference between the pixel value in the captured image and the pixel value in the background image may become small. In this case, the part that should originally be the background may be mistakenly extracted as the foreground area corresponding to the object. That is, in the conventional background subtraction method, it may not be possible to accurately extract the foreground region along the contour of the object. In this regard, Patent Document 1 discloses a technique of extracting a foreground region from a captured image and then shaping the extracted foreground region based on the edge information of the object.

Japanese Unexamined Patent Publication No. 10-23452

However, the method of Patent Document 1 has a problem that the shape of an object cannot be properly shaped if the edge information itself contains an error.

Therefore, the technique according to the present disclosure aims to obtain an accurate foreground image even when the boundary between the foreground object and the background is ambiguous and it is difficult to accurately extract the edge information.

The image processing device according to the present disclosure is an image processing device that generates a foreground image showing the shape of an object from a captured image, and corresponds to an acquisition means for acquiring the captured image and the shape of the object from the captured image. The extraction means for extracting the foreground region to generate a foreground image, the detection means for detecting the contour region of the object from the foreground image, and the pixels constituting the contour region in the foreground image correspond to the pixels. It is characterized by having a correction means for correcting based on whether or not the color of a pixel in a captured image is similar to the color of a background.

According to the technique of the present disclosure, it is possible to obtain an accurate foreground image even when the boundary between the foreground object and the background is ambiguous and it is difficult to accurately extract the edge information.

The figure explaining the outline of the process of generating a foreground image Diagram showing an example of the configuration of an image processing system that generates virtual viewpoint video Functional block diagram showing the internal configuration of the camera adapter Functional block diagram showing details of the image processing unit according to the first embodiment A flowchart showing the flow of the foreground image generation process according to the first embodiment. Explanatory drawing of process for determining whether or not a pixel of interest is a pixel of a contour region The figure explaining the effect of Embodiment 1. Functional block diagram showing details of the image processing unit according to the second embodiment A flowchart showing the flow of the foreground image generation process according to the second embodiment. The figure explaining the effect of Embodiment 2.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the following embodiments do not limit the present invention, and not all combinations of features described in the present embodiment are essential for the means for solving the present invention. The same configuration will be described with the same reference numerals.

[Embodiment 1]
In the present embodiment, after the foreground region corresponding to the object is extracted from the captured image, the contour region (boundary portion with the background) of the object is detected. Then, by correcting the color in the detected contour region based on the difference from the background color, a foreground image showing the shape of the object is generated. In the present embodiment, the foreground image is a binary image in which the foreground area is represented by “1” and the other background areas are represented by “0”, but a multivalued image in which a value indicating the certainty of the foreground region is further added. May be.

Further, in the present embodiment, the case where the foreground image of the object generated from the captured image is used for generating the virtual viewpoint image will be described as an example. That is, assume a use case in which the three-dimensional shape data is generated from the foreground image of an object, and the virtual viewpoint image including the object is generated based on the virtual viewpoint information.

(Outline of this embodiment)
The outline of the process for generating the foreground image in the present embodiment will be described with reference to FIG. First, an image (hereinafter, referred to as “target image”) 10 to be extracted in the foreground region is captured by the camera 11 which is an imaging device. The target image 10 shows a person object 12 as a foreground and a field 13 as a background. Further, from the same imaging viewpoint as the target image 10, an image (hereinafter, referred to as “background image”) 14 in which the background field 13 is captured without the person object 12 is imaged. Next, the target image 10 and the background image 14 are compared with each other, and the pixels having a large difference in pixel values are regarded as the constituent pixels of the person object 12, so that only the foreground region is extracted from the target image 10. To do. As a result, a foreground image representing the silhouette of the person object 12 is acquired. The foreground image showing the shape of the object is also called a silhouette image or a mask image, but in the present specification, the "foreground image" is used. Image 15 in FIG. 1 shows an ideal foreground image. However, in reality, the contour of the foreground region may become unclear due to the influence of lens aberration, image resolution, and the like. This is because pixels having various pixel values, from those close to the background color to those close to the object color, are mixed in the vicinity of the boundary between the object and the background on the target image 10. If the pixels near the boundary between the object and the background are in such a state, the pixels of the object and the pixels of the background cannot be correctly distinguished, resulting in a low-precision foreground region that does not follow the contour of the object. .. Image 15'in FIG. 1 shows an inaccurate (including error) foreground image that does not follow the contours of the object. Therefore, in the present embodiment, the contour portion 16 that is likely to contain an error is detected in the foreground image generated from the captured image. Then, when the color of the corresponding pixel in the captured image is similar to the color of the background, the pixel value of the pixel constituting the detected contour portion 16 is changed to the pixel value indicating the background. In this way, the foreground image in which the error in the contour portion is corrected is output as the final foreground image. The above is the outline of the foreground image generation processing performed in the present embodiment. Hereinafter, a specific configuration of the present embodiment will be described.

(System configuration)
FIG. 2 is a diagram showing an example of the configuration of an image processing system that generates a virtual viewpoint image. The image processing system 100 includes imaging modules 110a to 110z, a database (DB) 250, a server 270, a control device 300, a switching hub 180, and an end user terminal 190. That is, the image processing system 100 has three functional domains, that is, a video collection domain, a data storage domain, and a video generation domain. The video acquisition domain includes the photographing modules 110a to 110z, the data storage domain includes the DB 250 and the server 270, and the video generation domain includes the control device 300 and the end user terminal 190.

The control device 300 manages the operating state and controls the parameter setting for each block constituting the image processing system 100 through the network. Here, the network may be GbE (Gigabit Ethernet) or 10 GbE conforming to the IEEE standard (registered trademark), or may be configured by combining an interconnect Infiniband, an industrial local area network, or the like. Further, the network is not limited to these, and other types of networks may be used.

First, an operation of transmitting 26 sets of captured images of the photographing modules 110a to 110z from the photographing module 110z to the server 270 will be described. The photographing modules 110a to 110z each have one camera 112a to 112z. In the following, the 26 sets of systems from the photographing modules 110a to 110z may not be distinguished and may be simply referred to as “photographing module 110”. Similarly, the devices in each photographing module 110 may be described as "camera 112" and "camera adapter 120". The number of shooting modules 110 is 26 sets, but this is just an example and is not limited to this.

The photographing modules 110a to 110z are connected by a daisy chain. This connection form has the effect of reducing the number of connection cables and labor saving in wiring work in increasing the resolution of captured images to 4K or 8K and increasing the capacity of image data due to the increase in frame rate. The connection form is arbitrary. For example, a star-type network configuration in which the photographing modules 110a to 110z are connected to the switching hub 180 and data is transmitted / received between the photographing modules 110 via the switching hub 180 may be used.

In the present embodiment, each photographing module 110 is composed of a camera 112 and a camera adapter 120, but is not limited thereto. For example, it may have a microphone, a pan head, and an external sensor. Further, in the present embodiment, the camera 112 and the camera adapter 120 are separated from each other, but they may be integrated in the same housing. The captured image obtained by the camera 112a in the photographing module 110a is transmitted to the camera adapter 120b of the photographing module 110b after the image processing described later is performed on the camera adapter 120a. Similarly, the photographing module 110b transmits the captured image obtained by the camera 112b to the photographing module 110c together with the captured image acquired from the photographing module 110a. By continuing such an operation, 26 sets of captured images are transmitted from the photographing module 110z to the switching hub 180, and then transmitted to the server 270.

In this embodiment, it is assumed that the foreground image is generated in each camera adapter 120. However, the present invention is not limited to such an aspect, and the server 270 that has received 26 sets of captured images may be configured to generate a foreground image corresponding to each captured image.

(Camera adapter configuration)
Next, the details of the camera adapter 120 will be described. FIG. 3 is a functional block diagram showing the internal configuration of the camera adapter 120. The camera adapter 120 includes a network adapter 121, a transmission unit 122, an image processing unit 123, and a camera control unit 124.

The network adapter 121 performs data communication with another camera adapter 120, a server 270, and a control device 300. Further, for example, in accordance with the Ordinary Clock of the IEEE1588 standard, the time stamp of the data transmitted / received to / from the server 270 is saved and the time is synchronized with the server 270. In addition, time synchronization with a time server may be realized by another EtherAVB standard or an original protocol. In the present embodiment, a NIC (Network Interface Card) is used as the network adapter 121, but the present embodiment is not limited to this.

The transmission unit 122 controls the transmission of data to the switching hub 180 and the like via the network adapter 121. The transmission unit 122 has a function of performing compression by applying a predetermined compression method, compression rate, and frame rate to the transmitted / received data, and a function of decompressing the compressed data. It also has a function of determining the routing destination of the received data and the data processed by the image processing unit 123, and a function of transmitting the data to the determined routing destination. It also has a function of creating a message for transferring image data to another camera adapter 120 or server 270. The message contains meta information of image data. This meta information includes a time code or sequence number at the time of sampling for image capture, a data type, an identifier of the camera 112, and the like. The image data to be transmitted may be compressed. Further, a message is received from another camera adapter 120, and data information fragmented to a packet size defined by a transmission protocol is restored to image data according to the data type included in the message.

The image processing unit 123 performs a process of generating a foreground image showing the shape of the object based on the image data captured by the camera 112 under the control of the camera control unit 124. It also performs processing such as dynamic calibration. By generating the foreground image by each of the plurality of camera adapters 120, the load in the image processing system 100 can be distributed. Dynamic calibration is a calibration performed during shooting, and is a color correction process for suppressing color variation between cameras and a blur correction process for stabilizing the position of an image against blur caused by camera vibration. (Electronic vibration isolation treatment) etc. are included.

The camera control unit 124 is connected to the camera 112 to control the camera 112, acquire a captured image, provide a synchronization signal, set a time, and the like. For control of the camera 112, for example, setting and reference of shooting parameters (number of pixels, color depth, frame rate, white balance setting, etc.), state information of the camera 112 (shooting, stopped, synchronizing, error, etc.) ), Start and stop of shooting, focus adjustment, etc.

(Details of foreground image generation processing)
Subsequently, the foreground image generation process in the camera adapter 120 according to the present embodiment will be described in detail with reference to the functional block diagram shown in FIG. 4 and the flowchart shown in FIG. As shown in FIG. 4, the image processing unit 123 in the camera adapter 120 has five functional units related to the generation of the foreground image. Specifically, it has an image acquisition unit 401, a foreground area extraction unit 402, a contour area detection unit 403, a similarity determination unit 404, and a pixel value correction unit 405. Further, the series of processes shown in the flowchart of FIG. 5 is realized by the CPU (not shown) in the camera adapter 120 deploying a predetermined program in a work memory (RAM) (not shown) and executing the program. However, it is not necessary that all of the processes shown below are executed by one CPU, and a part or all of the processes may be performed by one or a plurality of processing circuits other than the CPU. The captured image input to the camera adapter 120 is a moving image composed of a plurality of frames. For example, when moving image data captured at a frame rate of 60 fps is input for 10 seconds, serial number images of all 600 frames are input as a stream, and the following processing is performed in order in frame units. .. However, the input captured image is not limited to the moving image, and may be a still image. In the following description, the symbol "S" represents a step.

In S501, the image acquisition unit 601 acquires the data of the frame (hereinafter, referred to as “target image”) to be extracted in the foreground region and the background image. Here, if the imaging scene is a soccer match, the background image data is captured under the same conditions as the target image at a stadium where there are no players, balls, or other objects, and the HDD is not shown. You can read what you have stored in. In this case, the same conditions include not only physical conditions such as the position, attitude, focal length, and optical center of the camera 112, but also environmental conditions such as weather and time zone. In addition, instead of reading the data of the background image captured in advance from the HDD or the like and acquiring it, it is created by performing filter processing using an intermediate value filter or an average value filter on a plurality of frames constituting the input moving image. You may. Alternatively, it may be created by performing clustering processing on a plurality of frames. The target image and background image data acquired in this step are output to the foreground region extraction unit 402 and the similarity determination unit 404.

In S502, the foreground region extraction unit 402 extracts the foreground region from the target image using the target image and the background image acquired in S501. Specifically, the pixel values of the corresponding pixels in the target image and the background image are compared, and the pixel positions having different pixel values are specified. Then, by setting the pixel value of the pixel of the coordinates having the same pixel value to "0" and the pixel value of the pixel having the coordinates not having the same pixel value to "1", a binary image showing the foreground region in the target image can be obtained. can get. The pixel values do not have to be exactly the same, and may be regarded as the same as long as they are within a predetermined threshold value (for example, about 5% of the maximum pixel value). The binary image thus obtained is output as a provisional foreground image (hereinafter, referred to as “provisional foreground image”) to the contour region detection unit 403, the similarity determination unit 404, and the pixel value correction unit 405. In the present embodiment, the description is made on the premise that the foreground region is extracted by using the background subtraction method, but for example, the inter-frame subtraction method may be used.

In S503, the contour area detection unit 403 initializes a map (hereinafter, referred to as “contour map”) for detecting the contour area of the object in the provisional foreground image obtained in S502. Specifically, the pixel values of all the pixels of the contour map are set to "0". This contour map is a binary image having the same number of pixels and image size as the provisional foreground image. In the provisional foreground image, "1" is used for the pixels corresponding to the pixels (or pixels that are likely to be) that constitute the contour area indicating the vicinity of the boundary between the object and the background, and "0" is used for the other pixels. A value is given. By the initialization in this step, the value "0" indicating that it is not the contour area is set as the initial value for all the pixels in the contour map. After the initialization process, the process proceeds to S504.

In S504, the contour area detection unit 403 indicates the vicinity of the boundary between the object and the background with respect to the pixel of interest (hereinafter, referred to as “focused pixel”) among the constituent pixels of the provisional foreground image obtained in S502. Determine if it is a pixel. At this time, the pixel of interest is sequentially selected from, for example, the upper left pixel of the foreground image. Here, with reference to FIG. 6, the details of the determination process of whether or not the pixel of interest is a pixel in the contour region will be described.

First, a block 600 having a predetermined size (here, 5 × 5) centered on the coordinates (u ₀ , v _{0) of the pixel of interest is set.} The predetermined size may be determined in advance according to the size of the target image (4K, full HD, etc.) and the size of the object (ratio to the target image). Next, it is determined whether the pixel of interest 601 is a pixel of the contour region based on the pixel value of the pixel of interest 601 in the provisional foreground image and the pixel value of each pixel included in the block 600. Specifically, when the pixel value of the pixel of interest 601 in the provisional foreground image is "1 (= object)" and there is one or more pixels with a pixel value of "0 (= background)" in the block 600. It is determined that the pixel of interest 601 is a pixel in the contour region. Now, it is assumed that the blocks 600 are in the state of 602 to 604, respectively. Each lattice in the blocks 602 to 604 represents one pixel, and the pixel value is A pixel having "0" is shown in black, and a pixel having a pixel value of "1" is shown in white. In the example of FIG. 6, in the case of block 604, the pixel of interest 601 is determined to be a pixel in the contour region. If the above conditions are not satisfied, such as block 602 in which the pixel of interest 601 is the background and block 603 in which the pixel of interest is not in the block 600, the pixel of interest 601 is not a pixel in the contour region. The method for determining whether or not the pixels are in the contour region is not limited to the above method. For example, a contraction image obtained by subjecting a provisional foreground image to a morphology process is generated to generate a provisional foreground image. When the pixel value of the pixel of interest in the above and the pixel value of the pixel corresponding to the pixel of interest in the contracted image are not the same, it may be determined that the pixel is a pixel in the contour region.

In S505, the contour area detection unit 403 updates the pixel value in the contour map for the pixel of interest according to the determination result in S504. That is, when the pixel of interest is determined to be a pixel of the contour region, corresponding to the same coordinates as the coordinates of the target pixel (u _0, v _0), the pixels of the pixel in the edge map E (u _0, v ₀₎ Change the value to "1". Also, if the pixel of interest is determined not to be a pixel of the contour regions, the pixels of the pixel of the coordinates of the target pixel (u _0, v ₀₎ corresponds to the same coordinates as the contour map E (u _0, v ₀₎ Keep the value at “0”.

In S506, it is determined whether or not all the pixels in the provisional foreground image have been processed as the pixels of interest. If there is an unprocessed pixel, the process returns to S504, the next pixel of interest is selected, and processing is continued. On the other hand, if the processing of all the pixels is completed, the contour map that has been updated is output from the contour area detection unit 403 to the pixel value correction unit 405, and the process proceeds to the next processing (S507).

In S507, the similarity determination unit 404 initially sets a map (hereinafter, referred to as “similarity map”) for determining whether or not the color of each pixel constituting the target image is similar to the background color. To become. Specifically, the pixel values of all the pixels of the similarity map are set to "0". This similarity map is a binary image having the same number of pixels and image size as the target image. Among the pixels in the target image, the pixel corresponding to the pixel determined to have a color similar to the background color is given a value of "1", and the other pixels are given a value of "0". By the initialization in this step, the value "0" indicating that the background color is not similar to all the pixels of the similarity map is set as the initial value. After the initialization process, the process proceeds to S508.

In S508, the similarity determination unit 404 determines the similarity between the color of the pixel of interest in the target image and the background color indicated by the background image using a predetermined evaluation value regarding the tint. The "color" here includes information on the so-called three attributes of color (hue, lightness, and saturation). Further, since the meaning and selection method of the pixel of interest are the same as those in S504, the description thereof will be omitted. Hereinafter, the similarity determination in this step will be described in detail.

First, a block centered on the coordinates (u ₀ , v _{0) of the pixel of interest is set.} The block size may be the same as or different from that of S504. _{Next, two evaluation values C F} and C _B for the pixel of interest, which are indexes for determining whether or not the pixel has a color similar to the background color, are obtained. For example, the mean square error (MSE) is used for these two evaluation values C _F and C _{B, and} they are represented by the following equations (1) and (2), respectively.

In the above formula (1) and (2), I (x, y) pixel values of the coordinates (x, y) in the target image, I _F (x, y) coordinates in the interim foreground image (x, y) Represents the pixel value of. Further, k represents a subscript for identifying an RGB3 channel, and B represents a block. Further, n _F is the total number of pixels (pixel value = 1) in the foreground region corresponding to the object in the provisional foreground image included in the block, and n _B is the pixels (pixel value = 0) in the background region not corresponding to the object. Represents the total number of. Here, the evaluation value C _F represents the difference between the pixel value of the pixel of interest and the pixel value of the foreground region included in the block, and the more similar the two, the smaller the value. Further, the evaluation value C _B represents the difference between the pixel value of the pixel of interest and the pixel value of the background region included in the block, and the more similar the two, the smaller the value. In this embodiment, the difference in color is evaluated by MSE (Mean Squared Error), but the evaluation index is not limited to this, and MAE (Mean Absolute Error), RMSE (Root Mean Square Error), etc. May be used. Next, it is determined whether the color of the pixel of interest is similar to the background color based on the obtained two evaluation values C _F and C _B. Specifically, if the evaluation value C _F obtained for the pixel of interest is the evaluation value C _B or more, it is determined that the color is similar to the background color. On the contrary, if the evaluation value C _F obtained for the pixel of interest is less than the evaluation value C _B , it is determined that the color is not similar to the background color. In the present embodiment, when the pixel value of the pixel of interest is closer to the pixel value of the background region than the pixel value of the foreground region included in the block centered on the pixel of interest, the color of the pixel of interest is It was judged to be similar to the background color. However, the method for determining similarity is not limited to this. For example, when the difference between the pixel value of the pixel of interest in the captured image and the pixel value of the background image having the same coordinates as the pixel of interest is small (when it is equal to or less than a predetermined threshold), the color of the pixel of interest is the background color. It may be determined that they are similar. In this case, a relatively large value is set as a predetermined threshold value in consideration of color mixing in the pixels in the contour region described above.

In S509, the similarity determination unit 404 updates the pixel value in the similarity map for the pixel of interest according to the determination result in S508. That is, when it is determined that the color of the pixel of interest is similar to the color of the background, the pixel value of the pixel in the similarity map M (x, y) corresponding to the same coordinates as the coordinates (x, y) of the pixel of interest is used. Change to "1". If it is determined that the color of the pixel of interest is not similar to the color of the background, the pixel of the pixel of the similarity map (x, y) corresponding to the same coordinates as the coordinates (x, y) of the pixel of interest. Keep the value at “0”.

In S510, it is determined whether or not all the pixels in the target image have been processed as the pixels of interest. If there are unprocessed pixels, the process returns to S508, the next pixel of interest is selected, and processing is continued. On the other hand, if the processing of all the pixels is completed, the similarity determination unit 404 outputs the updated similarity map to the pixel value correction unit 405, and proceeds to the next processing (S511).

In S511, the pixel value correction unit 405 corrects the pixel values of the pixels of interest (u ₁ , v ₁ ) in the provisional foreground image based on the contour map and the similarity map. Since the meaning and selection method of the pixel of interest are the same as those of S504 and S508, the description thereof will be omitted. Specifically, when the following three conditions are satisfied, it is determined that the pixel of interest is not a pixel constituting the foreground region (a pixel constituting the background region), and the pixel value of the pixel of interest in the provisional foreground image is set to ". Change to 0 ”. On the other hand, when the following three conditions are not satisfied, it is determined that the pixel of interest is a pixel constituting the foreground region, and the pixel value of the pixel of interest is not changed.
First condition: _IF (u ₁ , v ₁ ) = 1 (the pixel value in the provisional foreground image is "1").
Second condition: E (u ₁ , v ₁ ) = 1 (the pixel value in the contour map is "1").
Third condition: M (u ₁ , v ₁ ) = 1 (the pixel value in the similarity map is "1").

As described above, each pixel in the contour region in the provisional foreground image has a pixel value in which the background color and the object color are mixed, and there is a high possibility that an error is included. Therefore, using the above-mentioned first condition and second condition, it is determined whether or not it belongs to either the foreground region or the contour region. Then, when it is determined that the pixel belongs to any of the above, and if the third condition is also satisfied, it is determined that the pixel of interest is a pixel belonging to the background, and the pixel value of the pixel of interest is changed to "0". By the correction processing as described above, the error portion in the provisional foreground image is corrected.

In S512, it is determined whether or not all the pixels in the provisional foreground image have been processed as the pixels of interest. If there is an unprocessed pixel, the process returns to S511, the next pixel of interest is selected, and processing is continued. On the other hand, if the processing of all the pixels is completed, the pixel value correction unit 405 outputs the provisional foreground image after the correction processing as the final foreground image.

The above is the content of the foreground image generation processing according to the present embodiment. Here, the effect of the present embodiment will be described with reference to FIG. 7. Similar to the case of FIG. 1 described above, the target image 10 captured by the camera 11 shows the person object 12 as the foreground and the field 13 as the background. Then, a foreground image showing the silhouette of the person object 12 is generated using the background image obtained by capturing only the field 13 from the same imaging viewpoint as the target image 10. FIG. 7 shows three types of foreground images 701 to 703. First, in the case of the foreground image 701, in the vicinity of the boundary between the person object 12 and the field 13, the pixels that should originally be the background area are mistakenly determined to be the constituent pixels of the foreground area, and the silhouette of the person object 12 is It is expressed more expanded than the actual contour. Further, in the case of the foreground image 702, while the portion corresponding to the silhouette of the person object 12 can be accurately extracted, the noise portion included in the target image 10 is erroneously extracted as the foreground region. In the case of the foreground image 703 obtained by the method of the present embodiment with respect to these foreground images by the conventional method, only the silhouette of the person object can be accurately extracted without erroneously extracting the noise portion as the foreground region.

<Modification example>
In S508 described above, the RGB value information is used when determining the degree of similarity between the color of the pixel of interest and the color of the background in the target image. Instead of the RGB values, for example, the RGB color space may be converted into a different color space such as HSV or Lab, and the information of the HSV value or Lab value in the converted color space may be used.

As described above, according to the present embodiment, even if the boundary between the foreground object and the background is ambiguous, the foreground region can be extracted with high accuracy along the contour of the foreground object.

[Embodiment 2]
Next, a mode in which the degree of similarity with the background color is determined based on the appearance frequency of the color component will be described as the second embodiment. It should be noted that the description of the contents common to the first embodiment such as the configuration of the image processing system will be omitted, and the description will be given below focusing on the configuration and the processing contents of the image processing unit 123, which are the differences.

(Outline of this embodiment)
In the first embodiment, a pixel having a color close to the background color in the outline area of the object is regarded as a pixel in the background area, but a pixel having a color close to the color of the object is in the foreground area. It may be regarded as a pixel. That is, in the case of the judgment based on the degree of similarity of the entire RGB values based on the predetermined evaluation value, it may not be possible to detect the pixel with an error, and the accuracy of the finally obtained foreground image is lowered by that amount. .. Therefore, in the present embodiment, the color component having the highest frequency of appearance in each pixel of the background image is obtained, and in the pixel of interest in the target image, the value of the obtained color component is the largest based on whether or not the value of the obtained color component is the largest. Determine if the color is close to the background color. Here, the color component means a color component corresponding to each channel of R (red), G (green), and B (blue) in the case of the RGB color space. In the present embodiment, it is assumed that the target image has colors expressed in the RGB color space. For example, when a scene in which a person is standing on a lawn is imaged, the pixels near the boundary between the person and the lawn in the obtained image have pixel values of a color in which the color of the lawn and the color of the person are mixed. .. Further, the pixel value representing the color of the lawn is generally a pixel value such that the G component takes the maximum value. Therefore, among the pixels in the contour region, the pixel having the maximum G component can be regarded as the pixel constituting the background. As a specific processing content, first, the color analysis processing of each pixel in the background image is performed together with the extraction processing of the foreground region. By this color analysis processing, the color component having the highest frequency of appearance among RGB is specified. Then, when the color component having the maximum value in the pixels in the foreground region and the contour region in the target image is the same as the color component having the highest frequency of appearance in the background image, it is determined that the pixel is not a pixel in the foreground region. , Change that color to the background color. This makes it possible to obtain a highly accurate foreground image.

(Details of foreground image generation processing)
The foreground image generation process in the camera adapter 120 according to the present embodiment will be described in detail with reference to the functional block diagram shown in FIG. 8 and the flowchart shown in FIG. As shown in FIG. 8, the image processing unit 123'according to the present embodiment has six functional units related to the generation of the foreground image. Specifically, it has an image acquisition unit 401, a foreground area extraction unit 402, a contour area detection unit 403, a similarity determination unit 404', a pixel value correction unit 405, and a background color analysis unit 801.

S901 to S906 correspond to S501 to S506 in the flowchart of FIG. 5 of the first embodiment, and there are no differences, so the description thereof will be omitted.

In S907, the background color analysis unit 801 analyzes the background image acquired in S901 and derives the color component having the highest frequency of appearance in each pixel constituting the background image. More specifically, in the background image, the total number P _G, B component of the pixel total number of pixels the value of the R component is maximized P _R, the value of the G component is maximum among the RGB values as the pixel value the value Find the total number P _B of pixels that maximizes. Then, the obtained P _R, P _G, among the P _B, that number is the largest color component, the occurrence frequency is the highest color component. Although all the pixels of the background image are used here, the color component having the highest frequency of appearance may be specified in a part of the background image such as the peripheral area of the object which is the foreground. The derived information on the color component having the highest frequency of appearance is output to the similarity determination unit 404'.

In S908, the similarity determination unit 404'initializes the similarity map. This similarity map is as described in the first embodiment. After the initialization process, the process proceeds to S909.

In S909, the similarity determination unit 404'checks the RGB value of the pixel of interest in the target image and identifies the color component having the largest value. Then, in the next S910, the similarity determination unit 404'determines the similarity between the color of the pixel of interest and the background color in the target image based on the color component ratio. Specifically, when the "color component having the highest frequency of appearance in the background image" specified in S907 and the "color component having the largest value in the pixel of interest" specified in S909 are the same. Determines that the color of the pixel of interest and the color of the background are similar. On the other hand, if they are not the same, it is determined that the color of the pixel of interest and the color of the background are not similar. The meaning and selection method of the pixel of interest are the same as those of S504 and S508 in the flowchart of FIG. 5 of the first embodiment.

In S911, the similarity determination unit 404'updates the pixel value in the similarity map for the pixel of interest according to the determination result in S910. That is, when it is determined that the color of the pixel of interest is similar to the color of the background, the pixel value of the pixel in the similarity map M (x, y) corresponding to the same coordinates as the coordinates (x, y) of the pixel of interest is used. Change to "1". If it is determined that the color of the pixel of interest is not similar to the color of the background, the pixel of the pixel of the similarity map (x, y) corresponding to the same coordinates as the coordinates (x, y) of the pixel of interest. Keep the value at “0”.

In S912, it is determined whether or not all the pixels in the target image have been processed as the pixels of interest. If there is an unprocessed pixel, the process returns to S909, the next pixel of interest is selected, and processing is continued. On the other hand, if the processing of all the pixels is completed, the similarity determination unit 404'outputs the updated similarity map to the pixel value correction unit 405, and proceeds to the next processing (S913).

S913 and S914 correspond to S511 and S512 in the flowchart of FIG. 5 of the first embodiment, respectively, and there are no differences, so the description thereof will be omitted.

The above is the content of the foreground image generation processing according to the present embodiment. Here, the effect of the present embodiment will be described with reference to FIG. Similar to the case of FIG. 7 of the first embodiment, the target image 10 captured by the camera 11 shows the person object 12 as the foreground and the field 13 as the background. Then, foreground images 1001 and 1002 representing the silhouette of the person object 12 are generated using the background image obtained by capturing only the field 13 from the same imaging viewpoint as the target image 10. In FIG. 10, the 4 × 4 block 14 is an enlargement of a part of the target image 10, and each grid of the block 14 represents one pixel. Further, the pixel group 18 in the block 14 indicates a pixel group that is more similar to the background color than the color of the person object 12, and the pixel group 19 is similar to the color of the person object 12 and the background color. Indicates a group of pixels having the same degree. The foreground image 1001 is a foreground image obtained by applying the method of the first embodiment described above, and the foreground image 1002 is a foreground image obtained by applying the method of the present embodiment described above. In the foreground image 1001 obtained by the method of the first embodiment, as shown in the pixel group 1011 the pixel group 18 which is more similar to the background color than the color of the person object 12 can be correctly recognized as the background. There is. On the other hand, the pixel group 19 having the same degree of similarity with the color of the object and the degree of similarity with the color of the background is processed as a pixel constituting the foreground region and contains an error. On the other hand, in the foreground image 1002 obtained by the method of the present embodiment, the foreground region is accurately extracted along the contour of the object.

<Modification example>
In the above-mentioned S902, as in the case of S502 in the first embodiment, the provisional foreground image is obtained by comparing the background image captured in advance with the target image. Instead of this, the corrected output foreground image obtained by applying the first embodiment may be treated as a provisional foreground image in the present embodiment, and each process may be performed. In this case, a more accurate foreground image can be obtained as compared with the case where the first embodiment and the second embodiment are executed independently.

The method for determining the color component having the highest frequency of appearance in the background image in S901 described above is not limited to the above example. For example, the RGB color space is converted into a different color space such as HSV or Lab, the hue is divided into a plurality of groups by using the pixel values in the converted color space, and then the frequently appearing hue group has a high frequency of appearance. It may be decided as a color.

As described above, according to the present embodiment, even if the color difference between the background and the foreground is about the same in the pixels included in the outline of the object, the object in the foreground is extracted with high accuracy along the outline of the object. can do.

<Other Embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (16)

An image processing device that generates a foreground image showing the shape of an object from an captured image.
An acquisition means for acquiring the captured image and
An extraction means for generating a foreground image by extracting a foreground region corresponding to the shape of the object from the captured image, and
A detection means for detecting the contour region of the object from the foreground image, and
A correction means for correcting the pixels constituting the contour region in the foreground image based on whether or not the color of the pixels in the captured image corresponding to the pixels is similar to the background color.
An image processing device characterized by having. The acquisition means further acquires a background image in which the object does not exist, and obtains the background image.
Further, it has a determination means for determining whether or not the color of each pixel constituting the captured image is similar to the background color indicated by the background image.
The correction means makes the correction based on the determination result of the determination means.
The image processing apparatus according to claim 1. In the determination means, when the color of the pixel of interest in the captured image is closer to the color of the background indicated by the background image than the color of the object, the color of the pixel of interest is similar to the background color. Judging that it is
When the pixel in the foreground image corresponding to the pixel of interest determined by the determination means to be close to the background color is a pixel constituting the contour region, the correction means determines the pixel value of the pixel. Make corrections to the pixel values that indicate the background,
The image processing apparatus according to claim 2. The determination means is characterized in that it determines whether or not the color of the pixel of interest is closer to the background color than the color of the object by using a predetermined evaluation value regarding the tint. The image processing apparatus according to 3. The image processing apparatus according to claim 4, wherein the predetermined evaluation value is any one of MSE, MAE, and RMSE. The pixel values of the corresponding pixels between the foreground image and the background image are compared, and when the difference is equal to or less than a predetermined threshold value, it is determined that the color of the pixel of interest is similar to the background color. The image processing apparatus according to claim 3, wherein the image processing apparatus is characterized by the above. 4. The determination means is characterized in that the determination is performed using the pixel value in the color space of the captured image or the pixel value in the converted color space obtained by converting the color space into a different color space. The image processing apparatus according to any one of items 6 to 6. The captured image and the background image have a common color space represented by a plurality of color components, and have a common color space.
The determination means determines whether or not the color of the pixel of interest is closer to the background color indicated by the background image than the color of the object, and the color component ratio of the pixel values in the pixels constituting the captured image. The image processing apparatus according to claim 3, wherein the determination is made based on the above. Further, it has a derivation means for deriving the color component having the highest frequency of appearance in the color space of the background image.
When the color component with the highest appearance frequency in the background image derived by the derivation means and the color component having the maximum value in the attention pixel in the captured image are not the same, the determination means is described. It is determined that the color of the pixel of interest is close to the color of the background.
The image processing apparatus according to claim 8. The determination means is characterized in that the determination is performed using the pixel values according to the common color space or the pixel values according to the converted color space obtained by converting the color space into a different color space. The image processing apparatus according to claim 8. The determination means uses the corrected foreground image obtained by the image processing apparatus according to any one of claims 4 to 10 as the foreground image to be processed by the detection means and the correction means. The image processing apparatus according to claim 1. The detection means satisfies a predetermined condition that can be regarded as pixels constituting the vicinity of the boundary between the object and the background in the foreground image generated by the extraction means. The image processing apparatus according to any one of claims 1 to 11, wherein the pixel of interest detects the contour region. The predetermined condition is a case where the pixels in the foreground image are pixels constituting the object and one or more pixels constituting the background are present in a predetermined block including the pixels. The image processing apparatus according to claim 12. The predetermined condition is a case where the pixel value of a pixel in the foreground image and the pixel value of the corresponding pixel in the contracted image obtained by performing morphology processing on the foreground image are not the same. The image processing apparatus according to claim 12. An image processing method that generates a foreground image showing the shape of an object from a captured image.
The acquisition step of acquiring the captured image and
An extraction step of extracting a foreground region corresponding to the shape of the object from the captured image to generate a foreground image, and
A detection step for detecting the contour region of the object from the foreground image,
A correction step of correcting the pixels constituting the contour region in the foreground image based on whether or not the color of the pixel in the captured image corresponding to the pixel is similar to the background color.
An image processing method comprising. A program that causes a computer to function as each means of the image processing apparatus according to any one of claims 1 to 14.

Images