JP6611255B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to a technique for extracting an arbitrary region in an image.

  There is a demand for extracting an arbitrary region from an image in a wide range of fields. In TV stations, studios, etc., chroma key technology for extracting a subject area by placing a subject on a green background has been researched and developed for a long time and is still widely used. By extracting only the subject area, it is possible to obtain added value such as, for example, generating an image that cannot be realized by pasting the extracted subject image on another image. It can also be stored as teacher data for causing the artificial intelligence to recognize the subject. Because of these advantages, the area extraction technique using the chroma key technique is indispensable for virtual reality, CG creation in movies, big data generation, and the like.

  In the chroma key technology, since it is necessary to use a monochrome screen or the like for the background of the subject, the scenes where the technology can be used are limited. For example, it cannot be used in outdoor location images or sports competitions in stadiums. In order to solve this problem, a technique for extracting an arbitrary region in an arbitrary background has been studied. For example, a method using a background difference (for example, see Non-Patent Document 1), a method for inputting a histogram of each color of an extraction target region and a background (for example, see Non-Patent Document 2), a method for using a depth sensor (for example) : Non-patent document 3).

Satoshi Namibe, Toshikazu Wada, Takashi Matsuyama, “Background Difference Method Robust against Lighting Changes”, Information Processing Society of Japan, CVM115-3, 1999 Carsten Rother, Vladimir Kolmogorov, and Andrew Blake, “” GrabCut ”-Interactive foreground extraction using iterated graph cuts”, ACM Transactios on Graphics (SIGGRAPH), 2004, Volume 23, Issue 3, p. 309-314 Ryan Crabb, Colin Tracey, Akshaya Puranik, and James David, “Real-time Foreground Segmentation via Range and Color Imaging”, IEEE, CVPR Workshops, 2008

  However, the conventional techniques have the following problems.

  In the region extraction method using the background difference (hereinafter referred to as the background difference method), a background image that does not include the extraction target region is captured in advance, and the image including the extraction target region and the background image are compared for each pixel. Then, an extraction target region is obtained based on a difference value such as a color or luminance obtained as a result of the comparison. At this time, if the pixels of the background portion in the image including the extraction target area are equal to the pixels of the background image captured in advance and the pixels of the extraction target area are different from the pixels in the background image in color components and brightness, extraction is performed. The difference from the pixels of the background image can be obtained from only the pixels in the target area, and the extraction target area can be accurately obtained. However, depending on the usage scene, there is a difference between the background image taken in advance and the background part in the image including the extraction target area, such as the change in sunlight due to time, the fluctuation of illumination, or the generation of shadows due to subject movement, etc. The extraction target area cannot be obtained correctly. Also, depending on the movement of the subject and the clothes, the pixels in the extraction target region may have colors and brightness close to those of the pixels included in the background image captured in advance, and in this case, the extraction target region cannot be obtained accurately. Sometimes.

  Extraction target area and background area pixels are selected from the image including the extraction target, a histogram of the color of the selected pixel is obtained, and the extraction target area is calculated for all pixels of the image including the extraction target based on the distribution ratio. A method has been proposed in which the likelihood of whether or not it is a background is obtained and an extraction target region is obtained based on the likelihood. Although this method can determine the boundary between the extraction target area and the background area, it is based on color information, so the extraction target area can be obtained accurately depending on the selection result of the pixels in the extraction target area and the background area. There may not be.

  There is a method of obtaining only the subject as the extraction target region by using the distance to the subject existing in the extraction target region. In this method, since the extraction target area is determined based on the distance information, it is easy to accurately obtain the extraction target area even when the background color and the color of the extraction target area are close. However, in the method based on the depth, when a distance equal to the extraction target is detected in a region that is not the extraction target, it is difficult to separate this. In addition, when using an infrared depth sensor when measuring depth, the depth information that can be measured is sparse in the screen, and the extraction target area in the image must be specified precisely using only the depth result. Is difficult. In addition, the infrared depth sensor is difficult in principle to accurately measure a black object, fur, hair, etc., and depending on the type of subject, the extraction target area cannot be obtained accurately. There is.

  Thus, with the conventional technique, it is difficult to accurately obtain an extraction target region with respect to an arbitrary background.

  The present invention has been made in view of the above, and an object of the present invention is to obtain an extraction target region more accurately under an arbitrary background.

An image processing apparatus according to the first aspect of the present invention inputs an image input means for inputting an image, and the shape of the foreground area along the outline of the foreground area of the image, and corresponds to the shape of the foreground area from the image. Shape input means for cutting out pixels to obtain a processing area image, and the foreground or background probability of each pixel of the image from the pixels on the foreground area side of the processing area image and the pixels on the background area side of the processing area image An initial random field generating means for generating an initial random field for holding the contour, an outline extracting means for extracting the outline of the foreground area from the processing area image, and the initial random field weighted based on the outline is a cost. a separating means for separating the image into a foreground region and a background region using graph cuts as a function, the said contour extraction means, class pixels is close color and position from the process area image Superpixel calculating means for calculating a ringed superpixel and generating a boundary image representing a boundary of the superpixel, generating a differential filter based on the shape of the foreground region, and applying the differential filter to the boundary image Differential filter processing means for extracting contour candidate pixels in the foreground region, and contour interpolation means for extracting the contour line of the foreground region by interpolating between the contour candidate pixels along the boundary of the super pixel. It is characterized by.

An image processing method according to the second aspect of the present invention includes:
An image processing method executed by a computer, the step of inputting an image, the step of inputting the shape of a foreground region along the outline of the foreground region of the image, and the shape of the foreground region from the image A step of obtaining a processing region image by cutting out pixels, and maintaining the probability of foreground or background likelihood of each pixel of the image from the pixels on the foreground region side of the processing region image and the pixels on the background region side of the processing region image Generating an initial random field, extracting a contour line of the foreground region from the processing region image, and using the graph cut with the initial random field weighted based on the contour line as a cost function. the has a step of separating the foreground and background regions, a step of extracting the contour, the color and position of the processing region image Calculating a superpixel obtained by clustering a large number of pixels, generating a boundary image representing a boundary of the superpixel, generating a differential filter based on the shape of the foreground region, and applying the differential filter to the boundary image. Extracting the contour candidate pixels of the foreground region, and extracting the contour line of the foreground region by interpolating between the contour candidate pixels along a boundary of the superpixel .

  According to a third aspect of the present invention, there is provided an image processing program for operating a computer as each unit of the image processing apparatus.

  According to the present invention, an extraction target region can be obtained more accurately under an arbitrary background.

1 is a functional block diagram illustrating a configuration of an image processing apparatus according to a first embodiment. It is a flowchart which shows the flow of the process which produces | generates a process area image and an initial stage random field. It is a figure which shows the example of an original image. It is a figure which shows a mode that the shape of the foreground area | region was input into the original image. It is a flowchart which shows the flow of the process which extracts the outline of a foreground area | region. It is a figure which shows the example of the super pixel produced | generated from the process area | region image. It is a figure which shows the example of the binary image produced | generated from the super pixel of FIG. It is a figure which shows the inclination of each binary image divided by the grid, and each grid. It is a figure which shows the example which produced | generated and applied the differential filter orthogonal to 8 directions with respect to a binary image. It is the figure which expanded the chest vicinity of the mannequin of the original image. It is a figure which shows the pixel of the outline candidate extracted from the process area image of FIG. It is a figure which shows the example of the prosthetic line obtained by removing noise from FIG. It is a figure which shows the example of the outline candidate line obtained by interpolating between the probable lines of FIG. It is a figure which shows the example which extracted the outline from the outline candidate line of FIG. It is a flowchart which shows the flow of the process which interpolates between important lines. It is a figure which shows a mode that the label was provided to the super pixel and the adjacent information was provided to the pixel. It is a figure which shows a mode that the major line C1 was extended. It is a flowchart which shows the flow of the integration process of a major line. It is a figure which shows a mode that a significant line is integrated. It is a figure which shows a mode that the major line divided by the island-like super pixel is integrated. It is a flowchart which shows the flow of a foreground clipping process. It is a figure which shows the example of the output image which cut out the foreground area | region. It is a whole block diagram of the image recognition system containing the image processing apparatus in 2nd Embodiment. It is a functional block diagram which shows the structure of the image processing apparatus in 2nd Embodiment. It is a flowchart which shows the flow of the process which the image processing apparatus in 2nd Embodiment extracts a feature point. It is a flowchart which shows the flow of a process of the image recognition apparatus in 2nd Embodiment. It is a figure which shows the example of the result displayed on a client terminal.

[First Embodiment]
<Configuration of image processing apparatus>
FIG. 1 is a functional block diagram illustrating the configuration of the image processing apparatus according to the first embodiment.

  An image processing apparatus 1 shown in FIG. 1 includes an input unit 11, an initial random field generation unit 12, a superpixel calculation unit 13, a differential filter processing unit 14, a shape candidate extraction unit 15, a processing random field generation unit 16, and a graph cut calculation unit. 17 and an area logic operation unit 18. Each unit included in the image processing device 1 may be configured by a computer including an arithmetic processing device, a storage device, and the like, and the processing of each unit may be executed by a program. This program is stored in a storage device included in the image processing apparatus 1, and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or provided through a network.

  The input unit 11 inputs an original image (color image) to be processed for which separation of the foreground area and the background area is desired, and a processing area indicating the rough shape of the foreground area, including the boundary between the foreground area of the original image, An image corresponding to the processing region is cut out from the image to generate a processing region image. The input unit 11 displays the input original image. The user inputs a processing area with respect to the displayed original image by tracing the outline of the foreground area to be cut out with a thick pen tool or the like. For example, a display with a built-in touch sensor such as a tablet or a freehand input device such as a pen tablet can be used. The input unit 11 cuts out a part traced by the user from the original image and sets it as a processing region image. Alternatively, when capturing an original image, a shape of a target to be extracted may be detected using a depth sensor or a temperature sensor, and a processing region may be set based on the shape detected by the sensor. When inputting the shape of the processing area with the sensor, the selection area and the non-selection area are determined by the threshold value of the sensor, and a mask image is generated with the pixel value of the selection area set to 1 and the pixel value of the non-selection area set to 0. . However, it is assumed that this image reproduces the rough shape of the object.

  If the processing area is a closed area, the inside is the foreground area and the outside is the background area. When the processing area is an open area, which side is set as the foreground area is determined based on input from the user or sensor information.

  The initial random field generator 12 teaches the cost minimization problem using the edge pixels on the foreground area side of the processing area image as part of the foreground and the edge pixels on the background area side of the processing area image as part of the background. An initial random field having the probabilities of “foreground-like” and “background-like” of each pixel of the original image necessary for solving is generated. More precisely, the initial random field is a random field in which every pixel has a probability of having the same label (foreground or background) with neighboring pixels for every neighboring pixel. The initial random field is a known cost function used in graph cuts. For example, a cost function is such that a pixel value histogram is created from a specified area, and when an arbitrary pixel is appropriately labeled with the pixel value on the foreground or background, the overall cost is small. . In the present embodiment, a processing probability field obtained by weighting the outline of the foreground region with respect to the initial probability field is used at the time of graph cutting.

  The super pixel calculation unit 13 calculates a super pixel obtained by clustering pixels having similar colors and positions from the processing region image, and generates a binary image indicating the boundary of the super pixel.

  The differential filter processing unit 14 applies the differential filter generated based on the shape of the processing region to the binary image generated by the super pixel calculation unit 13 and extracts the contour candidate pixels of the foreground region.

  The shape candidate extraction unit 15 interpolates between the contour candidate pixels of the foreground region to extract the contour line of the foreground region, and generates a contour candidate probability field in which an infinitesimal cost is given to adjacent pixels on the contour line. .

  The processing random field generation unit 16 generates a processing random field by weighting the initial random field with the contour candidate random field.

  The graph cut calculation unit 17 uses the processing random field as a cost function to obtain a combination that minimizes the cost when labeling the foreground or the background on all pixels of the original image, and separates the foreground and the background, Extract the foreground area. A graph cut is a condition in which foreground and background areas are given in advance, assuming that there is a difference in pixel values between adjacent pixels in the foreground and background. Alternatively, it is a calculation method for efficiently obtaining a combination of labels that gives the minimum cost by giving a cost function that gives the lowest cost when the background label is appropriately attached.

  The area logic operation unit 18 cuts out an image corresponding to the foreground area from the original image and generates an output image.

<Operation of Image Processing Device>
Next, a processing flow of the image processing apparatus according to the first embodiment will be described. Hereinafter, the processing of the image processing apparatus 1 will be described by dividing it into three processing regions, that is, processing region image generation processing, contour extraction processing, and foreground clipping processing.

<Processing region image generation processing>
First, processing area image generation processing will be described. These processes are performed by the input unit 11 and the initial random field generation unit 12.

  FIG. 2 is a flowchart showing a flow of processing for generating a processing region image and an initial random field.

  The input unit 11 inputs an original image (step S11). FIG. 3 shows an example of the original image to be input.

  The input unit 11 displays the original image on the screen and inputs the shape of the foreground area (step S12). The user inputs a rough shape so as to trace the outline of the foreground area of the original image displayed on the screen. FIG. 4 shows a state in which the shape of the foreground area is input. The area along the outline of the mannequin in the figure is the shape of the foreground area input by the user.

  The input unit 11 cuts out a portion traced by the user from the original image as a processing region image (step S13). The input unit 11 stores a region including the outline of the foreground region input by the user as a mask image, and generates a processing region image by performing a logical operation on the original image and the mask image. The generated processing region image is transmitted to the initial random field generation unit 12 and the superpixel calculation unit 13.

  The initial random field generation unit 12 generates an initial random field from the processing region image (Step S14). In the example of FIG. 4, the processing area is a closed area surrounding the mannequin. The initial random field generation unit 12 generates an initial random field from the pixel values of the foreground region and the pixel values of the outer region, with the inside of the processing region image being the foreground region and the outside of the processing region image being the background region.

<Outline extraction processing>
Next, the contour extraction process will be described. This process is performed by the superpixel calculator 13, the differential filter processor 14, and the shape candidate extractor 15.

  FIG. 5 is a flowchart showing a flow of processing for extracting the outline of the foreground region.

  The super pixel calculation unit 13 generates a super pixel from the processing region image (step S21). FIG. 6 shows an example of a super pixel generated from the processing region image. A group of pixels of the same color in the figure is one super pixel.

  The super pixel calculation unit 13 generates a binary image obtained by extracting the outline of each super pixel (step S22). The superpixel calculator 13 extracts pixels adjacent to the superpixel boundary to generate a binary image. FIG. 7 shows an example of a binary image generated from the superpixel of FIG.

  The differential filter processing unit 14 generates a differential filter based on the shape of the processing region image (step S23). The differential filter processing unit 14 divides the processing region image with an arbitrary grid, obtains the inclination of the outer periphery of the processing region image in each grid, and generates a differential filter in a direction that folds perpendicular to the inclination of each grid. FIG. 8 shows the binary image divided by the grid and the inclination of each grid.

  The differential filter processing unit 14 applies the generated differential filter to the binary image (step S24). FIG. 9 shows an example in which a differential filter orthogonal to eight directions is generated and applied to a binary image. Pixels of contour candidates in the foreground region are extracted by the differential filter.

  The shape candidate extraction unit 15 removes noise from the contour candidate pixels (step S25). The shape candidate extraction unit 15 obtains the continuity of the contour candidate pixels, and if the contour candidate pixels are continuous than a predetermined threshold, the shape candidate extraction unit 15 leaves it as a constrained line, and if it is shorter than the predetermined threshold, removes it as noise. To do. FIG. 10 is an enlarged view of the vicinity of the chest of the mannequin in the original image. FIG. 11 is a diagram showing contour candidate pixels extracted from the processing region image of FIG. 10 by the superpixel calculator 13 and the differential filter processor 14. FIG. 12 shows an example of a significant line obtained by removing noise from FIG.

  The shape candidate extraction unit 15 interpolates between the significant lines along the superpixel boundary (step S26). Details of the process of interpolating between the significant lines will be described later. FIG. 13 shows an example of a contour candidate line obtained by interpolating between the significant lines in FIG. The contour candidate line indicated by reference numeral 20 in FIG. 13 is short and ignored.

  The shape candidate extraction unit 15 extracts a contour candidate line closest to the contour of the foreground region from the obtained contour candidate lines as a contour line (step S27). In this process, it is only necessary to extract the most likely contour line. For example, contour candidate lines whose shape, length, and inclination are approximate to the foreground area are extracted. Anything that is useful for extracting a correct contour line may be used. An outline candidate line may be presented to the user, and the user may arbitrarily select it. Alternatively, it is possible to allow the contour candidate line to be overlapped with pixels and calculate a line that is most likely to pass through the contour candidate line to obtain the contour line. FIG. 14 shows an example in which a contour line is extracted from the contour candidate lines in FIG. The white line along the mannequin in FIG. 14 is the extracted outline.

  The shape candidate extraction unit 15 generates a contour candidate probability field in which an infinitesimal cost is given to pixels adjacent on the contour line (step S28).

  Here, the process of interpolating between the significant lines will be described.

  FIG. 15 is a flowchart showing a flow of processing for interpolating between the significant lines.

  The superpixel calculator 13 assigns a label to each superpixel (step S31), and attaches adjacency information to pixels adjacent to the superpixel boundary (step S32). FIG. 16 shows a state in which a label is assigned to a super pixel and adjacent information is assigned to a pixel. The dotted rectangle in FIG. 16 represents one pixel, and the solid line represents a superpixel boundary. FIG. 16 shows three superpixels with labels L1, L2, and L3. Pixels P10 to P19 of the superpixel L1, pixels P20 to P16 of the superpixel L2, and pixels P30 to P38 of the superpixel L3 are pixels adjacent to the boundary of the superpixel. In FIG. 16, superpixels to which the device belongs and adjacent information are shown in parentheses. For example, information of (L1, L1: L3) is attached to the pixels P10 to P14. This indicates that the pixels P10 to P14 belong to the superpixel L1 and are adjacent to the boundary between the superpixels L1 and L3. Further, FIG. 16 shows two major lines C1 and C2. Pixels P10, P11, P30, and P31 belong to the major line C1, and pixels P18, P19, P20, and P21 belong to the major line C2.

  The shape candidate extraction unit 15 repeats the process of integrating the following significant lines until the number of significant lines becomes a certain number or less.

  The shape candidate extraction unit 15 includes the pixel having the same adjacent information as the pixel at the tip of the significant line in the significant line (step S33). The pixels at the tip of the significant line C1 in FIG. 16 are pixels P11 and P31. Pixels having the same adjacent information in the vicinity of 4 from the pixels P11 and P31 at the tip of the significant line C1 are searched and included in the significant line C1. The pixel P12 is a pixel having the same adjacency information (L1: L3) in the vicinity of 4 of the pixel P31. The pixel P12 is included in the major line C1 and is the tip of the major line C1. Subsequently, a pixel P32 having the same adjacency information in the vicinity of 4 of the pixel P12 is included in the significant line C1. The above process is repeated to extend the major line C1 along the boundary between the superpixels L1 and L3. FIG. 17 is a diagram illustrating a state in which the major line C1 is extended. Pixels P12 to P14 and P32 to P34 are included in the major line C1. The major line C2 is processed in the same manner and extended including the pixels P16, P17, P22, and P23.

  Subsequently, the shape candidate extraction unit 15 performs an integration process of the major lines whose tips are close to each other (Step S34).

  FIG. 18 is a flowchart showing the flow of the integration process of the major line.

  The shape candidate extraction unit 15 first determines whether or not the major line has come into contact with another major line (step S41). When the pixels in the vicinity of 4 at the leading end of the significant line are pixels of another significant line, it is determined that the significant line is in contact.

  When the potential line comes into contact (YES in step S41), the shape candidate extraction unit 15 adjoins the adjacent information attached to the pixel at the tip of the significant line and the adjacent information attached to the pixel of another meaningful line. Are the same (step S42).

  When the adjacent information is the same (YES in step S42), the shape candidate extraction unit 15 integrates the contacted major lines (step S43). When integrating the major lines, the major lines with the larger numbers are merged with the minor lines with the smaller numbers.

  When the adjacent information is different (NO in step S42), the shape candidate extraction unit 15 does not integrate the contacted major lines.

  On the other hand, when the potential line is not in contact (NO in step S41), the shape candidate extraction unit 15 determines whether the distance between the ends of the significant line is equal to or less than a certain value (step S44).

  When the distance between the tips is equal to or less than a certain value (YES in step S44), the shape candidate extraction unit 15 integrates the significant lines including the pixels on the shortest distance between the tips (step S45). The leading lines C1 and C2 in FIG. 17 are not in contact with each other, but the distance between the ends is a certain distance or less. Therefore, as shown in FIG. 19, the shape candidate extraction unit 15 integrates the major line C2 into the major line C1, including the pixels P15 and P35 on the shortest distance between the tips of the major lines C1 and C2. .

  If the distance between the tips is not less than a certain value (NO in step S44), the shape candidate extraction unit 15 determines whether or not the significant line is divided by the island-shaped superpixel (step and 46). Specifically, the shape candidate extraction unit 15 determines whether the leading pixel of each of the tangent lines belongs to a common super pixel, and whether the super pixel of the interpolation target pixel is a common item of adjacent information of each of the leading pixels. Judge whether or not. In FIG. 20, an island-shaped superpixel L2 is formed between the superpixels L1 and L3, and the major lines C1 and C2 along the boundary between the superpixels L1 and L3 are divided. In FIG. 20, the pixel P10 at the leading end of the significant line C1 belongs to the super pixel L1, and the pixel P11 at the leading end of the significant line C2 belongs to the super pixel L1. That is, the pixel at the tip of each of the important lines belongs to a common super pixel. The interpolation target pixel P20 in the vicinity of 4 of the pixel P10 belongs to the super pixel L2. The interpolation target pixel P29 in the vicinity of 4 of the pixel P11 belongs to the super pixel L2. The adjacent information of the pixel P10 is L1: L2: L3, and the adjacent information of the pixel P11 is L1: L2: L3. That is, the super pixel L2 to which the interpolation target pixels P20 and P29 belong is a common item of the adjacent information of the leading pixels P10 and P11. Therefore, the situation in FIG. 20 is determined as YES in step S46.

  If it is determined that the significant line is divided by the island-shaped superpixel (YES in step S46), the shape candidate extraction unit 15 connects the interpolation target pixels at the shortest distance and integrates the significant line (step S47). At this time, the significance of the interpolated pixel is set to a half value (for example, 0.5). When the significance is half-value, the influence on the initial random field is reduced. In FIG. 20, along the boundary of the super pixel L2, the pixels P21 to P24 and P25 to P28 between the interpolation target pixels P20 and P29 are included in the major line C1, and the major line C2 is integrated into the major line C1. To do.

  The above process will integrate the major lines and reduce the number of major lines.

<Foreground clipping process>
Next, the foreground clipping process will be described. This processing is performed by the processing random field generation unit 16, the graph cut calculation unit 17, and the region logic operation unit 18.

  FIG. 21 is a flowchart showing the flow of the foreground clipping process.

  The processing random field generation unit 16 generates a processing random field by weighting the contour candidate random field with respect to the initial random field (step S51). For example, when the background color and the foreground color are close to each other, the weight ratio can be determined to be significant, such as placing a weight on the shape. By weighting the contour candidate probability field with respect to the initial probability field, the shape of the foreground region can be weighted to the initial probability field generated from the color histograms of a part of the foreground region and a part of the background region.

  The graph cut calculation unit 17 assigns a foreground or background label to all pixels of the original image so as to minimize the cost, using the processing probability field as a cost function, and extracts a foreground region (step S52). At this time, in order to increase the processing speed, graph cuts between superpixels may be performed using the superpixels generated by the superpixel calculator 13.

  The area logical operation unit 18 generates a binary image of 1 and 0 divided into a foreground area and a background area, and performs an logical operation on the binary image and the original image, thereby outputting an output image obtained by cutting out the foreground area. Is generated (step S53). FIG. 22 shows an example of an output image obtained by cutting out the foreground area.

  As described above, according to the present embodiment, the input unit 11 inputs a processing region that includes the boundary of the foreground region of the original image and indicates the rough shape of the foreground region, and the initial random field generation unit 12 Then, an initial random field is generated by using the foreground area side of the processing area image as a part of the foreground and the background area side as a part of the background, and the shape candidate extraction unit 15 extracts the outline of the foreground area and cuts the graph. The calculation unit 17 separates the initial random field weighted based on the contour line into a foreground region and a background region using a graph cut as a cost function, so that a color histogram of a part of the foreground region and a part of the background region is obtained. By adding the shape of the foreground region to the initial random field generated from the above, the foreground region can be more accurately separated under an arbitrary background.

  According to the present embodiment, the super pixel calculation unit 13 generates a super pixel to generate a binary image indicating the boundary of the super pixel, and the differential filter processing unit 14 generates the differential based on the shape of the processing region image. The filter is applied to the binary image, and the shape candidate extraction unit 15 interpolates the contour candidate pixels in the binary image after applying the differential filter along the superpixel boundary to estimate the contour line of the foreground region. Thus, the outline of the foreground area can be estimated based on the input shape of the foreground area.

  When this embodiment is applied to a moving image, the probability field generated at the time of processing the first image is retained and updated. For example, it can be expected to follow the contour information by matching the key points of the corresponding pixels by following the movement of the feature points on the image by optical flow or the like.

[Second Embodiment]
<Configuration of image recognition system>
FIG. 23 is an overall configuration diagram of an image recognition system including an image processing device according to the second embodiment. The image recognition system shown in the figure includes an image processing device 1 and an image recognition device 3. The image processing apparatus 1 and the image recognition apparatus 3 are connected via a network.

  In the second embodiment, the image processing apparatus 1 is used for creating teacher data. Similar to the first embodiment, the image processing apparatus 1 is input with the original image and the shape of the foreground area cut out from the original image, and a foreground image with only the foreground area cut out is obtained. In the second embodiment, as shown in FIG. 24, the image processing apparatus 1 includes a feature point extraction unit 19, extracts feature points of the foreground image, and uses the feature points and the original image as teacher data. Send to.

  The image recognition device 3 includes a reception unit 31, a feature point extraction unit 32, a discrimination processing unit 33, a transmission unit 34, and an image data storage unit 35. Each unit included in the image recognition device 3 may be configured by a computer including an arithmetic processing device, a storage device, and the like, and the processing of each unit may be executed by a program. This program is stored in a storage device included in the image recognition device 3, and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or provided through a network.

  The receiving unit 31 receives an image to be recognized from the client terminal 5. The image recognition device 3 and the client terminal 5 are connected via a network.

  The feature point extraction unit 32 extracts feature points from the received image.

  The discrimination processing unit 33 collates the feature points extracted by the feature point extraction unit 32 with the feature points stored in the image data storage unit 35, and searches for data having similar feature points.

  The transmission unit 34 transmits the search result of the discrimination processing unit 33 to the client terminal 5. The search result is data associated with the feature point, for example, an image received from the image processing apparatus 1.

  The image data storage unit 35 stores the feature points received from the image processing apparatus 1 in association with the images.

<Operation of image recognition system>
Next, the operation of the image recognition system according to the second embodiment will be described.

  FIG. 25 is a flowchart showing a flow of processing in which the image processing apparatus extracts feature points.

  Assume that the image processing apparatus 1 cuts out the foreground image as described in the first embodiment.

  The feature point extraction unit 19 extracts feature points of the foreground image (step S61). The feature point may be any significant one such as an edge gradient or a color change. However, the feature points extracted by the feature point extraction unit 19 must be of the same type as the feature points extracted by the feature point extraction unit 32 of the image recognition device 3.

  The feature point extraction unit 19 transmits the feature points and the original image input by the image processing device 1 to the image recognition device 3 (step S62). The feature point extraction unit 19 transmits the coordinates of the feature point on the image, the positional relationship, and information on the object shown in the foreground image to the image recognition apparatus 3 in association with the original image. When the image processing apparatus 1 creates the teacher data, it is preferable to input foreground object information when the user inputs the shape of the foreground area.

  FIG. 26 is a flowchart showing the flow of processing of the image recognition apparatus.

  The receiving unit 31 receives an image from the client terminal 5 (step S71).

  The feature point extraction unit 32 extracts feature points of the image received from the client terminal 5 (step S72). The feature point extraction unit 32 extracts the same kind of feature points as the feature point extraction unit 19 of the image processing apparatus 1.

  The discrimination processing unit 33 searches the image data storage unit 35 for data having feature points similar to the feature points extracted by the feature point extraction unit 32 (step S73).

  The transmission unit 34 transmits the search result of the discrimination processing unit 33 to the client terminal 5 (step S74).

  FIG. 27 is a diagram illustrating an example of a result displayed on the client terminal.

  A result is obtained when the client terminal 5 transmits an inquiry image 51 taken by a camera or the like included in the client terminal 5 to the image recognition apparatus 3. In FIG. 27, object information 52 and related search result images 53A and 53B shown in the inquiry image 51 obtained as a search result are displayed. The object information 52 and the search result images 53 </ b> A and 53 </ b> B are data stored in the image data storage unit 35.

  According to the present embodiment, the image processing apparatus 1 cuts out the foreground area, extracts feature points of the foreground area, and creates teacher data, thereby cutting off the extra background and extracting only probable feature points. In the image recognition processing, errors can be suppressed and an improvement in matching accuracy can be expected.

DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus 11 ... Input part 12 ... Initial probability field generation part 13 ... Super pixel calculation part 14 ... Differential filter processing part 15 ... Shape candidate extraction part 16 ... Processing probability field generation part 17 ... Graph cut calculation part 18 ... Area Logical operation unit 19 ... feature point extraction unit 3 ... image recognition device 31 ... reception unit 32 ... feature point extraction unit 33 ... discrimination processing unit 34 ... transmission unit 35 ... image data storage unit 5 ... client terminal

Claims (5)

An image input means for inputting an image;
A shape input means for inputting a shape of a foreground area along an outline of a foreground area of the image, and cutting out pixels corresponding to the shape of the foreground area from the image to obtain a processing area image;
An initial random field generating means for generating an initial random field that holds a foreground-like or background-like probability of each pixel of the image from pixels on the foreground area side of the processing area image and pixels on the background area side of the processing area image; ,
Contour extraction means for extracting the contour of the foreground region from the processing region image;
Separating means for separating the image into a foreground region and a background region using a graph cut with the initial random field weighted based on the contour as a cost function ,
The contour line extracting means includes
Superpixel calculation means for calculating a superpixel obtained by clustering pixels having similar colors and positions from the processing area image and generating a boundary image representing a boundary of the superpixel;
Differential filter processing means for generating a differential filter based on the shape of the foreground region and applying the differential filter to the boundary image to extract contour candidate pixels of the foreground region;
An image processing apparatus comprising: a contour interpolation unit that interpolates between the contour candidate pixels along a boundary of the super pixel and extracts a contour line of the foreground region . The image processing apparatus according to claim 1 , wherein the separating unit separates a foreground region and a background region between the superpixels. The shape input means, the image processing apparatus according to claim 1 or 2, characterized in that inputting the shape of the foreground region using the depth sensor or temperature sensor. An image processing method executed by a computer,
Inputting an image;
Inputting the shape of the foreground region along the outline of the foreground region of the image;
Cutting out pixels corresponding to the shape of the foreground region from the image to obtain a processing region image;
Generating an initial random field that holds the foreground or background likelihood of each pixel of the image from the pixels on the foreground area side of the processing area image and the pixels on the background area side of the processing area image;
Extracting a contour line of the foreground region from the processing region image;
Separating the image into a foreground region and a background region using a graph cut with the initial random field weighted based on the contour as a cost function , and
Extracting the contour line comprises:
Calculating a superpixel obtained by clustering pixels having similar colors and positions from the processing region image, and generating a boundary image representing a boundary of the superpixel;
Generating a differential filter based on the shape of the foreground region, applying the differential filter to the boundary image, and extracting contour candidate pixels of the foreground region;
An image processing method comprising: interpolating between the contour candidate pixels along a boundary of the superpixel to extract a contour line of the foreground region . An image processing program that causes a computer to operate as each unit of the image processing apparatus according to claim 1 .