JP6407467B1 - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To provide an image processing apparatus, an image processing method, and a program capable of sufficiently and accurately assigning a foreground label and a background label to pixels in order to distinguish a target area from a background area and cut it out from an image. An image input unit that inputs an original image, and a primary target area is calculated using DNN for the input original image, and a background is set in an area other than the primary target area. A primary target area calculation unit that distributes labels, a small area slide unit that extracts a small area while sliding a small area frame within the primary target area, and an average value or variance of the characteristics of the small area The information included in the small area is converted into a vector by converting the information including the position information and features in the pixel in the small area into a small area characteristic digitizing unit that is statistically evaluated and digitized using A small area clustering unit that performs clustering of each of the small areas using a neural network and stores as a node for each similarity or feature of the original image in the small area; and By evaluating the relationship between the small region extracted by the node and the original image inputted, the nodes by the small region clustering unit, etc., the small region is positioned at each position in the primary target region. A small area linking unit for linking areas, a class of each small area by the small area clustering unit, and a positional relationship between the input images and the continuity of the small areas are evaluated to obtain an evaluation value. Using the evaluation values obtained by the small region connection evaluation unit and the small region connection evaluation unit to calculate the primary target region cut out by the primary target region calculation unit, the secondary target region and the background region The foreground label and the background label are assigned to each of the pixels, and the foreground label, By using the result of distributing the foreground label and the background label by the secondary target area background area cut-out section that performs background label distribution and the secondary target area background area cut-out section as the initial value of the graph cut. A graph cut unit for further distributing the foreground label and the background label in the secondary target region and the background region by the graph cut, and only the pixels that are made the foreground label by the graph cut unit An image processing apparatus comprising: an image generation unit that generates and outputs an image. [Selection] Figure 2

Description

  The present invention relates to an image processing technique, and more particularly to an image processing apparatus, an image processing method, and a program.

  Conventionally, it has been difficult to cut out only a fashion item (for example, clothes) from a photographic image as a target area separately from a background area. Judging from the gray scale or the like only by the shading of the pixels, the clothes are fibers and have irregularities, and are often detected as noise due to light intensity and patterns.

  Here, there are the following as related technologies.

Patent Document 1 discloses the following contents.
As a technique for extracting a target region with high accuracy, a graph cut method is known in which an image is divided into a target region and a background region by modeling a link between pixels. A graph that looks like a node is created, and a cut that divides the graph into a node group of the object region and a node group of the background region with the minimum energy is derived.
However, there is a problem that the extraction accuracy is lowered at a site where each feature amount is not suitable.
For example, when a person wearing a white shirt is walking in front of a shelf and a white wall, the color feature is not suitable for extraction near the shirt, and the extraction accuracy is reduced. Inadequate extraction accuracy decreases.
In other words, in the vicinity of the shirt, the energy of the color feature can be reduced in both the shirt area and the wall area in addition to the boundary between the shirt and the wall. It is easy to extract a person area including.
On the other hand, because the energy of the shape feature can be reduced near the leg edge and the edge of the shelf near the leg, a person area lacking a part of the foot is extracted or includes the area of the shelf It becomes easy to extract a person area.
In this way, there are parts where the color is similar between the person and the background, or there is a background edge near the person, but the extraction accuracy decreases, but the color of the person varies, and the person Since the background color and edges around the person change due to the movement of, it is difficult to preset the arrangement of appropriate contribution ratios of the energy of the color features and the energy of the shape features.
In view of the above problems, the contribution of the feature amount is adjusted for each part of even one target object, for example, color-based region division is performed at the head and shape-oriented region division is performed at the leg. Thus, a technique has been proposed in which a region of an object can be extracted with high accuracy even when a different factor of decreasing accuracy occurs for each part of the object (see Patent Document 1).

Patent Document 2 discloses the following contents.
The graph cut has a drawback that the seed must be set manually because the object and background representative pixels (hereinafter also referred to as seeds) and the object are cut out manually based on the seed.
On the other hand, there is a method that does not manually set the seed. There is a method that automatically calculates the seed. After learning the target to be extracted in advance using another method and roughly estimating the region, seed the pixels inside and outside the region. It is used as.
In graph cut, a graph is created with each image pixel as a node, plus two nodes: a source node corresponding to the background (or foreground) label and a sink node corresponding to the foreground (or background) label. The
A link called an N link is extended between adjacent pixel nodes, and a link called a T link is extended between the source node and the pixel node and between the sink node and the pixel node.
Here, in the prior art, correct labels for the background and the foreground are assigned to a part of the pixel nodes manually or by learning. The correct answer label is also called a seed.
A background seed is assigned a background correct label, and a foreground seed is assigned a foreground correct label. These seeds are specified by the user, the target object region is estimated by another method using these seeds, and the foreground seed or background seed is obtained from the inside or outside of the region.
After obtaining the seed, the cost to be given to the T link is calculated from the relationship between the characteristics of the seed and the characteristics of the pixels other than the seed. The graph cut technique is used to separate the foreground and the background by cutting the graph into two subgraphs so that the energy function value defined by the sum of the costs of the T link and N link is minimized. is there.
Since it is necessary to calculate the data cost without setting the seed by hand or collecting and learning a large amount of data, the graph cut method is used, but the seed is not set by the user by hand. In addition, a technique that enables pattern extraction that does not require learning has been proposed (see Patent Document 2).

JP 2014-10717 A JP 2014-132392 A

  However, even with the techniques disclosed in Patent Document 1 and Patent Document 2 described above, in order to distinguish a target area from a background area and extract it from an image, it is not possible to allocate foreground labels and background labels to pixels sufficiently accurately. It was still difficult.

  An object of the present invention is to provide an image processing apparatus, an image processing method, and a program capable of sufficiently and accurately assigning foreground labels and background labels to pixels in order to distinguish a target area from a background area and cut out from an image. There is.

The image processing apparatus of the present invention
An image input unit for inputting an original image;
A primary target area calculation unit that calculates a primary target area using DNN and assigns a background label to an area other than the primary target area for the input original image;
A small area slide unit that extracts the small area while sliding a frame of the small area in the primary target area;
A small region feature quantification unit that statistically evaluates and quantifies the features of the small region using an average value and variance;
By vectorizing the position information in the pixel in the small region and the information including the feature, the small region is clustered using a neural network based on the information included in the small region. A small area clustering unit that stores as a node for each similarity or feature of the original image,
By evaluating the relationship between the small region extracted by the small region slide unit and the input original image, and each node by the small region clustering unit, each position in the primary target region is evaluated. A small area link portion that links the small areas;
A small region connection evaluation unit that evaluates the positional relationship between the class of each small region by the small region clustering unit and the input image, evaluates the continuity of the small region, and calculates an evaluation value;
Using the evaluation value by the small region connection evaluation unit, the primary target region cut out by the primary target region calculation unit is divided into the secondary target region and the background region, and foreground A label, a background label, and a secondary target area background area separation unit that performs the foreground label and the background label individually for each pixel in the small area;
By using the result of sorting the foreground label and the background label as the initial value of the graph cut by the secondary target region background region segmentation unit, the secondary target region and the background region by the graph cut are used. Furthermore, the foreground label, the graph cut part for sorting the background label,
An image generation unit that generates and outputs only the pixels that have been used as the foreground label by the graph cut unit as a target image.

The image processing method of the present invention includes:
A step of inputting an original image by an image input unit;
A step of calculating a primary target area using DNN for the input original image by the primary target area calculation unit and allocating a background label to an area other than the primary target area;
Extracting the small area while sliding the frame of the small area in the primary target area by the small area slide unit;
A step of quantifying the characteristics of the small region statistically using an average value or variance by the small region feature digitizing unit;
The small area clustering unit vectorizes the position information in the pixel in the small area and information including the features, thereby performing clustering of the small areas using a neural network based on the information included in the small area. And storing as a node for each similarity or feature of the original image in the small region;
The small area link portion, the small region sliding part the original image input and extracted the small area by the to assess the relationship between each node of the Sub area clustering unit, the primary subject Linking the subregion to each position within the region;
A step of evaluating a positional relationship between the class of each small region by the small region clustering unit and the inputted image by a small region connection evaluation unit, evaluating the continuity of the small regions, and calculating an evaluation value; When,
By using the evaluation value obtained by the small region connection evaluation unit, the primary target region cut out by the primary target region calculation unit is determined as a secondary target region by the secondary target region background region cutting unit. Dividing into background areas, foreground labels and background labels are assigned to each, and foreground labels and background labels are individually assigned for each pixel in the small area;
By using the result of distributing the foreground label and the background label by the graph cut unit as the initial value of the graph cut in the secondary target region background region dividing unit, the secondary target region in the graph cut And further sorting the foreground label and the background label in the background region;
And a step of generating and outputting only the pixels set as the foreground label by the graph cut unit as a target image by an image generation unit.

The program of the present invention
In the information processing device,
A process of inputting an original image by the image input unit;
A process of calculating a primary target area using the DNN and distributing a background label to an area other than the primary target area for the input original image by the primary target area calculation unit;
A process of extracting the small area while sliding a frame of the small area in the primary target area by the small area slide unit;
A process of statistically evaluating and quantifying the features of the small region using an average value and variance by the small region feature digitizing unit;
The small area clustering unit vectorizes the position information in the pixel in the small area and information including the features, thereby performing clustering of the small areas using a neural network based on the information included in the small area. Processing to store as a node for each similarity or feature of the original image in the small area,
The small area link portion, the small region sliding part the original image input and extracted the small area by the to assess the relationship between each node of the Sub area clustering unit, the primary subject Linking the small area to each position within the area;
A process of evaluating a positional relationship between the class of each small region by the small region clustering unit and the input image by the small region connection evaluation unit, evaluating the continuity of the small regions, and calculating an evaluation value When,
By using the evaluation value obtained by the small region connection evaluation unit, the primary target region cut out by the primary target region calculation unit is determined as a secondary target region by the secondary target region background region cutting unit. A process of dividing the background area into the foreground label and the background label for each of the background areas, and separately assigning the foreground label and the background label to each of the pixels in the small area; ,
By using the result of distributing the foreground label and the background label by the graph cut unit as the initial value of the graph cut in the secondary target region background region dividing unit, the secondary target region in the graph cut And processing for further sorting the foreground label and the background label in the background region,
The image generation unit executes a process of generating and outputting only the pixels that have been set as the foreground label by the graph cut unit as a target image.

  According to the present invention, there is provided an image processing apparatus, an image processing method, and a program capable of sufficiently and accurately assigning a foreground label and a background label to a pixel in order to distinguish a target area from a background area from an image. be able to.

It is a block diagram which shows the structure of the image processing apparatus in embodiment of this invention. It is a functional block diagram of the image processing apparatus in an embodiment of the present invention. It is a figure explaining the slide of the small area | region in embodiment of this invention. 4 is a flowchart illustrating a processing operation of the image processing apparatus according to the embodiment of the present invention. It is a figure explaining the utilization example of the image processing method in embodiment of this invention. It is a figure explaining the utilization example of the image processing method in embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a diagram illustrating an example of a configuration of an image processing apparatus according to the present embodiment. In the example illustrated in FIG. 1, the image processing apparatus 10 is connected to the imaging unit 20.
The imaging unit 20 may be a camera or the like, and for example, images a model or a mannequin wearing a new fashion item.
The image processing apparatus 10 is shown here as a separate configuration from the imaging unit 20, but may be provided in the same apparatus, or may be provided with a display unit or the like.

  The image processing apparatus 10 is, for example, an information processing apparatus such as a personal computer, a tablet terminal, or a smartphone that performs image processing such as pattern extraction on an image captured by the imaging unit 20. As a hardware, a known configuration such as a control unit 101, a main storage unit 102, an auxiliary storage unit 103, a communication unit 104, and a drive device 105 is provided. Each unit is connected to each other via a bus so that data can be transmitted and received.

  The control unit 101 is a central processing unit (CPU) that performs control of each device, data calculation, and processing in a predetermined information processing device. The control unit 101 is an arithmetic device that executes a program stored in the main storage unit 102 or the auxiliary storage unit 103. The control unit 101 receives data from an input device or a storage device, calculates and processes the data, and then outputs a predetermined output device. (Not shown) or output to a storage device.

  The main storage unit 102 is, for example, a ROM (Read Only Memory) or a RAM (Random Access Memory). The main storage unit 102 is a storage device that stores or temporarily stores programs and data such as an OS and application software that are basic software executed by the control unit 101.

  The auxiliary storage unit 103 is, for example, an HDD (Hard Disk Drive) or the like, and is a storage device that stores data related to application software. The auxiliary storage unit 103 stores, for example, image data acquired from the imaging unit 20.

  The communication unit 104 performs data communication with peripheral devices by wire or wireless. For example, the communication unit 104 acquires an image including a pattern via a network and stores the acquired image in the auxiliary storage unit 103.

  The drive device 105 reads a predetermined program from the recording medium 106, for example, a flexible disk, a CD (Compact Disc), a magneto-optical disk, etc., and installs it in the storage device. The installed predetermined program can be executed by the image processing apparatus 10. Note that a predetermined program may be acquired from another device via a predetermined network.

  FIG. 2 is a functional block diagram of the image processing apparatus 10 in the present embodiment. In the example illustrated in FIG. 2, the image processing apparatus 10 includes an image input unit 201, a primary target region calculation unit 202, a small region slide unit 203, a small region feature digitization unit 204, a small region clustering unit 205, and a small region link unit 206. , A small region connection evaluation unit 207, a secondary target region background region segmentation unit 208, a graph cut unit 209, and an image generation unit 210.

  Each unit illustrated in FIG. 2 functions, for example, when the control unit 101 loads an image processing program stored in the auxiliary storage unit 103 into the main storage unit 102 and executes it. That is, each unit illustrated in FIG. 2 may be realized by the control unit 101 and the main storage unit 102 as a work memory, or may be realized by hardware.

  For example, the image input unit 201 has a function of inputting an image (original image) captured by the imaging unit 20. The input image may be a color image. The image input unit 201 stores the input image in the main storage unit 102.

  The primary target region calculation unit 202 performs a primary target region (final background region) on the input image (original image stored in the storage unit 102) using a technique using DNN (multilayer neural network). And a background label is assigned to an area other than the target area (primary).

  The small area slide unit 203 has a function of extracting the small area while sliding the frame of the small area within the target area. For example, as shown in FIG. 3, a total of 9 squares of 3 × 3 may be used as one small area frame, and may be slid by one square in the horizontal and vertical directions.

  The small region feature digitizing unit 204 has a function of quantitatively evaluating and digitizing the small region feature extracted by the small region slide unit 203.

  The small region clustering unit 205 has a function of clustering the features of each small region and storing them as nodes for each similarity or feature of the images in the small region. By vectorizing the position information in the pixel in the small area and the information including the feature, the information contained in the small area is used to perform clustering (classification) of each small area using a neural network.

  The small region link unit 206 links the relationship between the small region extracted by the small region slide unit 203 and the input image (the original image stored in the storage unit 102) and each node in the small region clustering unit 205 by a link. It has a function to express. By evaluating the connection with the area to which the background label is assigned and the connection of the small area located at the center of gravity in the target area (primary), the small area is set at each position in the target area (primary). Link.

  The small region connection evaluation unit 207 evaluates the positional relationship between each small region class in the small region clustering unit 205 and the input image (original image stored in the storage unit 102), and the continuity of the small regions. Has a function to perform evaluation. Evaluate the similarity of similar subregion connections and labels.

  The secondary target area background area segmentation unit 208 assigns labels to each small area using the evaluation value from the small area connection evaluation unit 207, that is, the target area (primary target area cut out by the primary target area calculation unit 202). The secondary foreground area and background area are divided into foreground labels and background labels, and foreground and background labels are also individually assigned to each pixel in the small area. It has the function of sorting and labeling.

  The graph cut unit 209 further sorts the foreground label and the background label between the target region (secondary) and the background region based on the information labeled by the secondary target region background region cutting unit 208, It has a function of labeling.

  The image generation unit 210 has a function of generating, as an image (target image), only the pixels that have been set as foreground labels by the graph cut unit 209.

  Next, the processing operation of the image processing apparatus 10 in the present embodiment will be described in detail with reference to the flowchart of FIG. In the following description, it is assumed that the foreground label and the background label are allocated to the target area and the background area of the image, respectively.

  First, the image input unit 201 of the image processing apparatus 10 inputs, for example, an image captured by the imaging unit 20 (step S01). The image input unit 201 stores the input image in the main storage unit 102.

  Next, the primary target area calculation unit 202 of the image processing apparatus 10 uses the DNN (multilayer neural network) for the image input by the image input unit 201 to perform a primary target area (final foreground area). (This may include a part of the background area.) Is calculated, and the background label is assigned to an area other than the target area (primary) (step S02). At this time, the foreground label is not yet distributed in the target area (primary).

  Next, the small area slide unit 203 of the image processing apparatus 10 extracts the small area while sliding the frame of the small area within the target area (primary) (see FIG. 3) (step S03). In addition, along with the slide of the frame of the small area, the connection with the area where the background label is distributed and the connection of the small area located at the center of gravity in the target area (primary) are evaluated by processing as described below. Thus, the foreground label and the background label are assigned to each of the secondary target area and the background area.

  The small area feature digitizing unit 204 of the image processing apparatus 10 quantitatively evaluates and digitizes the characteristics of the small area (step S04). For example, it is not grayscale (a black and white check is gray according to human vision, but the machine recognizes each, so we want to recognize a given square as gray on average). Statistically evaluated using or variance and digitized.

  The small region clustering unit 205 of the image processing apparatus 10 clusters the features of each small region and stores them as nodes for each similarity or feature of the images in the small region (step S05). That is, by vectorizing the information including the position information and features in the pixels in the small area, clustering (classification) of each small area is performed using a neural network based on the information included in these small areas.

  The small area link unit 206 of the image processing apparatus 10 includes a small area extracted by the small area slide unit 203, an input image (an original image stored in the storage unit 102), and each node by the small area clustering unit 205. By evaluating the relationship or the like, the small area is linked to each position in the target area (primary) (step S06). By evaluating the connection with the area to which the background label is assigned and the connection of the small area located at the center of gravity in the target area (primary), the small area is set at each position in the target area (primary). Link.

  The small region connection evaluation unit 207 of the image processing apparatus 10 evaluates the positional relationship between each small region class by the small region clustering unit 205 and the input image (original image stored in the storage unit 102). An evaluation value is calculated by evaluating the continuity of the region (step S07). Evaluate the similarity of similar subregion connections and labels.

  The secondary target region background region segmentation unit 208 of the image processing apparatus 10 uses the evaluation value from the small region connection evaluation unit 207 to distribute labels for each small region, that is, cut out by the primary target region calculation unit 202. The target area (primary) is divided into a secondary target area and a background area, and the foreground label and background label are assigned to each area. Also, each pixel in the small area is individually foregrounded. Labels and background labels are sorted and labeled (step S08).

At least two or more by the labeling of the background area by the pre-processing (primary) using DNN and the clustering of the small area associated with each position of the image using the label information and further using the neural network A multi-stage scheme is realized.
In this way, by separating in multiple stages, it becomes possible to gradually separate the foreground area and the background area in more detail. In addition, by assigning labels to small areas associated with the position of each image and clustering the small areas using a neural network, it is possible to extract the topology (relevance) of features for each small area. .

  Next, the graph cut unit 209 of the image processing apparatus 10 distributes the foreground label and the background label to the target region (secondary) and the background region of the input image by the secondary target region background region cutting unit 208. Results are used as initial values for graph cuts, so that more detailed target regions can be extracted by graph cuts, that is, foreground and background labels can be further divided between target regions (secondary) and background regions for labeling. Is performed (step S09). When cutting the graph, by minimizing the sum of the energy evaluation functions, the pixels included in the small area are further accurately separated into the foreground area and the background area. That is, the small area itself is not cut using the foreground label and the background label of the small area, but the pixels in each small area are cut into the foreground area and the background area using the foreground label and the background label in the pixels in the small area. In other words, it is possible to perform segmentation for the pixels included in the small region, instead of segmenting the small region block.

  Then, the image generation unit 210 of the image processing apparatus 10 generates only a pixel that has been set as the foreground label by the graph cut unit 209 as a target image and outputs the target image to a predetermined output device (image display device) or the like (step S10). Thus, only the final target area can be distinguished from the final background area from the input image. That is, for example, it is possible to cut out only a fashion item (for example, clothes) from a photographic image as a target image in distinction from the final background region.

  According to the present embodiment, the foreground label and the background label can be allocated to the pixel sufficiently accurately in order to distinguish the target region (final) from the image and distinguish it from the background region (final). .

  Further, according to the present embodiment, the input image is indirectly identified as the target region by labeling the target region and the background region of the small region associated with each position of the input image by the neural network. By classifying the features of the image associated with each small region without using a predetermined threshold (such as shading) as in the past, it is possible to classify the small region with a non-linear feature topology. It becomes possible. That is, classification and labeling can be performed in consideration of the connection of image features with the peripheral region.

  Further, according to the present embodiment, the image is not divided into the elementary regions as in the past, but the small region is linked with each position of the image by sliding the frame of the small region within the target region. By dividing the linked small region into either the target region or the background region, that is, the label is assigned to the associated small region, and the associated small region is labeled and used for clustering. It is possible to evaluate the connection of features in each small area in detail, considering the connection between the area determined to be the background in DNN and the small area linked to the target area. it can.

  In addition, the conventional method of performing classification (grouping) between adjacent pixels is suitable because it is highly likely to be detected as noise in images of fashion items having various patterns and textures (for example, clothes). However, according to the present embodiment, by associating each position of an image with a small region from the beginning, classification (grouping) is performed by a neural network in consideration of a wide range including a peripheral region of the image, not a pixel unit. Therefore, the contour can be extracted even when the similar pixels are not continuous in the peripheral region. For example, it is possible to extract a contour such that the concatenation angle of the small region with respect to the entire image is continuous by looking at the inclination of the boundary for each small region.

  In addition, according to the present embodiment, the small areas associated with the images in the foreground (target area) and the background are classified using a neural network, so that a plurality of small areas associated with each image are considered. Thus, it is possible to take into account statistical differences in image features in each of the plurality of regions.

  Further, according to the present embodiment, a more flexible nonlinear analysis using a neural network can be expected instead of using a specific probability model.

Hereinafter, an example of a service using the image processing method according to the present embodiment will be described.
Automatic tagging service: ATS (Auto Tagging Service), that is, it can be used for optimizing and improving the efficiency of operations with explanatory texts and search words associated with online store operations.

  First, the flow of conventional online store management operations is as follows: (1) product shooting, that is, shooting a new product on a model or mannequin, and (2) online store registration, that is, with a search word / description for each product. Was created manually (for example, it takes about 10 minutes for each wear).

Therefore, an automatic tagging service: ATS is used to automate explanations and search words.
As shown in FIG. 5, an optimal description and tag are automatically generated from image feature amounts of similar images by AI. By doing so, cost reduction and quality level standardization can be achieved.

As shown in FIG.
(1) Upload an image to ATS.
(2) A target image (target region) is extracted by an algorithm specialized for a clothing region (an image processing method according to the present embodiment).
(3) A feature amount is calculated by an image analysis algorithm. Vectorization, for example, clustering is performed by using the color feature quantity (RGB value) and shape feature quantity of each pixel as a feature vector (dimension number 3).
(4) A similar image is detected by a high-speed search algorithm for high-dimensional data. For example, it is possible to collect 1 million or more fashion item images and manually create teacher data to improve the accuracy of image recognition technology. Further, misclassified images may be stored in a DB, and misclassified features may be learned by AI to improve accuracy.
(5) Search the product master DB. For example, a comparison with a photographic image (similar in the past) is performed.
(6) An optimum product is selected from candidates of “category”, “tag (hash tag)”, and “description” from similar products and automatically generated.

  The above-described embodiment is a preferred embodiment of the present invention, and various modifications can be made without departing from the gist of the present invention. For example, a process for realizing the function of each device may be performed by causing each device or the like to read and execute a program for realizing the function of each device. Further, the program is transmitted to another computer system by a transmission wave through a CD-ROM or a magneto-optical disk as a computer-readable recording medium, or through the Internet or a telephone line as a transmission medium. Also good. Also, some systems may be realized through human actions.

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 20 Imaging part 101 Control part 102 Main memory part 103 Auxiliary memory part 104 Communication part 105 Drive apparatus 106 Recording medium 201 Image input part 202 Primary object area | region calculation part 203 Small area slide part 204 Small area characteristic digitization part 205 Small region clustering unit 206 Small region link unit 207 Small region connection evaluation unit 208 Secondary target region background region segmentation unit 209 Graph cut unit 210 Image generation unit

Claims (3)

An image input unit for inputting an original image;
A primary target area calculation unit that calculates a primary target area using DNN and assigns a background label to an area other than the primary target area for the input original image;
A small area slide unit that extracts the small area while sliding a frame of the small area in the primary target area;
A small region feature quantification unit that statistically evaluates and quantifies the features of the small region using an average value and variance;
By vectorizing the position information in the pixel in the small region and the information including the feature, the small region is clustered using a neural network based on the information included in the small region. A small area clustering unit that stores as a node for each similarity or feature of the original image,
By evaluating the relationship between the small region extracted by the small region slide unit and the input original image, and each node by the small region clustering unit, each position in the primary target region is evaluated. A small area link portion that links the small areas;
A small region connection evaluation unit that evaluates the positional relationship between the class of each small region by the small region clustering unit and the input image, evaluates the continuity of the small region, and calculates an evaluation value;
Using the evaluation value by the small region connection evaluation unit, the primary target region cut out by the primary target region calculation unit is divided into the secondary target region and the background region, and foreground A label, a background label, and a secondary target area background area separation unit that performs the foreground label and the background label individually for each pixel in the small area;
By using the result of sorting the foreground label and the background label as the initial value of the graph cut by the secondary target region background region segmentation unit, the secondary target region and the background region by the graph cut are used. Furthermore, the foreground label, the graph cut part for sorting the background label,
An image generation apparatus comprising: an image generation unit that generates and outputs only the pixels that have been used as the foreground label by the graph cut unit as a target image. A step of inputting an original image by an image input unit;
A step of calculating a primary target area using DNN for the input original image by the primary target area calculation unit and allocating a background label to an area other than the primary target area;
Extracting the small area while sliding the frame of the small area in the primary target area by the small area slide unit;
A step of quantifying the characteristics of the small region statistically using an average value or variance by the small region feature digitizing unit;
The small area clustering unit vectorizes the position information in the pixel in the small area and information including the features, thereby performing clustering of the small areas using a neural network based on the information included in the small area. And storing as a node for each similarity or feature of the original image in the small region;
The small area link portion, the small region sliding part the original image input and extracted the small area by the to assess the relationship between each node of the Sub area clustering unit, the primary subject Linking the subregion to each position within the region;
A step of evaluating a positional relationship between the class of each small region by the small region clustering unit and the inputted image by a small region connection evaluation unit, evaluating the continuity of the small regions, and calculating an evaluation value; When,
By using the evaluation value obtained by the small region connection evaluation unit, the primary target region cut out by the primary target region calculation unit is determined as a secondary target region by the secondary target region background region cutting unit. Dividing into background areas, foreground labels and background labels are assigned to each, and foreground labels and background labels are individually assigned for each pixel in the small area;
By using the result of distributing the foreground label and the background label by the graph cut unit as the initial value of the graph cut in the secondary target region background region dividing unit, the secondary target region in the graph cut And further sorting the foreground label and the background label in the background region;
An image processing method comprising: a step of generating and outputting only the pixels that have been used as the foreground label by the graph cut unit as an object image by an image generation unit. In the information processing device,
A process of inputting an original image by the image input unit;
A process of calculating a primary target area using the DNN and distributing a background label to an area other than the primary target area for the input original image by the primary target area calculation unit;
A process of extracting the small area while sliding a frame of the small area in the primary target area by the small area slide unit;
A process of statistically evaluating and quantifying the features of the small region using an average value and variance by the small region feature digitizing unit;
The small area clustering unit vectorizes the position information in the pixel in the small area and information including the features, thereby performing clustering of the small areas using a neural network based on the information included in the small area. Processing to store as a node for each similarity or feature of the original image in the small area,
The small area link portion, the small region sliding part the original image input and extracted the small area by the to assess the relationship between each node of the Sub area clustering unit, the primary subject Linking the small area to each position within the area;
A process of evaluating a positional relationship between the class of each small region by the small region clustering unit and the input image by the small region connection evaluation unit, evaluating the continuity of the small regions, and calculating an evaluation value When,
By using the evaluation value obtained by the small region connection evaluation unit, the primary target region cut out by the primary target region calculation unit is determined as a secondary target region by the secondary target region background region cutting unit. A process of dividing the background area into the foreground label and the background label for each of the background areas, and separately assigning the foreground label and the background label to each of the pixels in the small area; ,
By using the result of distributing the foreground label and the background label by the graph cut unit as the initial value of the graph cut in the secondary target region background region dividing unit, the secondary target region in the graph cut And processing for further sorting the foreground label and the background label in the background region,
A program that causes the image generation unit to execute a process of generating and outputting only the pixels that have been set as the foreground label by the graph cut unit as a target image.

Images