JP2019536164A - Image processing apparatus, image processing method, and image processing program - Google Patents

Abstract

An image processing apparatus capable of reducing the influence of a background on a similarity score or a similarity label of an object recognition task. An image processing apparatus includes: a feature extraction unit configured to acquire a feature in each sample of a scaled region of interest in a survey image; Saliency generator 12 that calculates the probabilities of the pixels in the sampled sample, and uses the calculated probabilities to remove non-essential features from the scaled sample to calculate the score or label of the object And a dropout processing unit 13. [Selection diagram] FIG.

Description

  The present invention relates to an image processing device, an image processing method, and an image processing program, and more particularly to an image processing device, an image processing method, and an image processing program for removing a background for the purpose of object recognition.

  The object recognition task has many practical applications in monitoring, biometrics, etc. The purpose of these tasks is to output a label or score that indicates the level of similarity between a pair of input images containing the object of interest. The object here may be a person, a vehicle, an animal, or the like. Metric learning is one of the most effective techniques for obtaining similarity scores. The purpose of this technique is to calculate the distance between input images. First, the input image is projected onto the feature space. The feature space is a space that can be learned or a space that can be manually generated. A metric or function is then learned that can compute distances in the new feature space by effectively separating similar and dissimilar features with a given margin.

  However, for robust object recognition, it is necessary to consider the influence of the background on the final score. The reason is that in an unconstrained environment, the background can cause false positive or false negative results. For example, if the objects are very different, the recognition algorithm may still yield a high similarity score just because the background is very similar. The reverse is also true, and if the object is similar but the background is different, the recognition score can be quite low. There is therefore a need to address this issue.

  However, little progress has been made in addressing this issue, and it remains an open issue to be solved. Many methods focus on improving the detection method so that the resulting image contains more objects than the background. Other methods are also focused on improving functionality and metrics. However, little systematic effort has been made regarding the background subtraction itself. Therefore, there is a need for a method that can cope with the influence of background in recognition work.

  One method of recognizing an object is to combine a plurality of measurement standards (see Patent Document 1). Patent Document 1 describes that after a plurality of manually generated features are extracted from an image, several similarity functions such as Bhattacharya coefficient and cosine similarity are used. Finally, RankBoost algorithm is used to combine them. This method provides high accuracy and combines the advantages of many metrics together.

  Another method for estimating the scale is to use a triangular graph (see Patent Document 2). Patent Document 2 describes a method of fitting a triangular graph inside an object (for example, a person) by minimizing an energy function using dynamic programming. This method describes the shape of a person. This method also increases the robustness of the method by combining color information using the HSV color space.

  Patent Document 3 describes a brightness transfer function (BTF). BTF is a function that maps the appearance of an object from one camera to another. These BTF maps found from each training image are weighted into a single model (WBTF) and used for prediction. BTF maps are suitable when lighting fluctuations are a major problem.

  In Non-Patent Document 1, a manually generated feature called a local maximum occurrence representation (LOMO) is calculated for each image. In this method, the projection matrix is efficiently learned with a metric function that is similar in principle to the secondary discriminant analysis technique.

  The method described in Non-Patent Document 2 discloses the similarity between a pair of images from end to end. This means that the entire feature generation, feature extraction, and metric learning pipelines are brought together by learning a deep neural network. A contrasting loss function has also been proposed to help improve discrimination.

US Patent Application Publication No. 2007/0211938 US Patent Application Publication No. 2013/0343642 US Patent Application Publication No. 2014/0098221 JP 2011-253354 A Special table 2013-541119 gazette

S. Liao, Y. Hu, Xiangyu Zhu, and SZ Li, "Person re-identification by Local Maximal Occurrence representation and metric learning," 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, 2015, pp 2197-2206. S. Chopra, R. Hadsell, and Y. LeCun, "Learning a Similarity Metric Discriminatively, with Application to Face Verification," 2005 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Washington, DC, 2005, 539-546.

  Object recognition includes extracting features from an input image for the purpose of representing the object in a more descriptive space. In this space, or a subspace thereof, a metric function or a distance function is learned. This function can be used to compare input images. However, after the features are extracted, not only the input image but also features belonging to the background are obtained. These background features can cause discrepancies in the recognition results. The metrics learned from these features are not robust enough. Thus, there is a need for a technique that can remove such features early in the learning process.

  In Patent Document 1, a plurality of measurement standards are learned or calculated from input features. These metrics are then combined using a ranking function that evaluates each of the metrics. This ranking function is learned in the same manner as the boosting algorithm. In this technique, the background feature problem is not directly addressed. In Patent Document 1, it is assumed that at least one of the metric functions has sufficient discriminating power to know the difference between the foreground feature and the background feature. However, this technology depends on applications, features, etc. and is not directly operated.

  The method described in Patent Document 2 models an object shape using a triangular graph and a color histogram. This improves the performance of identifying the foreground and background. However, if the object is non-rigid, it is difficult to model the shape effectively. For example, when the object is a human, the shape is arbitrary, and it is not a geometric shape such as a square or an ellipse. Therefore, this method is not very suitable in such a case.

  In Patent Document 3, a function for mapping the appearance of an object from one camera to another camera is learned. However, this learning is not always feasible if the environment cannot be accessed or controlled. Without calibration information, it is difficult to learn the projection matrix and hence the mapping function.

  The method described in Non-Patent Document 1 requires manually generated features. Manually generated features are features that are specifically designed with a particular application in mind. Such features work very well in certain applications, but are not generalized to suit other application areas.

  The apparatus described in Patent Document 4 removes unnecessary features using depth information. In order to find the depth information of the image, hardware or camera calibration information is required.

  The method described in U.S. Patent No. 6,057,054 uses pixel dispersion to remove unwanted (background) features. If the pixel variance is high, the pixel is considered as background and is removed. Also, if the pixel variance is low, the pixel is considered foreground. This method is not suitable for scenes with varying illumination, as it assumes that low dispersion pixels are necessarily foreground.

  An object of the present invention is to provide an image processing apparatus, an image processing method, and an image processing program that can reduce the influence of the background on the similarity score or similarity label of an object recognition task.

  An image processing apparatus according to the present invention comprises a feature extraction means for obtaining features in each scaled sample of a region of interest in a survey image and a scaled sample that contributes to the score or label of the object of interest in the region of interest. A saliency generator that calculates the probability of a pixel in it, and a dropout processor that uses the calculated probability to remove features that are not essential for calculating an object's score or label from the scaled sample It is characterized by providing.

  The image processing method according to the present invention obtains a feature in each scaled sample of the region of interest in the survey image and contributes to the score or label of the object of interest in the region of interest for the pixels in the scaled sample. Probabilities are calculated, and the calculated probabilities are used to remove features that are not essential for calculating object scores or labels from the scaled samples.

  A non-transitory computer readable recording medium having recorded an image processing program according to the present invention, when executed on a computer, obtains features in each scaled sample of a region of interest in a survey image, A feature that is not essential for calculating the probability of a pixel in a scaled sample that contributes to the object's score or label of interest and using the calculated probability to calculate the object's score or label An image processing program is stored to remove from the scaled sample.

  According to the present invention, the influence of the background on the similarity score or similarity label of the object recognition task can be reduced.

FIG. 1 is a block diagram showing a configuration example of an image processing apparatus 100 according to the first embodiment of the present invention. FIG. 2 is a flowchart illustrating an example of the operation of the image processing apparatus 100 according to the first embodiment of the present invention. FIG. 3 is a flowchart showing an estimation process performed by the image processing apparatus 100 according to the first embodiment of the present invention. FIG. 4 is a flowchart showing a dropout process performed by the image processing apparatus 100 according to the first embodiment of the present invention. FIG. 5 is a flowchart showing the saliency processing by the image processing apparatus 100 according to the first embodiment of the present invention. FIG. 6 is a block diagram illustrating a configuration example of the image processing apparatus 10 according to the second embodiment of the present invention. FIG. 7 is a block diagram illustrating a hardware configuration example of a computer 1000 capable of realizing the image processing apparatus according to the embodiment of the present invention.

  The overall approach for solving the above technical problems is summarized below. The performance of object recognition is affected by the characteristics of the background, especially when the background scene is complex. Therefore, it is required to suppress the background feature. Given the location of the object of interest in the image, several scaled samples are generated. Features are extracted from these scaled samples. Using the object detector, the saliency map is generated by taking the back propagation of the object detector output relative to the input image. By using the saliency map as a supplement, the probability of pixels belonging to the object or background is calculated. The use of this dropout is performed by removing the neurons belonging to the pixel feature whose probability is background. Next, a score is obtained using feature matching for the remaining features, and one target image with the highest score is selected as the output.

  The present invention is designed to solve these aforementioned problems. In addition to the above entities, other obvious and obvious disadvantages that the present invention can overcome will be apparent from the detailed description and drawings.

<Embodiment 1>
Hereinafter, a first embodiment of the present invention will be described in detail.

  FIG. 1 is a block diagram showing a configuration example of an image processing apparatus 100 according to the first embodiment of the present invention. Referring to FIG. 1, an image processing apparatus 100 includes an input unit 101, an object detection unit 102, a feature extraction unit 103, a learning unit 104, a model storage unit 105, a saliency generation unit 106, and a dropout process. Unit 107, feature matching unit 108, parameter update unit 109, output unit 110, and training data set storage unit 111.

  The input unit 101 receives a series of frames, that is, an image, for example, a video frame, a still image, etc., in the tracking stage. The input unit 101 receives a series of frames, that is, training frames, for example, at or before the learning stage. In the following description, a plurality of frames may be referred to as “plural images”, and frames within the plurality of frames may be referred to as “images”. Also, a plurality of training frames are denoted as “a plurality of training images”, and a training frame within the plurality of training frames is denoted as a “training image”.

  The object detection unit 102 detects one of a region of interest in the frame, that is, an object such as a face, or other object that may include some part. In the following description, the object detection unit 102 detects a person within the frame. The object detection unit 102 provides the position of the person in the frame, that is, the xy coordinates of the upper left and lower right bounding boxes. In the following description, the object detection unit 102 may be referred to as an “object detector”.

  The feature extraction unit 103 is used to extract features from the region of interest provided by the object detection unit 102. Using the position provided by the object detector 102, the feature extractor 103 generates a scaled sample. These samples are then normalized to be in the same coordinate system. The coordinates are defined in a coordinate system set in advance in the frame. Finally, features are extracted from these sample images. These features can be edges, texture, color, temporal, spatial and / or other higher level information or a combination of lower level information from the sample image.

  The learning unit 104 learns a model with one or more series of training frames. More specifically, the learning unit 104 learns a model that will be used to calculate a saliency map of the sample with features extracted from the training frame. The learning unit 104 may calculate an average vector and a covariance matrix from sample features as part of model parameter learning. The learning unit 104 may calculate the gradient of the output of the object detector with respect to the input image.

  This model basically captures the distribution of scaled sample features. More specifically, this model captures the possibility that pixels output from an object detector belong to a specific label. The object detector maximizes its output score so that a given input image matches the desired label. In the present embodiment, the reverse process is required. That is, given a label that is sought to be generated, the object detector maximizes its output score so that the image fits the given label. The model storage unit 105 stores model parameters used for inference purposes, and is used to evaluate the model with given inputs.

  The saliency generation unit 106 uses the model parameters stored in the model storage unit 105 to derive the probability that a certain pixel belongs to a specific label. The probability is calculated by obtaining the gradient of the output from the object detector with respect to the random image. This random image is repeatedly updated until the pixels in this image finally represent probabilities. This procedure repeatedly provides the required saliency map.

  The saliency map generated by the saliency generation unit 106 is input to the dropout processing unit 107. In the dropout processor 107, each feature of the sample is directly associated with their probability from the saliency map. If the probability of a feature is low, ie if the feature belongs to the background class, the feature is removed or excluded. If a feature belongs to the object, the feature is rescaled using probability. This generates the final feature used for matching.

  The feature matching unit 108 selects the sample having the highest score by comparing the feature of the target image with the feature of the survey image. The feature of the survey image at this scale matches the feature of the registered target image. A score is generated by the feature matching unit 108 for each target image. The model parameter is updated by the parameter update unit 109.

  In the following description, a set of the feature extraction unit 103, the saliency generation unit 106, the dropout processing unit 107, and the feature matching unit 108 may be referred to as an “estimation processing unit”.

  The output unit 110 outputs the final target image or ID. The output unit 110 puts a predetermined mark indicating the ID at a predetermined position on the frame represented by the scale (width, height) of the bounding box of the object in the output which is the frame in which the mark is drawn. Can draw.

  The training data set storage unit 111 stores one or more series of training samples including a set of target images and survey images and a label indicating whether or not they are the same object. Note that the input unit 101 may not be used to implement the training data set storage unit 111.

  Next, the operation of the image processing apparatus 100 according to the first embodiment will be described in detail with reference to the drawings.

  FIG. 2 is a flowchart illustrating an example of the operation of the image processing apparatus 100 according to the first embodiment of the present invention.

  The operation of the image processing apparatus 100 according to the first embodiment of the present invention is roughly divided into a training phase and an evaluation phase. In this paragraph, the outline of the present invention will be described with reference to FIG. 2, and the evaluation phase will be described. Object recognition begins with the detection of an object in an image or frame. As shown in FIG. 2, the input (step S101) to the system is an image of an object called a survey image. The object detection unit 102 confirms whether or not the target image exists (step S102). The target image is also called a gallery registered image. If the target image has not been selected (NO in step S102), the previously registered image is selected from the gallery (step S103). The object detection unit 102 may be a specific embodiment of a general object detector. Then, the detected object and the selected target image are provided to the estimation processing unit (step S104). If the target image exists (YES in step S102), the process directly proceeds to step S104. Thereafter, an output score is generated by the estimation processing unit (step S105). If all target images are finally compared, the process ends.

  Details of the estimation process will be described later with reference to FIG. Hereinafter, the estimation processing unit will be briefly described. The estimation processing unit scores each sample generated from the current frame and the target image, and outputs a sample having the maximum score.

  Next, the output unit 110 outputs the estimated ID or estimated label and score, that is, the final output described above (step S105). If the processing of the image processing apparatus 100 has not ended (NO in step S106), the input unit 101 receives the next frame (step S101). When the processing of the image processing apparatus 100 is completed by an instruction from the user of the image processing apparatus 100 via an input device (not shown) (YES in step S106), or when all target images have been processed, image processing The apparatus 100 stops the process illustrated in FIG.

  Next, the operation in the estimation processing phase of the image processing apparatus 100 according to the first embodiment will be described in detail with reference to the drawings.

  FIG. 3 is a flowchart illustrating an example of an operation in the estimation processing phase of the image processing apparatus 100 according to the first embodiment.

As described above, it is required that a model for estimation processing is learned. Accordingly, given a target image, a sample that is a scaled survey image or a scaled query image is generated in step S201. These samples are extracted around a given area at the location of the object and a scale provided by the object detector. Next, features are extracted from these samples (step S202). The extracted features indicate features such as HOG (Histogram of Oriented Gradients), LBP (Local Binary Patterns), normalized gradient, and color histogram. In step S203, dropout processing is executed. The dropout process will be described in detail later with reference to FIG. Next, in the training phase (YES in step S204), it means that the background feature is required to be removed using the mask in step S203 (step S206). The mask is an output of the dropout process in step S203. The background feature is removed using element-wise multiplication of the feature map generated by feature extraction and the mask (step S202). A score is obtained by feature matching using the reduced features from the feature map (step S207). Finally, the scale with the maximum score is selected as the final output from all the scales (step S208). In the case of NO in step S204, no masking operation is required, and only a forward path (classifier forward path) for transmitting data without being disturbed is executed (step S205). This procedure is described in detail below.

In equation (1), the left side is the average or average value of the samples. This is one of the parameters used to normalize features before the actual dropout procedure. 'x _i ' is the vector of features of the i th sample and 'N' is the total number of scaled samples.

In Equation (2), 'V' is a feature vector variance. When these two equations are used, the features are normalized so that the mean is zero and the variance is one. Normalization is performed using the following equation.

  After the features are normalized, the features are passed to the dropout process (step S203). Hereinafter, the procedure of the dropout process will be described in more detail with reference to FIG.

FIG. 4 is a flowchart of dropout processing steps of the image processing apparatus 100 according to the first embodiment of this invention. The first step of the dropout process is a saliency process (step S301). The saliency process is used to generate a saliency map and will be described in detail later with reference to FIG. The saliency map is used to obtain pixel probabilities. Next, in step S302, a feature map is acquired. In step S202, only the feature is extracted, but in step S302, the size of the feature is changed so as to form a three-dimensional map. Next, in step S303, the saliency map entry is checked against the threshold at each pixel location, ie, the x-axis and y-axis. This threshold is selected in advance. If the probability in the saliency map is greater than the threshold value 'T' (YES in step S303), the corresponding feature is simply renormalized using the following equation (4) (step S306).

  If the probability is less than or equal to the threshold value 'T' (NO in step S303), the corresponding feature is removed by being set to zero in step S304. Next, as in step S302, the feature map is updated by reshaping the map to the original dimensions instead of the three dimensions (step S305).

  Next, the saliency map generated in step S301 is stored in the model storage unit 105 in accordance with step S307. Thereafter, the image processing apparatus 100 stops the dropout process shown in FIG. Next, the saliency processing step will be described with reference to FIG.

FIG. 5 is a flowchart showing the saliency processing by the image processing apparatus 100 according to the first embodiment of the present invention. Referring to FIG. 5, the scaled samples are used as input, as shown in step S401. In step S402, the saliency map is initialized with random values using a Gaussian distribution with a mean of zero and a variance of one. This is shown in the following formula (5).

Here, 'P' is a random value for initialization, 'm' is the mean of the Gaussian distribution, 'S' is the standard deviation, and 'd' is the dimension of the saliency map. After initialization, a classifier forward path is calculated in step S403. Equation (6) represents computing a classifier forward path, ie, a class label, given an input image that is a randomly initialized saliency map.

  In Equation (6), 'L' is a classification function whose input is an image 'I', and 'c' is a constant used for regularization of maximization.

The next step is the classifier reverse path (step S404), which is used to obtain the slope of equation (6) for the input saliency map image. This step provides the direction in which the image of the saliency map should be updated so that equation (6) can be maximized. Step S404 is executed using the following equation.

  In Equation (7), '▽ L' is the gradient of the classifier function with respect to the image of the saliency map, and 'a' is a constant that controls the step size. In this equation, “I” is the saliency map updated in step S405. In the next step S406, the loss that occurs after one forward pass and one return pass is calculated. If the loss is sufficiently small (YES in step S407), the algorithm converges and the saliency processing can be stopped. If the loss is not yet sufficiently small (NO in step S407), the steps from step S403 are executed again. These steps are repeated until the loss of the saliency map image is reduced and the algorithm converges.

After the estimation process is completed by the steps described above, the features are renormalized again. This allows the feature matching step to be performed. Matching can be performed using kernel methods such as crossed kernels, Gaussian kernels, polynomial kernels and the like.

  Equation (8) gives a matching score 'r' between the feature of the target image 'I' and the feature of the survey image 'x'. Here, 'd' is the feature dimension length and 'j' is the dimension index. The target image with the lowest score is selected.

  One of the objects of the present invention is to provide an image processing apparatus capable of accurately recognizing an object and reducing the influence of the background on the similarity score or the distance score.

  The first effect of the present embodiment is that the object can be estimated accurately and the influence of the background on the recognition score can be reduced.

  Other effects of this embodiment will be described below. The advantage of this embodiment is that a plurality of metrics can still be used with this method, such as a combination of numerous metrics described in US Pat. This image processing apparatus can be used for reducing the background effect, which is a process for improving the performance of each measurement standard.

  Another effect of this embodiment is that, unlike Non-Patent Document 1 and Patent Document 2, model parameters do not require manually generated features. Manually generated features narrow the scope of the technology and prevent generalization. This image processing device can be used with any technique that requires background removal.

  A further effect of the present embodiment is that, unlike Patent Document 3, it is not necessary to calculate a projection matrix, so that camera calibration information is unnecessary.

  Another effect of this embodiment is that learning is performed from end to end as in Non-Patent Document 2, and a similarity score is directly output when a set of input images is given. However, unlike Non-Patent Document 2, the distance function for the image processing apparatus is not limited to the Euclidean distance.

  Also, heavy optimization techniques such as potential support vector machines are not required and therefore real-time operation is possible. Furthermore, the shape of a rigid body and the shape of a non-rigid body can also be recognized. Further, no sample changes in shape, orientation, and parts are required.

  The device described in Patent Document 4 and the method described in Patent Document 5 are deterministic and not probabilistic. However, this embodiment is a probabilistic method, and requires a probability map provided by the saliency generator 106. Moreover, this embodiment does not require hardware or calibration information, and is not assumed in Patent Document 5.

<Embodiment 2>
Next, a second embodiment of the present invention will be described in detail with reference to the drawings.

  FIG. 6 is a block diagram illustrating a configuration example of the image processing apparatus 10 according to the second embodiment of the present invention. Referring to FIG. 6, the image processing apparatus 10 includes a feature extraction unit 11 (for example, a feature extraction unit 103) that acquires features in each scaled sample of a region of interest in a survey image, and a target of interest in the region of interest. A saliency generator 12 (eg, saliency generator 106) that calculates the probability of a pixel in the scaled sample that contributes to the object score or label, and using the calculated probability, the object score or And a dropout processor 13 (eg, dropout processor 107) that removes non-essential features from the scaled sample to calculate the label.

  With this configuration, the image processing apparatus can reduce the influence of the background on the similarity score or similarity label of the object recognition task.

  The second embodiment has the same effect as the first effect of the first embodiment. The reason why the same effect as the first effect of the first embodiment is obtained is that the basic principle is the same in both embodiments.

  The image processing apparatus 10 acquires a similarity between a given target image and a scaled sample of a survey image, and selects a scaled sample having the maximum similarity as a final output (for example, a feature) A matching unit 108) may be included.

  With this configuration, the image processing apparatus can output the scaled sample with the maximum similarity.

  The dropout processor 13 uses the calculated probabilities to generate a mask for removing features that are not essential for calculating the object's score or label, and is scaled using the generated mask. Features may be removed from the collected samples.

  Here, the neurons belonging to the background pixels are calculated, and a mask that can serve as a threshold for the features belonging to such pixels is generated.

  With this configuration, the image processing apparatus can remove features from the scaled sample using the generated mask.

  The image processing apparatus 10 learns to learn model parameters from a series of one or more training samples including a set of target images and survey images, and labels indicating whether or not the objects indicated by the images are the same object. Part (for example, the learning part 104) may be included.

  With this configuration, the image processing apparatus can learn the relationship between the target image and the survey image.

  Further, the image processing apparatus 10 applies a mask generated by the dropout processing unit 13 to remove a feature of a pixel having a low probability in the saliency map, and updates the feature map (for example, , A dropout processing unit 107) may be included.

  With this configuration, the image processing apparatus can update the feature map using the mask.

  The image processing apparatus 10 may include a feature normalization unit (for example, the dropout processing unit 107) that removes features by the dropout processing unit 13 and then normalizes the remaining features again.

  With this configuration, the image processing apparatus can execute the feature matching step using the kernel method.

  Each of the image processing apparatus 100 and the image processing apparatus 10 can be realized using a computer, a program for controlling the computer, dedicated hardware, or a set of a computer and a program for controlling the computer and dedicated hardware.

  FIG. 7 is a block diagram illustrating a hardware configuration example of a computer 1000 that can implement the image processing apparatus 100 and the image processing apparatus 10 described above. Referring to FIG. 7, a computer 1000 includes a processor 1001, a memory 1002, a storage device 1003, and an interface 1004 communicatively connected via a bus 1005. The computer 1000 can access the storage medium 2000. Each of the memory 1002 and the storage device 1003 may be a storage device such as a RAM (Random Access Memory) or a hard disk drive. The storage medium 2000 may be a storage device such as a RAM or a hard disk drive, a ROM (Read Only Memory), or a portable storage medium. The storage device 1003 may function as the storage medium 2000. The processor 1001 can read data and programs from the memory 1002 and the storage device 1003 and write data and programs to the memory 1002 and the storage device 1003. The processor 1001 can communicate with a server (not shown) that provides a frame to the processor 1001, a terminal (not shown) that outputs a final output shape, and the like via the interface 1004 and the like. The processor 1001 can access the storage medium 2000. The storage medium 2000 stores a program for causing the computer 1000 to operate as the image processing apparatus 100 or the image processing apparatus 10.

  The processor 1001 loads a program for operating the computer 1000 as the image processing apparatus 100 or the image processing apparatus 10 stored in the storage medium 2000 into the memory 1002. The processor 1001 operates as the image processing apparatus 100 or the image processing apparatus 10 by executing a program loaded in the memory 1002.

  The input unit 101, object detection unit 102, feature extraction unit 103, learning unit 104, saliency generation unit 106, feature matching unit 108, dropout processing unit 107, and output unit 110 are loaded from the storage medium 2000 to the memory 1002. It can be realized by a dedicated program capable of realizing the functions of the above-described units. The processor 1001 executes a dedicated program. The model storage unit 105, the parameter update unit 109, and the training data set storage unit 111 can be realized by the memory 1002 and / or a storage device 1003 such as a hard disk device. An input unit 101, an object detection unit 102, a feature extraction unit 103, a learning unit 104, a model storage unit 105, a saliency generation unit 106, a dropout processing unit 107, a feature matching unit 108, a parameter update unit 109, an output unit 110, and The training data set storage unit 111 can be realized by a dedicated circuit that realizes the function of each unit described above.

  As a final point, it is clear that the processes, techniques and methodologies described and illustrated herein are not limiting or related to any particular apparatus and can be implemented using a combination of components. Also, various types of general purpose devices may be used in accordance with the instructions herein. The invention has also been described using a specific set of embodiments. However, these are merely exemplary and not limiting. For example, the described software may be implemented in a wide variety of languages such as C, C ++, Java, Python, Perl, and the like. Furthermore, other implementations of the technology of the present invention will be apparent to those skilled in the art.

  Although the invention has been particularly shown and described with reference to exemplary embodiments thereof, the invention is not limited to these embodiments. It will be appreciated by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the invention as defined by the claims.

  A part or all of the above embodiment can be described as the following supplementary notes, but is not limited to the following supplementary notes.

  (Supplementary Note 1) Obtain features in each scaled sample of the region of interest in the survey image and calculate the probability of the pixels in the scaled sample that contribute to the score or label of the object of interest in the region of interest And using the calculated probability to remove from the scaled sample features that are not essential for calculating the score or label of the object.

  (Supplementary note 2) A mask for removing features that are not essential for calculating the label or score of an object is generated, and the mask is applied to remove the feature of pixels in a saliency map with low probability The image processing method as described. Here, the neurons belonging to the background pixels are calculated and a mask is generated that can be a threshold for features belonging to such pixels.

  (Supplementary note 3) Supplementary note 1 or 1 in which model parameters are learned by one or more series of training samples including a set of target images and survey images and a label indicating whether or not the objects indicated by the images are the same object The image processing method according to attachment 2.

  (Supplementary note 4) The image processing method according to any one of supplementary note 1 to supplementary note 3, wherein a scaled sample of an image is obtained from a given region of interest.

  (Supplementary note 5) The image processing method according to any one of supplementary notes 1 to 4, wherein after the feature is removed, the remaining feature is normalized again.

  (Supplementary note 6) A scaled feature that, when executed on a computer, obtains a feature in each scaled sample of the region of interest in the survey image and contributes to the score or label of the object of interest in the region of interest Recorded an image processing program that calculates the probabilities of pixels in a sample and uses the calculated probabilities to remove non-essential features from the scaled sample to calculate the object's score or label A non-transitory computer-readable recording medium.

  (Supplementary note 7) Generates a mask for removing features that are not essential for calculating the label or score of an object when executed on a computer, and removes pixel features in a saliency map with low probability The recording medium according to appendix 6, wherein a mask is applied for the purpose. Here, the neurons belonging to the background pixels are calculated and a mask is generated that can be a threshold for features belonging to such pixels.

  (Supplementary note 8) When executed by a computer, by one or more series of training samples including a set of a target image and a survey image, and a label indicating whether or not the object represented by each image is the same object The recording medium according to appendix 6 or appendix 7, which learns model parameters.

  (Supplementary note 9) The recording medium according to any one of supplementary note 6 to supplementary note 8, which obtains a scaled sample of an image from a given region of interest when executed by a computer.

  (Supplementary note 10) The recording medium according to any one of supplementary note 6 to supplementary note 9, wherein when executed by a computer, after the feature is removed, the remaining feature is normalized again.

10, 100 Image processing device 11, 103 Feature extraction unit 12, 106 Saliency generation unit 13, 107 Dropout processing unit 101 Input unit 102 Object detection unit 104 Learning unit 105 Model storage unit 108 Feature matching unit 109 Parameter update unit 110 Output Unit 111 training data set storage unit 1000 computer 1001 processor 1002 memory 1003 storage device 1004 interface 1005 bus 2000 storage medium

The image processing program according to the present invention, the computer, characterized and acquires the features in each sample were scaled region of interest in the survey image, contributing to the score or label of the object of interest in the region of interest, scaled calculation processing for calculating the probability of pixels in the sample, and using the calculated probabilities, removal processing of removing features not essential to calculate the score or label of the object from the scaled samples Is executed .

Claims (10)

Feature extraction means for obtaining features in each scaled sample of the region of interest in the survey image;
Saliency generating means for calculating the probability of a pixel in the scaled sample that contributes to the score or label of the object of interest in the region of interest;
An image processing apparatus comprising: dropout processing means for removing features that are not essential for calculating a score or label of the object using the calculated probability from the scaled sample. The image processing according to claim 1, further comprising: a feature matching unit that obtains a similarity between a given target image and a scaled sample of the survey image, and selects a scaled sample having the maximum similarity as a final output. apparatus. Dropout processing means
Using the calculated probabilities to generate a mask to remove features that are not essential for calculating an object's score or label;
The image processing apparatus according to claim 1, wherein the feature is removed from the scaled sample using the generated mask. 3. A learning unit that learns model parameters from one or more series of training samples including a set of target images and survey images, and a label indicating whether or not the objects indicated by the images are the same object. The image processing apparatus described. The image processing apparatus according to claim 3, further comprising a feature map update unit that updates the feature map by applying a mask generated by the dropout processing unit in order to remove a feature of a pixel having a low probability in the saliency map. The image processing apparatus according to claim 1, further comprising a feature normalization unit that normalizes the remaining features again after removing the features by the dropout processing unit. Obtain features in each scaled sample of the region of interest in the survey image,
Calculating the probability of a pixel in the scaled sample that contributes to the score or label of the object of interest in the region of interest;
An image processing method comprising using the calculated probabilities to remove features from the scaled sample that are not essential for calculating the score or label of the object. Get the similarity between a given target image and a scaled sample of survey images,
The image processing method according to claim 7, wherein a scaled sample having the maximum similarity is selected as a final output. When run on a computer
Obtain features in each scaled sample of the region of interest in the survey image,
Calculating the probability of a pixel in the scaled sample that contributes to the score or label of the object of interest in the region of interest;
A non-transitory computer readable recording medium recording an image processing program that uses the calculated probability to remove features that are not essential for calculating the score or label of the object from the scaled sample . When run on a computer
Get the similarity between a given target image and a scaled sample of survey images,
The recording medium according to claim 9, wherein a scaled sample having the maximum similarity is selected as a final output.

Images