JP2020017082A - Image object extraction device and program - Google Patents

Abstract

To provide an image object extraction device and a program for extracting a specific object from an input image in a highly accurate manner for a relatively short period of time.SOLUTION: An image object extraction device according to the present invention comprises: a scale conversion section 11 which performs scale conversion against an input image; a computation area curving out section 12 which sequentially curves out an attention area and a context area including peripheral information thereof while scanning the input image after the scale conversion; a size conversion section 14 which reduces the size of the context area so as to become equal to the size of the attention area; and a neural network section 15 which combines each of those obtained from calculating a feature amount in parallel operation using a neural network against each of partial images in the attention area and the context area after the size conversion, extracts a specific object on the basis of a combined feature amount combined therein, and causes the scale conversion to repeat.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for extracting a specific object appearing in an image, and in particular, to an image object extraction apparatus for extracting a sky, a building, a vehicle, a person's face, and the like as an object from an image of, for example, a landscape using a neural network. And programs.

  For example, as a technique for extracting a sky, a building, a vehicle, a face of a person, or the like as an object from an image of a landscape, a technique using machine learning or a neural network is known.

  In particular, a technique for extracting a specific object using a neural network is disclosed (for example, see Non-Patent Documents 1 and 2).

  Neural networks are widely used in tasks such as object extraction and object recognition. When using a neural network to extract an object reflected in a part of an input image, a region of interest (also referred to as “ROI”) of a part of the input image (or a part of a feature map calculated from the input image) Is input to the neural network, and the extraction result is output (for example, see Non-Patent Document 3).

  FIG. 8 shows a schematic configuration of an image object extraction device 100 using a neural network in the related art. FIG. 9A is a diagram showing an outline of an object extraction process using a neural network, and FIG. 9B is one-dimensionally omitted in FIG. 9A for clarity. FIG.

  The image object extracting apparatus 100 according to the related art illustrated in FIG. 8 includes an attention area cutout unit 112, a scanning unit 113, and a neural network unit 115.

  The attention area cutout unit 112 receives the input image I, cuts out a partial image of the attention area (ROI) from the input image I based on the image coordinates specified by the scanning unit 113, and outputs the cutout image to the neural network unit 115. Therefore, the attention area cutout unit 112 functions as an input layer of the image object extraction device 100 shown in FIGS. 9A and 9B.

  The scanning unit 113 sequentially scans (eg, scans in pixel units) the reference value of the region of interest (ROI) from the input image I each time the neural network unit 115 at the subsequent stage performs the characteristic calculation. Generated and output certain image coordinates to the calculation area cutout unit 112.

  The neural network unit 115 includes an attention area feature calculation unit 1151 and an object extraction unit 1154 as a partial network which is a part of the structure of the neural network.

  The attention area feature calculation unit 1151 calculates the feature amount of the partial image of the attention area (ROI) input from the calculation area extraction unit 112 using a neural network, and outputs the feature amount to the object extraction unit 1154. Therefore, the attention area feature calculation unit 1151 functions as a feature calculation layer of the image object extraction device 100 shown in FIGS. 9A and 9B, and converts a partial image (NA1 shown) of the attention area (ROI) into a neural network. Is used to calculate a feature value (NA2 shown).

  Here, as the feature amount calculated by the attention area feature calculation unit 1151, any well-known feature that uses a neural network can be used and is represented by a feature map. As an example of calculating such a feature map, general object conversion (gradation conversion, sharpness / smoothing conversion, edge extraction conversion, morphing conversion, etc.) is performed on a partial image of a region of interest (ROI), and for example, binary, Although it can be represented by a scalar, vector, matrix, or the like, it can be represented more simply by a two-dimensional matrix calculated by a convolutional neural network. A convolutional neural network usually consists of a combination of a convolutional layer, a pooling layer, a fully connected layer, and the like.

  The object extracting unit 1154 determines from the feature amount of the region of interest (ROI) obtained from the region of interest feature calculating unit 1151 that the corresponding region of interest (ROI) is a specific object (vehicle, person's face, etc.) targeted by the neural network. ) Is determined, and if it is determined that the object is an object, the extraction result is output to the outside. Therefore, the attention area feature calculation unit 1154 functions as an object extraction / output layer of the image object extraction apparatus 100 shown in FIGS. 9A and 9B, and stores the purpose of the neural network in the corresponding attention area (ROI). It is determined whether or not a specific object (vehicle, person's face, etc.) is included, and an object extraction result (shown as ND) is output.

  As described above, the image object extracting apparatus 100 according to the related art inputs part of a region of interest (ROI) of an image to a neural network and finally obtains an object extraction result. Extracts objects without any consideration.

Q. V. Le, “Building High-level Features Using Large Scale Unsupervised Learning,” ICASSP, 2013 A. Krizhevsky, I. Sutskever and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” NIPS, 2012 Yamada, Watanabe, "Tracking by Feature Map Selection of Convolutional Neural Network", 79th Annual Convention of Information Processing Society of Japan, Proc. -08, March 16-18, 2017

  As described above, in the image object extraction device using the neural network in the related art, a region of interest (ROI) of an image is input to the neural network, and finally, the object extraction result is obtained. , ROI are not considered at all.

  Therefore, especially when the size of the object shown in the input image is small, it becomes difficult to extract the object, and there is room for improvement in the object extraction accuracy.

  Therefore, an object of the present invention is to provide an image object extraction device and a program that extract a specific object from an input image accurately and in a relatively short time in view of the above-described problem.

  That is, the image object extraction device of the present invention is an image object extraction device that extracts a specific object from an input image, and uses the predetermined initial scale as an initial value to step-by-step convert the input image at a predetermined magnification. Scale conversion means for sequentially generating an input image that has been subjected to scale conversion so as to reduce the size, and scanning the input image scale-converted by the scale conversion means, a partial image of a region of interest, the region of interest and its surroundings A calculation region cutout unit for sequentially cutting out the partial images of the context region including information in a predetermined size, and a size for performing size conversion so that the partial images of the context region sequentially cut out are reduced to the same size as the attention region. Transforming means, using a neural network for the partial image of the region of interest, A region-of-interest calculation unit for calculating a feature amount of the context region; a context region feature calculation unit for calculating a second feature amount of the partial image of the context region after the size conversion using a neural network; A combining unit that combines the first feature amount and the second feature amount to generate a combined feature amount, and determines whether the attention area includes the specific object based on the combined feature amount And an object extraction unit for extracting the specific object from the input image obtained through the scale conversion unit, wherein at least the attention region feature calculation unit, the context region feature calculation unit, the combining unit, The object extraction means is configured as a partial network in the neural network, and The calculating means and the context region feature calculating means are configured to be processed in parallel, and the object extracting means sets the scale of the input image obtained through the scale converting means within a range where the scale does not become smaller than a predetermined threshold value. It is characterized in that objects of different sizes are extracted by repeating the scale conversion by the conversion means.

  Further, in the image object extraction device of the present invention, the calculation region cutout unit cuts out the partial image of the attention region and the partial image of the context region with fixed values from the input image obtained through the scale conversion unit. And wherein the context region has a center of gravity that matches the center of gravity of the region of interest, and is cut out in a size enlarged by a predetermined amount so as to include information around the vertical and horizontal directions with respect to the region of interest. I do.

  Further, in the image object extracting apparatus according to the present invention, the calculation area cutout unit cuts out the area of the context area so as to satisfy the area of more than 1 time and 4 times or less with respect to the area of the attention area.

  Further, in the image object extracting apparatus according to the present invention, the first feature amount and the second feature amount are each represented by a feature map obtained by a feature amount calculation process of the same format. .

  Further, in the image object extraction device of the present invention, the attention area feature calculation means and the context area feature calculation means each execute the scale conversion means by parallel processing based on a convolutional neural network as feature amount calculation processing of the same format. It is characterized in that a feature map in which the positional relationship of each of the first feature amount and the second feature amount is correlated with reference to an input image obtained through the process.

  Furthermore, the program of the present invention is configured as a program for causing a computer to function as the image object extracting device of the present invention.

  According to the present invention, since an object is extracted in consideration of both a region of interest (ROI) in an input image and peripheral information (context region) including the ROI, the operation time is undesirably increased. It is possible to suppress an increase in the amount of calculation without performing this, and to improve the extraction accuracy of the object. In particular, according to the present invention, even when the size of an object with respect to an input image is relatively small so as to be difficult to extract with the related art, the object can be accurately extracted.

It is a block diagram showing a schematic structure of an image object extraction device of one embodiment by the present invention. FIGS. 3A to 3C are explanatory diagrams of a region of interest (ROI) and a context region with respect to an input image in an image object extraction device according to an embodiment of the present invention. 5 is a flowchart illustrating an operation of the image object extraction device according to the embodiment of the present invention. It is an explanatory view of a parallel processing type neural network concerning an image object extraction device of one embodiment by the present invention. It is an explanatory view about input-and-output when a convolutional neural network is used for an attention area feature operation part and a context area feature operation part in an image object extraction device of one embodiment of the present invention. It is a figure showing the example of processing of an example using a convolutional neural network in an image object extraction device of one embodiment by the present invention. It is a figure showing the example of processing of an example using a convolutional neural network in an image object extraction device of one embodiment by the present invention. FIG. 11 is a block diagram illustrating a schematic configuration of a conventional image object extraction device. (A), (b) is an explanatory view of a neural network in a conventional image object extraction device.

  Hereinafter, an image object extraction device 1 according to an embodiment of the present invention will be described with reference to the drawings.

(overall structure)
FIG. 1 is a block diagram showing a schematic configuration of an image object extraction device 1 according to an embodiment of the present invention. An image object extraction device 1 according to an embodiment of the present invention includes a scale conversion unit 11, a calculation region cutout unit 12, a scanning unit 13, a size conversion unit 14, and a neural network unit 15.

  The scale conversion unit 11 receives the input image I (W × H as horizontal × vertical size), temporarily stores the input image I in a memory (not shown), and outputs the input image I at a predetermined magnification (1 / k; k is an arbitrary real number). Is a functional unit that performs scale conversion so as to reduce step by step. The scale conversion unit 11 uses the first scale (W × H) as an initial value, and sequentially orders the individual input images I when the scale of the input image I is reduced stepwise within a range where the scale of the input image I does not become smaller than a predetermined threshold. It outputs to the calculation area cutout unit 12.

  In other words, the image object extraction device 1 of the present embodiment performs processing after the calculation area cutout unit 12 while reducing the size of the input image I little by little by the scale conversion unit 11 so that objects of various different sizes can be extracted. Apply

  The calculation area cutout unit 12 receives the input image I from the scale conversion unit 11 and temporarily stores the input image I in a memory (not shown) as illustrated in FIG. Based on the image coordinates (p (i), q (i)) at the time, a partial image of the region of interest (ROI) from the input image I (w × h as the horizontal × vertical size) and the region of interest (ROI) A partial image (w ′ × h ′ as a horizontal × vertical size) of a context region including surrounding information is cut out and output to the neural network unit 15 and the size conversion unit 14, respectively.

  The image size (w × h) of the region of interest (ROI) is a predetermined fixed value, and the image size (w ′ × h ′) of the context region is also a fixed value. However, the context region has a center of gravity that matches the center of gravity of the region of interest (ROI), and has a size enlarged by a predetermined amount so as to include information around the vertical and horizontal sides of the region of interest (ROI).

  For example, with respect to the area A of the image size (w × h) of the region of interest (ROI), the area A ′ of the image size (w ′ × h ′) of the context region is set to satisfy A <A ′ ≦ 4A. . It is confirmed from the experimental results described later that this range is preferable from the viewpoint of calculation time and detection accuracy.

  For example, FIGS. 2A to 2C are explanatory diagrams of a region of interest (ROI) and a context region for an input image I in the image object extraction device 1 according to an embodiment of the present invention. In the example illustrated in FIG. 2A, the operation area cutout unit 12 determines the image coordinates (p (i), q (i) at the i-th scanning time in the input image I in which, for example, two objects Obj1 and Obj2 appear. ) Shows a state in which a region of interest (ROI) and a context region are cut out. For example, when the region of interest (ROI) is located on the object Obj1 as shown in FIG. 2B, the calculation region cutout unit 12 sets the center of gravity of the region of interest (ROI) as shown in FIG. Cut out the context area with the center of gravity that matches.

  When the region of interest (ROI) is located at the end of the input image I, if there is no information around any of the vertical and horizontal directions with respect to the region of interest (ROI), the image of the context region is also displayed. The size (w ′ × h ′) has a barycenter that matches the barycenter of the region of interest (ROI) by complementing a fixed value (for example, an intermediate value of the dynamic range) in the nonexistent portion, and also has a region of interest (ROI). Is a fixed value size enlarged by a predetermined amount.

  By setting the attention area (ROI) and the context area to fixed values in this way, the subsequent processing of the neural network unit 15 can be stabilized and the processing can be simplified. Note that the input image I input to the calculation region cutout unit 12 is an individual size that is reduced stepwise with the scale (W × H) as an initial value by the scale conversion unit 11, so that the region of interest (ROI) and the Since the context region expands relatively stepwise, objects of various different sizes can be extracted.

  The scanning unit 13 generates a reference value of a region of interest (ROI) by sequentially scanning (for example, scanning in units of one pixel) from the input image I input to the calculation region extracting unit 12 and generating a certain i-th coordinate value. The image coordinates (p (i), q (i)) at the time of scanning are output to the calculation area cutout unit 12.

  The size conversion unit 14 reduces the partial image (w ′ × h ′) of the context region input from the calculation region extraction unit 12 so as to have the same size (w × h) as the region of interest (ROI), and Output to the network unit 15. Note that the reduction processing itself by the size conversion unit 14 can be executed in the neural network unit 15.

  The neural network unit 15 includes an attention area feature calculation unit 151, a context area feature calculation unit 152, a feature connection unit 153, and an object extraction unit 154 as a partial network that is a part of the structure of the neural network.

  The attention area feature calculation unit 151 calculates a feature amount of the partial image of the attention area (ROI) input from the calculation area cutout unit 12 using a neural network, and outputs the feature amount to the feature combination unit 153.

  The context region feature calculation unit 152 calculates a feature amount of the partial image of the context region input from the size conversion unit 14 using a neural network, and outputs the feature amount to the feature combination unit 153.

  Here, as the feature amounts calculated by the attention area feature calculation unit 151 and the context area feature calculation unit 152, known ones arbitrarily determined can be used as long as they use a neural network. It is assumed that the feature amounts are calculated in the same format and are represented by a feature map whose positional relationship is correlated. As an example of such feature amount calculation processing, general object conversion (gradation conversion, sharpness / smoothing conversion, edge extraction conversion, morphing conversion, etc.) is performed on each partial image of the region of interest (ROI) and the context region. Although it is possible to use a convolutional neural network, the calculation can be performed more easily. As will be described later as an example, a feature map based on a convolutional neural network can be represented by a two-dimensional matrix. A convolutional neural network usually consists of a combination of a convolutional layer, a pooling layer, a fully connected layer, and the like.

  The feature combining unit 153 combines the features of the ROI and the context area calculated by the attention area feature calculation unit 151 and the context area feature calculation unit 152, and outputs the combined features to the object extraction unit 154. The unit 13 is instructed to generate a reference coordinate value of the next region of interest (ROI) in the input image I.

  At this time, the scanning unit 13 determines whether or not the object extraction processing has been completed from the entire input image I. If the processing has not been completed, the scanning unit 13 determines the reference of the next region of interest (ROI) for the input image I Is generated, and if it has been completed, i corresponding to the image coordinates (p (i), q (i)) from the initial position for the input image I newly input to the calculation area cutout unit 12 Start the second scan.

  The object extracting unit 154 extracts a specific object which is a target of the neural network based on the ROI obtained from the feature combining unit 153 and the combined feature of the context region and the ROI. It is determined whether or not the specific object is included, and the specific object is extracted.

  That is, if the object extracting unit 154 determines that the corresponding region of interest (ROI) is the specific object, the object extracting unit 154 determines the position information of the region of interest (ROI) at the i-th scanning time with respect to the input image I or The partial image of the region (ROI) itself is output to the outside as an extraction result. This extraction result can be used for recognition processing such as vehicle recognition and face recognition.

  In addition, the object extracting unit 154 performs an object learning based on a pre-learning based on a combined feature amount obtained by combining the attention area (ROI) and the context area obtained from the feature combining unit 153 as a result of the scanning performed by the scanning unit 13. Is determined, and an object is extracted from the entire input image I.

  The object extraction unit 154 can be freely designed without restriction as an object extraction process, and the neural network unit 15 (especially, the object extraction unit 154) uses the ROI and ROI based on a large number of image samples in advance. Neural network parameters relating to object extraction are preliminarily learned based on the combined feature amount obtained by combining the context regions.

  When the object extraction processing is completed for the entire input image I, the object extraction unit 154 instructs the scale conversion unit 11 to perform a predetermined magnification (1 / k; (Real number) to generate the next input image I.

  Therefore, the image object extraction device 1 of the present embodiment can extract objects having various different sizes from the input image (W × H) to be input.

  In FIG. 1, in order to enhance the understanding of the present invention, an example in which a scale conversion unit 11, a calculation region cutout unit 12, a scanning unit 13, and a size conversion unit 14 are distinguished from a neural network unit 15 forming a neural network. However, the entire image object extracting apparatus 1 may be configured as a single neural network.

(Device operation)
Hereinafter, the image object extraction device 1 of the present embodiment will be described more specifically with reference to FIGS. FIG. 3 is a flowchart showing the operation of the image object extraction device 1 according to one embodiment of the present invention. FIG. 4 is an explanatory diagram of a parallel processing type neural network according to the image object extraction device 1 according to one embodiment of the present invention.

  First, as shown in FIG. 3, in the image object extracting apparatus 1, the scale converter 11 determines whether the scale of the input image (W × H) is smaller than a predetermined threshold (Step S1). .

  When the scale of the input image (W × H) is smaller than a predetermined threshold (in this example, smaller than w × h), the scale converter 11 ends the process (step S1: Y), and If not (step S1: N), the process first proceeds to step S3 as the input image I, and after that (step S1: N), the scale of the input image I is increased by a predetermined magnification (1 / k; k). (Step S2).

  Subsequently, the image object extraction device 1 causes the calculation area cutout unit 12 to input the input image I based on the image coordinates (p (i), q (i)) at the i-th scanning time specified by the scanning unit 13. Then, a partial image (w × h) of a region of interest (ROI) and a partial image (w ′ × h ′) of a context region including the region of interest (ROI) and information around the region are cut out (step S3).

  FIG. 4 is an explanatory diagram of a parallel processing type neural network according to the image object extraction device 1 of the present embodiment. The scale conversion unit 11 and the calculation region cutout unit 12 function as an input layer of the image object extraction device 1, and provide a certain region of interest (ROI) and a context region with respect to the input image I which is simply illustrated one-dimensionally in FIG. Cut out.

  Subsequently, the image object extracting apparatus 1 reduces the partial image (w ′ × h ′) of the context area to the same size (w × h) as the attention area (ROI) by the size conversion unit 14. Then, the respective partial networks of the attention area feature calculation unit 151 and the context area feature calculation unit 152 are applied in parallel (step S4). That is, the attention area feature calculation unit 151 and the context area feature calculation unit 152 calculate the feature amounts by parallel processing using the neural networks in the attention area (ROI) and the context area, respectively.

  Therefore, as shown in FIG. 4, the size conversion unit 14 functions as a size conversion layer of the image object extraction device 1, and converts the partial image (DS shown in the drawing) of the context region into the size (NA1 shown in the drawing) of the region of interest (ROI). The size is reduced so as to have the same size (NB1 in the drawing). Then, the attention area feature calculation unit 151 and the context area feature calculation unit 152 function as a feature calculation layer (a convolution layer or a pooling layer in the case of a convolutional neural network) of the image object extraction device 1, and From the partial image (NA1 shown) and the partial image of the context area after the size conversion (NB1 shown), feature amounts (NA2, NB2 shown) are calculated using neural networks.

  Subsequently, the image object extracting apparatus 1 uses the feature combining unit 153 to calculate the feature amounts of the ROI and the context area calculated by the partial networks of the attention area feature calculation unit 151 and the context area feature calculation unit 152, respectively. Combine (step S5).

  Therefore, as shown in FIG. 4, the feature connection unit 153 functions as a feature connection layer of the image object extraction device 1 (such as a fully connected layer (which may include a softmax layer in the case of a convolutional neural network)). A combined feature (NC in the figure) is generated by combining the feature of the region (ROI) and the feature of the context region.

  Subsequently, the image object extraction device 1 uses the object extracting unit 154 to determine a corresponding region of interest (ROI) of the neural network based on the combined feature amount obtained by combining the region of interest (ROI) and the context region. It is determined whether or not a target specific object (vehicle, person's face, etc.) is included, and if it is determined that the target image is an object, a region of interest (ROI) at the i-th scanning time for the input image I is determined. The position information or the partial image of the region of interest (ROI) is output to the outside as an extraction result (step S6).

  Therefore, as shown in FIG. 4, the object extraction unit 154 functions as an object extraction / output layer of the image object extraction device 1, and stores a specific object (vehicle) as a target of the neural network in a corresponding region of interest (ROI). , Etc.), and outputs an object extraction result (ND in the figure).

  Further, the image object extracting apparatus 1 determines whether or not the object extraction processing has been completed from the entire input image I by the scanning unit 13 (step S7), and if not completed (step S7: N), A coordinate value serving as a reference for the next region of interest (ROI) for the input image I is generated, and the process proceeds to step S3. On the other hand, if the object extraction processing has been completed for the entire input image I (step S7: Y), the process proceeds to step S1, and then the input image I newly input to the calculation region cutout unit 12 via step S2. To step S3 so as to start scanning from the initial position.

  As described above, the image object extraction device 1 according to the present invention cuts out the region of interest (ROI) and the context region including the ROI, reduces the image size of the context region to the image size of the ROI, and then performs the ROI A parallel processing type neural network is configured to perform parallel processing on the object and the context region, and the object is extracted with the image size of the ROI that is the original operation target.

(Example)
Hereinafter, an embodiment in which a convolutional neural network is used for the image object extraction device 1 according to the present invention will be described with reference to FIGS.

  FIG. 5 is an explanatory diagram regarding input and output when a convolutional neural network is used for the attention area feature calculation unit 151 and the context area feature calculation unit 152 in the image object extraction device 1 according to one embodiment of the present invention.

  First, as shown in FIG. 5, when a convolutional neural network is used as each partial network of the attention area feature calculation unit 151 and the context area feature calculation unit 152, which are the feature calculation layers, the size of the input image I in the input layer W × Each feature amount of the region of interest (ROI) and the context region with respect to H (number of pixels) is output as a feature map, for example, as an m × n two-dimensional matrix. Note that the output of the feature combination layer connected via the feature calculation layer is represented by m × n × 2.

  In other words, the feature combining unit 153, which is a feature combining layer, combines two types of feature amounts represented by values of m rows and n columns of a two-dimensional matrix as a feature map and outputs the combined feature amounts to the object extracting unit 154.

  Here, each value of m and n is a finite value, and m = 1, 2,..., M, and n = 1, 2,. The values of M and N are values determined by the configuration of the neural network.

  The object extraction unit 154, which is an object extraction / output layer, indicates that the receptive field (the region of interest (ROI) and the context region for the input image I) corresponding to the neurons forming the neural network is an object. For example, the larger the value of each of m and n, the higher the possibility of being an object.

  More specifically, an example of using the convolutional neural network in the image object extraction device 1 of the present embodiment will be described with reference to FIGS. FIG. 6 and FIG. 7 are diagrams illustrating a processing example of an example using a convolutional neural network in the image object extraction device 1 of the present embodiment. FIG. 7 is a one-dimensional representation of an input image for simplicity of FIG. 6, and the embodiment shown in FIGS. 6 and 7 performs parallel processing on a region of interest (ROI) and a context region. It is an example to which a convolutional neural network of the type is applied.

  In the examples shown in FIGS. 6 and 7, from the input image I, which is the output of the scale conversion unit 11, the calculation region extraction unit 12 outputs a region of interest (ROI) of 4 × 4 pixels and a context region of 8 × 8 pixels. It is assumed that a partial image is cut out (see FIG. 1).

  Here, the partial image of the context region of 8 × 8 pixels is cut out to have a center of gravity that matches the center of gravity of the region of interest (ROI) of 4 × 4 pixels.

  Then, the partial image of the context region of 8 × 8 pixels is down-sampled (DS shown in the figure) by a reduction ratio of 1/2 by the size conversion unit 14 and converted into the same size as the region of interest (ROI).

  Each partial image of the region of interest (ROI) of 4 × 4 pixels and the context region of 4 × 4 pixels after the size conversion is input to the neural network unit 15 configured by a convolutional neural network.

  The neural network unit 15 according to the present embodiment also includes an attention area feature calculation unit 151, a context area feature calculation unit 152, a feature combination unit 153, and an object extraction unit 154 (see FIG. 1).

  The attention area feature calculation unit 151 and the context area feature calculation unit 152 respectively include a convolution layer (Conv shown in the drawing) using (kernel size, stride) as a parameter and a maximum pooling layer (Conn) in which (kernel size, stride) is used as a parameter. MP).

  In each convolutional layer (Conv shown in the drawing) in the attention area feature calculation unit 151 and the context area feature calculation unit 152, each receptive field (4 × 4 pixel attention area (ROI) and 4 × 4 pixel For each partial image in the context area), the kernel size is set to 3 × 3 pixels, the stride is set to 1 (moving width for moving in units of one pixel), and convolution operation is performed while moving the kernel, and 2 × 2 two-dimensional Form a matrix feature map.

  In the pooling layer (MP shown in the drawing) in the attention area feature calculation unit 151, the kernel size is 2 × 2, the stride is 2, and the area having the largest value from the feature map of the attention area (ROI) after the convolution calculation is performed. To form a 1 × 1 two-dimensional matrix feature map.

  On the other hand, in the pooling layer (MP shown in the figure) in the context region feature calculation unit 152, the kernel size is also set to 2 × 2, but the stride is set to 1, and the region having the maximum value from the feature map after the convolution calculation regarding the context region is determined. Extraction, thereby forming a 1 × 1 two-dimensional matrix feature map.

  By the way, when the attention area feature calculation unit 151 and the context area feature calculation unit 152 via the size conversion unit 14 are configured as a convolutional neural network of a parallel processing type, each receptive field (the attention area (ROI for input image I) ) And the center (center of gravity) point of the context area) and the stride (movement width) of the receptive field coincide with each other. As a result, the correlation between the region of interest (ROI) and the context region can be increased, and the accuracy of subsequent object extraction can be improved.

  That is, the overall stride of the region of interest (ROI) in the region of interest calculation unit 151 based on the input image I is 2 pixels (based on stride 2 of the maximum pooling layer), and the region of interest corresponding to m rows and n columns ( If the receptive field of the (ROI) is a (x, y, x + 4, y + 4) 4 × 4 rectangular area as the image coordinates of the four corners with respect to the input image I, the image coordinates corresponding to (m + 1) rows and n columns are (x + 2) , Y, (x + 2) +4, y + 4).

  Similarly, the overall stride based on the input image I relating to the context region in the context region feature calculation unit 152 via the size conversion unit 14 is also 2 pixels (the stride 1 of the maximum pooling layer, Downsampling with a reduction ratio of 1/2).

  That is, for simplicity, in FIG. 7, the one-dimensional representation is used to calculate the pixel of the input image I (the receptive field) with respect to the calculation of the attention area feature calculation unit 151 and the calculation of the size conversion unit 14 and the context area feature calculation unit 152 ) And each output of the calculation. In the calculation of the attention area feature calculation unit 151 and the calculation of the size conversion unit 14 and the context area feature calculation unit 152, the output at the time of the calculation indicated by the solid line and the output at the time of the calculation indicated by the broken line next to the solid line are: It can be seen that the position corresponds to a position shifted by two pixels, and it can be confirmed that the positional relationship between the region of interest (ROI) and the context region can be maintained in a highly correlated (unbroken) state.

  In the example illustrated in FIG. 6, the feature maps of the 1 × 1 two-dimensional matrix output from the attention area feature calculation unit 151 and the context area feature calculation unit 152 are combined in the channel direction by the feature combining unit 153. The data is output to the object extracting unit 154 as a 1 × 1 × 2 feature map.

  The object extracting unit 154 determines whether or not an object is included based on the pre-learning based on the 1 × 1 × 2 feature map, and extracts the object from the entire input image I. The partial network constituting such an object extraction unit 154 can be freely designed without any restrictions. In general, a structure is used in which the convolutional layer and the pooling layer are repeated, and then connected to the fully connected layer and the softmax layer.

(Experimental results based on examples)
Here, the effect of the image object extraction device 1 according to the present invention was verified by experiments. In the experiment, the image object extraction device 1 according to the present invention cut out a region of interest (ROI) of 8 × 8 pixels and a partial image of a context region of 16 × 16 pixels from the input image I. Then, the convolution layer and the convolution layer are set so that the overall stride of the ROI in the ROI feature unit 151 is 2 and the overall stride of the context region in the ROI 152 via the size converter 14 is also 2. It was designed by combining the pooling layers. The total number of convolutional layers of the entire neural network unit 15 including the object extracting unit 154 is set to 3.

  On the other hand, as a comparative example, as in the image object extracting apparatus 100 according to the related art illustrated in FIG. 8, it is assumed that an object is extracted only in a region of interest (ROI) of 8 × 8 pixels. And

  Table 1 shows experimental results of object extraction according to the present invention and a comparative example. Table 1 shows a comparison based on the recall ratio for evaluating the small number of detection omissions. In this example, the number of samples is 3971 images. The recall according to the present invention was improved by about 1% as compared with the comparative example. Therefore, the use of the peripheral information of the region of interest (ROI) improves the accuracy of object extraction and confirms the effectiveness of the present invention.

  Table 2 shows a comparison result of the calculation time (required execution time) related to the object extraction according to the present invention and the comparative example. The operation time (required execution time) according to the present invention is within an allowable range, as compared with the comparative example. In particular, based on the conventional technology, an object is simply extracted based only on a region of interest (ROI) of 8 × 8 pixels, an object is further extracted based only on a context region of 16 × 16 pixels, and the results are collectively extracted. It is also conceivable to configure so as to make a judgment. In this case, even if the same level of accuracy as that of the present invention can be obtained, the operation time (required execution time) of the comparative example shown in Table 2 is expected to be twice or more. It is understood how the operation time (required execution time) is kept small.

  From the results in Tables 1 and 2, the image object extraction device 1 according to the present invention performs the parallel processing on the region of interest (ROI) and the context region, and the size of the context region is the same as that of the region of interest (ROI). In this way, it is considered that the increase in the amount of calculation was suppressed while improving the accuracy of object extraction.

  Therefore, the object in the input image is extracted or recognized as in the image object extraction device 1 according to the present invention, using the technique of extracting or recognizing the object using only the input region of interest (ROI) as in the related art. It can be seen that it is effective to use the region of interest (ROI) and its surrounding information (peripheral information) for recognition.

  In particular, in the related art, even when the size of an object with respect to an input image is relatively small so as to be difficult to extract, the configuration according to the present invention enables the object to be accurately extracted. The tendency becomes remarkable.

  To sum up, as a modification of the conventional technique, it is conceivable that only the context region including the peripheral information of the region of interest (ROI) is to be calculated from the beginning. In this case, the peripheral information of the region of interest (ROI) can be considered without changing the configuration of the related art shown in FIG. 8, but there are some problems.

  First, the object is extracted by the neural network while keeping the image size of the context region including the ROI as it is, and the calculation time for extracting the object increases. In other words, in this case, the image size of the context region is relatively larger than the original calculation target ROI image size, and the calculation time increases in proportion to the enlarged area. In particular, in the task of extracting an object from the input image, while the scale conversion as in the above-described scale conversion unit 11 is effective, it is executed many times for images to be calculated at various positions and sizes. Then, the calculation time significantly increases.

  Second, if an object is extracted by a neural network in a context region including an ROI, it is necessary to cut out an object to be extracted from the context region from which the object was extracted by the ROI of the original operation target by some method. It adversely affects the operation time.

  Therefore, the image object extracting apparatus 1 according to the present invention cuts out a context region including the region of interest (ROI) together with the ROI, reduces the image size of the context region to the image size of the ROI, and then performs the ROI and the context A parallel processing type neural network for processing an area in parallel is constructed, and an object is extracted with the image size of the ROI of the original operation target. Therefore, as shown in Tables 1 and 2 above, the accuracy of object extraction is clearly improved when there is peripheral information including the ROI, and the calculation is performed without undesirably increasing the operation time, as compared with the ROI alone. The increase in amount can be suppressed.

  With respect to the example of the above-described embodiment, a computer functioning as the image object extracting device 1 is configured, and a program for causing each unit of these devices to function can be suitably used. Specifically, a control unit for controlling each unit can be constituted by a central processing unit (CPU) in a computer, and at least a storage unit for appropriately storing a program necessary for operating each unit is provided. It can be configured with one memory. That is, by causing such a computer to execute the program by the CPU, the functions of the respective units described above can be realized. Furthermore, a program for realizing the function of each means can be stored in a predetermined area of the storage unit (memory). Such a storage unit can be configured by a RAM or a ROM inside the device, or can be configured by an external storage device (for example, a hard disk). Further, such a program can be configured as a part of software (stored in a ROM or an external storage device) on an OS used in a computer. Further, a program for causing such a computer to function as each means can be recorded on a computer-readable recording medium. In addition, each of the above-described units may be configured as a part of hardware or software, and each unit may be implemented in combination.

  Although the above embodiments and examples have been described as representative examples, it is apparent to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as limited by the above-described embodiments and examples, but only by the appended claims.

  According to the present invention, an object can be extracted from an input image with high accuracy and in a relatively short time, so that the present invention is useful for applications that extract or recognize an object from an image.

REFERENCE SIGNS LIST 1 image object extraction device 11 scale conversion unit 12 operation region cutout unit 13 scanning unit 14 size conversion unit 15 neural network unit 151 attention area feature calculation unit 152 context region feature calculation unit 153 feature connection unit 154 object extraction unit 100 image object extraction device 112 Attention area cutout section 113 Scanning section 115 Neural network section 1151 Attention area feature calculation section 1154 Object extraction section

Claims (6)

An image object extraction device for extracting a specific object from an input image,
Scale conversion means for sequentially generating an input image that has been subjected to scale conversion so as to reduce the input image stepwise at a predetermined magnification with a predetermined initial scale as an initial value,
An operation area for sequentially cutting out a partial image of a region of interest and a partial image of a context region including information of the region of interest and surrounding information while scanning the input image scale-converted by the scale converting means, respectively, in a predetermined size. Cutting means,
Size conversion means for performing size conversion so as to reduce the partial images of the context region sequentially cut out to the same size as the attention region,
Attention area feature calculation means for calculating a first feature amount using a neural network for the partial image of the attention area;
Context region feature calculation means for calculating a second feature amount using a neural network for the partial image of the context region after the size conversion,
Combining means for combining the first feature quantity and the second feature quantity to generate a combined feature quantity;
Object extraction means for extracting the specific object from an input image obtained through the scale conversion means, by determining whether the attention area includes the specific object based on the combined feature amount, With
At least the attention region feature calculation unit, the context region feature calculation unit, the combining unit, and the object extraction unit are configured as a partial network in a neural network,
The attention area feature calculation means and the context area feature calculation means are configured to be processed in parallel,
The object extracting unit extracts the objects of different sizes by repeating the scale conversion by the scale converting unit within a range in which the scale of the input image obtained through the scale converting unit does not become smaller than a predetermined threshold. Image object extraction device.   The calculation area cutout means cuts out a partial image of the attention area and a partial image of the context area with fixed values from the input image obtained through the scale conversion means, and the context area is the attention area. 2. The image object extraction method according to claim 1, wherein the image of interest is cut out in a size enlarged by a predetermined amount so as to include information about the vertical and horizontal sides of the attention area. apparatus.   3. The image object extraction device according to claim 1, wherein the calculation region cutout unit cuts out the area of the context region so that the area of the context region satisfies an area that is greater than 1 time and equal to or less than 4 times the area of the attention area. apparatus.   The said 1st feature-value and the said 2nd feature-value are each represented by the feature map obtained by the feature-value calculation process of the same format, The Claim 1 characterized by the above-mentioned. An image object extraction device according to item 1.   The attention area feature calculation means and the context area feature calculation means are each based on an input image obtained through the scale conversion means by performing parallel processing based on a convolutional neural network as feature quantity calculation processing in the same format. The image object extraction device according to any one of claims 1 to 4, wherein a feature map in which a positional relationship between each of the first feature amount and the second feature amount is calculated.   A program for causing a computer to function as the image object extraction device according to any one of claims 1 to 5.