KR102167011B1 - An image traning apparatus extracting hard negative samples being used to training a neural network based on sampling and a threshold adjusting adaptively and a method performed by the image training apparatus - Google Patents

Abstract

The image learning apparatus according to an embodiment may extract a hard negative sample used for learning a neural network from a training image. The hard negative sample can be used to learn an undesirable result from the neural network as a result of recognizing an object. The hard negative sample may be determined among search regions sampled from the training image. The image learning apparatus may determine a class score, which is a probability that each of the sampled search regions corresponds to an object of the training image, and then determine a hard negative sample from among the search regions based on the determined class score. Among the hard negative samples, the number of hard negative samples used for learning of the neural network may be determined based on at least one of the number of positive samples, a preset threshold compared to the class score, and a preset ratio between the positive samples and the hard negative samples. have.

Description

An image learning apparatus for extracting a hard negative sample used to learn a neural network based on sampling and an adaptively changed threshold, and a method performed by the apparatus BASED ON SAMPLING AND A THRESHOLD ADJUSTING ADAPTIVELY AND A METHOD PERFORMED BY THE IMAGE TRAINING APPARATUS}

The present invention relates to an apparatus and method for identifying an object included in an image using a neural network.

The neural network is modeled by mathematical expressions of the characteristics of human biological neurons, and can be used to extract or identify objects included in a learning image. The machine supervised learning method is a method of learning a neural network by using a training image and truth data including information related to an object included in the training image. That is, as the neural network is trained using the truth data, a result of the neural network identifying an object from the training image may converge to the truth data corresponding to the training image.

In the process of learning the neural network, positive samples and negative samples, which are part of the training image, may be used. The positive sample is a part of a training image including an object, and the neural network may be trained to identify a region in which an object exists in the training image using the positive sample. The negative sample is a part of a training image that does not include an object or includes a part of the object, and the neural network may be trained to identify an image in which an object does not exist in the training image by using the negative sample. In general, the number of negative samples acquired from the training image may be much larger than the number of positive samples acquired from the training image. If the number of negative samples is greater than the number of positive samples, it may cause imbalance of data used for learning by the neural network.

The present invention proposes an image learning apparatus and method for extracting a hard negative sample used to learn a neural network based on sampling and an adaptively changed threshold.

The present invention proposes an image learning apparatus and method for determining a hard negative sample based on search regions obtained by sampling a training image.

The present invention proposes an image learning apparatus and method for selecting a search area used to determine a hard negative sample based on an adaptively changed threshold.

According to an embodiment, in an image learning method using a neural network, sampling a plurality of search regions from a learning image, wherein each of the plurality of search regions calculates a class score that is a probability corresponding to an object included in the learning image. Determining, among the plurality of search areas, identifying a hard negative sample having a class score greater than a preset threshold, and learning the neural network based on the identified hard negative sample, the An image learning method in which the threshold is adjusted based on the number of hard negative samples used for learning of the neural network is provided.

According to an embodiment, the step of identifying the hard negative sample comprises: an image learning of identifying the hard negative sample from among the plurality of search regions, among search regions that do not correspond to the positive samples used to learn the neural network. A method is provided.

According to an embodiment, the learning of the neural network includes: identifying a positive sample determined by comparing a location of the object and all regions of the training image; and, among the identified hard negative samples, a hard class score having a large class score Including the step of sequentially selecting a hard negative sample used for learning the neural network from the negative sample, the number of hard negative samples used for learning the neural network, the number of the identified positive samples and the positive samples and An image learning method is provided that is determined based on at least one of preset ratios between hard negative samples.

According to an embodiment, the step of determining whether to change the threshold based on the number of positive samples determined by comparing the location of the object and all regions of the training image and the number of identified hard negative samples is further performed. A video learning method including is provided.

According to an embodiment, the determining whether to change the threshold may include a value obtained by applying the number of positive samples to a preset ratio between the positive sample and the hard negative sample than the number of the identified hard negative samples. In large cases, an image learning method is provided for determining to change the threshold.

According to an embodiment, when the threshold is changed, an image learning method having a value smaller than a value used in the training image in a training image different from the training image is provided.

According to an embodiment, in the sampling step, an image learning method of sampling the plurality of search regions among the learning images based on a preset probability is provided.

According to an embodiment, determining a positive sample by comparing all regions of the learning image with the positions of the objects of the learning image, and based on a class score and a preset threshold of each of a plurality of regions sampled from the learning image, Determining a hard negative sample, wherein the class score is a probability that each of the plurality of regions corresponds to the object, and based on the determined hard negative sample and the determined positive sample, learning a neural network, An image learning method in which the threshold is adjusted according to a result of comparing the number of hard negative samples and the number of positive samples is provided.

According to an embodiment, the determining of the positive sample includes identifying a location of the object based on truth data corresponding to the training image, and each of all regions of the training image is a location of the identified object. An image learning method comprising the step of selecting the positive sample from all regions of the training image based on whether or not the degree of overlap with is equal to or greater than a threshold value is provided.

According to an embodiment, the determining of the hard negative sample includes identifying a negative sample based on whether a degree of overlapping with a position of the object is less than or equal to a threshold, among a plurality of regions sampled from the training image; and An image learning method comprising the step of determining a negative sample having a class score greater than the threshold value from among the identified negative samples as the hard negative sample is provided.

According to an embodiment, there is provided a method of learning an image including a plurality of regions sampled from the training image including a soft negative sample having a class score less than or equal to the threshold value and a hard negative sample having a class score greater than the threshold value.

According to an embodiment, based on (1) the number of regions having a class score greater than the threshold, among the plurality of regions, and (2) a preset ratio between positive samples and hard negative samples and the number of positive samples. There is provided an image learning method further comprising the step of determining whether to change the threshold value by comparing target values of the calculated hard negative samples.

According to an embodiment, the determining whether to change the threshold value comprises: when the target value is greater than the number of regions having a class score greater than the threshold value, image learning of determining to change the threshold value to a smaller value A method is provided.

According to an embodiment, the determining of the hard negative sample comprises, when the target value is greater than the number of regions having a class score greater than the threshold value, one or more regions having a class score greater than the threshold value, among the plurality of regions. An image learning method for determining a region as the hard negative sample is provided.

According to an embodiment, the determining of the hard negative sample may include, when the target value is smaller than the number of regions having a class score greater than the threshold value, from among the plurality of regions, the region having the largest class score, in descending order. There is provided an image learning method including extracting an area as much as the target value and determining the extracted areas as the hard negative sample.

According to an embodiment, in the image recognition method using a neural network, identifying an input image, inputting the input image to the neural network, and based on the output of the neural network, included in the input image Recognizing an object, wherein the neural network learns in advance one or more hard negative samples selected based on a preset threshold from among a plurality of search regions sampled from the training image, and the threshold value is the training image An image recognition method is provided that is adjusted based on at least one of the number of positive samples determined by comparing all regions of and the position of the object and the number of the selected hard negative samples.

According to an embodiment, the hard negative sample is a search region having a class score greater than the threshold among negative samples, which is a search region excluding a search region corresponding to the positive sample, among the plurality of search regions, and the class score A method of recognizing an image is provided in which the probability that each of the plurality of search regions corresponds to the object.

According to an embodiment, a hard negative sample used to learn a neural network may be extracted based on sampling and a threshold value that is adaptively changed.

According to an embodiment, a hard negative sample may be determined based on search regions obtained by sampling a training image.

According to an embodiment, a search area used to determine a hard negative sample may be selected based on an adaptively changed threshold.

1 is a flowchart illustrating an operation of learning a neural network using a training image by an image learning apparatus according to an exemplary embodiment.
2 is an exemplary diagram for explaining a positive sample and a hard negative sample used by an image learning apparatus to learn a neural network according to an embodiment.
3 is an exemplary diagram for explaining an operation of aligning a plurality of search areas by an image learning apparatus according to an exemplary embodiment.
FIG. 4 is an exemplary diagram for explaining an operation of selecting a hard negative sample from among a plurality of search areas by an image learning apparatus according to an exemplary embodiment.
5 is a flowchart illustrating an operation of recognizing an object present in an input image using a neural network learned by an image learning apparatus according to an exemplary embodiment.
6 is a diagram illustrating a structure of an image learning apparatus according to an exemplary embodiment.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in this specification are exemplified only for the purpose of describing the embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention They may be implemented in various forms and are not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention can apply various changes and have various forms, the embodiments will be illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes changes, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are only for the purpose of distinguishing one component from other components, for example, without departing from the scope of rights according to the concept of the present invention, the first component may be named as the second component, Similarly, the second component may also be referred to as a first component.

When a component is referred to as being “connected” or “connected” to another component, it is understood that it may be directly connected or connected to the other component, but other components may exist in the middle. Should be. On the other hand, when a component is referred to as being “directly connected” or “directly connected” to another component, it should be understood that there is no other component in the middle. Expressions that describe the relationship between components, for example, “between” and “just between” or “directly adjacent to” should be interpreted as well.

The terms used in the present specification are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate that the specified features, numbers, steps, actions, components, parts, or combinations thereof exist, but one or more other features or numbers, It is to be understood that the presence or addition of steps, actions, components, parts or combinations thereof does not preclude the possibility of preliminary exclusion.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this specification. Does not.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these embodiments. The same reference numerals in each drawing indicate the same members.

1 is a flowchart illustrating an operation of learning a neural network using a training image by an image learning apparatus according to an exemplary embodiment. The neural network learned by the image learning apparatus may be used to recognize a training image or an object included in another image other than the training image.

Referring to FIG. 1, in step 101, the image learning apparatus according to an exemplary embodiment may identify a training image to be used for learning a neural network. The learning image may be transmitted from a network (eg, the Internet) connected to the video learning device or another electronic device (eg, a camera including an image sensor, a smartphone, etc.) connected to the video learning device. The image learning device may identify a training image transmitted from a network or another electronic device and stored in a memory of the image learning device. The training image may be divided into an area in which a subject (ie, an object) exists and an area in which the subject does not exist (eg, a background or another subject separated from the subject).

Referring to FIG. 1, in step 102, the image learning apparatus according to an embodiment may sample a search area that is a part of a training image. The image learning apparatus may determine a search area by dividing a portion of the training image. The image learning apparatus may sample the search area from the training image based on a preset probability (eg, 1/k (k is a real number greater than 1)). One or more search regions may be selected from the training image based on the probability. When a plurality of search regions are sampled, the sizes of the search regions may be different.

Referring to FIG. 1, in step 103, the image learning apparatus according to an exemplary embodiment may determine a class score, which is a probability corresponding to an object included in the training image in which one or more selected search regions correspond. That is, the class score may be a value indicating a probability that a corresponding search area corresponds to an object. When a plurality of search regions are sampled, the image learning apparatus may determine a class score for each of the search regions. Since the search area is a sample of a part of the training image, the video learning apparatus may not calculate a class score in all areas of the training image. Accordingly, the amount of calculation required for the video learning apparatus to calculate the class score can be reduced.

Referring to FIG. 1, in step 104, the image learning apparatus according to an embodiment may select and store a search region having a class score greater than a preset threshold θ from among one or more search regions. The threshold value θ is a real number of 0 or more and 1 or less, and, for example, the initial value may be determined as a real number between 0.5 and 0.9. The threshold value θ may be adaptively changed every time a neural network is trained using different training images. The image learning apparatus may select only negative samples from among the search regions having a class score greater than the threshold value θ. The negative sample is a part of a training image that does not include an object or includes a part of the object, and may be used to learn to the neural network that the object is not included in the negative sample. In step 102, when the image learning apparatus selects a plurality of search regions, one or more search regions having a class score greater than the threshold value θ may be selected.

The negative sample may be classified into a hard negative sample having a class score greater than the threshold value θ and a soft negative sample having a class score less than the threshold value θ. Accordingly, the search region stored by the image learning apparatus in step 104 may be a search region corresponding to the hard negative sample.

Referring to FIG. 1, in step 105, the image learning apparatus according to an embodiment may group stored search regions. Furthermore, when a plurality of search regions are selected and stored, the image learning apparatus may arrange the stored search regions based on a corresponding class score order. For example, a plurality of search areas stored in the video learning apparatus may be arranged in descending order of class scores.

Referring to FIG. 1, in step 106, the image learning apparatus according to an exemplary embodiment may identify a positive sample used to learn a region in which an object exists in the training image from the neural network as a part of the training image. . In other words, the positive sample is a part of the training image including the object, and may be used to learn to the neural network that the object is included in the positive sample. The image learning apparatus may identify one or more positive samples by searching all regions of the training image. More specifically, the image learning apparatus may determine one or more positive samples by comparing the location of the object and all regions of the training image. Hereinafter, it is assumed that the image learning apparatus has identified N _p positive samples.

The search region having a class score greater than the threshold value θ stored in step 104 may be selected from the remaining search regions excluding the search region corresponding to the positive sample from among the plurality of search regions. That is, the search region sampled from the training image may be used to extract only hard negative samples. The positive sample is determined by searching all regions of the training image, but the negative sample may be determined based on the search region sampled from the training image. Accordingly, the time and computation amount required to determine a negative sample can be reduced.

Referring to FIG. 1, in step 107, the image learning apparatus according to an embodiment may determine the number N _n of hard negative samples to be used for learning a neural network based on the number of positive samples. A ratio between the number of positive samples N _p and the number of hard negative samples N _n may be preset. In this case, the number of hard negative samples N _n may be determined based on a preset ratio and the number N _p of positive samples identified in step 106. For example, when the ratio between the number of positive samples N _p and the number of hard negative samples N _n is previously determined as 1:m, the number N _n of hard negative samples may be mN _p . In this case, the number N _n of hard negative samples may be proportional to the number N _p of positive samples.

Referring to FIG. 1, in step 108, the image learning apparatus according to an embodiment may compare the number of search regions grouped in step 105 with the number of hard negative samples N _n determined in step 107. have. The image learning apparatus may select a hard negative sample to be used to learn a neural network from among the grouped search regions, based on the class score and the number of hard negative samples N _n . That is, N _n may be a target value of a hard negative sample to be used for learning a neural network.

More specifically, when the number of grouped search regions is greater than the target value Nn of the hard negative sample, in step 109, the video learning apparatus according to an embodiment is based on a class score corresponding to each of the grouped search regions. , It is possible to select a hard negative sample from among the grouped search areas. The number of search regions selected as the hard negative samples may correspond to the target value Nn of the hard negative samples.

For example, the image learning apparatus may select Nn search regions having a relatively high class score as hard negative samples. When a plurality of stored search areas are sorted based on the class score order, for example, when the plurality of search areas are arranged and stored in descending order of the class score, the video learning apparatus is the Nn-th search area from the first stored search area. Up to Nn search regions can be selected as hard negative samples.

When the number of grouped search regions is less than the target value Nn of the hard negative sample, in step 110, the image learning apparatus according to an embodiment may adjust the threshold value θ used to select the search region. The threshold value θ may be changed corresponding to α × θ to which a preset real number α is applied. For example, the image learning apparatus may reduce the size of the threshold value θ so that a larger number of hard negative samples may be selected when learning the next training image. In this case, α applied to the threshold value θ may be a positive real number less than 1 (eg, 0.99). Accordingly, for the next training image, the image learning apparatus may use a larger number of hard negative samples to learn the neural network.

When the number of grouped search regions is smaller than the target value Nn of the hard negative samples, in step 111, the image learning apparatus according to an embodiment may select all of the search regions grouped in step 105 as hard negative samples. . The order of steps 110 and 111 may be different from that shown in FIG. 1, and the image learning apparatus may independently perform steps 110 and 111.

Referring to FIG. 1, in step 112, the image learning apparatus according to an embodiment may train a neural network using a selected hard negative sample and an identified positive sample. Accordingly, the neural network may be trained to identify a region in which an object exists in the training image based on the positive sample. Also, the neural network may identify a region in which an object does not exist in the training image based on the hard negative sample. As described above, since the hard negative sample is a negative sample having a class score greater than the threshold value θ, it can be used to learn an undesirable result of recognizing an object by the neural network.

The hard negative sample used to train the neural network may be a part of a training image having a class score equal to or greater than the threshold value θ. That is, the hard negative sample used to learn the neural network may be a negative sample that is relatively difficult to determine that the neural network correctly recognized the object among a plurality of negative samples that can be extracted from the training image. Since the image learning apparatus according to an embodiment trains a neural network by selecting only a hard negative sample, the neural network may be used to more accurately identify an object.

In addition, for the imbalance due to the difference in the number of positive samples and the number of negative samples, the number of hard negative samples used for learning a neural network is limited to a maximum of N _n , so that the time required to learn a neural network can be reduced. I can. Furthermore, instead of calculating the class scores of all regions of the learning image, a hard negative sample is used for a portion of the sampled training image (e.g., the search regions sampled in step 102). Since it is determined by calculating the class score, it is possible to reduce the amount of time and computation required to determine the hard negative sample.

FIG. 2 is an exemplary diagram for explaining positive samples 220 and 230 and hard negative samples 210 used by an image learning apparatus to learn a neural network according to an embodiment.

The image learning apparatus may determine a hard negative sample 210 to be used to learn a neural network from a search area obtained by sampling the training image 200 based on a preset probability. Referring to FIG. 2, the hard negative sample 210 may include a background of the training image 200. Alternatively, the hard negative sample 210 may include an object other than an object to be recognized through a neural network. Since the hard negative sample 210 is determined as a part of the training image having a class score greater than the threshold value θ or greater than the threshold value θ, it is relatively easy to determine the area in which the object exists. May be part.

The image learning apparatus may determine the positive samples 220 and 230 based on the class score determined in all areas of the training image 200. When the image learning apparatus identifies the truth data 240 corresponding to the training image 200, the positive samples 220 and 230 may be determined in consideration of the truth data 240. The truth data 240 may include information (eg, coordinate information) indicating a position in the training image 200 of an object to be recognized through a neural network.

For example, the image learning apparatus compares all regions of the learning image 200 with the positions of the objects identified from the truth data 240, and determines a specific region whose degree of overlapping with the object is equal to or greater than a preset threshold value by using the positive sample 220, 230). The degree of overlap with the object may be determined based on, for example, Intersection-of-Union (IOU). For example, if the IOU, which is the degree to which the specific area and the object overlap, is 0.5 or more, the video learning apparatus may determine the specific area as the positive samples 220 and 230. In summary, among all the regions of the training image 200, a region that relatively much overlaps with the position of the object identified from the truth data 240 may be determined as the positive samples 220 and 230. Alternatively, the image learning apparatus may determine, as a positive sample, one area that most closely overlaps the location of the object indicated by the truth data 240. Alternatively, the video learning apparatus determines whether each of the areas of the training image 200 overlaps with the location of the identified object (eg, can be evaluated by the IOU) is equal to or greater than a threshold (eg, 0.5). Based on whether or not, a positive sample can be selected.

When the neural network is learned to identify a region in which an object exists in the training image 200, the negative sample 210 may indicate an incorrect identification result that is not desired to be obtained through the neural network. For example, when comparing the positive samples 220 and 230 and the hard negative samples 210, the size of the area where the hard negative sample 210 and the object overlap is the area where the positive samples 220 and 230 and the object overlap. May be relatively smaller than the size of. That is, the hard negative sample 210 may represent a result of identifying the area where the object is present in less accurate than that of the positive samples 220 and 230. As the neural network learns the positive samples 220 and 230 as well as the hard negative samples 210, the neural network can more accurately identify a region in which an object exists in the training image 220.

3 is an exemplary diagram for explaining an operation of aligning a plurality of search areas by an image learning apparatus according to an exemplary embodiment.

Referring to FIG. 3, a plurality of search regions (search region 1 310 to search region 4 340) obtained by the image learning apparatus are illustrated. The search regions may be obtained by sampling the input image 300. The image learning apparatus may determine a class score, which is a probability that a plurality of search regions correspond to an object included in the learning image 300. For example, the class score of search area 1 (310) is 0.21, the class score of search area 2 (320) is 0.32, the class score of search area 3 (330) is 0.12, and the class score of search area 4 (340) is It is assumed to be 0.57.

The image learning apparatus may remove a search region (ie, a soft negative sample) having a class score less than or equal to a preset threshold (eg, threshold θ in FIG. 1) from among the plurality of search regions. For example, when the threshold is 0.2, the search area 3 330 having a class score of 0.12 may be removed. The image learning apparatus may perform clustering on search regions having a class score exceeding a threshold. While the image learning apparatus performs clustering, a search area having a relatively low class score may be removed. After performing clustering, the image learning apparatus may sort and store the remaining search regions based on the class score. Accordingly, a search area (ie, a hard negative sample) having a class score greater than the threshold value θ may be stored in the image learning apparatus.

After clustering is performed, it is assumed that search area 1 (310), search area 2 (320), and search area 4 (340) remain. The video learning apparatus may arrange the search regions in the order of the search region 4 340, the search region 2 320, and the search region 1 310 in the order of the highest class score. The image learning apparatus may select one or more search regions from among a plurality of sorted search regions based on the number of positive samples identified from the training image 300 and use them for learning the neural network. The number of search regions (ie, hard negative samples) selected by the video learning apparatus may be proportional to the number of positive samples.

FIG. 4 is an exemplary diagram for explaining an operation of selecting a hard negative sample from among a plurality of search areas by an image learning apparatus according to an exemplary embodiment.

Referring to FIG. 4, a plurality of search areas extracted from the training image 400 by the video learning apparatus are illustrated. The plurality of search regions may be regions sampled from the training image 400 based on a preset probability. The video learning apparatus may determine class scores of a plurality of sampled search regions. In other words, since the determination of the class score is performed in a plurality of sampled search regions instead of all regions of the training image 400, the amount of calculation required to determine the class score can be reduced.

When the class score of each of the plurality of search areas is determined, the video learning apparatus may extract one or more search areas having a class score exceeding a preset threshold value θ from the plurality of search areas. Referring to FIG. 4, search areas 420, 430, 440, 450, and 460 may have a class score exceeding a preset threshold θ. The image learning apparatus may discard the remaining search regions except for the search regions 420, 430, 440, 450, and 460 (discard). The image learning apparatus may cluster or align the search regions 420, 430, 440, 450, and 460. The clustering or alignment performed by the image learning apparatus on the search regions 420, 430, 440, 450, and 460 is similar to that described with reference to FIG. 3, and a detailed description thereof is omitted.

The image learning apparatus may perform the above-described clustering or alignment on the remaining search regions, except for the search region corresponding to the positive sample, among the plurality of search regions. Accordingly, the search regions 420, 430, 440, 450, and 460 may be hard negative samples having a class score exceeding a preset threshold value θ.

The image learning apparatus may use at least one of the search regions 420, 430, 440, 450, and 460 having a class score exceeding a preset threshold value θ to learn the neural network. Before determining at least one of the search regions 420, 430, 440, 450, and 460 as a hard negative sample to be used for learning a neural network, the image learning apparatus determines one positive sample 410 in the training image 400. You can decide more. An operation of determining one or more positive samples 410 from the training image 400 by the video learning apparatus is similar to that described with reference to FIG. 2, and thus a detailed description thereof will be omitted.

Referring to FIG. 4, it is assumed that the image learning apparatus extracts one positive sample 410 from the training image 400. The number of hard negative samples used for learning of a neural network among the search areas 420, 430, 440, 450 and 460 may be determined based on the number of positive samples 410. For example, so that the ratio between the number of hard negative samples and the number of positive samples 410 used for learning a neural network satisfies a preset ratio, the video learning apparatus includes search areas 420, 430, 440, and 450 , 460), a hard negative sample used for learning a neural network may be selected.

For example, when the ratio is 1:3, since the number of positive samples 410 is 1, the image learning apparatus includes five search areas 420, 430, and 440 having a class score exceeding the threshold value θ. , 450, and 460), three search regions may be selected and used for learning a neural network. The video learning apparatus selects three search regions in descending order from the search region having the largest class score based on the class scores of each of the search regions 420, 430, 440, 450, and 460 to use for learning of the neural network. I can.

The number of positive samples 410 or the number of hard negative samples determined based on a preset ratio is greater than or equal to the number of search regions 420, 430, 440, 450, 460 having a class score exceeding the threshold value θ. In this case, the image learning apparatus may use all of the search regions 420, 430, 440, 450, and 460 for learning of the neural network. For example, when the ratio is 1:6, since the number of positive samples 410 is 1, the image learning apparatus may determine six search regions as the number of hard negative samples to be used for learning a neural network. Since the number of search regions 420, 430, 440, 450, 460 is smaller than the determined number of hard negative samples, the image learning apparatus replaces all of the search regions 420, 430, 440, 450, 460 of the neural network. It can be determined as a hard negative sample to use for learning. Accordingly, although the number of hard negative samples is determined to be 6, the final number of search regions or hard negative samples used for learning a neural network may be 5.

The number of positive samples 410 or the number of hard negative samples determined based on a preset ratio is greater than or equal to the number of search regions 420, 430, 440, 450, 460 having a class score exceeding the threshold value θ. In this case, the image learning apparatus may change the threshold value θ. For example, after learning of the training image 400 is completed, when learning a new training image, the threshold value θ may be changed to a smaller value so that the number of hard negative samples corresponding to the new training image increases. have. In other words, if the threshold applied to the training image 400 is θ and the threshold applied to the new training image is θ', for a real number α less than 1 (for example, 0.99), θ'= It may be α × θ.

The positive sample 410 and one or more hard negative samples selected by the above-described operation may be used together with the training image 400 to learn a neural network. While the image learning apparatus learns the neural network by using a plurality of training images including the training image 400, the size of the threshold value θ sequentially applied to the plurality of training images may be gradually decreased. Accordingly, the class score of the hard negative sample determined from the plurality of training images may also be gradually decreased according to the size of the threshold value θ. Therefore, the accuracy of the neural network can converge rapidly.

5 is a flowchart illustrating an operation of recognizing an object present in an input image using a neural network learned by an image learning apparatus according to an exemplary embodiment. The neural network learned by the image learning apparatus according to an embodiment may be used to recognize input images in various application fields, such as, for example, intelligent cars, image security devices, games, and robots. Hereinafter, an operation of recognizing an input image by using a neural network learned by the image learning apparatus according to an exemplary embodiment by the image recognition apparatus according to an exemplary embodiment will be described. The image recognition device can be applied to intelligent cars, image security devices, games, and robots. The image learning apparatus may also recognize an object included in the input image based on the neural network learned by the operation of FIGS. 1 to 4.

Referring to FIG. 5, in step 510, the image recognition apparatus according to an embodiment may identify an input image. The input image may be received through a network connected to the image recognition device and may be stored in a memory of the image recognition device. Alternatively, it may be transmitted from another electronic device (eg, a camera including an image sensor, a smartphone, etc.) connected to the image recognition device and stored in a memory of the image recognition device. The image recognition device may identify an input image stored in the memory.

Referring to FIG. 5, in step 520, the image recognition apparatus according to an embodiment may pre-process the input image in order to input the input image to the neural network. The intensity, size, etc. of the input image may be changed in consideration of the input of the neural network, and feature vectors related to the input image may be extracted.

Referring to FIG. 5, in step 530, the image recognition apparatus according to an embodiment may input a preprocessed input image to a pre-learned neural network. The neural network may be learned by the operation of the image learning apparatus described in FIGS. 1 to 4. The neural network may include an input layer including a plurality of nodes, one or more hidden layers, and an output layer. Information related to the input image (eg, the feature vector) may be input as an input layer.

Referring to FIG. 5, in step 540, the image recognition apparatus according to an embodiment may determine a region in which an object exists in an input image based on an output of a neural network. In other words, the image recognition apparatus may recognize an object included in the input image based on the output of the neural network. The neural network may include an output layer including a plurality of nodes. The image recognition apparatus may obtain an output value of a node included in the output layer. The image recognition apparatus may determine a region in which the object exists in the input image based on the obtained output value.

Since the neural network learns the positive sample and the hard negative sample by the operation of the image learning apparatus described in FIGS. 1 to 4, the image recognition apparatus may obtain information related to an area where an object exists in the input image from the neural network. . The image recognition apparatus may display and output an area where an object exists in the input image as a bounding box based on the output of the neural network. While displaying and outputting an area where an object exists in the input image as a bounding box, the image recognition apparatus may output a probability that the object exists in the bounding box. Alternatively, the image recognition apparatus may extract and output a region having a high probability of the existence of an object from the input image.

In order to apply an image recognition device using a neural network to intelligent cars, image security devices, games, and robots, it is necessary to develop and verify a neural network optimized for the environment in the field while adjusting various parameters related to the neural network. . As the image recognition apparatus according to an embodiment uses the neural network learned by the image learning apparatus, time required for development and verification of the neural network may be reduced. Accordingly, the time required to develop the image recognition device can be shortened.

6 is a diagram illustrating a structure of an image learning apparatus according to an exemplary embodiment.

Referring to FIG. 6, an image learning apparatus according to an embodiment may include a memory 610 and a processor 620. The memory 610 and the processor 620 may communicate with each other through a bus 630.

The memory 610 may store instructions that can be read by a computer. A learning image used to learn the neural network, search regions sampled from the learning image, and a class score corresponding to each of the search regions may be stored in the memory 610. When the image learning apparatus sorts search regions greater than or equal to the threshold, search regions greater than or equal to the threshold may be stored in the memory 610 in descending order of the class score. In addition, parameters related to the neural network may be stored in the memory 610.

The processor 620 may perform the above-described operations as the instructions stored in the memory 610 are executed by the processor 620. The memory 610 may be a volatile memory such as a random access memory (RAM) or a nonvolatile memory such as a hard disk drive (HDD) or a solid state drive (SSD).

The processor 620 is a device that executes instructions or programs, or controls an image learning apparatus, and may include, for example, a CPU (Central Processing Unit) and a GPU (Graphic Processing Unit). The video learning device is connected to an external device (for example, an image capturing device, a personal computer, or a network) through an input/output device (not shown), and can exchange data. For example, the image learning apparatus may receive a learning image through an image sensor. The video learning device is implemented as at least a part of computing devices such as personal computers, tablet computers, netbooks, mobile devices such as mobile phones, smart phones, PDAs, tablet computers, and laptop computers, or electronic products such as smart televisions and security devices for gate control. Can be.

The processor 620 samples a plurality of search areas from the training image based on a preset probability, determines a class score indicating whether each of the plurality of search areas corresponds to an object included in the training image, and among the plurality of search areas, Search regions having a class score greater than a preset threshold may be selected, and the selected search regions may be sorted according to the class score and stored in the memory 610. As a part of the training image, the processor 620 may identify a positive sample used to learn the neural network a region in which an object exists in the training image, and based on the number of identified positive samples and a class score, the memory ( Among the search regions stored in 610), a hard negative sample used to learn a region in which an object does not exist in the training image by the neural network may be selected. The processor 620 may learn a neural network based on the selected hard negative sample and positive sample.

Since the above-described matters through FIGS. 1 to 5 are applied to each of the components shown in FIG. 6 as they are, a more detailed description will be omitted.

In summary, the image learning apparatus according to an embodiment may extract a hard negative sample used for learning a neural network from a training image. The hard negative sample can be used to learn to the neural network an undesirable result of recognizing an object. The hard negative sample may be determined among search regions sampled from the training image according to a preset probability. The image learning apparatus may determine a class score that is a probability that the sampled search regions correspond to an object, and then determine a hard negative sample to be used for learning a neural network among the search regions based on the determined class score. The number of hard negative samples to be used for learning of the neural network may be determined based on at least one of the number of positive samples, a preset threshold compared to the class score, and a ratio between the preset positive samples and the hard negative samples. Accordingly, it is possible to reduce the amount of time and computation required to align the hard negative samples.

Since a search region having a class score equal to or greater than the threshold is determined as a hard negative sample, a hard negative sample used for learning of a neural network may correspond to a search region where it is relatively difficult to determine whether an object exists. Since the image learning apparatus determines a hard negative sample from among the search regions sampled from the training image according to a preset probability, time required to determine the hard negative sample may be reduced.

The number of search regions determined as hard negative samples may not exceed the number of search regions having a class score exceeding the threshold. When it is determined that the number of search areas determined as hard negative samples is insufficient, the threshold may be changed to a smaller value. The video learning apparatus may determine whether the number of search regions determined as hard negative samples is insufficient based on the number of positive samples and a ratio between a preset positive sample and a hard negative sample. When the image learning apparatus sequentially uses a plurality of training images for learning a neural network, each time a hard negative sample is extracted from the training image, the threshold may be adaptively changed. Using the adaptively changed threshold and the search region sampled according to the probability, the image learning apparatus according to an embodiment may learn a neural network more quickly without deteriorating performance.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments are, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, such as one or more general purpose computers or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodyed in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description by those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the claims to be described later.

Claims (17)

In the image learning method using a neural network,
Sampling, by an image learning apparatus, a plurality of search regions from the training image;
An image learning apparatus compares whether each of the plurality of search regions corresponds to a position of an object included in the learning image with respect to each of the plurality of search regions, so that each of the plurality of search regions is an object included in the learning image. Determining a class score that is a probability corresponding to and;
Extracting, by an image learning apparatus, one or more search regions having a class score exceeding a preset threshold from among the plurality of search regions;
Clustering the extracted one or more search regions by an image learning apparatus;
Arranging, by an image learning apparatus, the one or more search regions according to an order of class scores determined in each of the clustered one or more search regions to select a search region used for learning a neural network;
Identifying, by an image learning apparatus, one or more positive samples used to learn, by a neural network, an area in which the object exists in the training image by comparing the location of the object in the training image and all areas of the training image;
An image learning apparatus is a neural region in which an object does not exist in the learning image among the sorted one or more search regions based on the number of the identified one or more positive samples and the class score determined for each of the aligned one or more search regions. Selecting one or more search areas used for network learning;
Identifying, by an image learning apparatus, a hard negative sample for each of the selected one or more search areas; And
Learning, by an image learning device, the neural network based on the identified one or more hard negative samples
Including,
The threshold is,
An image learning method that is adjusted based on the number of hard negative samples used for learning of the neural network. The method of claim 1,
Identifying the hard negative sample,
An image learning method for identifying the hard negative sample from among the plurality of search regions, among the remaining search regions in which an object is not included except for a search region corresponding to a positive sample used to learn the neural network. The method of claim 1,
Learning the neural network,
Identifying a positive sample determined by comparing the location of the object and all regions of the training image; And
Selecting a hard negative sample used for learning the neural network sequentially from the hard negative sample having a large class score among the identified hard negative samples
Including,
The number of hard negative samples used for learning of the neural network is,
An image learning method that is determined based on at least one of the identified number of positive samples and a preset ratio between a positive sample and a hard negative sample. The method of claim 1,
Determining whether to change the threshold based on the number of positive samples determined by comparing the location of the object and all regions of the training image and the number of identified hard negative samples
Video learning method further comprising a. The method of claim 4,
The step of determining whether to change the threshold value,
When a value obtained by applying the number of positive samples to a preset ratio between the positive samples and the hard negative samples is greater than the identified number of hard negative samples, it is determined to change the threshold. The method of claim 5,
The threshold is,
When the threshold value is changed, the learning image different from the training image has a value smaller than the value used in the training image. The method of claim 1,
The sampling step,
An image learning method for sampling a plurality of search regions having different sizes to divide a portion of the training image based on a preset probability determined as 1/k (k is a real number greater than 1). Determining, by an image learning apparatus, a positive sample by comparing all regions of the training image with positions of objects in the training image; And
Determining, by the video learning apparatus, a hard negative sample based on a class score of each of the plurality of regions in which the learning image is sampled and a preset threshold value-The class score is for each of the plurality of search regions, each of the plurality of search regions It is a probability that each of the plurality of regions corresponds to an object included in the training image by comparing whether it corresponds to the position of the object included in the training image; And
Learning, by an image learning device, a neural network based on the determined hard negative sample and the determined positive sample
Including,
The threshold is,
It is adjusted according to a result of comparing the number of hard negative samples and the number of positive samples,
The step of determining the hard negative sample,
The video learning apparatus extracts one or more search regions having a class score exceeding a preset threshold from among the plurality of search regions,
The image learning apparatus clusters the extracted one or more search regions,
The image learning apparatus arranges the one or more search regions in accordance with an order of class scores determined in each of the clustered one or more search regions in order to select a search region used for learning a neural network,
The video learning apparatus compares the position of the object in the training image and all regions of the training image to identify one or more positive samples used to learn the search for the existence of the object in the training image in a neural network,
An image learning apparatus is a neural region in which an object does not exist in the learning image among the sorted one or more search regions based on the number of the identified one or more positive samples and the class score determined for each of the aligned one or more search regions. Select one or more search areas used for learning of the network,
An image learning method in which an image learning apparatus identifies hard negative samples for each of the selected one or more search regions. The method of claim 8,
The step of determining the positive sample,
Identifying a location of the object based on truth data corresponding to the learning image; And
Selecting the positive sample from among all the regions of the learning image based on whether all regions of the learning image overlap with the position of the identified object equal to or greater than a threshold.
Video learning method comprising a. The method of claim 8,
The step of determining the hard negative sample,
Identifying a negative sample based on whether or not a degree of overlapping with the position of the object is less than or equal to a threshold value from among the plurality of regions sampled from the training image; And
Among the identified negative samples, determining a negative sample having a class score greater than the threshold as the hard negative sample
Video learning method comprising a. The method of claim 8,
A plurality of regions sampled from the training image,
A soft negative sample having a class score less than or equal to the threshold; And
Hard negative sample having a class score greater than the threshold
Video learning method comprising a. The method of claim 8,
Among the plurality of regions, the number of regions having a class score greater than the threshold value, and (2) a preset ratio between positive samples and hard negative samples and a target value of hard negative samples calculated based on the number of positive samples are compared Thus, determining whether to change the threshold value
Video learning method further comprising a. The method of claim 12,
The step of determining whether to change the threshold value,
When the target value is greater than the number of regions having a class score greater than the threshold value, it is determined to change the threshold value to a smaller value. The method of claim 12,
The step of determining the hard negative sample,
When the target value is greater than the number of regions having a class score greater than the threshold value, one or more regions having a class score greater than the threshold value from among the plurality of regions are determined as the hard negative samples. The method of claim 12,
The step of determining the hard negative sample,
If the target value is smaller than the number of regions having a class score greater than the threshold value, extracting regions as many as the target value from the region having the largest class score from among the plurality of regions in descending order; And
Determining the extracted regions as the hard negative samples
Video learning method comprising a. In the image recognition method using a neural network,
Identifying, by an image learning device, an input image;
Inputting, by an image learning device, the input image to the neural network; And
Recognizing, by an image learning apparatus, an object included in the input image based on the output of the neural network
Including,
The neural network,
Among the plurality of search regions sampled from the learning image, the image learning apparatus compares with respect to each of the plurality of search regions whether each of the plurality of search regions corresponds to a position of an object included in the learning image. Each search area determines a class score, which is a probability corresponding to an object included in the training image,
The video learning apparatus extracts one or more search regions having a class score exceeding a preset threshold from among the plurality of search regions,
The image learning apparatus clusters the extracted one or more search regions,
The image learning apparatus arranges the one or more search regions in accordance with an order of class scores determined in each of the clustered one or more search regions in order to select a search region used for learning a neural network,
The video learning apparatus compares the location of the object in the training image and all areas of the training image to identify one or more positive samples used to learn the area where the object exists in the training image in a neural network,
An image learning apparatus is a neural region in which an object does not exist in the learning image among the sorted one or more search regions based on the number of the identified one or more positive samples and the class score determined for each of the aligned one or more search regions. Select one or more search areas used for learning of the network,
The video learning device identifies a hard negative sample for each of the selected one or more search areas,
The video learning device learns the identified one or more hard negative samples in advance,
The threshold is,
An image recognition method that is adjusted based on at least one of the number of positive samples determined by comparing all regions of the training image and the position of the object and the number of the selected hard negative samples. The method of claim 16,
The hard negative sample,
Among the plurality of search regions, a search region having a class score greater than the threshold value among negative samples, which is a search region excluding a search region corresponding to the positive sample,
The class score is an object included in the training image by comparing whether each of the plurality of search areas corresponds to a position of an object included in the training image for each of a plurality of discovery areas. The image recognition method that is the probability of corresponding to.

Images