KR102444929B1 - Method for processing image of object for identifying animal and apparatus thereof - Google Patents

Abstract

The present invention provides a method for effectively detecting an object for identifying a companion animal while decreasing computational complexity, and an apparatus thereof. According to the present invention, the method for processing an image of an object for identifying a companion animal comprises: a step of acquiring an image in which the companion animal is included; a step of generating characteristic area candidates for determining a species of the companion animal in the image; a step of setting a first characteristic area with a determined location and size based on a reliability value of each of the characteristic area candidates; a step of setting a second characteristic area including the object for identifying the companion animal in the first characteristic area; and a step of acquiring the image of the object in the second characteristic area. According to the present invention, the apparatus generates a broader final characteristic area by considering the reliability value of each of the plurality of characteristic area candidates in a process of determining a characteristic area for determining the species of the companion animal to enable more accurate detection by detecting the object for identifying the companion animal in the final characteristic area.

Description

METHOD FOR PROCESSING IMAGE OF OBJECT FOR IDENTIFYING ANIMAL AND APPARATUS THEREOF

The present invention relates to a method and apparatus for processing an image of an object for identification of a companion animal, and more particularly, an image processing method and apparatus for effectively detecting an object capable of identifying a companion animal, such as a dog's inscription is about

In modern society, there is an increasing demand for companion animals that can be relied on emotionally while living with people. Accordingly, there is an increasing need to manage and manage information on various companion animals in a database for health management of companion animals. In order to manage the companion animal, identification information of the companion animal, such as a human fingerprint, is required, and objects that can be used according to the companion animal may be defined. For example, in the case of puppies, since the inscriptions (the shape of the nose folds) are different, the inscriptions may be used as identification information for each puppy.

As shown in Fig. 1 (a), the method of registering the inscription is to photograph the face including the nose of the companion animal (S110) and store the image including the inscription in the database, such as registering a human fingerprint or face, and It is performed by the registration process (S120). In addition, the method of inquiring the inscription is as shown in (b) of FIG. 1, by photographing the inscription of the companion animal (S130), searching for the inscription matching the photographed inscription and related information (S140), and It may be performed by the process of outputting information matching the inscription (S150). As shown in FIG. 1 , through the process of registering and inquiring inscriptions of companion animals, each companion animal can be identified and information of the companion animal can be managed. The pet's inscription information is stored in a database and can be used as data for AI-based learning or recognition.

However, there are several problems in obtaining and storing the inscriptions of companion animals.

First, it may be difficult to recognize a photo depending on a photographing angle, focus, distance, size, environment, and the like. There have been attempts to apply human facial recognition technology to inscription recognition, but there is a problem that sufficient data is accumulated for human facial information, whereas sufficient data is not secured for inscription information of companion animals, resulting in a low recognition rate. Specifically, in order to perform AI-based recognition, learning data processed in a form that can be learned by a machine is required, but the inscriptions of companion animals are difficult to recognize because sufficient data is not accumulated.

In addition, an image with clear nose wrinkles is required for inscription recognition of companion animals, but unlike humans, companion animals do not know how to perform actions such as stopping for a moment, so it is not easy to obtain a clear image of nose wrinkles. For example, it is very difficult to obtain an image of an inscription of the desired quality because a dog constantly moves its face and sticks out its tongue. For example, referring to FIG. 2 , an image in which the wrinkles of the nose are clearly photographed as in the images at the bottom is required, but most of the images actually photographed have the wrinkles in the nose clearly photographed due to shaking, like in the images at the top. In many cases it has not been In order to solve this problem, a method of taking pictures with the dog's nose forcibly fixed is being considered, but it is evaluated as inappropriate because it forces the companion animal to perform a forced action.

The present invention provides an image processing method and apparatus capable of effectively detecting an object for identification of a companion animal while reducing computational complexity.

The problems to be solved of the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

A method for processing an image of an object for identification of a companion animal according to the present invention includes: acquiring an image including the companion animal; and generating feature region candidates for determining the species of the companion animal from the image a second feature comprising: setting a first feature region whose location and size are determined based on the reliability values of each of the feature region candidates; and an object for identifying the companion animal in the first feature region setting a region; and acquiring an image of the object in the second feature region.

According to the present invention, the generating of the feature region candidates includes generating a plurality of feature images hierarchically using an artificial neural network, and applying predefined boundary regions to each of the feature images to each boundary. The method may include calculating a probability value in which a companion animal of a specific species is located in the regions, and generating the feature region candidates in consideration of the probability value.

According to the present invention, the generating of the feature region candidates includes: selecting a first boundary region having the highest probability of corresponding to a specific animal species in the feature image; Calculating the degree of overlap with the first boundary area for the remaining boundary areas except for the probability values according to the order of the probability values, and including a boundary area with the degree of overlap greater than the reference degree of overlap in the feature area candidate of the feature image; may include.

According to the present invention, the degree of overlap may be defined as a ratio of intersection to union area of each of the first boundary region and the remaining boundary regions.

According to the present invention, the central point at which the first feature region is located may be determined by a weighted sum of reliability values for the central points of the feature region candidate.

According to the present invention, the width of the first feature region is determined by a weighted sum of reliability values for the widths of the feature region candidates, and the height of the first feature region is a confidence value for the heights of the feature region candidates. can be determined by the weighted sum of

An apparatus for processing an image of an object for identification of a companion animal according to the present invention includes a camera for generating an image including the companion animal, and an object for identification of the companion animal by processing the image provided from the camera. and a processor for generating an image. The processor generates feature region candidates for determining the species of the companion animal from the image, sets a first feature region whose location and size are determined based on reliability values of each of the feature region candidates, and A second feature region including an object for identifying the companion animal may be set in the feature region, and an image of the object may be acquired from the second feature region.

According to the present invention, the processor generates a plurality of feature images hierarchically using an artificial neural network, and applies predefined boundary regions to each of the feature images, so that a companion animal of a specific species is applied to each of the boundary regions. It is possible to calculate a location probability value and generate the feature region candidates in consideration of the probability value.

According to the present invention, the processor selects a first boundary region with the highest probability of corresponding to a specific animal species in the feature image, and includes: The degree of overlap with the first boundary area may be calculated according to the order of the probability values, and a boundary area having the degree of overlap greater than the reference degree of overlap may be included in the feature area candidate of the feature image.

According to the present invention, the degree of overlap may be defined as a ratio of intersection to union area of each of the first boundary region and the remaining boundary regions.

According to the present invention, the central point at which the first feature region is located may be determined by a weighted sum of reliability values for the central points of the feature region candidate.

According to the present invention, the width of the first feature region is determined by a weighted sum of reliability values for the widths of the feature region candidates, and the height of the first feature region is a confidence value for the heights of the feature region candidates. can be determined by the weighted sum of

According to the present invention, in the process of determining the feature region for determining the species of the companion animal, the final feature region of a wider region is generated by considering the reliability values of each of the plurality of feature region candidates, and thereafter, the companion animal within the final feature region is generated. It enables more accurate detection by detecting an object for identification of

Effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 shows a schematic procedure for AI-based management of companion animals.
2 illustrates examples of images not suitable for learning or identification and images suitable for learning or identification in AI-based management of companion animals.
3 shows a procedure for managing inscriptions of an AI-based companion animal to which the suitability determination of an object image for learning or identification according to the present invention is applied.
4 illustrates a procedure for detecting an object for identification of a companion animal in the companion animal management system according to the present invention.
5 shows an example of a UI (User Interface) screen for detecting an identification object of a companion animal to which the present invention is applied.
6 illustrates a process of detecting an object for identification of a companion animal according to the present invention.
7 shows a process for setting a feature region according to the present invention.
8 illustrates a process of deriving a feature region for determining a species of a companion animal according to the present invention.
9 is a flowchart of a method for processing an image of an object for identification of a companion animal.
10 is a block diagram of an apparatus for processing an image of an object for identification of a companion animal.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. The present invention may be embodied in many different forms and is not limited to the embodiments described herein.

In order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals are assigned to the same or similar elements throughout the specification.

In addition, in various embodiments, components having the same configuration will be described using the same reference numerals only in the representative embodiment, and only configurations different from the representative embodiment will be described in other embodiments.

Throughout the specification, when a part is "connected (or coupled)" with another part, it is not only "directly connected (or coupled)" but also "indirectly connected (or connected)" with another member therebetween. combined)" is also included. In addition, when a part "includes" a certain component, this means that other components may be further included, rather than excluding other components, unless otherwise stated.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In this document, the contents of extracting identification information by utilizing the shape of a dog's nose wrinkle (inscription) are mainly described, but in the present invention, the scope of companion animals is not limited to dogs, and, as a feature used as identification information, It is not limited and various physical characteristics of companion animals may be used.

As described above, since the inscription images of companion animals are not sufficient for AI-based learning or identification, and the inscription images of companion animals are likely to be of low quality, the inscription images are selectively selected for AI-based learning or identification. You need to store it in the database.

FIG. 3 shows a procedure for managing inscriptions of companion animals based on AI to which judgment of suitability of inscription images for learning or identification according to the present invention is applied. The present invention first determines whether the photographed inscription image is suitable as data for AI-based learning or identification after photographing the inscription of the companion animal, and if it is determined that it is suitable, it is transmitted to the server for AI-based learning or recognition and stored to be used as data for later learning or identification.

As shown in FIG. 3 , the inscription management procedure according to the present invention largely includes an inscription acquisition procedure and an inscription recognition procedure.

According to the present invention, when an inscription of a companion animal is newly registered, an image containing a companion animal is taken, and an image of the inscription is extracted from the face region of the companion animal, and in particular, the inscription image is used to identify or learn the companion animal. First determine whether it is suitable for When it is determined that the captured image is suitable for identification or learning, the image is transmitted to the server (artificial intelligence neural network) and stored in the database.

In the case of inquiring identification information of companion animals through inscriptions, in the same way, after shooting an image containing companion animals, the inscription image is extracted from the companion animal's face area. In particular, the inscription image is used for identification or learning of the companion animal. First determine whether it is suitable or not. When it is determined that the photographed image is suitable for identification or learning, the image is transmitted to the server and the identification information of the companion animal is extracted through matching with the previously stored inscription images.

In the case of the inscription registration procedure, a companion animal is photographed as shown in (a) of FIG. 3 (S305), and a face region (hereinafter described as a first feature region) is first detected in the photographed companion animal image (S310), Detects the area occupied by the nose within the face area (hereinafter referred to as the second characteristic area) and outputs an inscription image through a quality check on whether the captured image is suitable for learning or identification (S315), and the output image is It is transmitted to the server constituting the artificial neural network, stored and registered (S320).

In the case of the inscription inquiry procedure, a companion animal is photographed (S330), a face region is detected from the image of the companion animal (S335), and an area occupied by a nose is detected within the face region as shown in FIG. An inscription image is output through a quality check on whether the image is suitable for learning or identification (S315), which is similar to the inscription registration procedure. Thereafter, the process of searching for matching information by comparing the output inscription image with the previously stored and learned inscription images (S345) and outputting the search result (S350) are performed.

4 illustrates a procedure for detecting an object corresponding to a nose of a companion animal in the system for managing inscriptions of a companion animal according to the present invention.

Referring to FIG. 4 , an initial image is generated by first photographing a companion animal (S405), and a face region is first detected from the initial image (S410). Thereafter, the nose region is detected in consideration of the companion animal species within the face region (S415). Detecting the face region first and secondarily detecting the nose region can lower the computational complexity and improve detection accuracy than detecting the nose region considering all species through cascaded detection. Because. Thereafter, a quality check is performed to check whether the detected image of the nose area is suitable for identification or learning of a future inscription (S420), and if it is determined as a suitable image as a result of the quality check, the image is transmitted to the server to It may be used for identification or stored for future learning or identification (S425).

Also, according to the present invention, it is possible to control the camera to focus on the detected nose region so that the image of an object for identification of a companion animal, such as a dog's nose wrinkle (inscription), is not taken blurry (S430). This is to prevent the image quality from being deteriorated due to the nose being out of focus by making the camera focus on the nose area.

5 shows an example of a UI (User Interface) screen for obtaining an image of an inscription of a companion animal to which the present invention is applied. 5 shows a case for obtaining a dog's inscription among several companion animals.

Referring to FIG. 5 , it is determined whether the companion animal currently being photographed is a dog by identifying the species of the companion animal in the image being photographed. If the companion animal being photographed is not a dog, a phrase such as 'cannot find a dog' is output as shown in FIG. 5, and if the companion animal being photographed is a dog, a procedure for obtaining the dog's inscription is performed. In order to determine whether the companion animal being photographed is a dog, the face region of the companion animal included in the image is first extracted, and the image included in the face region is compared with the existing learned data to determine the species of the companion animal. can decide

Thereafter, after setting an area corresponding to the dog's nose on the dog's face, focusing on the region corresponding to the nose, and shooting is continued. As shown in FIG. 5 , even when the dog is continuously moving, the photographing may be performed by focusing on the nose while tracking the dog's nose. In this case, it may be determined whether the image of the dog's inscription in each captured image is suitable for identification or learning of a companion animal, and the degree of suitability may be output as shown in FIG. 5 . The degree of whether the inscription image is suitable for identification or learning of a companion animal can be calculated as a numerical value, and according to the numerical value for the fit, the lower the fit, the better the graphic is. can be output. When the image of the inscription of the dog is sufficiently obtained, the shooting may end and the identification information may be stored in the database together with the image of the inscription of the dog or the identification information of the dog may be output.

In the present invention, the face region of the companion animal is first detected, and then the nose region is detected within the face region. This is to reduce the difficulty of object detection while reducing computational complexity. In the process of capturing an image, an object other than an object to be detected or unnecessary or erroneous information may be included in the image. Therefore, in the present invention, it is first determined whether a desired object (the nose of a companion animal) exists in the image being photographed.

In addition, in order to identify the inscription of the companion animal, an image having a resolution higher than a certain level is required, but as the resolution of the image increases, the amount of computation for image processing increases. In addition, as the types of companion animals increase, there is a problem in that the computational difficulty of artificial intelligence further increases because a learning method is different for each type of companion animal. In particular, since similar types of animals have similar shapes (eg, a dog's nose and a wolf's nose are similar), classifying a nose with an animal type for a similar animal may have a very high computational difficulty.

Accordingly, the present invention uses a cascaded object detection method to reduce the computational complexity. For example, while photographing a companion animal, the face region of the companion animal is detected first, and the type of the companion animal is first identified. detect the area. This involves first performing the process of identifying the type of companion animal at low resolution with relatively low computational complexity, and then applying the object detection method determined according to the type of companion animal to maintain high resolution in the companion animal's face area and perform nose area detection. will be. Thus, the present invention can effectively detect the nose region of a companion animal while relatively reducing computational complexity.

6 illustrates an overall image processing process for identification of a companion animal according to the present invention. As shown in FIG. 6 , the method for processing an input image according to the present invention includes a step of receiving an input image from a camera (S605), a first pre-processing step of generating a primary processed image by adjusting the size of the input image ( S610), the first feature region detection step of detecting the position and species of the animal from the processed image generated in the first preprocessing step (S615), and the first feature value of the animal image from the results of the first feature region detection step A first post-processing step of extracting (S620), a step of determining a detector for detecting an object (eg, nose) for identification of a companion animal according to the species of the companion animal in the image processed through the first post-processing step ( S625), a second pre-processing step of adjusting the size of an image for image processing for identification of a companion animal (S630), and at least one second feature corresponding to an animal species that can be detected in the first feature detecting step, respectively A second post-processing step ( S640 ) of extracting a second feature value of the animal image corresponding to the region detection step ( S635 ) and each second feature region detection step is included.

A method for processing an image of an object for identification of a companion animal according to the present invention comprises the steps of: acquiring an image including a companion animal; generating feature region candidates for determining a species of a companion animal from the image; , setting a first feature region whose location and size are determined based on the reliability values of each of the feature region candidates, and setting a second feature region including an object for identifying a companion animal in the first feature region; , acquiring an image of the object in the second feature region. Hereinafter, an image processing method of an object for identification of a companion animal according to the present invention will be described with a focus on recognizing a dog's inscription.

first pretreatment step

The step of applying the first pre-processing to the original image ( S610 ) is a step of converting the image into a form suitable for object detection by adjusting the size, ratio, direction, etc. of the original image.

With the development of camera technology, the input image is mostly composed of millions to tens of millions of pixels, and it is not desirable to directly process such a large image. In order for object detection to work efficiently, preprocessing must be performed to make the input image suitable for processing. This process is mathematically performed as a coordinate system transformation.

It is clear that an arbitrary processed image can be generated by matching any four points in the input image to the four vertices of the processed image and performing an arbitrary coordinate system transformation process. However, when an arbitrary nonlinear transformation function is used in the coordinate system transformation process, inverse transformation to obtain the feature region of the input image from the bounding box obtained as a result of the feature region detector must be possible. For example, it is preferable to use an affine transformation (Affine Transformation), which linearly transforms four arbitrary points of an input image by matching four vertices of the processed image, because the inverse transformation process can be easily obtained.

As an example of a method of determining four arbitrary points in the input image, a method of using four vertices of the input image as it is may be considered. Alternatively, a method of adding a blank space to the input image or cutting out a part of the input image may be used so that the horizontal and vertical lengths are converted at the same ratio. Alternatively, various interpolation methods may be applied to reduce the size of the input image.

first feature region detection step

In this step, by first detecting the region where the companion animal exists and the species of the animal in the pre-processed image, the first characteristic region that can be used in the second characteristic region detection step to be described later is set, and each companion animal It aims to improve the final feature point detection performance by selecting the second feature region detector optimized for the animal species.

In this process, the object detection and classification method can be easily combined by any person with ordinary knowledge in the related field. However, since the method based on the artificial neural network is known to have superior performance compared to the conventional method, it is preferable to use a feature detection technique based on the artificial neural network as much as possible. For example, as shown in FIG. 7 , a feature detector of a single-shot multibox detection (SSD) type, which is an algorithm for detecting objects of various sizes with respect to one image, may be used in an artificial neural network.

The input image normalized according to the preprocessor described above is hierarchically generated from the first feature image to the nth feature image by the artificial neural network. That is, the step of generating a plurality of feature images hierarchically using an artificial neural network is performed. In this case, a method of extracting a feature image for each layer may be mechanically learned in the learning step of the artificial neural network.

The hierarchical feature image extracted in this way is combined with a list of predefined boxes corresponding to each layer to generate a list of bounding boxes, object types, and confidence values (probability values). That is, by applying predefined boundary regions (bounding boxes) to each of the feature images, a probability value of a specific species of companion animal being located in each of the boundary regions is calculated. This computational process may also be mechanically learned in the learning stage of the artificial neural network. For example, the result is returned in the format shown in Table 1 below. At this time, the number of species that the neural network can determine is determined in the neural network design stage, and when an object does not exist implicitly, that is, “background” is defined.

These result boxes are finally returned as object detection results in the image by merging overlapping result boxes through a non-maximum suppression (NMS) step. NMS is a process of deriving a final feature region from a plurality of feature region candidates, and the feature region candidates may be generated in consideration of the probability values shown in Table 1 according to the procedure shown in FIG. 7 .

A detailed description of this process is as follows.

1. Perform the following procedure for each species except for the background.

A. From the list of bounding boxes, exclude boxes whose species probability is lower than a certain threshold. If there are no boxes left, exit with no results.

B. In the bounding box list, the box with the highest probability of the species is designated as the first box (first bounding area) and excluded from the bounding box list.

C. For the rest of the bounding box list, each of the following processes is performed in the order of highest probability.

i. Calculate the degree of overlap with the first box. For example, an intersection over union area ratio may be used.

ii. If the overlap value is higher than a certain threshold, this box is a box overlapping the first box. Merge with the first box.

D. Add the first box to the list of result boxes.

E. If there are still boxes in the bounding box list, repeat from step C again for the remaining boxes.

For two boxes A and B, for example, the intersection to union area ratio can be effectively calculated as in Equation 1 below.

That is, according to the present invention, generating the feature region candidates includes selecting a first boundary region (first box) that has the highest probability of corresponding to a specific animal species in the feature image, and the first boundary region selected from the feature image. For the remaining boundary regions except (the first box), the intersection-to-union area ratio (IoU) with the first boundary region is calculated according to the order of the probability values, and the boundary region where the intersection-to-union area is larger than the reference area ratio of the feature image It may include including the feature region candidate.

A method of merging the first box and the overlapping box in the above process will be described as follows. For example, you can merge the first box by keeping it as it is and deleting the second box from the bounding box list (Hard NMS). Alternatively, the first box is maintained as it is, and the second box reduces the probability of a specific species by weighting it by a value between (0, 1), and if the attenuated result value is smaller than a specific threshold value, it is only deleted from the bounding box list. method (Soft NMS).

As an embodiment proposed by the present invention, as shown in Equation 2 below, a new method (Expansion NMS) for merging a first box (a first feature region candidate) and a second box (a first feature region candidate) according to a probability value ) can be used.

In this case, p ¹ and p ² are the probability values of the first box (first feature region candidate) and the second box (first feature region candidate), respectively, and C _(x,y) ¹ , C _(x,y) ² , C _(x,y) ⁿ represents the (x,y) coordinate value of the center point of the first box, the second box, and the merged box, respectively. In the same way, W ₁ , W ₂ , and W _n represent the horizontal width of the first box, the second box, and the merged box, respectively, and H ¹ , H ² , H ⁿ represent the vertical height. The probability value of the merged box may use the probability value of the first box. The first feature region derived by the extended NMS according to the present invention is determined in consideration of the reliability value at which a specific species is to be located in each of the feature region candidates.

That is, the central point (C _(x,y) ⁿ ) at which the first feature region is located is the center point (C _(x,y) ¹ , C _(x,y) ² ) of the feature region candidate as shown in Equation 2 It can be determined by the weighted sum of the reliability values (p ¹ , p ² ).

In addition, the width (W _n ⁾ of the first feature region is determined by the weighted sum of the reliability values (p ¹ , p ² ) for the widths (W ₁ , W ₂ ) of the feature region candidates as in Equation 2, The height H ⁿ of the first feature region may be determined by a weighted sum of reliability values p ¹ , p ² with respect to the heights H ¹ , H ² of the feature region candidates.

By creating a new box according to the above embodiment, a box having a larger width and width is obtained as compared to the existing Hard-NMS or Soft-NMS method. According to the present embodiment, it is possible to add a certain part of a blank in the preprocessing detection for performing the multi-stage detector. By using the expansion NMS according to the present invention, this blank can be adaptively determined.

8 shows an example of detecting a feature region of a companion animal by applying the extended NMS according to the present invention compared to the existing NMS. (a) of FIG. 8 shows a plurality of feature region candidates generated from the original image, (b) of FIG. 8 is an example of a first feature region derived by the conventional NMS, and (c) of FIG. An example of the second feature area derived by applying the extended NMS according to the present invention is shown. As shown in (b) of FIG. 8, the conventional NMS (Hard NMS, Soft NMS) selects one box (feature area candidate) with the highest reliability from among a plurality of boxes (feature area candidates). In the process of detecting the second feature region, there is a possibility that a region necessary to obtain an inscription, such as a nose region, may deviate.

Accordingly, the present invention applies a weighted average based on the reliability value to a plurality of boxes (feature area candidates) to convert one box having a large width and height as shown in FIG. face region), and a second feature region (nose region) for identification of a companion animal may be detected within the first feature region. By setting a more extended first feature region as in the present invention, it is possible to reduce the occurrence of an error in which a subsequent second feature region is not detected.

Finally, it goes without saying that the feature region in the original image can be obtained by performing an inverse transformation process with respect to an arbitrary transformation process used in the pre-processing step for one or a plurality of bounding boxes determined in this way. Depending on the configuration, by adding a certain amount of blank space to the feature region in the original image, the second detection step, which will be described later, may be adjusted to be well performed.

first post-processing step

The first feature value may be generated by performing an additional post-processing step on each feature region of the input image obtained in the step S720 of setting the first feature region described above. For example, in order to obtain brightness information (first feature value) on the first feature region of the input image, an operation as in Equation 3 below may be performed.

In this case, L is the Luma value according to the BT.601 standard, and V is the brightness value defined in the HSV color space. M and N are the horizontal width and vertical height of the target feature region.

By using the additionally generated first feature value, it may be predicted whether the first feature region obtained in the step of detecting the first feature region ( S615 ) is suitable for use in an application field combined with the present patent. It is obvious that the additionally generated first feature value should be appropriately designed according to an application field. If the condition of the first feature value defined in the application field is not satisfied, the system may be configured to selectively omit the second feature region setting and object detection steps, which will be described later.

second feature region detection step

The purpose of this step is to extract a feature region specifically required in the application field from the region where the animal exists. For example, in an application field that detects the positions of eyes, nose, mouth, and ears in an animal's face region, in the first feature region detection step, the animal's face region and animal species information are first distinguished, and the 2 In the feature area detection step, the purpose of detecting the positions of the eyes, nose, mouth, and ears according to the animal species.

In this process, the step of detecting the second characteristic region may consist of a plurality of independent characteristic region detectors specialized for each animal species. For example, if dogs, cats, and hamsters can be distinguished in the first characteristic region detection step, it is preferable to design three second characteristic region detectors to be specialized for dogs, cats, and hamsters. In this way, it is possible to reduce the learning complexity by reducing the types of features to be learned by the individual feature region detector, and it is self-evident that neural network learning is possible even with a smaller number of data in terms of learning data collection.

Since each of the second feature region detectors is configured independently of each other, an ordinary knowledge holder can easily configure an independent individual detector. Preferably, each feature region detector is individually configured to suit the feature information to be detected in each species. Alternatively, in order to reduce the complexity of the system configuration, some or all of the second feature region detectors share the feature region detectors having the same structure, but a method of configuring the system suitable for each species by replacing the learning parameter values may be used. have. Furthermore, a method of further reducing system complexity by using a feature region detector having the same structure as the first feature region detection step as the second feature region detector, but replacing only the learning parameter values and the NMS method may be considered.

For one or a plurality of feature regions set through the first feature region detection step and the first post-processing step, determine which second feature region detector to use using the species information detected in the first feature region detection step; A second feature region detection step is performed using the determined second feature region detector.

First, preprocessing is performed. In this case, in the process of transforming the coordinates, it is obvious that a transformation process capable of inverse transformation should be used. In the second preprocessing process, since the first feature region detected in the input image must be converted into the input image of the second feature region detector, it is preferable to define the four points necessary for designing the transformation function as four vertices of the first feature region. do.

Since the second feature region obtained through the second feature region detector is a value detected using the first feature region, the first feature region must be considered when calculating the second feature region within the entire input image.

A second feature value may be generated by performing an additional post-processing step similar to the first post-processing step on the second feature area obtained through the second feature area detector. For example, information such as the posture of an animal to be detected may be obtained by applying a Sobel filter to obtain image sharpness, or by using detection and relative positional relationship between feature regions.

By using the additionally generated second feature value, it can be predicted whether the feature region obtained in the second object detection step is suitable for use in an application field combined with the present patent. It is obvious that the additionally generated second feature value should be appropriately designed according to an application field. When the condition of the second feature value defined in the application field is not satisfied, it is preferable to design so that data can be acquired suitable for the application field, such as excluding the first detection area as well as the second detection area from the detection result. .

system expansion

In the present invention, by configuring two detection steps, the first feature location detection step detects the position and species of an animal, and the system and method of configuring a detector to be used in the second feature location detection step are selected according to the results. For example.

This cascade configuration can be easily extended to a multi-layer cascade configuration. For example, in the first feature position detection step, the entire body of the animal is detected, in the second feature location detection step, the face position and limb position of the animal are detected, and in the third feature position detection step, the eyes, nose, Applications such as detecting the position of the mouth and ears are possible.

By using such a multi-layer cascading configuration, it is possible to easily design a system capable of acquiring feature locations of several layers at the same time. In designing a multi-layer cascading system, in determining the number of layers, the hierarchical domain of the feature location to be acquired, the operating time and complexity of the entire system, and the resources required to construct each individual feature region detector should be considered. It is obvious that it is possible to design

9 is a flowchart of a method for processing an image of an object for identification of a companion animal.

A method for processing an image of an object for identification of a companion animal according to the present invention includes: acquiring an image including a companion animal ( S910 ); and generating feature region candidates for determining a species of a companion animal from the image ( S920 ), setting a first feature region whose location and size are determined based on the reliability values of each of the feature region candidates ( S930 ), and including an object for identifying a companion animal in the first feature region The method includes the steps of setting a second characteristic area (S940) and obtaining an image of the object in the second characteristic area (S950).

According to the present invention, the generating of the feature region candidates includes generating the feature images hierarchically using an artificial neural network, and applying predefined boundary regions to each of the feature images to obtain a specific species in each boundary region. The method may include calculating a probability value at which the companion animal will be located, and generating feature region candidates in consideration of the probability value.

The input image normalized according to the preprocessor is hierarchically generated from the first feature image to the nth feature image by the artificial neural network, and the method of extracting the feature image for each layer may be mechanically learned in the learning step of the artificial neural network. .

The extracted hierarchical feature image is combined with a list of predefined boundary regions (bounding boxes) corresponding to each layer, and a list of probability values where a specific animal type is located for each boundary region is generated as shown in Table 1. Here, when it is not possible to determine whether it is a specific animal type, it may be defined as "background".

Thereafter, by applying the extended NMS according to the present invention, candidates (feature region candidates) of the first feature region for identifying a species such as the face of a companion animal in each feature image are generated. Each of the feature region candidates may be derived using the previously derived probability value for each specific animal species.

According to the present invention, the generating of the feature region candidates includes: selecting a first boundary region having the highest probability of corresponding to a specific animal species from the feature image; and the remaining boundary regions excluding the first boundary region selected from the feature image. , calculating an overlap with the first boundary region according to the order of probability values, and including a boundary region having a greater overlap than a reference overlapping degree in a feature region candidate of the feature image. In this case, in order to evaluate the degree of overlap, for example, a ratio of intersection to union area between two boundary regions may be used.

That is, a feature region candidate as shown in FIG. 8A may be generated through the following procedure.

1. Perform the following procedure for each species except for the background.

A. From the list of bounding boxes, exclude boxes whose species probability is lower than a certain threshold. If there are no boxes left, exit with no results.

B. In the bounding box list, the box with the highest probability of the species is designated as the first box (first bounding area) and excluded from the bounding box list.

C. For the rest of the bounding box list, each of the following processes is performed in the order of highest probability.

i. Calculate the degree of overlap with the first box. For example, you can use the intersection over union ratio.

ii. If the overlap value is higher than a certain threshold, this box is a box overlapping the first box. Merge with the first box.

D. Add the first box to the list of result boxes.

E. If there are still boxes in the bounding box list, repeat from step C again for the remaining boxes.

For two boxes A and B, for example, the intersection to union area ratio can be effectively calculated as in Equation 1 described above.

As described with reference to FIG. 8 , a first feature region (eg, a face region of a companion animal) may be derived from each of the feature region candidates derived according to the present invention based on the reliability value of each feature region candidate.

According to the present invention, the central point (C _(x,y) ⁿ ) at which the first feature region is located is the central point (C _(x,y) ¹ , C _(x,y) ² of the feature region candidate as shown in Equation 2) ) can be determined by the weighted sum of the reliability values (p ¹ , p ² ).

According to the present invention, the width (W _n ⁾ of the first feature region is determined by the weighted sum of the reliability values (p ¹ , p ² ) for the widths (W ₁ , W ₂ ) of the feature region candidates as shown in Equation (2). is determined, and the height H ⁿ of the first feature region may be determined by a weighted sum of reliability values p ¹ , p ² with respect to the heights H ¹ , H ² of the feature region candidates.

The present invention applies a weighted average based on the reliability value to a plurality of boxes (feature area candidates) to set one box having a large width and height as a first feature area (a face area of a companion animal), and the first feature A second feature region (nose region) for identification of a companion animal may be detected within the region. By setting a more extended first feature region as in the present invention, it is possible to reduce the occurrence of an error in which a subsequent second feature region is not detected.

Thereafter, detection of a second feature region (eg, a nose region) for identification of a companion animal is performed within the first characteristic region (eg, a face region of a dog). This step is performed in consideration of the species of companion animal performed previously. When the companion animal is a dog, object detection for identification optimized for the dog may be performed. The optimized object detection may be different for each animal type.

The step of setting the second characteristic region may include the second characteristic region (eg, nose) based on the probability that an object (eg, nose) for identification of the companion animal is located in the first characteristic region (eg, face region) according to the species of the companion animal. e.g. nose area).

In addition, post-processing may be performed to check whether the image of the object for identification of the companion animal detected as the second feature region is suitable to be used for subsequent learning or identification. As a result of the post-processing, it may be derived as a second feature value indicating the fitness of the corresponding image. If the second feature value is greater than the reference value, the image including the second feature region is transmitted to the server. If the second feature value is smaller than the reference value, the image including the second feature region is discarded.

10 is a block diagram of an apparatus for processing an image of an object for identification of a companion animal.

The camera 1010 may include an optical module such as a lens and a charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) that generates an image signal from input light, and generates image data through image capturing. Thus, it can be provided to the processor 1020 .

The processor 1020 controls each module of the apparatus 1000 and performs an operation necessary for image processing. The processor 1020 may be configured with a plurality of microprocessors (processing circuits) according to their functions. As described above, the processor 1020 may detect an object (eg, nose) for identification of a companion animal (eg, dog) and determine the validity of an image for the corresponding object.

The communication module 1030 may transmit or receive data with an external entity through a wired/wireless network. In particular, the communication module 1030 may exchange data for processing based on artificial intelligence through communication with a server for learning or identification.

Additionally, the apparatus 1000 may include various modules according to uses, including a memory 1040 for storing image data and information necessary for image processing, and a display 1050 for outputting a screen to a user.

An apparatus for processing an image of an object for identification of a companion animal according to the present invention includes a camera 1010 for generating an image including a companion animal, and processing an image provided from the camera 1020 to identify a companion animal. and a processor 1030 for generating an image of an object for The processor 1030 generates feature region candidates for determining the species of the companion animal in the image, sets a first feature region whose location and size are determined based on the reliability values of each of the feature region candidates, and the first feature region may set a second characteristic area including an object for identifying the companion animal, and obtain an image of the object from the second characteristic area.

According to the present invention, the processor 1020 generates a plurality of feature images from the image hierarchically using an artificial neural network, and applies predefined boundary regions to each of the feature images to specify a specific value in each boundary region. The feature region candidates may be generated by calculating a probability value where the companion animal of the species is located, and taking the probability value into consideration.

According to the present invention, the processor 1020 selects a first boundary region that has the highest probability of corresponding to a specific animal species in the feature image, and selects the remaining boundary regions in the feature image except for the selected boundary region. The degree of overlap with the first boundary area may be calculated according to the order of the probability values, and a boundary area having the degree of overlap greater than the reference degree of overlap may be included in the feature area candidate of the feature image. In this case, the intersection to union area ratio between the two boundary regions may be used to calculate the degree of overlap.

According to the present invention, the degree of overlap may be defined as a ratio of intersection to union area of each of the first boundary region and the remaining boundary regions.

According to the present invention, the central point at which the first feature region is located may be determined by a weighted sum of reliability values for the central points of the feature region candidate.

According to the present invention, the width of the first feature region is determined by the weighted sum of reliability values for the widths of the feature region candidates, and the height of the first feature region is the weighted sum of the reliability values for the heights of the feature region candidates. can be determined by

According to the present invention, the processor 1020 detects the changed position of the object in the next image, determines whether the image of the object whose position is changed in the next image is suitable for AI-based learning or identification, and returns to the changed position. In a state in which the focus is set, the camera 1010 may be controlled to perform the next photographing.

According to the present invention, the processor 1020 sets a first feature area for determining the species of the companion animal in the image, and a second feature including an object for identification of the companion animal within the first feature area. You can set the area.

According to the present invention, the processor 1020 may determine whether an image of an object for identification of a companion animal is suitable for AI-based learning or identification.

According to the present invention, the processor 1020 determines whether the quality of the image of the object satisfies the reference condition, and if the quality satisfies the reference condition, transmits the image of the object to the server, and the quality does not satisfy the standard condition If not, the camera 1010 may be controlled to discard the image of the object and perform photographing for the next image.

This embodiment and the drawings attached to this specification merely clearly show some of the technical ideas included in the present invention, and those skilled in the art can easily infer within the scope of the technical ideas included in the specification and drawings of the present invention. It will be apparent that all possible modifications and specific embodiments are included in the scope of the present invention.

Therefore, the spirit of the present invention should not be limited to the described embodiments, and not only the claims to be described later, but also all those with equivalent or equivalent modifications to the claims will be said to belong to the scope of the spirit of the present invention. .

Claims (12)

A method for processing an image of an object for identification of a companion animal, the method comprising:
acquiring an image including the companion animal;
generating feature region candidates for determining the species of the companion animal from the image;
generating a first feature region whose location and size are determined based on a weighted sum of reliability values at which a specific animal species is to be located in each of the feature region candidates;
setting a second characteristic area including an object for identifying the companion animal in the first characteristic area based on the species of the companion animal; and
acquiring an image of the object in the second feature region,
The step of creating the first feature region comprises:
calculating a reliability value at which a specific animal species will be located in each of the feature region candidates through an operation on a hierarchical feature image generated by an artificial neural network; and
and generating the first feature region by merging each of the feature region candidates by assigning a weight according to a reliability value of each of the feature region candidates.
According to claim 1,
The generating of the feature region candidates includes:
generating a plurality of feature images hierarchically using an artificial neural network;
calculating a probability value that a companion animal of a specific species is located in each of the boundary regions by applying predefined boundary regions to each of the feature images; and
and generating the feature region candidates in consideration of the probability value.
3. The method of claim 2,
The generating of the feature region candidates includes:
selecting a first boundary region having a highest probability of corresponding to a specific animal species from the feature image;
For boundary regions other than the selected first boundary region in the feature image, a degree of overlap with the first boundary region is calculated according to the order of the probability values, and a boundary region whose overlap is greater than a reference overlap is determined. and including in the feature region candidate of the feature image.
4. The method of claim 3,
The degree of overlap is defined as a ratio of intersection to union area of each of the first boundary region and the remaining boundary regions.
According to claim 1,
The central point at which the first feature region is located is determined by a weighted sum of reliability values for the central points of the feature region candidate.
According to claim 1,
The width of the first feature region is determined by a weighted sum of reliability values for the widths of the feature region candidates,
The height of the first feature region is determined by a weighted sum of reliability values for the heights of the feature region candidates.
An apparatus for processing an image of an object for identification of a companion animal, the apparatus comprising:
a camera that generates an image including the companion animal;
a processor for processing the image provided from the camera to generate an image of an object for identification of the companion animal;
The processor is
generating feature region candidates for determining the species of the companion animal from the image;
setting a first feature region whose position and size are determined based on a weighted sum of the reliability values of a specific animal species to be located in each of the feature region candidates;
setting a second characteristic area including an object for identifying the companion animal in the first characteristic area based on the species of the companion animal;
acquiring an image of the object in the second feature area,
To set the first characteristic area, the processor,
calculating the reliability value at which a specific animal species will be located in each of the feature area candidates through operation on the hierarchical feature image generated by the artificial neural network,
The apparatus for generating the first feature region by merging the feature region candidates by weighting them according to the reliability value of each of the feature region candidates.
8. The method of claim 7,
The processor is
Generates a plurality of feature images hierarchically using an artificial neural network,
calculating a probability value that a companion animal of a specific species is located in each of the boundary regions by applying predefined boundary regions to each of the feature images;
An apparatus for generating the feature region candidates in consideration of the probability value.
9. The method of claim 8,
The processor is
selecting a first boundary region with the highest probability of corresponding to a specific animal species from the feature image,
For boundary regions other than the selected first boundary region in the feature image, a degree of overlap with the first boundary region is calculated according to the order of the probability values, and a boundary region whose overlap is greater than a reference overlap is determined. An apparatus for including in a feature region candidate of the feature image.
10. The method of claim 9,
The degree of overlap is defined as a ratio of an intersection to an intersection area of the first boundary region and each of the remaining boundary regions.
8. The method of claim 7,
The central point at which the first feature region is located is determined by a weighted sum of reliability values for the central points of the feature region candidate.
8. The method of claim 7,
The width of the first feature region is determined by a weighted sum of reliability values for the widths of the feature region candidates,
The height of the first feature region is determined by a weighted sum of reliability values for the heights of the feature region candidates.

Images