KR20200044209A - Computer program and theminal for providing individual animal information based on the facial and nose pattern imanges of the animal - Google Patents

Abstract

Disclosed is a computer program stored in a computer-readable storage medium. According to several embodiments of the present invention, the computer program includes instructions to allow a computer to perform the following operations. The operations include: an operation of acquiring a first image including the face of an animal photographed by a camera of a user terminal; an operation of inputting the first image into a pre-learned species classification model to acquire first information for the species of the animal; an operation of recognizing the position of a first region including the muzzle of the animal in the first image; an operation of determining a region of interest in the first region if a preset condition is satisfied; an operation of extracting a second image including the region of interest in the first image if the region of interest is determined; and an operation of transmitting the first information and the second image to a server if the second image is extracted.

Description

COMPUTER PROGRAM AND THEMINAL FOR PROVIDING INDIVIDUAL ANIMAL INFORMATION BASED ON THE FACIAL AND NOSE PATTERN IMANGES OF THE ANIMAL}

The present disclosure relates to a computer program and a terminal stored in a computer-readable storage medium, and more particularly, to a program and a terminal stored in a computer-readable storage medium that provides object information of an animal based on the image of the animal.

The demand for pets is increasing. And, as the demand for pets increases, the number of organic and lost animals per year also increases.

To solve this problem, the government is implementing an animal registration system to make it easier to find the owners of the abandoned animals. For animal registration, it is divided into a built-in type that inserts a wireless identification chip into a pet and an external type that hangs the identification tag with a wireless identification chip attached to the pet's neck.

The identification chip, which is a built-in type, is inhuman due to the characteristic that it is injected into the body of an animal, and there is a problem that side effects may occur in the body after insertion. In addition, there is a fear that the external identification tag may be lost, and if the identification tag is intentionally removed and abandoned, there is a problem that tracking is impossible.

Therefore, in recent years, research on technology using an image of an object has been actively conducted. Among them, the inscription is a unique pattern that can be found on the nose of an animal and can be used as a human fingerprint, so it is required as an alternative technique to the animal registration system described above.

Korean Patent Registration No. 10-1788272 provides a method for managing animal recognition and registration through biometric recognition.

The present disclosure has been devised in correspondence with the background art described above, and is intended to provide a program and a terminal stored in a computer-readable storage medium that provides individual information of an animal based on the image of the animal.

The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In accordance with some embodiments of the present disclosure for solving the above-described problems, a computer program stored in a computer-readable storage medium is disclosed. The computer program includes instructions for causing the user terminal to perform the following operations, the operations comprising: obtaining a first image including a face of an animal photographed by a camera of the user terminal; Inputting the first image into a pre-trained species classification model to obtain first information about the animal species; Recognizing a position of a first region including the nose of the animal in the first image; Determining a region of interest within the first region when a preset condition is satisfied; When the region of interest is determined, extracting a second image including the region of interest from within the first image; And when the second image is extracted, transmitting the first information and the second image to a server. It may include.

In addition, in some embodiments of the present disclosure, the first learning data for training the species classification model uses image data including an animal's face as an input of learning data, and image data including the animal's face. Can be generated by labeling the first classification class for a species of animal. And the species classification model, using the first training data set including the first training data to learn the classification of the animal species that match the face of the animal, based on the first image, the first 1 You can get information.

In addition, in some embodiments of the present disclosure, the species classification model, when receiving the first image as an input node, outputs a confidence score for each of the first classification classes at an output node, and The first classification class having the highest value among the confidence scores may be obtained as first information.

In addition, in some embodiments of the present disclosure, the operation of recognizing the position of the first region including the animal's nose in the first image may include inputting the first image into a pre-trained nose region recognition model. And recognizing a position of the first region within the first image.

In addition, in some embodiments of the present disclosure, the nose region recognition model uses the coordinates for a bounding box and a second training data set including image data including an animal's face. The location of the nose region matching the image of the face may be learned, and the location of the first region within the first image may be recognized.

In addition, in some embodiments of the present disclosure, the operation of recognizing the position of the first region including the nose of the animal in the first image may include: a second including the animal's eye in the first image. Recognizing a location of the region and a third region including the animal's mouth; And recognizing the position of the first area based on the position of the second area and the position of the third area. It may include.

In addition, in some embodiments of the present disclosure, when the preset condition is satisfied, the operation of determining a region of interest in the first region may include: right nostril of the animal within the first region (nostril) and left nostril recognition; Recognizing a pose in which the animal is photographed based on at least one of a circularity of the right nostril and the left nostril and an inertia ratio of the nostril; And when the pose matches a preset pose, recognizing that the preset condition is satisfied.

In addition, in some embodiments of the present disclosure, the operation of determining a region of interest that satisfies a preset condition within the first region may include, when the preset condition is satisfied, the right nostril and Calculating a distance between the left nostrils; And determining the size of the region of interest and the location of the region of interest based on the distance.

In addition, in some embodiments of the present disclosure, the operation of determining a region of interest that satisfies a preset condition in the first region may be performed when the preset condition is not satisfied. The control may further include controlling the display unit to display an indicator requiring re-shooting so that a first image that satisfies.

In addition, in some embodiments of the present disclosure, the operation of determining a region of interest that satisfies a preset condition within the first region may be performed when the region of interest is not satisfied. It may further include an operation of making the pose match the preset pose by frontalizing the image.

In addition, in some embodiments of the present disclosure, the operation of frontizing the first image so that the pose matches the preset pose includes: extracting feature points of nostrils on the image of the first region; And correcting the first image such that the extracted feature points of the nostrils match the feature points of the nostrils on an image photographed from the front of the animal pre-stored in the memory.

In addition, in some embodiments of the present disclosure, after transmitting the first information and the second image to the server, the transmitted first information from the server and the individual information of the animal corresponding to the second image Receiving an operation; And displaying the entity information.

In addition, in some embodiments of the present disclosure, the entity information is mapped to a first inscription image matching the second image pre-processed based on a local histogram at the server. The object information, and the first inscription image may be one of a plurality of inscription images stored in the server.

Meanwhile, a computer program stored in a computer readable storage medium according to some other embodiments of the present disclosure is disclosed. The computer program includes instructions for causing the server to perform the following operations, the operations being: a species of the animal obtained from a first image including a face of an animal captured by the camera of the user terminal from a user terminal. Receiving first information about and a second image including a region of interest within a first region including the nose of the animal within the first image; Pre-processing the second image based on a local histogram; Retrieving a first inscription image matching the preprocessed second image from a plurality of inscription images stored in the memory of the server; And obtaining at least one of a first inscription image matching the pre-processed second image and object information mapped to the first inscription image; And transmitting at least one of the first inscription image and the object information to the user terminal.

In addition, in some other embodiments of the present disclosure, the first inscription image having second information about a nose wrinkle corresponding to the first information at a second characteristic point corresponding to the first characteristic point among the plurality of inscription images The searching may include correcting the second image to move the position of the first feature point when the first feature point and the second feature point are misaligned; And retrieving the first inscription image using the corrected second image. It may further include.

In addition, in some other embodiments of the present disclosure, when the first feature point and the second feature point are misaligned, the operation of correcting the second image to move the position of the first feature point may include: Moving the second image vertically or horizontally or rotating the second image so that the location of the feature point and the second feature point coincide; It may further include.

The technical solutions that can be obtained in the present disclosure are not limited to the above-mentioned solutions, and other solutions that are not mentioned are clearly understood by those skilled in the art from the description below. Will be understandable.

The present disclosure can provide programs and terminals stored in a computer-readable storage medium that provides animal information based on an animal image.

The effects obtainable in the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those skilled in the art from the following description. .

Various aspects are now described with reference to the figures, in which like reference numerals are used collectively to refer to similar elements. In the following embodiments, for illustrative purposes, a number of specific details are presented to provide a holistic understanding of one or more aspects. However, it will be apparent that such aspect (s) may be practiced without these specific details.
1 is a schematic diagram of a user terminal and a server according to some embodiments of the present disclosure.
2 is a block diagram of a user terminal in accordance with some embodiments of the present disclosure.
3 is a flowchart illustrating an operation of transmitting first information and second images of animals to a server according to some embodiments of the present disclosure.
4 is a schematic diagram illustrating a convolutional neural network according to some embodiments of the present disclosure.
5 is a diagram for schematically describing a species classification model according to some embodiments of the present disclosure.
6 is a view for explaining an example of a method for obtaining information about a species of an animal using a species classification model according to some embodiments of the present disclosure.
7 is a diagram for describing a method of recognizing a first region including a nose of an animal according to some embodiments of the present disclosure.
8 is a diagram for describing a method of recognizing a first region including a nose of an animal according to some embodiments of the present disclosure.
9 is a flowchart illustrating an operation of recognizing whether a preset condition is satisfied according to some embodiments of the present disclosure.
10 is a diagram illustrating an example of an operation of recognizing a nostril according to some embodiments of the present disclosure.
11 is a view for explaining an example of a method for measuring the circularity of the nostrils according to some embodiments of the present disclosure.
12 is a view for explaining an example of a method for measuring the inertia of the nostrils according to some embodiments of the present disclosure.
13 is a flowchart illustrating an example of a method for determining a region of interest within a first region according to some embodiments of the present disclosure.
14 is a flowchart for explaining an example of a method in which animal information of an animal is provided to a user terminal according to some embodiments of the present disclosure.
15 is a diagram illustrating an example of preprocessing a second image according to some embodiments of the present disclosure.

Various embodiments and / or aspects are now disclosed with reference to the drawings. In the following description, for purposes of explanation, a number of specific details are disclosed to assist in the overall understanding of one or more aspects. However, it will also be appreciated by those skilled in the art of this disclosure that this aspect (s) can be practiced without these specific details. The following description and the annexed drawings set forth in detail certain illustrative aspects of the one or more aspects. However, these aspects are exemplary and some of the various methods in the principles of the various aspects may be used, and the descriptions described are intended to include all such aspects and their equivalents. Specifically, as used herein, "an embodiment", "yes", "a good", "an example", and the like, any aspect or design described is not to be construed as being advantageous or advantageous over other aspects or designs. It may not.

Hereinafter, the same or similar components are assigned the same reference numbers regardless of reference numerals, and overlapping descriptions thereof are omitted. In addition, in describing the embodiments disclosed in this specification, detailed descriptions of related well-known technologies are omitted when it is determined that the gist of the embodiments disclosed in this specification may be obscured. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed herein, and the technical spirit disclosed herein is not limited by the accompanying drawings.

The terminology used herein is for describing the embodiments and is not intended to limit the present disclosure. In this specification, the singular form also includes the plural form unless otherwise specified in the phrase. As used herein, “comprises” and / or “comprising” does not exclude the presence or addition of one or more other components other than the components mentioned.

Although the first, second, etc. are used to describe various elements or components, it goes without saying that these elements or components are not limited by these terms. These terms are only used to distinguish one element or component from another element or component. Therefore, it goes without saying that the first element or component mentioned below may be the second element or component within the technical spirit of the present disclosure.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings commonly understood by those skilled in the art to which the present disclosure pertains. In addition, terms defined in the commonly used dictionary are not ideally or excessively interpreted unless specifically defined.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless specified otherwise or unclear in context, "X uses A or B" is intended to mean one of the natural inclusive substitutions. That is, X uses A; X uses B; Or, if X uses both A and B, "X uses A or B" can be applied in either of these cases. It should also be understood that the term “and / or” as used herein refers to and includes all possible combinations of one or more of the listed related items.

In addition, the terms "information" and "data" as used herein can often be used interchangeably with each other.

The suffixes "modules" and "parts" for components used in the following description are given or mixed only considering the ease of writing the specification, and do not have a meaning or a role distinguished from each other.

The objectives and effects of the present disclosure, and technical configurations for achieving them, will be apparent with reference to embodiments described below in detail together with the accompanying drawings. In describing the present disclosure, when it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present disclosure, the detailed description will be omitted. In addition, terms to be described later are terms defined in consideration of functions in the present disclosure, which may vary according to a user's or operator's intention or practice.

However, the present disclosure is not limited to the embodiments disclosed below, and may be implemented in various different forms. Only the present embodiments are provided to make the present disclosure complete, and to fully inform the person of ordinary skill in the art to which the present disclosure belongs, and the present disclosure is only defined by the scope of the claims. . Therefore, the definition should be made based on the contents throughout this specification.

1 is a schematic diagram of a user terminal and a server according to some embodiments of the present disclosure.

As illustrated in FIG. 1, the user terminal 100 may be connected to the server 200 through a network 300 through wired / wireless.

According to an embodiment of the present disclosure, the user terminal 100 may obtain a first image including the face of an animal photographed by the camera unit 110. The user terminal 100 may obtain a second image including first information on a species of an animal photographed from the first image and a region of interest in the first image. The user terminal 100 may transmit the acquired first information and second image to the server 200 through the network 300. The user terminal 100 may receive the object information of the photographed animal from the server 200 based on the first information and the second image transmitted to the server 200. The user terminal 100 may display the object information of the received animal. Here, the individual information of the animal may include at least one of animal species, gender, and singularity. In addition, the animal's individual information may include at least one of the animal's owner's name, contact information, and address. Such animal information can be used to identify the owner of the photographed animal. The above-mentioned animal information is only examples, and the present disclosure is not limited thereto.

A detailed description of transmitting the first information and the second image of the user terminal 100 to the server 200 and receiving animal information from the server 200 will be described in detail with reference to FIGS. 2 to 13 below.

Next, the server 200 according to some embodiments of the present disclosure may receive the first information and the second image from the user terminal 100. The server 200 may search for the first inscription image stored in the memory of the server 200 based on the first information and the second image. The server 200 may acquire object information mapped to the searched inscription image. The server 200 may transmit the mapped object information to the user terminal 100 through the network 300. The method in which the above-described server 200 transmits the object information of the animal to the user terminal 100 based on the first information and the second image is only an example, and the present disclosure is not limited.

The detailed operation of the above-described server 200 transmitting the individual information of the animal to the user terminal 100 will be described in detail with reference to FIGS. 14 to 15 below.

Although not shown in the drawings, the server 200 may include a control unit, a network unit, and a memory. Each component of the above-described server 200 is similar to the components included in the user terminal 100, which will be described later, a detailed description thereof will be omitted. However, the present invention is not limited thereto, and the server 200 may include more or less components than the above-described components.

According to some embodiments of the present disclosure, the memory of the server 200 may store a computer program for performing operations of the server 200 according to some embodiments of the present disclosure, which will be described later. The computer program stored in the memory of the server 200 may be read and driven by the control unit of the server 200. In addition, the memory of the server 200 may store a program for the operation of the control unit of the server 200.

Next, the network 300 according to some embodiments of the present disclosure includes a public switched telephone network (PSTN), an x Digital Subscriber Line (xDSL), a rate adaptive DSL (RADSL), a multi rate DSL (MDSL), Various wired communication systems such as Very High Speed DSL (VDSL), Universal Asymmetric DSL (UDSL), High Bit Rate DSL (HDSL), and Local Area Network (LAN) can be used.

In addition, the network 300 presented here is Code Division Multi Access (CDMA), Time Division Multi Access (TDMA), Frequency Division Multi Access (FDMA), Orthogonal Frequency Division Multi Access (OFDMA), Single Carrier-FDMA (SCFDMA) And other wireless communication systems, such as other systems.

In addition, Wireless LAN (WLAN), Wireless-Fidelity (Wi-Fi), Wireless Fidelity (Wi-Fi) Direct, Digital Living Network Alliance (DLNA), Wireless Broadband (WiBro), World Interoperability for Microwave Access (WiMAX), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), etc., communication with the user terminal 100 and the server 200 Can be used.

The techniques described herein can be used in the networks mentioned above, as well as other networks.

2 is a block diagram of a user terminal in accordance with some embodiments of the present disclosure.

According to some embodiments of the present disclosure, the user terminal 100 may include a camera unit 110, a memory 120, a control unit 130, a network unit 140, and a display unit 150. However, the above-described components are not essential in implementing the user terminal 100, and thus the user terminal 100 may have more or fewer components than those listed above. Here, each component may be composed of a separate chip or module or device, or may be included in one device.

The user terminal 100 according to some embodiments of the present disclosure may be a personal computer (PC), a notebook, a mobile terminal, a smart phone, or a tablet PC. However, the present invention is not limited thereto, and the user terminal 100 may be any type of terminal equipped with a camera unit 110 capable of capturing an image.

The user terminal 100 may include one or a plurality of camera units 110 for input of image information. The camera unit 110 may process an image frame such as a still image or a video image obtained by an image sensor in a shooting mode. The processed image frame may be displayed on the display unit 150 or stored in the memory 120.

Meanwhile, the plurality of camera units 110 provided in the user terminal 100 may be arranged to form a matrix structure. Through the camera unit 110 forming the matrix structure, the user terminal 100 may have various angles or A plurality of image information having focus may be input.

The camera unit 110 according to some embodiments of the present disclosure may acquire a first image including an animal's face. Here, the first image may be a video image or a preview image. The above-described first image is only an example, and the present disclosure is now not limited.

The first image may be used by a user to obtain first information about a species of animal photographed by the user terminal 100. In addition, the first image may be used to extract a second image including a region of interest from within the first region including the nose of the photographed animal.

The first image acquired by the camera unit 110 may be stored in the memory 120. Further, the first image may be transmitted to the server 200 through the network unit 140.

The memory 120 may store data supporting various functions of the user terminal 100. The memory 120 may store a number of application programs or applications that are driven in the user terminal 100 and data and commands for the operation of the user terminal 100. At least some of these applications may be downloaded from external servers via wireless communication. In addition, at least some of these application programs may exist on the user terminal 100 from the time of shipment for basic functions of the user terminal 100. Meanwhile, the application program may be stored in the memory 120 and installed on the user terminal 100 to be driven by the controller 130 to perform an operation (or function) of the user terminal 100.

The memory 120 includes a flash memory type, a hard disk type, a solid state disk type, an SDD type (Silicon Disk Drive type), and a multimedia card micro type. ), Card-type memory (e.g. SD or XD memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read (EPMROM) It may include a storage medium of at least one type of -only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk. The user terminal 100 may be operated in connection with a web storage that performs a storage function of the memory 170 on the Internet.

The memory 120 according to some embodiments of the present disclosure stores a computer program for performing the operation of obtaining the first information and extracting the second image and transmitting the first information and the second image to the server 200. You can. The computer program stored in the memory 120 may be read and driven by the controller 130. In addition, the memory 120 may store a program for the operation of the control unit 130, and input / output data (eg, a first image including an animal's face, first information about an animal species) , The second image including the region of interest and the object information of the animal) may be temporarily or permanently stored. The memory 120 may store data related to a display and sound.

The control unit 130 may be composed of one or more cores, a central processing unit (CPU) of a computing device, a general purpose graphics processing unit (GPGPU), and tensor processing (TPU) unit), and a processor for data analysis and deep learning.

The control unit 130 according to some embodiments of the present disclosure reads a computer program stored in the memory 120 and includes first information about a species of an animal captured by a user from a first image and a region of interest within the first image The second image can be extracted.

According to some embodiments of the present disclosure, the controller 130 may perform calculation for learning a neural network. The controller 130 is configured to process the input data for learning in deep learning (DN), extract features from the input data, calculate errors, and update the weight of the neural network using backpropagation. You can perform calculations for learning.

At least one of the CPU, GPGPU, and TPU of the control unit 130 may process learning of the model. For example, the CPU and GPGPU process computations for learning a model, acquiring first information about a species of an animal photographed using the model, and extracting a second image including a region of interest within the first image. can do. In addition, in some embodiments of the present disclosure, operations on at least one of learning a model, acquiring the first information through a model, and extracting the second image may be processed by using processors of a plurality of computing devices together. . Further, a computer program executed in a computing device according to some embodiments of the present disclosure may be a CPU, GPGPU, or TPU executable program.

In some embodiments of the present disclosure, the user terminal 100 may distribute and process the model using at least one of a CPU, GPGPU, and TPU. In addition, in some embodiments of the present disclosure, the user terminal 100 may distribute and process the model together with other computing devices.

The network unit 140 enables wireless communication between the user terminal 100 and the wireless communication system, between the user terminal 100 and another terminal (not shown), or between the user terminal 100 and the server 200. It may include one or more modules. In addition, the network unit 140 may include one or more modules connecting the user terminal 100 to one or more networks.

Wireless Internet technologies include, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World) Interoperability for Microwave Access (HSDPA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), etc. The module for transmitting and receiving data according to at least one wireless Internet technology in a range including the Internet technology not listed above.

The display 150 displays (outputs) information processed by the user terminal 100. For example, the display 150 may display execution screen information of an application program driven by the user terminal 100, or UI (User Interface) or GUI (Graphic User Interface) information according to the execution screen information. .

The display unit 150 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display (flexible). display), a three-dimensional display (3D display), and an electronic ink display (e-ink display).

Hereinafter, a control unit 130 for controlling the operation of the user terminal 100 according to some embodiments of the present disclosure will be described in detail with reference to FIGS. 3 to 13.

3 is a flowchart illustrating an operation of transmitting first information and second images of animals to a server according to some embodiments of the present disclosure.

According to some embodiments of the present disclosure, the controller 130 may acquire a first image including the face of an animal photographed from the camera unit 110 (S210).

Specifically, the controller 130 may acquire a first image including the face of the animal photographed from the camera unit 110 based on the image photographing command. For example, when a user photographs an animal with the camera unit 110 of the user terminal 100, the control unit 130 includes all the faces of the animal (eg, an animal's eye photographed by the user, A user interface representing a photographing command including nose, mouth, and ears) may be displayed through the display unit 150. Here, the first image may be a video image or a preview image.

Next, the controller 130 may input the first image into the pre-trained species classification model to obtain first information on the species of the photographed animal (S220).

Hereinafter, a method of obtaining first information according to some embodiments of the present disclosure will be described in detail with reference to FIGS. 4 to 6.

4 is a schematic diagram illustrating a convolutional neural network according to some embodiments of the present disclosure.

Throughout this specification, neural networks, network functions, and neural networks may be used in the same sense. A deep neural network (DNN) may mean a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can be used to identify potential latent structures in data. In other words, it is possible to grasp the potential structure of photos, text, video, voice, and music (for example, what objects are in the picture, what the content and emotions of the text are, what the content and emotions of the speech, etc.) . Deep neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), and deep belief networks (DBNs). , Q network, U network, Siam network, and the like.

The convolutional neural network illustrated in FIG. 4 is a type of deep neural network and includes a neural network including a convolutional layer. Convolutional neural networks are a type of multilayer perceptorns designed to use minimal preprocessing. The CNN may consist of one or several convolutional layers and artificial neural network layers combined with it. CNN may additionally utilize weights and pooling layers. Thanks to this structure, CNN can fully utilize the input data of the two-dimensional structure. Convolutional neural networks can be used to recognize objects in an image. The convolutional neural network can process image data as a matrix with dimensions. For example, in the case of red-green-blue (RGB) -encoded image data, each of R, G, and B colors may be represented as a two-dimensional (for example, two-dimensional image) matrix. That is, the color value of each pixel of the image data may be a component of the matrix, and the size of the matrix may be the same as the size of the image. Therefore, the image data can be represented by three two-dimensional matrices (three-dimensional data array).

A convolutional process (input / output of a convolutional layer) can be performed by moving a convolutional filter in a convolutional neural network and multiplying the convolutional filter and matrix components at each position of the image. The convolutional filter may be configured as an n * n matrix. The convolutional filter may generally consist of a fixed type filter smaller than the total number of pixels in the image. That is, when an m * m image is input to a convolutional layer (for example, a convolutional layer having a convolutional filter size of n * n), a matrix representing n * n pixels including each pixel of the image It can be multiplied by this convolutional filter (ie, the product of each component of the matrix). A component matching the convolutional filter in the image may be extracted by multiplication with the convolutional filter. For example, a 3 * 3 convolutional filter for extracting a vertical line component from an image may be configured as [[0,1,0], [0,1,0], [0,1,0]] You can. When a 3 * 3 convolutional filter for extracting vertical components from the image is applied to the input image, vertical components that match the convolutional filter from the image may be extracted and output. The convolutional layer may apply a convolutional filter to each matrix for each channel representing an image (ie, R, G, and B colors in the case of R, G, and B coded images). The convolutional layer may extract a feature matching the convolutional filter from the input image by applying a convolutional filter to the input image. The filter value of the convolutional filter (that is, the value of each component of the matrix) may be updated by backpropagation in the learning process of the convolutional neural network.

A subsampling layer is connected to the output of the convolutional layer, thereby simplifying the output of the convolutional layer to reduce memory usage and computation. For example, if the output of a convolutional layer is input to a pooling layer having a 2 * 2 max pooling filter, the maximum value included in each patch is output for each 2 * 2 patch in each pixel of the image to compress the image. You can. The aforementioned pooling may be a method of outputting a minimum value in a patch or outputting an average value of patches, and any pooling method may be included in the present disclosure.

The convolutional neural network may include one or more convolutional layers and sub-sampling layers. The convolutional neural network can extract features from an image by repeatedly performing a convolutional process and a subsampling process (for example, the Max pooling described above). Through the iterative convolutional and subsampling process, the neural network can extract global features of the image.

The output of the convolutional layer or subsampling layer may be input to a fully connected layer. A full connected layer is a layer in which all neurons in one layer are connected to all neurons in a neighboring layer. The full connected layer may refer to a structure in which all nodes of each layer are connected to all nodes of the other layer in the neural network.

In some embodiments of the present disclosure, the neural network may include a deconvolutional neural network (DCNN) to perform segmentation of image data. The deconvolutional neural network performs an operation similar to that of calculating the convolutional neural network in the reverse direction. The deconvolutional neural network may output features extracted from the convolutional neural network as a feature map related to original data. Descriptions of specific configurations for the convolutional neural network are discussed in more detail in US patent US9870768B2, which is incorporated herein by reference in its entirety.

According to some embodiments of the present disclosure, the pre-trained species classification model corresponds to a convolutional neural network, and the controller 130 may recognize the species of the animal photographed using the above-described convolutional neural network.

5 is a diagram for schematically describing a species classification model according to some embodiments of the present disclosure. 6 is a view for explaining an example of a method for obtaining information about a species of an animal using a species classification model according to some embodiments of the present disclosure.

Referring to FIG. 5, the pre-trained species classification model of the present disclosure is a pre-trained network function 520, which is among supervised learning, unsupervised learning, and semi supervised learning. It can be learned in at least one way.

 Learning network functions is intended to minimize errors in the output. In the learning of the network function, learning data is repeatedly input to the network function. In the training of the network function, the output of the network function for the training data and the error of the target are calculated. The weight of each node of the network function can be updated by back propagating the error of the network function from the output layer of the network function to the input layer in the direction of reducing errors in the learning of the network function.

In some embodiments of the present disclosure, the species classification model may be the network function 520 described above.

In some embodiments of the present disclosure, the first learning data for training the species classification model includes image data including an animal's face as an input of training data, and image data including an animal's face for an animal species. It can be created by labeling the first classification class.

In some embodiments of the present disclosure, the species classification model uses the first training data set including the first training data to learn classification for the animal species that matches the animal's face, and based on the first image. It can be generated to obtain first information about the species of the animal.

Here, the species classification model may be generated from an external server or other computing device. In addition, the external server or other computing device may perform web crawling to acquire image data including an animal's face. The image data can be used in the first set of training data for training the species classification model. In addition, the external server or other computing device may acquire image data including an animal face from the user terminal 100 or another user terminal. The method of obtaining the above-described image data is only an example, and the present disclosure is not limited now.

The user terminal 100 according to some embodiments of the present disclosure may acquire an externally generated species classification model and store it in the memory 120. Meanwhile, according to some other embodiments of the present disclosure, the user terminal 100 may generate a species classification model.

For example, the size, shape, and location of each of the eyes, nose, mouth, and ears may differ depending on the species of dog. The dog species can be distinguished based on the characteristics of the dog species. An external server or other computing device may acquire image data including the animal's face in the manner described above. The external server or other computing device may generate the first learning data by labeling information on the animal species based on the characteristic that can distinguish the animal species from the acquired image data. The external server or other computing device may generate a species classification model by learning the classification of the animal species that matches the animal's face in the network function using the first training data set including the first training data. Meanwhile, the method for generating the above-described species classification model is only an example, and the present disclosure is not limited thereto, and the user terminal 100 may directly generate the species classification model.

The control unit 130 according to some embodiments of the present disclosure may obtain information about a species of an animal included in the first image by inputting the generated species classification model into the first image.

Specifically, the controller 130 may calculate by inputting a first image including an animal's face in a species classification model. The controller 130 may output a confidence score for each of the first classification classes for the animal species using the species classification model in the output node 530. The controller 130 may obtain the first classification class having the highest value among the confidence scores as the first information based on the output confidence scores. The method for obtaining the first information described above is merely an example, and the present disclosure is not limited thereto.

For example, referring to FIG. 6, the controller 130 may calculate an confidence score by inputting an image of an animal in a species classification model. The control unit 130 provides a confidence score of each of the first classification classes classified according to types of animals (eg, dogs) included in the image based on characteristics such as shape, size, and location of each of the eyes, nose, mouth, and ears. Can be calculated using the species classification model.

When the image input to the species classification model in the user terminal 100 is the image of the beagle, the confidence score for the beagle may have the highest value among the calculated confidence scores as shown in FIG. 6. In this case, the controller 130 may acquire the first information by recognizing the animal species captured from the first image as a beagle. The method of obtaining the above-described first information is only an example, and the present disclosure is not limited now.

Meanwhile, referring to FIG. 3 again, when the controller 130 acquires the first information on the animal species in step S220, the position of the first region including the nose of the animal photographed in the first image It can be recognized (S230).

Hereinafter, a method of recognizing the position of the first region of the animal by the control unit 130 according to some embodiments of the present disclosure will be described in detail with reference to FIGS. 7 and 8.

7 and 8 are diagrams for describing a method of recognizing a first region including a nose of an animal according to some embodiments of the present disclosure.

According to some embodiments of the present disclosure, the controller 130 may recognize the position of the first region including the nose of the animal captured in the first image by inputting the first image into the pre-trained nose region recognition model. have. The nose recognition model may be a model using a You Only Look Once (YOLO) algorithm. For a detailed configuration of the model using the YOLO algorithm, a paper entitled "Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp .779-788 ".

Here, the nose region recognition model may be generated from an external server or other computing device. As image data for training the nose region recognition model, image data for training the above-described species classification model may be used. In addition, an external server or other computing device may acquire image data for learning the nose region as a method for obtaining the image data described above. The method of obtaining image data for training the nose region recognition model described above is merely an example, and the present disclosure is not limited.

The user terminal 100 may acquire an externally generated nose region recognition model and store it in the memory 120. Meanwhile, according to some other embodiments of the present disclosure, the user terminal 100 may generate a nose region recognition model.

The nose region recognition model can learn the location of the nose region that matches the image of the animal's face using coordinates for a bounding box and a second training data set that includes image data that includes the animal's face. have.

For example, an external server or other computing device may generate the second learning data by labeling the second information about the location of the first area including the nose in the image data including the face of the animal. Here, the second information may include information about the shape, position, and size of the animal's nose for identification with other objects (eg, eyes, mouth, ears, etc.) in the face image of the animal. The labeled learning data may be that the user directly labels the second information within the image containing the animal face. The labeled learning data may be labeling second information on image data including an animal face using another network function model. For example, the second information may be labeled at the position of the first region including the nose in the image including the animal face. An external server or other computing device may obtain second information about the location of the first region including the nose by calculating each of the second training data included in the labeled second training data set using a nose region recognition model. have.

The external server or other computing device may calculate an error by comparing information about the location of the first region obtained by calculating using the nose region recognition model and labeled second information of the second training data. The external server or other computing device can calculate the error based on the degree of overlap of regions. The external server or other computing device is based on the degree of overlap of the bounding box of the information about the location of the first area obtained by calculating using the nose area recognition model and the bounding box of the second labeled information of the learning data. The error can be calculated. The error may be obtained by dividing the intersection of the bounding box of the information on the position of the first region calculated using the bounding box of the labeled second information and the nose region recognition model by the union. The external server or other computing device may back propagate the error calculated in the direction of the input layer from the output layer of the nose region recognition model, which is the reverse direction. The external server or other computing device may reverse the error from the output layer to the input layer to update the connection weight of each node of each layer according to the back propagation. The connection weight of each node to be updated may be determined according to a learning rate. The calculation of the nose region recognition model for the first image, which is the input data, and the back propagation of errors may constitute a learning cycle. The learning rate may be applied differently according to the number of iterations of the learning cycle of the network function. For example, in the early stages of learning a network function, a high learning rate may be used to rapidly increase the efficiency of the network function by ensuring a certain level of performance. For example, a low learning rate can be used to increase accuracy in a later learning period of a network function. For example, an external server or other computing device may increase the learning rate for learning the nose region recognition model from 0.001 to 0.1 per epoch. For example, an external server or other computing device may train the nose region recognition model by setting the learning rate during the last 30 epochs to 0.0001 for training the nose region recognition model. An external server or other computing device may set drop out so that a part of the output of the hidden node is not delivered to the next hidden node in order to prevent overfitting in learning the nose region recognition model. For example, the controller 130 may perform learning of the nose region recognition model by setting the dropout rate of the nose region recognition model to 0.5. The description of the method for generating the nose region recognition model described above is merely an example, and the present disclosure is not limited thereto. The user terminal 100 may generate a nose region recognition model in a manner in which the above-described external server or other computing device generates a nose region recognition model.

Referring to FIG. 7, the controller 130 is based on the first image, first information about the species of the animals included in the first image, and scoring values for each of the second classification classes in the above-described nose recognition model YOLO algorithm. Thus, the second information on the location 710 of the first region including the nose may be determined. The second classification class may be for recognizing the position of the first region including the nose region.

According to some other embodiments of the present disclosure, the controller 130 may recognize the location 710 of the first region including the animal's nose based on the location of the animal's eyes and the animal's mouth.

Specifically, referring to FIG. 8, the controller 130 may recognize the position of the second region including the animal's eye and the third region including the animal's mouth in the first image (S231).

For example, the controller 130 uses the eye area recognition model and the mouth area recognition model stored in the memory 120 to determine the position of the second area including the animal's eye and the position of the third area including the animal's mouth. Can be recognized. Here, the eye region recognition model (or the mouth region recognition model) can recognize the eye region (or the mouth region) in the same manner as the nose region recognition model described above, so a detailed description thereof will be omitted. However, the present invention is not limited thereto, and the controller 130 may recognize the location of the second region including the animal's eye and the location of the third region including the animal's mouth in various ways.

Then, the controller 130 may recognize the location of the first region including the nose of the animal photographed in the first image based on the location of the second region and the location of the third region (S232).

Specifically, the nose position recognition model for recognizing the position of the nose based on the position of the eye and the position of the mouth may be stored in the memory 120. Accordingly, the control unit 130 may include the nose of the animal by inputting the position of the second area acquired in step S231 and the position of the third area acquired in step S231 into the nose position recognition model stored in the memory. The position of the first area can be recognized. In this case, the nose position recognition model may recognize the nose position by additionally inputting information about the animal species.

The method of recognizing the position of the first region including the nose of the animal described above is merely an example, and the present disclosure is not limited thereto.

Meanwhile, referring to FIG. 3 again, if the position of the first region including the nose of the animal is recognized in the first image in step S230, the controller 130 may determine whether the preset condition is satisfied. Yes (S240).

Hereinafter, an operation of recognizing whether a preset condition is satisfied will be described in detail with reference to FIGS. 9 to 12.

9 is a flowchart illustrating a method of recognizing whether a preset condition is satisfied according to some embodiments of the present disclosure. 10 is a diagram illustrating an example of an operation of recognizing a nostril according to some embodiments of the present disclosure. 11 is a view for explaining an example of a method for measuring the circularity of the nostrils according to some embodiments of the present disclosure. 12 is a view for explaining an example of a method for measuring the inertia of the nostrils according to some embodiments of the present disclosure.

The controller 130 according to some embodiments of the present disclosure may recognize the right nostril and the left nostril of the animal in the first region (S241).

Specifically, referring to FIG. 10, the controller 130 may recognize the right nostril and the left nostril of the animal in the first region using a feature extraction algorithm.

For example, the region where the nostril is located in the first region of the first image may be the darkest. Accordingly, a difference between the brightness value of pixels in the nostril region on the image in the first region and the brightness value of pixels in the nostril region located at the edge of the nostril may have a maximum value. In this case, the brightness value may vary depending on the pixel depth of the first image data. The controller 130 may extract the brightness values of the pixels of the image of the first region, and recognize the region having the maximum difference in brightness values of pixels between adjacent regions as the edge of the nostril. In addition, the controller 130 may extract pixels in a region where the difference in brightness value has the maximum value as feature points of the nostrils. The method for recognizing the nostril described above is merely an example, and the present disclosure is not limited thereto.

Meanwhile, referring to FIG. 9 again, the controller 130 may photograph the animal based on at least one of a right nostril and a circularity of the left nostril and an inertia ratio of the nostril. The pose can be recognized (S242). Here, the pose in which the animal is photographed may be information indicating from which side the user has photographed the face of the animal.

According to some embodiments of the present disclosure, a method in which the controller 130 recognizes a pose in which an animal is photographed based on a right nostril and a circularity of the left nostril is as follows. Here, the circularity is the upper / lower / nostril of the nostril corresponding to the species of the animal in which the nostril curvature at the feature point existing on the upper / lower / left / right side of the nostril of the animal included in the image is stored in the memory 120 It may mean the degree similar to the curvature of the nostril at the feature points present on the / left / right side.

The controller 130 may recognize a pose in which the animal is photographed based on first information about the animal species obtained from the first image and circularity according to the animal species stored in the memory 120. .

For example, referring to FIG. 11, the controller 130 is on the upper / lower / left / right side of the nostrils of the animals included in the first image including the face of the animal obtained by shooting with the camera unit 110. The difference between the curvature of the nostril measured at each of the characteristic points P1, P2, P3, and P4 and the curvature of the nostril measured at each of the feature points existing at the top / bottom / left / right stored in the memory 120 You can calculate the value. The control unit 130 may recognize that the pose of the animal is front when the calculated difference value is less than a preset threshold value. Here, the curvature of the nostril measured at each of the feature points existing in the upper / lower / left / right side stored in the memory 120 is the upper / lower of the nostril according to the species of the animal when the face of the animal is photographed from the front. It may be the curvature of the nostril measured at each of the feature points present on the / left / right side.

On the other hand, when the user uses the user terminal 100 to photograph the face of the animal from the right side (or the left side) rather than the front side, the nostrils of the nostrils at the feature point P4 located on the left side from the center of the nostril of the animal The curvature and the curvature of the nostril at the feature point P2 existing on the right side may be smaller than the curvature of the feature points existing on the left side and the right side stored in the memory. Further, the curvature at the feature point P1 existing at the upper side of the nostril and the curvature at the feature point P3 existing at the lower side of the nostril may be greater than the curvature of the feature points existing at the upper and lower sides stored in the memory. In this case, the control unit 130 may recognize that the pose of the animal is right (or left).

According to some other embodiments of the present disclosure, a method in which the controller 130 recognizes a pose in which an animal is photographed based on a right nostril and an inertia of the left nostril is as follows. Here, the inertia rate means the ratio of the length (a) of the horizontal axis to the length (b) of the vertical axis at the center of the nostril.

Referring to FIG. 12, the controller 130 may measure a first length (a) between the feature points of two nostrils located on the horizontal axis from the center of the nostril among the feature points of the nostril. In addition, the controller 130 may measure a second length (b) between the feature points of the two nostrils located on the vertical axis from the center of the nostril. The control unit 130 may measure the inertia rate of the nostril based on the ratio of the first length (a) and the second length (b).

The controller 130 poses where the animal is photographed based on the first information on the species of the animal obtained from the first image, the inertia rate corresponding to the first information stored in the memory, and the inertia rate measured by the first image Can recognize.

Meanwhile, the inertia rate according to the animal species may be stored in the memory 120. The controller 130 may recognize a pose in which the animal is photographed based on a difference value between the inertia rate measured from the first image and the inertia rate mapped to the first information on the animal species stored in the memory. Here, the inertia rate stored in the memory 120 may be an inertia rate that may be obtained when an animal of the corresponding species is photographed from the front.

For example, the control unit 130 is the difference value between the inertia rate of the nostrils of the animal and the inertia rate stored in the memory 120 included in the first image including the face of the animal obtained by shooting with the camera unit 110 Can be calculated. The control unit 130 may recognize that the pose of the animal is front when the calculated difference value is less than a preset threshold value. Here, the inertia rate stored in the memory 120 may be an inertia rate when an animal's face is photographed from the front.

On the other hand, when the user uses the user terminal 100 to photograph the face of the animal from the right side than the front, the length (a) of the horizontal axis located in the right nostril of the animal is the right nostril stored in the memory 120. The length of the horizontal axis is longer and the length (b) of the vertical axis can be measured in the same way. On the other hand, when the user uses the user terminal 100 to photograph the face of the animal from the right side rather than the front, the length (a) of the horizontal axis located in the left nostril of the animal is the left nostril stored in the memory 120. It is shorter than the length of the horizontal axis and the length of the vertical axis can be measured equally. Accordingly, when the pose is the right side, the controller 130 may recognize that the pose is the right side by comparing the inertia rate recognized through the first image with the inertia rate stored in the memory 120.

On the other hand, referring back to FIG. 9, when the photographed pose matches the preset pose, the controller 130 may recognize that the preset condition is satisfied (S243).

Specifically, when the controller 130 recognizes the pose of the animal photographed in step S242, the controller 130 can recognize that the preset condition is satisfied when the pose of the photographed animal matches the preset pose. have. Here, the preset pose means that the animal is photographed from the front.

For example, when it is recognized that the pose recognized in step S243 is front, the controller 130 may recognize that the preset condition is satisfied. On the other hand, if it is recognized that the pose recognized in step S243 is not front, the controller 130 may recognize that the preset condition is not satisfied.

Meanwhile, referring back to FIG. 3, the controller 130 may determine a region of interest in the first region when the preset condition is satisfied in step S240 (S250).

The region of interest within the first region according to some embodiments of the present disclosure may be a region containing an inscription, which is a unique pattern found in the nose of an animal. When the animal is swollen, the size of the nose outside the animal's nostrils expands or contracts, and the shape of the inscription located outside the animal's nostrils can be altered. However, when the animal has a nostril, the size between the left and right nostrils changes less than the outside of the nostrils, and the shape of the inscription located between the left and right nostrils is also located outside the nostrils. Deformation may occur smaller than the shape of. Therefore, it is desirable to determine the region of interest between the left and right nostrils.

13 is a flowchart illustrating an example of a method for determining a region of interest within a first region according to some embodiments of the present disclosure.

Referring to FIG. 13, when the preset condition is satisfied in step S243 of FIG. 9, the controller 130 may calculate the distance between the right nostril and the left nostril (S251).

For example, the control unit 130 may recognize the left / right nostrils by extracting the feature points of the left / right nostrils on the image in the first region in step S241 described above. In addition, the distance between the right nostril and the left nostril may be calculated by the control unit 130.

Next, the controller 130 may determine the size and position of the region of interest based on the distance between the nostrils (S252).

If the distance between the left and right nostrils calculated in step S251 is long, the controller 130 may largely determine the size of the region of interest. On the other hand, if the calculated distance between the left and right nostrils is short, the controller 130 may determine the size of the region of interest to be small. In addition, the controller 130 may determine a position that is the center of the distance between the left and right nostrils as the position of the region of interest. This is because, as described above, the position that is the center of the distance between the left and right nostrils is the position with the smallest change in shape according to the movement of the nostrils.

On the other hand, according to some embodiments of the present disclosure, the control unit 130 displays the display unit 150 to display an indicator requiring re-shooting so that a first image satisfying the preset condition is obtained when the preset condition is not satisfied. ) Can be controlled.

For example, when it is recognized that the pose taken in step S240 of the controller 130 is taken from the right side, it may be recognized that the preset condition is not satisfied. The control unit 130 may control the display unit 150 to display an indicator requesting re-image by moving the camera unit 110 to the left side of the animal's face so as to satisfy a predetermined condition. The controller 130 may again determine whether the re-photographed first image satisfies a preset condition. The indicator method for re-taking the first image described above is merely an example, and the present disclosure is not limited now.

In addition, according to some embodiments of the present disclosure, when the preset condition is not satisfied, the controller 130 may frontize the first image to satisfy the preset condition.

Specifically, the control unit 130 may extract the feature points of the left and right nostrils on the image in the first region in the above-described step S241. The memory 120 may store information regarding the location of the feature points of the left / right nostrils based on the image of the front face of the animal. The controller 130 may correct the first image to match the feature points of the nostrils extracted on the image in the first region to the stored feature points based on information on the location of the feature points of the nostrils stored in the memory 120. . For example, when the user photographs the animal from the right side, the image of the area where the right nostril is located on the first image is acquired wider than the area where the right nostril is located on the actual animal, while the left nostril is The image of the region to be located is obtained narrower than the region where the left nostril of the real animal is located. The control unit 130 may determine that the face of the animal is photographed from the right side on the first image. The controller 130 may further correct the size of the pixels positioned on the right side of the first image or move the position to the left. In addition, correction may be made to increase the size of the pixels located on the left side of the first image or to move it to the right side. The control unit 130 may extract the feature points of the left and right nostrils again on the corrected first image. The controller 130 may recognize that the preset condition is satisfied when the feature points of the extracted left / right nostrils match the stored feature points. The above-described method of correcting and frontizing the first image is merely an example, and the present disclosure is not limited thereto.

Meanwhile, referring to FIG. 3 again, when the region of interest is determined, the controller 130 may extract a second image including the region of interest from the first image (S260).

According to some embodiments of the present disclosure, the controller 130 may generate second image data including a region of interest within the first image. The second image may be transmitted to the server and used to obtain object information of the photographed animal.

Next, the control unit 130 may control the network unit 140 to transmit the first information and the second image to the server when the second image is extracted in step S260.

The control unit of the server 200 obtains the object information of the animal photographed by the user terminal 100 based on the received first information and the second image, and transmits it to the user terminal 100, the network unit of the server 200 Can be controlled.

On the other hand, when the user terminal 100 receives the animal's individual information from the server 200 through the network unit 140, the control unit 130 displays to display the animal's individual information through the display unit 150 The unit 150 can be controlled.

Here, the individual information of the animal may include at least one of animal species, gender, and singularity. Further, the animal's individual information may include at least one of the animal's owner's name, contact information, and address. The individual information of the animal can be used to identify the owner of the photographed animal. The above-mentioned animal information is only examples, and the present disclosure is not limited thereto.

The controller 130 may control the overall operation of the user terminal 100 in addition to the operations related to the application program. The controller 130 may provide or process appropriate information or functions to the user by processing signals, data, and information input or output through the above-described components or by driving an application program stored in the memory 120.

In addition, the controller 130 may control at least some of the components discussed with reference to FIG. 2 to drive the application program stored in the memory 120. Furthermore, the control unit 130 may operate by combining at least two or more of the components included in the user terminal 100 to drive the application program.

14 is a flowchart for explaining an example of a method in which animal information of an animal is provided to a user terminal according to some embodiments of the present disclosure. 15 is a diagram illustrating an example of preprocessing a second image according to some embodiments of the present disclosure.

Referring to FIG. 14, the control unit 130 of the user terminal 100 may extract the first information and the second image based on the first image acquired through the camera unit 110 (S310). This is described above with reference to FIGS. 3 to 13 and detailed description thereof will be omitted.

As described above, the first information may include information on the species of the photographed animal. In addition, the second image may include an inscription image of the animal that can confirm the object information of the photographed animal.

The control unit 130 of the user terminal 100 may control the network unit 140 to transmit the first information and the second image to the server 200 through the network unit 140 (S320).

The control unit of the server 200 may preprocess the received second image based on the local histogram (S330).

When the user acquires the first image by photographing the face of the animal using the user terminal 100, a difference in quality of the first image may occur according to an environment in which the user photographs the animal. For example, the quality of the first image may be different depending on the model of the user terminal 100. Alternatively, other noises, such as foreign matter (eg, animal sweat) in the nose of the animal and reflected light of the camera unit 110 of the user terminal 100 may be included in the first image. Accordingly, noise may also be included in the second image extracted based on the first image that may include noise.

Referring to FIG. 15, the control unit of the server 200 according to some embodiments of the present disclosure may preprocess the second image 1510 including noise based on a local histogram including an image processing algorithm.

Meanwhile, according to some embodiments, the image processing algorithm of the server 200 may include canny edge detection, Harris corner detection, etc., but the present disclosure is not limited thereto.

For example, through the canny edge detection, the server 200 blurs the second image 1510 to remove noise, and uses the mask edge to detect the edges of the animal's nose patterns (inscriptions). , By removing the Non-Maximum Value and dividing the size by Double Threshold, the edges can be extracted. Also, the server 200 may measure the width and depth of the pixels of the second image. The server 200 may preprocess the second image 1510 based on the width and depth of the edges and pixels. When the second image 1510 is preprocessed by the server 200, the contrast difference between the patterns on the preprocessed second image 1520 may increase. The method of preprocessing the second image described above is merely an example, and the present disclosure is not limited now.

When the server 200 pre-processes the second image 1510, the pre-processed second image 1520 may reduce the quality difference of the second image 1510 according to the type of the user terminal 100. In addition, noise included in the second image 1510 before the pre-processing (for example, noise due to foreign matter on the nose of an animal or noise due to reflected light from the camera) can be reduced. In addition, the shape of the nose pattern and the splitting of the nose pattern in the pre-processed second image 1520 may be more clear. Through this, the first inscription image matching the pre-processed second image 1520 as described later may be more accurately searched.

Meanwhile, referring to FIG. 14 again, the server 200 may search for a first inscription image matching the second preprocessed image from a plurality of inscription images stored in the memory of the server 200 (S340).

A plurality of inscription images may be classified and stored according to animal species in the memory of the server 200. The control unit of the server 200 may search for the first inscription image matching the second image based on the first information on the type of the animal photographed in the classified inscription image. Since the inscription images are classified according to the animal species in the memory of the server 200 and stored, the speed of retrieving the first inscription image matching the second image can be improved. The method of retrieving the first inscription image described above is merely an example, and the present disclosure is not limited now.

The method for searching for the first inscription image in step S340 is as follows.

The control unit of the server 200 may recognize the first feature point from the pre-processed second image.

The control unit of the server 200 may recognize the first feature point in the pre-processed second image using a feature extraction algorithm. Here, the first feature point means a point where valleys of the nose wrinkles (for example, a dark portion in the second image) meet in the second image. Accordingly, the brightness value of pixels in which the first feature point is located in the pre-processed second image is higher than the brightness value of pixels in which patterns (eg, a bright portion in the second image) are located in the second image. Can have At this time, the brightness value may be changed according to the pixel depth of the pre-processed second image. The control unit of the server 200 may extract the brightness values of pixels of the pre-processed second image. When the extracted brightness value exceeds a preset brightness value according to the depth of a pixel stored in the memory of the server 200, the control unit of the server 200 sets the position of pixels having a brightness value exceeding the preset brightness value. It can be recognized as the first feature point in the second image. The method for recognizing the first feature point described above is merely an example, and the present disclosure is not limited thereto.

The control unit of the server 200 may recognize the first information about the nose wrinkles at the first characteristic point. Here, the first information about the nose wrinkles may include information about a direction in which the nose wrinkles face from the feature point, information about the number of nose wrinkles, and the like. The first information about the above-mentioned nose wrinkles is merely an example, and the present disclosure is not limited thereto.

The control unit of the server 200 may search for a first inscription image having second information about the nose wrinkles corresponding to the first information about the nose wrinkles at a second characteristic point corresponding to the first characteristic point among the plurality of inscription images. .

Specifically, information about the nose wrinkles at each of the plurality of feature points included in each of the plurality of inscription images may be stored in the memory of the server 200. The control unit of the server 200 searches the memory for information about the nose wrinkles, including information about the direction in which the nose wrinkles are directed at the first feature point, and information corresponding to the number of the nose wrinkles, and the first inscription described above. It can be recognized as an image. That is, the control unit of the server 200 may search for an inscription image having information about the same nose wrinkle at the same feature point among a plurality of inscription images stored in the memory of the server 200.

Meanwhile, the control unit of the server 200 may correct the second image for moving the position of the first feature point when the positions of the first and second feature points are misaligned. Here, the control unit of the server 200 is the first feature point when the coordinates of the first feature point and the coordinates of the second feature point are different from each other or the direction in which the nose wrinkles in the first feature point is different from the direction in which the nose wrinkles in the second feature point is different. It can be recognized that the positions of the and the second feature point are misaligned.

Specifically, the control unit of the server 200 performs a correction to vertically or horizontally move the second image or rotate the second image so that the positions of the first feature points in the second image and the second feature points in the first inscription image match. You can.

Meanwhile, the control unit of the server 200 may acquire the object information mapped to the first inscription image matching the second image (S350). However, the present invention is not limited thereto, and the controller of the server 200 may acquire at least one of a first inscription image matching the second image and object information mapped to the first inscription image.

The control unit of the server 200 may transmit the object information obtained in step S350 to the user terminal (S360). However, the present invention is not limited thereto, and the control unit of the server 200 may transmit at least one of the object information and the first inscription image to the user terminal 100.

The control unit 130 of the user terminal 100 may control the display unit 150 to display object information received from the server 200 (S370). However, the present invention is not limited thereto, and at least one of the object information and the first inscription image may be displayed.

Here, the individual information of the animal may include at least one of animal species, gender, and singularity. Further, the animal's individual information may include at least one of the animal's owner's name, contact information, and address. The individual information of the animal can be used to identify the owner of the photographed animal. The above-mentioned animal information is only examples, and the present disclosure is not limited thereto.

Descriptions of the presented embodiments are provided to enable any person of ordinary skill in the art to use or practice the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art of the present disclosure, and the general principles defined herein can be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure should not be limited to the embodiments presented herein, but should be interpreted in the broadest scope consistent with the principles and novel features presented herein.

Claims (17)

A computer program stored in a computer-readable storage medium,
The computer program includes instructions for a user terminal to perform the following operations, which include:
Acquiring a first image including a face of an animal photographed by a camera of the user terminal;
Inputting the first image into a pre-trained species classification model to obtain first information about the animal species;
Recognizing a position of a first region including the nose of the animal in the first image;
Determining a region of interest within the first region when a preset condition is satisfied;
When the region of interest is determined, extracting a second image including the region of interest from within the first image; And
When the second image is extracted, transmitting the first information and the second image to a server;
Containing,
A computer program stored on a computer readable storage medium.
According to claim 1,
The first training data for training the species classification model,
The image data including the face of the animal is used as input of learning data, and the image data including the face of the animal is generated by labeling the first classification class for the animal species, and
The species classification model,
Learning to classify the species of the animal that matches the face of the animal by using the first set of learning data including the first learning data, and to obtain the first information based on the first image,
A computer program stored on a computer readable storage medium.
According to claim 2,
The species classification model,
When the first image is received as an input node, a confidence score for each of the first classification classes is output from an output node, and the first classification class having the highest value among the confidence scores is first information. Acquired by,
A computer program stored on a computer readable storage medium.
According to claim 1,
Recognizing the position of the first region including the nose of the animal in the first image,
And inputting the first image into a pre-trained nose region recognition model to recognize the location of the first region within the first image,
A computer program stored on a computer readable storage medium.
The method of claim 4,
The nose region recognition model,
A second learning data set including image data including an animal's face and coordinates for a bounding box are used to learn the location of the nose region matching the image of the animal's face, and the first image Recognizing the position of the first region within,
A computer program stored on a computer readable storage medium.
The method of claim 4,
Recognizing the position of the first region including the nose of the animal in the first image,
Recognizing a position of a second area including the animal's eye and a third area including the animal's mouth within the first image; And
Recognizing the position of the first area based on the position of the second area and the position of the third area;
Containing,
A computer program stored on a computer readable storage medium.
According to claim 1,
When the predetermined condition is satisfied, the operation of determining a region of interest within the first region may include:
Recognizing a nostril and a left nostril of the animal within the first region;
Recognizing a pose in which the animal is photographed based on circularity of the right nostril and the left nostril and inertia ratio of the nostril; And
Recognizing that the preset condition is satisfied when the pose matches a preset pose;
Containing,
A computer program stored on a computer readable storage medium.
The method of claim 7,
The operation of determining a region of interest that satisfies a preset condition in the first region may include:
Calculating a distance between the right nostril and the left nostril when the preset condition is satisfied; And
Determining a size of the region of interest and a location of the region of interest based on the distance;
Containing,
A computer program stored on a computer readable storage medium.
The method of claim 7,
The operation of determining a region of interest that satisfies a preset condition in the first region may include:
Controlling the display unit to display an indicator requesting re-shooting so that a first image satisfying the preset condition is obtained when the preset condition is not satisfied;
Containing more,
A computer program stored on a computer readable storage medium.
The method of claim 7,
The operation of determining a region of interest that satisfies a preset condition in the first region may include:
If the preset condition is not satisfied, frontalizing the first image to make the pose match the preset pose;
Containing more,
A computer program stored on a computer readable storage medium.
The method of claim 10,
The operation of making the pose match the preset pose by frontalizing the first image,
Extracting feature points of nostrils on the image of the first region; And
Correcting the first image so that the feature points of the extracted nostrils match the feature points of nostrils on an image photographed from the front of an animal pre-stored in memory;
Containing more,
A computer program stored on a computer readable storage medium.
According to claim 1,
After transmitting the first information and the second image to the server, the server receives at least one of the first information and the animal information corresponding to the second image and the first inscription image from the server. To do; And
Displaying at least one of the object information and the first inscription image;
Containing more,
A computer program stored on a computer readable storage medium.
The method of claim 12,
The individual information,
The second image is object information mapped to a first inscription image matching the second image pre-processed based on a local histogram at the server,
The first inscription image,
One of a plurality of inscription images stored in the server,
A computer program stored on a computer readable storage medium.
A computer program stored in a computer-readable storage medium,
The computer program includes instructions for the server to perform the following actions, which include:
In the first area including the nose of the animal in the first image and the first information about the species of the animal obtained from the first image containing the face of the animal photographed by the camera of the user terminal from the user terminal Receiving a second image including the region of interest in the;
Pre-processing the second image based on a local histogram;
Retrieving a first inscription image matching the preprocessed second image from a plurality of inscription images stored in the memory of the server; And
Obtaining at least one of a first inscription image matching the pre-processed second image and object information mapped to the first inscription image; And transmitting at least one of the first inscription image and the object information to the user terminal.
Containing,
A computer program stored on a computer readable storage medium.
The method of claim 14,
Searching for a first inscription image matching the second pre-processed image from a plurality of inscription images stored in the memory of the server,
Recognizing a first feature point from the pre-processed second image;
Recognizing first information about nose wrinkles at the first feature point of the second image;
Retrieving the first inscription image having second information about a nose wrinkle corresponding to the first information at a second characteristic point corresponding to the first characteristic point among the plurality of inscription images;
Containing more,
A computer program stored on a computer readable storage medium.
The method of claim 15,
The operation of retrieving the first inscription image having the second information on the nose wrinkle corresponding to the first information from the second characteristic point corresponding to the first characteristic point among the plurality of inscription images may include:
Correcting the second image to move the position of the first feature point when the positions of the first feature point and the second feature point are shifted; And
Retrieving the first inscription image using the corrected second image;
Containing more,
A computer program stored on a computer readable storage medium.
The method of claim 16,
When the position of the first feature point and the second feature point are misaligned, the operation of correcting the second image to move the position of the first feature point may include:
Moving the second image vertically or horizontally to rotate the second image so that the positions of the first feature point and the second feature point coincide;
Containing more,
A computer program stored on a computer readable storage medium.

Images