KR102349851B1 - System and method for providing multi-object recognition service using camera for pet - Google Patents

Abstract

Provided is a system for providing a multi-object recognition service for companion animals using a camera, which comprises: a camera installed or provided to face a direction of a space where one or more companion animals live to photograph the companion animals, and outputting image data obtained by photographing the companion animals; a user terminal for outputting result data obtained by recognizing the companion animals when the image data collected by the camera is output; and a recognition service providing server including a registration unit for mapping and registering the user terminal and camera from the user terminal, a recognition unit for identifying and tracking multiple objects corresponding to the companion animals included in the image data received from the camera, and a transmission unit for transmitting the identified and tracked result data with the image data. Therefore, demands of a plurality of companion animals can be accurately recognized and immediately resolved.

Description

System and method for providing multi-object recognition service for companion animals using a camera

The present invention relates to a system and method for providing a multi-object recognition service for companion animals using a camera, and provides a platform for identifying and tracking multi-objects using a camera.

As the social structure is changed due to the development of industry, the number of households raising companion animals is increasing due to the aging population, the number of nuclear families, and the rapid increase of single-person households. According to the industrial economic analysis report of the Korea Institute for Industrial Economics and Trade (KIET), the domestic companion animal-related market is steadily increasing. It is reported that monitoring systems and wearable products exclusively for companion animals are being commercialized through domestic and foreign companies. If the owner goes out for a long time, caring for a pet left alone can be a complicated problem. This refers not only to health problems, but also to large and small economic losses caused by pet behavior. Examples are electric shocks or explosions caused by biting electric cords or batteries, and fire accidents caused by the touch of a battery range. According to the report of the Fire Department, the number of fires caused by companion animals is gradually increasing, and the amount of property damage is also increasing.

At this time, a platform for controlling IoT devices to identify and track companion animals and analyze abnormal behaviors to stop abnormal behaviors was researched and developed. 26th), Korean Patent Publication No. 2020-0055821 (published on May 22, 2020) and Korean Patent Application Publication No. 2015-0101760 (published on September 04, 2015), various types of separation anxiety behavior The companion animal’s separation anxiety behavior pattern and tracking space are stored so that the companion animal can relieve separation anxiety by enabling tracking and tracking, and the fixed camera records the companion animal’s behavior in the following space and following space and sends it to the server. When the companion animal's behavior is determined to be separation anxiety by the server, it receives a follow command from the server to follow the companion animal, and each of the image data set based on the companion animal image captured by the camera Generates a CNN output value that generates a CNN output value for a CNN that uses image data as an input value of a multi-stream, generates an LSTM output value for an LSTM that uses a CNN output value as an input value, and rejects based on the LSTM output value A configuration that detects dangerous behavior of animals, and detects when a companion animal left alone at home is in an abnormal state, and determines whether the companion animal is in an abnormal state from the companion animal sound signal so that it can immediately notify the owner of the companion animal in a remote location. Each configuration is disclosed to inform the .

However, the above-described configuration is only a configuration that can be used in a household of about one or two animals, and in the absence of a recent owner, a companion animal care service, for example, a plurality of companion animals such as a companion animal kindergarten, a dog cafe, or a dog playground It is not a usable configuration in industries that need to manage all at once. Also, even if it is installed at home, it is always connected to a smartphone recently and the owner, who has to wait for 5 minutes to receive work orders or reminders from the boss or workplace, can run to the house immediately by telling him that he is doing something abnormal. Also, the owner who can't go home is just launching a technology that makes it sad that he has to watch his pet's pitiful crying with his smartphone. Accordingly, research and development of a control platform that can designate and recognize a plurality of companion animals as multi-objects, and analyze them with artificial intelligence when abnormal behavior occurs, and take appropriate actions is required.

An embodiment of the present invention identifies and tracks at least one companion animal by designating it as a multi-object, extracting behavioral data from the companion animal's body data, facial expression data from facial data, and sound data from image data for companion animal By controlling at least one IoT device to understand the condition of the animal and provide each companion animal with what they currently want, companies that provide care for companion animals or multi-dog or multi-cat households can meet the needs of multiple companion animals. It is possible to provide a method of providing a multi-object recognition service for companion animals using a camera that can accurately identify and solve problems immediately. However, the technical task to be achieved by the present embodiment is not limited to the technical task as described above, and other technical tasks may exist.

As a technical means for achieving the above-described technical problem, an embodiment of the present invention is installed or provided to face at least one companion animal in the direction of the space in which the companion animal lives so as to photograph at least one companion animal, and photographing at least one companion animal A camera that outputs one image data, a user terminal that recognizes at least one companion animal when outputting image data collected from the camera, and a user terminal that outputs data as a result of recognizing at least one companion animal, and a registration unit that maps and registers the user terminal and the camera from the user terminal, the camera A cognitive service providing server including a recognition unit for identifying and tracking multiple objects corresponding to at least one companion animal included in the image data received from the .

Another embodiment of the present invention includes the steps of mapping and registering the user terminal and the camera from the user terminal, identifying and tracking multiple objects corresponding to at least one companion animal included in image data received from the camera, and identification and transmitting the tracking result data together with the image data.

According to any one of the above-described problem solving means of the present invention, at least one companion animal is designated as a multi-object, identified and tracked, behavior data from body data of the companion animal, expression data from facial data, and expression data from image data By extracting sound data to understand the status of companion animals and controlling at least one IoT device so that each companion animal can immediately supply what they want, companies that provide care for companion animals or multiple or multi-cat households Accurately identify the needs of pets and provide immediate solutions.

1 is a view for explaining a system for providing a multi-object recognition service for companion animals using a camera according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating a cognitive service providing server included in the system of FIG. 1 .
3 and 4 are diagrams for explaining an embodiment in which a companion animal multi-object recognition service using a camera according to an embodiment of the present invention is implemented.
5 is an operation flowchart illustrating a method for providing a multi-object recognition service for companion animals using a camera according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

The terms "about", "substantially", etc. to the extent used throughout the specification are used in or close to the numerical value when manufacturing and material tolerances inherent in the stated meaning are presented, and are intended to enhance the understanding of the present invention. To help, precise or absolute figures are used to prevent unfair use by unconscionable infringers of the stated disclosure. As used throughout the specification of the present invention, the term "step of (to)" or "step of" does not mean "step for".

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware. Meanwhile, '~ unit' is not limited to software or hardware, and '~ unit' may be configured to be in an addressable storage medium or to reproduce one or more processors. Thus, as an example, '~' denotes components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functions provided in the components and '~ units' may be combined into a smaller number of components and '~ units' or further separated into additional components and '~ units'. In addition, components and '~ units' may be implemented to play one or more CPUs in a device or secure multimedia card.

Some of the operations or functions described as being performed by the terminal, apparatus, or device in the present specification may be performed instead of by a server connected to the terminal, apparatus, or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal means mapping or matching the terminal's unique number or personal identification information, which is the identification data of the terminal. can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a system for providing a multi-object recognition service for companion animals using a camera according to an embodiment of the present invention. Referring to FIG. 1 , a system 1 for providing a multi-object recognition service for companion animals using a camera includes at least one user terminal 100 , a cognitive service providing server 300 , at least one camera 400 , and at least one The IoT device 500 may be included. However, since the companion animal multi-object recognition service providing system 1 using the camera of FIG. 1 is only an embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1 .

At this time, each component of FIG. 1 is generally connected through a network (Network, 200). For example, as shown in FIG. 1 , at least one user terminal 100 may be connected to the cognitive service providing server 300 through the network 200 . In addition, the cognitive service providing server 300 may be connected to at least one user terminal 100 , at least one camera 400 , and at least one IoT device 500 through the network 200 . Also, the at least one camera 400 may be connected to the cognitive service providing server 300 through the network 200 . In addition, at least one IoT device 500 may be connected to at least one user terminal 100 , a cognitive service providing server 300 , and at least one camera 400 through the network 200 .

Here, the network refers to a connection structure in which information exchange is possible between each node, such as a plurality of terminals and servers, and an example of such a network includes a local area network (LAN), a wide area network (WAN: Wide Area Network), the Internet (WWW: World Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television networks, and the like. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), 5th Generation Partnership Project (5GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi (Wi-Fi) , Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth (Bluetooth) network, NFC ( Near-Field Communication) networks, satellite broadcast networks, analog broadcast networks, Digital Multimedia Broadcasting (DMB) networks, and the like are included, but are not limited thereto.

In the following, the term at least one is defined as a term including the singular and the plural, and even if the at least one term does not exist, each element may exist in the singular or plural, and may mean the singular or plural. it will be self-evident In addition, that each component is provided in singular or plural may be changed according to embodiments.

At least one user terminal 100, when a companion animal is registered using a web page, an app page, a program or an application related to the companion animal multi-object recognition service using the camera and the camera 400 is purchased, the camera 400 and It may be a terminal that registers information of its own companion animal. In addition, the user terminal 100 may be a terminal that outputs a result of identifying and tracking multiple objects on image data when there are a plurality of companion animals. And, when the abnormal behavior is collected, the user terminal 100 receives information on the abnormal behavior from the cognitive service providing server 300 or receives an alarm and approves the control command of at least one IoT device 500 . It may be a terminal.

Here, the at least one user terminal 100 may be implemented as a computer that can access a remote server or terminal through a network. Here, the computer may include, for example, navigation, a laptop equipped with a web browser, a desktop, and a laptop. In this case, the at least one user terminal 100 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one user terminal 100 is, for example, as a wireless communication device that guarantees portability and mobility, navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS(Personal Handyphone System), PDA(Personal Digital Assistant), IMT(International Mobile Telecommunication)-2000, CDMA(Code Division Multiple Access)-2000, W-CDMA(W-Code Division Multiple Access), Wibro(Wireless Broadband Internet) ) terminal, a smart phone, a smart pad, a tablet PC, etc. may include all kinds of handheld-based wireless communication devices.

The cognitive service providing server 300 may be a server that provides a companion animal multi-object cognitive service web page, an app page, a program, or an application using a camera. In addition, the cognitive service providing server 300 may be a server in which at least one artificial intelligence algorithm for identifying multiple objects is embedded. In addition, the cognitive service providing server 300 identifies a companion animal within the image data collected through the camera 400, and when there are a plurality of companion animals, the recognition service identifies and tracks the plurality of companion animals by designating them as multi-objects. It may be a server running the process. In addition, the cognitive service providing server 300 separates the face and body of the companion animal, extracts expression data from facial data, behavior data from body data, and sound data from image data, and then analyzes the state of the companion animal. Then, it may be a server that transmits the corresponding control to the IoT device 500 .

Here, the cognitive service providing server 300 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, navigation, a laptop equipped with a web browser, a desktop, and a laptop.

The at least one camera 400 may be a device for photographing a companion animal using a web page, an app page, a program or an application related to the companion animal multi-object recognition service using the camera and transmitting it to the cognitive service providing server 300 have.

Here, the at least one camera 400 may be implemented as a computer that can connect to a remote server or terminal through a network. Here, the computer may include, for example, navigation, a laptop equipped with a web browser, a desktop, and a laptop. In this case, the at least one camera 400 may be implemented as a terminal capable of accessing a remote server or terminal through a network. The at least one camera 400 is, for example, a wireless communication device that ensures portability and mobility, such as navigation, Personal Communication System (PCS), Global System for Mobile communications (GSM), Personal Digital Cellular (PDC), and PHS. (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) It may include all types of handheld-based wireless communication devices such as terminals, smartphones, smartpads, and tablet PCs.

The at least one IoT device 500 may be a device driven according to control received from the cognitive service providing server 300 using a web page, an app page, a program, or an application related to a companion animal multi-object recognition service using a camera. . In this case, the IoT device 500 may be, but is not limited to, a food feeder, a companion animal care robot, a lighting, a TV, a speaker, and the like.

Here, the at least one IoT device 500 may be implemented as a computer that can connect to a remote server or terminal through a network. Here, the computer may include, for example, navigation, a laptop equipped with a web browser, a desktop, and a laptop. In this case, the at least one IoT device 500 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one IoT device 500 is, for example, a wireless communication device that guarantees portability and mobility, including navigation, Personal Communication System (PCS), Global System for Mobile communications (GSM), Personal Digital Cellular (PDC), PHS(Personal Handyphone System), PDA(Personal Digital Assistant), IMT(International Mobile Telecommunication)-2000, CDMA(Code Division Multiple Access)-2000, W-CDMA(W-Code Division Multiple Access), Wibro(Wireless Broadband Internet) ) terminal, a smart phone, a smart pad, a tablet PC, etc. may include all kinds of handheld-based wireless communication devices.

2 is a block diagram illustrating a cognitive service providing server included in the system of FIG. 1, and FIG. 3 is an embodiment in which a companion animal multi-object recognition service using a camera according to an embodiment of the present invention is implemented. It is a drawing for explanation.

Referring to FIG. 2 , the cognitive service providing server 300 may include a registration unit 310 , a recognition unit 320 , a transmission unit 330 , a tagging unit 340 , and an IoT control unit 350 .

At least one user terminal 100, at least one camera 400, and at least one IoT device (not shown) operating in conjunction with the cognitive service providing server 300 according to an embodiment of the present invention When transmitting a companion animal multi-object recognition service application, program, app page, web page, etc. using a camera to 500), at least one user terminal 100, at least one camera 400, and at least one IoT device ( 500) may install or open a companion animal multi-object recognition service application, program, app page, web page, etc. using a camera. In addition, the service program may be driven in at least one user terminal 100 , at least one camera 400 , and at least one IoT device 500 using a script executed in a web browser. Here, the web browser is a program that enables the use of a web (WWW: World Wide Web) service, and refers to a program that receives and displays hypertext written in HTML (Hyper Text Mark-up Language), for example, Netscape. , Explorer, Chrome, and the like. In addition, the application means an application on the terminal, for example, includes an app (App) executed in a mobile terminal (smartphone).

Referring to FIG. 2 , the registration unit 310 may register the mapping of the user terminal 100 and the camera 400 from the user terminal 100 . The camera 400 may be installed or provided to face the direction of a space in which the companion animal lives so as to photograph at least one companion animal, and may output image data obtained by photographing the at least one companion animal. When outputting the image data collected from the camera 400 , the user terminal 100 may output data as a result of recognizing at least one companion animal together.

For image-based biometric recognition, first, biometric images must be acquired. At this time, the most important thing is to acquire an image when the face direction is the front. This is because biometric information can be completely collected when the face is facing the front, and the object recognition rate can be increased only when images are collected in the same shape as possible. Unlike humans, dogs do not look straight ahead when they want, so in order to increase the probability of obtaining biometric images, it is necessary to use a technology to determine whether the dog is facing the front in real-time images of 30 frames per second or more. In addition, when the face direction is the front, it is necessary to provide location information for segmentation of the eye and nose regions to be used for iris recognition or inscription recognition in the image. In addition, unlike humans, dogs have a lot of hair, so each individual has a large external change that is not related to the direction of the face, and there are restrictions in using precise geometric information such as heatmaps. Therefore, in one embodiment of the present invention, based on a method of extracting information suitable for estimating the direction of the dog's face, it is determined whether the dog's face is frontal, and when the face direction is the front, it is possible to provide eye and nose position information for biometric image segmentation. make it possible

To this end, the dog's eyes and nose are detected from the image through a deep learning algorithm, and 5 pieces of information for estimating the face direction are extracted from the relative positions and sizes of the detected eyes and nose, and whether it is a frontal view or not through a machine learning classifier. to judge If the face direction is the front, the position information of the eyes and nose detected through deep learning is provided to segment the eye and nose images to be used for biometric recognition.

In the face-to-face identification method of companion animals, first, the eyes and nose are detected from the dog image input through a deep learning algorithm, and the respective coordinates are output. Based on the coordinates of the position of the eyes and nose, information to be used for face frontal identification is extracted. By inputting the extracted information into the machine learning classifier, it is determined whether the dog is facing the front or not. When the face direction is the front, the detected eye and nose position information is provided so that the eye and nose regions to be used for biometric recognition can be segmented from the image.

It is possible to collect images of Maltese, Poodle, Shih Tzu, Yokoshire Terrier, and Pomeranian, which are the representative dog breeds most bred in Korea, and classify them into frontal and non-frontal. An image in which at least one of the two eyes and the nose is not visible can be easily determined as not the front by checking the number after detecting the eyes and nose. Therefore, only images in which both eyes and nose are visible can be collected. The data for training can be labeled for the dog's eyes and nose using LabelImg, an open-source program distributed by tzutalin GitHub. When labeling is complete, the class number and normalized coordinate values (x, y, w, h) of the bounding box for each image are saved as a text file.

<Deep learning based pet eye and nose detection>

In a method that uses geometric information to determine the direction of a face based on the relative positions and sizes of facial features, how accurately each feature is located is very important. In addition, in order to increase the probability of collecting biometric images suitable for biometric recognition from uncooperative dogs, real-time image processing of at least 30 frames per second should be possible. Therefore, in one embodiment of the present invention, real-time processing is possible for dog eyes and nose detection, and YOLOv3 and YOLOv4 algorithms with excellent accuracy can be used. YOLO (You Only Look Once) is a deep learning algorithm optimized for real-time processing by simultaneously performing localization and classification of an object by one convolutional network viewing an image once. This algorithm exists in several forms as it continues to be modified and developed, and YOLOv3 and YOLOv4 are the most recently published models.

The structure of the object detection algorithm consists of three parts: Backbone, Neck, and Head (Dense Prediction). Backbone is an already learned network used for feature map extraction, Neck is a layer that exists between Backbone and Head to extract different feature maps at different stages of Backbone, and Head is actually an object. This is the part responsible for detection. YOLOv3 applies darknet53 as a backbone and uses FPN (Feature Pyramid Network) as a neck, and the head calculates the score for determining whether there is an object in the bounding box and the class-specific probability for classifying the object. It is structured to predict at the same time. YOLOv4 is an improved model of YOLOv3 by grafting the latest deep learning technology. CSPdarknet53 is applied to the backbone, SPP (Spatial Pyramid Pooling) and PAN (Path Aggregation Network) are used as the neck, and the head has the structure of YOLOv3.

<Information extraction for frontal identification of pets>

The direction of the pet's face movement is indicated by Yaw, Pitch, and Roll. Yaw, Pitch, and Roll refer to rotation based on the Z axis, rotation based on the X axis, and rotation based on the Y axis, respectively. The misalignment in the roll direction on the dog's face does not significantly affect the deformation of the biometric image, and it can be easily corrected by finding the misaligned angle of the line connecting the two eyes. Therefore, whether the face is frontal is judged only by the deviation of the yaw and pitch directions. By applying the geometric method used for estimating the direction of a person's face, five factors that provide information on the yaw and pitch direction misalignment are extracted and marked as A~E. The values of the three interior angles of the triangle connecting the center points of the eyes and the nose provide information on the yaw and pitch directions. The ratio of the length of the line connecting the center points of the two eyes to the length of the line connecting the nose vertically from the line provides information on the pitch direction. The ratio of the length from the point where the line connecting the center points of the two eyes to the line connecting the nose to the vertical line meets the left eye to the right eye provides information in the yaw direction. The detected bounding box area ratio of the left eye and the right eye provides information in the yaw direction. The ratio of the area of the left eye and the area of the right eye to the area of the nose in the bounding box of the eye and nose also provides information in the yaw direction.

<Machine learning-based dog face detection>

It can be applied to the proposed methods of deep learning methods such as MLP (Multi-Layer Perceptron), RF (Random Forest), and SVM (Support Vector Machine) to determine whether the face direction is the front by using the extracted 5 dog face direction information. have. The MLP classifier is a deep learning algorithm composed of an input layer, a hidden layer, and an output layer. Factors that affect the performance of the MLP classifier include the number of hidden layers, the number of nodes in each layer, activation function, optimization algorithm, L2 generalization parameter (Alpha), and learning rate. The RF classifier is an algorithm that uses multiple decision tree classifiers that classify data in a tree form and finally classifies them through voting. The factors that affect the RF classifier performance include the number of decision trees (Estimator), the maximum depth (Max Depth), the minimum number of samples for sub-node splitting (Min Samples Split), and the minimum number of samples to become a node (Min Samples Leaf). ), etc. The SVM classifier is an algorithm that classifies data by finding a decision boundary that maximizes the distance between adjacent samples. Factors that affect SVM classifier performance include kernel, generalization parameter (C), and gamma. Hyper-parameters to be adjusted for performance optimization for each classifier can be selected as variables that have a large impact on performance. The grid search algorithm can be used to find the combination with the best performance, and the K-fold cross-validation method, which is commonly used recently, can be used for performance verification. When the registration and identification of the companion animal is completed, the next stage is the recognition stage.

The recognizer 320 may identify and track multiple objects corresponding to at least one companion animal included in the image data received from the camera 400 . The recognition unit 320 may identify and track at least one companion animal based on the facial data, body data, and pattern data of the at least one companion animal. The pattern data may be data including a color or a pattern. The facial data may include facial expression data. Here, in order to draw an outline by extracting edge points from the facial data of at least one companion animal, edge points between at least one component constituting the face are connected, and multiple objects can be identified based on the distance between the connecting lines between each component. . Also, the recognition unit 320 may identify the multi-object through nose print recognition of at least one companion animal.

In this case, in one embodiment of the present invention, two methods may be used to identify and track a plurality of companion animals that are multi-objects. The first is a method of identifying and tracking multi-objects using LiDAR and heterogeneous sensor data from the camera 400, and the second is a method in which the companion animal often shows a sudden turn or action, and the camera 400 ) is fixed, the object called companion animal has a characteristic that is too small to be caught by the camera, that is, has a small size and irregular movement, so a method using a parallelized CPU for object recognition and automatic tracking can be used.

<LiDAR and Camera>

The LiDAR according to an embodiment of the present invention may be, for example, a LiDAR that acquires an optical signal through an APD by irradiating a pulse laser of 905 nm band with a 4-channel LiDAR system and converts it into a distance value. In the case of the scanning optical system, a flat mirror rotation optical system is applied, and through this, a maximum horizontal viewing angle of 120 degrees is secured. The vertical resolution has a final range of 5 degrees when considering the radiation angle of the laser. The camera 400 generally does not match the horizontal viewing angle with the LiDAR by about 50 degrees based on the standard angle. To solve this, a wide-angle lens having a range of 120 degrees is applied to match the viewing angle between the image and the LiDAR. Both sensor signals can have a processing speed of 30 fps, which is processed signals in an embedded environment. Embedded system operates in Nvidia's Jetson TX2 board environment, and LiDAR receives signals through Ethernet communication and images through USB3.0 communication.

LiDAR and camera data must be integrated into one space as data from different spaces. To this end, a mathematical relationship between the two sensors is defined. In order to synchronize the space of the camera and the space of the LiDAR, the point data that the LiDAR is looking at detects the points that are taken from the pixels of the image to establish a linear mathematical relationship between the two sensors. Among the coordinate systems of the two sensors, one point of LiDAR and one pixel of the image are matched by estimating the rotation matrix and movement matrix based on the camera's coordinate system, and estimating the primary distance and main point values corresponding to the internal factors of the camera. In the method of detecting and recognizing multiple objects from fused data, 900 point data is acquired in one channel per frame for 3D point data acquired through LiDAR. In case of one-to-one operation, the burden on data processing increases. In order to optimize the signal processing, it is necessary to cluster the objects detected in the point data first.

Only the center value of the clustered point data is converted into the image space, and recognition can be performed based on the corresponding pixel. At this time, the YOLO algorithm, one of the deep learning methods, can be applied to recognize the object. The YOLO algorithm divides the image range into a grid and then selects the class with the highest probability. It is difficult for the algorithm to accurately recognize an object when an object obtained from an image becomes smaller, that is, when the amount of acquired pixel data becomes smaller. In addition, although the recognition accuracy is very high by applying the deep learning technique, there is a limit to applying only images because there are questions about the reliability.

Accordingly, in an embodiment of the present invention, object recognition is performed based on a region acquired through LiDAR, so that a limitation of a general image can be solved. In addition, since the detection part is based on the value physically measured by the estimation algorithm, reliability can be secured. As a result, the two heterogeneous data correspond to one space, and the object can be recognized based on the position obtained through LiDAR. The above-described method is a method of recognizing multiple objects using LiDAR and the camera 400, and through the convergence of low-cost LiDAR and camera so that each small business owner can easily purchase and use it at each home or providing companion animal care services. It can recognize multiple objects. By taking the advantages of heterogeneous sensors, it is possible to increase the recognition accuracy and increase the reliability of the system.

<Multi-object tracking using parallelization>

Most of the deep learning or real-time object tracking or automatic tracking is performed using GPU, but in an embodiment of the present invention, a multi-object tracker with improved speed is implemented only by CPU parallelization without utilizing GPU. For multi-object tracking, a model for each object is created based on color, speed, and size, and the movement of each object can be predicted based on the movement of the entire object in order to apply the specialized technology to the companion animal observation image. The behavior or pattern of companion animals continuously changes with time, and the shape and color of objects also change according to the location of the object. You can set the learning rate to be slow for this purpose. To improve tracking speed performance, it is assumed that each object moves within a window of N times the width and length of the current location, and object tracking is executed only within the corresponding range, so that unnecessary calculations can be avoided.

Updating and tracking object models in dog cafes, dog hotels, and dog playgrounds that provide companion animal care services can be performed independently, enabling high speed performance improvement without loss of accuracy performance through parallelization. However, prediction through the entire object movement is possible after all object tracking is finished, so it is not subject to parallelization. If parallelization using a high-performance GPU is used, the tracking speed can be increased, but the price of the tracking server rises, so it is not suitable for small businesses. Accordingly, in one embodiment of the present invention, it is possible to track a plurality of companion animals in the shooting area using only the CPU so that it can be applied to each home or each small business establishment that provides a care service. OpenMP API can be used for parallelization, and OpenCV can be used for image processing tasks and comparative experiments.

In dog cafes, dog parks, dog hotels, etc., pets of different sizes are mixed, so even a slight glance by the manager can lead to a dog bite accident. It leads to an untimely death that occurs within seconds. Dogs use their mouths with their hands, but due to the nature of the dog, until the prey stops breathing (until the movement of the prey stops), it bites the prey and passes its teeth under the hide, then shaking and tearing remains as an instinct. The part is inevitably a vital point, and if you bite and shake it, the organ will puncture and eventually die, or even if you bite a part that is not a vital part, it leads to serious injuries, such as tearing the part. For this reason, the manager cannot go to the bathroom or eat even for a short period of time, so he has to take turns. many.

Therefore, first, it is possible to designate a spatial area for tracking object coordinates in a space that dogs or cats can travel and to designate an initial object area, and to re-designate the tracking area in real-time and pause situations to correct tracking errors. In addition, a model initialization function may be further included in order to respond to a model change due to a sudden environmental change or incorrect learning. In order to analyze multiple objects in real-time in multi-cat or multi-dog households or dog playgrounds or cafes, each object can be tracked in real time. .

The transmitter 330 may transmit identification and tracking result data together with image data. The image data may include sound data. Accordingly, it is possible to extract expression data from facial data, behavior data from body data, and sound data from image data, and analyze them. At this time, the method of collecting each data to identify companion animals and extracting status information of companion animals according to the behaviors, expressions, and sounds of each companion animal is described in the Applicant's Pre-Registered Patent No. 10-2176934 (November 2020). Notice on the 10th), so a detailed description will be omitted.

At this time, a matter not disclosed in the prior patent is a case of analyzing sound data, which is an important element for knowing the state of a dog or cat. In order to analyze this, the pre-stored artificial intelligence algorithm includes sound data and vibration data of at least one companion animal in a normal state included in the list of at least one dog breed or cat breed, and sound data for each type of abnormal state of the at least one companion animal. And it may be an algorithm modeled by using vibration data as a training data set and a training data set.

To this end, first, the transmitter 330 may perform pre-processing including noise removal on the collected sound data and vibration data. This is because, if noise is not removed, the results of data analysis methods, which are highly dependent on data, will be greatly affected, and eventually the results will be meaningless. In this case, when the noise has already been removed by the transmitter 330, this process may be skipped. Second, the transmitter 330 may apply a pre-mapped and stored weight to the selected companion animal with respect to the sound data collected after pre-processing. At this time, the weight is the weight when the error with the label is the smallest while modeling the artificial intelligence algorithm, and is mapped to an identifier such as the type of each companion animal and stored. Accordingly, when the user terminal 100 searches for the type and breed of the companion animal or selects the type, a corresponding identifier may be extracted and a weight mapped to the identifier may be extracted.

Third, the transmitter 330 may convert the collected sound data into a spectrogram image using Short-Time Fourier Trasnform (STFT). In this case, there is a problem in that an error occurs in the frequency analysis technique for an abnormal signal of the Fourier transform, and a method is needed to perform frequency analysis even at the moment when such an error occurs. Accordingly, STFT is a method of dividing a target signal into frame units in consideration of signal stability and performing Fourier transform while moving a window of a certain size. The formula for STFT in the time domain is STFT(t,w)=∫x(τ-t)e- ^jwτ dτ, and the integral range is from -infinity to +infinity. Here, x(τ) is a signal to be analyzed, and w(τ-t) is a window function. In addition, a spectrogram is a tool for visualizing and grasping a sound or a wave, and the characteristics of a waveform and a spectrum are combined. In the waveform, you can see the change in the time axis, that is, the change in the amplitude axis in the time domain. In the spectrum, you can see the change in the amplitude axis according to the change in the frequency axis. In the spectrogram, the difference in amplitude according to the change in the time and frequency axes is printed It is expressed as a difference in density or display color.

Fourth, the transmitter 330 may extract feature data from the transformed spectrogram image using a Convolutional Neural Network (CNN). Since the spectrogram is output as an image, as a result, when comparing sound data recorded by a user through image analysis with pre-stored sound data, the image should be used for comparison. At this time, when trying to calculate the similarity between different images, it is difficult to determine the degree of similarity between images through pixel-by-pixel comparison. There is a method of extracting and comparing images, which is deep learning technology, CNN. CNN learns the characteristics of images by itself, can classify images and recognize patterns of images based on the learned characteristics, and is suitable for image processing with a network structure using a convolutional layer. Since an image can be classified based on the features in it, it is used in an embodiment of the present invention. However, it is not limited to CNN, and any method that can distinguish an image and extract its features is possible.

CNN extracts the features of the spectrogram in the form of vectors and can measure the similarity between images by using the extracted features. In one embodiment of the present invention in order to extract the features of the spectrogram, a model using the basic structure of a convolutional neural network consisting of a convolution layer, an activation function layer, and a max pooling layer is used. It can be used, and before measuring the similarity, it is possible to extract and use the feature vector of the spectrogram from the layer before the Softmax layer used to classify the image. Basically, it can be used to measure similarity by extracting features from a model having a CNN having a fully-connected layer and a global average layer (GAP) that brings average values on a feature map. . As another model for measuring the similarity of images, a CNN-based autoencoder model can be used. The encoder consists of a convolutional neural network structure, and a decoder ( Decoder) may be included. Spectrogram features are extracted from a specific layer of the learned autoencoder model, and the feature vector generated through the GAP layer can be used to measure image similarity. Euclidean distance and cosine similarity can be measured from the image feature vectors extracted through the above three models, and using these, the degree of similarity for each spectrogram, i.e., can be sorted according to the degree of similarity. .

After the feature data for the sound data has been extracted, it is necessary to extract the feature data for the vibration data. Fifth, the transmitter 330 transmits the collected vibration data to the XYZ axis in the time domain. After performing the restructuring, the feature data can be extracted using CNN. The basic concept for this returns to the principle of pendulum movement with respect to the time axis (time domain). The weight on the spring moves up and down over time, leaving a graph while moving back and forth between the upper and lower limits based on the central point. Likewise, animal sounds may include vibration data, such as growls or low barking sounds. At this time, when using a geomagnetic sensor, a 9-axis sensor, an acceleration sensor, etc., data generation and graph visualization of which direction of the XYZ axis vibration is applied, how strong the vibration is, and how the pattern is made becomes possible. Since the generated graph is also an image, the transmitter 330 may extract feature data for the vibration data using the CNN described above.

After the characteristic data for the measured sound data and vibration data is extracted, sixthly, the transmitter 330 may compare the characteristic data extracted from the sound data and the vibration data with the previously stored characteristic data of the selected companion animal. have. For at least one companion animal, characteristic data of sound data and vibration data for normal or abnormal are already learned and stored, and even if it is abnormal, depending on the type of abnormality, for example, the dog is stressed and compulsive behavior Characteristic data of sound data and vibration data has already been learned and stored depending on whether it is a case of snoring, whether it is a case of being hungry or whining to see the owner, whether it is a case of being hungry, or a case of barking with anxiety due to sensitivity to external noise. have. Accordingly, when comparing the pre-learned characteristic data with the characteristic data of the measured sound data and vibration data, it is possible to determine whether the characteristic data is normal or abnormal, and in which case the abnormality belongs. In this case, the feature data comparison can use the Euclidean distance as described above as described above.

Finally, the transmitter 330 may select pre-stored feature data having the highest similarity as a result of the comparison. If pre-stored feature data having the highest similarity is selected in this way, a result report may be generated using the selected feature data and transmitted to the user terminal 100 . This is because, in the previously stored feature data, whether it is normal or abnormal, the cause of the abnormality in the case of abnormality, the type of abnormality, and a solution are mapped and stored in advance. At this time, when a human intervention is required, the transmitter 330 contacts a pet sitter currently capable of pet sitting from the list of pet sitters designated in advance by the user terminal 100 , and opens the gate or front door with the IoT device 500 . You can also have a pet sitter take care of your pet.

In order to learn, the sound data of the normal state and the sound data of each type of the abnormal state are converted into a spectrogram image using STFT (Short-Time Fourier Trasnform), and the converted spectrogram image is converted into a Convolutional Neural Network (CNN). ) can be used to extract feature data. After restructuring the vibration data of the steady state and the vibration data of each type of abnormal state in the time domain with respect to the XYZ axis, the feature data may be extracted using CNN. Here, when extracting feature data from the collected sound data and vibration data in a steady state and sound data and vibration data for each type of abnormal state, the extracted feature data is used based on LSTM (Long Short-Term Memory). can be re-extracted.

The tagging unit 340 may transmit a tag corresponding to the facial expression data extracted from the facial data and the behavior pattern data extracted from the body data to the user terminal 100 . Hunger, depression, stress, boredom, hunger, thirst, joy, sadness, etc. can be expressed in facial expressions, but animals also express them in actions. When you are stressed, you can also determine the condition by behavioral patterns, such as biting your own tail or biting parts of your body. Recently, a channel called YouTube that dogs watch has been developed in various ways. If you are bored, you can play a video that makes various sounds to stimulate curiosity, or if you are currently anxious, you can play a video of natural sounds or white noise that relieves stress. In addition, if it is too dark, the light may be turned on if the IoT device 500 is a light, and if ventilation is insufficient, the window may be opened if the window is automatically opened and closed by the IoT device 500 . In addition, if the IoT device 500 is a companion animal care robot, it can relieve boredom by moving along a path that may be of interest to a dog or cat, shaking the robot body, or moving in a specific pattern.

When an abnormal behavior is detected as a result of analyzing the image data collected from the camera 400, the IoT controller 350 may perform a control process previously mapped to the abnormal behavior and stored. In this case, the IoT controller 350 may control the turn-on or turn-off of at least one IoT device 500 that is linked through the home network to which the camera 400 is connected.

Hereinafter, an operation process according to the configuration of the cognitive service providing server of FIG. 2 will be described in detail with reference to FIG. 3 as an example. However, it will be apparent that the embodiment is only one of various embodiments of the present invention and is not limited thereto.

Referring to FIG. 3 , (a) the cognitive service providing server 300 collects various information according to the type of companion animal, divides it into a face, a body, and a sound, and converts it into an expression from a face, an action from a body, etc. It is divided into facial data, behavior data, and sound data, and the AI algorithm is completed through training and testing so that it can be used for facial recognition, behavior pattern analysis, and sound analysis, respectively. And, (b) the cognitive service providing server 300 maps and stores the registration information collected from the user terminal 100 and the camera 400 with the user terminal 100, and starts shooting in the camera 400 in real time to collect and analyze it.

And, referring to (c) and (d), the cognitive service providing server 300 transmits a result of recognizing each identified object to the user terminal 100 based on the analysis result, and The IoT device 500 may be controlled according to the analyzed state. As shown in (b), if the modeled artificial intelligence algorithm erroneously analyzed, for example, if feedback was received by the user terminal 100, the manager terminal (not shown) identified whether it was an obvious error or a user's recognition error. Next, if there is an obvious error, the modeled AI algorithm can be retrained with the feedback as a new input value.

The matters not described for the method for providing multi-object recognition service for companion animals using the camera of FIGS. 2 to 4 are the same as those described for the method for providing multi-object recognition service for companion animals using the camera in FIG. 1 , or Since it can be easily inferred from the described content, the following description will be omitted.

5 is a diagram illustrating a process in which data is transmitted/received between components included in the system for providing multi-object recognition service for companion animals using the camera of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of a process in which data is transmitted and received between each component will be described with reference to FIG. 5, but the present application is not limited to such an embodiment, and the example shown in FIG. 5 according to various embodiments described above will be described. It is apparent to those skilled in the art that the data transmission/reception process may be changed.

Referring to FIG. 4 , the cognitive service providing server maps and registers the user terminal and the camera from the user terminal ( S5100 ), and identifies multiple objects corresponding to at least one companion animal included in the image data received from the camera, and Track (S5200).

Then, the cognitive service providing server transmits the identification and tracking result data together with the image data (S5300).

The order between the above-described steps ( S5100 to S5300 ) is merely an example and is not limited thereto. That is, the order between the above-described steps ( S5100 to S5300 ) may be mutually changed, and some of these steps may be simultaneously executed or deleted.

The matters not described for the method for providing multi-object recognition service for companion animals using the camera of FIG. 5 are the same as those described for the method for providing multi-object recognition service for companion animals using the camera through FIGS. 1 to 4 above, or Since it can be easily inferred from the described content, the following description will be omitted.

The method of providing a companion animal multi-object recognition service using a camera according to an embodiment described with reference to FIG. 5 is also in the form of a recording medium including instructions executable by a computer, such as an application or program module executed by a computer. can be implemented. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

In the method for providing a multi-object recognition service for companion animals using a camera according to an embodiment of the present invention described above, an application basically installed in a terminal (which may include a program included in a platform or an operating system, etc. basically installed in the terminal) may be executed by the application store server, an application or an application (ie, a program) directly installed in the master terminal through an application providing server such as a web server related to the application or the corresponding service by the user. In this sense, the method for providing multi-object recognition service for companion animals using a camera according to an embodiment of the present invention described above is implemented as an application (ie, a program) installed by default in a terminal or directly installed by a user, and is installed in the terminal. It may be recorded on a computer-readable recording medium.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

Claims (10)

delete delete delete delete delete delete a camera installed or provided to face at least one companion animal in a space direction in which the companion animal lives to photograph at least one companion animal, and outputting image data of at least one companion animal;
a user terminal for outputting data as a result of recognizing the at least one companion animal when outputting the image data collected from the camera; and
A registration unit for mapping and registering the user terminal and the camera from the user terminal, a recognition unit for identifying and tracking multiple objects corresponding to the at least one companion animal included in the image data received from the camera, the identification and tracking a cognitive service providing server including a transmission unit for transmitting the result data together with the image data;
including,
The recognition unit,
Identifies and tracks the at least one companion animal based on facial data including facial expression data, body data, and pattern data including color or pattern of the at least one companion animal,
Between at least one component forming the face to identify the multi-object through nose print recognition of the at least one companion animal, or draw an outline by extracting edge points from the facial data of the at least one companion animal Connect edge points, identify multi-objects based on the connecting line distance between each component,
The cognitive service providing server,
a tagging unit for transmitting a tag corresponding to the facial expression data extracted from the facial data and the behavior pattern data extracted from the body data to the user terminal;
an IoT controller for performing a control process previously mapped to and stored in the abnormal behavior when an abnormal behavior is detected as a result of analyzing the image data collected from the camera;
further comprising,
The IoT control unit controls turn-on or turn-off of at least one IoT device that is linked through a home network to which the camera is connected,
The image data is a companion animal multi-object recognition service providing system using a camera, characterized in that it includes sound data.
delete delete delete

Images