KR20210009778A - Method and Apparatus for Recognizing Animal State using Video and Sound - Google Patents

Abstract

Disclosed are a method and an apparatus for recognizing animal conditions by using images and sounds. The method for recognizing animal conditions by using images and sounds comprises the steps of: collecting pluralities of pieces of image and sound data including the behavior, expression, and sound of animals of a target animal group, and performing annotation to determine the condition of a plurality of target animal groups corresponding to each of the pieces of the image and sound data, setting predetermined state values to groundtruth (GT) by using the results of the annotation, and constructing a training data set; receiving the training data set for performing machine learning, determining corresponding GTs as outputs, and learning weight values and parameters for the output GTs; and collecting image and sound data of animals of which the condition is to be recognized, inputting the same to a machine-learned classifier in advance, calculating a plurality of condition values for the input image and sound data to determine a condition corresponding to the highest condition value as the condition of the animals, and transmitting the determined condition of the animals to the user.

Description

Method and Apparatus for Recognizing Animal State using Video and Sound}

The present invention relates to a method and apparatus for recognizing animal conditions using images and sounds.

Today, the field of recognizing a person's facial expressions or emotions is rapidly developing, and as deep learning techniques are used to obtain facial expression information of various characters, it has become possible to more efficiently grasp a person's emotions.

Meanwhile, the number of companion animals such as dogs and cats is increasing rapidly, and communication between humans and companion animals becomes more important as the days go by. PET TV or systems that monitor companion animals from a distance have appeared, but no system has been developed that automatically recognizes the condition of animals, that is, easily communicates with animals.

According to the prior art, just as people relieve stress by watching TV, PET TV such as dog TV has a function of relieving anxiety by showing animal-related images to companion animals.

Some services also provide the ability to interact with the owner and pet. For example, when a companion animal is alone at home, it stabilizes the companion animal by showing the owner's face in real time through a remote monitor or listening to a voice.

A company called Null Information has developed an IoT interface called Pet Pulse that uses the heart rate of animals. However, since it must be closely attached to the neck of the animal, it is not only uncomfortable to the animal to be tightened, but also it is virtually impossible to recognize various animal states due to the nature of the heart rate.

In addition, as a thesis, a study on animal state recognition has been conducted using the animal's electroencephalogram (EEG) information. However, it is difficult to commercialize EEG because it is more difficult to obtain and expensive than heart rate.

Companion animals, like humans, have an inner state and express this through facial expressions, actions, and sounds. It expresses hunger, pain, joy, sadness, and various physiological phenomena, but there is a limit to how the owner responds with sense. It is difficult to communicate with animals unless you are an expert because you cannot communicate with words.

The present invention proposes a system for recognizing a state of a companion animal based on an image and sound, that is, an audio-visual (AV) signal.

As the number of companion animals such as dogs and cats increases rapidly, and communication between humans and companion animals becomes more important day by day, the technical problem to be achieved by the present invention is to monitor the expression, behavior, and sound of the target animal using image sensors and microphones. It is to provide a system that automatically recognizes the state of the animal and communicates with the animal. The present invention proposes a system for recognizing the state of a companion animal based on an image and sound, that is, an audio-visual (AV) signal.

A learning process must be preceded for the animal state recognition method using images and sounds proposed by the present invention. In order to learn, a data set must first be constructed. The data set construction process is as follows: Sufficiently large amounts of image and sound data are collected in pairs from animals similar to the target animal, for example, from several different animals of the same species. For convenience, assume that the number of animals is N. It is assumed that a sufficient number of data pairs have been collected for each state. The step in which the expert determines the state of each (video, sound) data is called annotation, and this task is performed on the collected data. In the training data set, the state specified for each data is called GT (ground-truth).

When the learning data set for the target animal is prepared, the following learning process is performed. For learning, traditional machine learning methods such as SVM can be used, or deep learning such as CNN can be used. Take CNN as an example. The CNN-based learning process includes inputting the data of the data set to the CNN, determining corresponding state values, that is, GTs as output, and learning CNN weight values and various parameters through a predetermined CNN learning method such as back propagation. It consists of steps. The learning process can usually be performed off-line in advance. In addition, it is possible to perform the above process with the image data set alone. In addition, it is possible to perform the above process with the sound data set alone. In addition, it is possible to perform the above process by considering the image and sound pair as one data.

The operation of the system to recognize the state of an animal in real time using a machine learning-based classifier is as follows. Acquiring image data and sound data of a target animal for which the state is to be recognized, inputting the acquired image and sound data to a pre-machine-learned classifier, calculating N state values, among the calculated state values It consists of selecting the best state and delivering the result value in the form of sound or video to the user.

The step of classifying the state of the animal for which the state is to be recognized using the image and sound in real time may include recognizing the state of the animal using only image data, or recognizing the state of the animal using only sound data, or It is possible to recognize the state of the animal by using image data and sound data.

The state of the animal includes an emotional state including sadness and joy, and a physiological phenomenon including hunger and bowel movements. The image information expresses the corresponding animal's expression or behavior, and the sound information expresses the corresponding animal's sound.

In another aspect, the animal state recognition apparatus using images and sounds proposed by the present invention receives a sensing unit that acquires image data and sound data of an animal to recognize the state, and receives the acquired image data and sound data. It includes an animal state determination unit that classifies the state of an animal to be recognized in real time, and a state transmission unit that transmits the determined state of the animal to a user through audio or video.

The present invention can recognize the state of a companion animal based on an image and sound, that is, an audio-visual (AV) signal. The present invention is developed as an app for a smartphone so that a user can communicate with an animal in real time. The present invention may be implemented by dedicated hardware for real-time animal communication consisting of a camera, a microphone, a speaker, a display, and a processor for calculation. When a target animal is photographed at a short distance using the device, the user can grasp the state of the target animal in real time.

1 is a flowchart illustrating a method of constructing a learning data set for recognizing an animal condition using images and sounds according to an embodiment of the present invention.
2 is a conceptual diagram of a method for recognizing an animal condition according to an embodiment of the present invention.
3 is a flowchart illustrating a learning method for recognizing an animal state using images and sounds according to an embodiment of the present invention.
4 is a view for explaining a process of classifying the state of an animal according to an embodiment of the present invention.
5 is a flowchart illustrating a method of recognizing an animal condition using images and sounds according to an embodiment of the present invention.
6 is a flowchart illustrating a method for recognizing an animal condition using images and sounds according to another embodiment of the present invention.
7 is a diagram illustrating a configuration of an apparatus for recognizing an animal condition using images and sounds according to an embodiment of the present invention.

The present invention proposes a method for classifying and recognizing the state of animals such as dogs and cats. Animals also express their condition through actions, facial expressions, and sounds. Therefore, sufficient image and sound data are obtained for the target animal group, and annotation is performed through animal experts. Through this, a data set for learning is constructed, and the data for learning is classified by a limited number of states of the animal group. In the learning stage, learning is performed with the training data set of a given target animal group using machine learning such as SVM (Support Vector Machines) or deep learning. In the inference step, images and sounds are acquired in real time from random animals belonging to the same target animal group as learning. The acquired images and sounds are input to the learned classifier and classified in real time into a specific state. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating a method of constructing a learning data set for recognizing an animal condition using images and sounds according to an embodiment of the present invention.

The proposed method for constructing a learning data set for animal state recognition using images and sounds is the step of collecting (110) a plurality of image data and sound data including animal behavior, facial expressions, and sounds for a target animal group. The step 120 of performing an annotation to determine the state of a plurality of target animals corresponding to each of the image data and sound data, and the state value specified using the result of the annotation is set to GT (groundtruth) And, it includes a step 130 of building a data set for learning.

In step 110, a plurality of image data and sound data including animal behavior, facial expressions, and sounds are collected for the target animal group. For example, the sensing of image data is possible through a fixed camera around an animal or an image acquisition device portable by a user such as a smartphone. Sensing of sound data is possible through a microphone built into a fixed camera around, a microphone attached to an animal's body, or a microphone built into a user-portable device such as a smartphone.

In step 120, an annotation is performed on each of the collected image data and sound data to determine the states of the corresponding plurality of target animal groups. Thereafter, in step 130, a defined state value is set to GT (groundtruth) using the result of performing the annotation, and a data set for learning is constructed. A training data set is constructed to determine the state of the target animal group by performing annotation, the prescribed state value of the training data set is set to GT (groundtruth), and the training data set is configured for each specific species.

2 is a conceptual diagram of a method for recognizing an animal condition according to an embodiment of the present invention.

It is largely divided into a video-based network and a sound-based network. Image information 210 is collected through an image-based network to efficiently analyze changes in expressions or behaviors of animals (230), and if sound exists at the same time, sound information 220 is collected and further analyzed through a sound network Do (240). In this way, the final state of the animal is determined by using both image data and sound data (250).

3 is a flowchart illustrating a learning method for recognizing an animal state using images and sounds according to an embodiment of the present invention.

The proposed learning method for animal state recognition using images and sounds includes the step of receiving a training data set for performing machine learning (310), and determining the GTs corresponding to the received training data set as outputs (320). ) And learning weight values and parameters for the output GTs (330).

As described above, in order to construct a data set for learning, a sufficient number of video recordings (including sound) are performed on animals of the same species as the target animal. Annotation to determine the state of animals in each data is performed using the subject animal expert. The prescribed state value of the acquired training data set is set to GT (groundtruth). Since the characteristics of each animal are different for each species, it is desirable to configure the learning data set for each specific species. A data set for learning according to an embodiment of the present invention is composed of images or sounds of animals of a specific species, and each data must be annotated in one state.

In step 310, a learning data set for performing machine learning is input. A sufficient amount of image data and sound data are required for machine learning. For example, suppose a chihuahua is a typical companion animal, a dog species. Also, suppose that there are N states of the target animal, that is, Chihuahua.

In step 320, the GTs corresponding to the inputted training data set are determined as outputs.

Using the configured training data set, learn animal state classification. In order to perform machine learning, the GT corresponding to each input of the training data set is determined as an output.

In step 330, weight values or parameters in the corresponding machine learning module are learned through an optimization process.

For example, an image part may be learned by being configured with a typical HOG + SVM module, or it may be learned by being configured with a CNN such as RESNET or DenseNet. When using a video rather than a single video, it may have a CNN + LSTM structure.

Image features and sound features for determining the state of the target animal group are extracted from each of the image data and sound data, and the state of the target animal group is classified using the extracted image features and sound features.

At this time, machine learning is performed using only the image data of the learning data set, or machine learning is performed using only the sound data of the learning data set, or machine learning is performed using the image data and sound data of the learning data set. can do.

4 is a view for explaining a process of classifying the state of an animal according to an embodiment of the present invention.

The image part 430 is divided into an image feature extraction 431 and a classification 432. The image feature extraction 431 may receive an image as an input 410 and extract a hand-crafted feature such as HoG, LBP, or Haar, or may use a neural network such as CNN.

Classification 432 classifies the input features into a machine learning method such as SVM and random forest.

A CNN, RNN, LSTM, etc. in which the image feature extraction 431 and classification 432 are integrated at one time may be used.

The sound part 440 is divided into sound feature extraction 441 and classification 442. The sound feature extraction 441 may receive a sound input 420 and extract hand-craft features such as HoG, LBP, and Haar, or may use a neural network such as CNN. The sound part 440 has a similar method to the image part 430, but only the input is different. However, in consideration of the one-dimensional characteristics of sound, DNN, RNN, LSTM, GRU, etc. may be used instead of CNN.

Meanwhile, a final classification result 480 is produced by fusion 470 of the image score 450 and the sound score 460 according to the machine learning result of the image part 430 and the machine learning result of the sound part 440. Fusion 470 usually weights averages the video scores 450 and the sound scores 460.

According to an embodiment of the present invention, it is possible to classify the state of an animal using only an image. When only an image is used, the state of the animal may be detected using only the image part learned in FIG. 4, that is, only the image during camera output. For example, the camera may be a camera fixed around an animal like a CCTV, or a moving camera like a smartphone.

According to another embodiment of the present invention, it is possible to classify the state of an animal using only sound. When only sound is used, the state of the animal may be detected using only the sound part learned in FIG. 4, that is, only the microphone output. For example, the microphone may be attached to an animal, may be fixed around it, or may be a microphone built into a smartphone.

According to another embodiment of the present invention, it is possible to classify the state of an animal using both images and sounds. The most common case of using both video and sound is the same as the combined case of using only video and using only sound.

5 is a flowchart illustrating a method of recognizing an animal condition using images and sounds according to an embodiment of the present invention.

The proposed animal state recognition method using images and sounds includes the steps of collecting image data and sound data of an animal that wants to recognize the state (510), and inputting the image data and sound data collected into a machine-learned classifier ( 520), calculating a plurality of state values for the input image data and sound data (530), determining a state corresponding to the highest state value among the calculated state values as the state of the animal (540), and It includes the step 550 of transmitting the determined animal state to the user.

In step 510, image data and sound data of an animal for which the state is to be recognized are collected. For example, the sensing of image data is possible through a fixed camera around an animal or an image acquisition device portable by a user such as a smartphone. Sensing of sound data is possible through a microphone built into a fixed camera around, a microphone attached to an animal's body, or a microphone built into a user-portable device such as a smartphone.

In step 520, the collected image data and sound data are input to a pre-machine-learned classifier.

In step 530, a plurality of state values for the input image data and sound data are calculated.

In step 540, a state corresponding to the highest state value among the calculated state values is determined as the state of the animal. By using the training data set of the pre-machine-learned classifier, the video and sound scores of the video data and the sound data are weighted and averaged, and the state value with the highest score is selected. The state corresponding to the selected state value is determined as the state of the animal.

At this time, the state of the animal is recognized using only image data of the animal for which the state is to be recognized, or the state of the animal is recognized using only sound data, or the state of the animal is recognized using image data and sound data. Recognize.

The state of the animal determined in step 550 is transmitted to the user. The state of the animal can be communicated to the user through audio or video. The state of the animal includes emotional states including sadness and joy, and physiological phenomena including hunger and bowel movements, and the image data expresses the animal's expression and behavior corresponding to the state of the animal, and the sound data includes the state of the animal. Animal sounds corresponding to are expressed.

6 is a flowchart illustrating a method for recognizing an animal condition using images and sounds according to another embodiment of the present invention.

The proposed animal state recognition method using images and sounds collects a plurality of image data and sound data including animal behavior, facial expressions, and sounds for a target animal group, and a plurality of targets corresponding to each of the image data and sound data. Performing an annotation to determine the state of the fauna, setting a prescribed state value to GT (groundtruth) using the result of the annotation execution, and constructing a data set for learning (610), performing machine learning In step 620 of learning the weight values and parameters for the output GTs and determining the corresponding GTs as output by receiving the data set for learning to be input, and collecting image data and sound data of an animal to recognize the state It inputs into a pre-machine-learned classifier, calculates a plurality of state values for the input image data and sound data, determines the state corresponding to the highest state value as the state of the animal, and delivers the determined state of the animal to the user. Step 630 is included.

In step 610, by collecting a plurality of image data and sound data including animal behavior, facial expression, and sound for the target animal group, the state of the plurality of target animal groups corresponding to the image data and sound data is determined. To do this, annotation is performed, and the specified state value is set to GT (groundtruth) using the result of the annotation execution, and a data set for learning is constructed.

For example, the sensing of image data is possible through a fixed camera around an animal or an image acquisition device portable by a user such as a smartphone. Sensing of sound data is possible through a microphone built into a fixed camera around, a microphone attached to an animal's body, or a microphone built into a user-portable device such as a smartphone.

Thereafter, the specified state value is set to GT (groundtruth) using the result of the annotation execution, and a training data set is constructed. A training data set is constructed to determine the state of the target animal group by performing annotation, the prescribed state value of the training data set is set to GT (groundtruth), and the training data set is configured for each specific species.

In step 620, a training data set for performing machine learning is received, corresponding GTs are determined as outputs, and weight values and parameters for the output GTs are learned.

A sufficient amount of image data and sound data are required for machine learning. For example, suppose a chihuahua is a typical companion animal, a dog species. Also, assume that the state of animals is N. Using the configured training data set, learn animal state classification. In order to perform machine learning, a training data set is input and the corresponding GTs are determined as outputs.

In machine learning, for example, in the case of an image part, it is possible to learn typical HOG + SVM or CNN such as RESNET or DenseNet. When using a video rather than a single video, it may have a CNN + LSTM structure.

Image features and sound features for determining the state of the target animal group are extracted from each of the image data and sound data, and the state of the target animal group is classified using the extracted image features and sound features.

At this time, machine learning is performed using only the image data of the learning data set, or machine learning is performed using only the sound data of the learning data set, or machine learning is performed using the image data and sound data of the learning data set. can do.

In step 630, the image data and sound data of the animal for which the state is to be recognized are collected and input to a pre-machine-learned classifier, and a plurality of state values for the input image data and sound data are calculated to obtain the highest state value. The corresponding state is determined as the state of the animal, and the determined state of the animal is transmitted to the user.

For example, the sensing of image data is possible through a fixed camera around an animal or an image acquisition device portable by a user such as a smartphone. Sensing of sound data is possible through a microphone built into a fixed camera around, a microphone attached to an animal's body, or a microphone built into a user-portable device such as a smartphone.

By using the training data set of the pre-machine-learned classifier, the video and sound scores of the video data and the sound data are weighted and averaged, and the state value with the highest score is selected. The state corresponding to the selected state value is determined as the state of the animal.

At this time, the state of the animal is recognized using only image data of the animal for which the state is to be recognized, or the state of the animal is recognized using only sound data, or the state of the animal is recognized using image data and sound data. Recognize.

The state of the animal includes emotional states including sadness and joy, and physiological phenomena including hunger and bowel movements, and the image data expresses the animal's expression and behavior corresponding to the state of the animal, and the sound data includes the state of the animal. Animal sounds corresponding to are expressed.

7 is a diagram illustrating a configuration of an apparatus for recognizing an animal condition using images and sounds according to an embodiment of the present invention.

The proposed animal condition recognition apparatus using an image and sound includes a sensing unit 710, an animal condition determining unit 720, and a state transmitting unit 730.

The sensing unit 710 collects image data and sound data of an animal for which a state is to be recognized. For example, the sensing of image data is possible through a fixed camera around an animal or an image acquisition device portable by a user such as a smartphone. Sensing of sound data is possible through a microphone built into a fixed camera around, a microphone attached to an animal's body, or a microphone built into a user-portable device such as a smartphone.

Image data and sound data of an animal for which the state is to be recognized are collected through a sensing unit, and an operation is performed to classify the animal state in real time through a CPU or GPU linked to the sensing unit. The image data and sound data collected by the sensing unit can be transmitted wirelessly or wired. Categorizing the animal's condition in real time like this can also be developed with a smartphone app.

The animal state determination unit 720 collects image data and sound data of an animal for which the state is to be recognized, inputs it to a pre-machine-learned classifier, and calculates a plurality of state values for the input image data and sound data, The state value corresponding to the value is selected and determined as the state of the animal.

By using the training data set of the pre-machine-learned classifier, the video and sound scores of the video data and the sound data are weighted and averaged, and the state value with the highest score is selected.

The video and sound scores according to the machine learning results of the video parts and the machine learning results of the sound parts of the training data set are weighted and averaged. For example, the result of Fusion results in N final scores. The state with the highest score is determined as the state of the animal.

The state transmission unit 730 transmits the determined animal state to the user. The emotional state including sadness and joy, and the state of the animal including physiological phenomena including hunger and bowel movements are transmitted to the user through audio or video.

The state of the animal includes emotional states including sadness and joy, and physiological phenomena including hunger and bowel movements, and the image data expresses the animal's expression and behavior corresponding to the animal's state, and the sound data includes the animal's state. Animal sounds corresponding to

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It can be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodyed. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description by those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the claims to be described later.

Claims (10)

Collecting a plurality of image data and sound data including animal behavior, facial expressions, and sounds for the target animal group;
Performing an annotation on each of the collected image data and sound data to determine a state of a corresponding plurality of target fauna; And
The step of setting the specified state value as GT (groundtruth) using the result of annotation execution and constructing a training data set
A method of constructing a learning data set for animal state recognition using images and sounds, including. Receiving a learning data set for performing machine learning;
Determining, as outputs, corresponding GTs for the received training data set; And
Learning weight values and parameters for the output GTs
Learning method for animal state recognition using images and sounds, including. The method of claim 2,
CNN machine learning including HOG and SVM, RESNET, and DenseNet is performed for image data using the training data set, or CNN and LSTM structure machine learning is performed for video data.
Machine learning is performed using only the image data of the learning data set, or machine learning is performed using only the sound data of the learning data set, or machine learning is performed using the image data and sound data of the learning data set.
Learning method for animal state recognition using images and sounds. Collecting image data and sound data of an animal whose state is to be recognized;
Inputting collected image data and sound data to a pre-machine-learned classifier;
Calculating a plurality of state values for the input image data and sound data;
Determining a state corresponding to the highest state value among the calculated state values as the state of the animal; And
Delivering the determined animal condition to the user
Animal state recognition method using images and sounds comprising a. The method of claim 4,
The step of determining the state corresponding to the highest state value among the calculated state values as the state of the animal,
By using the training data set of the pre-machine-learned classifier, the video and sound scores of the video and sound data are weighted and averaged, and the state value with the highest score is selected.
Recognizing the state of the animal using only the image data of the animal for which the state is to be recognized, or recognizing the state of the animal using only sound data, or recognizing the state of the animal using image data and sound data.
Animal condition recognition method using images and sound. The method of claim 4,
The state of the animal includes emotional states including sadness and joy, and physiological phenomena including hunger and bowel movements, and the image data expresses the animal's expression and behavior corresponding to the animal's state, and the sound data includes the animal's state. Animal sounds corresponding to
Animal condition recognition method using images and sound. Annotation to determine the state of the corresponding plurality of target fauna for each of the image data and sound data by collecting a plurality of image data and sound data including animal behavior, facial expression, and sound for the target fauna Performing, and setting a prescribed state value to GT (groundtruth) using the result of performing the annotation, and constructing a training data set;
Receiving a training data set for performing machine learning, determining corresponding GTs as outputs, and learning weight values and parameters for the output GTs; And
The image data and sound data of the animal for which the state is to be recognized are collected and input to a machine-learned classifier, and a plurality of state values for the input image data and sound data are calculated to determine the state corresponding to the highest state value. Determining the state of the animal and transmitting the determined animal state to the user
Animal state recognition method using an image and sound comprising a. A sensing unit that collects image data and sound data of an animal to recognize a state;
The image data and sound data of the animal for which the state is to be recognized are collected and input into a machine-learned classifier, and a plurality of state values for the input image data and sound data are calculated and the state value corresponding to the highest value is selected. An animal state determination unit that determines the state of the animal; And
Status transmission unit that delivers the determined animal status to the user
Animal state recognition device using an image and sound comprising a. The method of claim 8,
The animal condition determination unit,
By using the training data set of the pre-machine-learned classifier, the weighted average of the video and sound scores for the video and sound data, and the state value with the highest score is selected
Recognizing the state of the animal using only the image data of the animal for which the state is to be recognized, or recognizing the state of the animal using only sound data, or recognizing the state of the animal using image data and sound data.
Animal condition recognition device using images and sound. The method of claim 8,
The state transmission unit,
Delivering the emotional state including sadness and joy and the state of the animal including physiological phenomena including hunger and bowel movements to the user through audio or video
Animal condition recognition device using images and sound.

Images