KR102279958B1 - Method and Apparatus for Recognizing Animal State using Video and Sound - Google Patents

Abstract

A method and apparatus for recognizing animal states using images and sounds are presented. The learning method for animal state recognition using images and sounds proposed in the present invention collects a plurality of image data and sound data including an animal's behavior, expression, and sound for a target animal group, and stores the image data and sound data respectively. performing an annotation to determine the state of a plurality of target animal groups corresponding to each other, setting a prescribed state value to GT (groundtruth) using the annotation execution result, and constructing a data set for learning, a machine By receiving a training data set for performing learning, determining corresponding GTs as outputs, learning weight values and parameters for the output GTs, and collecting image data and sound data of animals to recognize the state It is input to the machine-learning classifier in advance, calculates a plurality of state values for the input image data and sound data, determines the state corresponding to the highest state value as the state of the animal, and delivers the determined state of the animal to the user. includes steps.

Description

Method and Apparatus for Recognizing Animal State using Video and Sound

The present invention relates to a method and apparatus for recognizing animal states using images and sounds.

Today, the field of recognizing human expressions or emotions is rapidly developing, and as deep learning techniques are used to obtain facial expression information of various people, it has become possible to more efficiently grasp the emotions of a person.

On the other hand, the number of companion animals such as dogs and cats is rapidly increasing, and communication between humans and companion animals becomes more important day by day. Although PET TV or systems for monitoring companion animals from a distance are emerging, no system has been developed that automatically recognizes the state of the animal, that is, communicates easily with the animal.

According to the prior art, just as people relieve stress by watching TV, PET TV, such as a dog TV, functions to relieve anxiety while showing animal-related videos to companion animals.

Some services also provide the ability for owners and pets to interact. For example, when the pet is alone in the house, it can be stabilized by showing the owner's face in real time through the remote monitor or by playing the voice.

A company called Swell Information has developed an IoT interface that uses the heart rate of animals called Pet Pulse. However, since it has to be attached closely to the animal's neck, it not only causes inconvenience to the animal, but also it is virtually impossible to recognize various animal states due to the characteristics of heart rate.

In addition, as a thesis, animal state recognition research using animal electroencephalogram (EEG) information has been conducted. However, EEG is more difficult to obtain and expensive than heart rate, making it difficult to commercialize.

Companion animals, like humans, have an inner state and express this through facial expressions, actions, and sounds. Hunger, pain, joy, sadness, and various physiological phenomena are expressed, but there is a limit to how the owner responds with his senses. It is not easy to communicate with animals unless you are an expert because you cannot communicate with words.

The present invention proposes a system for recognizing the state of a companion animal based on an image and sound, that is, an AV (Audio-Visual) signal.

Korean Patent Publication No. 10-2019-0028022 (2019.03.18.) Korean Patent Publication No. 10-1873926 (2018.07.04.) Korean Patent Publication No. 10-1785888 (2017.10.17.)

The technical problem to be achieved by the present invention is that as the number of companion animals such as dogs and cats increases rapidly, and communication between humans and companion animals becomes more important day by day, the expression, behavior, and sound of the target animal using an image sensor and a microphone, etc. The goal is to provide a system that automatically recognizes the state of the animal and communicates with the animal. The present invention proposes a system for recognizing the state of a companion animal based on an image and sound, that is, an AV (Audio-Visual) signal.

For the animal state recognition method using images and sounds proposed in the present invention, a learning process should be preceded. For learning, a data set must first be constructed. The data set construction process is as follows: Collect enough image data and sound data in pairs from animals similar to the target animal, for example, several different animals of the same species. For convenience, assume that there are N states of animals. It is assumed that a sufficient number of data pairs have been collected for each state. The step in which the expert determines the state of each (video, sound) data is called annotation, and this operation is performed on the collected data. In the training data set, the state specified for each data is called GT (ground-truth).

When the training data set for the target animal is prepared, the following learning process is performed. Traditional machine learning methods such as SVMs can be used for training, or deep learning such as CNNs can be used. Take CNN as an example. The CNN-based learning process includes the steps of inputting the data of the data set into the CNN, determining the corresponding state values, that is, GTs as outputs, and learning the weight values and various parameters of the CNN through a predetermined CNN learning method such as back propagation. consists of steps. The learning process may be performed in advance off-line. In addition, it is possible that the above process is performed with an image data set alone. In addition, it is possible that the above process is performed with a sound data set alone. It is also possible to perform the above process by considering the image and sound pair as one data.

The operation of the system for recognizing the state of an animal in real time using a machine learning-based classifier is as follows. Acquiring image data and sound data of a target animal for which state is to be recognized, inputting the obtained image and sound data to a pre-machine-learning classifier, calculating N kinds of state values, among the calculated state values It consists of a step of selecting the best state, and a step of delivering the result value in the form of sound or video to the user.

The step of classifying the state of the animal whose state is to be recognized using the image and sound in real time may include recognizing the state of the animal using only image data, or recognizing the state of the animal using only sound data, or It is possible to recognize the state of the animal using image data and sound data.

The state of the animal includes an emotional state including sadness and joy, and physiological phenomena including hunger and defecation. The image information expresses the expression or action of the animal corresponding thereto, and the sound information expresses the sound of the animal corresponding thereto.

In another aspect, the apparatus for recognizing an animal state using an image and sound proposed by the present invention includes a sensing unit that acquires image data and sound data of an animal to recognize a state, and receives the acquired image data and sound data. and an animal state determining unit that classifies the state of an animal whose state is to be recognized in real time, and a state transmitting unit that transmits the determined animal state to a user through voice or image.

According to the present invention, the state of the companion animal can be recognized based on an image and sound, that is, an audio-visual (AV) signal. The present invention can be developed as an app for a smartphone so that a user can communicate with animals in real time. The present invention may be implemented as hardware dedicated to real-time animal communication including a camera, a microphone, a speaker, a display, and a processor for calculation. When the target animal is photographed at a short distance using the device, the user can grasp the state of the target animal in real time.

1 is a flowchart illustrating a method of constructing a learning data set for recognizing an animal state using images and sounds according to an embodiment of the present invention.
2 is a conceptual diagram of a method for recognizing an animal state according to an embodiment of the present invention.
3 is a flowchart illustrating a learning method for animal state recognition using images and sounds according to an embodiment of the present invention.
4 is a view for explaining a process of classifying the state of an animal according to an embodiment of the present invention.
5 is a flowchart illustrating a method for recognizing an animal state using images and sounds according to an embodiment of the present invention.
6 is a flowchart illustrating a method for recognizing an animal state using an image and sound according to another embodiment of the present invention.
7 is a diagram illustrating the configuration of an apparatus for recognizing animal states using images and sounds according to an embodiment of the present invention.

The present invention proposes a method for classifying and recognizing the status of animals such as dogs and cats. Animals also express their state through actions, expressions, and sounds. Therefore, sufficient image and sound data are obtained for the target animal group and annotation is performed by animal experts. Through this, a training data set is constructed, and the training data is classified into a limited number of states of the corresponding animal group. In the learning stage, using machine learning such as SVM (Support Vector Machines) or deep learning, learning is performed with a training data set for a given target animal group. In the inference stage, images and sounds are acquired in real time from any animal belonging to the same target animal group as the learning. The acquired image and sound are input to the learned classifier and classified in a specific state in real time. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating a method of constructing a learning data set for recognizing an animal state using images and sounds according to an embodiment of the present invention.

The proposed method for constructing a learning data set for animal state recognition using images and sounds includes the steps of collecting a plurality of image data and sound data including the behavior, expression, and sound of animals for a target group of animals (110), the collected Performing annotation (120) to determine the state of a plurality of target animal groups corresponding to each of the image data and the sound data, and setting a prescribed state value to GT (groundtruth) using the annotation execution result and constructing a training data set (130).

In step 110 , a plurality of image data and sound data including an animal's behavior, expression, and sound are collected for a target group of animals. For example, sensing of image data is possible through a fixed camera around an animal or a user-portable image acquisition device such as a smartphone. Sensing of sound data is possible through a microphone built into a nearby fixed camera, a microphone attached to an animal body, or a microphone built into a user-portable device such as a smartphone.

In step 120, annotation is performed to determine the state of a plurality of target animal groups corresponding to each of the collected image data and sound data. Afterwards, in step 130 , a prescribed state value is set to GT (groundtruth) using the annotation execution result, and a training data set is constructed. Annotation is performed to construct a training data set for determining the state of the target animal group, set the prescribed state value of the training data set to GT (groundtruth), and configure the training data set for each specific species.

2 is a conceptual diagram of a method for recognizing an animal state according to an embodiment of the present invention.

Broadly, it is divided into an image-based network and a sound-based network. Image information 210 is collected through an image-based network to efficiently analyze changes in an animal's expression or behavior (230), and sound information 220 is collected when sound is present at the same time, and further analyzed through a sound network do (240). As described above, the final animal state is determined by using both image data and sound data information ( 250 ).

3 is a flowchart illustrating a learning method for animal state recognition using images and sounds according to an embodiment of the present invention.

The proposed learning method for animal state recognition using images and sounds includes the steps of receiving an input of a learning data set for performing machine learning (310), and determining the GTs corresponding to the received learning data set as an output (320). ) and learning weight values and parameters for the output GTs ( 330 ).

As described above, in order to construct a training data set, sufficiently many video recordings (including sound) are performed for animals of the same species as the target animal. Annotation is performed to determine the state of the animal in each data by using a subject animal expert. The prescribed state value of the acquired training data set is set to GT (groundtruth). Since each animal has different characteristics for each species, it is desirable to configure the training data set for each specific species. A learning data set according to an embodiment of the present invention consists of images or sounds of animals of a specific species, and each data must be annotated in one state.

In step 310, a training data set for performing machine learning is received. A sufficient amount of image data and sound data is required for machine learning. For example, consider the Chihuahua, one of the dog breeds that is a representative companion animal. Also, suppose that there are N states of the target animal, that is, the Chihuahua.

In step 320, GTs corresponding to the received training data set are determined as outputs.

The animal state classification is learned using the configured training data set. To perform machine learning, a GT corresponding to each input of the training data set is determined as an output.

In step 330, weight values or parameters in the corresponding machine learning module are learned through an optimization process.

For example, the image part may be trained by configuring a typical HOG + SVM module, or may be trained by being configured with a CNN such as RESNET or DenseNet. If you use a video instead of a single video, it can be a CNN + LSTM structure.

An image feature and a sound feature for determining the state of the target animal group are extracted from each of the image data and the sound data, and the state of the target animal group is classified using the extracted image feature and sound feature.

In this case, machine learning is performed using only the image data of the training dataset, or machine learning is performed using only the sound data of the training dataset, or machine learning is performed using the image data and sound data of the training dataset. can do.

4 is a view for explaining a process of classifying the state of an animal according to an embodiment of the present invention.

The image part 430 is divided into image feature extraction 431 and classification 432 . The image feature extraction 431 may receive an image 410 and extract hand-crafted features such as HoG, LBP, and Haar, or may use a neural network such as CNN.

The classification 432 classifies the input features using a machine learning method such as SVM or a random forest.

A CNN, RNN, LSTM, etc. that integrate image feature extraction 431 and classification 432 at once may be used.

The sound part 440 is divided into a sound feature extraction 441 and a classification 442 . The sound feature extraction 441 may receive a sound input 420, extract hand-craft features such as HoG, LBP, and Haar, or use a neural network such as CNN. The sound part 440 is similar in method to the image part 430 except that only the input is different. However, in consideration of the one-dimensional characteristics of sound, DNN, RNN, LSTM, GRU, etc. may be used instead of CNN.

Meanwhile, a final classification result 480 is generated by fusion 470 of the image part 430 machine learning result and the image score 450 and sound score 460 according to the sound part 440 machine learning result. Fusion 470 usually weights averages image score 450 and sound score 460 .

According to an embodiment of the present invention, it is possible to classify the state of an animal using only an image. In the case of using only the image, the state of the animal may be detected using only the image part learned in FIG. 4 , that is, using only the image from the camera output. For example, the camera may be a fixed camera around an animal like CCTV, or a moving camera like a smartphone.

According to another embodiment of the present invention, it is possible to classify the state of an animal using only sound. When only sound is used, the state of the animal may be detected using only the sound part learned in FIG. 4 , that is, using only the microphone output. For example, the microphone may be attached to an animal, may be fixed around it, or may be a microphone built into a smartphone.

According to another embodiment of the present invention, it is possible to classify the state of an animal using both an image and a sound. The case of using both video and sound is the most common, and it is the same as combining the case of using only the image and the case of using only the sound.

5 is a flowchart illustrating a method for recognizing an animal state using images and sounds according to an embodiment of the present invention.

The proposed method for recognizing an animal state using an image and sound includes the steps of collecting image data and sound data of an animal for which the state is to be recognized (510), and inputting the collected image data and sound data into a pre-machine-learned classifier ( 520), calculating a plurality of state values for the input image data and sound data (530), determining a state corresponding to the highest state value among the calculated state values as an animal state (540), and and transmitting (550) the determined state of the animal to the user.

In step 510, image data and sound data of an animal whose state is to be recognized are collected. For example, sensing of image data is possible through a fixed camera around an animal or a user-portable image acquisition device such as a smartphone. Sensing of sound data is possible through a microphone built into a nearby fixed camera, a microphone attached to an animal body, or a microphone built into a user-portable device such as a smartphone.

In step 520, the collected image data and sound data are input to the machine-learning classifier in advance.

In step 530, a plurality of state values for the input image data and sound data are calculated.

In step 540, a state corresponding to the highest state value among the calculated state values is determined as the state of the animal. By using the training data set of the machine-learning classifier in advance, the image score and the sound score for the image data and the sound data are weighted averaged, and a state value with the highest score is selected. The state corresponding to the selected state value is determined as the state of the animal.

In this case, the state of the animal is recognized using only the image data of the animal whose state is to be recognized, or the state of the animal is recognized using only the sound data, or the state of the animal is recognized using the image data and sound data. Recognize.

The state of the animal determined in step 550 is transmitted to the user. The state of the animal may be transmitted to the user through voice or video. The animal state includes an emotional state including sadness and joy, and physiological phenomena including hunger and defecation. The image data expresses the animal's expression and behavior corresponding to the state of the animal, and the sound data contains the animal state. The corresponding animal sounds are expressed.

6 is a flowchart illustrating a method for recognizing an animal state using images and sounds according to another embodiment of the present invention.

The proposed animal state recognition method using image and sound collects a plurality of image data and sound data including the behavior, expression, and sound of an animal for a target animal group, and collects a plurality of targets corresponding to each of the image data and sound data. Annotation is performed to determine the state of the animal group, a state value defined using the annotation execution result is set to GT (groundtruth), and a data set for learning is constructed (610), and machine learning is performed. In step 620 of receiving a training data set for learning, determining corresponding GTs as outputs, learning weight values and parameters for the output GTs, and collecting image data and sound data of animals to recognize the state It is input to the machine-learning classifier in advance, calculates a plurality of state values for the input image data and sound data, determines the state corresponding to the highest state value as the state of the animal, and delivers the determined state of the animal to the user. Step 630 is included.

In step 610, a plurality of image data and sound data including an animal's behavior, expression, and sound are collected for the target animal group, and the state of a plurality of target animal groups corresponding to each of the image data and sound data is determined. To do this, annotation is performed, a prescribed state value is set to GT (groundtruth) using the annotation execution result, and a training data set is constructed.

For example, sensing of image data is possible through a fixed camera around an animal or a user-portable image acquisition device such as a smartphone. Sensing of sound data is possible through a microphone built into a nearby fixed camera, a microphone attached to an animal body, or a microphone built into a user-portable device such as a smartphone.

Thereafter, a prescribed state value is set to GT (groundtruth) using the annotation execution result, and a training data set is constructed. Annotation is performed to construct a training data set for determining the state of the target animal group, set the prescribed state value of the training data set to GT (groundtruth), and configure the training data set for each specific species.

In step 620 , a training data set for performing machine learning is received, corresponding GTs are determined as outputs, and weight values and parameters for the output GTs are learned.

A sufficient amount of image data and sound data is required for machine learning. For example, consider the Chihuahua, one of the dog breeds that is a representative companion animal. Also, suppose that there are N animal states. The animal state classification is learned using the configured training data set. In order to perform machine learning, a training data set is received as an input and corresponding GTs are determined as outputs.

Machine learning can learn a typical HOG + SVM for video parts, for example, or a CNN such as RESNET or DenseNet. If you use a video instead of a single video, it can be a CNN + LSTM structure.

An image feature and a sound feature for determining the state of the target animal group are extracted from each of the image data and the sound data, and the state of the target animal group is classified using the extracted image feature and the sound feature.

In this case, machine learning is performed using only the image data of the training dataset, or machine learning is performed using only the sound data of the training dataset, or machine learning is performed using the image data and sound data of the training dataset. can do.

In step 630, the image data and sound data of the animal for which the state is to be recognized are collected and input to the machine-learning classifier in advance, and a plurality of state values for the input image data and sound data are calculated to obtain the highest state value. The corresponding state is determined as the state of the animal, and the determined state of the animal is transmitted to the user.

For example, sensing of image data is possible through a fixed camera around an animal or a user-portable image acquisition device such as a smartphone. Sensing of sound data is possible through a microphone built into a nearby fixed camera, a microphone attached to an animal body, or a microphone built into a user-portable device such as a smartphone.

By using the training data set of the machine-learned classifier in advance, the image score and the sound score for the image data and sound data are weighted averaged, and a state value with the highest score is selected. The state corresponding to the selected state value is determined as the state of the animal.

In this case, the state of the animal is recognized using only the image data of the animal whose state is to be recognized, or the state of the animal is recognized using only the sound data, or the state of the animal is recognized using the image data and sound data. Recognize.

The animal state includes an emotional state including sadness and joy, and physiological phenomena including hunger and defecation. The image data expresses the animal's expression and behavior corresponding to the state of the animal, and the sound data contains the animal state. The corresponding animal sounds are expressed.

7 is a diagram showing the configuration of an apparatus for recognizing animal states using images and sounds according to an embodiment of the present invention.

The proposed animal state recognition apparatus using images and sounds includes a sensing unit 710 , an animal state determining unit 720 , and a state transmitting unit 730 .

The sensing unit 710 collects image data and sound data of an animal whose state is to be recognized. For example, sensing of image data is possible through a fixed camera around an animal or a user-portable image acquisition device such as a smartphone. Sensing of sound data is possible through a microphone built into a nearby fixed camera, a microphone attached to an animal body, or a microphone built into a user-portable device such as a smartphone.

The image data and sound data of the animal for which the state is to be recognized are collected through the sensing unit, and an operation is performed to classify the state of the animal in real time through the CPU or GPU interlocked with the sensing unit. The image data and sound data collected by the sensing unit may be transmitted wirelessly or wired. This real-time classification of animal states can also be developed as a smartphone app.

The animal state determiner 720 collects image data and sound data of an animal for which state is to be recognized, inputs it to a pre-machine-learning classifier, calculates a plurality of state values for the input image data and sound data, and calculates the highest The state value corresponding to the value is selected to determine the state of the animal.

By using the training data set of the machine-learned classifier in advance, the image score and the sound score for the image data and sound data are weighted averaged, and a state value with the highest score is selected.

The image score and sound score according to the image part machine learning result and the sound part machine learning result of the training data set are weighted averaged. For example, Fusion results in N final scores. The state with the highest score is determined as the state of the animal.

The state transfer unit 730 transmits the determined state of the animal to the user. The state of the animal including the emotional state including sadness and joy and the physiological phenomenon including hunger and defecation is delivered to the user through voice or video.

The animal state includes an emotional state including sadness and joy, and physiological phenomena including hunger and defecation. The image data expresses the animal's expression and behavior corresponding to the state of the animal, and the sound data contains the animal state. The corresponding animal sounds are expressed.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and carry out program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (10)

For each of the plurality of image data and sound data including the behavior, expression, and sound of the animal for the target animal group collected through the sensing unit, an annotation ( annotation), setting a prescribed state value to GT (groundtruth) using the annotation execution result, and constructing a data set for learning;
receiving, by an animal state determining unit, a learning data set for performing machine learning, determining corresponding GTs as outputs, and learning weight values and parameters for the output GTs; and
By using the image data and sound data of the animal to recognize the state collected through the sensing unit, it is input to the machine-learned classifier through the animal state determiner, and a plurality of state values for the input image data and sound data are calculated. determining the state corresponding to the highest state value as the state of the animal, and transmitting the determined state of the animal to the user through the state transmitting unit
including,
The step of the animal state determining unit receiving a learning data set for performing machine learning as an input, determining corresponding GTs as outputs, and learning weight values and parameters for the output GTs includes:
Using the training dataset, perform CNN machine learning including HOG and SVM, RESNET, and DenseNet for image data, or perform machine learning with CNN and LSTM structure for video data,
Image features and sound features for determining the state of the target animal group are extracted from each of the image data and sound data, and image data machine learning results and sound data machine learning results using the extracted image features and sound features image scores and By averaging the sound scores by weighted average, fusion is performed to generate the final classification result.
A method for recognizing animal states using images and sounds. According to claim 1,
By using the image data and sound data of the animal to recognize the state collected through the sensing unit, it is input to the machine-learned classifier through the animal state determiner, and a plurality of state values for the input image data and sound data are calculated. The step of determining the state corresponding to the highest state value as the state of the animal, and transmitting the determined state of the animal to the user through the state transmitting unit,
By using the training data set of the machine-learning classifier in advance, the image and sound scores for the image data and sound data are weighted averaged, and the state value with the highest score is selected.
Recognizing the state of the animal using only the image data of the animal whose state is to be recognized, or recognizing the state of the animal using only sound data, or recognizing the state of the animal using image data and sound data
A method for recognizing animal states using images and sounds. According to claim 1,
The animal state includes an emotional state including sadness and joy, and physiological phenomena including hunger and defecation. The image data expresses the animal's expression and behavior corresponding to the state of the animal, and the sound data contains the animal state. animal sounds corresponding to
A method for recognizing animal states using images and sounds. a sensing unit for collecting image data and sound data of an animal to recognize a state;
The image data and sound data of the animal for which the state is to be recognized are collected and input to a machine-learning classifier in advance, and a plurality of state values for the input image data and sound data are calculated and the state value corresponding to the highest value is selected. an animal state determination unit to determine the state of the animal; and
A state transmitting unit that transmits the determined state of the animal to the user
including,
The animal condition determination unit,
For each of the plurality of image data and sound data including the behavior, expression, and sound of the animal for the target animal group collected through the sensing unit, an annotation ( annotation), set the prescribed state value to GT (groundtruth) using the annotation execution result, build a data set for learning,
Receives a training data set for performing machine learning as an input, determines corresponding GTs as outputs, learns weight values and parameters for the output GTs,
Using the image data and sound data of the animal to recognize the state collected through the sensing unit, it is input to the machine-learning classifier in advance, and a plurality of state values for the input image data and sound data are calculated to obtain the highest state value. determining the corresponding condition as the condition of the animal;
Using the training dataset, perform CNN machine learning including HOG and SVM, RESNET, and DenseNet for image data, or perform machine learning with CNN and LSTM structure for video data,
Image features and sound features for determining the state of the target animal group are extracted from each of the image data and sound data, and image data machine learning results and sound data machine learning results using the extracted image features and sound features image scores and By averaging the sound scores by weighted average, fusion is performed to generate the final classification result.
Animal state recognition device using image and sound. 5. The method of claim 4,
The animal condition determination unit,
By using the training data set of the machine-learning classifier in advance, the image and sound scores for the image data and sound data are weighted averaged, and the state value with the highest score is selected.
Recognizing the state of the animal using only the image data of the animal whose state is to be recognized, or recognizing the state of the animal using only sound data, or recognizing the state of the animal using image data and sound data
Animal state recognition device using image and sound. 5. The method of claim 4,
The state transfer unit,
It is a system that conveys the state of animals, including emotional states including sadness and joy, and physiological phenomena including hunger and defecation, to users through audio or video.
Animal state recognition device using image and sound. delete delete delete delete

Images