KR101170339B1 - System and method for management of emergency situation using recognizing expression of face, and terminal and remote monitoring apparatus - Google Patents

Abstract

The present invention relates to a system and method for managing an abnormal situation using facial expression recognition, a terminal and a remote monitoring apparatus, wherein the remote monitoring apparatus analyzes an image photographing a management object, extracts a facial image, and analyzes the extracted facial image. After recognizing the facial expression of the object, and confirming that the abnormal expression, and transmits the abnormal message to the terminal, the terminal receives a user input, and includes a configuration for remotely controlling the remote control device, according to the present invention, the sick facial expression, etc. By recognizing an abnormal facial condition, it can quickly cope with an emergency situation, and can also provide medicine or food to a managed object remotely.

Description

System and method for managing abnormal situation using facial expression recognition, terminal and remote monitoring apparatus {system and method for management of emergency situation using recognizing expression of face, and terminal and remote monitoring apparatus}

The present invention relates to an abnormal situation management method using facial expression recognition, in particular, an abnormal situation management system using facial expression recognition for identifying an expression state of a management object, such as an elderly person or a dog, and transmitting an abnormal message to a terminal in case of an abnormal facial expression; It relates to a method, a terminal and a remote monitoring apparatus.

As the age of information and aging family grows more and more, aging elderly living alone or there are no elderly caregivers in their homes, and many elderly people spend long hours alone. In many cases, accidents in which elderly people or pets suddenly fail to respond quickly when an emergency situation occurs and lose valuable lives occur.

Thus, when an emergency occurs, a service for quickly responding to an emergency using a mobile phone is provided to a person who needs protection such as an elderly person, and a phone number set in advance by an operation such as a speed dial when an emergency occurs. , 112, etc.), and an emergency telephone for notifying an emergency situation by sending an address when the other party receives the call.

However, in the case of the conventional apparatus, when the elderly person suddenly falls down and cannot press the speed dial, or when there is no ability to cope alone, such as a pet, there is a problem in that proper measures cannot be taken.

On the other hand, if pets are left alone in their homes, those who raise their pets will prepare enough food for the pets left alone in the afternoon until they return home for dinner. Compared to the lack of intelligence in the food bowl is enough food to continue to eat, and if the food bowl is not enough food to eat all the food, even if hungry pets can not eat their own food.

Due to this, there is a problem that the pet's health problems occur because the pet's food could not be packed on time, and if the pet is sick, the owner can not know the condition of the pet at any time and can cope quickly. none.

The present invention has been proposed to solve the above-mentioned conventional problems, by using a remote monitoring device provided in the home to detect the expression of the management object, for example, the elderly or pets, and if it is confirmed that the abnormal expression to the terminal of the guardian An object of the present invention is to provide an abnormal situation management system and method using facial expression recognition that can take appropriate measures by sending an abnormal message, and a terminal and a remote monitoring device.

The abnormal situation management system using the facial expression recognition of the present invention for achieving the above object, the image of the management object is analyzed to extract the face image of the management object, the face image is analyzed to analyze the After recognizing a facial expression, if the facial expression of the management object is confirmed that the abnormal expression, a remote monitoring device for transmitting an abnormal message with reference to the predetermined terminal identification information, when receiving the abnormal message from the remote monitoring device the abnormal message Outputs, receives input for controlling the remote control device through an input device, generates control information, and when the terminal receives the control information and the control information, performs an operation corresponding to the control information. It includes a remote drive device.

The remote monitoring device in the abnormal situation management system using the facial expression recognition of the present invention for achieving the above object, the image capture unit, the transmission and reception unit for transmitting and receiving data, the device storage unit for storing the terminal identification information and After analyzing the management object image photographed through the photographing unit, extracting the facial image of the management object, analyzing the face image to recognize the expression of the management object, and confirming that the expression of the management object is an abnormal expression, And a device controller for transmitting an abnormal message to the terminal through the transceiver, by referring to the terminal identification information.

In the remote monitoring apparatus in the abnormal situation management system using the facial expression recognition of the present invention, the device storage unit further stores the reference expression information, the device controller, after analyzing the face image to generate the facial expression information, The facial expression information is compared with reference expression information defining a facial expression state, characterized in that to recognize the expression of the management object.

In the remote monitoring apparatus in the abnormal situation management system using the facial expression recognition of the present invention, the device control unit, characterized in that for acquiring the face image of the management object through the photographing unit to set the reference expression information.

In the remote monitoring device in the abnormal situation management system using the facial expression recognition of the present invention, the device control unit, by analyzing the extracted face image to generate the management object identification information for identifying the management object, and the abnormal message and And transmitting the management object identification information together.

In the remote monitoring device in the abnormal situation management system using the facial expression recognition of the present invention, the device control unit, if it is confirmed that the expression of the management object is the abnormal expression, the image of the management object or the face image of the management object It is characterized in that the transmission with the abnormal message.

In the abnormal situation management system using the facial expression recognition of the present invention, the remote monitoring device further comprises a sensing unit for detecting whether the management object is close, the device control unit through the sensing unit the management object to the remote monitoring device; When detecting the proximity to the predetermined distance or more, it characterized in that for activating the photographing unit.

In the remote monitoring device in the abnormal situation management system using the facial expression recognition of the present invention, the detection unit is characterized in that it comprises at least one sensor of the infrared sensor, ultrasonic sensor and temperature sensor.

In the abnormal situation management system using the facial expression recognition of the present invention, the remote monitoring device further includes a device input unit for receiving information input, wherein the device controller receives the terminal identification information through the device input unit and receives the terminal identification information. It is characterized by storing.

In the abnormal situation management system using the facial expression recognition of the present invention, the remote monitoring device further includes a device input unit for receiving information input, and the device control unit inputs personal information including past history of the management object through the device input unit. It receives and stores in the device storage unit, and transmits the personal information with the abnormal message.

In the remote monitoring apparatus in the abnormal situation management system using the facial expression recognition of the present invention, if the device control unit does not receive a confirmation message for the abnormal message from the terminal for a predetermined time, the transmission and reception unit to the terminal through the The message may be retransmitted.

The remote monitoring device in the abnormal situation management system using the facial expression recognition according to the present invention further includes a short range communication unit performing short range communication, and the photographing unit is detachable and performs short range communication when separated from the remote monitoring apparatus. And a short range communication device, wherein the device control unit obtains the image from the photographing unit using the short range communication unit.

In the remote monitoring apparatus in the abnormal situation management system using the facial expression recognition of the present invention, the photographing unit is composed of a plurality, the device control unit obtains the image of the management object from the plurality of photographing unit, respectively, It is characterized by recognizing the expression of the management object by setting a weight to any one of the image.

In the remote monitoring apparatus in the abnormal situation management system using the facial expression recognition of the present invention, the device storage unit stores the location information of the remote monitoring device, and the device control unit transmits the location information with the abnormal message. It is done.

The terminal in the abnormal situation management system using the facial expression recognition of the present invention for achieving the above object, the communication unit for transmitting and receiving data, an output unit for expressing information, an input unit for receiving information input and a remote monitoring device through the communication unit If the remote monitoring device receives an abnormal message that recognizes the expression of the management object and confirms that the abnormal expression, the remote monitoring device outputs the abnormal message through the output unit, and receives an input for controlling the remote control unit through the input unit. And a terminal control unit which transmits the control information to the remote driving apparatus through the communication unit.

 In the abnormal situation management system using the facial expression recognition of the present invention, the terminal further includes a terminal storage unit for storing management object information, wherein the terminal control unit, when receiving the management object identification information together with the abnormal message, the terminal storage And collecting the management object information corresponding to the management object identification information from the unit and outputting the management object information through the output unit.

In the terminal in the abnormal situation management system using the facial expression recognition of the present invention, the terminal control unit, when receiving the image of the management object or the face image of the management object with the abnormal message, the management object image through the output unit Or outputting the face image of the management object.

In the terminal in the abnormal situation management system using the facial expression recognition of the present invention, the terminal control unit, water, medicine or feed selection information provided to the management object through the input unit, medicine or feed type information and water, The control information is generated by receiving information including at least one of quantity information of medicine or feed.

An abnormal situation management method using facial expression recognition of the present invention for achieving the above object, an extraction step of extracting a face image of the management object by analyzing the image captured by the remote monitoring device, the remote monitoring device Recognizing step of recognizing the expression of the management object by analyzing the face image, if the remote monitoring device determines that the recognition result is abnormal expression, transmitting step of transmitting an abnormal message to the terminal with reference to the predetermined terminal identification information It includes.

 In the abnormal situation management method using the facial expression recognition of the present invention, the recognition step, after the remote monitoring device analyzes the facial image to generate the facial expression information, and the reference expression information to define the facial expression information and facial expression state; In comparison, the facial expression of the management object is characterized.

The abnormal situation management method using facial expression recognition of the present invention further includes a setting step of setting the reference expression information by using the face image of the remote monitoring apparatus photographing the management object before the extraction step. It is done.

In the abnormal situation management method using facial expression recognition of the present invention, after the extraction step, the remote monitoring apparatus analyzes the extracted face image and generates an identification information generation step of generating management object identification information for identifying the management object. The transmitting step may further include transmitting, by the remote monitoring apparatus, the management object identification information together with the abnormal message.

In the abnormal situation management method using the facial expression recognition of the present invention, the transmitting step, if the remote monitoring apparatus determines that the facial expression of the management object is the abnormal expression, the image of the management object or the face image of the management object It is characterized in that the transmission with the abnormal message.

In the abnormal situation management method using facial expression recognition of the present invention, after the transmitting step, the remote monitoring device receiving a confirmation message for the abnormal message from the terminal and the remote monitoring device for a predetermined time If it does not receive a message, characterized in that it further comprises a retransmission step of retransmitting the abnormal message to the terminal.

In the abnormal situation management method using the facial expression recognition of the present invention for achieving the above object, when the terminal receives an abnormal message from the remote monitoring device to recognize the facial expression of the management object and confirmed that the abnormal expression, An output step of outputting an abnormal message, a generation step of generating control information by receiving an input for controlling the remote control device by the terminal, and transmitting the control information generated by the terminal to the remote control device; .

In the abnormal situation management method using facial expression recognition of the present invention, in the outputting step, when the terminal receives the management object identification information together with the abnormal message, the management object information corresponding to the management object identification information is collected from the inside. It characterized by outputting.

In the abnormal situation management method using the facial expression recognition of the present invention, the output step, when the terminal receives the image of the management object or the face image of the management object with the abnormal message, the management object image or the management And outputting an object face image.

According to the present invention, by detecting the facial expression state of the management object, for example, the elderly or pets, and recognizes the abnormal facial expression state, such as a sick facial expression, it can send an abnormal message to the terminal of the guardian, there is an advantage that can respond quickly to an emergency situation. .

In addition, the medicine can be provided to the elderly remotely, and when the pet is in a hungry facial expression state, the terminal can provide food remotely.

1 is a block diagram of an abnormal situation management system using facial expression recognition according to an embodiment of the present invention.
2 is a block diagram of a remote monitoring apparatus according to the embodiment of FIG. 1.
3 is a configuration diagram of a terminal according to the embodiment of FIG. 1.
4 is a flowchart illustrating a method for managing an abnormal situation using facial expression recognition according to an embodiment of the present invention.
5 is a flowchart illustrating a method for managing an abnormal situation using facial expression recognition according to another embodiment of the present invention.
6 is a flowchart illustrating an abnormal situation management method using facial expression recognition according to another exemplary embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. However, the detailed description of known functions and configurations incorporated herein will be omitted when it may unnecessarily obscure the subject matter of the present invention. In addition, the same reference numerals are used for parts having similar functions and functions throughout the drawings.

In the present specification, when one component 'transmits' data or a signal to another component, it may directly transmit data or a signal, and may transmit data or a signal through at least one other component. it means.

1 is a block diagram of an abnormal situation management system using facial expression recognition according to an embodiment of the present invention. Referring to FIG. 1, the abnormal situation management system using facial expression recognition according to the present embodiment includes a remote monitoring apparatus 10, a terminal 20, a remote driving apparatus 30, and a network 40.

The network 40 refers to a data communication network for data transmission and information exchange between the remote monitoring apparatus 10, the terminal 20, and the remote driving apparatus 30. The network 40 includes the Internet, an intranet, an extranet, or other wired, wireless, or combinations thereof well known or later developed.

The remote monitoring apparatus 10 is a device capable of transmitting and receiving data using the network 40. The remote monitoring apparatus 10 may include, but is not limited to, a computer, a notebook, a home appliance, and the like, and in particular, a message may be transmitted and received using a service such as a short message service (SMS) or a multimedia message service (MMS). It may be a device having a camera that can take an image of an object.

The terminal 20 is a terminal of a computer, a laptop, a home appliance, or the like capable of transmitting and receiving data using the network 40. In particular, the terminal 20 may be a personal digital assistant (PDA) capable of transmitting and receiving wireless data using the network 40, a portable multimedia player (PMP) that is a portable multimedia player, a mobile communication terminal, and the like. ) Can be a terminal capable of transmitting and receiving messages using a service such as SMS, MMS.

The remote drive device 30 is a device capable of receiving data using the network 40. The remote drive device 30 of the present embodiment receives the remote control signal and performs an operation such as rotating or stopping the motor.

The remote monitoring apparatus 10 according to the present embodiment captures an image of a management object, extracts a management object's face image by analyzing the captured management object's image, and analyzes the extracted management object's face image to express an expression of the management object. Recognize.

The management object may be a senior citizen or a pet in general, but is not limited thereto, and may include a human or animal in need of protection.

When the remote monitoring apparatus 10 confirms that the recognized expression of the management object is an abnormal expression, that is, confirms that the management object is making a hungry or painful expression, the terminal monitoring information of the preset terminal 20 is referred to. The abnormal message is transmitted to the terminal 20 through the network 40.

The terminal 20 identification information is information indicating a unique identifier (terminal identifier) of the terminal, such as a mobile identification number (MIN), an electronic serial number (ESN), or an Internet Protocol (IP) address of the terminal 20.

The abnormal message is information that can inform the user of the terminal 20 that the current state of the management object is different from the normal state, and may be a message mainly using a service such as SMS, MMS, but is not limited thereto. Obviously, the message can be included if the user can recognize the state of the management object.

The terminal 20 outputs the abnormal message. At this time, it may be mainly output as text, image, etc., but may be other forms as long as the user of the terminal 20 can recognize.

In this case, the terminal 20 may be configured in plural and may be not only a terminal of a guardian of a management object, but also 119, 112, a terminal of a hospital, and the like.

The terminal 20 receives a user input and generates control information for controlling the remote control device 30 and transmits the generated control information to the remote control device 30.

The control information refers to information for controlling the remote control device 30 to perform a specific operation, and the control information is information including water, medicine or feed selection information, medicine or feed type information, quantity information provided, and the like. .

When the remote control device 30 receives the control information, the remote drive device 30 performs an operation corresponding to the control information.

In this case, the remote control device 30 may provide the water, the medicine and the feed separately, and provide the medicine or the feed according to the type of the feed, in order to perform an operation corresponding to the control information. The quantity of can be adjusted.

2 is a block diagram of the remote monitoring apparatus 10 according to the embodiment of FIG. 1. 1 and 2, the remote monitoring apparatus 10 according to the present embodiment includes a photographing unit 11, a device input unit 12, a transceiver unit 13, a near field communication unit 14, a device storage unit 15, It comprises a sensing unit 17, the device control unit 19.

The photographing unit 11 photographs an image photographed through a lens, and a camera sensor (not shown) converting the photographed image signal into an analog signal and a signal processor converting the analog image signal photographed from the camera sensor into digital data. (Not shown). The camera sensor may be a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) image sensor, and the signal processor may be implemented as a digital signal processor (DSP), but is not limited thereto.

The photographing unit 11 of the present embodiment captures an image of the management object and may be mounted inside the remote monitoring apparatus 10 or may be separately configured outside. In addition, the photographing unit 11 includes a plurality, and each photographing unit 11 may photograph an image of a management object.

The photographing unit 11 includes a short range communication device that provides an interface for transmitting / receiving data with the remote monitoring apparatus 10 when configured separately.

The device input unit 12 provides a plurality of keys for the operation of the remote monitoring apparatus 10 and generates a selection signal according to the key selection of the user of the remote monitoring apparatus 10. In addition, the device input unit 12 may be a keypad, a pointing device such as a touch pad, or an input device such as a touch screen.

The device input unit 12 of the present exemplary embodiment may receive terminal identification information of the terminal 20 or input personal information including a past history of a management object.

The transceiver 13 has an interface for transmitting and receiving data with the terminal 20 via the network 40.

The short range communication unit 14 has an interface for transmitting and receiving data with the photographing unit 11 using wired / wireless communication when the photographing unit 11 is separated from the remote monitoring apparatus 10. In this case, short-range wireless communication technologies such as ZigBee, PicoCast, Bluetooth, Ultra Wide-Band (WWB), and Wi-Fi (Wireless LAN) may be used for the wireless communication session.

The device storage unit 15 stores an application program necessary for controlling the operation of the remote monitoring apparatus 10 and data generated while executing the program.

Device storage unit 15 of the present embodiment is a storage for storing the terminal identification information of the terminal 20. In addition, the reference expression information may be further stored, and location information of the remote monitoring apparatus 10 may be stored.

The sensing unit 17 includes a sensor for sensing a management object proximate the remote monitoring apparatus 10. For this operation, the sensing unit 17 includes at least one of an infrared sensor, an ultrasonic sensor, and a temperature sensing sensor, but is not limited thereto. There is no mechanical contact with a management object approaching or being present in the remote monitoring apparatus 10. Any sensor can be detected.

Infrared sensor is divided into two types, active and passive, depending on the principle of operation. Active infrared sensor is installed in a fixed place at a certain place in a fixed direction. Infrared detection of the receiver is disturbed and detects, and passive infrared sensors are commonly referred to as PIR (Pyroelectric Infrared Ray) sensors, which have a wide field of view, and when an object emitting infrared rays approaches this field of view, Detect the infrared.

Ultrasonic sensors are classified into direct reflection type, mirror reflection type, and permeability type, and the direct reflection type periodically emits ultrasonic waves, and when the ultrasonic waves are reflected by the object, it receives the resulting echo and detects the object. A fixed reflector is installed on the opposite side of the sensor to detect the object reflected by the ultrasonic wave reflected by the reflector, and the permeable type is located in the opposite direction of the transmitter and receiver.

The temperature sensor is divided into contact and non-contact type, and is classified into an optical pyrometer using a non-contact red filter and a radiation pyrometer using the principle of emitting radiant energy as much as the temperature emitted from an object. The radiation thermometer detects the object by detecting the radiant energy emitted by the object with a detection device such as a thermofile.

The device controller 19 controls overall operations of the photographing unit 11, the device input unit 12, the transceiver unit 13, the near field communication unit 14, the device storage unit 15, and the detection unit 17. Do it.

The device controller 19 of the present embodiment activates the photographing unit 11 and acquires an image of the management object photographed by the photographing unit 11.

In this case, the device controller 19 detects whether the management object is close to the remote monitoring apparatus 10 through the sensing unit 17, and photographs when the management object is close to the remote monitoring apparatus 10 by a predetermined distance or more. The unit 11 may be activated.

By activating the photographing unit 11 only when the management object is in proximity to the remote monitoring apparatus 10, it is possible to prevent unnecessary power waste and to recognize the expression of the management object for a longer time.

When the sensing unit 17 of the remote monitoring apparatus 10 includes an infrared sensor, the device controller 19 detects the infrared rays irradiated from the body of the management object proximate the remote monitoring apparatus 10 to detect the management object. Can be.

When the sensing unit 17 of the remote monitoring device 10 includes an ultrasonic sensor, the device control unit 19 manages to approach the remote monitoring device 10 while irradiating ultrasonic waves perpendicularly to the remote monitoring device 10. The management object may be detected by detecting ultrasonic waves reflected from the body of the object.

In addition, when the sensing unit 17 of the remote monitoring apparatus 10 includes a temperature sensor, the device controller 19 may detect the management object by detecting the radiant energy emitted from the body of the management object.

The device controller 19 captures an image of the management object through the photographing unit 11 and extracts a face image of the management object by analyzing the captured image of the management object.

In this case, the device controller 19 uses face recognition algorithms such as Principle Component Analysis (PCA), Linear Discriminant Analysis (LDA), Fisher Discriminant Analysis (FDA), Independent Component Analysis (ICA), and Support Vector Machine (SVM). The face image may be extracted from the image of the management object, but the algorithm for extracting the face image of the management object is not limited thereto.

The device controller 19 analyzes the extracted face image to recognize the expression of the management object.

At this time, the facial expression of the management object may be recognized by using facial recognition algorithms such as FACS, AAM (Active Apprearance Model), Adaboost, etc., but the algorithm for recognizing the expression of the management object is not limited thereto. .

In addition, the device controller 19 may generate facial expression information by using the facial expression recognition algorithm and recognize the facial expression of the management object in comparison with the reference expression information stored in the device storage unit 15.

The facial expression information is information used as a parameter to recognize the facial expression of the management object, such as a smiley expression, angry expression, sad expression, and no expression. In this case, the management object face image may include values of coordinates of feature points such as eyes, noses, mouths, change values of coordinates of feature points, relative distances between eyes and mouth, distance between the glans, and the like. In addition, color coordinate values, such as binary red-green-blue coordinates (RGB) or hue / saturation / brightness coordinates (HSI: Hue, Saturation, Intensity Coordinates) It may include.

The reference expression information is information used as a reference parameter for defining the facial expression state and is information to be compared with the facial expression information. The standard expression information can be set based on the general face for each person or species of animal as the default value, and the position, relative distance, shape, and color of eyebrows, eyes, nose, mouth, etc. for each person or even the same species of animal. Since the back face is different, the face image unique to the management object may be obtained through the photographing unit 11, and the face image may be set based on the face image unique to the management object.

When the device controller 19 confirms that the recognized management object has an abnormal expression, that is, the management object makes a hungry facial expression, a sick expression, etc., the device controller 19 refers to the terminal identification information of the preset terminal 20, and the network The abnormal message is transmitted to the terminal 20 through the 40.

When the remote monitoring apparatus 10 includes the device input unit 12, the device control unit 19 receives a user input through the device input unit 12 and transmits terminal identification information of the terminal 20 to the device storage unit 15. Can be stored. In addition, the terminal identification information of the terminal 20 can be stored in a plurality, it is obvious that it can be deleted or changed.

In this case, there may be a plurality of managed objects that require protection, and analyzes the face image extracted from the image of the managed object to generate the managed object identification information for identifying the managed object, and transmits the managed object identification information with the abnormal message to the terminal 20. ) Can be sent.

For example, if there are a plurality of senior citizens, the type and quantity of medicines are different, so the guardian can know who is sick or who should take the medicine from the identification information of the management object. to be.

In addition, the user of the terminal 20 may transmit the image of the management object or the face image of the management object when the abnormal message is transmitted so that the user can directly check the image of the management object or the face image of the management object.

In another embodiment of the present invention, when the remote monitoring device 10 includes the device input unit 12, the device control unit 19 receives the personal information including the past history of the management object through the device input unit 12 device If it is stored in the storage unit 15 and confirms that the expression of the management object is an abnormal expression, the personal information of the management object may be transmitted together with the abnormal message.

In another embodiment of the present invention, the photographing unit 11 may be composed of a plurality, the device controller 19 receives a plurality of images of the management object, by setting a weight to any one of the plurality of received images Recognize the expression of a managed object.

When installed in various places in the house in consideration of the moving object of the management object by using the plurality of photographing units 11, the expression of the management object can be monitored at all times, and the weight of one of the plurality of images is set to express the expression of the management object. Recognition can have the effect of more accurately recognizing the facial expression state of the management object.

For example, the device controller 19 extracts a face image larger than the face image of the management object using a face recognition algorithm from the images of the plurality of management objects or extracts the color coordinates of the surrounding images and the face images. A high weight can be set for a face image with a large difference. Also, in a process of recognizing the expression of a managed object using a facial expression recognition algorithm in a face image of a managed object, a feature image such as eyebrows, eyes, and lips is extracted more. By setting the weight high, the facial expression of the management object can be recognized.

In another embodiment of the present invention, if the device controller 19 does not receive the confirmation message for the abnormal message received from the terminal 20 through the transceiver 13 for a predetermined time, the abnormal message terminal 20 ), And this process can be repeated until a confirmation message is received.

If the confirmation message has not been received for a certain time, the abnormal message may be retransmitted to have an effect that the user of the terminal 20 quickly confirms the abnormal message.

In another embodiment of the present invention, the device storage unit 15 may store the location information of the remote monitoring device 10, and when transmitting the abnormal message to the terminal 20, it may be transmitted along with the location information.

The location information is information indicating the position of the remote monitoring device 10, and the address where the remote monitoring device 10 is installed or the remote monitoring device 10 includes a GPS (Global Positioning System) receiver (not shown). It may include the location of the remote monitoring device 10 received through.

3 is a configuration diagram of a terminal 20 according to the embodiment of FIG. 1. 1 and 3, the terminal 20 according to the present embodiment includes an input unit 21, a communication unit 23, an output unit 25, a terminal storage unit 27, and a terminal control unit 29. .

The input unit 21 provides a plurality of keys for the operation of the terminal 20, and generates a selection signal according to the key selection of the user of the terminal 20. In addition, the input unit 21 may be a keypad, a pointing device such as a touch pad, or an input device such as a touch screen.

The input unit 21 of the present embodiment receives an input for controlling the driving of the remote drive device 30. You can also receive input to send a confirmation message.

The communication unit 23 has an interface for transmitting and receiving data with the remote monitoring device 10 or the remote driving device 30 via the network 40.

The output unit 25 displays information including various function menus executed in the terminal 20. In this case, an LCD (Liquid Crystal Display) or a touch screen may be used as the output unit 25, and the touch screen simultaneously functions as an output device and an input device.

The terminal storage unit 27 stores an application program necessary for controlling the operation of the terminal 20 and data generated while executing the program.

Terminal storage unit 27 of the present embodiment is a storage for storing the information of the management object.

The management object's information refers to basic information about the personal information including the name and age of the management object. Especially, if the management object is a pet, the pet includes the characteristics of the species, pet training information, and foods that are not good for the pet. It may contain useful information related to raising an animal.

The terminal controller 29 controls overall operations of the input unit 21, the communication unit 23, the output unit 25, and the terminal storage unit 27.

When the terminal controller 29 receives an abnormal message from the remote monitoring apparatus 10 that the remote monitoring apparatus 10 analyzes the expression of the management object and confirms that it is an abnormal expression, the terminal controller 29 outputs the abnormal message through the output unit 25. At this time, it may be mainly output as text, image, etc., but may be other forms as long as the user of the terminal 20 can recognize.

In this case, the remote monitoring apparatus 10 may transmit the management object identification information together with the abnormal message, and when the terminal control unit 29 receives the management object identification information, the terminal monitoring unit may transmit the management object information corresponding to the management object identification information. 27 may be collected from the output unit 25.

In addition, the remote monitoring apparatus 10 may transmit an image of the management object or a face image of the management object together with the abnormal message, and the terminal controller 29 together with the abnormal message or when there is a user input of the terminal 20. The image or face image of the management object can be output.

The terminal controller 29 receives the user's input through the input unit 21, generates control information for controlling the remote control device 30, and transmits the control information to the remote drive device 30.

In this case, the terminal controller 29 may receive data such as water, medicine or feed selection information, medicine or feed type information, water, medicine or feed quantity information, and the like through the input unit 21. 29) may generate control information including data received from the user.

The terminal controller 29 may receive a user's input in the form of a menu for the convenience of the user's input of the terminal 20, and the name, configuration, etc. of the menu may be changed or received by the user through the input unit 21. Can be reconfigured.

On the other hand, the terminal control unit 29 may output the abnormal message, and receive the user's input to transmit a confirmation message indicating that the user of the terminal 20 confirms the abnormal message to the remote monitoring device (10).

At this time, since the input from the user is a simple input for confirming that the user has received the abnormal message, it may be in any form.

4 is a flowchart illustrating a method for managing an abnormal situation using facial expression recognition according to an embodiment of the present invention. For convenience of description, the configuration shown in FIGS. 1 to 3 will be described with reference to the corresponding reference numerals. 1 and 4, the remote monitoring apparatus 10 captures an image of a management object through the photographing unit 11 (S51).

At this time, the remote monitoring device 10 detects the proximity of the management object to the remote monitoring device 10 through the sensing unit 17, and when the management object is close to the remote monitoring device 10 more than a predetermined distance, The photographing unit 11 may be activated to capture an image of the management object.

In operation S51, the remote monitoring apparatus 10 that captures an image of the management object extracts a face image of the management object by analyzing an image of the management object using a face recognition algorithm (S53).

The remote monitoring apparatus 10 extracting the face image in step S53 analyzes the face image using an expression recognition algorithm and recognizes the expression of the management object in step S55.

In this case, the remote monitoring apparatus 10 may generate facial expression information by using the facial expression recognition algorithm, and recognize the facial expression of the management object in comparison with the reference expression information stored in the device storage unit 15.

In this case, the reference expression information may be set based on a general face for each person or species of animal as a default value, and for each person or the same species, the position of the eyebrows, eyes, lips, nose, etc., relative distance, shape, Since the face color and the like are different, the remote monitoring apparatus 10 may set reference expression information by obtaining a unique face image through the photographing unit 11 and analyzing the acquired face image.

When the remote monitoring apparatus 10 that recognizes the expression of the management object in step S55 confirms that the expression of the recognized management object is an abnormal expression, that is, confirms that the management object makes a hungry facial expression, a sick expression, a preset setting The abnormal message is transmitted to the terminal 20 with reference to the terminal identification information of the terminal 20 (S57).

In this case, the remote monitoring apparatus 10 may store the terminal identification information in response to a user input, and the terminal identification information may be plural, and the terminal identification information may be deleted or changed.

In addition, there may be a plurality of management objects, and the remote monitoring apparatus 10 may generate management object identification information for identifying the management object by analyzing a face image extracted from the image of the management object and transmit the abnormal message together with the abnormal message.

Meanwhile, the remote monitoring apparatus 10 may transmit the image of the management object or the face image of the management object when the abnormal message is transmitted so that the user of the terminal 20 may directly check the image of the management object or the face image of the management object. have.

The terminal 20 that has received the abnormal message in step S57 outputs the abnormal message (S59).

In this case, when the terminal 20 receives the management object identification information together with the abnormal message, the terminal 20 may collect and output the management object information corresponding to the management object identification information from the terminal storage unit 27.

In addition, when the terminal 20 receives the image of the management object or the face image of the management object together with the abnormal message, the terminal 20 may output the image or the face image of the management object.

The terminal 20 outputting the abnormal message in step S59 receives data such as water or feed selection information, feed type information, and water or feed quantity information through the input unit 21 (S61).

At this time, the terminal 20 may receive the user's input in the form of a menu for the convenience of the user's input of the terminal 20, the name, configuration, etc. of the menu is changed by receiving the user's input through the input unit 21 Or reconfigure.

The terminal 20 receiving the user input in step S61 generates control information for controlling the remote control apparatus 30 including the data input from the user (S63), and remotely drives the generated control information. The apparatus 30 transmits to the apparatus 30 (S65).

The remote drive device 30 that has received the control information in step S65 performs an operation corresponding to the control information (S67).

In this case, the remote control device 30 may provide the water and the feed separately, to provide the feed according to the type of feed, and to adjust the amount of feed provided to perform an operation corresponding to the control information. have.

5 is a flowchart illustrating a method for managing an abnormal situation using facial expression recognition according to another embodiment of the present invention. For convenience of description, the configuration shown in FIGS. 1 to 3 will be described with reference to the corresponding reference numerals. 1, 4, and 5, the plurality of images of the management object are photographed through a plurality of photographing units 11 that are not physically mounted on the remote monitoring device 10 of the remote monitoring device 10. (S71).

In operation S71, the remote monitoring apparatus 10 that captures images of the plurality of management objects extracts the face images of the plurality of management objects from the images of the plurality of management objects by analyzing the images of the management objects using a face recognition algorithm ( S73).

In step S73, the remote monitoring apparatus 10 extracting the face images of the plurality of management objects sets a weight to any one of the plurality of face images, and recognizes the expression of the management object from the face image in which the weight is set to be high (S75). ).

For example, the remote monitoring apparatus 10 extracts a larger face image in the process of extracting a face image of a managed object using a face recognition algorithm from images of a plurality of managed objects, or color coordinate values of surrounding images and face images. The face image can be set to high weight on the face image with large difference, and the feature image such as eyebrows, eyes and lips is extracted more in the process of recognizing the expression of the management object using the facial expression recognition algorithm in the face image of the management object. You can recognize the expression of the management object by setting the weight to high.

Since steps S77 to S87 are the same processes as steps S57 to S67 of FIG. 4, description thereof is omitted.

6 is a flowchart illustrating an abnormal situation management method using facial expression recognition according to another exemplary embodiment of the present invention. For convenience of description, the configuration shown in FIGS. 1 to 3 will be described with reference to the corresponding reference numerals. 1, 4, and 6, steps S91 to S95 are the same processes as steps S51 to S55 of FIG. 4, and thus descriptions thereof will be omitted.

When the remote monitoring apparatus 10 that recognizes the expression of the management object in step S95 confirms that the expression of the recognized management object is an abnormal expression, that is, confirms that the management object makes a hungry expression, a sick expression, and the like, the device is stored. The location information is collected from the unit 15 (S97).

The remote monitoring apparatus 10 having collected the location information in step S97 refers to the terminal identification information of the terminal 20, and transmits the abnormal message and the location information to the terminal 20 (S99).

In this case, the remote monitoring apparatus 10 may receive the personal information including the past history of the management object through the device input unit 12 and store it in the device storage unit 15, and the personal object of the management object from the device storage unit 15. The information may be collected and transmitted along with the abnormal message and the location information to the terminal 20.

The terminal 20 receiving the abnormal message and the location information in step S99 outputs the abnormal message and the location information (S101).

At this time, the terminal 20 may output the address of the remote monitoring device 10 in text form when displaying the location information, and also outputs the surrounding map of the remote monitoring device 10 as an image and remote monitoring on the map. The current location of the device 10 may be displayed as a highlight or the like so that a user of the terminal 20 may recognize the output.

The terminal 20 outputting the abnormal message and the location information in step S101 receives an input from the user (S103).

In this case, since the input is a simple input for confirming that the user of the terminal 20 has received the abnormal message, the input may be in any form.

The terminal 20 receiving the input from the user in step S103 transmits a confirmation message to the remote monitoring apparatus 10 (S105).

At this time, the remote monitoring device may set a predetermined time in advance, and if the confirmation message is not received for a predetermined predetermined time, branching to step S99 and retransmitting the abnormal message and the location information, this process is confirmed You can repeat until you receive the message.

As described above and described with reference to a preferred embodiment for illustrating the technical idea of the present invention, the present invention is not limited to the configuration and operation as shown and described as such, without departing from the scope of the technical idea Those skilled in the art will appreciate that many variations and modifications to the present invention are possible. And all such modifications and changes as fall within the scope of the present invention are therefore to be regarded as being within the scope of the present invention.

According to the present invention, it is possible to appropriately manage the management objects, such as the elderly or pets, so that a separate device is not required in the portable terminal, thereby minimizing the occurrence of costs. In particular, in view of the trend of increasing the number of elderly people living alone and the number of households raising pet dogs, the way to detect the facial expression state of the management object of the present invention and take appropriate measures will not be less useful.

10: remote monitoring device 11: the recording unit
13: transceiver 12: device input
14: short-range communication unit 15: device storage unit
17: detection unit 19: device control unit
20: terminal 21: input unit
23: communication unit 25: output unit
27: terminal storage unit 29: terminal control unit
30: remote control device 40: network

Claims (27)

A plurality of face images of the management object are extracted by analyzing a plurality of images photographing the management object, a weight is set to any one of the plurality of face images, and the expression of the management object is extracted from a face image having a high weight. After recognizing, if it is confirmed that the facial expression of the management object is an abnormal expression, a remote monitoring device for transmitting an abnormal message with reference to the predetermined terminal identification information;
The terminal outputs the abnormal message when receiving the abnormal message from the remote monitoring apparatus, receives an input for controlling the remote driving apparatus through an input device, generates control information, and transmits the control information to the remote driving apparatus. ; And
A remote driving device that performs an operation corresponding to the control information when receiving the control information;
Ideal situation management system using a facial recognition. A photographing unit photographing a management object through a plurality of cameras;
Transceiver for transmitting and receiving data;
Device storage unit for storing the terminal identification information;
Analyzing a plurality of images of the management object to extract a plurality of face images of the management object, set a weight to any one of the plurality of face images, the expression of the management object from the face image is set to a high weight A device control unit for transmitting an abnormal message to the terminal through the transceiver by referring to the terminal identification information after recognizing that the expression of the management object is an abnormal expression;
Remote monitoring device comprising a. The apparatus of claim 2, wherein the device storage unit further stores reference expression information.
The apparatus controller generates a facial expression information by analyzing the facial image, and then compares the facial expression information with reference expression information defining a facial expression state to recognize the facial expression of the management object. The apparatus of claim 3, wherein the device control unit comprises:
The remote monitoring device, characterized in that for setting the reference expression information by obtaining a face image of the management object through the photographing unit. The apparatus of claim 2, wherein the device control unit comprises:
And analyzing the extracted face image to generate management object identification information for identifying the management object, and to transmit the management object identification information together with the abnormal message. The apparatus of claim 2, wherein the device control unit comprises:
And confirming that the facial expression of the management object is the abnormal facial expression, transmitting a video of the management object or a face image of the management object together with the abnormal message. The method of claim 2,
Further comprising: a sensing unit for detecting whether the management object is close,
And the device controller activates the photographing unit when the management object detects that the management object is close to the remote monitoring apparatus by a predetermined distance or more. The method of claim 7, wherein
The sensing unit remote monitoring apparatus comprising at least one sensor of the infrared sensor, the ultrasonic sensor and the temperature sensor. The apparatus of claim 2, further comprising a device input unit configured to receive information input.
And the device control unit receives the terminal identification information through the device input unit and stores the terminal identification information in the device storage unit. The apparatus of claim 2, further comprising a device input unit configured to receive information input.
And the device control unit receives personal information including the past history of the management object through the device input unit, stores the personal information in the device storage unit, and transmits the personal information with the abnormal message. The apparatus of claim 2, wherein the device control unit comprises:
If the confirmation message for the abnormal message is not received from the terminal for a predetermined time, the remote monitoring device, characterized in that for retransmitting the abnormal message to the terminal via the transceiver. The method of claim 2,
And a short range communication unit performing short range communication.
The photographing unit is detachable and includes a short range communication device that performs short range communication when separated from the remote monitoring apparatus.
And the apparatus controller acquires the image from the photographing unit by using the short range communication unit. The method of claim 2, wherein the photographing unit is composed of a plurality
And the apparatus controller acquires images of the management object from the plurality of photographing units, respectively, and sets weights to any one of the plurality of images to recognize the expression of the management object. According to claim 2, wherein the device storage unit stores the location information of the remote monitoring device,
And the device control unit transmits the location information together with the abnormal message. Communication unit for transmitting and receiving data;
An output unit for expressing information;
An input unit to receive information input;
A plurality of face images of the management object are extracted by analyzing a plurality of images photographing the management object, a weight is set to any one of the plurality of face images, and the expression of the management object is extracted from a face image having a high weight. When receiving the abnormal message confirming that the abnormal expression from the remote monitoring device to recognize, outputs the abnormal message through the output unit, receives the input for controlling the remote control device through the input unit to generate control information, the communication unit A terminal control unit which transmits the control information to the remote driving apparatus;
Terminal comprising a. The method of claim 15, further comprising a terminal storage for storing the management object information,
And receiving the management object identification information together with the abnormal message, wherein the terminal controller collects the management object information corresponding to the management object identification information from the terminal storage unit and outputs the management object information through the output unit. The method of claim 15, wherein the terminal control unit,
And receiving the image of the management object or the face image of the management object together with the abnormal message and outputting the management object image or the management object face image through the output unit. The method of claim 15, wherein the terminal control unit,
Generating the control information by receiving information including at least one of water, medicine or feed selection information, medicine or feed type information, and water, medicine or feed quantity information provided to the management object through the input unit; Terminal, characterized in that. A photographing step of photographing, by the remote monitoring apparatus, the management object through a plurality of cameras;
Extracting, by the remote monitoring apparatus, a plurality of face images of the management object by analyzing a plurality of images photographing the management object;
A recognition step of setting, by the remote monitoring apparatus, a weight to any one of the plurality of face images, and recognizing an expression of the management object from the face image having a high weight;
A transmission step of transmitting an abnormal message to the terminal with reference to preset terminal identification information when the remote monitoring apparatus confirms that the recognition result is an abnormal expression;
An abnormal situation management method using facial expression recognition comprising a. The method of claim 19, wherein the recognition step,
The remote monitoring apparatus analyzes the facial image to generate facial expression information, and then compares the facial expression information with reference expression information defining the facial expression state to recognize the facial expression of the management object. How to manage your situation. The method of claim 20, wherein before the extraction step,
A setting step of setting, by the remote monitoring apparatus, the reference expression information by using a face image photographing the management object;
An abnormal situation management method using facial expression recognition, characterized in that it further comprises. The method of claim 19, wherein after the extraction step,
An identification information generation step of generating, by the remote monitoring apparatus, management object identification information for identifying the management object by analyzing the extracted face image;
The transmitting step, the abnormal situation management method using the facial recognition, characterized in that the remote monitoring device transmits the management object identification information with the abnormal message. The method of claim 19, wherein the transmitting step,
If the remote monitoring apparatus determines that the expression of the management object is the abnormal expression, the abnormal situation management method using facial expression recognition, characterized in that for transmitting the image of the management object or the face image of the management object with the abnormal message. . 20. The method of claim 19, after the transmitting step,
A reception step of receiving, by the remote monitoring apparatus, a confirmation message for the abnormal message from the terminal;
A retransmission step of retransmitting the abnormal message to the terminal if the remote monitoring apparatus does not receive the confirmation message for a predetermined time;
An abnormal situation management method using facial expression recognition, characterized in that it further comprises. The terminal analyzes a plurality of images of the management object to extract a plurality of face images of the management object, sets a weight to any one of the plurality of face images, and sets the weight of the management object from the face image having a high weight. An output step of outputting the abnormal message upon receiving the abnormal message confirming that the abnormal facial expression is received from the remote monitoring apparatus for recognizing the facial expression;
A generation step of receiving, by the terminal, an input for controlling a remote driving apparatus and generating control information;
A transmission step of transmitting, by the terminal, the generated control information to the remote driving apparatus;
An abnormal situation management method using facial expression recognition comprising a. The method of claim 25, wherein the output step,
And receiving, by the terminal, management object identification information together with the abnormal message, collecting and outputting management object information corresponding to the management object identification information from the inside. The method of claim 25, wherein the output step,
And the terminal outputs the management object image or the management object face image when the terminal receives an image of the management object or a face image of the management object together with the abnormal message.

Images