KR20210079823A - Electronic device and Method for controlling the electronic device thereof - Google Patents

Abstract

Disclosed are an electronic device and a method for controlling the same. The electronic device includes a visual sensor, a memory configured to store at least one instruction, and a processor connected to the visual sensor and the memory, and configured to control the electronic device. The processor, by executing the at least one instruction, may: identify a static object from a plurality of image frames obtained through the visual sensor to obtain a reference image comprising the identified static object; when a fall down event is detected through a trained neural network model, identify a static object from at least one image frame obtained through the visual sensor after the fall down event is detected; obtain an event image comprising the identified static object from the at least one image frame; and identify a fall down presence of a person by comparing the reference image with the event image. According to the present invention, the electronic device can provide a more accurate fall down event detection fact to a user.

Description

Electronic device and method for controlling the same

The present disclosure relates to an electronic device and a control method thereof, and the present disclosure relates to an electronic device capable of detecting a fall event based on an image obtained through a visual sensor and a control method thereof.

Recently, artificial intelligence systems have been used in various fields. Unlike the existing rule-based smart system, an artificial intelligence system is a system in which a machine learns, judges, and becomes smarter by itself. The more the artificial intelligence system is used, the better the recognition rate and the more accurate understanding of user preferences, and the existing rule-based smart systems are gradually being replaced by deep learning-based artificial intelligence systems.

In particular, in recent years, an artificial intelligence system for detecting a person's fall or occupancy by inputting an image obtained using a visual sensor into a trained neural network model has been provided.

However, in the current artificial intelligence system, there was an error in detecting a fall event even in an unintended situation even when an image is input to the trained neural network model. For example, in the case of an event in which a person is suddenly obscured by an object larger than a person in the image, or an event in which an action of a person trying to leave the room at high speed is detected, it is a situation where there is no human movement suddenly after a rapid human movement There was an error that the artificial intelligence system could not distinguish between these events and the fall event.

Accordingly, there is a need for a method for detecting a fall event more accurately by an artificial intelligence system for detecting a fall event using an image obtained through a visual sensor.

The present disclosure has been devised to solve the above problems, and an object of the present disclosure is to compare a reference image including a fixed object with an event image including a fixed object after detecting a fall event to identify whether a person has fallen. It relates to an electronic device and a method for controlling the same.

According to an embodiment of the present disclosure, an electronic device includes: a visual sensor; a memory storing at least one instruction; and a processor connected to the visual sensor and the memory to control the electronic device, wherein the processor executes the at least one instruction to obtain a fixed object from a plurality of image frames obtained through the visual sensor. A reference image including the identified fixed object is obtained by identification, and when a fall event is detected through the learned neural network model, the fixed object is obtained from at least one image frame obtained through the visual sensor after the fall event is detected. may be identified, an event image including the identified fixed object may be obtained from the at least one image frame, and whether a person has fallen may be identified by comparing the reference image with the event image.

Further, according to an embodiment of the present disclosure, a method of controlling an electronic device includes: identifying a fixed object from a plurality of image frames obtained through a visual sensor to obtain a reference image including the identified fixed object; when a fall event is detected through the learned neural network model, identifying a fixed object from at least one image frame obtained through the visual sensor after the fall event is detected; obtaining an event image including the identified fixed object from the at least one image frame; and comparing the reference image with the event image to identify whether a person has fallen.

According to an embodiment of the present disclosure as described above, by reducing the false positive probability of a fall event, the electronic device can provide a more accurate fall event detection fact to the user.

1 is a view for explaining an operation of an electronic device for detecting a fall event, according to an embodiment of the present disclosure;
2 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure;
3 is a flowchart illustrating a method for an electronic device to detect a fall event according to an embodiment of the present disclosure;
4 is a flowchart illustrating a method of adjusting a threshold value for detecting a light change amount in a dynamic vision sensor based on a brightness value obtained through an illuminance sensor according to an embodiment of the present disclosure;
5A and 5B are diagrams illustrating images acquired through a dynamic vision sensor, according to an embodiment of the present disclosure;
6 is a view for explaining a method of generating a reference image, according to an embodiment of the present disclosure;
7A and 7B are diagrams for explaining a false positive embodiment and a reconnaissance embodiment of a fall event, according to an embodiment of the present disclosure;
8 is a sequence diagram for an electronic device to provide an alarm and an image according to a human event, according to an embodiment of the present disclosure.

Hereinafter, various embodiments of the present disclosure are described. However, this is not intended to limit the technology of the present disclosure to specific embodiments, and it should be understood to include various modifications, equivalents, and/or alternatives of the embodiments of the present disclosure. .

In this document, expressions such as "has," "may have," "includes," or "may include" refer to the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.

In this document, expressions such as "A or B," "at least one of A or/and B," or "one or more of A or/and B" may include all possible combinations of the items listed together. . For example, "A or B," "at least one of A and B," or "at least one of A or B" means (1) includes at least one A, (2) includes at least one B; Or (3) it may refer to all cases including both at least one A and at least one B.

As used herein, expressions such as "first," "second," "first," or "second," may modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components. For example, the first user equipment and the second user equipment may represent different user equipment regardless of order or importance. For example, without departing from the scope of the rights described in this document, a first component may be named as a second component, and similarly, the second component may also be renamed as a first component.

As used herein, terms such as “module”, “unit”, “part”, etc. are terms used to refer to a component that performs at least one function or operation, and such component is implemented in hardware or software or may be implemented as a combination of hardware and software. In addition, a plurality of "modules", "units", "parts", etc. are integrated into at least one module or chip, and are integrated into at least one processor, except when each needs to be implemented in individual specific hardware. can be implemented as

A component (eg, a first component) is "coupled with/to (operatively or communicatively)" to another component (eg, a second component); When referring to "connected to", it should be understood that the certain element may be directly connected to the other element or may be connected through another element (eg, a third element). On the other hand, when it is said that a component (eg, a first component) is "directly connected" or "directly connected" to another component (eg, a second component), the component and the It may be understood that other components (eg, a third component) do not exist between other components.

As used herein, the expression "configured to (or configured to)" depends on the context, for example, "suitable for," "having the capacity to ," "designed to," "adapted to," "made to," or "capable of." The term “configured (or configured to)” may not necessarily mean only “specifically designed to” in hardware. Instead, in some circumstances, the expression “a device configured to” may mean that the device is “capable of” with other devices or parts. For example, the phrase “a processor configured (or configured to perform) A, B, and C” refers to a dedicated processor (eg, an embedded processor) for performing the operations, or by executing one or more software programs stored in a memory device. , may mean a generic-purpose processor (eg, a CPU or an application processor) capable of performing corresponding operations.

Terms used in this document are only used to describe specific embodiments, and may not be intended to limit the scope of other embodiments. The singular expression may include the plural expression unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meanings as commonly understood by one of ordinary skill in the art described in this document. Among the terms used in this document, terms defined in a general dictionary may be interpreted with the same or similar meaning as the meaning in the context of the related art, and unless explicitly defined in this document, ideal or excessively formal meanings is not interpreted as In some cases, even terms defined in this document cannot be construed to exclude embodiments of this document.

Hereinafter, the present disclosure will be described in more detail with reference to the drawings. However, in describing the present disclosure, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present disclosure, a detailed description thereof will be omitted. In connection with the description of the drawings, like reference numerals may be used for like components.

Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

1 is a diagram for describing an operation of an electronic device for detecting a fall event, according to an embodiment of the present disclosure. The electronic device of the present disclosure may be a CCTV or a home gateway, but this is only an example, and may be implemented as home appliances such as TV, refrigerator, washing machine, artificial intelligence speaker, etc., portable terminal such as smart phone, tablet PC, etc. have. In this case, the electronic device may include a visual sensor to acquire an image frame for detecting a fall event occurring in a specific space.

The electronic device may acquire a plurality of image frames 10 - 1 , 10 - 2 , and 10 - 3 by using a visual sensor. In this case, the visual sensor is a dynamic vision sensor (DVS), and the dynamic vision sensor is a sensor capable of acquiring an image by detecting pixels that have changed due to movement in a pixel unit. is the sensor. However, the visual sensor being a dynamic vision sensor is only an exemplary embodiment, and may also be implemented as a general image sensor.

The electronic device may identify the fixed object from the plurality of acquired image frames 10-1, 10-2, and 10-3. Specifically, in general, a dynamic vision sensor is a sensor capable of detecting a change in light due to movement, and may detect a moving object rather than a fixed object. However, the electronic device according to an embodiment of the present disclosure may adjust a threshold for detecting a change in light in the dynamic vision sensor. In particular, the electronic device may adjust the threshold value according to the illuminance of the external environment. When the threshold value of the change in light detectable by the dynamic vision sensor is lowered, the electronic device may extract a fixed object as well as a moving object from the image frame.

In addition, the electronic device may identify an object commonly detected in the plurality of image frames 10 - 1 , 10 - 2 and 10 - 3 as a fixed object.

According to another embodiment, the electronic device may include an IR light source that can irradiate IR. In addition, the electronic device may control the IR light source to emit IR while changing the intensity of the emitted IR, and the electronic device may detect the changed IR through the dynamic vision sensor to identify the moving object and the fixed object. . In addition, the electronic device may identify an object commonly detected in the plurality of image frames 10 - 1 , 10 - 2 and 10 - 3 as a fixed object.

According to another embodiment, the electronic device detects a change in light using a shutter or applies a change to a pixel value using a vibrating element such as an actuator and a motor to each of a plurality of image frames. It is possible to identify a moving object and a fixed object included in the . In addition, the electronic device may identify an object commonly detected in the plurality of image frames 10 - 1 , 10 - 2 and 10 - 3 as a fixed object.

According to another embodiment, the electronic device may acquire pixel values of a plurality of images through an image sensor, and identify a fixed object based on a fixed pixel value among the acquired pixel values.

In addition, the electronic device may acquire the reference image 15 based on the fixed object identified from the plurality of images 10 - 1 , 10 - 2 and 10 - 3 . In this case, the reference image may be an image including a fixed object in a specific space.

In particular, the electronic device 100 may obtain a reference image including a fixed object based on a data value per pixel included in a plurality of image frames. Specifically, the electronic device 100 obtains a reference image including a fixed object by using representative values (eg, average values, modes, values obtained by AI, etc.) of pixels included in a plurality of image frames. can

The electronic device may detect a fall event by inputting the image frames 20 - 1 and 20 - 2 acquired through the visual sensor into the learned neural network model. In this case, the trained neural network model is a neural network model trained to detect a human fall event based on an image frame acquired through a visual sensor, and may be implemented as a deep neural network (DNN).

When the fall event is detected, the electronic device may identify the fixed object from at least one image frame acquired through the visual sensor after the fall event is detected. In this case, the electronic device 100 may identify the fixed object from one image frame after a predetermined time after the fall event is detected, but this is only an example, and as described above, a fall is a method of acquiring a reference image. After an event is detected, a fixed object may be identified from a plurality of image frames. Thereafter, the electronic device may acquire the event image 30 including the identified fixed object from at least one image frame. In this case, the event image 30 may be an image including a fixed object obtained after a fall event is detected.

In addition, the electronic device may compare the reference image and the event image to identify whether a person has fallen. That is, the electronic device may acquire a degree of similarity between the reference image and the event image.

Specifically, as shown in FIG. 1 , when the reference image 15 and the event image 30 are different, that is, when the similarity between the reference image 15 and the event image is less than a threshold value, the electronic device detects a fall It can be determined that the event is espionage. That is, as shown in FIG. 1 , if it is identified that the additional object 35 that was not included in the reference image 15 is further included in the event image 30 acquired after the fall event is detected, the electronic device It may be determined that a human fall event has occurred.

However, when the reference image and the event image are the same, that is, when the similarity between the reference image and the event image is equal to or greater than a threshold, the electronic device may determine that the detected fall event is a false positive and may continuously monitor the fall event. For example, an event similar to a fall event in which a person leaves a field of view (FOV), an event in which a person is covered by an object, an event in which a person flies out of a door, etc. may not be judged as a fall event.

If it is determined that the detected fall event is a spy, the electronic device may provide an alarm message to an external user terminal. In this case, the alarm message may include at least one of a message including information on the fall event and an image frame after the fall event.

As described above, by comparing the fixed object before and after the fall event is detected, the electronic device may reduce the probability of false positives of the fall event.

2 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure. As shown in FIG. 2 , the electronic device 100 may include a visual sensor 110 , an illuminance sensor 120 , an IR light source unit 130 , a memory 140 , a processor 150 , and a communication unit 160 . can Meanwhile, it goes without saying that some components may be added or omitted depending on the type of the electronic device 100 in the configuration of the electronic device 100 illustrated in FIG. 2 .

The visual sensor 110 is configured to acquire an image for a specific space. In particular, according to an embodiment of the present disclosure, the visual sensor 110 may be implemented as a dynamic vision sensor (DVS), but this is only an example and may be implemented as a general image sensor.

The illuminance sensor 120 is configured to sense the illuminance of the external environment. In particular, the electronic device 100 may adjust a threshold value of a change in light detectable by the dynamic vision sensor based on the illuminance sensed by the illuminance sensor 120 .

The IR light source unit 130 is configured to emit IR light. In particular, the IR light source unit 130 may change the intensity of the IR light under the control of the processor 150 , but this is only an example and may change the emission period of the IR light.

The memory 140 may store instructions or data related to at least one other component of the electronic device 100 . In particular, the memory 140 may include a non-volatile memory and a volatile memory, and may be implemented as, for example, a flash-memory, a hard disk drive (HDD), or a solid state drive (SSD). The memory 140 is accessed by the processor 150 , and reading/writing/modification/deletion/update of data by the processor 150 may be performed. Also, the memory 140 may store a trained neural network model for detecting a fall event. In this case, the learned neural network model may be an artificial intelligence model trained to detect whether a person falls by inputting a plurality of image frames obtained through the visual sensor 110 . In particular, the trained neural network model may be executed by an existing general-purpose processor (eg, CPU) or a separate AI-only processor (eg, GPU, NPU, etc.). In addition, the memory 140 may store a plurality of components (or modules) for detecting the fall event shown in FIG. 2 . When a program for detecting a fall event is executed or the electronic device is powered on, a plurality of components stored in the memory 140 may be loaded into the processor 150 as shown in FIG. 2 .

The communication unit 160 is configured to communicate with various types of external devices according to various types of communication methods. The communication unit 160 may include a Wi-Fi module, a Bluetooth module, an infrared communication module, and a wireless communication module. In particular, the processor 150 may communicate with various external user terminals using the communication unit 160 . Specifically, the communication unit 160 may transmit at least one of information on the fall event and an image frame acquired after the fall event to the external user terminal. Also, the communication unit 160 may transmit to the external user terminal at least one of a message including information that a moving object is not detected for a threshold time and an image frame acquired after the threshold time. Also, when the electronic device 100 is not provided with the visual sensor 110 , the communication unit 160 may receive an image from an external camera device.

The processor 150 may be electrically connected to the memory 140 to control the overall operation of the electronic device 100 . In particular, the processor 150 executes at least one instruction stored in the memory 140 , thereby identifying a fixed object from a plurality of image frames acquired through the visual sensor 110 to generate a reference image including the identified fixed object. can be obtained And, when a fall event is detected through the learned neural network model, the processor 150 identifies a fixed object from at least one image frame acquired through a visual sensor after the fall event is detected, and identifies from the at least one image frame An event image including the fixed object is acquired, and by comparing the reference image and the event image, it is possible to identify whether a person has fallen.

In particular, as shown in FIG. 2 , the processor 150 includes a fixed object detection module 151 , an event detection module 152 , a reference image acquisition module 153 , an event image acquisition module 154 , and a comparison module 155 . ), and an alarm module 156 . In this case, the plurality of modules 151 to 156 may be implemented by software, but this is only an embodiment and may be implemented by a combination of software and hardware.

As an example, the fixed object detection module 151 may detect (or identify) a fixed object from a plurality of image frames. Specifically, the fixed object detection module 151 obtains the boundary surface of the object by adjusting the threshold value of the change in light that can be detected by the dynamic vision sensor, and the object included in each of the plurality of image frames based on the boundary surface of the object. may be identified, and an object commonly included in a plurality of image frames may be identified as a fixed object. In this case, the threshold value of the change in light may be changed according to the illuminance around the electronic device 100 obtained through the illuminance sensor 120 .

As another example, the fixed object detection module 151 controls the IR light source unit 130 to emit IR while changing the intensity of the IR emitted by the IR light source unit 130, and detects the changed IR intensity through a dynamic vision sensor. object can be identified. In addition, the fixed object detection module 151 may identify an object included in common among objects included in a plurality of image frames as a fixed object.

According to another embodiment, the fixed object detection module 151 detects a change in light using a shutter or uses an actuator and a motor to change a pixel value to identify an object. can In addition, the fixed object detection module 151 may identify an object included in common among objects included in a plurality of image frames as a fixed object.

As another example, when the visual sensor 110 is implemented as a general image sensor, the fixed object detection module 151 obtains pixel values of a plurality of images obtained through the image sensor, and pixels obtained from a plurality of image frames A fixed object may be identified based on a fixed pixel value among the values.

The event detection module 152 may detect a human fall event using the learned neural network model 157 . In particular, the event detection module 152 may detect a fall event by inputting a plurality of image frames acquired through the visual sensor 110 to the neural network model 157 in real time.

The reference image acquisition module 153 may acquire a reference image including the fixed object detected by the fixed object detection module 151 . In this case, the reference image may be acquired as a plurality of image frames acquired before detecting the fall event. In particular, the reference image acquisition module 153 may acquire a reference image including a fixed object based on data values per pixel included in a plurality of image frames. Specifically, the reference image acquisition module 153 uses a representative value (eg, an average value, a mode value, a value obtained by AI, etc.) of pixels included in a plurality of image frames to obtain a reference image including a fixed object. can be obtained

The event image acquisition module 154 may acquire an event image including the fixed object detected by the fixed object detection module 151 . In this case, the event image may be acquired as at least one image frame acquired after detecting the fall event. In an embodiment, the event image acquisition module 154 may acquire one image frame acquired after detecting the fall event as the event image. As another example, the event image acquisition module 154, like the reference image acquisition module 153, is an event including a fixed object based on data values per pixel included in a plurality of image frames acquired after a fall event is detected. image can be obtained. That is, the event image acquisition module 154 is fixed using representative values (eg, average values, modes, values acquired by AI, etc.) of pixels included in a plurality of image frames acquired after a fall event is detected. An event image including an object may be acquired.

The comparison module 155 may compare the reference image acquired from the reference image acquisition module 153 with the event image acquired from the event image acquisition module 154 . That is, the comparison module 155 may determine the similarity between the reference image and the event image. In this case, when the visual sensor 110 is a dynamic vision sensor, the similarity may be the similarity of the position of a pixel where a change in light is sensed. When the visual sensor !10 is a general image sensor, the similarity is the pixel value. It may be similarity.

And, if the similarity is less than the threshold, the comparison module 155 identifies the human fall event as a spy, and if the similarity is greater than or equal to the threshold, the comparison module 155 identifies the human fall event as a false positive. have. For example, if the similarity between the reference image and the event image is less than 98%, the comparison module 155 identifies that the human fall event is spying, and if the similarity is 98% or more, the comparison module 155 determines that the human fall event is It can be identified as a false positive.

If it is determined that the fall event is spying, the alarm module 156 may provide an alarm message to the user. Specifically, the alarm module 156 may transmit an alarm message including at least one of a message including information on the fall event and an image frame obtained after detecting the fall event to the external user terminal through the communication unit 160 .

Meanwhile, functions related to artificial intelligence according to the present disclosure are operated through the processor 150 and the memory 140 . The processor 150 may include one or a plurality of processors. In this case, one or more processors may be a general-purpose processor such as a CPU, an AP, a digital signal processor (DSP), or the like, a graphics-only processor such as a GPU, a vision processing unit (VPU), or an artificial intelligence-only processor such as an NPU. One or a plurality of processors control to process input data according to a predefined operation rule or artificial intelligence model stored in the memory 140 . Alternatively, when one or more processors are AI-only processors, the AI-only processor may be designed with a hardware structure specialized for processing a specific AI model.

The predefined action rule or artificial intelligence model is characterized in that it is created through learning. Here, being made through learning means that a basic artificial intelligence model is learned using a plurality of learning data by a learning algorithm, so that a predefined action rule or artificial intelligence model set to perform a desired characteristic (or purpose) is created means burden. Such learning may be performed in the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and/or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

The artificial intelligence model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and a neural network operation is performed through an operation between an operation result of a previous layer and a plurality of weight values. The plurality of weights of the plurality of neural network layers may be optimized by the learning result of the artificial intelligence model. For example, a plurality of weights may be updated so that a loss value or a cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network may include a deep neural network (DNN), for example, a Convolutional Neural Network (CNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), There may be a Deep Belief Network (DBN), a Bidirectional Recurrent Deep Neural Network (BRDNN), or a Deep Q-Networks, but is not limited to the above-described example.

Meanwhile, in addition to the configuration shown in FIG. 2 , the electronic device 100 may further include an output device such as a display (not shown) or a speaker (not shown). That is, when a fall event is detected, the electronic device 100 may output information about the fall event using an output device such as a display or a speaker.

3 is a flowchart illustrating a method for an electronic device to detect a fall event, according to an embodiment of the present disclosure.

First, the electronic device 100 may acquire a plurality of image frames through a visual sensor (S310). Specifically, the electronic device 100 may acquire a plurality of image frames through a visual sensor that captures a specific space. In this case, the electronic device 100 may acquire a plurality of image frames through a dynamic vision sensor (DVS), but this is only an example and may acquire a plurality of image frames through a general image sensor.

The electronic device 100 may obtain a reference image including the fixed object by identifying the fixed object from the plurality of image frames ( S320 ).

As an embodiment of the present disclosure, when the visual sensor is a dynamic vision sensor, the electronic device 100 may acquire a plurality of image frames including both a moving object and a fixed object by using the dynamic vision sensor. A method of acquiring a plurality of image frames including both a moving object and a fixed object will be described with reference to FIG. 4 .

Specifically, as shown in FIG. 4 , the electronic device 100 may detect the illuminance around the electronic device ( S410 ).

Then, the electronic device 100 may adjust a threshold value of a change in light for detecting an object according to the sensed illuminance ( S420 ). Specifically, the electronic device 100 may adjust the threshold value of a change in light for detecting an object to be larger as the sensed illuminance value is higher, and as the sensed illuminance value is lower, the change in light for detecting the object is increased. The threshold can be adjusted to be small.

Then, the electronic device 100 may determine whether the boundary surface of the object is identified in the image frame obtained by adjusting the threshold value of the change of light ( S430 ). Specifically, when the image frame obtained by adjusting the threshold value of light change is the first image frame 510 as shown in FIG. 5A , the electronic device 100 determines that the boundary surface of the object is not identified. can Alternatively, when the image frame obtained by adjusting the threshold value of light change is the second image frame 520 as shown in FIG. 5B , the electronic device 100 may determine that the boundary surface of the object is identified.

When the boundary surface of the object is identified (S430-Y), the electronic device 100 may identify the object from the plurality of image frames by fixing the threshold (S440). If the boundary surface of the object is not identified (S430-N), the electronic device 100 may determine whether the boundary surface of the object is identified by adjusting (reducing) the threshold again.

That is, the electronic device 100 may acquire a plurality of image frames including the object by adjusting the threshold value of the change in light for detecting the object through the method illustrated in FIG. 4 .

In another embodiment, the electronic device 100 detects an object included in a plurality of image frames by using the IR light source unit 130 to emit IR by changing the intensity of IR or by changing the emission period of IR to emit IR. can do. That is, the electronic device 100 may detect a change in light for detecting an object through the dynamic vision sensor by changing the light intensity or the light emission period of the IR light source unit 130 . Accordingly, the electronic device 100 may acquire a plurality of image frames including the object.

As another embodiment, the electronic device 100 may identify an object by detecting a change in light by using a shutter or by changing a pixel value by using an actuator and a motor.

Then, as shown in FIG. 6 , the electronic device 100 uses a plurality of image frames 610 - 1 to 610 - 6 acquired at a time before the event occurs, and a reference image 620 including a fixed object. ) can be obtained. Specifically, the electronic device 100 may identify an object commonly detected from the plurality of image frames among the objects included in the plurality of image frames as the fixed object, and obtain the reference image 620 including the fixed object. can In this case, the electronic device 100 may acquire the reference image 620 based on an image frame acquired at a specific period (eg, 10 minutes) among image frames acquired at a time before the fall event occurs, but this This is only an embodiment, and the reference image 620 may be acquired based on an image frame acquired at a specific point in time (eg, afternoon time) before the fall event occurs.

According to another embodiment of the present disclosure, when the visual sensor is a general image sensor, the electronic device 100 sets a fixed object based on a region in which pixel values are constantly maintained among pixel values of a plurality of image frames acquired through the image sensor. can be identified. In this case, the electronic device 100 may identify a fixed object based on an image frame obtained during a time having an illuminance value within a predetermined range. That is, the electronic device 100 may identify a fixed object based on pixel values of an image frame acquired in the same time period (eg, morning or evening). Then, the electronic device 100 may acquire a reference image including the identified fixed object.

Referring back to FIG. 2 , the electronic device 100 may determine whether a fall event has been detected using the learned neural network model ( S330 ). In this case, the trained neural network model may be an artificial intelligence model trained to detect a human fall event by inputting a plurality of image frames.

If a fall event is not detected (S330-N), the electronic device 100 may acquire (or update) a reference image using a plurality of image frames acquired after a predetermined time (S320). When a fall event is detected (S330-Y), the electronic device 100 may acquire at least one image frame through a visual sensor (S340).

Then, the electronic device 100 may obtain an event image including a fixed object from at least one image frame (S350). As an embodiment, the electronic device 100 may acquire an image frame at a specific point in time (eg, 1 minute after detection of the fall event) as the event image after detecting the fall event, but this is only an example, and detecting the fall event Then, an event image including a fixed object may be acquired using the method described in S320 based on the plurality of image frames acquired afterward.

Then, the electronic device 100 may compare the reference image and the event image (S360). That is, the electronic device 100 may determine whether the reference image and the event image are the same by determining the similarity between the reference image and the event image.

Then, the electronic device 100 may determine whether to fall based on the comparison result determined in S360 (S370). Specifically, as shown in FIG. 7A , when the reference image 710 and the event image 720 are different from each other, that is, when the similarity between the reference image 710 and the event image 720 is less than a threshold, the electronic device ( 100 ) may determine that the fall event is espionage and determine that a human fall exists. However, as shown in FIG. 7B , when the reference image 730 and the event image 740 are identical to each other, that is, when the similarity between the reference image 710 and the event image 720 is equal to or greater than a threshold, the electronic device ( 100) may determine that the fall event is a false positive so that there is no human fall.

If it is determined that there is a fall (S370-Y), the electronic device 100 may provide an alarm message (S380). However, if it is determined that there is no fall ( S370 -N ), the electronic device 100 may monitor a fall event ( S330 ).

8 is a sequence diagram for an electronic device to provide an alarm and an image according to a human event, according to an embodiment of the present disclosure.

The electronic device 100 may detect an event (S810). In this case, the electronic device 100 may detect the fall event described with reference to FIGS. 1 to 7B , but this is only an example and may detect an event in which a moving object is not detected for a specific time.

The electronic device 100 may transmit information about the event to the user terminal 800 (S820). In this case, the information about the event may include information on the fact that the event occurred, the type of the event, the time when the event occurred, the place where the event occurred, and information about the person where the event occurred.

The user terminal 800 may output information on the received event (S830). In this case, the user terminal 800 may visually output information about the event through an output device such as a display, but this is only an embodiment, and information about the event may be output aurally through a speaker. , can be output visually through a vibrating device.

The user terminal 800 may receive a user command for checking an image (S840).

When a user command is received, the user terminal 800 may request an image from the electronic device 100 (S850). The electronic device 100 may transmit an image obtained after detecting the event to the user terminal 800 in response to the image request (S860).

The user terminal 800 may display the transmitted image (S870).

That is, as described above, by transmitting the event information and the image after the event is detected after the event is detected, the user can more quickly check whether the event has occurred and the details of the event. Thereby, the user may be able to respond to an emergency situation more quickly.

Meanwhile, although the above-described embodiment has been described as detecting a fall event, this is only an exemplary embodiment, and the technical idea of the present disclosure may be applied to an embodiment that detects another event (eg, an occupancy event).

In addition, in the above-described embodiment, a reference image is acquired based on a plurality of image frames acquired before event detection, and it is determined whether an event is spying by comparing the acquired reference image and the event image, but this is only an example. In addition, by comparing the image frame acquired before the event detection with the image acquired after the event detection, it is possible to determine whether the event is spying. Specifically, an image frame acquired at a specific time before the event was detected (eg, 1 day before the event was detected) and acquired at a specific time after the event was detected (eg, 1 minute after the event was detected) By comparing the image frames, it is possible to determine whether the event is spying or not.

On the other hand, the term "unit" or "module" used in the present disclosure includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic block, part, or circuit. can A “unit” or “module” may be an integrally constituted part or a minimum unit or a part thereof that performs one or more functions. For example, the module may be configured as an application-specific integrated circuit (ASIC).

Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage media readable by a machine (eg, a computer). As a device capable of calling and operating according to the called command, it may include an electronic device (eg, the electronic device 100) according to the disclosed embodiments. When the command is executed by the processor, the processor directly, Alternatively, a function corresponding to the instruction may be performed using other components under the control of the processor. The instruction may include code generated or executed by a compiler or an interpreter. , may be provided in the form of a non-transitory storage medium, where 'non-transitory' means that the storage medium does not include a signal and is tangible, but data is stored on the storage medium It does not distinguish between semi-permanent or temporary storage.

According to an example, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as merchandise. The computer program product may be distributed in the form of a machine-readable storage medium (eg, compact disc read only memory (CD-ROM)) or online through an application store (eg, Play Store™). In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily created in a storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

Each of the components (eg, a module or a program) according to various embodiments may be composed of a singular or a plurality of entities, and some sub-components of the aforementioned sub-components may be omitted, or other sub-components may be It may be further included in various embodiments. Alternatively or additionally, some components (eg, a module or a program) may be integrated into a single entity to perform the same or similar functions performed by each corresponding component prior to integration. According to various embodiments, operations performed by a module, program, or other component are executed sequentially, parallel, iteratively, or heuristically, or at least some operations are executed in a different order, are omitted, or other operations are added. can be

110: visual sensor 120: illuminance sensor
130: IR light source 140: memory
150: processor 160: communication unit

Claims (18)

In an electronic device,
visual sensor;
a memory storing at least one instruction; and
a processor connected to the visual sensor and the memory to control the electronic device;
The processor, by executing the at least one instruction,
identifying a fixed object from a plurality of image frames obtained through the visual sensor to obtain a reference image including the identified fixed object;
When a fall event is detected through the learned neural network model, a fixed object is identified from at least one image frame obtained through the visual sensor after the fall event is detected,
obtaining an event image including an identified fixed object from the at least one image frame;
An electronic device that compares the reference image and the event image to identify whether a person has fallen. According to claim 1,
The visual sensor is a dynamic vision sensor,
The processor is
obtaining a boundary surface of an object by adjusting a threshold value of a change in light detectable by the dynamic vision sensor, and identifying an object included in each of the plurality of image frames based on the boundary surface of the object;
An electronic device for identifying an object included in common among objects included in the plurality of image frames as a fixed object. 3. The method of claim 2,
The threshold value of the change in light is changed according to the illuminance around the electronic device. According to claim 1,
It further includes an IR light source unit;
The visual sensor is a dynamic vision sensor,
The processor is
The IR light source emits IR while changing the intensity of the IR emitted,
The dynamic vision sensor detects the IR with the changed intensity and identifies an object included in each of a plurality of image frames,
An electronic device for identifying an object included in common among objects included in the plurality of image frames as a fixed object. According to claim 1,
The visual sensor is an image sensor,
The processor is
Obtaining pixel values of a plurality of images obtained through the image sensor,
An electronic device for identifying the fixed object based on a fixed pixel value among the obtained pixel values. According to claim 1,
The learned neural network model is,
An electronic device which is an artificial intelligence model trained to detect whether a person has fallen by inputting a plurality of image frames obtained through the visual sensor. According to claim 1,
The processor is
determining the similarity between the reference image and the event image,
If the similarity is less than a threshold value, the human fall event is identified as a false positive, and if the similarity is greater than or equal to the threshold value, the human fall event is identified as a false positive. According to claim 1,
It further includes; a communication unit including a circuit;
The processor is
When it is determined that the fall event of the person is a spy, the electronic device transmits at least one of a message including information on the fall event and an image frame after the fall event to the user terminal through the communication unit. According to claim 1,
It further includes; a communication unit including a circuit;
The processor is
When a moving object is not detected from the image acquired through the visual sensor for a threshold time, at least one of a message including information indicating that the moving object is not detected and an image frame acquired after the threshold time is transmitted to the user through the communication unit An electronic device that transmits to a terminal. A method for controlling an electronic device, comprising:
identifying a fixed object from a plurality of image frames obtained through a visual sensor to obtain a reference image including the identified fixed object;
when a fall event is detected through the learned neural network model, identifying a fixed object from at least one image frame obtained through the visual sensor after the fall event is detected;
obtaining an event image including the identified fixed object from the at least one image frame;
Comparing the reference image and the event image to identify whether a person has fallen; Control method comprising a. 11. The method of claim 10,
The visual sensor is a dynamic vision sensor,
The step of identifying a fixed object from the plurality of image frames comprises:
A boundary surface of an object is obtained by adjusting a threshold value of a change in light detectable by the dynamic vision sensor, an object included in each of the plurality of image frames is identified based on the boundary surface of the object, and the plurality of images A method of identifying and controlling objects included in common among objects included in a frame as fixed objects. 12. The method of claim 11,
The threshold value of the change in light is changed according to the illuminance around the electronic device. 11. The method of claim 10,
The electronic device further includes an IR light source,
The visual sensor is a dynamic vision sensor,
The step of identifying a fixed object from the plurality of image frames comprises:
The IR light source emits IR while changing the intensity of the IR emitted,
The dynamic vision sensor detects the IR with the changed intensity and identifies an object included in each of a plurality of image frames,
A control method for identifying an object included in common among objects included in the plurality of image frames as a fixed object. 11. The method of claim 10,
The visual sensor is an image sensor,
The step of identifying a fixed object from the plurality of image frames comprises:
Obtaining pixel values of a plurality of images obtained through the image sensor,
A control method for identifying the fixed object based on a fixed pixel value among the obtained pixel values. 11. The method of claim 10,
The learned neural network model is,
A control method that is an artificial intelligence model trained to detect whether a person has fallen by inputting a plurality of image frames obtained through the visual sensor. 11. The method of claim 10,
The step of detecting whether the fall is
determining a similarity between the reference image and the event image; and
and identifying that the human fall event is a false positive if the similarity is less than the threshold value, and identifying the human fall event as a false positive if the similarity is greater than or equal to the threshold value. 17. The method of claim 16,
When it is determined that the fall event of the person is a spy, transmitting at least one of a message including information on the fall event and an image frame after the fall event to the user terminal. 11. The method of claim 10,
When a moving object is not detected from the image obtained through the visual sensor for a threshold time, at least one of a message including information that the moving object is not detected and an image frame obtained after the threshold time is transmitted to the user terminal Control method further comprising;

Images