WO2024096463A1

WO2024096463A1 - Electronic device providing service using edge computing and method for operating same

Info

Publication number: WO2024096463A1
Application number: PCT/KR2023/016953
Authority: WO
Inventors: 김용범
Original assignee: 김용범
Priority date: 2022-10-31
Filing date: 2023-10-30
Publication date: 2024-05-10
Also published as: KR20240061109A

Abstract

The present invention relates to an electronic device providing a service using edge computing and a method for operating same. A method for operating an electronic device providing a store management service may comprise the steps of: obtaining a monitoring video by capturing the interior of a store using a camera; detecting an abnormal video containing abnormal behavior of a visitor on the basis of the monitoring video and an abnormal behavior detection model; transmitting the abnormal video to the electronic device of the manager of the store; and outputting warning information within the store as the abnormal video is detected.

Description

Electronic device providing services using edge computing and method of operating the same

This relates to electronic devices that provide services using edge computing and methods of operating the electronic devices.

Edge computing is a technology that processes data in real time through small distributed servers rather than centralized servers. Edge computing can mean processing data at the edge of the network, unlike cloud computing where a central server processes all data.

Such edge computing is being applied to various technological fields such as autonomous driving and smart factories that require real-time response.

By monitoring the inside of the store in real time, we want to detect abnormal behavior of visitors in real time.

We want to quickly detect visitors behaving abnormally through electronic devices installed within the store.

According to one side, obtaining a monitoring image by photographing the interior of the store through a camera; Detecting an abnormal video containing abnormal behavior of a visitor based on the monitoring image and the abnormal behavior detection model; transmitting the abnormal video to an electronic device of the store manager; and outputting warning information within the store as the abnormal image is detected.

According to one side, the camera; communication device; processor; and a memory storing instructions executable by the processor, wherein the processor, by executing the instructions, acquires a monitoring image by photographing the interior of the store through a camera, and uses the monitoring image and an abnormal behavior detection model to Based on this, an electronic device detects an abnormal image containing abnormal behavior of a visitor, transmits the abnormal image to the electronic device of the manager of the store, and outputs warning information within the store as the abnormal image is detected. A device is provided.

By monitoring the inside of the store in real time, abnormal behavior of visitors can be detected in real time.

Electronic devices installed within the store can quickly detect visitors behaving abnormally.

The invention may be readily understood by combination of the following detailed description and accompanying drawings, where reference numerals refer to structural elements.

1 is a conceptual diagram illustrating the operation of monitoring the behavior of visitors to a store and detecting abnormal behavior using an electronic device within the store, according to an embodiment.

Figure 2 is a flowchart showing a method of operating an electronic device that detects abnormal behavior of a visitor through monitoring inside a store, according to an embodiment.

Figure 3 is a flowchart showing a method of operating an electronic device that detects an abnormal image, according to an embodiment.

Figure 4 is a flowchart showing a method of operating an electronic device that verifies a candidate image, according to an embodiment.

FIG. 5 is a diagram illustrating a process for updating an abnormal behavior detection model, according to an embodiment.

FIG. 6 is a flowchart illustrating a method of operating an electronic device that generates and provides statistical data based on results of monitoring a store, according to an embodiment.

Figure 7 is a block diagram showing the configuration of an electronic device, according to an embodiment.

Hereinafter, various embodiments will be described in detail with reference to the drawings. The embodiments described below may be modified and implemented in various different forms. In order to more clearly explain the characteristics of the embodiments, detailed descriptions of matters widely known to those skilled in the art to which the following embodiments belong will be omitted.

Meanwhile, in this specification, when a configuration is said to be “connected” to another configuration, this includes not only the case of being “directly connected,” but also the case of being “connected with another configuration in between.” In addition, when a configuration “includes” another configuration, this means that other configurations may be further included rather than excluding other configurations, unless specifically stated to the contrary.

Additionally, terms including ordinal numbers such as 'first' or 'second' used in this specification may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

Throughout this specification, “electronic devices” include mobile phones, smart phones, laptop computers, PCs, desktops, home appliances, tablet PCs, e-book terminals, digital broadcasting terminals, and personal digital assistants (PDAs). ), PMP (Portable Multimedia Player), navigation, MP3 player, digital camera, etc., but is not limited to these.

FIG. 1 is a conceptual diagram illustrating an operation of monitoring the behavior of a visitor to a store and detecting abnormal behavior using an electronic device 10 within the store, according to an embodiment.

Referring to FIG. 1, the electronic device 10 can monitor the interior of the store. By monitoring the interior of the store, the electronic device 10 can detect abnormal behavior of visitors to the store. In this case, the electronic device 10 may be installed inside the store. The electronic device 10 has a built-in camera or is connected to a camera, so the electronic device 10 can capture the inside of the store in real time.

Meanwhile, in order to determine whether there is any abnormal behavior of a visitor in the monitoring video acquired by the electronic device 10, the monitoring video is transmitted from the electronic device 10 to the server 20, and when the analysis result is provided, the store stores in real time. There may be difficulties in monitoring. For example, if there are limitations in network connection, waiting time, bandwidth, etc. between the electronic device 10 and the server 20, the monitoring image can be viewed directly from the electronic device 10 without requesting monitoring image analysis from the server 20. By performing analysis, we can provide fast processing and customized services.

The electronic device 10 may request detailed analysis from the server 20 for images that are not easy to analyze with the electronic device 10. The server 20 may perform analysis on the requested image and transmit the analysis result to the electronic device 10 as feedback information. Additionally, the server 20 may request confirmation of the analysis result from the manager's electronic device 30. The manager's electronic device 30 may transmit feedback information about the analysis results of the server 20 to the server 20 .

The electronic device 10 may transmit monitoring result information to the manager's electronic device 30. The electronic device 10 may receive feedback information about the monitoring result information from the manager's electronic device 30.

FIG. 2 is a flowchart showing a method of operating an electronic device 10 that detects abnormal behavior of a visitor through monitoring inside a store, according to an embodiment.

Referring to FIG. 2 , in step S210, the electronic device 10 may acquire a monitoring image by photographing the interior of the store using a camera. The camera in the electronic device 10 can photograph the interior of the store in real time.

In step S220, the electronic device 10 may detect an abnormal image containing the visitor's abnormal behavior based on the monitoring image and the abnormal behavior detection model. The electronic device 10 may download an abnormal behavior detection model from the server 20. The abnormal behavior detection model may be a model that detects preset abnormal behavior in a video. In other words, the abnormal behavior detection model can receive a video as input data and output a video of a visitor corresponding to a preset abnormal behavior in the video as output data.

Here, the abnormal behavior detection model can be updated by learning the monitoring image acquired within the electronic device 10. Additionally, the electronic device 10 may periodically obtain an updated version of the abnormal behavior detection model from the server 20.

The electronic device 10 may detect a candidate image predicted to be abnormal behavior based on an abnormal behavior detection model and determine the candidate image as an abnormal image through verification of the candidate image. The process of detecting candidate images and determining abnormal images is explained in FIG. 3.

In step S230, the electronic device 10 may transmit the abnormal image to the store manager's electronic device 30. The electronic device 10 may transmit notification information indicating that an abnormal image has been detected to the manager's electronic device 30. The electronic device 10 may transmit information on the type of abnormal behavior corresponding to the abnormal image and information on measures to be taken regarding the abnormal behavior to the electronic device 30 of the manager.

In step S240, the electronic device 10 may output warning information within the store as an abnormal image is detected. For example, the electronic device 10 may output warning information corresponding to an abnormal image. Specifically, if the abnormal behavior included in the abnormal image is lying down in the store, the electronic device 10 may output warning information indicating that lying down in the store is prohibited.

For example, the electronic device 10 may output first warning information corresponding to abnormal behavior based on the abnormal image and the number of visitors visiting the store. Specifically, when the only visitors visiting the store are those who engage in abnormal behavior, the electronic device 10 may output warning information notifying the prohibition of abnormal behavior and warning of legal action. On the other hand, if the number of visitors to the store exceeds a preset number, the electronic device 10 may output warning information to prohibit abnormal behavior and provide caution.

Additionally, the electronic device 10 may transmit options for countermeasures against abnormal images to the manager's electronic device 30.

FIG. 3 is a flowchart showing a method of operating the electronic device 10 for detecting an abnormal image, according to an embodiment.

Referring to FIG. 3, in step S310, the electronic device 10 may acquire a monitoring image as input data for an abnormal behavior detection model. For example, the electronic device 10 may remove overlapping images from the monitoring image and then obtain the preprocessed monitoring image as input data for an abnormal behavior detection model.

In step S320, the electronic device 10 may detect a candidate image corresponding to a reference image representing a preset abnormal behavior based on the abnormal behavior detection model.

For example, preset abnormal behaviors include lying down in the store, destroying facilities in the store, climbing on facilities in the store, fighting in the store, stealing items in the store, and physically touching the store. It may include at least one of the following actions: exposing the user, and making a fuss in the store, and is not limited to the above examples.

For example, a reference image is an image showing preset abnormal behavior. Additionally, a video that meets the criteria for abnormal behavior that can be judged as abnormal behavior can be used as a reference video.

For example, the abnormal behavior detection model may be a learning model that learns a reference image showing abnormal behavior and detects whether the input image is an image showing abnormal behavior based on the learning results. The abnormal behavior detection model may be downloaded in advance from the server 20 and may be updated periodically.

In step S330, the electronic device 10 may verify the candidate image based on the matching level of the candidate image with respect to the reference image. For example, the matching level may indicate the degree to which a candidate image matches a reference image. Therefore, the higher the matching level, the more similar the candidate image may be to the reference image.

If the matching level is high, the candidate image may be determined to be similar to the reference image, so the electronic device 10 can immediately detect the candidate image as an abnormal image. On the other hand, if the matching level is not high, since verification is required as to whether the candidate image corresponds to an abnormal image, the electronic device 10 may verify the candidate image based on the server 20 or the manager's management device. there is. The process of verifying the candidate image based on the matching level of the candidate image is explained in FIG. 4.

In step S340, if the candidate image is verified, the electronic device 10 may detect the candidate image as an abnormal image. The electronic device 10 may determine the candidate image as an abnormal image and transmit the abnormal image to the manager's electronic device 30. Additionally, the electronic device 10 may transmit notification information notifying the visitor's abnormal behavior to the manager's electronic device 30.

FIG. 4 is a flowchart showing a method of operating the electronic device 10 for verifying a candidate image, according to an embodiment.

Referring to FIG. 4 , in step S410, the electronic device 10 may obtain the matching level of the candidate image with respect to the reference image. In step S420, the electronic device 10 may check whether the matching level of the candidate image is low level. For example, if the matching level of the candidate image is low level, the electronic device 10 may perform verification of the candidate image through the server 20 in step S421. On the other hand, if the matching level of the candidate image is high level, the electronic device 10 can verify the candidate image as an abnormal image in step S430 without going through the server 20. In other words, there is an effect that abnormal behavior can be monitored in real time on the electronic device 10 without transmitting the candidate image detected by the electronic device 10 to the server 20.

In step S421, the electronic device 10 may request verification of the candidate image from the server 20. The server 20 can check whether the candidate video contains abnormal behavior of the visitor. The server 20 may determine whether the candidate image is an abnormal image using a higher version model than the abnormal behavior detection model transmitted to the electronic device 10. In step S424, the server 20 may transmit the determined result to the electronic device 10 as feedback information. Additionally, in step S422, the server 20 may transmit the verification result of the candidate image to the manager's electronic device 30 and request confirmation of the verification result.

In step S423, the manager's electronic device 30 may display a screen for requesting feedback information about the candidate image. For example, the manager's electronic device 30 may display a message requesting confirmation whether the visitor's behavior in the candidate video is abnormal behavior. The manager's electronic device 30 may display an icon to select whether the behavior is abnormal. Additionally, the manager's electronic device 30 may display an icon that registers the behavior in the candidate video as abnormal behavior. For example, the manager's electronic device 30 may receive an input for selecting an icon indicating that the visitor's behavior in the candidate video is abnormal behavior. That is, the manager's electronic device 30 may receive feedback information about the candidate image and transmit the feedback information to the server 20.

In step S424, the server 20 may obtain feedback information about the candidate image directly or from the manager's electronic device 30. The server 20 may transmit feedback information to the electronic device 10.

In step S430, the electronic device 10 may verify the candidate image based on the feedback information. For example, when verification is completed that the candidate image is an abnormal image, the electronic device 10 may determine the candidate image as an abnormal image and perform the next process according to the detection of the abnormal image. On the other hand, when verification is completed that the candidate image is not an abnormal image, the electronic device 10 may determine that the candidate image is a normal image.

Meanwhile, the electronic device 10 may use the results of verifying the candidate image as learning data. The electronic device 10 may learn a candidate image and information indicating whether the candidate image is an abnormal image, and update the abnormal behavior detection model based on the learning result.

In step S510, the server 20 may transmit an abnormal behavior detection model to the electronic device 10. The electronic device 10 can receive an abnormal behavior detection model and monitor the store in real time using the abnormal behavior detection model.

In step S520, the manager's electronic device 30 may transmit reference information about abnormal behavior to the electronic device 10 within the store. The electronic device 10 may update the abnormal behavior detection model based on reference information input from the manager's electronic device 30.

In step S530, the electronic device 10 may obtain reference information about abnormal behavior. For example, the reference information may include a type of abnormal behavior and at least one behavior that can be determined to be similar to the abnormal behavior.

In step S540, the electronic device 10 may generate a reference image based on reference information. The electronic device 10 may generate a reference image that can be judged as abnormal behavior within the store based on the structure and reference information within the store. The reference image may be obtained from an image captured within the store or may be obtained from a virtually generated image.

In step S550, the electronic device 10 may update the abnormal behavior detection model based on the result of learning the reference image. The electronic device 10 may provide an abnormal behavior detection model suitable for the store by updating the abnormal behavior detection model based on reference information input by the manager.

FIG. 6 is a flowchart showing a method of operating the electronic device 10 that generates and provides statistical data based on the results of monitoring a store, according to an embodiment.

Referring to FIG. 6 , in step S610, the electronic device 10 may generate statistical data on preset abnormal behavior based on the results of monitoring the store.

For example, the electronic device 10 may generate statistical data about the type of at least one abnormal behavior that occurred during a preset period and the time when the at least one abnormal behavior occurred.

Additionally, the electronic device 10 may obtain an analysis result of analyzing status information in the store corresponding to a time when at least one abnormal behavior did not occur. The electronic device 10 compares the state information of the store at the time when the abnormal behavior occurred with the state information of the store at the time when the abnormal behavior did not occur, and provides information about the cause or environment in which the abnormal behavior occurred based on the comparison result. can be obtained.

In step S620, the electronic device 10 may transmit statistical data to the manager's electronic device 30.

FIG. 7 is a block diagram showing the configuration of the electronic device 10, according to one embodiment.

Referring to FIG. 7 , the electronic device 10 may include a camera 710, a communication device 720, a memory 730, and a processor 740. The electronic device 10 may include more components than those shown in FIG. 7 . Additionally, the electronic device 10 may include fewer components than those shown in FIG. 7 .

The camera 710 can obtain an image or video by photographing a predetermined object. The camera 710 can photograph a predetermined space in real time. For example, the camera 710 can obtain monitoring images by filming the inside of the store in real time.

The communication device 720 can communicate with an external device. The communication device 720 may include at least one of a Wi-Fi chip, a Bluetooth chip, an NFC chip, and a wireless communication chip. The processor 740 can communicate with various external devices using the communication device 720.

The memory 730 may store a plurality of applications or applications running on the electronic device 10, data for operating the electronic device 10, and commands.

The processor 740 controls the overall operation of the electronic device 10 and may include at least one processor such as a CPU.

The processor 740 may obtain a monitoring image by photographing the interior of the store through the camera 710.

The processor 740 may detect an abnormal image containing the visitor's abnormal behavior based on the monitoring image and the abnormal behavior detection model. The processor 740 may download the abnormal behavior detection model from the server 20. The abnormal behavior detection model may be a model that detects preset abnormal behavior in a video.

The processor 740 may detect a candidate image predicted to be abnormal behavior based on an abnormal behavior detection model and determine the candidate image as an abnormal image through verification of the candidate image.

Specifically, the processor 740 may acquire the monitoring image as input data for an abnormal behavior detection model. For example, the processor 740 may remove overlapping images from the monitoring image and then obtain the preprocessed monitoring image as input data for an abnormal behavior detection model.

The processor 740 may detect a candidate image corresponding to a reference image representing a preset abnormal behavior based on the abnormal behavior detection model.

For example, the abnormal behavior detection model may be a learning model that learns a reference image showing abnormal behavior and detects whether the input image is an image showing abnormal behavior based on the learning results.

For example, the abnormal behavior detection model can be analyzed through hardware processes (CPU, GPU, memory, etc.) by installing a dedicated application in the electronic device. In addition, to increase compatibility with various electronic devices, even without installing the dedicated application, you can run the web application in a web browser and detect abnormal behavior through the hardware processor.

The processor 740 may verify the candidate image based on the matching level of the candidate image with respect to the reference image. For example, the matching level may indicate the degree to which a candidate image matches a reference image. Therefore, the higher the matching level, the more similar the candidate image may be to the reference image.

Specifically, if the matching level of the candidate image is low level, the processor 740 may request verification of the candidate image from the server 20. The server 20 can check whether the candidate video contains abnormal behavior of the visitor. The server 20 may determine whether the candidate image is an abnormal image using a higher version model than the abnormal behavior detection model transmitted to the electronic device 10. The processor 740 may perform verification of the candidate image through the server 20. On the other hand, if the matching level of the candidate image is high level, the electronic device 10 can verify the candidate image as an abnormal image without going through the server 20. That is, abnormal behavior can be monitored in real time on the electronic device 10 without transmitting the candidate image detected by the electronic device 10 to the server 20.

The processor 740 may verify the candidate image based on feedback information obtained from the server 20 or the manager's electronic device 30. For example, when verification is completed that the candidate image is an abnormal image, the processor 740 may determine the candidate image to be an abnormal image. On the other hand, if verification is completed that the candidate image is not an abnormal image, the processor 740 may determine that the candidate image is a normal image.

Meanwhile, the processor 740 may use the results of verifying the candidate images as learning data. The processor 740 may learn a candidate image and information indicating whether the candidate image is an abnormal image, and update the abnormal behavior detection model based on the learning result.

Additionally, the processor 740 may update the abnormal behavior detection model based on reference information input from the manager's electronic device 30. Specifically, the processor 740 may obtain reference information about abnormal behavior. For example, the reference information may include a type of abnormal behavior and at least one behavior that can be determined to be similar to the abnormal behavior.

The processor 740 may generate a reference image based on reference information. The processor 740 may generate a reference image that can be judged as abnormal behavior within the store based on the structure and reference information within the store. The reference image may be obtained from an image captured within the store or may be obtained from a virtually generated image.

The processor 740 may update the abnormal behavior detection model based on the results of learning the reference image. The processor 740 may provide an abnormal behavior detection model suitable for the store by updating the abnormal behavior detection model based on reference information input by the manager.

When the candidate image is verified, the processor 740 may determine the candidate image as an abnormal image and transmit the abnormal image to the manager's electronic device 30. Additionally, the processor 740 may transmit notification information notifying the visitor's abnormal behavior to the manager's electronic device 30 through the communication device 720.

Additionally, the processor 740 may transmit information on the type of abnormal behavior corresponding to the abnormal image and information on measures to be taken regarding the abnormal behavior to the manager's electronic device 30 through the communication device 720 .

As an abnormal image is detected, the processor 740 may output warning information within the store. For example, the processor 740 may output warning information corresponding to an abnormal image. Specifically, if the abnormal behavior included in the abnormal image is lying down in the store, the electronic device 10 may output warning information indicating that lying down in the store is prohibited.

For example, the processor 740 may output first warning information corresponding to abnormal behavior based on the abnormal video and the number of visitors visiting the store. Additionally, the processor 740 may transmit options for countermeasures against abnormal images to the manager's electronic device 30 through the communication device 720.

The above-described method of operating the electronic device and server may be implemented in the form of a computer-readable storage medium that stores instructions or data executable by a computer or processor. Such computer-readable storage media may include ROM, RAM, hard disk, solid-state disk (SSD), and store instructions or software, related data, data files, and data structures.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the claims described below as well as equivalents to these claims.

The present invention can be used in the field of electronic devices that provide services using edge computing and methods of operating electronic devices.

Claims

In a method of operating an electronic device,

Obtaining a monitoring image by photographing the interior of the store using a camera;

Detecting an abnormal video containing abnormal behavior of a visitor based on the monitoring image and the abnormal behavior detection model;

transmitting the abnormal video to an electronic device of the store manager; and

An operating method comprising the step of outputting warning information within the store as the abnormal image is detected.
According to paragraph 1,

The step of detecting the abnormal image is,

Obtaining the monitoring image as input data for the abnormal behavior detection model;

Detecting a candidate image corresponding to a reference image representing a preset abnormal behavior based on the abnormal behavior detection model;

Verifying the candidate image based on a matching level of the candidate image with the reference image; and

When the candidate image is verified, an operating method comprising detecting the candidate image as the abnormal image.
According to paragraph 2,

The step of verifying the candidate image based on the matching level of the candidate image with the reference image includes:

If the matching level of the candidate image is a low level, receiving feedback information of the candidate image from a server or the manager's electronic device, and verifying the candidate image based on the received feedback information, an operating method comprising: .
According to clause 3,

An operating method further comprising updating the abnormal behavior detection model based on a result of verifying the candidate image.
According to paragraph 1,

As the abnormal image is detected, the step of outputting warning information within the store includes:

outputting first warning information corresponding to abnormal behavior based on the abnormal video and the number of visitors visiting the store; and

An operating method comprising transmitting an option for a response to the abnormal image to the electronic device of the manager.
According to paragraph 1,

Receiving reference information on preset abnormal behavior from the manager's electronic device;

generating a reference image for the preset abnormal behavior based on the reference information;

An operating method further comprising updating the abnormal behavior detection model based on a result of learning the reference image.
According to paragraph 1,

Generating statistical data on preset abnormal behavior based on the results of monitoring the store;

A method of operation further comprising transmitting the statistical data to the manager's electronic device.
In clause 7,

The step of generating the statistical data is,

generating the statistical data about the type of at least one abnormal behavior that occurred during a preset period and the time of occurrence of the at least one abnormal behavior;

An operating method comprising obtaining an analysis result of analyzing status information in the store corresponding to a time when the at least one abnormal behavior does not occur.
camera; communication device;

processor; and

and a memory storing instructions executable by the processor, wherein the processor executes the instructions,

Obtain monitoring video by filming the interior of the store through a camera,

Based on the monitoring video and the abnormal behavior detection model, detect an abnormal video containing the visitor's abnormal behavior,

Transmitting the abnormal video to the electronic device of the manager of the store,

An electronic device that outputs warning information within the store as the abnormal image is detected.
According to clause 9,

The processor, by executing the instructions,

Obtaining the monitoring image as input data for the abnormal behavior detection model,

Based on the abnormal behavior detection model, detect a candidate image corresponding to a reference image representing a preset abnormal behavior,

Verify the candidate image based on the matching level of the candidate image to the reference image,

When the candidate image is verified, the electronic device detects the candidate image as the abnormal image.