KR20230152961A - Electronic device to prevent theft of goods stored in cashierless store and operation method thereof - Google Patents

Abstract

An electronic device that prevents the theft of products stored in unmanned stores is disclosed. According to an embodiment of the present invention, an electronic device for preventing the theft of products displayed in an unmanned store includes a camera unit, a storage unit that stores images of a buyer purchasing a product displayed in the unmanned store, and a storage unit that stores images of a buyer purchasing a product displayed in the unmanned store. When entering a store, the camera unit is controlled to capture the buyer's face to create a facial image, perform deep learning on the buyer's appearance based on the stored image, and perform deep learning based on the deep learning results. The buyer is identified and matched with the face image, and based on the image captured by the camera unit, it is determined whether the product is picked by the buyer, and the product's ID (identification) and When matching the buyer, and a predefined time has elapsed from the time the buyer leaves the payment area of the unmanned store without paying for the product, the facial image of the buyer, the ID of the product, It may include a processor that transmits information about the time the buyer left the payment area of the unmanned store to the store owner's user terminal.

Description

Electronic device for preventing theft of products stored in unmanned stores and method of operating the same {ELECTRONIC DEVICE TO PREVENT THEFT OF GOODS STORED IN CASHIERLESS STORE AND OPERATION METHOD THEREOF}

The present invention relates to an electronic device that prevents theft of products stored in an unmanned store and a method of operating the same. More specifically, an electronic device that prevents theft in an unmanned store in advance and detects theft when it occurs, and a method of operating the same. It's about how it works.

Recently, with the development of technologies such as big data, artificial intelligence, and the Internet of Things, overall improvements are being made to services provided in real life. In particular, technology is developing to provide convenience in real life and reduce business costs by deep learning human behavior based on big data.

These technological developments are bringing innovation to the offline shopping sector, reflecting the desire to improve the traditional purchase and payment process of lining up at the checkout counter and paying with a card or cash from the wallet.

Amazon, an American e-commerce company, is expanding its 'Amazon Go' service, which combines an unmanned payment system. Amazon Go has introduced a system in which payment is completed by linking with a smartphone application by simply scanning the barcode corresponding to one's personal information, entering the store, picking up an item, and leaving the store.

As an example of the technology behind the present invention, Republic of Korea Patent Publication No. 10-2021-0075833 (2021.06.23.) is a user recognition module that acquires biometric information of a user who wants to enter an unmanned store, and Launch of a user authentication system in an unmanned store that includes a card recognition module that extracts biometric information stored in the card and a user authentication module that authenticates the user by comparing the biometric information obtained from the user recognition module with the biometric information extracted from the card recognition module. do.

As mentioned above, unmanned stores have recently become active, but there are some issues that unmanned store technology has not been able to solve. For example, in relation to unmanned stores, there are still many areas that need to be supplemented in terms of adult authentication technology for products that require adult authentication, such as cigarettes and alcohol, and theft prevention technology.

The present invention was devised to solve the above-mentioned problems. The present invention is a method of preventing the theft of products by tracking the behavior of buyers in unmanned stores and quickly responding in the event of theft through profiling of non-payment events. The purpose is to provide.

An electronic device for preventing the theft of products displayed in an unmanned store according to various embodiments of the present invention includes a camera unit, a storage unit that stores images of a buyer purchasing a product displayed in the unmanned store, and a storage unit that stores images of a buyer purchasing a product displayed in the unmanned store. When entering a store, the camera unit is controlled to capture the buyer's face to create a facial image, perform deep learning on the buyer's appearance based on the stored image, and perform deep learning based on the deep learning results. The buyer is identified and matched with the face image, and based on the image captured by the camera unit, it is determined whether the product is picked by the buyer, and the product's ID (identification) and When matching the buyer, and a predefined time has elapsed from the time the buyer leaves the payment area of the unmanned store without paying for the product, the facial image of the buyer, the ID of the product, It may include a processor that transmits information about the time the buyer left the payment area of the unmanned store to the store owner's user terminal.

In addition, a method of operating an electronic device for preventing theft of products stored in an unmanned store according to various embodiments of the present invention includes the process of storing a video of a buyer purchasing a product stored in the unmanned store, When entering an unmanned store, a process of generating a facial image by photographing the face of the buyer, a process of performing deep learning on the appearance of the buyer based on the stored image, and the process of performing deep learning on the appearance of the buyer based on the performed deep learning results. A process of specifying a buyer and matching the face image, a process of determining whether the product is picked by the buyer based on the image captured by the camera unit, a process of matching the ID of the product and the buyer, and If a predefined time has elapsed from the time the buyer leaves the payment area of the unmanned store without paying for the product, the facial image of the buyer, the ID of the product, and the buyer are returned to the unmanned store. It may include a process of transmitting information about the time of departure from the payment area to the store owner's user terminal.

According to various embodiments of the present invention, the present invention prevents the theft of products by tracking the behavior of buyers within an unmanned store, and provides a method of quickly responding in the event of theft through profiling of non-payment events. .

1 is a diagram of an unmanned store theft prevention system according to an embodiment of the present invention.
Figure 2 is a block diagram of an unmanned store server according to an embodiment of the present invention.
Figure 3 is an image taken of a purchaser according to an embodiment of the present invention.
Figure 4 is an image of a buyer's hand according to an embodiment of the present invention.
Figure 5 is a detailed configuration diagram of an electronic device according to an embodiment of the present invention.
Figure 6 is a flowchart of a method of operating an electronic device that prevents the theft of products stored in an unmanned store according to an embodiment of the present invention.

Hereinafter, the operating principle of a preferred embodiment of the present invention will be described in detail with reference to the attached drawings. Additionally, when describing embodiments of the invention, if it is determined that a detailed description of a related known function or configuration may obscure the gist of the present disclosure, the detailed description will be omitted. The terms used below are defined in consideration of the functions in the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, the definitions of terms used should be interpreted based on the content and corresponding functions throughout this specification.

1 is a diagram of an unmanned store theft prevention system according to an embodiment of the present invention.

The unmanned store theft prevention system 10 may include a store owner user terminal 10, an unmanned store server 200, and an external server 300. Here, the store owner user terminal 10, the unmanned store server 200, and the external server 300 can be implemented with various devices such as PCs, tablet PCs, smartphones, and servers.

The store owner user terminal 100 is a user terminal used by the store owner of an unmanned store. In general, the store owner user terminal 100 can perform unmanned store management, inventory management, cost management, and store situation monitoring outside the unmanned store.

As an example, the store owner user terminal 100 can receive the real-time situation of the unmanned store through communication with the in-store server 200, which will be described later. The store owner user terminal 100 can be notified of relevant information when theft (or theft) occurs in an unmanned store. The store owner user terminal 100 may process a theft case (or non-payment case) by transmitting a control signal to the unmanned store server 200 based on the store owner's input.

The unmanned store server 200 can generally control the unmanned store theft prevention system 10. Here, the unmanned store server 200 may be defined by various terms such as electronic device, electronic device, server, and user terminal.

The unmanned store server 200 can photograph the buyer from the moment the buyer enters the unmanned store to the moment the buyer leaves. The unmanned store server 200 can store a video of a buyer purchasing a product provided in an unmanned store or delete the stored video.

For example, when a buyer enters an unmanned store, the unmanned store server 200 may capture the buyer's face and generate a facial image. In this case, the unmanned store server 200 may perform deep learning on the buyer's appearance based on the stored image. Additionally, the unmanned store server 200 can specify a buyer based on deep learning results and match it with the facial image.

Deep learning technology, a type of machine learning, can involve the process of learning down to a deep level in multiple stages based on data. Deep learning can represent a set of machine learning algorithms that extract key data from data at increasing levels.

The deep learning structure may include an artificial neural network (ANN). For example, the deep learning structure consists of deep neural networks (DNN) such as convolutional neural network (CNN), recurrent neural network (RNN), and deep belief network (DBN). It can be.

The deep learning structure according to this embodiment can use various known structures. For example, the deep learning structure according to the present invention may include CNN, RNN, DBN, etc. RNN is widely used in natural language processing, etc., and is an effective structure for processing time-series data that changes over time. It can build an artificial neural network structure by stacking layers at every moment. DBN may include a deep learning structure composed of multiple layers of restricted boltzman machine (RBM), a deep learning technique. By repeating RBM learning, when a certain number of layers are reached, a DBN with that number of layers can be constructed.

CNN can include a model that simulates human brain function, which is created based on the assumption that when a person recognizes an object, the basic features of the object are extracted, then complex calculations are performed in the brain, and the object is recognized based on the results. there is.

Meanwhile, learning of an artificial neural network can be accomplished by adjusting the weight of the connection lines between nodes (adjusting the bias value if necessary) to produce the desired output for a given input. Additionally, artificial neural networks can continuously update weight values through learning. Additionally, methods such as back propagation can be used to learn artificial neural networks.

In addition, the unmanned store server 200 determines that the buyer has picked the product when the weight of the shelf on which the product is displayed changes by the weight of the product and the distance between the buyer and the shelf is within a predefined distance. And, the ID (identification) of the product and the buyer can be matched.

Additionally, the unmanned store server 200 can determine whether a product is being picked by a buyer based on the image captured by the camera unit 210.

If a predefined time has elapsed from the time the buyer leaves the payment area of the unmanned store without paying for the product, the unmanned store server 200 may determine the buyer as a person who has not paid for the picked product. In this case, the unmanned store server 200 may transmit information about the face image of the buyer, the ID of the product, and the time the buyer left the payment area of the unmanned store to the store owner's user terminal 100. When the unmanned store server 200 receives a request for notification of a non-payment event from the store owner's user terminal, the unmanned store server 200 may display a face image of the buyer to notify the fact that a non-payment event has occurred in the unmanned store.

The external server 300 may be, for example, a National Police Agency server. If it is determined that theft has occurred, the unmanned store server 200 may create a profile of the person responsible for the non-payment incident and transmit the created profile to the external server 300. In this case, the unmanned store server 200 can request police action by transmitting the fact that non-payment occurred, location information of the unmanned store where non-payment occurred, etc. along with profile information to the external server 300.

According to an embodiment of the present invention described above, it is possible to prevent product theft by tracking the buyer's behavior within an unmanned store, and to provide a method of quickly responding in case of theft through buyer profiling.

In general, the devices, systems, and methods disclosed herein may be used in, for example, general-purpose computing systems, server-client computing systems, consumer-merchant computing systems, mainframe computing systems, cloud computing infrastructure, and telephony computing systems. , can include a number of different devices and computer systems, including laptop computers, desktop computers, smartphones, cellular phones, personal digital assistants (PDAs), tablet computers, and other mobile devices, many of which may be implemented within different devices and computer systems. Devices and computing systems may include one or more databases and other storage devices, servers, and additional components such as processors, modems, terminals and displays, computer-readable media, algorithms, modules and applications, and other computer-related components. It may be possible. Devices and computer systems and/or computing infrastructure are configured, programmed, and adapted to perform the functions and processes of the systems and methods as disclosed herein.

Figure 2 is a block diagram of an unmanned store server according to an embodiment of the present invention.

Referring to FIG. 2, the unmanned store server 200 may include a camera unit 210, a storage unit 220, and a processor 230.

The camera unit 210 may include at least one camera. Here, the camera unit 210 may be placed at the entrance of the unmanned store, the ceiling of the unmanned store, or a portion of a shelf.

As an example, the camera unit 210 may be placed on a portion of a shelf and photograph at least one product stored (displayed, placed) on the shelf or stored on another shelf. In addition, the camera unit 210 is disposed on a part of the shelf and detects objects that enter to pick products stored on the shelf, for example, a part of the buyer's body, or the upper, lower, left and right sides of the shelf. An object entering another shelf placed on the side, for example, a part of the buyer's body, can be photographed. Part of the buyer's body may be the buyer's hands.

As an example, the camera unit 210 can capture space within a store. For example, the space within a store may be divided into a plurality of spaces, and a plurality of cameras included in the camera unit 210 may be arranged to cover each of the plurality of spaces described above. In this case, the plurality of cameras can photograph each of the plurality of spaces described above in real time. In this case, a plurality of cameras can track the buyer and photograph the buyer's appearance.

The storage unit 220 may include volatile or non-volatile recording media.

The storage unit 220 is connected to one or more processors and can store codes that, when executed by the processor 220, cause the processor 220 to control the unmanned store server 200.

Here, the storage unit 220 may include magnetic storage media or flash storage media, but the scope of the present invention is not limited thereto. This storage unit 220 may include internal memory and/or external memory, volatile memory such as DRAM, SRAM, or SDRAM, one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, and flash ROM. , non-volatile memory such as NAND flash memory, or NOR flash memory, SSD. It may include a flash drive such as a compact flash (CF) card, SD card, Micro-SD card, Mini-SD card, or memory stick, or a storage device such as an HDD.

The storage unit 220 can store various data or information. As an example, the storage unit 220 may store real-time images captured by the camera unit 210. Additionally, the storage unit 220 may store the user's profile created by the processor 230. Additionally, the results of judgments, decisions, or calculations performed by the processor 230 may be stored.

The processor 230 can generally control the unmanned store server 200. In particular, the processor 230 can prevent the theft of products stored in unmanned stores and take action when theft occurs.

Specifically, the processor 230 may control the camera unit 210 to capture a picture of a buyer purchasing a product displayed in an unmanned store, and control the storage unit 220 to store the captured image.

Additionally, the processor 230 may control the camera unit 210 to capture the buyer's face and generate a facial image when the buyer enters the unmanned store. The processor 230 may store the buyer's face image as the buyer's profile information in the storage unit 210.

Additionally, the processor 230 may specify a buyer who purchased the product based on an image pre-stored in the storage unit 220 or an image captured by the camera unit 210.

As an example, the processor 230 may specify a buyer through deep learning using a pose estimation algorithm. For example, referring to FIG. 3, the processor 230 may perform deep learning on the buyer's image to identify a buyer who performs various actions while moving as the same buyer.

Additionally, the processor 230 may extract the movement (movement) and direction of movement of the buyer's body through a skeleton extraction algorithm. Additionally, the processor 230 may extract the movement of the human body using computer vision technology, for example, optical flow.

Here, the processor 230 can identify the buyer based on the deep learning results and match the pre-stored facial image. Here, the pre-stored face image may be an image that includes the buyer's face and has at least part of the background screen deleted.

Meanwhile, the processor 230 may determine whether the buyer purchased the product.

As an example, the above-described shelf may include a weight sensor unit. The weight sensor unit can sense changes in the weight of the product or the weight of the shelf. The weight sensor unit may sense a change in weight when a product is added to or removed from the shelf, and transmit the sensing result to the processor 230. Here, if the weight of the shelf on which the product is displayed changes by the weight of the product and the distance between the buyer and the shelf is within a predefined distance, the processor 230 determines that the buyer has picked the product. You can.

As an example, referring to FIG. 4, it is assumed that the camera unit 210 photographs the top of a product stored on the shelf. The processor 230 can determine various directions (A, B, C) of the hand entering the shelf.

Here, the processor 230 may determine whether the product is picked by the buyer based on the determined hand direction. Here, the direction of the hand may be determined by a vector connecting the start point of movement of the purchaser's hand and the stop point of movement of the purchaser's hand.

Specifically, the processor 230 determines the hand movement start point and the hand movement stop point, and creates a first vector connecting the hand movement start point and the hand movement stop point. can be created. Additionally, the processor 230 generates a second vector between the location of the picked product and the location of at least one buyer present in the unmanned store. The processor 230 may determine whether the at least one buyer has picked the product depending on whether the direction of the first vector and the direction of the second vector match. For example, if the direction of the first vector and the direction of the second vector are within (or coincide with) a predefined angle, the processor 230 determines that the product has been picked by the at least one buyer. can do.

Here, the processor 230 can specify the buyer who picked the product by further considering the timing of the weight change of the shelf, in addition to the result of comparing the direction of the first vector and the direction of the second vector. there is.

In one embodiment of the present invention described above, if the buyer who picked the product is specified, the processor 230 may match the ID (identification) of the product with the buyer.

Additionally, the processor 230 may determine whether the purchaser has not made payment.

For example, if a predefined time has elapsed from the time the buyer leaves the payment area of the unmanned store without paying for the product, the processor 230 may determine the buyer as someone who has not paid for the picked product. Here, the payment area can be appropriately determined according to the circumstances of the unmanned store, such as a specific range around the unmanned payment kiosk or the entrance line of the unmanned store.

In this case, the processor 230 may transmit information about the face image of the buyer, the ID of the product, and the time the buyer left the payment area of the unmanned store to the store owner's user terminal 100.

When the processor 230 receives a request for notification of the occurrence of a non-payment event from the store owner's user terminal 100, the processor 230 may display a face image of the buyer and output a voice guidance related to the occurrence of the non-payment event.

For example, when a buyer picks multiple products, the processor 230 may match each ID of the multiple products with the buyer. If the number of products paid by the buyer is smaller than the number of products, the processor 230 may identify the missing product by comparing the IDs of the plurality of products with the IDs of the products paid by the buyer. Additionally, the processor 230 may generate and display a graphic user interface (GUI) requesting additional payment for the missing product, and generate and output voice guidance requesting additional payment for the missing product.

As an example, the processor 230 may lock the door when the buyer leaves the payment area of the unmanned store while the product has not been paid. Additionally, a notification indicating that the door has been locked can be transmitted to the store owner's user terminal 100.

Meanwhile, the processor 230 may create a profile of a buyer who has not paid or a profile of a non-payment event.

Specifically, the processor 230 may control the camera unit to photograph the buyer's hand approaching the shelf from at least one direction, thereby generating at least one hand image. In this case, the processor 230 may control the storage unit 220 to store at least one hand image.

Here, the processor 230 may acquire at least one palm image among at least one hand image based on the results of deep learning performed on the human hand image.

The processor 230 may distinguish or extract fingerprints from the acquired palm image. To this end, the processor 230 may perform machine learning, such as deep learning, on fingerprint detection. In this case, the above-described storage unit 220 can store data used for machine learning, result data, etc.

As an example, the processor 230 may extract the boundary of the fingerprint area. The processor 220 may extract the outline of the fingerprint from the two-dimensional array based on the level value of the fingerprint image. In this case, the processor 230 may convert the fingerprint area into a grayscale image. Additionally, the processor 230 can extract the boundary of the fingerprint area by calculating the median value of the pixels within the fingerprint area and specifying the median value of the array.

Specifically, the processor 230 can calculate the intensity and direction of the outline by obtaining a differential value from a two-dimensional array based on the level value of the fingerprint area. Here, the outline (edge) dictionary refers to a line representing the outer edge of an object, and at the level of image processing, it can refer to line elements that characterize the image. In other words, recognizing the boundary line and finding the pixel corresponding to the outline is called edge detection.

Contours are places in an image where pixel values suddenly change. In other words, the outline can mean the boundary between two areas with different brightness, and can mean the part where the brightness of the pixel changes more than the threshold. Here, the threshold may mean an arbitrary reference value for determining the presence or absence of a boundary.

The processor 230 may determine the algorithm for extracting the outline based on the size of a gradient vector obtained by differentiating the image. That is, the processor 230 can obtain the slope based on the calculation of the partial differential operator. Since the contour is a part where the light/darkness value changes rapidly, a differential operation that takes the change in the function can be used to extract the contour.

Differentiation includes first-order differentiation (gradient) and second-order differentiation (laplacian). The presence or absence of a contour in an image is checked using the size of the graph gradient in the first-order differential value, and the sign of the graph gradient is used to identify the outline pixel in the second-order differential value. You can check the location of the bright and dark parts of .

Algorithms for extracting contours may include sobel edge detection, canny edge detection, etc. Additionally, the processor 230 may convert the fingerprint area into a grayscale image to find 'light values' and 'dark values' in order to find the exact boundary of the fingerprint area.

The processor 230 can extract the boundary of the fingerprint area by calculating the median value of the pixels within the fingerprint area and specifying the median value of the array. That is, the processor 230 can binarize the image converted to grayscale, extract the intermediate value, and extract the outline.

For example, a mask can be used as a way to apply differentiation to an image. The processor 230 can detect a contour line by applying a mask to 3x3 pixels of the image and determining the center pixel. Here, the general size of the mask is 3x3, but masks of 5x5 and 7x7 sizes are also applicable, and as the mask becomes larger, the edges become thicker and can appear clearly.

Additionally, the processor 230 may perform filtering by taking the median value among pixels within the mask area as a representative value. By using the intermediate value and filtering it by setting an adaptive weight according to the difference from the surrounding value, impulsive noise such as spot noise can be reduced, image smoothing (edge or border protection) is possible, and sharpness differences are large. Noise removal can be performed without it.

In one embodiment of the present invention, processor 230 may apply a smoothing filter, such as a median filter, to remove noise before final edge detection of the fingerprint area and a spatial filter for edge detection. there is. That is, the processor 230 may perform a recursive call on the candidate homogeneous region determined based on the median value of the array, and calculate the average value of the change in pixel value of the fingerprint image as a result of the recursive call.

Finally, the processor 230 may obtain the buyer's fingerprint image and store it in the storage unit 220.

Meanwhile, the processor 230 deletes at least one stored palm image when the buyer normally pays for the picked product, and acquires a fingerprint image from at least one stored palm image when the buyer does not pay for the picked product. Thus, the storage unit 212 can be controlled to match and store the purchaser.

In addition, the processor 230 determines at least one palm image as one of a left-hand palm image and a right-hand palm image, based on the deep learning results performed on the human hand image, and determines whether the fingerprint image corresponds to the left or right hand. You can create and save the indicating indicator.

In addition, when the buyer does not pay for the picked product, the processor 230 provides a fingerprint image, an indicator indicating whether the fingerprint image corresponds to the left or right hand, the facial image of the buyer, the ID of the product, the amount of unpaid damage, and the buyer. A non-payer profile or a non-payment case profile containing at least one of information about the time of leaving the payment area of the unmanned store may be created and transmitted to the National Police Agency server.

According to the various embodiments of the present invention described above, the theft of products can be prevented by tracking the buyer's behavior within an unmanned store, and rapid response in the event of theft can be achieved through profiling of the buyer or non-payment events.

Figure 5 is a detailed configuration diagram of an electronic device according to an embodiment of the present invention.

Referring to FIG. 5 , the electronic device 500 includes a communication unit 510, a storage unit 520, and a processor 530.

The communication unit 510 performs communication. The communication unit 510 can communicate with external electronic devices through various communication methods such as BT (BlueTooth), WI-FI (Wireless Fidelity), ZigBee, IR (Infrared), NFC (Near Field Communication), etc.

The storage unit 520 may store O/S (Operating System) software modules for driving the electronic device 500, data for configuring various UI screens provided in the display area, etc.

Additionally, the storage unit 520 is capable of reading and writing.

The processor 530 generally controls the operation of the electronic device 500 using various programs stored in the storage unit 520.

Specifically, the processor 530 includes RAM 531, ROM 532, main CPU 533, graphics processing unit 534, first to n interfaces (535-1 to 535-n), and bus 536. Includes.

Here, RAM 531, ROM 532, main CPU 533, graphics processing unit 534, first to n interfaces (535-1 to 535-n), etc. may be connected to each other through bus 536. .

The first to n interfaces 535-1 to 535-n are connected to the various components described above. One of the interfaces may be a network interface connected to an external device through a network.

The ROM 532 stores a set of instructions for booting the system. When the turn-on command is input and power is supplied, the main CPU 533 copies the O/S stored in the storage unit 520 to the RAM 531 according to the command stored in the ROM 532 and executes the O/S. Boot the system.

When booting is complete, the main CPU 533 copies various stored application programs to the RAM 531 and executes the application programs copied to the RAM 531 to perform various operations.

The main CPU 533 accesses the storage unit 530 and performs booting using the O/S stored in the storage unit 530. And, the main CPU 533 performs various operations using various programs, content, data, etc. stored in the storage unit 530.

The graphics processing unit 534 uses a calculation unit and a rendering unit to create a screen including various objects such as icons, images, and text.

Figure 6 is a flowchart of a method of operating an electronic device that prevents the theft of products stored in an unmanned store according to an embodiment of the present invention.

Referring to FIG. 6, the method of operating an electronic device that prevents the theft of products displayed in an unmanned store includes a process of storing a video of a buyer purchasing a product displayed in an unmanned store (S610), and a process of storing a video of a buyer purchasing a product displayed in an unmanned store. When entering, a process of generating a facial image by photographing the buyer's face (S620), a process of performing deep learning on the buyer's appearance based on the stored image (S630), and identifying the buyer based on the deep learning results. A process of specifically matching a face image (S640), a process of determining whether a product is picked by a buyer based on the image captured by the camera unit (S650), and matching the ID of the product and the buyer. Process (S660), if a predefined time has elapsed from the time the buyer leaves the payment area of the unmanned store without paying for the product, the facial image of the buyer, the ID of the product, and the buyer's payment area of the unmanned store The process of transmitting information about the time of departure from the store owner to the store owner's user terminal (S670) and upon receiving a request for notification of a non-payment event from the store owner's user terminal, a face image of the buyer is displayed and information regarding the non-payment event occurs It may include a process of outputting voice (S680).

The process of determining whether the above-mentioned product is picked is based on the image captured by the camera unit, the process of determining the direction of the buyer's hand entering the shelf where the product is displayed, and the determined direction of the hand. , may include a process of determining whether the product is picked by the buyer.

Here, the direction of the hand described above can be determined by a vector connecting the start point of movement of the purchaser's hand and the stop point of movement of the purchaser's hand.

In addition, the process of determining whether the above-mentioned product is picked is when the weight of the shelf on which the product is displayed is changed by the weight of the product, and the distance between the buyer and the shelf is within a predefined distance, the buyer picks the product. ) may further include a process of determining that

The method of operating the above-described electronic device includes a process of matching each ID of a plurality of products with the buyer when the buyer picks multiple products, and when the number of products paid by the buyer is less than the number of the plurality of products. By comparing the ID of the product and the ID of the product paid by the buyer, the missing product is identified, a GUI is created and displayed to request additional payment for the missing product, and a voice guidance is generated to request additional payment for the missing product. This may further include an output process.

In addition, the method of operating the above-mentioned electronic device is to lock the door when the buyer leaves the payment area of the unmanned store while the product has not been paid, and to transmit a notification indicating the locking of the door to the store owner's user terminal. Additional processes may be included.

In addition, the method of operating the above-described electronic device further includes the steps of controlling the camera unit to capture the hand of a buyer approaching the shelf from at least one direction, generating at least one hand image, and storing the at least one hand image. can do.

Additionally, the method of operating the electronic device described above may further include a process of acquiring at least one palm image among at least one hand image based on the results of deep learning performed on the human hand image.

In addition, the method of operating the above-described electronic device includes a process of deleting at least one stored palm image when the buyer normally pays for the picked product, and a process of deleting at least one stored palm image from the stored palm image when the buyer does not pay for the picked product. It may further include the process of acquiring a fingerprint image, matching it with the purchaser, and storing it.

In addition, the method of operating the above-described electronic device includes a process of determining at least one palm image as one of a left-hand palm image and a right-hand palm image based on deep learning results performed on a human hand image, and a process of determining whether the fingerprint image is left or right-handed. It may further include the process of generating and storing an indicator indicating whether it corresponds to .

In addition, the method of operating the above-mentioned electronic device is, in the case where the buyer has not paid for the picked product, a fingerprint image, an indicator indicating whether the fingerprint image corresponds to the left or right hand, a facial image of the buyer, the ID of the product, and the amount of unpaid damage. And it may further include the process of creating a non-payment event profile including information about the time the buyer left the payment area of the unmanned store and transmitting it to an external server.

Meanwhile, the method of operating an electronic device for preventing theft of products stored in an unmanned store according to various embodiments of the present invention described above is implemented as a computer-executable program code and can be used in various non-transitory computer readable media. medium) and can be provided to each server or device to be executed by a processor.

For example, the process of storing a video of a buyer purchasing a product displayed in an unmanned store, the process of generating a facial image by photographing the buyer's face when the buyer enters an unmanned store, and the process of generating a facial image of the buyer based on the stored video Process of performing deep learning on the appearance of the product, specifying the buyer based on the deep learning results and matching the face image, process of matching the ID of the product and the buyer (S650), state in which the buyer has not paid for the product If a predefined time has elapsed from the time the buyer left the payment area of the unmanned store, the face image of the buyer, the ID of the product, and information about the time the buyer left the payment area of the unmanned store are sent to the store owner's user terminal. When a request for notification of the occurrence of a non-payment event is received from the transmission process (S660) and the store owner's user terminal, a non-temporary program is stored that performs the process of displaying the face image of the buyer and outputting a voice guidance related to the occurrence of the non-payment event. A readable medium (non-transitory computer readable medium) may be provided.

A non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as registers, caches, and memories. Specifically, the various applications or programs described above may be stored and provided on non-transitory readable media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, etc.

Although embodiments of the present invention have been shown and described above, those skilled in the art will recognize that various changes in form and details may be made without departing from the spirit and scope of the present embodiments as defined by the appended claims and their equivalents. You will understand that you can.

Unmanned store theft prevention system: 10
Store owner user terminal: 100
Unmanned store servers: 200
Electronics: 500
External servers: 300
Camera unit: 210
Storage: 220, 520
Processor: 230, 530
Department of Communications: 510

Claims (22)

In an electronic device that prevents the theft of products stored in an unmanned store,
camera unit;
a storage unit that stores images captured of buyers purchasing products displayed in the unmanned store; and
When a buyer enters the unmanned store, control the camera unit to capture the buyer's face and create a facial image,
Perform deep learning on the buyer's appearance based on the stored image,
Based on the deep learning results, the buyer is identified and matched with the face image,
Based on the image captured by the camera unit, determine whether the product is picked by the buyer,
Match the ID (identification) of the product and the buyer,
If a predefined time has elapsed from the time the buyer leaves the payment area of the unmanned store without paying for the product, the facial image of the buyer, the ID of the product, and the buyer are returned to the unmanned store. An electronic device that includes a processor, which transmits information about the time of departure from the payment area to the store owner's user terminal.
According to paragraph 1,
The processor,
Based on the image captured by the camera unit, determine the direction of the buyer's hand entering the shelf where the product is displayed,
An electronic device that determines whether the product is picked by the buyer, based on the determined hand direction.
According to paragraph 2,
The direction of the hand is,
An electronic device determined by a vector connecting a start point of movement of the hand of the purchaser and a stop point of movement of the hand of the purchaser.
According to paragraph 2,
The processor,
An electronic device that determines that the buyer has picked the product when the weight of the shelf on which the product is displayed changes by the weight of the product and the distance between the buyer and the shelf is within a predefined distance. .
According to paragraph 1,
The processor,
When the buyer picks multiple products, each ID of the plurality of products is matched with the buyer,
If the number of products paid by the buyer is less than the number of the plurality of products, the IDs of the plurality of products are compared with the IDs of the products paid by the buyer to specify the missing product, and the missing product is added. An electronic device that generates and displays a GUI (graphic user interface) requesting payment, and generates and outputs voice guidance requesting additional payment for the missing product.
According to paragraph 1,
The processor,
When the buyer leaves the payment area of the unmanned store while the product is not paid, the electronic door is locked and a notification indicating the locking of the door is transmitted to the store owner's user terminal. Device.
According to paragraph 2,
The processor,
An electronic device that controls the camera unit to capture the buyer's hand approaching the shelf from at least one direction to generate at least one hand image, and controls the storage unit to store the at least one hand image.
In clause 7,
The processor,
An electronic device that acquires at least one palm image among the at least one hand image based on a deep learning result performed on a human hand image.
According to clause 8,
The processor,
If the buyer normally pays for the picked product, the at least one stored palm image is deleted,
An electronic device that controls the storage unit to obtain a fingerprint image from the stored at least one palm image, match it with the purchaser, and store the fingerprint image when the purchaser has not paid for the picked product.
According to clause 9,
The processor,
Based on deep learning results performed on the human hand image, determining the at least one palm image as one of a left palm image and a right palm image,
An electronic device that generates and stores an indicator indicating whether the fingerprint image corresponds to a left or right hand.
According to clause 10,
The processor,
If the buyer has not paid for the picked product, the fingerprint image, an indicator indicating whether the fingerprint image corresponds to the left or right hand, the face image of the buyer, the ID of the product, the amount of unpaid damage, and the buyer An electronic device that generates a non-payment event profile including information on the time of leaving the payment area of the unmanned store and transmits it to an external server.
In a method of operating an electronic device that prevents the theft of products stored in an unmanned store,
A process of storing a video of a buyer purchasing a product provided in the unmanned store;
When a buyer enters the unmanned store, a process of generating a facial image by photographing the buyer's face;
A process of performing deep learning on the buyer's appearance based on the stored image;
A process of specifying the buyer and matching the face image based on the deep learning results performed;
A process of determining whether the product is picked by the buyer based on the image captured by the camera unit;
A process of matching the ID (identification) of the product and the buyer; and
If a predefined time has elapsed from the time the buyer leaves the payment area of the unmanned store without paying for the product, the facial image of the buyer, the ID of the product, and the buyer are returned to the unmanned store. A method of operating an electronic device, including a process of transmitting information about the time of departure from the payment area to the store owner's user terminal.
According to clause 12,
The process of determining whether the product is picked is:
Based on the image captured by the camera unit, determining the direction of the buyer's hand entering the shelf where the product is displayed; and
A method of operating an electronic device comprising: determining whether the product is picked by the buyer, based on the determined hand direction.
According to clause 13,
The direction of the hand is,
A method of operating an electronic device, determined by a vector connecting a start point of movement of the hand of the purchaser and a stop point of movement of the hand of the purchaser.
According to clause 13,
The process of determining whether the product is picked is:
If the weight of the shelf on which the product is displayed changes by the weight of the product and the distance between the buyer and the shelf is within a predefined distance, determining that the buyer has picked the product; Further comprising: a method of operating an electronic device.
According to clause 12,
When the buyer picks a plurality of products, matching each ID of the plurality of products with the buyer; and
If the number of products paid by the buyer is less than the number of the plurality of products, the IDs of the plurality of products are compared with the IDs of the products paid by the buyer to specify the missing product, and the missing product is added. A method of operating an electronic device further comprising: generating and displaying a graphic user interface (GUI) requesting payment, and generating and outputting voice guidance requesting additional payment for the missing product.
According to clause 12,
If the buyer leaves the payment area of the unmanned store while the product is not paid, locking the door and transmitting a notification indicating that the door has been locked to the store owner's user terminal; A method of operating an electronic device, further comprising:
According to clause 13,
Controlling the camera unit to capture the buyer's hand approaching the shelf from at least one direction to generate at least one hand image, and storing the at least one hand image; further comprising: How it works.
According to clause 18,
A method of operating an electronic device, further comprising: acquiring at least one palm image among the at least one hand image based on a deep learning result performed on a human hand image.
According to clause 19,
If the buyer normally pays for the picked product, deleting the stored at least one palm image; and
If the buyer has not paid for the picked product, obtaining a fingerprint image from the stored at least one palm image, matching it with the buyer, and storing it.
According to clause 20,
Based on a deep learning result performed on the human hand image, determining the at least one palm image as one of a left palm image and a right palm image; and
A method of operating an electronic device further comprising: generating and storing an indicator indicating whether the fingerprint image corresponds to a left or right hand.
According to clause 21,
If the buyer has not paid for the picked product, the fingerprint image, an indicator indicating whether the fingerprint image corresponds to the left or right hand, the face image of the buyer, the ID of the product, the amount of unpaid damage, and the buyer A method of operating an electronic device further comprising: generating a non-payment event profile including information on the time of leaving the payment area of the unmanned store and transmitting it to an external server.

Images