TW202004619A - Self-checkout system, method thereof and device therefor - Google Patents

Abstract

A self-checkout system capable of product identification and customer abnormal behavior detection, a method therewith and a device therefor are provided herein. The self-checkout detection system includes a product identification device and a customer abnormal behavior detection device. The product identification device is capable of product identification and further capable of determining whether the products on a platform are correctly arranged or placed. The customer abnormal behavior detection device is capable of detecting a customer abnormal checkout behavior.

Description

Self-checkout system, method and device

This disclosure provides a self-checkout system, method and device.

At present, there are two main self-service checkout systems, namely, manual bar code self-checkout system and computer vision self-checkout system. The self-checkout system for manual bar code scanning reduces the incidence of customer theft by determining whether the weight of the product is abnormal, analyzing after the video, and sending people to conduct regular inspections. The computer vision self-checkout system can only identify the goods on the counter, and cannot detect whether the customer put the goods on the platform to checkout truthfully. When the product recognition status is not good, it can only be manually eliminated by the clerk.

The present disclosure provides a self-checkout system, method and device.

The self-checkout system in one of the embodiments of the present disclosure includes a platform, a product identification device, and a customer abnormal behavior detection device. The platform is configured to place at least one product. The commodity identification device is configured to perform commodity identification on at least one commodity placed on the platform. The customer abnormal behavior detection device is configured to detect abnormal checkout behavior based on the customer image obtained in front of the platform to obtain an abnormal behavior detection result, wherein, when the abnormal behavior detection result is confirmed as an abnormal behavior, A notice of abnormal behavior is issued to adjust this abnormal behavior.

The self-checkout method in one of the exemplary embodiments of the present disclosure includes commodity identification of at least one commodity placed on a platform. Get a customer image. The abnormal checkout behavior is detected based on the customer image, and an abnormal behavior detection result is obtained based on the customer image. When it is judged that the abnormal behavior detection result is an abnormal behavior, an abnormal behavior notification is issued to adjust the abnormal behavior accordingly.

The self-checkout device in one of the embodiments of the present disclosure includes a platform, an image capture device, and a processor. The platform is configured to place at least one product. The image capturing device is used to obtain a platform image and a customer image. The processor is configured to perform a commodity identification process and/or an abnormal checkout behavior detection process on the products placed on the platform. The product recognition process includes obtaining a recognition result based on the image, wherein when a recognition result cannot be obtained, a prompt notification is issued to adjust the placement of the product on the platform. The abnormal checkout behavior detection process performs abnormal checkout behavior detection based on customer images to obtain an abnormal behavior detection result. When the abnormal behavior detection result is confirmed as an abnormal behavior, an abnormal behavior notification is issued to adjust the abnormal behavior accordingly.

In order to make the above-mentioned features and advantages of the present invention more obvious and understandable, the embodiments are specifically described below in conjunction with the accompanying drawings for detailed description as follows.

The self-checkout system, one of the disclosed embodiments, includes a commodity identification device and a customer abnormal behavior detection device. The commodity identification device is used to identify the commodity, including detecting whether the commodity is placed on the platform correctly, and confirming whether the identification can be completed. Detecting the type of goods can use weight and/or depth detection to help identify the goods. The customer abnormal behavior detection device is used to detect whether the customer has abnormal checkout behavior. Based on the above, the present disclosure can also identify abnormal checkout behaviors, identify skeletons, behavior patterns, and detect goods held. It can exclude personal items such as leather bags, mobile phones, etc., and determine whether customers still hold the goods on hand. In addition, in another alternative embodiment, the self-checkout system and method can automatically identify the items and quantities of products purchased by customers, especially whether the product placement method can display enough products within the camera's perspective Feature, and prompt the customer to turn the product over or separate it to complete the product identification.

The following will illustrate the self-checkout system and method proposed in this disclosure with different implementation examples, but not limited thereto.

Please refer to FIG. 1A, which is a schematic diagram of a self-checkout system according to one of the disclosed embodiments. In this embodiment, the self-checkout system 100 includes a customer abnormal behavior detection device 110, a commodity identification device 120, and a platform 130. A clear checkout area 132 is included on the platform 130 to allow customers to place merchandise.

The customer abnormal behavior detection device 110 and the commodity identification device 120 may be connected to each other or operate independently in a separate manner. In an embodiment, the components in the customer abnormal behavior detection device 110 and the commodity identification device 120 may be shared. In an embodiment of the present disclosure, the commodity identification device 120 can be operated in preference to the customer abnormal behavior detection device 110, this method can be after the customer places all items on the checkout platform 130, before performing the checkout calculation You can confirm whether the customer still holds the product in hand. In addition, the abnormal customer behavior detection device 110 and the product identification device 120 can also operate simultaneously as required.

In one embodiment, the aforementioned abnormal customer behavior detection device 110 may include a processor 112, a storage device 114, and an image capture device 116. The processor 112 described above may be a computer central processing unit (CPU) of a general architecture, which can provide various functions by reading and executing programs or instructions stored in a storage device. Part or all of the functions of the processor 112 may be replaced by a dedicated circuit such as an application specific integrated circuit (Application Specific Integrated Circuit, ASIC). The above storage device 114 may be a nonvolatile memory (Nonvolatile Memory), such as a hard disk, a solid state drive, or a flash memory, etc., which may be used to store the acquired image. The storage device 114 can also be used to store program software or instruction sets required for the customer abnormal behavior detection device 110 to perform the customer abnormal behavior detection operation. The above-mentioned image capturing device 116, such as a camera or a video camera, is used to take a picture to obtain an image when the customer checks out.

The program software required for the detection of abnormal behavior of customers includes, for example, real-time skeleton positioning program, behavior recognition program, handheld product recognition program, etc. In an alternative embodiment, the storage device may also store multiple databases, and these databases are used to store multiple checkout behavior data and deep learning data. In another alternative embodiment, the aforementioned multiple or partial databases can be stored in a remote host server or cloud database, and the customer abnormal behavior detection device 110 can include a network access device, It can be accessed online or downloaded from a remote host server or cloud database as needed.

In one embodiment, the aforementioned product identification device 120 may include a processor 122, a storage device 124, an image capture device 126, and/or a display device 128. The processor 122 can be a computer central processing unit (CPU) with a general architecture, and can provide various functions by reading and executing programs or instructions stored in a storage device. In addition, part or all of the functions of the processor 122 may be replaced by a dedicated circuit such as a dedicated integrated circuit (ASIC). The storage device 124 may include non-volatile memory, such as a hard disk, a solid state drive, a flash memory, and so on. The storage device 124 is used to store programs required for the operation of the product identification device 120, including, for example, a product object cutting program, a product feature identification program, a product placement determination program, a product orientation determination program, and a part or all of a product connection detection program. The image capturing device 126, such as a camera or a video camera, is used to take photos of the checkout area to generate images in the checkout area 132 on the platform 130.

In an alternative embodiment, the storage device 124 may also store multiple databases, and these databases are used to store multiple checkout behavior data and deep learning data. In another alternative embodiment, the aforementioned multiple or partial databases can be stored in a remote host server or cloud database, and the commodity identification device 120 can include a network access device, which can be used as required Online access or download from a remote host server or cloud database to use. The above storage device may also include a database for storing multiple commodity data and deep learning data.

In addition, the product identification device 120 can also be configured with a display device 128 such as a screen or a projector, etc., for displaying a customer interface or displaying prompt messages, etc. The display device 128 may be a touch screen to provide a customer interface to facilitate interaction with customers. In another embodiment, the display device 128 may also be a different device other than the independent product recognition device 120, or other devices The display and the like are not limited by this embodiment. The commodity identification device 120 may also be configured with a sound playback device such as a speaker, etc., to emit sounds such as music, prompt sounds, or other instructions. Both of the above can be used simultaneously or alternatively.

For a practical application example of the self-checkout system proposed in this disclosure, refer to FIG. 1B. FIG. 1B is a flow chart illustrating a self-checkout service in a computer vision collaboration. In this computer vision collaborative self-checkout service process, the entire self-checkout process is completed with the self-checkout system 100 and/or other peripheral devices according to the following process.

Referring to FIG. 1B, during the standby in step S01, the display device of the self-checkout system 100 is in a standby state, for example, displaying a description of the use procedure. When the customer approaches, in step S02, the self-checkout system 100 is awakened. Next, in step S03, the customer places multiple products on the platform, and the self-checkout system 100 uses the image capturing device 126 of the product identification device 120 to identify the product. In one embodiment, weight and/or depth detection assistance may also be used. Identify the product. Following step S04, the corresponding information is displayed on the display device (multiple product information can be displayed simultaneously). Then, in step S05, the payment amount is displayed, and then in step S07, the customer is allowed to make a payment. And in step S08, a receipt is obtained.

In the above computer vision collaborative self-checkout service process, the computer vision commodity recognition technology used may be to detect the image characteristics of the countertop commodity through computer vision and deep learning technology, based on the shape, color, text, trademark, barcode, etc. of the commodity Features make joint decisions, instantly identify the items and quantities purchased by customers, and implement self-checkout in conjunction with mobile payments. If the product in the view angle of the image capturing device 126 does not show enough product features, such as the product is not laid flat, and the products are stacked and shielded on each other, the product identification device 120 can automatically detect and project through the screen or projector Face, separation". After turning over and separating the product, the customer can complete the identification of the product. This reminder can use color, text, and any other reminder content that can attract attention to remind customers.

In the above computer vision collaborative self-checkout service process, the computer vision commodity recognition technology features used can interact with customers to make the checkout smoothly. After the customer places the product, in one embodiment, the camera can recognize the customer's gesture to start detecting the product, or determine whether the customer is close to the checkout counter through infrared, ultrasonic, or microwave sensors. When recognizing a product, each product number can be projected on the product, and the product number and name are displayed on the display device 128 so that the customer knows the recognized product. If the product is not placed correctly, the customer is prompted to place the product correctly, and recognize the customer's gesture to start detecting the product. If the self-checkout system 100 detects that there are still products not in the hands of the customer, the customer is reminded to place the products.

In the above computer vision collaborative self-checkout service process, the abnormal checkout behavior judgment technology used includes abnormal behavior judgment and reminders, the items held by customers are not all placed in the checkout area, and the product weight does not match the recognition result and// Or the customer's operation mistakes will be actively judged and the staff will be prompted to actively assist and so on. The technologies used to judge abnormal checkout behaviors may include real-time skeleton positioning technology modules, behavior/posture recognition technology modules, handheld object recognition technology modules, etc., which will be described in detail below.

Please refer to FIG. 2, which is a schematic structural diagram of a self-checkout system according to one of the disclosed embodiments. In this embodiment, the self-checkout system 100 includes a customer abnormal behavior detection device 210, a commodity identification device 220, and a platform 230. A clear checkout area 232 is included on the platform 230 to allow customers to place merchandise. The locations of the abnormal customer behavior detection device 210 and the product identification device 220 are only schematic diagrams, and can be located at any location of the self-checkout system 100.

In an actual application example, since the customer's image is to be obtained, the customer abnormal behavior detection device 210 may include image capturing devices 212 and 214 on both sides, and the positions of the two image capturing devices 212 and 214 may be Adjust as needed, not limited to the position in the illustration. The image capturing devices 212 and 214 are used to capture a customer image obtained in front of the platform 230. The customer abnormal behavior detection device 210 is configured to detect abnormal checkout behavior based on the customer image to obtain an abnormal behavior detection result. When it is judged that the abnormal behavior detection result is an abnormal behavior, an abnormal behavior notification is issued to adjust the abnormal behavior accordingly.

The commodity identification device 220 may include an image capturing device 222 and a projection device 224 in an embodiment. The projection device 224 can, for example, project each product number on the product, and display the product number and name on the display device to let the customer know the identified product. In addition, if the product is not placed correctly, it can also prompt the customer to place the product correctly through projection, and recognize the customer's gesture to start detecting the product. The positions of the image capturing devices 212 and 214, the image capturing device 222, or the projection device 224 can be adjusted as needed, and can also be shared and shared. Therefore, for example, the customer abnormal behavior detection device 210 or the product recognition device 220 All of these devices can be driven together to achieve the necessary operations.

In one embodiment, the self-checkout system 100 may include a display device 240, which may interact with the customer via the display content 242, and may also communicate with the customer via a device such as a touch panel of the display device 240. In one embodiment, the self-checkout system 100 can communicate with an external server host 250 through a network access device. In the above embodiment, multiple or partial databases of the customer abnormal behavior detection device 210 or the commodity identification device 220 may be stored in a remote server host 250 or a cloud database (not shown).

In another embodiment, as shown in FIG. 2, the self-checkout system 100 may include at least one processor 216, a plurality of image capturing devices 212, 214, 222, a projection device 224, a storage device (not shown), and Display device 240. The processor 216 is used to execute a customer abnormal behavior detection module and a commodity identification module. The customer abnormal behavior detection module and the commodity identification module are programs or software stored in the storage device.

In one embodiment, the functions of the above-mentioned customer abnormal behavior detection module include abnormal behavior judgment and reminder, the items held by the customer are not all placed in the checkout area, the product weight does not match the recognition result, and/or the customer operation error is actively judged And prompt the staff to take the initiative to assist, etc., that is, the above several functional modules can adjust different combinations according to different needs. The technologies used to judge abnormal checkout behaviors may include real-time skeleton positioning technology modules, behavior/posture recognition technology modules, and/or hand-held item recognition technology modules, etc., some or all of them.

In one example, the functions of the aforementioned product identification module include detecting the image characteristics of countertop products through computer vision and deep learning technology, and making a joint decision based on the characteristics of the product's shape, color, text, trademark, barcode, etc. to identify the customer's purchase in real time Item, quantity, combined with mobile payment to achieve self-checkout. If the product in the camera's view does not show enough product features, such as the product is not lying flat, and the products are stacked and shielded on each other, the recognition system can automatically detect and project the "Please turn the product over and separate" prompt through the projector. After turning over and separating the product, the customer can complete the identification of the product. This reminder can use color, text, and any other reminder content that can attract attention to remind customers.

The operation flow of the device 210 for detecting abnormal customer behavior in the self-checkout system proposed in this disclosure will be described below. Please refer to FIG. 3A, which is a schematic diagram illustrating a customer abnormal behavior detection process according to an embodiment of the present disclosure. After step S310 of product identification completion or product identification is in progress, step S320 is performed to obtain a customer image in the checkout area. Next, in step S330, a customer gesture recognition process is performed according to the obtained customer image and a gesture recognition result is obtained. Then, based on the gesture recognition result, it is determined whether the customer has abnormal checkout behavior, as in step S340. If it is determined in step S340 that the customer has an abnormal checkout behavior, then proceed to step S350 to issue a notice of abnormal checkout behavior. If it is determined in step S340 that the customer has no abnormal checkout behavior, then step S360 is performed, and checkout can be performed.

Please refer to FIG. 3B and FIG. 3C, which are exemplary schematic diagrams illustrating the step S340 of the operation flow of the customer abnormal behavior detection device 210 of the disclosed embodiment, based on the customer image recognition process. The above-mentioned customer posture recognition process based on the customer image may adopt the process shown in FIG. 3B, including the behavior/posture recognition process S334 and the hand-held item recognition process S336 to obtain the above-mentioned gesture recognition result. In another embodiment, as shown in FIG. 3C, it may include performing a real-time skeleton positioning process S332, and then performing a behavior/posture recognition process S334 and a hand-held item recognition process S336 to obtain the aforementioned gesture recognition result.

For the above-mentioned real-time skeleton positioning process S332, please refer to FIG. 3D. In one embodiment, it includes executing a real-time skeleton positioning module (Realtime Human 2D Pose Estimation). The real-time skeleton positioning process S332 includes using the acquired customer image 361 as an input of a 2 branch (Convolutional Neural Network, CNN). As shown in FIG. 3D, the customer image 361 is input into the first branch and the second branch. After a two-stage operation, a confidence map (Confidence Map) for jointly predicting Body Part Detection and Part Affinity Field is used to obtain partial associations. Part Affinity Field (Part Affinity Field) is a set of 2D vector areas used to encode the position and orientation of limbs in the image domain. The two-branch model is trained through the image marking of Body Part and Part Affinity Field. In the 2-branch multi-stage CNN architecture, the stage t in the first branch predicts the reliability mapping table St, and the stage t in the second branch predicts the PAFs Lt. After each stage, the predictions and image features from the two branches are connected together in the next stage, and then the next stage of prediction is performed. Obtain real-time skeleton positioning information according to the above process.

For the above-mentioned behavior/posture recognition process and hand-held item recognition process, please refer to FIGS. 4A and 4B and refer to FIG. 3B or 3C for description. Please refer to FIG. 4A, in this embodiment, a behavior/posture identification (Human Pose Identification) module is executed, and FIG. 4B illustrates five common checkout gestures. First, according to the acquired customer image 410, after detecting the key points of the body (step S332), the key points of the shoulders, elbows and wrists are used as a pattern to identify the behavior of the monitored person (step S334), as shown in the figure The key point line 412 of the shoulder, elbow and wrist in 4A. After human body gesture recognition, the candidate region 414 in the image is extracted to detect hand-held objects. And according to such an architecture, within this range, the YOLO algorithm such as step 416 is used as a method for the object detector to locate the object and identify the object type for palm/handheld commodity detection and recognition (as in step S336). YOLO refers to "You Only Look Once", which can be used to identify objects. In one embodiment, using the YOLO model to perform a CNN on a picture can determine the type and location of the objects inside, which can greatly improve the recognition speed. In this implementation example, through the YOLO algorithm as a method of locating objects and identifying object types, the confidence index and bounding box information of five common checkout behaviors are obtained, and the behavior/posture recognition result 411 is obtained. In the YOLO algorithm, the customer image 410 is divided into multiple bounding boxes (bounding-box), and the position of each boundary box in the customer image 410 is composed of two coordinate points (x1, y1) and (x2, y2) Defined, and for each bounding box to calculate the probability of what kind of object. Each bounding box has five prediction parameters, including x, y, w, h, and confidence index. (x, y) represents the displacement of the center of the box, and w, h are the length and width of the bounding box, which can be defined by the coordinate points (x1, y1) and (x2, y2). The confidence index (Confidence) contains the degree of confidence in the predicted object and the accuracy of the object in this bounding box. This step can detect whether people still hold commodities when using the automatic checkout system. The five recognized object types include, for example, the recognition result R1 mobile phone, R2 wallet, R3 handbag, R4 bottle or R5 canned beverage, so as to identify whether the hand-held item is a commodity.

In this embodiment, the key points of the body can be detected to obtain the posture category of the human body, as shown in FIG. 4B, to recognize the checkout behavior of the monitored person and perform hand-held commodity detection and identification. Taking the customer's image 420 or 422 as an example, the bounding box of the hand-held object can be marked through the above-mentioned behavior/posture recognition module. After skeleton positioning and behavior/posture recognition, define a range (such as the junction of the hand, arm and body) as the area where the product and/or palm may appear, and according to the shoulder, elbow and wrist key point lines 412 and the image The candidate area 414 (the area indicated by the dashed line) can determine the detection of handheld products in different gesture categories. For example, 431 to 435 can distinguish the types of human poses, such as the shoulder, elbow, and wrist key point lines 412 of the pose 431 can be judged as the gesture of holding an item, and the candidate area 414 (the area indicated by the dotted line) in the image ) You can judge whether you have an item or not. Therefore, the posture 431 can be classified into the category of “human posture”. In addition, posture 432 can be categorized as a "posture in both hands" human body posture category. The key point line 412 of the shoulder, elbow and wrist of the posture 433 can be judged as the posture of holding an item in one hand and holding another item under the shoulder, so it can be classified as "the item in one hand and the other item in the shoulder of one hand The "posture" category of human pose. The key point line 412 of the shoulder, elbow, and wrist of the posture 434 can determine that both hands are sagging, so it can be classified as the "posture of both hands" human body posture category. Or five different posture categories such as "other postures" of posture 435. After recognizing the posture category of the person to be monitored, the handheld goods can be detected and identified.

In an embodiment of the present disclosure, personal items such as leather bags, mobile phones, etc. are excluded through palm tracking and detection of held commodities, so as to identify whether the handheld items are commodities. In detail, after the body skeleton is detected, the body skeleton line is obtained, the multiple nodes of the shoulder, elbow and wrist in the body skeleton line are identified, that is, the junction of the hand, arm and body, and then the body skeleton line Compare with the preset model to obtain the posture category of the hand-held object. For example, please refer to the customer's image 420 in FIG. 4B. The character in the image 420 is in accordance with his body skeleton line and line node, and is held with another object in one hand. The default model under the shoulder of one hand is the most similar, so it is judged that the customer may hold the product in one hand and hold the other item under the shoulder of the other hand, and then perform the step of delineating the candidate area of the handheld item, which can be identified using behavior and gesture recognition technology For example, determine the end node of the body skeleton line (representing the position of the hand), and then define the range of the right-hand candidate area including the end node of the body skeleton line and the nodes that will contain objects such as the shoulder and elbow, and the range of the left hand area includes the body skeleton line End nodes and wrist nodes. After demarcating the hand-held item candidate area, you can identify whether there is an item in the hand-held item candidate area. In one embodiment, if it is determined that there is an item in the hand-held item candidate area, you can identify whether the item in the hand-held item candidate area is commodity.

Please refer to FIG. 5, which is a schematic diagram illustrating a computer vision commodity recognition process proposed by the disclosed content embodiment. Here, the computer vision commodity identification process includes at least the commodity image feature identification process and the commodity image feature analysis. The product identification device 220 of this embodiment can store different applications or can communicate with the external server host 250 or cloud database (not shown) through the network access device to access the required data or Is a software program. The programs required for the operation of the commodity identification device 220 of this embodiment include, for example, commodity object cutting programs, commodity feature identification programs, commodity placement judgment programs, commodity-oriented judgment programs, and/or part or all of commodity connection detection programs, and so on.

In step S510, the commodity recognition device starts to operate, and the image on the platform 230 is obtained through the image capturing device 222. In step S520, a product image feature recognition process is performed. In one embodiment, the processor 216 loads the commodity object cutting program stored in the storage device into the memory device, and executes the commodity object cutting program to cut the commodity image, to recognize and capture the commodity image features, such as shape, Color distribution, text, trademark location or content. In one embodiment, a plurality of commodities are placed on the platform 230. Therefore, the captured image includes images of the plurality of commodities. The image feature recognition process may include cutting the images of the plurality of commodities; the processor 216 will store them in the storage The product object cutting program of the device is loaded into the memory device, and the product object cutting program is executed to cut the obtained image to find the image of each product. In one embodiment, the commodity object cutting process, for example, cuts out multiple commodity regions from the image by edge detection to obtain each commodity image. The cutting process of the goods will be described later, and will be explained in conjunction with FIGS. 6A and 6B. After obtaining the product image, the product image features are identified based on the product image for subsequent comparison and analysis.

After identifying the features of the commodity image, the process of analyzing the features of the commodity image is performed according to these features, as shown in step S530. In step S530, the acquired product image features, such as shape, color distribution, text, trademark, barcode position or content, etc., are analyzed with a feature database for product image recognition, for example, referring to the established features Database to analyze the items and quantities of products purchased by customers.

In step S540, the product recognition result is confirmed. In one embodiment, it is determined whether the product of the product image is consistent with the product of the database, for example, whether the image feature of the product is consistent with the image feature of the product of the feature database, and if it is the same, the product of the product image is determined to be the feature database Product and proceed to step S560 to complete the product identification. In one embodiment, if it is determined that the image feature of the product is inconsistent with the image feature of the product in the feature database, or whether the product image feature cannot be determined as a product in the feature database, then proceed to step S550 to notify the customer to adjust the platform The position of the commodity returns to step S510 to capture the adjusted commodity image on the platform. In one embodiment, in step S540, if there are a plurality of identified products, and one of the products cannot be determined whether the product image feature is a product in the feature database, it will proceed to step S550.

The following describes the product image feature recognition process of step S520 in detail with an embodiment. In one embodiment, the image is processed first, for example, the obtained product object image is cut, and then the features of the product image are extracted. Please refer to FIGS. 6A and 6B, which are schematic diagrams for explaining the image cutting process of the commodity object according to the disclosed embodiment. In FIG. 6A, the commodity object cutting program cuts the commodity area according to the acquired image 610 by edge detection, adjusts the lightness characteristics in the image to increase the contrast between the background and the commodity, and uses, for example, Sobel Edge Detection (Sobel Edge Detection) method Find out the product boundary, and then use the Run Length algorithm to reinforce the boundary and suppress noise, and then determine the boundary and divide the product area. Referring to FIG. 6B, by marking the product area of the acquired image 620, the coordinates of the product area can be calculated to obtain the area where the product image exists, and then the features of the product image can be found according to the area of the product image. Then, based on these features, the product image feature analysis process of step S530 is performed.

In step S530, the acquired product image features can be referred to the established feature database to analyze the items and quantities of the products purchased by customers. FIG. 6C illustrates a schematic diagram of product feature identification proposed by the disclosed embodiment. In an embodiment, for example, the object cutting program described above can be executed to obtain the product image features. Then the processor 216 loads the commodity feature recognition program stored in the storage device into the memory device and executes the commodity feature recognition program to detect multiple features in the commodity areas using deep learning or other algorithms, and Based on these characteristics, identification is performed to obtain multiple product identification results. In one embodiment, through the detection of the characteristics of the product area, deep learning technology is used to perform product rotation and image angle recognition to extract the whole (such as shape and color distribution) of high-resolution images and detailed features (such as text) And trademarks) to identify the products purchased by customers. Different commodities 630 to 660 as shown in FIG. 6C.

In an embodiment of the present disclosure, commodity classification may be performed in the commodity image feature analysis process of step S530. The processor 216 loads the commodity classification program stored in the storage device into the memory device and executes the commodity classification process. Please refer to FIG. 7A, which is a schematic diagram illustrating the classification process of the commodities proposed by the disclosed embodiment. This classification process includes the step of setting the confidence value of the classification result in step S710, the step of identifying the product orientation in step S720, and the product connection detection step of step S730.

First, in step S710, the confidence value of the classification result is first established. Please refer to FIG. 7B, which is a schematic diagram for explaining the confidence value table of the classification result established by the embodiment of the disclosure. The commodity classification program calculates the confidence value of the classification result of the commodity classification based on the image characteristics of the commodity. For example, based on the image characteristics of the commodity, the highest three classification results of the commodity 1 are calculated. The confidence values are 0.956, 0.022, 0.017, which may be the highest of the commodity 2 The confidence values of the three classification results are 0.672, 0.256, 0.043, to establish the confidence value table of the classification results as shown in FIG. 7B, and use the confidence values of the classification results to determine whether they are credible, such as the confidence value of the classification results (Confidence Value) ) Whether it is greater than the threshold value, if it is greater than the threshold value, it has credibility. Taking FIG. 7B as an example, if the threshold value is 0.7, because the confidence value of the highest classification result of commodity 1 may be 0.956, then the product image feature is judged For goods 1. In an embodiment, when the confidence value of the classification result has credibility or the commodity can be judged according to the confidence value of the classification result, step S720 is no longer required. If the confidence value of the classification result is less than the threshold, step S720 is performed.

In step S720, the product identification is performed. In an embodiment of the present disclosure, after executing the product feature identification program, the processor loads the product placement determination program stored in the storage device into the memory device and executes it. The product placement judgment program is used to determine whether the object placed on the platform is a product, whether the upward-facing side of the product placed on the platform is a surface with fewer features, or whether the product is clearly characterized by the image capture unit of the platform Way.

Please refer to FIG. 7C, which is a schematic diagram for explaining the product orientation determination process proposed by the embodiment of the disclosure content to determine the orientation of the product. Please refer to steps S720 and 7C in FIG. 7A, the product orientation determination program can determine the orientation of the products placed on the platform, for example, using deep learning technology for image recognition, and determine whether the captured product image is a surface with fewer features, such as The top of the Le Bao 722, the bottom of the Tetra Pak 724 or the cap surface of the Baote bottle 726, etc. If it is judged that the number of upward-facing features of the product image is insufficient or too small, it is judged that there are fewer features and it is difficult to identify which product. In an embodiment, when it is determined that there are fewer features, that is, the number of features is insufficient, the customer may be notified to adjust the orientation of the product placement, and it is not necessary to perform step S730.

Please refer to FIG. 7D, which is a schematic diagram for explaining the connection detection of commodities provided by the embodiment of the present disclosure. Please refer to FIG. 7A and FIG. 7D at the same time. Taking the beverage bottle 732 of FIG. 7D as an example, after the product orientation determination program is executed, if the number of features of the product facing upward is sufficient to identify, it can be determined that the product is lying on the platform. Then the processor loads the commodity connection detection program stored in the storage device into the memory device and executes it to perform the commodity connection detection step of step S730. The product connection detection program is used to detect whether several products are connected or overlapped by the aspect ratio of the product. For example, if the normal (or database) canned beverage has an aspect ratio of 2:1, when it is recognized that it is flat When the canned beverage is lying and the aspect ratio of the canned beverage is detected to be 1:1, it can be judged that the canned beverage is connected to another product. In one embodiment, a prompt message may be issued to inform the customer that adjustment is required Product location.

Please refer to FIG. 7E, which is a schematic diagram illustrating the method for prompting the customer to adjust the product arrangement proposed in an embodiment of the present disclosure. In this embodiment, a prompt of "please place the product on the platform" may be projected through the projector, or a prompt such as voice or on-screen text may be used to request the customer to place the product on the platform and then execute the product identification process again. Prompt messages can use sound, graphics, colors, text, bar codes and other prompts to remind customers.

In another exemplary embodiment, a prompt message prompting the customer to adjust the way the goods are displayed uses a projector to project a different color label on the platform 740, for example, the product 734 is projected with a first color light that is different from other areas of the platform 740. The first color area 742. The second color area 744 can also be generated by simultaneously projecting light of a second color that is different from other areas of the platform 740 and the first color onto another product 722 and 726. This will allow customers to clearly know which products need to be adjusted. In addition to this embodiment, the message that prompts the customer to adjust the placement of the product can also be used to project the "Please turn the product over and separate" prompt, for example, a voice, screen text and other prompts can be used to ask the customer to turn the product Face or separate, and then re-execute the commodity identification process. The prompt message can use sound, graphics, color, text and other prompts to remind customers.

In summary, this disclosure proposes to detect the regional characteristics of commodities through computer vision and deep learning, and to identify the items and quantities of products purchased by customers. If the product in the camera's viewing angle does not show enough product features, the customer can be reminded to turn the product over and separate it by sound, graphics, color, text and other prompts. In the detection of abnormal checkout behavior, through real-time skeleton positioning, using the nodes of shoulders, elbows and wrists as a pattern to recognize the checkout behavior of the monitored person, and carry out hand-held object detection, and use sound, graphics, color, text After reminding the customer to place the product on the platform, repeat the product identification step.

This disclosure proposes a self-checkout system and method, which includes the functions of commodity identification and judgment of abnormal customer behavior. The self-checkout system includes a product identification function and a customer abnormal behavior detection function. The commodity identification function is used to identify the commodity, including detecting whether the commodity is placed on the platform correctly and confirming whether the identification can be completed. The customer abnormal behavior detection function is used to detect whether the customer has abnormal checkout behavior.

The self-checkout system and method proposed in this disclosure can instantly identify the items and quantities purchased by customers, combine with mobile payment to achieve self-checkout, and reduce the theft rate. Based on the above, this self-checkout system and method can identify the items and quantities of products purchased by customers, especially whether the product placement method can display sufficient product features within the camera's perspective, and prompt the customer to turn over or separate the products To complete the identification of goods. In addition, the present disclosure can also identify abnormal checkout behaviors, identify skeletons, behavior patterns, and detect goods held. It can exclude personal items such as leather bags, mobile phones, etc., and determine whether customers still hold the goods on hand.

Although the present invention has been disclosed as above with examples, it is not intended to limit the present invention. Any person with ordinary knowledge in the technical field can make some changes and modifications without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be subject to the scope defined in the appended patent application.

100‧‧‧Self-checkout system 110‧‧‧Customer abnormal behavior detection device 112‧‧‧processor 114‧‧‧‧storage device 116‧‧‧image capture device 120‧‧‧commodity recognition device 122‧‧‧ processing 124 ‧ ‧ ‧ storage device 126 ‧ ‧ ‧ image capture device 128 ‧ ‧ ‧ display device 130 ‧ ‧ ‧ platform 132 ‧ ‧ ‧ checkout area S01 ~ S08 ‧ ‧ ‧ computer vision collaborative self-checkout service flow 210 ‧ ‧ ‧Customer's Abnormal Behavior Detection Device 212, 214‧‧‧ Image Capture Device 216‧‧‧ Processor 220‧‧‧Commodity Recognition Device 222‧‧‧Image Capture Device 224‧‧‧Projection Device 230‧‧‧‧232 ‧‧‧ Checkout area 240‧‧‧Display device 242‧‧‧Display content 250‧‧‧Server host S310~S360‧‧‧Operation flow of customer abnormal behavior detection device S332~S336‧‧‧Customer attitude recognition flow 410‧‧‧Customer image 411‧‧‧Posture recognition result 412‧‧‧Shoulder, elbow and wrist key point lines 414‧‧‧Candidate area 416‧‧‧YOLO algorithm 431~435‧‧‧Products and palm may appear 610, 620‧‧‧ acquired images 630~660‧‧‧Product features S710~S730‧‧‧Product classification process 722, 724‧‧‧Top and bottom images of square beverage packaging 726‧‧‧ Bottled beverage lid 732‧‧‧Beverage bottle image 734‧‧‧Beverage bottle and other products image 740‧‧‧Platform 742‧‧‧First color area 744‧‧‧Second color area

FIG. 1A shows a schematic structural diagram of a self-checkout system according to one of the disclosed embodiments. FIG. 1B is a schematic diagram illustrating the process architecture of a computer vision collaborative self-checkout service. FIG. 2 is a schematic structural diagram of a self-checkout system according to one of the disclosed embodiments. FIG. 3A is a schematic diagram illustrating a customer abnormal behavior detection process according to an exemplary embodiment of the present disclosure. FIGS. 3B to 3D are schematic diagrams illustrating the process of performing customer pose recognition based on customer images in the disclosed exemplary embodiment. 4A and 4B are schematic diagrams illustrating the behavior/posture recognition process and the handheld item recognition process of the disclosed exemplary embodiment. FIG. 5 is a schematic diagram illustrating a computer vision commodity recognition process proposed by the disclosed content embodiment. 6A and 6B are schematic diagrams respectively illustrating a cutting process of a commodity object according to an embodiment of the present disclosure. FIG. 6C is a schematic diagram illustrating the identification of product features proposed by the disclosed content embodiment. FIG. 7A is a schematic diagram illustrating the classification process of commodities proposed by the disclosed content embodiment. FIG. 7B is a schematic diagram illustrating a confidence value table for classification results established according to an embodiment of the disclosed content. FIG. 7C is a schematic diagram illustrating the product orientation determination process proposed by the disclosed content embodiment to determine the orientation of the product. FIG. 7D is a schematic diagram illustrating the connection detection of commodities proposed by the embodiment of the disclosed content. FIG. 7E is a schematic diagram illustrating a method of prompting a customer to adjust a product display according to an embodiment of the disclosed content.

100‧‧‧ Self-checkout system

110‧‧‧Customer abnormal behavior detection device

112‧‧‧ processor

114‧‧‧Storage device

116‧‧‧Image capture device

120‧‧‧Commodity identification device

122‧‧‧ processor

124‧‧‧Storage device

126‧‧‧Image capture device

128‧‧‧Display device

130‧‧‧platform

132‧‧‧ Checkout area

Claims (36)

A self-checkout system includes: a platform configured to place at least one commodity; a commodity identification device configured to perform commodity identification on the at least one commodity placed on the platform; and a customer abnormal behavior detection The testing device is configured to perform abnormal checkout behavior detection based on a customer image obtained in front of the platform to obtain an abnormal behavior detection result, wherein, when the abnormal behavior detection result is determined to be an abnormal behavior, A notice of abnormal behavior is issued to adjust the abnormal behavior accordingly. The self-checkout system as described in item 1 of the patent application scope, wherein the device for detecting abnormal customer behavior includes: at least one image capture unit for obtaining the customer image; and a processor configured to The customer image performs the abnormal checkout behavior detection to obtain the abnormal behavior detection result, wherein the abnormal checkout behavior detection includes performing a gesture recognition process to confirm a checkout posture in the customer image, and then based on the checkout Posture, a hand-held item recognition process is performed on an area to obtain the abnormal behavior detection result. The self-checkout system as described in item 2 of the patent application scope, wherein the processor of the customer abnormal behavior detection device is configured to perform a real-time skeleton positioning process on the customer image before the gesture recognition process is configured, In order to obtain the skeleton positioning information of the customer in the customer image, to perform the gesture recognition process. A self-checkout system as described in item 3 of the patent application scope, wherein the processor is configured to obtain the customer's body skeleton lines from the customer's image, and compare the body skeleton lines with a preset model to obtain the Skeleton positioning information. The self-checkout system as described in item 2 of the patent application scope, wherein the processor of the customer abnormal behavior detection device is configured to obtain a plurality of key points in the customer's image according to The formed key point line is compared with the preset model to obtain the checkout posture corresponding to the customer. The self-checkout system as described in item 5 of the patent application scope, wherein the processor of the customer abnormal behavior detection device further obtains the human posture category according to the checkout posture, and is used to determine the location and range of a hand-held item candidate area, To carry out the process of identifying the handheld item. The self-checkout system as described in item 1 of the patent application scope, wherein the commodity identification device performs the commodity identification on the at least one commodity placed on the platform to obtain an identification result, wherein when the identification result cannot be obtained, A reminder notice is issued to adjust the placement of the at least one commodity on the platform. The self-checkout system as described in item 1 of the patent application scope, in which the product identification device recognizes the customer gesture of the customer image through a camera to start the product identification, or determines whether the customer is approaching through infrared, ultrasonic or microwave sensing The platform starts to identify the product. The self-checkout system as described in item 1 of the patent application scope, wherein the commodity identification device is configured to project the serial number on the at least one commodity. The self-checkout system as described in item 7 of the patent application scope, wherein the commodity identification device includes: an image capturing unit that acquires a platform image of the at least one commodity placed on the platform; a processor The platform image performs the product identification, obtains a plurality of features corresponding to the at least one product, and compares the features with a product feature database to obtain the identification result. The self-checkout system as described in item 10 of the patent application scope, wherein the processor of the commodity identification device is configured to perform the commodity identification on the platform image and obtain the features corresponding to the at least one commodity for When comparing and obtaining the recognition result, if the number of the features is not enough for judgment, the prompt notification is issued to adjust the placement method of the at least one product on the platform. The self-checkout system as described in item 11 of the patent application scope, wherein the processor of the commodity identification device is configured to cut out multiple commodity areas by edge detection in the image of the platform, and to the commodity areas Among them, the features of the at least one commodity are detected, and the features of the at least one commodity are identified. The self-checkout system as described in item 12 of the patent application scope, wherein the processor of the product identification device is configured to perform the product identification on the platform image based on the image in the product area and the product characteristics The database compares and obtains the corresponding confidence level of a classification result, determines whether the confidence level of the classification result is greater than a threshold, and determines whether the identification result is obtained. A self-checkout method includes: product identification for at least one product placed on a platform; obtaining a customer image; and detecting abnormal checkout behavior based on the customer image, and obtaining an abnormal behavior detection based on the customer image Test results, wherein when it is determined that the abnormal behavior detection result is an abnormal behavior, an abnormal behavior notification is issued to adjust the abnormal behavior accordingly. The self-checkout method as described in item 14 of the patent application scope, wherein the abnormal checkout behavior detection includes performing a gesture recognition process to confirm a checkout posture in the customer's image, and then performing an area on the checkout posture A hand-held item identification process to obtain the abnormal behavior detection result. The self-checkout method described in item 15 of the patent application scope, wherein before the gesture recognition process, a real-time skeleton positioning process is performed on the customer image to obtain the customer's skeleton positioning information in the customer image, so that To perform the gesture recognition process. The self-checkout method described in item 16 of the patent application scope, wherein the real-time skeleton positioning process obtains the customer's body skeleton lines from the customer image, and compares the body skeleton lines with a preset model to obtain the skeleton positioning News. The self-checkout method as described in item 15 of the patent application scope, wherein the hand-held item recognition process includes obtaining a plurality of key points in the customer image, and the key point lines formed by the key points and the preset The models are compared to obtain the checkout posture corresponding to the customer. The self-checkout method as described in item 18 of the patent application scope, wherein the position and range of a hand-held item candidate area are further determined according to the check-out posture to perform the hand-held item identification process. The self-checkout method as described in item 14 of the patent application scope further includes acquiring a platform image of the at least one product on the platform, obtaining a recognition result based on the platform image, and issuing when the recognition result cannot be obtained A prompt notification to adjust the placement of the at least one product. The self-checkout method as described in item 14 of the patent application scope, which further includes starting the product identification by recognizing the customer gesture of the customer's image or judging whether the customer is close to the platform by infrared, ultrasonic or microwave sensing Start the product identification. The self-checkout method as described in item 14 of the patent application scope, which further includes projecting a serial number on the at least one commodity. The self-checkout method as described in item 20 of the patent application scope, wherein the commodity identification includes obtaining a plurality of features corresponding to the at least one commodity based on the platform image, and comparing the features with a commodity feature database Obtain the recognition result. The self-checkout method as described in item 23 of the patent application scope, wherein the product identification is performed on the platform image, and the features corresponding to the at least one product are obtained for comparison to obtain the identification result. If the number of features is insufficient to judge, then the prompt notification is issued to adjust the placement of the at least one commodity on the platform. The self-checkout method as described in item 24 of the patent application scope, wherein performing the product identification on the platform image to obtain the features corresponding to the at least one product includes cutting a plurality of edge detections from the platform image A commodity area, and the features of the at least one commodity are detected in the commodity areas, and the features of the at least one commodity are identified. The self-checkout method as described in item 25 of the patent application scope, wherein the number of the features obtained when the product identification is performed on the platform image is the image of the product areas and the product features cut out based on the platform image The database is compared, and the confidence of a corresponding classification result is obtained, and it is judged whether it is greater than a threshold to determine whether the recognition result is obtained. A self-checkout device includes: a platform configured to place at least one commodity; an image capture device to obtain a platform image and a customer image; and a processor configured to place on the The at least one commodity on the platform undergoes a commodity identification process or an abnormal checkout behavior detection process, wherein the commodity identification process includes obtaining a recognition result based on the platform image, wherein when the recognition result cannot be obtained, a prompt notification is issued to Adjusting the placement method of the at least one product on the platform, wherein the abnormal checkout behavior detection process performs abnormal checkout behavior detection based on the customer image to obtain an abnormal behavior detection result, wherein, when the abnormal behavior is detected When the test result is confirmed to be an abnormal behavior, an abnormal behavior notification is issued to adjust the abnormal behavior accordingly. The self-checkout device of claim 27, wherein the processor is configured to identify the product on the platform image, obtain a plurality of features corresponding to the at least one product, and according to the features and A product feature database is compared to obtain the recognition result. The self-checkout device as described in item 28 of the patent application scope, wherein the processor is configured to perform the product identification on the platform image, obtain the number of features corresponding to the at least one product for comparison, and obtain the identification As a result, if the number of the features is not enough for judgment, the prompt notification is issued to adjust the placement of the at least one product on the platform. The self-checkout device as described in item 29 of the patent application scope, wherein the processor is configured to perform the product identification on the platform image to obtain the features corresponding to the at least one product, including The edge detection cuts out a plurality of commodity areas, and detects the characteristics of the at least one commodity in the commodity areas, and performs identification to obtain the characteristics of the at least one commodity. The self-checkout device as described in item 30 of the patent application scope, wherein the processor is configured to identify the image of the platform, and the number of the features is obtained based on the images of the product areas cut out Compare with the product feature database to get the confidence of a corresponding classification result, determine whether it is greater than a threshold, and determine whether to obtain the recognition result. The self-checkout device of claim 27, wherein the processor is configured to detect the abnormal checkout behavior of the customer image to obtain the abnormal behavior detection result, wherein the abnormal checkout Behavior detection includes performing a gesture recognition process to confirm a checkout posture in the customer image, and then performing a handheld item recognition process on an area according to the checkout posture to obtain the abnormal behavior detection result. The self-checkout device as described in item 32 of the patent application scope, wherein the processor is configured to perform a real-time skeleton positioning process on the customer image before obtaining the gesture recognition process to obtain the customer image Customer's skeleton positioning information, so as to carry out the gesture recognition process accordingly. A self-checkout device as described in item 33 of the patent application scope, wherein the processor is configured to obtain the customer's body skeleton line from the customer image, and compare the body skeleton line with a preset model to obtain the Skeleton positioning information. The self-checkout device described in item 34 of the patent application scope, wherein the processor is configured to obtain a plurality of key points in the customer image, and the key point lines formed by the key points and the pre-order Set the model for comparison to obtain the checkout posture corresponding to the customer. The self-checkout device described in item 35 of the patent application scope, wherein the processor is configured to perform the hand-held item identification process and further includes obtaining a human posture category according to the check-out posture, and used to determine the position of a hand-held item candidate area And range to carry out the process of identifying the handheld item.

Images