JP5054670B2 - Method and apparatus for detecting suspicious behavior using video analysis - Google Patents

Description

  The present invention relates to a method and apparatus for detecting suspicious behavior using video analysis.

  Retail stores typically use other transaction terminals, often referred to as point-of-sale terminals or cash registers, to enable the purchase of items by customers at these retail stores. For example, in traditional department stores, supermarkets, or other retail stores, customers walk around the store to collect purchased items and store these items in shopping carts, baskets, or simply point these items at the point-of-sale terminal. To the point of purchase and purchase these items included in the transaction. An operator such as a cashier can be arranged at the point-of-sale information management terminal, and the cashier is a person who is employed by the retail store to assist the customer and complete the transaction. In some cases, the retail store has introduced a self-sale point-of-sale information management terminal in which the customer is an operator. In either case, the operator typically places the purchased item on a bill of accounts, conveyor belt, or other item loading area. The point-of-sale information management terminal includes a scanning device such as a laser scanner or an optical scanner, and the scanner is a standard product code (UPC) label or bar code attached to each item that the customer desires to purchase. Works to identify. A laser scanner is typically a peripheral device connected to a computer that is part of a POS terminal. To scan the items, the operator picks each item one by one from the item carry-in area and passes the item over a scanning area such as a glass window built into a billboard or checkout area, The UPC code can be detected by the laser scanner. Once the point-of-sale information management computer enables the UPC code to be detected. Once the point-of-sale management computer identifies the UPC code on the item, the computer can look up the database to determine the price and identification information for the scanned item. Alternatively, in each case where the operator can scan the item, the operator can similarly enter the UPC code or product identification code into the terminal manually or through an automatic product identification device such as an RFID reader. can do. The term “scan” is usually defined to include all means of entering transaction items into a transaction terminal. Similarly, the term “scanner” is usually defined as all automatic and / or manual transaction terminals that record transaction information.

  Since the operator scans or inputs each purchased item one by one, the point-of-sale information management terminal maintains the accumulated total purchase price for all of the items included in the transaction. For each item that the operator has successfully scanned or entered, the point-of-sale terminal typically sounds a beep or beep to notify the operator that the item has been scanned by the point-of-sale terminal, and In response, the operator can place the item on an item unloading area, such as a downstream conveyor belt, or other area, where the customer can pick up the item or pack the item into a shopping bag. Like that. Once all items included in the transaction have been scanned in this way, the operator informs the point-of-sale terminal that the scanning process is complete, and the point-of-sale terminal manages the total purchase price. The customer pays for the item purchased in the transaction to the retail store.

  Conventional systems that allow the purchase of items using a scanner or other automatic item identification means by code have various drawbacks. In particular, the operation of such a system is compromised in such a way that a customer can possess one or more items without payment, either intentionally or unintentionally by the operator. It can happen. In particular, such a system can also be used as “sweating” where the operator does not scan the item as it moves through the transaction area, either intentionally or at any time. It is easy to cause the known “pass-throughs”. In such cases, the POS system cannot detect any items that are not scanned, so the items are never added to the purchase price. In such cases, the customer effectively obtains the item free of charge. Retail store chains can cost millions of dollars a year due to operator error or this type of fraud.

  In an example where fraud does not occur, the operator unknowingly passes the item through the scanning area during the transaction and places the item in an item delivery area, such as a downstream conveyor belt, but the item is scanned. There wasn't. Perhaps the operator is not paying attention and has not noticed (or did not care) that the scanner did not ring during the scan of the item.

  As an example of a “relationship transaction” symbolizing fraud in a retail store, an operator who can personally help a customer familiar with the operator (eg, an operator friend or conspirator) deliberately uses, for example, a UPC label. By covering with the operator's hand or moving the UPC code out of the scanning range of the scanner, the operator moves the item past the transaction area and prevents the POS system from scanning the item. In such a situation, the item blends into other items that have been scanned or not scanned in the same way, and the customer or operator continues to behave as if nothing happened. Often found on low-priced items, due to actual scanning where transactions are legitimately visible to anyone, or in situations where unscanned items are grouped with items that appear scanned With “Nature Trading”, after the operator has processed all items included in the transaction, the customer pays the operator a purchase price that represents only the total amount of all scanned and entered items. After making a payment, the customer takes all scanned / entered and unscanned items from the store and pays only for the scanned or entered items.

  In another fraudulent example known as “label replacement”, the operator causes the POS system to scan an item that is different from the item that is passing through the transaction area during the transaction. In such cases, the customer or operator replaces the UPC label of the original item, often expensive, with another low-cost UPC label. In such cases, a scan is performed, but the wrong item is identified by the POS system. In this way, the system will scan the item for a price that is much lower than the value of the item received by the customer.

  Security system designers have attempted to develop prior art to detect fraudulent operator errors or sudden operator errors when using POS terminals. Conventional systems in the field of detecting “slip-through” and “near trade” allow for the detection of unusually long “scan-gaps”. The “scan gap” is the length of time between successive scans at the point-of-sale information management terminal. If an item is passed through without scanning the item, the scan gap becomes longer before the next scan is performed. By comparing the scan gap as the relationship between the scanned item of the transaction for a given employee operator versus the average scan gap, the traditional method of using a scan gap is when the item bypasses the scanner without being scanned. Try to identify the event.

  The conventional detection method using the scan gap has been found to be a “noisy” indicator even under the best conditions of the scan gap, and as a general view, is considered impractical. This is largely due to delays such as those caused by full legitimate scan gips resulting from product loading, manual entry of unlabeled or unscanned articles, and rescans of items that were not scanned on the first pass. Based on the fact that you get. As a result, since the scan gap is no longer a reliable indicator, conventional systems that attempt to use the scan gap as a method of detecting fraudulent behavior are prone to problems.

  The system of the present invention generally performs a video count or video identification of items included in a transaction, as captured in video data, and obtains this item identification information from a transaction terminal such as a point-of-sale management register. By reporting the suspicious state compared to the transaction data being reported, it notifies of illegal behavior or operator error. In addition, by obtaining video data sent from at least one video camera that monitors the transaction area, the system automatically analyzes the video data (ie, does not require human intervention) and enables transactions in the transaction area. Keep track of items included in the. Using this information, the system can compare the video analysis result of the tracked item with transaction data generated by the transaction terminal to identify suspicious behavior.

  Unlike conventional systems, the system of the present invention uses video data analysis methods to detect behaviors such as “retail transactions” or “pass through”, as described below. In particular, the system scans a fraud code when an operator, such as a customer or store clerk, passes near a scanner (or RFID reader) without scanning one or more items, or the operator scans for an item. Or the fact of inventory theft or loss at a cash register, POS, or other transaction terminal when manually entered into the transaction terminal. The system of the present invention can also detect items that could be incorrectly pasted by an incorrect bar code and read incorrectly by the scanner, or entered by the operator as an incorrect item. In some embodiments, video analysis is utilized in association with transaction scan data for items actually scanned by the POS terminal.

  It can be appreciated that point-of-sale information management terminals or cash registers that utilize scan operations are merely examples of transaction terminals and that the system is not limited to fraud detection only in a retail store environment. Further, the scanning operation is not limited to laser scanning with a fixed scanner, but can include scanning with a portable scanner or a radio frequency identification (RFID) reader. The system can also be applied in situations where an operator manually enters codes or other item identification information into the transaction terminal through a keyboard. The system of the present invention can be widely applied to all environments in which transaction data can be used for comparison with video data related to the transaction.

  For example, a system for identifying items using an RFID tag can exhibit the advantages of the system of the present invention. For example, a toll booth system provides video data for vehicles passing through the toll booth, and by means of such a system, an operator such as a human or an RFID vehicle transceiver reading system, such as an RFID vehicle transceiver reading system. Automatic scanners can collect tolls from vehicles traveling on highways. Fraud can occur in such systems, for example, when an illegal transceiver is installed in a vehicle (eg, a car transceiver is installed in a truck). Further, a terminal operator can refer to either a store clerk or a customer as in a situation such as a self-checkout transaction terminal.

  More particularly, the system of the present invention includes a method and apparatus for determining transaction outcomes such as suspicious behavior associated with a transaction (eg, purchase, refund, cancellation, etc.) of an item at a transaction terminal by a customer. The system obtains video data associated with the transaction area. The video data can be obtained from an altitude camera, for example, focused on a cash register checkout area or other transaction area in a supermarket or other retail store. The system executes a machine-based automated video analysis algorithm disclosed as part of the system to analyze at least a portion of the video data to obtain at least one video parameter for at least a portion of the transaction associated with the transaction area To do. For example, the system can analyze the video data to track items included in a transaction associated with the transaction area (eg, identify the presence of an item). This process can automatically identify the presence of items included in the transaction based on video data analysis. This can be done, for example, by automatically detecting item behavior in the transaction area and / or by automatically detecting operator behavior in the transaction area. The detection of the presence of an item can detect an event where the item is transported from the attention area of the transaction area and / or an event where the item is brought into the attention area of the transaction area.

  In one configuration, the video transaction parameters are converted into video counts for how many items the video analysis algorithm has identified as processed by the operator in the transaction. For example, an item is processed when an operator moves the item past the transaction area, regardless of whether the item has been scanned or entered. Thus, the video count can detect and count items that have been processed and scanned / input and items that have been processed but not scanned / input.

  In another configuration, the video transaction parameter is a series of detection events generated by one or more detectors that perform analysis of all or part of the video data. The detector is typically an automatic image processing algorithm, which is applied to a region of interest in the video data. For example, the video data may reflect a large portion including the interior of the transaction area, the operator (eg, store clerk and / or customer), the item loading area, the scanning area, and the item loading area. The detector can analyze all or a part of this area, for example the carry-in conveyor belt area of the video data of the point-of-sale information management terminal. An image segmentation and comparison process can be applied to a video data frame for one or more regions of interest to detect the presence of items that have been brought into or transferred from the region of interest. When an item is detected in this way, a detection event can be generated to notify other features such as the presence of the item, detection time, and item size. There can be more than one region of interest, and a series of events can include multiple events from different detectors, or there can be only one detector to make a single set of events into one region of interest Can only be generated. By analyzing the video data in this way, the system can detect the presence of individual items in the transaction based on the video analysis. In one configuration, the system can determine how many items of visual processing have been performed during the entire transaction.

  The system obtains at least one transaction parameter sent from the transaction terminal associated with the transaction area. The predicted transaction parameter in one configuration is a transaction count or other item presence indication obtained based on transaction data generated by a transaction terminal (eg, point-of-sale information management terminal). As each item is scanned or identified, data is sent from the scanner to the transaction terminal processor. The system of the present invention accesses this data on a scan-by-scan basis or as aggregate set data from a database (this operation can be performed in various ways as will be described) and is processed by transactions. Find the presence of multiple items (and identification information in some cases).

  Using the video analysis results in combination with the transaction data, the system can determine if the presence of the item identified in the analysis of the video data matches the presence in the transaction data, and the match If not, identify suspicious behavior. For example, for each of the detectors, the system compares a set of detection events for that detector with at least a portion of the transaction data and indicates in a number of items detected by the detector in a portion of the transaction data. It is possible to identify at least one distinct discrepancy that is different from a large number of items. Thus, transaction data such as a transaction count (eg, scan count) or transaction item identification information represents the presence or number of items scanned (with respect to the entire transaction) and detection event data or video count based on video analysis is It represents the presence (or number) of items that the operator has moved to pass through the transaction area. Many other embodiments are described below, and these embodiments are described in more detail below. .

  Other embodiments of the invention may include any type of computing device, workstation, handheld or laptop computer, POS or transaction terminal, etc., which may include software and / or circuitry (eg, a processor). Configured to perform any or all of the operations of the method of the present invention. The system can include video camera (s) that capture video, or the system can be a stand-alone computer that receives as input one or more POS terminals at one or more retail locations. Receive video data and scan data collected from In other words, a computing device or processor programmed or configured to operate in any way, as described herein, is considered an embodiment of the invention. The system need not include a video camera and POS terminal, but can instead be an off-site computer system operated by a security service provider that receives video data and transaction data.

  Processing can be done in real-time when video data and transaction data are collected to identify just the fraud or suspicious behavior that has just occurred (in addition, processing can be done where the operator is located and the transaction item and The security officer who can examine the receipt and determine whether fraud has occurred can be notified), or, alternatively, the processing can involve video data and transaction data for one or more transactions It can be done at some point after being collected (ie it can be post-processed). When performing post processing, operator identification information can be maintained and tracked, and history or suspicious behavior regarding the operator can be accumulated. During the next analysis of the transaction for the operator (the operator logs in with identification information specific to the POS terminal so that all transactions performed by the operator can be associated with the operator), the system of the present invention The suspicious level assigned to the transaction result can be adjusted to take into account the history of the transaction. Therefore, it is not necessary to set an illegal flag in the first suspicious judgment result, but the second, third, and fourth transaction results over a predetermined time (for example, several hours, several days, etc.) If detected for the same operator, the video associated with these transactions is automatically identified and forwarded and scrutinized by another means such as a human review, and fraudulent behavior is included in the video I can confirm that.

  In one configuration, transaction data from one or more POS terminals at one or more retail locations is processed for a number of transactions and processed as described herein to notify fraudulent or suspicious behavior. When these transactions become flagged, video data relating only to these transactions will be scrutinized using other methods such as a human review and will be described automatically (i.e., manually or machine) It is possible to confirm the illegal behavior or suspicious behavior first identified by the process. Other variations and embodiments are described more fully in the detailed description section below.

  Other embodiments of the present invention include a software program that performs the steps and operations outlined above and disclosed in detail below. One such embodiment includes a computer program product that includes a computer-readable medium, the computer-readable medium including computer program logic encoded on the medium, the computer program logic being interconnected. When executed on a computing device having a memory and a processor, the processor is programmed to cause the processor to perform the operations of the present invention. Such an arrangement is typically provided as software, code, and / or other data (eg, a data structure) that is stored or encoded on a computer-readable medium, such as an optical Media (eg, CD-ROM), floppy disk or hard disk, or other media can be mentioned, such as one or more ROM chips, RAM chips, or firmware stored on a PROM chip Or microcode, or application specific integrated circuit (ASIC). Software or firmware, or other configurations similar to these, may be implemented on a computing device to cause the computing device to perform the methods described herein as embodiments of the invention.

  The system of the present invention can be strictly embodied as a software program, as software or hardware, or as hardware only within a processor or operating system. Illustrative embodiments of the present invention include computer systems, processors, and computer program products, and / or software applications provided by Stoplift, Inc., based in Burlington, Massachusetts, USA. Can be implemented.

  FIG. 1 shows an exemplary environment 300, which is suitable for use in the description of exemplary embodiments of the invention. An exemplary environment 300 depicts a retail store where a customer 305 can purchase an item 307. A transaction terminal 34, such as a point-of-sale information management terminal or cash register, is controlled by an operator 308, such as a store clerk, to allow the customer 305 to purchase an item 307. The transaction terminal 34 includes a scanning device 36 that scans the UPC barcode symbol or RFID tag that is affixed to each item 307 when these items 307 are brought into a predetermined neighborhood of the scanner of the scanning device 36. Can be detected and scanned or read.

  With the environment 300 operating normally, the customer 305 has a series of items 307 to purchase and approaches the transaction area 301. The item 307 can be stored, for example, in a shopping cart 311 or other article transport device that the customer 305 pulls to the transaction area 301. Alternatively, customer 305 can carry individual items 307 to transaction area 301. In order to purchase the item 307, the customer 305 picks up the item 307 from the shopping cart 311 (or from the customer's hand when carrying the item) and collects the items as a region 302-1 in the transaction area 301. Bring it to the displayed item carry-in area. The item carry-in area 302-1 may be a conveyor belt, counter top, or other surface area to which items to be purchased are brought in before being detected and read by the scanner 36 of the transaction terminal 34.

  An operator 38, such as a store clerk, interacts with the transaction terminal 34 by logging into the transaction terminal 34 or activating the transaction terminal. In this process, the operator 308 can present unique operator identification information to the transaction terminal 34. While the operator 308 operates the transaction terminal 34, the body of the operator 308 usually stays inside the operator area 302-4 of the transaction area 301. Once logged in, the operator 308 can begin selecting purchased items 307 in the item carry-in area 302-1 by, for example, picking up individual items 307 by hand. The operator 308 causes each item 307 from the item carry-in area 302-1 to pass through the scanner 36 that is normally located in the item reading area 302-2. Assuming that the operator performs a correct operation (that is, an operation without fraud and no error), the operator 308 determines the position of the item 307 and the scanner 36 detects the item identification information 310 attached to the item. , And be able to scan or read. When the scanner 36 detects the item identification information 310 for the item 307 that was scanned immediately before, the transaction terminal 34 register registers the item 307 as an item to be purchased, and usually appears to the operator 308 as a beep or beep. A notification sound is generated to notify that the item 307 has been successfully identified. In response to the notification sound, the operator 308 moves the item 307 to the item carry-out area 302-3, but the item carry-out area has been successfully scanned or read by the transaction terminal 34, or has been safely taken by the transaction terminal 34. It can be another counter top that holds the item 307 input to the, a downstream conveyor belt, or the like.

  The operator 308 repeats this process for each individual item 307 so that all items 307 to be purchased pass over the item reading area 302-2 from the item carry-in area 302-1 or the item reading area 302-2. Pass through and move (during this time the item is scanned) and reach the item unloading area 302-3. In some cases, item 307 need not include affixed item identification information 310 such as fruit, vegetables, and the like. In such a case, after moving the item 307 out of the item carry-in area 302-1, the operator 308 manually inputs item identification information into the transaction terminal 304 using a keyboard or other manual input device. Therefore, the transaction terminal 34 can register the item 307. In this way, after all items 307 have been identified by transaction terminal 34, operator 308 notifies transaction terminal 34 that the transaction has been completed, and transaction terminal 34 provides the total price of item 307 to be purchased. calculate. Next, the customer 305 pays the corresponding amount to the operator 308, and then proceeds to pick up the item 307 from the item carry-out area 302-3 and carry it out of the retail store.

  As shown in FIG. 1, the environment 300 further includes a transaction monitor 32 configured in accordance with an embodiment of the present invention to detect suspicious behavior associated with a transaction. The environment 300 further includes a video source 30 such as one or more upside video cameras that capture video of the transaction area 301. Typically, the video source 30 is installed at a height that is sufficiently higher than the transaction area 301 to capture various areas 302 and to shoot videos in the various areas 302. The transaction monitor 32 in this example receives as input video data 320 from the video source 30 and further transaction data 34 from the transaction terminal 34. In the exemplary environment 300, the transaction monitor 32 is shown as receiving transaction data 330 and video data 320 directly from the video source 30 and transaction terminal 34, but the transaction monitor 32 may process the item or the entire transaction by the operator 308. It can be appreciated that these inputs can be received in real time or after a certain amount of time has passed. Further, the transaction monitor 32 need not receive transaction data 330 and video data 320 directly from the video source 30 and transaction terminal 34. In another configuration, these inputs may be received from video tape equipment (or from a digital recording medium) or from a transaction database maintained by another computer system beside transaction terminal 34. Thus, the video source 30 can be a real-time source such as a camera or a storage source such as a recording device such as a VCR or DVR. Similarly, the transaction terminal 34 can supply real-time transaction data directly from a POS (eg, a cash accounting terminal or scanner), or the transaction data can be stored data from a transaction log database that stores POS data. be able to.

  While the system of the present invention is in operation, the transaction monitor 32 compares the video data 320 and corresponding transaction data 330 for suspicious behavior associated with the transaction area 301 such as “near deal” or “pass-through”. To identify and report suspicious behavior. Typically, this causes the transaction monitor 32 to collect video data 320 from the transaction area 301 that includes the transaction terminal 34, where the customer 305 purchases the item 307 during the transaction. A video source 30, such as a camera, is installed at a height that is sufficiently higher than the transaction area 301 to allow the video of region 302 to be filmed from above, but the system is limited to such a configuration. Not.

  Generally, the transaction monitor 32 performs automatic (i.e., non-human) video analysis on at least a portion or at least one segment of video data 320 from an upright camera to provide at least one associated with the transaction. The presence of the item 307 is detected. Transaction monitor 32 detects the presence of transaction-related items based on video data (automatically detected by the image processing techniques described herein) that the customer actually purchases at the transaction terminal (ie, terminal 34 reads, Or compared to transaction data 330 indicating items to be scanned 307) that were not purchased at the transaction terminal (ie, not scanned by or entered into transaction terminal 34, or by transaction terminal 34). The item that has passed through the transaction area 301 without being read is specified in the possession of the customer 305. The discrepancy due to the comparison between the presence of one or more items identified by automatically processing video data 320 and the items identified in transaction data 330 is a suspicious behavior that the system of the present invention can detect. Suggest. The suspicious behavior can be the result of an operator error due to the operator 308, or the result of an actual fraudulent behavior including “near deal” or “pass through”.

  Depending on the configuration, the transaction monitor 32 analyzes all or part of the video data obtained by filming the transaction area, eg, the behavior of objects passing through the transaction area, within a particular area of interest within the transaction area. The behavior of an object in a transaction area, the behavior of an object within a plurality of specific areas of interest within a transaction area, the behavior of an object entering a specific area of interest within a transaction area, and / or a specific area of interest within a transaction area Items can be automatically detected based on the behavior of the outgoing object.

  By analyzing all or a portion of the video data, in one configuration, one or more items are detected by at least one detector within at least one region of interest 302 of at least a portion of the video data. A series of detection events to suggest is generated. Depending on the embodiment, the transaction monitor 32 may perform item detection based on video analysis in only one region of interest 302 or in many regions 302. In FIG. 1, the transaction area 301 is divided into several regions 302-1 to 302-N or listed as regions 302-1 to 302-N. Each of these areas or regions can be considered as one region of interest 302, and the video data 320 can indicate that an item is included in the transaction. The behavior in some, all, or only one region can be captured.

  The transaction monitor 32 uses the detector to perform image processing in the region of interest. The detector is usually an image processing algorithm that can detect the presence of an item in that region. The presence of an item can be detected by applying a detection process to the item carry-in area 302, for example. With each detector (one or more detectors), the transaction monitor 32 compares a series of detection events for that detector with at least a portion of the transaction data (ie, the portion containing transaction information that matches the video data). Thus, at least one obvious discrepancy is identified among the multiple items detected by the detector from among the multiple items indicated in the portion of the transaction data. The transaction monitor 32 can recognize the overall suspicious level for the transaction based on the apparent discrepancy identified by the detector (s).

  In some configurations, in video processing or video analysis, the transaction area 301 is divided into a plurality of regions (eg, 302-1 to 302-3) through which the objects are sequentially placed in at least one transaction. Move during the part period. The transaction monitor 32 can perform automatic video detection on an item to obtain a pattern represented by one or more video parameters as the item is moving sequentially through multiple regions. Thus, the pattern represents a video event of items that have moved through the region during the entire period of the transaction or during some period. The transaction monitor 32 can obtain transaction data 330 that identifies items detected by the transaction terminal 34 during a portion of the transaction corresponding to the video data being analyzed, and automatically converts the video parameters as transaction parameters. This comparison indicates that the item representing the video event of the group of items for all or part of the transaction is detected by the transaction terminal 34 during the entire period of the transaction or during the same part of the period. Is determined by determining whether or not the transaction data does not match the specified transaction data. If there is a mismatch, the transaction monitor 32 determines that the transaction result is an illegal transaction.

  By providing different regions, a series of different detection events can be used to confirm that the item is included in the video data. In such a case, the transaction monitor 32 can simultaneously identify a series of detected events by the detector and transaction data to identify inconsistencies among the multiple items processed in the transaction area. For example, in the process of performing automatic video detection on an item, a “remove event” indicating that the operator 308 has transferred the item 307 from the item carry-in area 302-1 is entered in the item carry-in area 302-1. In the process, an “introduction event” indicating that the operator has placed the item in the item carry-out area can be specified in the item carry-out area. Thus, events that occur in sequence, such as transfer, carry-in, transfer, carry-in, etc., can be generated based on video analysis when multiple regions of video data are monitored. This order can be synchronized in time with, for example, transaction data indicating a scan for a group of items so that a regular pattern appears as transfer, scan, carry-in, transfer, scan, carry-in, etc. Suspicious patterns appear as transfer, scan, carry-in, transfer, and carry-in. Here, since the second scan event does not occur, it is suggested that an illegal behavior or a suspicious behavior is lurking. Applying automatic video analysis technology using a machine to detect video events of items related to the transaction area, and further compare this event with transaction data indicating transaction events of items related to the transaction area. Thus, the transaction monitor 32 determines whether the video event is not acknowledged as a transaction event in the transaction data, and in response, a specific segment indicating where a video event that is not acknowledged as a transaction event occurs. 328 can be identified in the video data.

  As shown in FIG. 1, once suspicious behavior is identified, the transaction monitor 32 identifies a particular segment in the video data that indicates where a video event that is not identified as a transaction event is occurring (ie, (In a video clip 328) and then sent to the inspector, who inspects the segment 328 of the video data to review the operator's suspicious behavior regarding item purchases during the transaction.

  In one configuration, the system of the present invention provides a way to actually count items, or item detection events, and is more robust in identifying contingencies that pass through the transaction area without being scanned. And it is an accurate method. The system is immune to scan delays, and the system can determine that more items were part of the transaction than the items scanned, so the system can steal due to the operator. Alternatively, it can function as a very accurate notification means indicating that an error has occurred.

  Another type of error that occurs independently of item detection comparisons is item identification of items. This misidentification is due to fraud, such as “ticket or label replacement”, where the bar code or “discount” bar code of the reduced item is superimposed on the bar code or other item identification information 310. Once pasted, the operator 308 deliberately manually enters a code or item identification information into the transaction terminal 34 regarding the price reduction item. In one configuration, the system of the present invention provides a technique for cropping item images directly from a commonly performed transaction video for comparison. In such a configuration, the transaction monitor 32 performs an image comparison and an image of the item associated with a certain detection event (eg, an event indicating that the item is included in the video data) is already stored. It can be determined whether or not the item image substantially matches. If these images do not nearly match, the transaction monitor 32 may confirm that the transaction may include a label replacement event, suggesting that suspicious behavior is lurking. it can. This approach allows the operator 308 to handle the item 307 normally, so there is no need to change the way transactions are normally performed. The system also does not affect or reduce the transaction processing capacity of the operator. This characteristic of the system is particularly important for experienced operators. The reason is that the operator is evaluated based on the processing speed of the operator. There is no need for the operator to change his or her behavior, so the system can be deployed in the correct position and dishonest employees can be deployed to these employees with some new structure in the correct position. Can be detected without recognizing that

  Described below is a series of flowcharts of various embodiments of the present invention. These embodiments are exemplary embodiments presented to illustrate a given operation of a given configuration and are not exhaustive of all variations of the system.

  FIG. 2 is a flowchart of an operation of one configuration of processing by the transaction monitor 32 that detects a slip-through event in the transaction area. In this configuration example, the transaction monitor 32 acquires video data transmitted from at least one video camera that monitors the transaction area 301. Analyzing the video clip 2 of the video data 320 for at least a portion of one transaction and the corresponding transaction data 8 (of transaction data 330 in FIG. 1) for that transaction, the item 307 included in the transaction in the transaction area 301 is analyzed. Chase. Any time range of video data 320 and corresponding transaction data 330 (ie, part of a transaction or more than one transaction) can be handled, but the description is clear and simple For this reason, in this example, it is assumed that one transaction is handled at a time.

  In one configuration, the presence of an item can be identified using a method that utilizes area differences. In this method using the area difference, the transaction monitor 32 defines a region of interest such as the item carry-in area 302-1 in the transaction area. If the video is for this area, the transaction monitor 32 operates the detector to capture the first frame of video data (i.e., first filmed) indicating the first set of items in the area of interest 302-1. ) Automatically. These items are, for example, the first set of items 307 placed in the area 302-1 for purchase by the customer 305. Thereafter, the transaction monitor 32 automatically identifies the second frame of video data (ie, taken at the second or later time) that represents the second set of items in the region of interest 302, but this If so, the first set of items is visually distinct from the second set of items. This is because items are transported from region 302-1 by operator 308. The transaction monitor 32 automatically detects that the first set of items is visually distinct from the second set of items as an event indicating that the item was in the attention area 302-1 of the video data. Can be notified.

  In step 10, the transaction monitor 32 acquires transaction data related to the transaction terminal 34 connected to the transaction area 301. The transaction data indicates whether or not the item 307 is registered in the transaction terminal 34 as a purchase item.

  In step 14, the transaction monitor 32 compares the video analysis result of the tracked item with transaction data generated by the transaction terminal to identify suspicious behavior. In one configuration, the transaction monitor 32 is transaction data (missing transaction data) that does not exist for the target item when the event indicates that the item exists in the attention area 302. Identify suspicious behavior in case. This method is an event-based comparison that associates each video detection event with a transaction event of transaction data.

  In another configuration of FIG. 2, video clip 2 is analyzed in step 4 to visually detect the presence of an item 307 that is actually included in the transaction (event detection can be used). Transaction monitor 32 analyzes at least a portion of the video data to obtain at least one video parameter for at least a portion of the transaction associated with transaction area 301. Thus, the video parameter can be the video count of the item for which the presence of the item is detected. The transaction monitor 32 also obtains at least one transaction parameter sent from the transaction terminal 34 connected to the transaction area. The transaction monitor 32 analyzes the transaction data 330 in step 10 to obtain a record of the item 307 included in the transaction. Based on these records, a predicted count of items reflected in the transaction data can be obtained. In step 14, the transaction monitor 32 compares the actual count, or video count 6, with the predicted count, scan count, or transaction count 12. If these counts match, the transaction monitor 32 can set a flag at step 16 indicating a “non-suspension” state for the transaction. If these counts do not match, the transaction monitor 32 may set a flag at step 18 indicating a “suspicious (eg, fraudulent) state” in the transaction.

  As explained above, the transaction monitor 32 identifies video counts for items found in the transaction area using video analysis and sets the transaction counts for items in the transaction area to at least a portion of the transaction. Identify by analyzing relevant transaction data. By comparing the video count with the transaction count, if the video count is different from the transaction count, the transaction monitor 32 can notify a transaction result representing a suspicious behavior such as an illegal transaction or an operator error. Depending on the configuration, the transaction monitor 32 can provide additional information, such as suspicious levels, based on an indicator, such as the difference between the actual count and the predicted count.

  In some configurations, other information may be used to rank suspicious levels, such as “low” or “high” or “in range”. For example, in a retail environment, each cashier operating a transaction terminal or cash machine typically logs into the system by entering unique identifying information through his or her register (eg, with a keyboard) and logging for the customer Prove that you are processing a transaction. Once logged in, the system of the present invention can perform the operations described herein, and if a fraudulent transaction is detected (eg, if the transaction count does not match the video count), the system Investigate the past history of this particular cashier, which can be stored in a database that shows how often the transaction marked “suspicious” is being performed (in the unique identification information of the teller Based on). In this way, if a fraudulent transaction occurs very rarely, an event where the transaction count does not match the video count can be a mere error of the cashier (i.e. If the suspicious transaction history of the cashier indicates that suspicious transactions are occurring frequently, events whose transaction count does not match the video count are considered suspicious levels. Can be displayed and this event can be used for further processing (eg, flagging this state so that it can be recognized by the human operator, allowing the operator to examine the video in more detail) ).

  In step 20 of FIG. 2, transaction monitor 32 may assign a suspicious level to video segment 328. The suspicious level indicates a suspicious level generated based on an automatic video analysis result obtained by comparing video data with transaction data. Transaction monitor 32 can adjust the suspicious level associated with the transaction outcome based on a number of factors. For example, the following elements can be mentioned.

  The absolute value of the difference between transaction count and video count. For example, if 10 of 20 detected items are not traded by the operator, the suspicious level is much higher than if only 1 of 20 was not traded.

  The history of the operator processing the transaction indicates that at least one past transaction has been confirmed as suspicious with respect to the operator indicated in the history. This example has been described above.

  The number of regions of interest in which an item has been detected in the video data. For example, if an item that is not traded is visually detected for all areas by all detectors, the suspicious / confidence level is detected for one area by only one detector that is not traded Much higher than would be done.

  The order in which an item is detected in the video data in different areas of interest in the transaction area (eg, an item transfer event followed by an item carry-in event).

  A discrepancy as a result of comparing the video identification information of an item in the video data with the transaction identification information of the item obtained from the transaction data. The transaction identification information may be, for example, an item price and / or item identification information.

  A history of cash registers that execute transactions. This history may indicate a faulty cash register or may indicate a cash register that is prone to suspicious behavior by location or other factors.

  A history of other items identified in the transaction that can indicate items of the transaction that do not require transaction data. For example, if a salad is identified as an item, and then the fork is not scanned (and therefore is initially recognized as suspicious), the fork is free by adjusting and lowering the suspicious level, so there is no need to scan the fork This is reflected in the suspicious level.

  FIG. 3A is a flowchart of processing executed by the transaction monitor 32 in a configuration example in which an identification error item is detected. In this configuration, the video data 40 of one transaction and the corresponding transaction data 46 of the transaction are analyzed. Any time range of video and corresponding transaction data (i.e. at least part of a transaction or more than one transaction) can be handled, but for clarity and simplicity of explanation, In this example, description will be made assuming that one transaction is handled at a time.

  In step 42, the actual item image 44 is cut out from the video data 40. One method of image cropping is described as part of the transfer / carry-in detection method (described below). In step 48, the predicted item image 52 corresponding to the item included in the transaction data is extracted from the item image 50 database. The item image database can be organized in any way, but one convenient method in a retail environment is by SKU number.

  From the viewpoint of the item image database, the system of the present invention can use the image pre-collection database. However, the transaction monitor 32 advances the processing and collects the image-added database while cutting out more images from each continuous transaction. You can also. In this way, the retailer does not have to waste time and money and deploy a database in advance. Instead, the database can basically be trained by taking and storing a video of a sufficiently large number of transactions for the same item.

  In step 54, the actual item image is compared with the predicted item image. Here, the actual item images can be cut out and compared with the corresponding predicted item images of these images at once or as a group. If the transaction data contains only a full list of items and no data on the order or number of times these items were scanned, there is no underlying material from which a priori correspondence between individual images can be obtained. Therefore, it is necessary to compare all sets of actual item images with all sets of predicted item images. However, if sequential transaction data or timing transaction data is available (e.g., the scan time stamp is approximately synchronized with the video time stamp) that allows the scanned item data to be associated with the video data through a synchronization process, the images For example, the actual first item image is compared with the first predicted item image, and the other item images are compared with the corresponding predicted item image in the same manner. In this case, a selection can be made that each individual actual item image is compared with one corresponding predicted item image of the image.

  If it is determined in step 56 that the images do not match, the transaction is flagged in step 58 as suspicious of the transaction. As mentioned above, if the actual item image is compared to the predicted item image one by one, you can choose to flag a particular item as suspicious rather than the entire transaction .

  If in step 60 it is found that the images actually match, the transaction is considered suspicious. The actual item image or image group 44 can be loaded into the item image group database as required (step 50). For example, it is not desirable to store new images in a pre-collected database from a professional point of view, but new images are necessary for a “learning” database. Nevertheless, in the next step 62, the transaction (or the individual item described above) is flagged as “not suspicious”.

  In this manner, the transaction monitor 32 performs image recognition of items detected within the transaction area using automatic video analysis to generate video identification information for the items. In order to perform this operation, the transaction monitor 32 acquires at least one transaction parameter by specifying predicted item identification information of items detected in the transaction area. The transaction monitor 32 then automatically compares the video parameter with the transaction parameter by comparing the item's video identification information with the item's predicted identification information. If the video identification information is different from the predicted identification information, the transaction monitor 32 notifies a transaction result indicating suspicious behavior.

  As described above, items included in a transaction can be counted within a specific region of interest or within a combination of regions of interest 302. A region of interest may include all areas in which transaction items are accommodated. In a normal retail form of a supermarket, for example, the area of interest is a shopping cart 302-N, a customer area 302-5, a carry-in conveyor belt area 302-1 (ie, an object carry-in area or area), a scan area 302-2, and a carry-out conveyor. A belt area 302-3 or a bagging area (ie, an object carry-out area), and an operator area 302-4 can be included.

  The operator can change for each region of interest. For example, in the supermarket form, if the item is counted at the time of transfer from the cart 311 to the carry-in conveyor belt area 302-1, the customer 305 performing the transfer can be considered as an operator. Similarly, if the cashier operator 308 is scanning items at the transaction terminal 34, the cashier 308 can be considered an operator.

  As described above, in a certain configuration, a more reliable counting method is provided by combining the counts performed over these attention regions 302. For example, by comparing the number of detections performed across the carry-in area 302-1 of items brought in in one configuration, the scanner or item reading area 302-2, and the bagging or other item carry-out area 302-3, Investigate whether these times match. If these numbers do not match, the average number of detections can be used. Similarly, the order of counting or detection events can be taken into account. For example, each item that is accurately counted is first counted as an item transfer event in the carry-in area, then counted as a scan event in the scanner, and then again counted as an item carry-in event in the bagging area (each event Or the count is detected and generated by a detector that analyzes the corresponding region of interest in the video data 320).

  In another configuration, the item count is in one area, such as an object transfer event when the operator transfers an object from the belt or object carry-in area 302-1, followed by the operator Observed as an order or pattern of another type of event, such as a carry-in event brought into the object carry-out area 302-3 or downstream conveyor belt. Therefore, the event that the transport event (the operator picks up the object for scanning and transports the object from the object carry-in area) followed by the carry-in event (the operator drops the item to the carry-out belt or the object carry-out area) Observe or interpret as one video count.

  The count is registered only in one stage or area 302 of the sequence (eg, item event detection in area 302-1) but not in the other one or two areas (area 302-3 and / or). Or if it is not detected in 302-2), the single count or event can be considered an error and / or can be labeled "suspicious". For example, if video data is analyzed and scan data or transaction data is analyzed, the system may determine which scan detection event (ie, transaction event) or scan count matches which video count. (E.g., when analyzing two areas, the video count may be performed for an event that an item transfer event is followed by an item load event, or only a single area, such as the item load area 302-1). In the case of analysis, the video count or event may be a single transport event that transports the item 307 from the area 302-1 as detected by video analysis). In such cases, if the scan count or event matches the event's video count on a one-to-one basis, there is no obvious fraudulent behavior and the transaction is not flagged for further investigation, or the notation Do not attach. However, the video data is time-synchronized with the scan data (such time synchronization can be an inherent process in the data collection process when the video data is collected simultaneously with the transaction data or the scan data, or In processing, the scan time stamp can be algorithmically compared with the video frame or pattern of the detection event) followed by the transport event followed by the carry event (for the first object, the item is included in the transaction data) Is invalid for a transaction if a pattern is detected, such as an event followed by another transport event (for the second object) It can be subjected notation that there is potential, or is suspected possible. In one configuration, the transaction monitor 32 associates the video timestamp of the event based on the analysis of the video data with the transaction timestamp of the item represented as being traded in the transaction data, and does not have a corresponding record in the transaction data. Since an event indicating an item of video data is specified, a suspicious behavior can be notified.

  There are a number of methods that can be used individually or in combination with each other, by which video analysis can be performed to place item counts or detection events within one or more regions of interest 302. Can be identified.

  In one such processing method according to the present invention, the items currently included in the attention area are counted. One method of counting items currently included in the attention area counts items that can be observed in the still image of the attention area. Because this operation considers that individual items can be segmented to sufficiently eliminate the ambiguity of the existence of individual items, this approach is especially useful when items are connected or overlapped. It becomes difficult to execute. In a supermarket form or other retail form, for example, a few items that are slowly placed one by one on a moving conveyor belt are actually scattered on the belt, so these items touch each other, or There is no overlap. In such a case, the exact number of items in the transaction can be determined by counting the items from the still image of the belt against the background of the belt. However, if a large number of items are on the belt, the items may begin to stack on each other at the end of the belt. When these items are stacked, segmentation of specific items by video analysis becomes very difficult, and still image analysis yields less reliable results regarding the count of items on the belt.

  In another configuration, the transaction monitor 32 can count periodic behavior. Periodic behavior within the region of interest can indicate the loading, transfer, or slipping of items. Activity measures include simple detection and measurement of motion in a region. In “tripwire: device line: a thin iron wire that automatically explodes when touched by an enemy in relation to explosives such as landmines” (ie, observe the motion along the edge of the region of interest) The direction of entry into the area can be taken into account. In the supermarket form, for example, a device along the customer facing side of the carry-in belt area can be used to count each time a customer comes in and brings a new item into the belt in the item carry-in area 302-1. .

  Two or more trip wires may be installed and used to determine that item motion is occurring across the region of interest on one side and toward the region of interest on the other side. In the vicinity of the scanner area 302-2 of the supermarket checkout register, for example, two trip wires on either side of the scanner area can detect motion in a specific direction that varies depending on the order in which these trip wires are activated.

  Another way to use trip wires is to place a series of trip wires over a large area of interest (perpendicular to the direction of the motion of interest) and detect the progress of motion from one side to the other. In the supermarket form, using such a series of trip wires, the forward direction of the object from the carry-in area to the bagging area even when the object is handed over the scanner area from one hand to another. Progress can be detected.

  If an operator object (ie, the portion of video data 320 that includes operator 308) can be identified from the image, the count is that the operator object itself (eg, his or her arm or hand) enters the area, and Can only be incremented when going out. By using skin detection and / or identifying the limbs of the operator object 308, the detector can detect or count items only when the operator's hand enters the area of interest (eg, 302-1). It can operate more accurately to start. If the edge of an operator object (i.e. his or her hand) has a color other than the object's unique color as part of the object's color histogram, the hand enters or leaves the item in the area of interest. Can be viewed as likely.

  FIG. 3B shows one method for detecting the behavior in the region of interest and notifying the presence of the item. Based on the image from video source 430, step 432 identifies the object and generates an object map. In step 434, association filtering (eg, skin detection, skin + object color histogram, etc.) is performed to confirm that only the object of interest or behavior of interest is being considered. For example, in a video image of a retail store environment, the operator operates the touch screen of the transaction terminal past the scanner area. Since such behavior in the region of interest does not suggest the presence of an item, it can be ignored by removing the case where the operator object is connected to the graphical area of the touch screen in the video image. In step 436, the current object map is taken into the motion map of the time-varying motion in the image. Next, in step 442, the motion map is analyzed to identify the motion in the motion direction (step 438) associated with the region of interest (step 440). If it is determined in step 444 that the motion has completely completed across the area of interest, this motion is recorded in step 446 as a video detection event. If it is determined at step 448 that more video remains, the process proceeds to the next video frame at step 450 and processing continues at step 432. If it is not determined at step 448 that more video remains, a record of video detection events is returned at step 452. For example, if the attention direction is directed from the item carry-in area to the item carry-out area and across the attention area as the scanner area, the transaction monitor 32 indicates the scanner area and the item is included in the transaction. Observe the behavior.

  In yet another configuration, the transaction monitor 32 may count intrusions or exits of objects that may be items or operator objects, or customer objects. In one configuration, an event where an object newly enters or leaves one or more areas of interest 302 suggests item detection or a count change. For example, in a supermarket configuration, when an item is transferred from a carry-on conveyor belt, or when an item is brought into a bagging area (or in both cases where a combination is possible), the event is It suggests that another item is included in the transaction.

  One method of detecting intrusion or exit detects color histogram changes in the region of interest. Yet another method of detecting the entry or exit of another object (eg, item or operator object) detects the appearance or disappearance of the object. Such an image detection algorithm described herein continuously captures still objects into the background image used for object segmentation. When a new object is added to the image, it appears as the only object in the foreground. Similarly, when an object is removed, a “ghost” (that is, the result of a change in the image location that the object once occupied) remains in the part of the object, and this ghost appears in the foreground in the same way. It will be. In either case, the object is subsequently counted and then incorporated into the background and ready for counting the next item.

  One advantage of this method is that it is very easy to cut out item images. When an object such as an item is brought in, an image of the item can be cut out as a cut-out item image. When an item is transferred (ie, a ghost object appears), the image of the item can be clipped from the frame preceding the “transfer”. One very difficult problem with the above type of method is that the arm of the operator itself appears in the region of interest as an object. In the two configuration examples according to the present invention, this problem is dealt with by ignoring the arm of the operator, or only an image having no arm in the area is used.

  In order to ignore the arm object, the object is first identified. This operation checks all of the objects contained in the video data to determine which objects extend from the large operator object 308 located in the operator area 302-4 outside the attention area 302-1 or attention. This can be done by investigating whether it extends into region 302-1. Therefore, it can be estimated that this object is the arm of the operator 308. Skin detection can also be performed to further confirm that the object is actually an arm and / or hand. Next, the arm object is removed from the object map of the region of interest, leaving only the merchandise item 307 being brought in or transported. To use only images with no arms in the photo, a trip wire can be used along the edge of the region of interest closest to the operator 308 to investigate whether any objects are crossing the trip wire. it can. If the object does not cross the trip wire in a particular video frame, this suggests that the arm is not included in the region of interest 302-1 of that frame. Using this logic, it is possible to determine the presence of an item by cutting out the frames before and after the arm intrusion and withdrawal. These frames can then be compared to identify item delivery or item transfer.

  From the above description, the transaction monitor 32 indicates that the motion of the operator 308 within the attention area (eg, 302-1) of the transaction area, suggesting the presence of an item 307 traded (eg, purchased) within the attention area. It can be seen that can be specified. The transaction monitor 32 can notify whether the record of the item is included in the transaction data corresponding to the operation that identifies the operator's motion, and within the attention area 302-1 of the transaction area. Although the operator's motion suggests the presence of the item 307 being traded, it is possible to automatically identify situations where the record of the item is not included in the transaction data. Accordingly, the transaction monitor 32 can notify the suspicious behavior.

  As noted above, each item 307 included in the transaction is utilized using operator modeling such as cashier 308 (or customer 305 when environment 300 of FIG. 1 is self-checkout terminal 34). The motion associated with the handling of can be identified. In a supermarket configuration when looking down from above, for example, the operator can be modeled as a torso (torso statue) with two arms extending from the torso. The operator's torso can be identified as present at the transaction terminal 34 by its location. Thus, the arm can be identified as two limbs that extend from the torso and move quickly. Next, handling of the item 307 extends one hand to the carry-in belt area or the item carry-in area 302-1 and then passes the item from one hand to the other hand in the vicinity of the scanner area or area 302-2. The two hands can then be combined and then modeled as a behavior in which the other hand extends towards the bagging area or item carry-out area 302-3.

  As noted above, a single region of interest can be monitored with a single item counting method, or multiple counting methods can be used in a single region of interest 302 to count items in that region 302. be able to. Similarly, a more accurate count can be performed on the entire transaction using a combination of counts and count sequences obtained from multiple regions of interest.

  FIG. 4 shows a process in which the transaction monitor 32 uses the analysis result of the video data 320 to count the items included in the transaction in one configuration. In one method of counting the presence of an item, an operation in which the operator moves the item from the attention area or an operation in which the operator carries the item into the attention area is counted. For example, whenever an operator releases an item and places it in a region of interest such as a bagging area, it can be safely estimated that the item is a merchandise item carried into the bagging area. On the other hand, one method of counting the transfer of items counts the operations in which items are transferred from the item carry-in area 302-1.

  In FIG. 4, when the video source 70 outputs a video frame, a first step 72 identifies the object in the image. One way to normally do this is to compare the current frame with the background model and extract the foreground element or item in the image. In the next step 74, an operator object (ie, the outline of the operator 308 in the video frame) is cut out. One method for performing this operation uses a priori information regarding the location where the operator 308 is likely to be. For example, if it is known that the operator 308 is standing in a specific area such as the operator area 302-4, it can be estimated that the largest moving object in the area is the operator 308. Another method for identifying and clipping an operator object in video data is that the operator object moves into the bagging area and places another item 307, for example when the operator object is in a region of interest. A state of moving across the boundary edge of 302-3 is specified.

  In the next step 76, a graphical import or transfer event is performed, such as when operator objects and items (or object ghosts as described below) in the region of interest are graphically separated from each other. To check. The transfer event suggests that the operator has picked up an item in the item carry-in area 302-1. If a transfer is made at step 78, the count is incremented at step 80. If more video remains in step 82, processing proceeds to the next video frame in step 84 and the loop continues again from step 72. When the transaction video ends, the count is returned in step 86. When processing for each item is performed, each time an object is transferred from the item carry-in area 302-1 (or an item is carried into the item carry-out area 302-3), a corresponding transfer (or carry-in) event is generated. Events can be associated with transaction data (or event pairs can be associated if multiple regions of interest are used).

  FIG. 5 illustrates one method of processing that can be performed by the transaction monitor 32, which enables detection of object transfer events and object carry-in events. In FIG. 5, the current image 90 and the updated background image 92 are captured as inputs by step 94, and these images are compared (by subtraction processing and threshold processing) to generate a binary object map 96. This object map can include operator objects in the image that are not part of the updated background, as well as any new objects (ie, items) that are not part of the updated background. In the next step 98, the current operator object 100 is cut out from the object map. In a supermarket configuration in which the cashier operator 308 stands in the compartment space (eg, area 302-4) in front of the cash register, the cutout of the operator object is the largest movement that stands in that space in one configuration. (In another configuration, the operator object can be cut out by finding an object that overlaps a small “tag” area that the operator may touch). This object is considered to be an operator object.

  In step 104, the region of interest 106 is directly defined using the current operator object 100 and the previous operator object 102. In the area where the previous operator object does not overlap, the direct attention area is configured for the following reason. If an item is held by the operator 308 in the previous video data frame, the item was a part of the previous operator object. Therefore, if the item has been released in the current frame, the item remains somewhere in the area of the previous operator object. Also, since the item has been released, the item is not part of the current operator object. Therefore, if a certain item is released after the previous frame, the object of the item appears in a certain area of the previous operator object, and the area does not overlap with the current operator object. Similarly, if an item was picked up in the previous frame, the ghost of that item in this frame (ie, the missing item where the item was removed from the background layer) was explained above. Appear in the same direct attention area.

  In the next step 110, the object map 96 is checked to determine whether a new object (ie, item) or an object (ie, item) ghost appeared directly in the area of interest 106. If not, in the next step 120, all areas outside the operator object are taken into the background update and used for the next frame. Finally, return a value that does not cause transfer / loading.

  If a new object actually appears directly in the region of interest in step 110, in the next step 112, the object (ie, item) image is cut out, that is, the image is copied from the current frame. If object transfer (cutting out a ghost portion) is detected, the image is taken from the frame before the previous frame when the item was picked up.

  In the next step 114, the cut-out image of the actual item is compared with a database made up of predicted object groups (for example, a database obtained by adding a photograph of these items to the item group). This is part of the item identification error detection method described above. Next, the object image is taken into the background update and used for the next frame. Again, in the next step 116, all the areas outside the operator object, including the transfer object or the carry-in object, are taken into the background update and used for the next frame. Finally, a value that does not cause a transfer (or carry-in) is returned.

  FIG. 11 shows an example of a video data frame 501 and a video data frame 502, which show how the transaction area 301 appears before event detection (video data frame 501) and after event detection (frame 502). Show. The detection event in this example happens to be a carry-in event, in which the operator 308 places an item 307 (milk jug in this example). Each frame 501 and 502 is divided into four quadrants: upper right, upper left, lower right, and lower left. The upper left quadrant of each frame indicates the original frame of the video data. In this case, the lower left quadrant indicates the updated background image generated by the above-described processing, as described above. The upper right quadrant of each frame 501 and 502 shows the operator object 308, and the lower right quadrant shows the difference binary image or map.

  In the upper left quadrant of the frame 501, it can be seen that the operator 308 has placed the item in the item carry-out area 302-3, but the operator's hand has not yet been released from the item 307. In the corresponding quadrant of the frame 502, the operator 308 has released the item 307 at that time by carrying it out to the item carrying-out area 302-3. Here, unlike the upper right quadrant of the frame 502 containing the operator object 308, the operator object 308 in the upper right quadrant of the frame 501 showing how the operator object is considered to contain items in the operator's hands; This is different from the item object 307 separated at the time of the frame 502. By the video analysis described above, this difference can be detected in the image group, and this behavior can be notified as an item detection event, in this case, an item carry-in event, and the item is included in the transaction. It becomes clear. In this way, video analysis can detect the items included in the transaction, and, as explained above, the transaction monitor 32 uses this information for comparison with the transaction data from the transaction terminal 34. Thus, it can be verified that each item detected in the video data has a corresponding entry (eg, scan, read, item identification information, or price) in the transaction data. If this corresponding transaction data does not exist for this object, the transaction monitor 32 can notify the suspicious behavior.

  Here, if the region of interest includes a moving background, such as a supermarket conveyor belt, a method for adapting the background image is described in detail below in the description of moving background adaptation. The system may incorporate shopping basket bottom detection or shopping basket internal detection to identify items stored, for example, under or in a movable shopping cart from a high elevation view.

  As illustrated above, the transaction monitor 32, in one configuration, defines at least one region of interest in the video data, and updates the current image of the region of interest (group) and the region of interest (group). At least a portion of the video data can be analyzed by calculating an object map that identifies changes between the finished background image. Accordingly, by cutting out at least one operator object inside the attention area (s), the transaction monitor 32 analyzes the object map and the operator object, thereby transferring the object from the attention area 302 or to the attention area 302. It is possible to detect whether it is possible to specify the import of the object. In this way, a detection event indicating a count of the presence of objects in the video data can be maintained. This operation can be performed on a single region or many regions 302, and can be performed on an item-by-item basis, for the majority of a transaction, or for an entire transaction (ie, all items).

  In the process of analyzing at least a portion of the video data, the system needs to adjust the auto gain characteristics that are installed in some cameras or other video sources. In auto gain, the brightness of the three color channels of the camera is balanced across the entire image. This causes an undesirable color shift when noisy objects (ie, objects that appear in the scene but do not appear in the transaction or part of the transaction area) appear in the camera's field of view. The system needs to suppress the color shift as much as possible. A very difficult problem to solve is that the system produces the same results as those obtained by adjusting the auto gain characteristics of the camera, so it cannot perform statistical calculations across the entire image. is there. Instead, the system should focus the system on the transaction area and place a relatively high specific gravity on objects in the transaction area that remain stationary but exhibit color or intensity shifts. One way in which this goal can be achieved is as follows. Compare the current image with the current updated background image (eg, by subtraction and thresholding) to obtain a noisy foreground / background pixel binary map. Next, the color statistics of the current image and the background image are collected by calculation using only the background pixels. These background pixels are used because they correspond to the same stationary object in the shooting area and indicate a color shift that needs to be corrected. Once the statistical calculation of the pixel labeled background has been made, changes are made to the current image to match the color of that image to the color of the background image, so the rest of the image being processed by the system To match. Once this pre-processing step is performed, normal video processing steps proceed. In this way, prior to video analysis, the transaction monitor 32 can adjust the auto gain characteristics working on the video source.

  In a configuration in which the attention area is monitored for both item transfer and entry, the transaction monitor 32 defines the object carry-in area 302-1 as the first attention area, and the object carry-out area 302-3 as the second attention area. Can be defined. In this configuration, in the detection of the object transfer event and / or the object carry-in event, the operator detects an object transfer event when the object is transferred from the object carry-in area, and / or the operator places the object in the object carry-out area. An object carry-in event can be detected. By detecting a carry-in event following a transfer event, in one configuration, the transaction monitor 32 can increment the video count as at least one video transaction parameter, or in another configuration, the transaction monitor 32 can detect a transfer event. And import events can be generated, and these events are then processed during comparison with the transaction data to identify items that are not included in the transaction data but appear in the video data.

  In one configuration, at least one of the plurality of regions of interest includes a conveyor belt such that the position of the group of objects is captured in the current video data frame and the previous video data frame by the conveyor belt. , Move on the conveyor belt in time. For example, the item carry-in area when the transfer detection event is generated as the operation of transferring the item 307 by the operator can be a conveyor belt that conveys the item 307 toward the operator. In such cases, the transaction monitor 32 can correct for the movement or translation of the items as they move from one position to the next in a continuous video frame. Specifically, the transaction monitor 32 can analyze at least a portion of the video data and correlate the previous video data frame and the current video data frame within the region of interest to move the position of the objects on the conveyor belt. The amount can be determined. When the transaction monitor 32 detects an object transfer event and / or an object carry-in event, the transaction monitor 32 can grasp the movement amount of the position of the object group on the conveyor belt.

  FIG. 8 is a flowchart of processing operations that can be performed by the transaction monitor 32. By executing the processing operations, an area comparison method used by the detector when performing video analysis can be provided.

  When video source 200 generates a video frame, video source (200) outputs the video frame. In the first step 202, an object group is specified in the frame image. In one method for performing this operation, the current frame is compared with the background model to extract foreground elements or objects in the image.

  In the next step 204, an operator object is cut out. One method for performing this operation uses a priori information about the location where the operator is likely to be. For example, when it is known that the operator is standing at a specific section position, it can be estimated that the moving object having the largest position is the operator. In another method for performing this operation, when the operator moves to the bagging area and places another item, an object moving across the boundary edge of the region of interest is identified.

  In the next step 206, the system checks whether the operator object extends into the region of interest. If at the next step 208 the operator object is located within the region of interest, the system checks at step 210 whether the operator was also included in the region of interest of the previous frame. If the operator is not included in the attention area of the previous frame, the previous frame is stored as key frame 1 in step 212. Therefore, the key frame 1 is an image of the attention area before the operator object enters the attention area. When comparing with an image of a region of interest after the operator object has moved out of the region of interest, the system can determine whether a new object has been brought into or transferred from the region of interest . Regardless of the determination at step 210, the system then proceeds to step 224 to check if more video remains.

  If it is determined in step 208 that the operator object is not included in the region of interest, it is checked in step 214 whether the key frame 1 has already been saved. If key frame 1 has not already been saved, the event suggests that the operator object has not yet entered the region of interest at the time of the previous frame. Since the operator object has not yet entered the attention area of the current frame, there is no need to analyze the attention area for new object loading / transfer by the operator. The system then proceeds to step 224 to check if more video remains.

  If the determination in step 214 is yes, yes indicates that the operator object has already entered the attention area before and has not yet left the attention area at the time of the last frame. Therefore, when the operator object is no longer included in the attention area, the operator object has just left the attention area. Accordingly, the current frame is saved as key frame 2 at step 216.

In step 218, the object areas of key frame 1 and key frame 2 are compared. This process is described in further detail below.
If it is determined at step 220 that the actual large change (eg, greater than the threshold amount) was between keyframe 1 and keyframe 2 in the object area, step 222 increments the item count or detection event Is generated.

  Regardless of the determination at step 220, the system proceeds to step 223, where the system resets (or deletes) keyframes 1 and 2 so that the operator object is no longer within the ROI (region of interest). Notify that you can think.

  The system then proceeds to step 24 to check if there are more videos remaining. Returns item count if there are no videos left. If more video remains, step 228 advances the video to the next frame and the system resumes the loop from step 202.

  FIG. 9 is a flowchart of processing steps performed by the transaction monitor 32 to enable keyframe area comparison. This figure shows one exemplary method, in which two key frames are compared to detect a change in the object area within the region of interest and to notify the object carry-in or object transfer.

  In this example, key frame 1 (step 250) is a video frame before the operator object enters the region of interest. Key frame 2 (step 254) is a video frame after the operator object has left the region of interest. The empty base image (step 252) is an image of a scene that does not include an object in the attention area. The attention area (step 259) is a binary map that emphasizes which area of the image is to be noted.

Steps 256, 258, 260, and 262 describe processing related to the key frame 1.
In step 256, the key frame 1 and the empty base image are compared (by subtraction processing and threshold processing) to generate a binary object map (step 258). This object map can include all new objects in the image that are not part of an empty base image that includes operator objects and other objects in the region of interest.

  In step 260, an object map (step 258) is taken in the object in the attention area, and the map is cut out by masking the map with the attention area (step 259) so that only the object in the attention area remains. One way to perform masking is by performing a logical AND operation on the binary object map and the binary region of interest map. The newly obtained object map is the object map of key frame 1 (step 262).

  In steps 264, 266, 268, and 270, an object map of key frame 2 (step 270) is generated from key frame 2 (step 254) by a similar method.

  In step 272, the amount of movement between the key frame 1 and the key frame 2 within the attention area is obtained. Such operations occurred by using standard techniques such as correlation, sum of squares of differences, flow analysis, etc. to overlap the region of interest and occurred between keyframe 1 and keyframe 2 The amount of movement can be determined. In step 272, a movement amount 274 is generated.

  The amount of movement 274 is applied at step 276 to superimpose the object map 270 of key frame 2 with the object map of key frame 1. An object map 278 of the superimposed key frame 2 is generated.

  In step 280, the area of the object map 262 of the key frame 1 is compared with the area of the object map 278 of the key frame 2. The area of each of these object maps can be calculated as the sum of the binary object maps.

  If there is a sufficiently large change in the area in step 282 (ie, exceeds the threshold), Yes is returned in step 284 indicating that the area has changed. If the area does not change, No indicating that the area has not changed is returned in step 286.

  FIG. 10 is a flowchart of the processing steps described for the processing by the transaction monitor 32. By the processing by the transaction monitor 32, a slip-through item detection method is provided. A slip-through can be detected by comparing the predicted item count and the actual item count over the period of the time window of all transactions, so that the period is such that a comparison can be made approximately item by item. As long as the time window is relatively short.

  During a time window that includes the duration of each individual transaction item entry (ie, scan, key entry, RFID read, etc.), it is expected that the item will be processed through only one transaction. If the visual count is greater than 1, the event suggests that suspicious behavior has occurred. Similarly, a corresponding POS item entry needs to be made during a time window that includes the duration of each individual visual item detection. If no POS entry is made, this means that the actually processed item has not been predicted according to the data, thus suggesting suspicious behavior (eg, potential slip-through). This embodiment will be described in further detail below.

  Item detection process 402 captures video from video source 400 and proceeds until it detects the next item included in the transaction. This operation can be performed by any one of various methods, and these methods include, but are not limited to, carry-in / transfer detection, comparison by area difference, scanner motion detection, and the like. Thereby, an item detection time 404 is generated.

  Next, at step 410, the system retrieves transaction item time data (step 408) from the transaction data source (step 406) and compares the data to the item detection time (step 410). This comparison indicates whether the transaction item time matches the item detection time.

  In step 412, the comparison result is checked. If the transaction time matches the item detection time, the predicted behavior and the actual behavior match, and the current visually detected item is flagged as “not suspicious” (step 414). If the transaction time does not match the item detection time, the predicted and actual behavior do not match, and the current visually detected item is flagged as “suspicious” and entered into the suspicious item log (step 416). .

  One way in which a match can be determined is whether the transaction item time that has not yet been matched is included in a time sufficiently close to item detection (eg, below a certain threshold or a dynamic threshold) It is done by judging. If the transaction item time is included in a time sufficiently close to item detection, the detection can be matched to the transaction time. If the transaction item time is not included in a time sufficiently close to item detection, the item detection is considered inconsistent. Another method keeps track of the total number of transaction items, not just the total number of items detected. If the detected item is greater than a trade item by a certain threshold number (eg, if there is only one item), the last detected item is considered not to match, and the record is Reset until it occurs. In another method, as shown in the example of FIG. 12, it is investigated whether visual item detection is performed more than once within the same item transaction time interval (inter-transaction-item interval). If more than one number of visual item detections are made within the same item transaction time interval, at least one of these number of item detections should not match a transaction item time. . In such a case, the item transaction time interval itself can be considered suspicious.

  If more video remains at step 418, the system proceeds to step 402 to resume the loop. If no video remains, the system proceeds to step 420 and returns a log of suspicious items.

In this way, the system provides a slip-through item detection method that can identify “pass-through” as suspicious behavior.
Other configurations include the ability to utilize customer and / or employee presence and tracking. Such a configuration can use a perspective shot of a “shoulder type” camera in addition to a simple looking down shot of the camera. In the case where the cashier is installing the camera behind the cash register and the camera is shooting the bill, cash register, and customer over his or her shoulder, the method By identifying an object that obstructs the bill of accounts, the employee can be identified as an object on the side close to the bill of accounts. In contrast, the customer object appears beyond the bill of accounts, but is identified as a broken object by the top edge of the bill of accounts. In one configuration, once a customer or employee is identified, his or her location is tracked and labeled by the system.

  FIG. 6 is a flowchart illustrating the processing steps used by the transaction monitor 32 in one configuration, which allows for customer and / or employee presence detection and tracking. In step 254, the input image 250 and the background base image 252 are compared to generate all object binary maps showing all non-background objects in the image. One exemplary method of making such a comparison is to “reset” the noise binary map by performing subtraction processing followed by threshold processing, followed by morphological operations. Next, in a next step 256, employee and / or customer object segmentation is described in the next section describing employee and / or customer object segmentation for all object binary maps and in FIG. To be done.

  If an employee object exists at step 268, the employee object is labeled at step 270. If the customer object does not exist at step 258, the process proceeds to the next video frame at step 284. Otherwise, if the same customer object is included in the previous frame in step 260, then in step 262, it is determined whether the customer object is stationary in time. If the customer object is stationary in time, the customer object may actually be an inanimate object such as a shopping cart. In that case, the process resets the customer flag at step 280 and then incorporates the stationary object into the background base image at step 282. If the customer object is not stationary in time, it is assumed that the object is actually a human and the object is labeled customer in step 266. The process then proceeds to the next video frame at step 284, whether or not it is stationary.

  If at step 260 the same customer object was not included in the previous frame, the customer object is tracked at step 272. The tracking is performed by tracking the shift of the center of mass of the target object group. If the customer object is located close to the bill base area (eg, 302-1, 302-2, 302-3), that is, standing at the bill stand position, a stationary object such as a cart exists. Determine whether or not. If a stationary object exists, the process proceeds to step 282 as previously described. If no stationary object exists, a customer presence flag is raised at step 278 and the customer is labeled at step 266.

  FIG. 7 is a flowchart of processing steps showing the processing performed by the transaction monitor 32, which enables segmentation of employee / customer objects.

  In step 204, the input image 200 and the background base image 202 are compared to generate all object binary maps that represent all non-background objects in the image. One exemplary method of making such a comparison is to “reset” the noise binary map by performing subtraction processing followed by threshold processing, followed by morphological operations.

  In step 206, the binary object map is checked to see if it contains any objects. If the object is not included, in step 208 the video is advanced to the next frame and processing is continued again from the beginning of the loop. If the object group is included in the binary object map in step 206, a labeling operation is performed on these objects in step 210. Next, in step 212, the first labeled object is selected. If in step 214 it is found that the object is only included in employee focus area 302-4, the object is added in step 218 to the employee-only binary map. The employee attention area 302-4 is usually on the front side of the bill of accounts, where only employees or other operators (in a position facing the customer or items on the bill of accounts) appear.

  If in step 216 it is found that the object is only included in the customer attention area 302-5, the object is added in step 228 to the customer only binary map. The customer attention area 302-5 is usually located immediately above the top edge of the top of the bill in the image, and appears at this position if the customer is standing at the bill.

  Next, if it is determined in step 220 that the object is included in the bill of interest area (eg, 302-1, 302-2, or 302-3) (upper bill), the object is an item on the bill. There is a possibility, and the object is added in step 222 to the binary map of the bill only. In step 224, the object appears in both the employee attention area 302-4 and one or more of the account base attention areas 302-1, 302-2, and 302-3, but the customer attention area 302-5. If not, the object is determined to be an employee or operator and is added to the employee-only binary map at step 218.

  At step 226, the object appears in both the customer attention area 302-5 and one or more of the bill-base attention areas 302-1, 302-2, or 302-3, but the employee attention area 302-4. If not, the object is determined to be a customer and the object is added to the customer-only binary map at step 228.

  In step 230, the object is one of all three attention areas (one or more of employee attention area 302-4, customer attention area 302-5, and bill attention area 302-1, 302-2, and 302-3). The object may be a combined object in which the employee image partially overlaps with the customer image. Accordingly, the object is added to the employee and customer binary map at step 232.

  After the object has been processed, in step 234 the object is removed from the full object binary map so that the next labeled object can be processed.

  If the object remains in the all object binary map at step 206, processing continues at step 210 for the next object. Otherwise, if all objects in the image have been processed, processing proceeds to the next video frame at step 208.

  In this way, the system can identify the operator object or customer object and clearly distinguish this object from the item object in the video analysis.

  Although the system and method have been shown and described in detail with reference to system and method configurations, those skilled in the art will recognize various changes in the form and details of the system and method in the accompanying patents. It can be understood that additions can be made without departing from the technical scope of the present invention defined by the claims. Therefore, the present invention is not limited by the configuration example presented above.

1 illustrates an example configuration of a network environment, the network environment including a video surveillance system and a computer system configured with a transaction monitor configured in accordance with the present disclosure. It is a flowchart which shows operation | movement of one structure regarding the process by the transaction monitor which detects the slipping-in behavior in a transaction area. It is a flowchart of the process which a transaction monitor performs in the structural example in which a misidentification item is detected. One method for detecting the behavior in the region of interest and notifying the presence of the item is shown. In one configuration, the transaction monitor shows a process of counting items included in a transaction using a video data analysis result. One method of processing that can be performed by the transaction monitor is shown, which allows detection of object transfer events and import events. FIG. 6 is a flow chart illustrating processing steps that a transaction monitor can use in one configuration to detect and track the presence of customers and / or employees. It is a flowchart of the processing step which shows the process which the transaction monitor 32 performs in order to implement | achieve employee and / or customer object classification. FIG. 6 is a flowchart of processing operations that a transaction monitor can perform to provide an area comparison method that a detector uses when performing video analysis. FIG. 6 is a flowchart of processing steps performed by a transaction monitor to enable keyframe area comparison. It is a flowchart of the process step explaining the process by a transaction monitor, and the passing-through item detection method is provided by the process by a transaction monitor. An example of a frame group of video data is shown, and these frames show how a transaction area appears before an event is detected and after an event is detected. FIG. 4 is a timeline diagram of events illustrating how a transaction manager can determine whether more than one visual item detection occurs within the same transaction-item time interval.

Claims (32)

A method for detecting suspicious behavior,
Obtaining video data transmitted from at least one video camera monitoring the transaction area;
Analyzing the video data to track items included in a transaction in the transaction area , the analyzing automatically analyzing video frames from at least one region of interest in at least a portion of the video data. Tracking, including identifying each event indicative of the presence of an item associated with the transaction ,
Comparing the video analysis result of the tracked item with transaction data generated at a transaction terminal to identify suspicious behavior , the comparing comprising: generating an event generated by analyzing the video data; and the transaction data Analyzing the set of items identified as being traded in the transaction, wherein the item represented by the at least one event generated by the analysis of the video data is at least one of the transaction data Determining whether a transaction item in a portion is indicated as a transaction item, and analyzing the video data for a video timestamp of the event represented as having been traded in the transaction data By associating the transaction time stamp, it includes identifying an event indicating an item in the video data with no record corresponding to the inside of said transaction data, the method comprising to the specific. Analyzing the video data to track items included in the transaction includes automatically identifying the presence of an item included in the transaction;
Comparing the video analysis result of the tracked item with transaction data determines whether the presence of the item identified in the analysis of the video data matches the corresponding presence in the transaction data and does not match The method of claim 1, comprising identifying suspicious behavior in the case. Automatically identifying the presence of an item included in the transaction,
Detecting the behavior of items in the transaction area; and
Detecting operator behavior in the transaction area;
The method of claim 2, comprising at least one of: Automatically identifying the presence of an item included in the transaction,
Detecting the transfer of items from the attention area of the transaction area; and
Detecting the delivery of an item to the attention area of the transaction area;
The method of claim 2, comprising at least one of: A method for determining a transaction result,
Obtaining video data related to the transaction area,
Analyzing at least a portion of the video data to obtain at least one video parameter for at least a portion of a transaction associated with the transaction area , wherein the analyzing comprises at least one of at least a portion of the video data. Automatically obtaining video frames from one region of interest to identify each event indicative of the presence of an item associated with the transaction, said obtaining
Obtaining at least one transaction parameter sent from a transaction terminal associated with the transaction area , wherein the obtaining comprises analyzing at least a portion of transaction data from the transaction terminal and transacting in the transaction. Identifying, including identifying a series of items identified as being
Automatically comparing the at least one video parameter with the at least one transaction parameter to confirm a transaction result , the comparing comprising generating an event generated by analyzing the video data and the transaction data; Analyzing the series of items identified as being traded in the transaction, and the item represented by the at least one event generated by the analysis of the video data is Determining whether at least a portion is indicated as a trade item, and analyzing the video data as a video time stamp of the event as being traded in the transaction data By associating the transaction time stamp of the item to be I, it includes identifying an event indicating an item in the video data with no record corresponding to the inside of said transaction data, the method comprising to the confirmation. Obtaining the video data includes obtaining video data sent from at least one high-profile video camera that monitors the transaction area;
Analyzing at least a portion of the video data includes analyzing the video data to track items included in a transaction in the transaction area;
The automatic comparison of the at least one video parameter with the at least one transaction parameter to confirm a transaction result compares the video analysis result of the tracked item with the transaction data generated at the transaction terminal, 6. The method of claim 5, comprising identifying Analyzing at least a portion of the video data includes generating, by at least one detector, a set of detection events indicating detection of an item within at least one region of interest in at least a portion of the video data;
Automatically comparing the at least one video parameter with the at least one transaction parameter;
Each detector compares the set of detection events for the detector with at least a portion of the transaction data and is indicated in at least a portion of the transaction data among a plurality of items detected by the detector. Identify at least one obvious discrepancy that differs from multiple items;
6. The method of claim 5, comprising identifying an overall suspicious level for the transaction based on an apparent discrepancy identified by the at least one detector. Each detector is a video processing algorithm applied to a region of interest of the video data in the transaction area;
Comparing the set of detection events includes comparing at least one set of detection events from at least one detector with at least a portion of the transaction data to determine a plurality of items to be processed in the transaction area. 8. The method of claim 7, comprising identifying a discrepancy in. Analyzing at least a portion of the video data includes detecting video events of items associated with the transaction area by a machine-based automated video analysis technique;
Obtaining at least one video parameter for a transaction performed in the transaction area includes identifying transaction data indicative of a transaction event of an item associated with the transaction area;
The automatic comparison of the at least one video parameter with the at least one transaction parameter determines whether the video event is identified as a transaction event of the transaction data, and according to the determination, the transaction event 6. The method of claim 5, comprising identifying a specific portion of video data indicating where a video event not specified as occurring occurs.   Identifying the specific portion of the video data includes sending the specific portion of the video data indicating where a video event not specified as the transaction event occurs to an inspector inspecting the segment of the video data. 10. The method of claim 9, comprising scrutinizing an operator's suspicious behavior regarding trading of items during the transaction period.   The automatic comparison of the at least one video parameter with the at least one transaction parameter may include determining a suspicious level generated by performing an automatic video analysis of the video data on the transaction data. The method of claim 10, comprising assigning to: Analyzing at least a portion of the video data comprises:
Defining the area of interest within the transaction area,
Automatically identifying a first video data frame representing a first set of items in the region of interest;
Automatically identifying a second set of items in the region of interest that are second video data frames representing the second set of items visually different from the first set of items;
Automatically notifying the visual difference of the first set of items with respect to the second set of items as an event indicating that an item was present in a region of interest of the video data;
Acquiring the at least one transaction parameter is transaction data associated with a transaction terminal associated with the transaction area, the transaction data indicating whether the item is registered in the transaction terminal as a transaction item. Including
The automatic comparison of the at least one video parameter with the at least one transaction parameter is missing the transaction data for an item corresponding to an event indicating that the item was present in the region of interest. 6. The method of claim 5, comprising identifying suspicious behavior in the case. Analyzing at least a portion of the video data comprises:
Defining the area of interest within the transaction area,
Automatically identifying a first set of items in the region of interest for a first time;
Automatically identifying a second set of items in the region of interest, the second set of items being visually different from the first set of items for a second time;
Automatically notifying the visual difference of the first set of items with respect to the second set of items as an event indicating that an item was present in a region of interest of the video data;
Acquiring the at least one transaction parameter is transaction data associated with a transaction terminal associated with the transaction area, the transaction data indicating whether the item is registered in the transaction terminal as a transaction item. Including
The automatic comparison of the at least one video parameter with the at least one transaction parameter is missing the transaction data for an item corresponding to an event indicating that the item was present in the region of interest. 6. The method of claim 5, comprising identifying suspicious behavior in the case. Analyzing at least a portion of the video data comprises performing a video analysis of the video taken of the transaction area to automatically detect and generate a video count for at least one of the next plurality of items. Including the plurality of items,
The behavior of objects passing through the transaction area,
The behavior of the object within a particular area of interest within the transaction area,
The behavior of the object within a plurality of specific areas of interest within the transaction area;
The behavior of an object that enters a particular area of interest within the transaction area,
The behavior of an object leaving a particular area of interest within the transaction area,
The method of claim 5 comprising: Acquiring the video data includes extracting the video data including an indication that an item is being traded within the transaction area from at least one region of interest in a transaction area that is approximately the same height as the scanner area of the transaction area. Including obtaining,
Performing a video analysis of the video taken of the transaction area to automatically detect and generate a count of the behavior of the object,
Detecting a behavior associated with an object including at least one of a customer, an operator, and an item in the transaction area, the behavior indicating the presence of an item traded in the transaction area; and
After detecting the behavior, maintaining a record of the presence of the detected item in the transaction area;
15. The method of claim 14, comprising at least one of: Analyzing at least a portion of the video data includes performing image recognition of an item detected within the transaction area using automatic video analysis to generate video identification information for the item;
Obtaining the at least one transaction parameter includes identifying predicted item identification information for items detected within the transaction area;
The automatic comparison of the at least one video parameter with the at least one transaction parameter compares the video identification information of the item with the predicted identification information of the item, and the video identification information is different from the predicted identification information. 6. The method of claim 5, comprising notifying a transaction result representing suspicious behavior. Automatically comparing the at least one video parameter with the at least one transaction parameter;
Detecting a transaction result indicative of fraudulent behavior associated with the transaction area; and
Detecting a transaction result indicative of an operator error associated with the transaction area;
6. The method of claim 5, comprising at least one of: Analyzing at least a portion of the video data includes identifying an operator motion indicative of the presence of an item being traded within an area of interest in the transaction area;
Obtaining the at least one transaction parameter comprises notifying whether a record of the item is included in transaction data corresponding to identifying the motion of the operator;
The automatic comparison of the at least one video parameter with the at least one transaction parameter indicates that an operator motion within a region of interest in the transaction area indicates the presence of an item to be traded. 6. The method of claim 5, comprising identifying that the record of the item is not included in the data and notifying the suspicious behavior according to the identification. Analyzing at least a portion of the video data includes identifying video counts of items detected within the transaction area using video analysis;
Obtaining at least one transaction parameter for at least a portion of a transaction performed in the transaction area identifies a transaction count of an item in the transaction area by analyzing transaction data associated with at least a portion of the transaction. Including
The automatic comparison of the at least one video parameter with the at least one transaction parameter compares the video count with the transaction count and, if the video count is different from the transaction count, out of fraudulent transactions and operator errors. 6. The method of claim 5, comprising notifying a transaction result representing at least one of the following. Automatically comparing the at least one video parameter with the at least one transaction parameter includes adjusting a suspicious level associated with the transaction result based on at least one of the following items: The multiple items are
i) the number of differences between the transaction count and the video count;
ii) a history of an operator processing the transaction, the history indicating that at least one previous transaction has been identified as suspicious with respect to the operator indicated in the history;
iii) the number of regions of interest in which items are detected in the video data;
iv) the order in which items in the video data are detected in different attention areas in the transaction area;
v) mismatch when comparing the video identification information of the item in the video data with the transaction identification information of the item obtained from the transaction data, wherein the transaction identification information is the price of the item and the identification information of the item A discrepancy when at least one of
vi) the history of the terminal executing the transaction;
vii) a history of other items identified in the transaction that suggests items in the transaction that do not require the transaction data;
viii) a set of items included in the transaction data;
20. The method of claim 19 , comprising: Analyzing at least a portion of the video data comprises:
Detecting the transfer of items from a first region of interest of the video data;
Detecting the transfer of the item to the second region of interest of the video data after detecting the transfer of the item from the first region of interest;
Obtaining the at least one transaction parameter comprises:
Obtaining transaction data generated by the transaction terminal;
The automatic comparison of the at least one video parameter with the at least one transaction parameter is a comparison with the transaction data whether the item detected to be transferred and loaded is present in the transaction data as a transaction item. The method according to claim 5, further comprising: notifying of a suspicious behavior related to an item when the item is identified and the item does not exist. Analyzing at least a portion of the video data includes incrementing a video count to notify the presence of an item associated with the transaction in response to the detection of the transfer of the item and the detection of the import of the item;
The method further includes:
Repetitively detecting the transfer of the item, detecting the arrival of the item, and incrementing the video count until the transaction ends, and the video count at the end of the transaction is determined based on the analysis of the video data. It represents the total number of items identified in the transaction, according to claim 2 1 the method according. Analyzing at least a portion of the video data comprises:
Defining at least one region of interest in the video data;
Calculating an object map that identifies a change between a current image and an updated background image of the at least one region of interest;
Cutting out at least one operator object in the at least one region of interest;
Detecting whether at least one of transfer and carry-in of the object for the at least one region of interest is specified by analysis of the object map and the operator object;
The method of claim 5 comprising: Analyzing the video data includes performing image adjustment processing on the video data to correct auto gain characteristics over the entire area of the video data,
The image adjustment processing, the applied is limited to a portion of the region of interest that are captured in the video data, according to claim 2 3 A method according. Applying the image adjustment process,
Wherein a object of the transaction area, remains stationary, and including limiting the focus to the object indicating at least one of a color shift and brightness shift, claims 2 to 4 The method according. Defining the at least one region of interest includes
Defining the object carry-in area as the first region of interest;
Defining the object carry-out area as a second region of interest,
Detecting at least one of a transfer event and a carry-in event of the object,
Detecting an object transfer event when an operator transfers an object from the object carry-in area;
Detecting an object carry-in event when an operator places an object in the object carry-out area;
In response to detecting an incoming event following the transfer event, incrementing the video count as at least one video transaction parameter;
Including, claim 2 3 A method according. At least one of the at least one region of interest includes a conveyor belt, and the position of objects on the conveyor belt captured in the current video data frame and the previous video data frame is over time by the conveyor belt. Move and
Analyzing at least a portion of the video data comprises:
Associating the previous video data frame with the current video data frame in the region of interest to determine the amount of movement of the position of the object group on the conveyor belt;
Grasping the amount of movement of the position of the object group on the conveyor belt when detecting at least one of an object transfer event and an object carry-in event;
Including, claims 2 to 4 The method according. Analyzing at least a portion of the video data comprises:
Determining whether an image of an item related to at least one of a carry-in event and a transfer event substantially matches an image of an already stored item by performing an image comparison;
Identifying the transaction as potentially containing a label replacement event indicating one of potential suspicious behavior and potential operator error if the images do not substantially match;
Including, claim 2 3 A method according. Analyzing at least a portion of the video data comprises:
Dividing the transaction area into a plurality of areas in which an object group sequentially moves during a period of at least a part of the transaction;
Performing automatic video detection of an item as the item moves sequentially through the plurality of regions to obtain a pattern represented by the at least one video parameter, wherein the pattern is at least a portion of the transaction. Obtaining the pattern representing a video event of an item moving through a plurality of areas,
Obtaining the at least one transaction parameter comprises:
Obtaining transaction data identifying items detected by the transaction terminal during a period of at least a portion of the transaction;
The automatic comparison of the at least one video parameter with the at least one transaction parameter means that the pattern representing an item's video event in at least a portion of the transaction is a period of at least a portion of the transaction. Determining whether the item detected by the transaction terminal indicates a mismatch with the transaction data that identifies the item, and identifying the transaction result as a suspicious transaction if the pattern indicates a mismatch, The method of claim 5. Performing automatic video detection of the item
Identifying a transfer event in an item carry-in area that indicates that an operator has transferred an item from the item carry-in area;
Identifying a carry-in event in an item carry-out area, wherein the operator indicates that the carry-in event indicates that an item has been placed in the item carry-out area;
30. The method of claim 29 , comprising: A method of identifying suspicious transaction activity in a retail store,
Collecting aerial video data from a transaction area that includes a transaction terminal, where the customer trades items during a transaction,
Performing automatic video analysis on at least a portion of the aerial video data to detect the presence of at least one item associated with the transaction, wherein performing the automatic video analysis comprises: Said detecting comprising automatically analyzing video frames from at least one region of interest in at least a portion of said information to identify each event indicative of the presence of an item associated with the transaction ;
By comparing the presence of at least one item associated with the transaction, obtained based on the video data, with transaction data indicating an item that the customer actually trades at the transaction terminal, the customer possesses and the transaction Identifying items that were not traded on the terminal , wherein the comparing is based on comparing the event generated by the analysis of the video data with the transaction data, and is being traded in the transaction Analyzing a series of identified items, an item represented by at least one event generated by analyzing the video data is indicated as a trade item in at least a portion of the transaction data. Determining whether or not, and associating a video timestamp of an event from the analysis of the video data with a transaction timestamp of an item represented as being traded in the transaction data, Identifying, including identifying an event indicative of an item in the video data that does not have a corresponding record therein ,
A method comprising: A system for identifying suspicious transactions in retail stores,
At least one aerial video source that operates to shoot video data in a transaction area;
At least one transaction terminal that operates to obtain transaction data from the item when the item is placed in the vicinity of the transaction terminal during the transaction;
The video data transmitted from the at least one high-camera video camera is acquired, the video data is automatically analyzed to track items included in the transaction in the transaction area, and the video analysis result of the tracked item is obtained. A transaction monitor that operates to identify suspicious behavior associated with the item as compared to transaction data generated by a transaction terminal;
The transaction monitor
Analyzing a series of items identified as being traded in the transaction by comparing the transaction data with the events generated by the analysis of the video data;
Determining whether an item represented by at least one event generated by analysis of the video data is indicated as a trade item in at least a portion of the transaction data;
A video that does not have a corresponding record in the transaction data by associating a video timestamp of the event from the analysis of the video data with a transaction timestamp of an item represented as being traded in the transaction data A system that operates to identify events that indicate items in the data .

Images