JP2018526945A - Video identification and analysis recognition system - Google Patents

Abstract

The analysis recognition system includes a video camera, an antenna, and a data analysis module. The video camera is configured to capture video data. The antenna is configured to capture mobile communication device data. The data analysis module is configured to correlate the video data and the mobile communication device data and generate a person profile associated with the video data and the mobile communication device data. The profile has profile data that includes any one or combination of captured video data and captured mobile communication data.

Description

(Cross-reference of related applications)
This application claims priority to and benefit from US patent application Ser. No. 14 / 817,871, filed Aug. 4, 2015. This application is also related to US patent application Ser. No. 14 / 256,385, filed Apr. 18, 2014, which includes US Provisional Application No. 61 / filed on Apr. 19, 2013. No. 813,942, U.S. Patent Application No. 14 / 213,548 filed March 14, 2014, and claims its benefit, which was filed on March 15, 2013. Claiming priority and benefit to US Provisional Application No. 61 / 798,740. The disclosure of each of these prior applications is hereby incorporated by reference as if fully set forth herein.

  The following relates to video viewing, monitoring and verification systems and methods of use. Specific applications may work with surveillance systems, street cameras, personal videos, in-store camera systems, parking camera systems, etc., providing real-time and / or post-time data analysis of one or more video streams. Configured as follows.

  Companies continue to try to identify specific user behaviors in order to improve their throughput and efficiency. For example, by understanding user behavior in the retail industry context, companies can both improve product sales and reduce product shrinkage. Focusing on the latter, employee theft is one of the biggest causes of retail inventory reduction. Thus, companies are trying to understand user behavior to reduce and ultimately eliminate inventory shrinkage.

  Companies use a variety of methods to prevent employee shrinkage. Passive electronics devices attached to items that are prone to theft in retail stores are used to trigger alarms, but customers and / or employees may disable these devices before taking items away from the store . Some retailers perform bag and / or cart inspections for both customers and employees, while other retailers have implemented and implemented loss prevention systems that incorporate video surveillance of POS transactions. Identifies transactions that may have been executed in violation of the procedure. Most procedures and techniques focus on identifying individual events instead of understanding the potential user behavior that occurs during these events. Thus, companies are unable to deal with potential conditions that allow individuals to commit theft.

  Surveillance systems, street camera systems, store camera systems, parking lot camera systems, and the like are widely used. In some cases, the camera video is continuously streamed, for example, a buffer period of 8, 12, 24, 48 hours is used, and then overwriting should not have to occur for the video. In other systems, longer time periods may be utilized, or a buffer is weeks or months of data stored and stored for a particular purpose. As can be appreciated, when an event occurs, the video is available for review and analysis of the video data. In some instances, the video stream captures data and analyzes various pre-determined scenarios based on automatic, user input, or programming, depending on the particular purpose. For example, the video can be programmed to track objects moving from the entrance into the store and throughout the store for inventory control and / or customer video surveillance.

  In other cases, police, FBI, and rescue workers are identified for investigation purposes, for example, to track suspects, for car accident investigations, or for other video evidence required for the investigation. It is necessary to scrutinize the various camera systems in the area or active area. In most cases, video fragments from various camera systems throughout the area can be important in splicing the visual maps of the event together. In other scenarios, an individual's habits or behavior may become suspicious and may require monitoring or tracking for real-time analysis and alert and / or post-time research analysis.

  There is a need to further develop this analysis technique and provide real-time and post-time analysis of video streams for security and research and marketing purposes.

  According to one aspect of the present disclosure, an analysis recognition system is provided. The analysis recognition system includes a video camera, an antenna, and a data analysis module. The video camera is configured to capture video data. The antenna is configured to capture mobile communication device data. The data analysis module is configured to correlate the video data and the mobile communication device data and generate a person profile associated with the video data and the mobile communication device data. The profile has profile data that includes any one or combination of captured video data and captured mobile communication data.

  In any one of the foregoing aspects, the data analysis module is configured to determine the arrival time of the person at the location and the location based on any one or a combination of the video data and the mobile communication device data. Determining any one or combination of person departure times and correlating video data and mobile communication device data based on any one or combination of arrival time and departure time Consists of.

  In any one of the aforementioned aspects, the video camera is one of a plurality of video cameras included in the system and configured to capture a plurality of video data, wherein the antenna is , One of a plurality of antennas included in the system and configured to capture a plurality of mobile communication device data. The data analysis module is further configured to correlate the plurality of video data and the plurality of mobile communication device data to generate a plurality of profiles of the plurality of persons that are respectively associated with the plurality of video data and the plurality of mobile communication device data. Can be done.

  According to another aspect of the present disclosure, the plurality of video cameras and the plurality of antennas are located at a plurality of facilities.

  In another exemplary aspect of the specification, the analysis and recognition system further includes a user interface configured to allow a plurality of profiles to be mined based on user input criteria.

  According to still further aspects of the specification, at least one of the plurality of antennas is attached to at least one of the plurality of video cameras, and at least one of the plurality of antennas is from the plurality of cameras. Located remotely.

  In any one of the foregoing aspects, the profile includes captured video data, captured mobile communication device data, captured video data or temporal data associated with captured mobile communication device data, and capture. Or any combination of location data associated with captured video data or captured mobile communication device data. The captured video data can include any one or combination of captured still images and video images. The mobile communication device data includes a WiFi identifier, a medium access control (MAC) identifier, a Bluetooth (registered trademark) identifier, a cellular identifier, a short-range wireless communication identifier, and a radio frequency identifier associated with the mobile communication device that communicates with the antenna. Any one of or a combination thereof. The temporal data can include any one or a combination of the time at which video data is captured and the time at which mobile communication device data is captured. The location data may include any one or a combination of where video data is captured and where mobile communication device data is captured.

  In any one of the foregoing aspects, the data analysis module further includes, based on the correlated video data and mobile communication device data, a profile that indicates the number of visits to the facility and the person's visit to the facility. It is configured to add any one or a combination of visit frequencies.

  In any one of the aforementioned aspects, the data analysis module further includes data from the first source, captured video data, captured mobile communication device data, correlated video data, and mobile communication device. Configured to identify the person based on the comparison between the data and any one or combination of profiles, the first source being a non-government database, a government database, and one or Any one or a combination of previously generated profiles above that is included.

  In any one of the aforementioned aspects, the analysis and recognition system further receives survey criteria, and based on the criteria, any one of video data, mobile communication device data, and profile data or their Includes a survey module configured to mine the combination.

  In a further aspect of the specification, the survey criteria includes a time frame, and the survey module is further configured to generate a list of persons whose presence was detected at a facility that is in the time frame.

  According to another aspect of the present disclosure, the survey module is further configured to determine the current location of the person by detecting a signal that matches the mobile communication device data obtained at the first location.

  In any one of the aforementioned aspects, the antenna is a WiFi antenna, a medium access control (MAC) antenna, a Bluetooth antenna, a cellular antenna, a near field communication antenna, and a radio frequency identification antenna. Including any one or a combination thereof.

  In any one of the aforementioned aspects, the data analysis module is further based on any one or combination of video data, mobile communication device data, profile data, and user entered criteria, The person is configured to be assigned to a positive list, an undecided list, or a negative list.

  According to another aspect of the present disclosure, the data analysis module further includes any one or combination of video data, mobile communication device data, and profiles on the list of employees or pre-determined persons. A person is configured to be assigned to a positive list based on a determination that it corresponds to a person.

  In any one of the aforementioned aspects, the antenna is configured to capture mobile communication device data by receiving data wirelessly from mobile communication devices located within the antenna.

  In any one of the aforementioned aspects, the data analysis module is further configured to detect the behavior of the person and store behavior data corresponding to the behavior in the profile.

  According to another aspect of the present disclosure, the data analysis module is configured to detect a person's behavior by extracting behavior information from any one or combination of video data and mobile communication device data. Is done. Behaviors include looking at a direction, reaching for a product item, purchasing a product item, traveling along a route at a facility, visiting a passage or place at a facility, and a certain amount of time at a facility. Any one or a combination of spending, spending an amount of time at a location at the facility, and visiting the facility over several separate cases.

  In a further aspect of the specification, the data analysis module classifies the person as a new customer or repeat customer at the facility based on the facility visit data stored in the profile, and the person is a new customer at the facility, or An indicator for being a repeat customer is configured to be added to or updated in the profile.

  According to another aspect of the present specification, the data analysis module may be configured to map any one or a combination of video data, mobile communication device data, and profile data to a passage location in a facility, a merchandise location in the facility. And the person's behavior by correlating with any one or a combination of the mapping of the shelves and the shelf location at the facility.

  In a further aspect of the specification, the analysis recognition system is further configured to mine any one or combination of video data, mobile communication device data, profile data, and sales data based on the reporting criteria. Consists of a survey module. The survey module is further configured to generate a report based on the mining of any one or combination of video data, mobile communication device data, and profile data. The report includes a sales close rate corresponding to any one or combination of merchandise items, merchandise categories, aisles at the facility, shelves at the facility, or pre-determined locations at the facility.

  According to another aspect of the specification, the data analysis module further generates location data corresponding to the behavior based on any one or a combination of the video data and the mobile communication device data, and the behavior data Configured to store location data in the profile.

FIG. 1 is a system block diagram of an embodiment of a video viewing, monitoring and verification system according to this disclosure.

FIG. 2 is a data / video / image sequencer according to an embodiment of the present disclosure.

FIG. 3 is an illustration of an image map and associated timeline generated by the sequencer of FIG.

FIG. 4 is a schematic illustration of an analysis recognition system used for object identification and tracking according to another embodiment of the present disclosure.

FIG. 5 is a schematic illustration of an analysis recognition system used for collective tracking according to another embodiment of the present disclosure.

FIG. 6 is a schematic illustration of an analysis recognition system used for person characteristic recognition according to another embodiment of the present disclosure.

FIG. 7 is a schematic illustration of an analysis recognition system used for a community monitoring network according to another embodiment of the present disclosure.

FIG. 8 is a screenshot of an embodiment of a survey module displaying a survey according to the present disclosure.

FIG. 9 is a flowchart of an analysis recognition method according to an exemplary embodiment of the present disclosure.

(Definition)
The following definitions are applicable throughout this disclosure (including above).

  A “video camera” may refer to a device for visual recording. Examples of video cameras include video imagers and lens devices, video cameras, digital video cameras, color cameras, monochrome cameras, cameras, camcorders, PC cameras, webcams, infrared (IR) video cameras, low intensity video cameras, thermal video It may include one or more of cameras, closed circuit television (CCTV) cameras, pan / tilt / zoom (PTZ) cameras, and video sensing devices. The video camera may be positioned to perform observation of the area of interest.

  “Video” may refer to a motion picture obtained from a video camera, represented in analog and / or digital form. Examples of videos include television, video, image sequence from a video camera or other observer, image sequence from a live feed, computer generated image sequence, image sequence from a computer graphics engine, computer readable medium, digital video disc (DVD) or an image sequence from a storage device such as a high resolution disk (HDD), an image sequence from an IEEE 1394 based interface, an image sequence from a video digitizer, or an image sequence from a network.

  “Video data” is the visual portion of the video.

  “Non-video data” is non-visual information extracted from video data.

  A “video sequence” may refer to a selected portion of video data and / or non-video data.

  “Video processing” may refer to any manipulation and / or analysis of video data, including, for example, compression, editing, and implementation of algorithms that generate non-video data from video.

  A “frame” may refer to a particular image or other discrete unit within a video.

  A “computer” receives one or more structured inputs and processes one or more devices and / or devices that can process the structured inputs according to defined rules and produce the results of the processing as outputs. Or it may refer to one or more systems. Computer embodiments include computers, stationary and / or portable computers, computers having a single processor, multiple processors, or multi-core processors that can operate in parallel and / or non-parallel, general purpose computers, supercomputers, mainframes , Super minicomputer, minicomputer, workstation, microcomputer, server, client, interactive television, web appliance, telecommunications device with internet access, hybrid combination of computer and interactive television, portable computer, tablet personal computer (PC) ), Personal digital assistant 123 (PDA), portable telephone, eg digital signal processor (DSP), field Application specific to emulate a computer and / or software such as a programmable gate array (FPGA), application specific integrated circuit (ASIC), application specific instruction set processor (ASIP), chip, multiple chips, or chipset Hardware, system-on-chip (SoC) or multiprocessor system-on-chip (MPSoC), optical computer, quantum computer, biocomputer, and data can be received according to one or more stored software programs It can process and produce results, and can typically include devices that can include input, output, storage, arithmetic, logic, and control units.

  “Software” may refer to rules defined for operating a computer. Examples of software may include software, code segments, instructions, applets, precompiled code, compiled code, interpreted code, computer programs, and programmed logic. In this description, the terms “software” and “code” may be applicable to software, firmware, or a combination of software and hardware.

  “Computer-readable medium” may refer to any storage device used to store data accessible by a computer. Examples of computer readable media include magnetic hard disks, floppy disks, optical disks such as CD-ROMs and DVDs, magnetic tape, flash removable memory, memory chips, and / or others that can store machine readable instructions thereon. Types of media may be included. “Non-transitory” computer-readable media includes any computer-readable media, the only exception being temporarily propagated signals.

  A “computer system” can refer to a system having one or more computers, and each computer can include a computer-readable medium embodying software for operating the computer. Embodiments of a computer system are connected together via a network to transmit and / or receive information between computer systems, a distributed computer system for processing information via a computer system linked by a network 2 One or more computer systems, and can receive data, process data according to one or more stored software programs, generate results, typically input, output, storage, It may include one or more devices and / or one or more systems that may include arithmetic, logic, and control units.

  A “network” can refer to a number of computers and associated devices that can be connected by a communication facility. The network may involve permanent connections such as cables or temporary connections such as those made over telephones or other communication links. The network may further include wired connections (eg, coaxial cables, twisted pairs, optical fibers, waveguides, etc.) and / or wireless connections (eg, radio frequency waveforms, free space optical waveforms, acoustic waveforms, etc.). Examples of networks may include the Internet, such as the Internet, an intranet, a local area network (LAN), a wide area network (WAN), and a combination of networks such as the Internet and an intranet. Exemplary networks include Internet Protocol (IP), Asynchronous Transfer Mode (ATM), and / or Synchronous Optical Network (SONET), User Datagram Protocol (UDP), IEEE 802. It can operate using any of several protocols such as x.

  “Real-time” analysis or analytics generally processes real-time or “live” video and is based on the behavior of elements (customers, patrons, crowds, etc.) in one or more video streams. Programmed conditions), anomalous scenarios (such as trapping, concentration, clothing or backpack separation, briefcases, unusual time groceries, etc.), or providing near-instantaneous reports or warnings of other scenarios .

  “Post-time” analysis or analytics is generally stored or stored from a camera source (from a specific camera system (eg, store, parking, on the street) or other video data (cell phone, home movie, etc.)) Based on the behavior of elements (customers, patrons, crowds, etc.) in one or more stored video streams that process the video, abnormal conditions (pre-programmed conditions), abnormal scenarios (徘徊, Concentration, clothing or backpack separation, briefcase, unusual time groceries, etc.), or other scenario reporting or warning.

“Mobile communication device data” generally refers to data transmitted and / or obtained from a mobile communication device using a wireless or wired communication protocol.
(Detailed explanation)

  Particular embodiments of the present disclosure are described hereinbelow with reference to the accompanying drawings, however, the disclosed embodiments may be embodied in various forms. I want you to understand that it is only. Well-known functions or constructions are not described in detail to avoid obscuring the present disclosure in unnecessary detail. Accordingly, the specific structural and functional details disclosed herein are not to be construed as limiting, but merely as the basis for a claim and virtually appropriately detailed by those skilled in the art. It should be construed as a representative basis for teaching various uses of the present disclosure in construction. In this description and in the drawings, like reference numbers indicate elements that can perform the same, similar, or equivalent functions.

  In addition, the present disclosure may be described herein in terms of functional block components, code enumeration, optional selection, page display, and various processing steps. It should be understood that such functional blocks may be implemented by any number of hardware and / or software components that are configured to perform the defined functions. For example, the present disclosure describes various integrated circuit components, such as memory elements, processing elements, logic elements, lookups, that may perform various functions under the control of one or more microprocessors or other control devices. Tables and the like may be employed.

  Similarly, the software elements of the present disclosure can be implemented using any programming or scripting language such as C, C ++, C #, Java, COBOL, assembler, PERL, Python, PHP, or the like, The algorithms may be implemented using any combination of data structures, objects, processes, routines, or other programming elements. The object code created may be on various operating systems including, but not limited to Windows®, Macintosh OSX®, iOS®, Linux®, and / or Android®. Can be executed in

  Furthermore, it should be noted that this disclosure may employ any number of conventional techniques for data transmission, signaling, data processing, network control, and the like. It is to be understood that the specific implementations shown and described herein are illustrative of the present disclosure and its best mode and are not intended to limit the scope of the present disclosure in any other way. Examples are presented herein that may include sample data items (eg, names, dates, etc.) that are intended as examples and should not be construed as limiting. Indeed, for the sake of brevity, conventional data networking, application development, and other functional aspects of the system (and components of the individual operational components of the system) may not be described in detail herein. is there. Further, the connecting lines shown in the various figures contained herein are intended to represent exemplary functional relationships and / or physical or virtual connections between various elements. It should be noted that many alternative or additional functional relationships, or physical or virtual connections, may exist within a practical electronic data communication system.

  As will be appreciated by one skilled in the art, the present disclosure may be embodied as a method, data processing system, device for data processing, and / or a computer program product. Thus, the present disclosure may take the form of an entirely software embodiment, an entirely hardware embodiment or an embodiment combining both software and hardware aspects. Further, the present disclosure may take the form of a computer program product on a computer readable medium having computer readable program code means embodied in the storage medium. Any suitable computer readable medium may be utilized including hard disks, CD-ROMs, DVD-ROMs, optical storage devices, magnetic storage devices, semiconductor storage devices (eg, USB thumb drives), and / or the like.

  In the discussion contained herein, the terms “user interface element” and / or “button” are understood to be non-limiting, including but not limited to hyperlinks, clickable images, and the like. Including other user interface elements.

  The present disclosure is described below with reference to block diagrams and flowchart illustrations of methods, apparatus (eg, systems), and computer program products according to various aspects of the disclosure. It should be understood that each functional block in the block diagrams and flowchart illustrations, and combinations of functional blocks in the block diagrams and flowchart illustrations, respectively, can be implemented by computer program instructions. These computer program instructions generate machines such that instructions executed on a computer or other programmable data processing device generate means for implementing the functions specified in the flowchart block or blocks. To be loaded onto a general purpose computer, special purpose computer, mobile device, or other programmable data processing device.

  These computer program instructions may also be computer or other such that the instructions stored in the computer readable memory include an instruction means that implements the functions defined in the flowchart block or blocks. It can be stored in a computer readable memory that can instruct the programmable data processing device to function in a particular manner. A computer program instruction also includes a series of operational steps such that instructions executed on a computer or other programmable device provide steps for implementing the functions defined in the flowchart block or blocks. Or it can be loaded on a computer or other programmable data processing device for execution on another programmable device to produce a computer-implemented process.

  Therefore, the functional blocks shown in the block diagrams and flowcharts are composed of a combination of means for performing a specified function, a combination of steps for performing the specified function, and a program instruction for performing the specified function. Support means. In addition, each functional block in the block diagram and the flowchart illustration, and a combination of the functional blocks in the block diagram and the flowchart illustration are suitable for a dedicated hardware-based computer system or a dedicated hardware and computer instruction for performing a specified function or step. It should be understood that it can be implemented by any combination.

  One of ordinary skill in the art may also note that for security reasons, any database, system, or component of the present disclosure may consist of any combination of databases or components in a single location or multiple locations, It will be understood to include any of a variety of suitable security features such as firewalls, access codes, encryption, decryption, compression, decompression, and / or the like.

  The scope of the present disclosure should be determined by the appended claims and their legal equivalents, rather than by the examples given herein. For example, the steps recited in any method claim may be performed in any order and is not limited to the order presented in the claims. Further, unless specifically stated herein as “important” or “essential”, no element is essential to the practice of this disclosure.

  With reference to FIG. 1, an analysis recognition system including video viewing, monitoring and verification is shown as 100 according to an embodiment of the present disclosure. The system 100 includes the ability to record video from one or more cameras 110 (eg, analog and / or IP cameras) and other data obtained using one or more antennas 150. Network video and data recorder. Video camera 110 connects to computer 120 across connection 130. Connection 130 may be an analog connection that provides video to computer 120, may be a digital connection that provides a network connection between video camera 110 and computer 120, or connection 130 includes an analog connection and a digital connection. obtain.

  Each video camera 110 connects to a computer 120 and a user interface 122 and provides a user connection to the computer 120. One or more video cameras 110 may each connect through individual connections, may connect through a common network connection, or any combination thereof.

  One or more antennas 150 may be attached to or included in one or more video cameras 110 or computers 120 and / or one or more video cameras 110 and computers 120. Can be located remotely. One or more antennas 150 may be communicatively coupled to computer 120 using connection 130 or may communicate wirelessly with computer 120 using the antenna of computer 120.

  One or more antennas 150 may be any one of various types of antennas or a combination thereof. Exemplary types of one or more antennas 150 are WiFi antennas, medium access control (MAC) antennas, Bluetooth antennas, cellular antennas, near field communication antennas, radio frequency identification (RFID) antennas, and Includes a Global Positioning System (GPS) antenna. It should be understood that the exemplary arrangement of antennas 150 shown in FIG. 1 is provided for purposes of illustration only, and other configurations of antennas 150 are envisioned. For example, a single camera 110 may include multiple different types of antennas.

  As will be discussed in more detail herein, one or more antennas 150 are positioned within the range of one or more antennas 150 according to various exemplary embodiments herein. It is configured to capture mobile communication device data from or above a mobile communication device (eg, a smartphone) and transmit the captured mobile communication device data to the data analysis module 140 for processing. The antenna 150 may be configured to capture mobile communication device data by wirelessly receiving data transmitted by mobile communication devices located within the range of the antenna. The antenna 150 listens and captures data transmitted by neighboring mobile communication devices periodically or continuously by pinging the mobile communication device and / or without using ping. By being configured, it can be configured to receive data wirelessly from nearby mobile communication devices.

  The system 100 includes at least one data analysis module 140. Data analysis module 140 may reside within computer 120 and / or within one or more of video cameras 110. The data analysis module 140 performs processing of video and / or mobile communication device data. For example, the data analysis module 140 implements one or more algorithms to generate non-video data from video and / or from mobile communication data. Non-video data includes, for example, an object identified in the frame, one or more properties of the object identified in the frame, and one or more properties associated with a predefined portion of the frame, etc. , Including non-video frame data describing the content of individual frames. Non-video data may also include non-video temporal data that describes temporal content between two or more frames. Non-video time data may be generated from video and / or non-video frame data. Non-video time data includes time data such as the time properties of objects identified in two or more frames and the time properties of one or more predefined portions of two or more frames. Including. Non-video frame data includes a count of identified objects (eg, an object may include a person and / or any portion thereof, an inanimate object, an animal, a vehicle, or a user-defined and / or developed object) and 1 One or more object properties (eg, the position of the object, the position of any part of the object, the dimensional property of the object, the dimensional property of the part of the object and / or the identified feature) and the related properties (eg, the second Position of the first object relative to the other object), or any other object that can be identified within the frame. An object may be identified as an object appearing in the video or removed from the video. An object can be identified as a virtual object that does not actually appear in the video, but can be added for research purposes, training purposes, or other purposes.

  In various exemplary embodiments herein, data analysis module 140 correlates video data and mobile communication device data captured by a video camera and antenna, respectively, and is associated with the video data and mobile communication device data. Configured to generate a profile. Profiles are profile data such as captured video data, captured mobile communication data, and / or other types of data associated with a person (eg, name, date of birth, residential address, and / or the like). Can be included.

  Profiles include captured video data, captured mobile communication device data, time data associated with captured video or mobile communication device data, and / or location data associated with captured video or mobile communication device data. May be included. The captured video data may include captured still images and / or captured video images. Mobile communication device data includes a WiFi identifier, a medium access control (MAC) identifier, a Bluetooth identifier, a cellular identifier, a near field communication identifier, and a radio frequency identifier associated with the mobile communication device that communicates with the antenna, and / Or any other identifier or data may be included. The time data may include a time at which corresponding video data is captured and / or a time at which corresponding mobile communication device data is captured. The location data may include a location where video data is captured and / or a location where mobile communication device data is captured.

  The data analysis module 140 may be configured to add to the profile the number of person visits and / or the frequency of person visits to the facility based on the correlated video data and mobile communication device data. . The data analysis module 140 also captures captured video data, data captured from a first source (eg, non-government databases, government databases, and one or more previously generated profiles). Mobile communication device data, correlated video and mobile communication device data, and / or profiles may be configured and configured to identify a person based on the comparison.

  Data analysis module 140 may also be configured to determine the arrival time and / or departure time of a person at a particular facility or location based on the captured video and / or mobile communication device data. Data analysis module 140 may correlate video data and mobile communication device data based on arrival time and / or departure time. This time-based correlation may, for example, allow the data analysis module 140 to associate a particular item of mobile communication device data (eg, a WiFi identifier) with a particular person captured on the video.

  In one example, video camera 110 may be one of a plurality of video cameras 110 included in system 100, each video camera 110 being configured to capture multiple sets of video data. obtain. Similarly, antenna 150 may be one of multiple antennas 150 included in the system, and multiple antennas 150 may each be configured to capture multiple sets of mobile communication device data. . The data analysis module 140 may also be configured to correlate multiple sets of video data and mobile communication device data and generate separate profiles for multiple persons associated with the individual video data and mobile communication device data. Video cameras and antennas may be located at a number of different locations and / or facilities.

  In another example, the data analysis module 140 is responsive to video data, mobile communication device data, profile data, and / or user input criteria (eg, input via the survey module 800 described below). Based on, the person may be configured to be assigned to a positive list, an undecided list, or a negative list. The data analysis module 140 may also be configured to determine that the video data, mobile communication device data, and / or profile correspond to an employee or a person on a list of predetermined persons. Based on this, people can be assigned to a positive list.

  In some exemplary embodiments herein, the data analysis module 140 may be configured to detect a person's behavior and store behavior data corresponding to the behavior in a profile. The data analysis module 140 may be configured to detect a person's behavior, for example, by extracting behavior information from video data and / or mobile communication device data. Behaviors look at a specific direction, reach for an item, purchase an item, proceed along a route at a facility, visit a passage or place at a facility, spend a certain amount of time at a facility, It may include a person who spends a certain amount of time at the location and / or who visits the facility over several separate cases. The data analysis module 140 may be further configured to classify the person as a new customer or repeat customer at the facility based on the facility visit data stored in the profile. Data analysis module 140 may add or update in the profile an indicator of whether the person is a new customer or a repeat customer at the facility. The data analysis module 140 may, in some cases, convert video data, mobile communication device data, and / or profile data to aisle location mapping, facility location mapping, and / or shelf location mapping at the facility. Can be configured to detect the behavior of the person.

  In some exemplary aspects herein, the data analysis module 140 generates location data corresponding to a particular behavior based on the captured video and / or mobile communication device data, and the corresponding behavior data and Relatedly, it may be configured to store location data in the profile.

  The data analysis module 140 may be located in the camera 110, convert video to video data and non-video data, and provide the video data and non-video data to the computer 120 via a network. Thus, the system 100 distributes video processing at the edge of the network, thereby minimizing the amount of processing required to be performed by the computer 120.

  Computer 120 includes a computer readable medium comprising software for monitoring user behavior, which when executed by computer 120 causes computer 120 to perform operations. User interface 122 provides an interface to computer 120. User interface 122 may connect directly to computer 120 or indirectly to computer 120 through a user network. The system user interface 122 may also be configured to receive one or more criteria from the user and may allow the profile to be mined based on the criteria. In some exemplary embodiments herein, the user interface 122 may host a survey module 800, as described in further detail below.

  User behavior is defined by activity, inactivity, movement, multiple event occurrences, temporal events, externally generated events, or any combination thereof. Specific user behavior is defined and provided to the computer 120.

  An activity can include picking up an object, which is placed or left in a particular location. Activities may include moving certain objects, such as opening doors, drawers, or compartments. Activities include positioning (or repositioning) a body part, such as placing a hand in a pocket or touching itself repeatedly in a specific location (an indication that a weapon may be hidden) Can be included. The activity may include moving to a specific location, a first individual engaging with a second individual, and / or moving hands, arms, legs, and / or feet in a specific movement. Activities may also include positioning the head in a particular direction, such as looking directly at security personnel or security camera 110, for example. Various other embodiments are discussed above in this specification.

  Inactivity can include failing to reach the object, the object falls or is positioned, and the individual (eg, the object) does not collect the dropped object. Inactivity may also include failure to walk to a specific location or failure to perform a specific task. For example, confirming that the security door is locked will require activity to approach the door and to tap the door to ensure that it will not be opened. Thus, user behavior can be defined as inactivity of approaching the door and / or inactivity of hitting the door to confirm that the door will not be opened. Various other examples of inactivity are discussed above in this specification.

  Temporal events may include the identification of customers who suddenly leave the store, individuals who stay at the store entrance or exit, and individuals who stay at a particular location for a time period that exceeds a certain threshold. Various other examples of temporal events are discussed above in this specification.

  A user may identify specific user behavior and provide and / or define specific user behavior characteristics within the computer 120. The computer 120 receives non-video data from the camera 110, and the non-video data includes behavior information data. The specific user behavior may be defined by a behavior model 143, which may be size, shape, length, width, aspect ratio, or any other suitable identifiable or identifiable attribute (eg, tattoo or Including one or more attributes, such as various other embodiments discussed in the book. The computer 120 includes a matching algorithm or matching module 141, such as a comparator, that compares the defined characteristics and / or model 143 of a particular user behavior with the user behavior in the defined non-video data. Matching indications by the matching algorithm or module 141 generate a survey, which includes video data and / or non-video data identified by the matching algorithm 141. A survey is a collection of data related to an identified event and generally records the behavior of interest. Surveys therefore require further scrutiny and survey to understand the specific behavior.

  The survey may be sent to other cameras or systems on a given network or provided across the network community to probe and alert for matches or identification. The matching algorithm 141 can be configured as a separate module or incorporated into the data analysis module 140 in the computer 120 or any camera 110. The data analysis module 140 may also include a comparator module 142 configured to compare the model 143 of specific user behavior and non-video data.

  Specific user behavior is that positioning the head towards the observation camera 110 exceeds a preset period, or positioning the head directly toward the manager's office exceeds a preset period. Can be defined. This particular user behavior may be that the customer is trying to prevent being detected during theft, trying to identify the observation camera 110 in the store, or whether the employee is observing his behavior Indicates that it is going to be judged. The data analysis module 140 implements an algorithm for generating non-video data that identifies the head position of the object. The video analysis module 140 may also provide a vector indicating the face and / or eye direction. The match algorithm 141 searches the non-video data to determine whether a vector indicating head position and / or face orientation exceeds a preset time period. A match results in the generation of a survey.

  Referring to FIG. 2, a data / video / image sequencer according to an embodiment of the present disclosure is shown as 200. The sequencer 200 can receive video, video data, from various sources of video (e.g., one of one or more video cameras 110) and / or from one or more of one or more antennas 150, It is configured to receive non-video data, video sequences, still images, and / or mobile communication device data. For example, continuous video may be provided from locations 1 and 2 while motion only data may be provided from location 7. Short duration video clips can be provided from locations 3 and 6, and still images can be provided from locations 4 and 5. Mobile communication device data may be provided from locations 1-7. This data may be communicated to the sequencer 200 via any suitable communication medium (eg, LAN, WAN, intranet, Internet, wired, modem connection, wireless, etc.). As shown in FIG. 2, the data from each location can be arbitrary, eg, continuously, periodically at a predetermined rate, asynchronously, in response to receiving a trigger, and the like. It can be communicated in any suitable manner.

  The sequencer 200 generates a time stamp from data provided with video data, image data, and / or mobile communication device data. The time stamp may be provided as part of the video data, image data, and / or mobile communication device data embedded in the video data, image data, and / or mobile communication device data, or the time stamp is Files containing video data, image data, and / or mobile communication device data may be provided. Alternatively, the sequencer 200 is configured to receive user input data, and the included timestamp information can be associated with each input.

  The sequencer 200 may additionally or alternatively generate a geographical location from data provided with video data, image data, and / or mobile communication device data. Geographic location information is provided as part of video data, image data, and / or mobile communication device data, embedded in video data, image data, and / or mobile communication device data, or video data, images Files containing data and / or mobile communication device data may be provided. For example, video, images, and / or mobile communication device data may contain landmark features that can be used to identify where the images were taken.

  The sequencer 200 may additionally or alternatively generate field-of-view data (hereinafter “FOV data”) for video data and / or image data. FOV data may be obtained from camera location information, obtained from information contained within the video (eg, landmark identification), and / or input by a user. The sequencer 200 may additionally or alternatively generate antenna range data for mobile communication device data. Antenna range data may be obtained from location information of antenna 150, obtained from information contained within mobile communication device data, and / or input by a user.

  FIG. 3 is an illustration of an image map 300 generated by the sequencer 200 and an associated timeline 310. The sequencer 200 utilizes time stamp data, geographical location data, FOV data, and / or antenna range data to provide all video data, image data, and / or mobile communication device data (or those) provided to the sequencer 200. Can be configured to assemble the image map 300 and the timeline 310.

  A user may provide a specific time and / or time frame to the sequencer 200, which provides all video, images, and / or mobile communication device data associated with the specific time. The time and / or time frame may be selected on the timeline 310, and the image map 300 displays all video data, image data, and / or mobile communication device data associated with the selected time and / or time frame. Can be updated to include.

  A user may additionally or alternatively provide the selected location to the sequencer 200, which provides all video data, image data, and / or mobile communications device data associated with that particular location. . The selected location may be selected on the image map 300 or provided to the sequencer 200 as geographic location data.

  The user additionally or alternatively provides sequencer 200 with a specific time and / or time frame in addition to the geographical location, and all video data, image data, and / or associated with that specific location. Or mobile communication device data may be further refined and separated.

  After a certain time, the time frame and / or geographic location is used to identify video data, image data, and / or mobile communication device data, and the user can search algorithms, methods described herein. And the system may be utilized to identify particular items of interest, patterns, and / or individuals contained within video data, image data, and / or mobile communication device data.

  The present disclosure provides additional algorithms and analysis for tracking and / or research purposes, as described below, beyond the scope of face recognition software (which can be used in conjunction therewith). In addition, in some cases, face recognition does not need to be used to flag or track someone or something, and the described system is sensitive to certain moral, federal, or local laws. It can be employed without any facial recognition software or algorithms that can be understood.

  The present disclosure also provides real-time / post-time object tracking based on pre-programmed parameters, such as the shape, color, size, number of certain objects on a person, oddities for a particular situation (eg, when hot at 80 degrees Fahrenheit) Winter coats), similarity of specific objects over a series of specific time frames (similar items in specific areas, eg backpacks), objects that require attention from a person over a preset time period Separation, strange objects in specific areas, objects placed close to objects that need attention, similar objects placed in similar areas and separated from people, contrasts and combinations of specific colors Various pre-programmed (or things based on, for example, a red shirt exposed under a black shirt, or a white hat over black hair) Programmed) real-time and post-time analysis of the known object or manually programmed object recognition, associated with analyzing recognition system for tracking.

  The object to be programmed is either a specific known based on the number of people carried, how the person walks, how the person is carried, etc., or an object lookup library and mapping algorithm It may include objects with shape, size, color, or weight. These objects can be pre-programmed into analysis software and tracked in real time and / or post time for analysis. Manually programmed objects are entered into the software by color, size, shape, weight, etc. and analyzed and tracked in real time and / or post time to determine abnormal conditions or for other purposes. obtain. Manually programmed objects such as face recognition images, tattoos, piercings, logos, or other indicia as described in more detail below may be uploaded in real time for analysis. In addition, user-generated items and / or images can be generated from video data (eg, frame data) and / or still images and provided for analysis. For example, as shown in the analysis and recognition system 400 of FIG. 4, identified within a still image and / or video frame (or identified as a result of one of the abnormal conditions described herein). Object 410 (eg, hat, backpack, clothes, or any identifiable feature) is separated from individual 405 for a pre-set amount of time (temporal event) and user generated for identification in live video 420 Can be retrieved and identified in a video 425, such as a video frame and / or a still image, provided or stored as a selected item 410 '.

  Person 405 may include a mobile communication device 440 that includes one or more antennas (not shown in FIG. 4) through which one or more signals (eg, mobile communication device data) are transmitted wirelessly. (For example, a smartphone). Examples of such mobile communication device data include when the mobile communication device 440 is a WiFi communication protocol, a medium access control (MAC) based communication protocol, a Bluetooth protocol, a cellular protocol, a near field communication protocol, and Includes a signal (eg, a handshake signal) that is transmitted according to one or more wireless communication protocols, such as a radio frequency identification protocol. As discussed above, one or more antennas 150 are mobile communication devices when they are located within the range of one or more antennas 150 in accordance with various exemplary embodiments herein. Is configured to capture the mobile communication device data transmitted by and transmit the captured mobile communication device data to the data analysis module 140 for processing.

  System 400 performs real-time and / or post-time analysis of video and non-video data (eg, mobile communication device data) and tracking all persons with backpack 410 in a specific area or a specific camera view. A data analysis module 140 may be included configured. The suspicious and / or attentional behavior of one or more persons can be tracked and recorded and analyzed either in real time or post time. For example, as identified in FIG. 4, if the backpack 410 is separated from the person 405 and left for a predetermined time period, the video may be flagged for real-time alerts and / or post-time analysis. Can be Objects, such as backpack 410, may be flagged, time stamped for later analysis, and / or separated into individual video streams. The user can zoom in for real-time or post-time analysis, for high resolution tracking, or for incorporation into the data / video / image sequencer 200 as discussed herein. Pre-programmed suspicious objects, eg, backpack 410 (or image library 430, user-generated image / object 435 (via input device), or any other mapping algorithm or module 140 recognized The person 405 who dropped the object may be tracked and analyzed for real-time alerts and / or post-time analysis. System 400 can be used for real-time or post-time analysis through one or more cameras 110, a network of cameras 110, 110a, 110b, one or more antennas 150, and / or a network of antennas 150, etc. Both tracking object 410 and flagging and tracking person 405 may be performed.

  In another example, the system 400 may flag and track a person who wears a winter coat in summer, a long raincoat when clear, etc. in real time for alerting purposes or post-time analysis. This will also be classified as an alert or an abnormal condition.

  System 400 may be able to combine pre-programmed analysis to alert on one or more abnormal scenarios (or combinations thereof). For example, a person carrying a case that can carry a semi-automatic or automatic rifle, and a person who sits outside a building that needs attention for a pre-determined time period, is automatically flagged and tracked. An alert can be sent to the security.

  The system 400 may be able to track and analyze specific objects, and the software or data analysis module 140 may pre-identify the same object in a video stream and / or still image that is acquired later. Can be programmed. For example, a particular important person is scheduled to have a press conference or scheduled to arrive at a particular location at a specific time. Scheduled events are postponed (intentionally or unintentionally). The software or data analysis module 140 is pre-programmed to recognize certain objects (or persons with objects 410 or user-generated objects 435) that appear in newly generated videos due to rescheduled events. obtain. In one instance, the original video from the original time of the scheduled event can be reviewed and the user can select a “repeat” object 410 (backpack, court, Software or data analysis module 140 may be pre-programmed to look for hats, clothing, briefcases, persons, etc.). A person can also be classified as a deaf and flagged for review at a later scheduled event. If it is the person of interest, a warning can be sent in real time to the security team reviewing the tape.

  The data analysis module 140 may be configured to recognize abnormal or unexpected pattern behavior and alert security or investigators to potentially abnormal scenarios, events, or conditions. The video and / or data can be configured for real-time analysis or post-event analysis. For example, the data analysis module 140 can be programmed to recognize a concentration pattern towards a specific geographic area and / or a diffusion pattern away from a particular geographic area. Global positioning software and vectorization can be utilized to accomplish this purpose. Recognition of concentration patterns and / or diffusion patterns can be useful in automatically recognizing potential flashmobs, mass robberies, or other unusual events. For example, as shown in FIG. 5, the analysis recognition system 500 may be configured to track an unusual number of regular customers 504a-504l arriving at a specific location 520 at or near a specific time 522. An analysis module 140 is included. The data analysis module 140 may also be configured to track the abnormal speed of regular customers 504a-504l and / or individuals arriving at or departing from a particular location 520. Typical arrival and / or departure speeds are preset or from previous individual algorithms that may have arrived at or departed from a specific location over a preset time or variable amount of time Can be acquired. Deviations from arrival and / or departure speeds can trigger abnormal conditions.

  Although not explicitly shown in FIG. 5, one or more of persons 504 may use one or more signals over which one or more signals (eg, mobile communication device data) are transmitted wirelessly. One or more corresponding mobile communication devices 440 (eg, smartphones) with more antennas may be owned. As discussed above, according to various exemplary embodiments herein, one or more antennas 150 are when they are located within individual ranges of one or more antennas 150. The mobile communication device data transmitted by the mobile communication device 440 is captured and configured to transmit the captured mobile communication device data to the data analysis module 140 for processing.

  The security system 500 with the data analysis module 140 and one or more camera arrays or systems 510a-510g recognizes an anomalous number of people concentrating towards a particular geographic area 520 over a preset time. Can be configured as follows. Data analysis module 140 may be configured to utilize vector analysis and / or image and data vector analysis algorithms and / or machine learning algorithms to evaluate one or more concentration patterns. Furthermore, the system 500 recognizes similarities in clothing, age, and items being carried (eg, briefcases, backpacks, and other similar items), and is an abnormal condition that could potentially be a security or investigator possibility. May be configured to alert. This can be useful in recognizing so-called “flash mobs” or other very sensitive situations during parades, marathons, political speeches, etc.

  The diffusion pattern and / or velocity can be used to identify an unusual pattern for an individual starting from a particular area 520. For example, in the case of a panic situation, the individual's diffusion rate is expected to exceed the preset or calculated average diffusion rate. Thus, the identification of one or more individuals leaving a specific area and / or situation at a speed below average speed or speed, such as panic, in some cases caused the person to have a specific panic situation, Or it may indicate that the individual was not in a panic-like state due to the fact that he was aware of it. Furthermore, a person who leaves the area faster than the average speed may “escape from an event”, eg, escape from a robber or escape from the next explosion.

  The data analysis module 140 also monitors web traffic and / or social media sites (Facebook (R), MySpace (R), LinkedIn (R)) related to specific locations and / or events and alerts on their nature. It can be configured to provide security or to combine web traffic associated with an event or geographic area with video analysis that recognizes a centralized pattern to alert potential flash mobs or gang robberies. The data analysis module 140 also works in reverse when a concentration pattern is recognized, accessing web traffic or various social media sites, pinging one or more of these sites, Some may collect additional information to reveal more pattern activity, or to reveal a flash mob event at a particular location.

  The data analysis module 140 may also be configured to monitor web traffic or social media sites for activity that precedes a particular time stamp. For example, a social media post that accompanies or precedes a specific event and conveys sorrow about the specific event may indicate a prognosis of that event and indicate prior knowledge of the upcoming event.

  The system 500 and the data analysis module 140 may use one or more street cameras, parking cameras, store / mall cameras, or to determine pre-programmed abnormal or manually programmed conditions in real time It may be configured to analyze video and / or mobile communication device data from other camera systems 510a-510g. The system 500 provides an alert when an unusual number of cars are concentrated at a particular spot (eg, a shopping mall) to determine how many people are concentrated at a particular store or location. The information can be combined with the video from the parking surveillance camera and the analysis can be configured to be combined with the in-store camera to determine wrinkles at a particular spot at a particular time or delta time. This is the typical behavior of a flash mob or gang robber. Again, the system 500 may connect to one or more social media sites for additional information and / or verification.

  Similarly, the speed patterns of approaching cars and / or the speed at which individuals depart from those cars, obtained from video and / or mobile communication device data, may also indicate abnormal conditions.

  Other examples of analyzes that the data analysis module 140 may perform in real time and / or post time may relate to gang type recognition. For example, the analysis and recognition system 600 of FIG. 6 recognizes gang colors and / or color combinations and / or patterns, and an unusual number of individuals (or unusual) with a particular color or color combination and / or pattern. % Of individuals) may be configured to flag video 618 and / or alert security if they are aggregated or hesitated over a particular geographic area. The data analysis module 140 may be used to identify certain characteristics or attributes 615 of the individual or individuals 605a, such as clothing, hats, pants style, shirt / coat color, style in which it is worn, symbol, coat logo, tattoo, It can be preprogrammed to recognize piercings, hairstyles, hand gestures, cars, motorbikes, etc. and alert to abnormal conditions in security or previous investigations stored as previous images 625 in computer 620. Alternatively and / or additionally, the data analysis module 140 recognizes the mobile communication device data (eg, cellular identifier) of the mobile communication device 440a of the individual 605a and the presence of the individual 605a that may be a known person known to security. Can be preprogrammed to alert. These individuals 605a can be flagged and tracked over a pre-set time period or until the person leaves the area. Overall image and characteristics 615 (color, uniform, hats, clothing style, hairstyle, logo, piercing, tattoo, symbol, other gang related signs, car, motorbike, or clothing in a crowd Etc.) may be recognized and trigger an alert. The data analysis module 140 is specific to x% individuals in a particular crowd having specific attributes 615, such as having the same tattoo, red shirt, same logo, and hairstyle. An alert such as an object attached may be provided. Data analysis module 140 may be configured to provide alerts based on an assessment that a predetermined number of individuals in a particular crowd have a particular characteristic 615.

  The data analysis module 140 provides a graphical representation of a number of abnormal conditions to better recognize one or more abnormal condition patterns or very high levels (and / or pre-determined levels). Can be configured to facilitate. This may allow a greater number of patterns to be tracked and analyzed by one or more individuals. The data analysis module 140 can also recognize contact between individuals, the contact can be a physical contact (eg, a handshake, a hug, or an object exchange), or the contact can be a non-contact (eg, to a conversation) Engagement, long-term eye contact, or other non-physical contact that would show cognition between them).

  Other alert type conditions may be associated with abnormal scenarios, and the data analysis module 140 recognizes objects carried by an individual 605b that are unusual for a particular area. For example, as shown in FIG. 6, a person carrying a pitch fork or shovel (not shown) in the mall 623 or a group carrying the bat 616 in the mall 623 and concentrating on a specific area (605b and 605c). ). Again, real-time analysis of the video is most useful and will provide alerts for unusual conditions in security. Post-mortem analysis can be useful for determining criminals when an event occurs, when authorities are called for assistance.

  Using any of the aforementioned scenarios or alerts noted herein, the data analysis module 140 cooperates with a video library 650 of images or algorithms and / or one of the aggregated mobile communication device data. Or it can collaborate with more databases to trigger alerts or respond to queries. Additional images, such as library images and / or user-generated images 650 can be provided as input to the data analysis module 140 and used to analyze the video through the recognition aspect of the data analysis module 140. This can all occur in real time or during post time analysis. Again, a query can be entered depending on the particular purpose, and the system 100, 200, 300, 400, 500, and / or 600 can analyze the video for the condition under which the query was made, in real time or post time. Can do.

  The systems 100, 200, 300, 400, 500, and / or 600 may be configured to perform 3D face recognition. The system 100, 200, 300, 400, 500, and / or 600 is manually programmed to recognize an individual or attention person 605a (or practitioner) in a survey based on clothing type, piercing, tattoo, hairstyle, etc. (Other than face recognition, which may be used depending on the authority of the organization (FBI vs. local mall security)). An image of the attention person 705a can be scanned into the data analysis module 140 and items such as piercings, tattoos, hairstyles, logos, and hats can be flagged and imaged for later analysis in real-time or post-time analysis. Can be uploaded to the database. For example, if a thief 605a plunders a convenience store and a person's face image is captured on one or more cameras 610, not only the person's image, but also other identifying information or characteristics or attributes 615 For example, hairstyles, tattoos, earrings, jewelry, clothing logos, etc. may be uploaded to all store cameras 610 as well. If the thief 605a enters the store again, an alert will be automatically sent to the security. Even if the system recognizes a similar tattoo or piercing pattern or logo 615 on a different person, that person can be considered a watched person for questioning by the authorities. Again, this goes beyond the mere face recognition, and so-called different people will not necessarily be flagged and tracked.

  The systems 100, 200, 300, 400, 500, and / or 600 also generate a library of individuals and / or patrons who regularly visit or visit certain locations, thereby enabling these specific individuals. System 100, 200, 300, 400, 500, and / or 600 focus on identifying and tracking individuals who have not been previously identified and stored in the library Can make it possible. A library of regular customers (not shown) may also link to a point of sale (POS) system, thereby verifying that the individual identified and stored in the library is a regular regular customer.

  As best shown in FIG. 7, another analysis and recognition system 700 is shown with a data analysis module 140 used in conjunction with a chain of stores, a mall, or a series of stores 750 in a town or community. The store community or store chain 750a-750e utilizes the same data analysis module 140 (or uploads images 724 and identifies information on individual store analysis systems 740a-740e) of the cameras 710a-710e. It is possible to share video images 724, mobile communication device data, and / or other identification information of known criminal 705 characteristics or characteristics across the network. These local store owners or store chains 750a-750e may have (prior characteristics or characteristics and / or images and / or information as described above) once a person 705 enters a store, eg, store 750a. It may be possible to prevent additional loss by flagging and tracking a specific known individual of interest 705 (based on identification information entered into the database). Alerts can be sent to the local authorities of these individuals (or groups of individuals), who are tracked throughout the network of cameras 710a-710e, including parking cameras, street cameras, etc. along the community network. obtain. Once the person 705 is flagged and there is an alert, other information related to cars, car types, car routes, accomplices, etc. can also be captured. Further, all cameras 710a-710e and / or antennas 150a-150e in system 700 may be alerted to flag and track individuals 705 and accomplices in real time and / or for post-time analysis. .

  The various described systems 100, 200, 300, 400, 500, 600, 700, and 800 can also be utilized to identify individuals with “contactless” conditions. For example, a building occupant may have an access prohibition order issued by a court that prevents certain individuals from entering a certain proximity range. An image, eg, 724, can be input to the system, eg, system 700, and the data analysis module 140 can identify the individual 705 and provide notifications and / or evidence to the building dwellers and / or authorities. Similarly, a government generated database 720 may be provided to the system 700, which includes a library 724 of images of individuals 705 identified in a particular legally required registration program.

  The community may choose to set up a community network for cameras 710a-710e for this purpose. A new owner of a local company may have a specific criminal image 724 for analysis (ie, for local alerts) with a single subscription (eg, amount), for example, a particularly violent crime While uploading a person's image 724, while additional identifying information may be particularly focused on the entire community for uploading on all networked cameras 710a-710e (or even a standalone system), Short-time shoplifters may not be noticed.

  The data analysis module 140 may also use walking methods such as a drag foot, a skate, a head angle, a stride, a hand swing, a hand gesture, etc. as an indicator of an individual or attention person. A person's way of walking is as individual as a fingerprint and can be used to identify disguised criminals. Many variables contribute to an individual's way of walking and this information can be uploaded to the data analysis module 140 (eg, walking speed, frequency of steps, angle between legs, hand / arm position, hand / arm). Arm swings, dragged feet, slippers, etc.).

  The data analysis module 140 also allows a certain number of known images or events or habits to occur within a certain time period (eg, touching its particular area X times within a preset time period, Touch or hold a known area to carry or hide, tense and convulsions, or rapidly turn head, glance with side eyes, within preset time period (Such as watching a video camera X times) may be configured to alert security. The data analysis module 140 alerts the security based on the occurrence of abnormal or excessive habits or events within a preset time limit, or any combination of events within a preset time period. Or may be configured to provide information to the user. For example, a person who walks through a store holding his hand over trousers and rapidly turning his head can trigger an alert or an abnormal situation. In another embodiment, security is flagged or emphasized (or otherwise identified within an area by systems 100, 200, 300, 400, 500, 600, 700, and / or 800) An attention person who repeatedly looks at the side or turns his head repeatedly in that direction can trigger an alert or an abnormal situation. In another example, typically an area with a short residence time (e.g. in a cosmetic area, the female residence time is typically, but not necessarily, long, while the male residence time is typically In particular, they are individuals who shop and / or stay in a store area that is short).

  As mentioned above, the analysis recognition system 100, 200, 300, 400, 500, 600, 700, and / or 800 of the present disclosure pre-programs habitual behaviors in the data analysis module 140 to Can be utilized to determine gun or weapon detection by analyzing (in real time and / or post time). For example, a person who repeatedly grabs a known area for storing weapons and walks in a certain way of walking (for example, walking with a dragged foot may indicate carrying a shotgun) is an indicator of the person carrying the weapon. Can be This information can be analyzed along with other identifying information or indicia (eg, tattoos, gang colors, gang symbols, logos, etc.) to trigger alerts or unusual situations. In another embodiment, the individual wears a trench coat and is looking sideways or turning his head when it is not raining or on a sunny summer day. In this case, the data analysis module 140 would require some sensory input regarding rain or temperature or sunlight (sunlight) and / or a connection to a system that provides such data. The time is also analyzed by the data analysis module 140, for example, to increase the “recognition” of the data analysis module 140 when triggering an alert much later in the night when more robberies tend to occur or after midnight. Can be preprogrammed triggers or additional events.

  In other embodiments, the data analysis module 140 allows security personnel to query the analysis recognition system 100, 200, 300, 400, 500, 600, 700, and / or 800 in real time or post time. obtain. “How many people wearing red baseball caps entered the store or area within 5-10 minutes of the delta?”, “How many people concentrated at the central fountain at this time or over this delta time?” Other queries may include the following instructions: “How many people are staying in the fountain for Delta minutes?” “Scanning and recognizing / flagging / tracking / tracking a person wearing long trousers or a winter coat (on a summer day at 90 degrees Fahrenheit)”, “Scanning and recognizing a person wearing a red hat / Flagging / tracking / tracking ”,“ scanning and recognizing / flagging / tracking / tracking a person carrying multiple backpacks ”,“ persons leaving objects (eg, backpacks without owners) Scan and recognize / flagging / tracking / tracking. Track people across systems, multiple systems, object flag locations, etc. "," Turn close to areas that need attention, close to areas that need attention And scan / recognize / flagging / tracking / tracking a person who has left an object, tracking a person across systems, multiple systems, flag locations, and / or To the time the absence of the object is pre-set, or if it is left for a preset time, the alert ".

  In another embodiment, the data analysis module 140 performs real-time video processing and analysis to determine crowd parameters (eg, real-time crowd count or real-time crowd density estimation) through automated processing of a physical space video sequence. Can be configured. Video analysis module 140 may include one or more algorithms configured to determine a rate of change in crowd parameters. The rate of change in crowd parameters can indicate crowd concentration or crowd spread.

  Data analysis module 140 automatically issues an alert when the rate of change in the crowd parameter exceeds a pre-determined threshold. For example, if the rate of change in crowd parameters indicates crowd concentration, the data analysis module 140 may alert a security potential flash mob or gang robber. Data analysis module 140 may be configured to utilize vector analysis and / or image and data vector analysis algorithms and / or machine learning algorithms to evaluate one or more concentration patterns.

  Data analysis module 140 may be connected to an array of cameras 510a-510g organized in the network, and each camera in the network may respond to one or more objects or individuals (eg, FIG. Can be used to track regular customers 504a-504l) shown in FIG. If the rate of change in the crowd parameter indicates crowd spread, the data analysis module 140 may alert a potentially dangerous situation or criminal activity to security.

  FIG. 8 is a screenshot of a survey module 800 displaying a generated survey, according to an embodiment of the present disclosure. Survey module 800 is configured to generate and store information required to record specific user behavior.

  In addition, survey module 800 may be configured to receive survey criteria entered by a user or machine and mine video data, mobile communication device data, and / or profile data based on the criteria. In one example, the survey criteria may include a time frame, and the survey module 800 may be configured to generate a list of people whose presence was detected at a facility that is in the time frame. The survey module 800 may also be configured to determine the current location of the person by detecting a signal that matches the previously acquired mobile communication device data at another location.

  In some exemplary embodiments herein, the survey module 800 receives report criteria entered by a user requesting a particular type of report and is obtained from the video camera 110 and / or antenna 150. Data, mobile communication device data, and / or profile data, and / or sales data (eg, obtained from a sales database) are configured to generate a report based on the data mining. The report may include, for example, a calculated sales close rate corresponding to a merchandise item, merchandise category, a passageway at the facility, a shelf at the facility, and / or a predetermined location at the facility.

  The survey module 800 includes a browsing window 810 with upper and lower browsing control bars 812a, 812b, a text input window 814, a timeline window 820, a camera window 830, a search window 840, a playback options window 850, A clip option window 860 and a file maintenance window 870 are included.

  The survey automatically generated by the system 100 is populated with information related to specific user behavior as discussed herein above. For example, the study illustrated in FIG. 8 includes a first video sequence 820a and a second video sequence 820b, where the first video sequence 820a is from a staircase camera and the second video sequence 820b is From the camera located in the elevator. In one embodiment, the first video sequence 820a was provided through an automatically generated survey, and the automatically generated survey was provided to loss prevention personnel.

  The first video sequence 820a is selected in the timeline window 820 and played in the viewing window 810. To proceed with this description, for example, assume that a loss prevention officer observes an individual who takes away a company laptop computer from the stair area in response to viewing the first video sequence 820a on the PDA. In generating the survey, the system identifies this user behavior as a specific user behavior, and in response to a scrutiny, the loss prevention officer agrees that the automatically generated survey has the benefits, The automatically generated investigation has been expanded to theft investigation.

  The automatically generated survey is provided to loss prevention personnel in near real time, so individuals who have a company laptop at this point will only take a few steps from where the laptop was taken away. It was n’t.

  Using a PDA, a loss prevention officer can automatically generate a survey (here a theft survey) by observing the temporally relevant video and video data available through the survey module 800 on the PDA. To proceed.

  Search window 840 may automatically select a time frame associated with the survey. The timeline can be manually controlled through the PDA.

  Video and / or video data from one or more cameras listed in camera window 830 may be selected for viewing in viewing window 810. Multiple video streams from individual cameras (see FIG. 1) may be viewed simultaneously by selecting an alternate viewing screen from the upper viewing control bar 812a.

  Lower viewing control bar 812b allows viewing video in viewing window 810 at real time or other selected speed. The survey module 800 provides a survey playback speed, which is automatically calculated to play the video at a playback speed that requires the loss prevention personnel to view all frames of the video sequence. The video is recorded and stored at a speed that exceeds the capabilities of the human eye to detect slight movements. In addition, the playback device may also have hardware and / or software limitations that prevent the playback device from displaying all frames of the video. Thus, video playback in “real time” results in the loss of individual frames of the video due to human viewing limits and / or computer display limits. The survey playback speed is calculated based on the human viewing limit and the display limit of the particular device being used to browse the survey module 800.

  The playback options window 850 allows the video sequence and / or video from each camera to be played in various modes. The full frame display mode plays the video at the calculated survey playback speed, and all the frames are displayed during playback and can be viewed. The motion only display mode provides a video sequence of video that includes motion. The trigger only display mode includes a video sequence that is temporally related to the trigger.

  The trigger includes an internal trigger and / or an external trigger. Internal triggers can be defined by the user and determined by the data analysis module 140, movement triggers, POS triggers generated by the POS module 141, and trip lines and / or areas (eg, entering an area and / or Analysis events that are defined by and then determined by the data analysis module 140. The external trigger is generated by an external hardware device that is directly or indirectly connected to the computer 120.

  At any point in the survey, loss prevention personnel can assign video sequences to the timeline. For example, in FIG. 8, the loss prevention officer has added a second video sequence 820b to the survey. The second video sequence 820b includes video provided from an elevator and a camera located on the stairs. To proceed with the scenario described hereinabove, assume that the loss prevention officer has identified a person of attention carrying the laptop and approaching the elevator displayed in the second video sequence 820b. . To facilitate a theft investigation, the loss prevention officer included a second video sequence 820b in the investigation timeline.

  The loss prevention officer may select various options from the video clip window 860. The timeline window 820 may be populated with one or more video sequences, still images generated from the video, or video clips containing text entered through the text input window 814. A video clip may include a continuous video sequence. Alternatively, a video clip that uses motion-only playback options (selected in the playback options window 850) includes multiple video sequences that include motion (eg, non-motion parts of the video are excluded from the video clip). ). Finally, loss prevention personnel capture a still image of the frame and personal features such as facial images, specific tools, or objects used during theft, or any other that may be required to proceed with the investigation. Of significant images can be captured.

  Finally, because investigations are generated in near real time, loss prevention personnel can notify security before the thief can leave the facility in response to confirmation of an ongoing theft, It is possible to arrest a thief.

  Reference is now made to FIG. 9, which shows a flowchart of an analysis recognition method 900, according to an exemplary embodiment of the present disclosure. According to exemplary embodiments herein, method 900 can be performed, at least in part, with video data captured by one or more video cameras 110 and / or one or more antennas 150 and / or respectively. Or, based on mobile communication device data, may be employed to populate a database with data that may be useful for security purposes, research purposes, marketing purposes, and / or the like.

  At block 902, video data is captured by one or more video cameras 110 and mobile communication device data is transmitted by one or more antennas 150, as described above in more detail with respect to FIG. Be captured. The video data, in one embodiment, includes images of one or more persons who were located in the field of view of one or more cameras 110 at a time, and the mobile communication device data is one at a time. It includes data captured from one or more mobile communication devices 440 that were located within the wireless communication range of one or more antennas 150. Each item of mobile communication device data may be associated with an individual mobile communication device that was carried by an individual one of the persons whose one or more images were captured.

  In block 904, the item of video data captured in block 902 is captured in block 902 based on the captured video data and one or more keys included in the mobile communication device data. Correlated with individual items of data. In one example, based on the correlation at block 904, individual profiles of persons associated with one or more individual items of video data and / or mobile communication device data are generated. Each profile may have profile data that includes any one or a combination of captured video data, captured mobile communication data, and / or additional data.

  The one or more keys utilized in block 904 may include any one or combination of attributes included in the item of video data or mobile communication device data, which is video It can be used to identify items of data or mobile communication device data and / or to correlate multiple items of video data and / or mobile communication device data as interrelated. For example, a human face image contained within an item of video data captured on a first date and time may be used as a key so that that item of video data is captured on a second date and time Can be correlated with another item of video data. As another example, an IP address included in an item of mobile communication data captured at a first location (eg, by the first video camera 110 or antenna 150) may be used as a key, thereby That item of mobile communication data can be correlated with another item of mobile communication data captured at a second location (eg, by the second video camera 110 or antenna 150). Exemplary types of keys include, but are not limited to, a collector / beacon identifier (eg, a unique identifier for a particular video camera 110 or antenna 150 that captured an item of video data or mobile communication device data), from a mobile device 440 Mobile communication device address (eg, Wi-Fi address, Bluetooth (registered trademark) address, NFC address, RFID address, cellular address, GPS device address, MAC address, mobile subscriber identification included in the captured mobile communication device data Number (IMSI) identifier, and / or any other suitable address or identifier), signal strength of mobile communication device data captured from mobile device 440, video data or mobile communication device The date when the item of data is captured, the time when the item of video data or mobile communication device data is captured, the location where the item of video data or mobile communication device data is captured, thereby capturing the item of mobile communication device data Media (e.g., certain wireless communication protocols) and / or the like. The key utilized in block 904 and any other data that is captured, generated, or otherwise derived from the steps of method 900 is encrypted using one or more suitable encryption algorithms in a known manner. Can be

  At block 906, the profile data generated at block 904 is normalized based on one or more normalization criteria. For example, the profile data may include (1) a store where one person has one or more cameras 110 and antennas 150 where a particular location (eg, video data and / or mobile communication data was captured in block 902). Normalized) based on the number of visits to (location), (2) the duration of time that a person stayed at a particular location, and / or (3) the frequency at which the person visited a particular location or its repetition rate Can do. This can be useful for identifying repeat customers, criminals who pretend to be a store before committing a burglary, and / or the like.

  At block 908, the profile data generated at block 904 and normalized at block 906 is one or more generated for individual profiles based on data aggregated per profile / person over time. Updated to include attributes. Examples of such attributes are that a person may have returned to the store, a first-time shopper, an employee (eg, with a very short duration of stay in a location, for example, the person may simply pass the store Is a passerby (as determined when passing within the camera 110 and / or antenna 150 located at the store), or the person's video data and / or mobile communication data has been captured May include whether to shop elsewhere in a particular retailer, whether a person was involved with an employee while located in a store, and / or the equivalent. After block 908, the method 900 may return to block 902 to continuously maintain video data and / or mobile communication device data as described above.

  As described above, data captured, generated, or otherwise resulting from the various steps of method 900 may be used for security and / or research purposes (eg, after a burglary in a store), for marketing purposes. And / or for many other purposes. For example, the data may include sales associated with a pre-determined event, such as a holiday shopping season, for example, to generate a virtual lineup for criminal investigation purposes in one or more positive or confidential applications. Or, based on the actual sales history in the parcels in the store, to analyze customer stay statistics, to calculate the true customer conversion rate of sales encountered, to enable analysis of marketing data Based on data captured from multiple cameras 110 and / or antennas 150 to generate a map, to identify a person who stays in the store after the store's closing time to generate a route through which the user passes And / or for the equivalent).

  Since various changes may be made in the above structure without departing from the scope of the present disclosure, all matters contained within the above description are to be construed as illustrative rather than limiting. Is intended to be. It will be appreciated that certain objects of the disclosure are achieved and other advantageous results are achieved, as defined by the scope of the following claims.

Claims (22)

An analysis recognition system,
A video camera configured to capture video data;
An antenna configured to capture mobile communication device data;
A data analysis module configured to correlate the video data and the mobile communication device data to generate a profile of a person associated with the video data and the mobile communication device data, wherein the profile is captured And a data analysis module comprising profile data comprising any one or a combination of said captured video data and said captured mobile communication data. The data analysis module includes:
One or more of the person's arrival time at a location and the person's departure time at the location based on any one or combination of the video data and the mobile communication device data Determining the combination of
The correlation of the video data and the mobile communication device data based on any one of or a combination of the arrival time and the departure time. system. The video camera is one of a plurality of video cameras included in the system and configured to capture a plurality of video data, and the antenna is included in the system and includes a plurality of One of a plurality of antennas configured to capture mobile communication device data;
The data analysis module further includes:
Correlating the plurality of video data and the plurality of mobile communication device data to generate a plurality of profiles of a plurality of persons associated with the plurality of video data and the plurality of mobile communication device data, respectively. The system of claim 1, configured.   The system of claim 3, wherein the plurality of video cameras and the plurality of antennas are located at a plurality of facilities.   The system of claim 4, further comprising a user interface configured to allow the plurality of profiles to be mined based on user input criteria.   At least one of the plurality of antennas is attached to at least one of the plurality of video cameras, and at least one of the plurality of antennas is remotely located from the plurality of cameras. Item 4. The system according to Item 3. The profile includes the captured video data, the captured mobile communication device data, the captured video data or time data associated with the captured mobile communication device data, and the captured video data or the Including any one or a combination of location data associated with captured mobile communication device data;
The captured video data includes any one or combination of captured still images and video images,
The mobile communication device data is any of a WiFi identifier, a medium access control (MAC) identifier, a Bluetooth identifier, a cellular identifier, a near field communication identifier, and a radio frequency identifier associated with a mobile communication device that communicates with the antenna. Including one or a combination thereof,
The temporal data includes any one or a combination of the time at which the video data is captured and the time at which the mobile communication device data is captured;
The location data includes any one or a combination of where the video data is captured and where the mobile communication device data is captured.
The system of claim 1.   The data analysis module may further include, based on the correlated video data and mobile communication device data, in the profile, the person's visit frequency to a facility and the person's visit frequency to the facility. The system of claim 1, configured to add one or a combination thereof.   The data analysis module further includes: data from a first source and the captured video data, the captured mobile communication device data, the correlated video data and mobile communication device data, and the profile Configured to identify the person based on a comparison between any one or combination thereof, wherein the first source is a non-government database, a government database, and one or more previous The system of claim 1, comprising any one of the generated profiles or a combination thereof.   A survey module configured to receive survey criteria and to mine any one or combination of the video data, the mobile communication device data, and the profile data based on the criteria The system of claim 1, comprising:   11. The system of claim 10, wherein the survey criteria includes a time frame, and the survey module is further configured to generate a list of persons whose presence was detected at a facility that is in the time frame.   The survey module of claim 10, wherein the survey module is further configured to determine a current location of the person by detecting a signal that matches the mobile communication device data obtained at a first location. system.   2. The antenna of claim 1, wherein the antenna includes any one or combination of a WiFi antenna, a medium access control (MAC) antenna, a Bluetooth antenna, a cellular antenna, a near field communication antenna, and a radio frequency identification antenna. The described system.   The data analysis module is further configured to positively identify the person based on any one or combination of the video data, the mobile communication device data, the profile data, and user-entered criteria, The system of claim 1, configured to assign to a list, or negative list.   The data analysis module further states that any one or combination of the video data, the mobile communication device data, and the profile corresponds to an employee or a person on a list of pre-determined persons. The system of claim 14, wherein the system is configured to assign the person to the positive list based on a determination.   The system of claim 1, wherein the antenna is configured to capture the mobile communication device data by receiving data wirelessly from mobile communication devices located within the antenna.   The system of claim 1, wherein the data analysis module is further configured to detect a behavior of the person and store behavior data corresponding to the behavior in the profile. The data analysis module is configured to detect the person's behavior by extracting behavior information from any one or combination of the video data and the mobile communication device data;
The behavior includes looking at a direction, reaching for a product item, purchasing the product item, traveling along a route at the facility, visiting a passage or place at the facility, the facility Any one or a combination of spending a certain amount of time at the facility, spending a certain amount of time at the location at the facility, and visiting the facility over several separate cases,
The system of claim 17.   The data analysis module classifies the person as a new customer or repeat customer at the facility based on facility visit data stored in the profile, and the person is a new customer at the facility or a repeat customer. The system of claim 18, wherein the system is configured to add to or update in the profile.   The data analysis module is configured to convert any one or a combination of the video data, the mobile communication device data, and the profile data, a passage location mapping in a facility, a product location mapping in the facility, and the The system of claim 17, configured to detect the person's behavior by correlating with any one or a combination of shelf location mappings at a facility. A survey module,
Mining any one or combination of the video data, the mobile communication device data, the profile data, and sales data based on reporting criteria;
Generating a report based on the mining of any one or a combination of the video data, the mobile communication device data, and the profile data, and
The report includes a sale close rate corresponding to any one or combination of product items, product categories, aisles at the facility, shelves at the facility, or pre-determined locations at the facility. The system of claim 20, further comprising a module. The data analysis module further includes:
Generating location data corresponding to the behavior based on any one or a combination of the video data and the mobile communication device data;
The system of claim 17, wherein the system is configured to: store the location data in the profile in association with the behavior data.

Images