WO2005045747A1

WO2005045747A1 - Autonomous flow-control system using artificial intelligence

Info

Publication number: WO2005045747A1
Application number: PCT/ES2004/000496
Authority: WO
Inventors: José Luis SERRANO RUIZ
Original assignee: Serrano Ruiz Jose Luis
Priority date: 2003-11-06
Filing date: 2004-11-05
Publication date: 2005-05-19
Also published as: MXPA06005056A; WO2005045747B1; ES2232301A1; ES2232301B2

Abstract

The invention relates to a simple, low-cost autonomous device with robust functionality, which is based on an analysis of signals captured by a device with an image sensor. According to the invention, the aforementioned signals are transmitted to a microprocessor which analyses said signals using an embedded expert system. The inventive system also comprises an autonomous memory element and a communication port. The invention is intended to be used for flow and capacity studies or as a security system.

Description

AUTONOMOUS AFFECTION CONTROL SYSTEM BY ARTIFICIAL INTELLIGENCE

FIELD OF THE TECHNIQUE

The present invention relates to a system for detecting people and objects in motion, and can therefore be used for counting people and objects, or as a security system. The system is embodied in a single and simple device with low cost and robust operation. It is designed to cover the gap that exists in the field of inflow and capacity studies, providing effective knowledge of data. The device is composed of an optical system, an image capturer, a microprocessor, a storage submodule and a communication block. The operation of this unique device is based on obtaining the signals collected by the image capturer and the subsequent processing of them by the expert program based on neural networks that is executed in the microprocessor. This program detects objects (or, where appropriate, people) and their sense of passage. The use of the system is aimed at a market segment that lacks a device that is reliable and economical to know the inflows and capacity of surfaces and obtain data that allows them to know and increase their benefits, in addition to helping to ensure safety from establishments such as stores, large commercial areas, small businesses, fairs, passenger transport, etc.

STATE OF THE TECHNIQUE

The constant evolution of image treatment techniques is opening new fields of application for people's daily lives. In an increasingly competitive market, companies need to know the influx of customers in their establishments in order to reduce costs, plan templates and know the success of advertising campaigns or promotions. Inflow count systems are known, there are currently systems / devices capable of counting objects or, where appropriate, people, whose characteristics can be included in one of the following groups: - Simple: These are systems in which the effectiveness It is average, which allow knowing slots in which there is more activity than in others, but the error in the data collected can be very large. Within this group, manual, infrared lateral position systems, etc. are included. - Complexes: These are systems that have a high complexity, where the data obtained has a greater ability, but at the same time they are expensive, difficult to maintain, not robust, etc. These systems, due to their high cost, are not accessible to the small merchant or to small and medium-sized businesses.

Spanish patent P 2125798, from the Polytechnic University of Madrid, describes an automatic counting system for people on the move, which has two independent cameras, a computer for data processing and needs a light source to illuminate the passage area .

The present invention is an autonomous, embedded system that integrates an image capturer with a memory, a communication block and the processor in the same device. In addition, the device is designed to perform counting using natural lighting but has the ability to operate using infrared lighting integrated into the same device in low light situations.

Japanese patents JP 7028969 and JP 7085242, of KUNIO HIUGA, protect the same counting system, based on an algorithm; the second deals with the main counting program, while the first refers to the algorithm for normalizing the lighting conditions to carry out the system of count described in the second. It consists of a camera installed in a door or similar, analyzing only its field of vision and has some non-embedded processing blocks.

The invention referred to in the present application, in addition to the differences indicated with the Spanish patent, uses Artificial Intelligence models based on neural networks. In addition, this device has the ability to work in a mode of cooperation through which it can work in collaboration with other devices referred to in the present invention to cover wide fields of vision with the aim of not duplicating or losing objects (or where appropriate people) in that field of vision.

The English patent GB 2180642, of HOCHIKI CO, for the counting of moving bodies, is based on two light bulbs on the floor, reflected by mirrors and dissected through a lens towards linear CCD's and by means of a counting algorithm they are processed , determining number of people. The present invention, in addition to all the differences indicated with the previous patents, does not necessarily require one or several light sources to illuminate the area and collect the data.

EXPLANATION OF THE INVENTION

It is a system that counts entries and exits of objects (or, where appropriate, people) based on the study of the image obtained by an image grabber. This image is subjected to a digital treatment, and the signals and characteristics obtained are analyzed by the expert functional block of artificial intelligence based on neural networks. This functional block is the one that detects the objects (or, where appropriate, people) and determines to which set of the previously established belong (object enters, object leaves, noise, shopping cart, vehicle leaves, vehicle enters, person leaves, person enters, person enters etc). The movement of the objects under evaluation may be longitudinal, diagonal or transverse to the device. The system works without the need for lighting but has the ability to operate through lighting infrared incorporated in the same device to count in areas of poor lighting.

For a security application, it detects behavior and can trigger the most appropriate alarms for each case. The system has the ability to send images and even video.

The passage of the objects (or in their case people) can be done at any speed and trajectory, without influencing neither the previous parameters nor the sense in the recognition process.

The microcontroller includes an internal program, conveniently protected, that analyzes the signal obtained by the sensor and provides the result of the recognition operation.

The system also includes an internal memory to store the influx data autonomously. In addition, these can be exchanged with an external computer system through one or more communications ports for later analysis. The device has the ability to communicate with the computer system through physical or wireless transmission media, at unlimited distances, through various communications technologies. For this, the device is implemented with the communications block of the appropriate communications technology. Likewise, it is worth noting the fact that the system includes self-calibration means transparent to the user, so that every time the system identifies changes on stage, it allows automatic adaptation to these changes or different working conditions. In this way, the positioning of fixed objects (such as carpets, mats, floor changes, changes in the paint of the ground or surroundings, ...) or obstacles (various furniture, fire extinguishers, architectural constructions, ditches ...) in the counting zone it is detected by the system and it adapts automatically, without the intervention of the client or specialized technicians. The internal program, which is conveniently protected in the microcontroller, can be updated through the communications port, which allows the knowledge of the expert artificial intelligence expert system resident in the device to be updated. The system uses an external power supply that takes the energy from the existing electrical network and transforms the current into the operating conditions of the same, being able to operate also by means of an electric vehicle battery outlet, such as a bus, a train, an airplane, etc., and autonomously taking the energy of some batteries.

The configuration of the device at the time of installation is simple. The only parameters necessary for the operation of the system is the height at which the device is located, the distance with the left limit of the passage zone and the distance with the right limit of the passage zone. In addition, if the device works in cooperation mode (that is, when an access is covered by more than one device), a cooperation label (value 1 2 or 3) is added for the left limit and a label for the right edge. These cooperation labels determine the counting area associated with each device that acts in cooperation

BRIEF DESCRIPTION OF THE DRAWINGS

To complete the description that is being made, and in order to help a better understanding of the features of the invention, according to a preferred example of practical implementation thereof, a set of drawings is accompanied as an integral part of said description. where for illustrative and non-limiting purposes, the following has been represented:

Figure 1 shows a representation of the preferred position of use of the counter device object of the invention.

Figure 2 represents a block diagram with the components of the device.

Figure 3a is a front view and 3b another rear of the device in question.

Figure 4 is an explanatory flow chart of the system workflow. Figure 5a represents the coverage area of one device and 5b that of several collaborative devices.

Figure 6 is a block diagram of the algorithm that includes the system. DETAILED EXHIBITION OF AN EMBODIMENT The device in which the system of the invention materializes should preferably be placed in the zenith position represented in Figure 1, thereby avoiding interference with customers.

The passage of the objects (or in their case people) is carried out longitudinally, diagonally or transversely to the device, where these are illuminated by natural light or by the aforementioned infrared light emitters incorporated in the device, which illuminate the person or object. The device, which is unique, integrates (Figure 2) an optical system SO, a Cl image capturer, a microprocessor, a MEM data storage submodule, a COM communication block and if necessary an infrared illumination module for areas of Low or no visibility. The image obtained from the optical system SO-image capture system Cl is analyzed by a digital processing subprogram of the TS signal that is executed in the microprocessor. This digital processing subprogram of the TS signal returns a set of signals and characteristics. These signals and characteristics enter into an expert subprogram of artificial intelligence based on SE neural networks that runs on the same microprocessor. This expert subprogram returns the final verdict.

The final verdict, that is, the number of entries and exits of objects (or persons, if any) is stored in a MEM data storage submodule. In addition there is a COM communications subsystem with standard outputs to other systems, in order to facilitate the exchange of data, update the embedded expert system, perform remote tests in case of failure, alarm management, download of stored data, etc. , also being able to reload the internal program from a remote system. Among others, the system has the ability to transmit through this communications subsystem the captured image and even real-time video to other external systems in order to detect alarms in security systems, to configure the system at the time of installation , verify the proper functioning of the device or check its efficiency. Figure 3a shows the rear part of the device and it shows the output connections for the communications submodule, CON.COM, and the connection to the power supply, 9 VDC. Figure 3b shows the lens L that belongs to the optical system of the device that protrudes slightly from the front of the device. The shape and components of the communications module and its output connections vary depending on the communications technology used between the device and the external computer system.

The device does not need a period of adaptation to the scenario in which it will work. In addition, the device is robust enough to be installed in a mobile vehicle and automatically adapt to any change of scenery, greatly facilitating manipulations by the user or installer. The changes of scenery to which it adapts automatically include, for example, the changes of natural light or those of surrounding furniture.

The special characteristics in terms of size and autonomy of the device mean that it can be integrated into mobile systems or in locations where previously such systems would be unthinkable. The system has the ability to save data for long periods of time in its storage memory block. To do this, it uses a natural storage system based on STM / LTM algorithms (short time memory / long time memory)

The STM / LTM theory (from English Short Time Memory / Long Time Memory) is a psychological theory of natural storage of knowledge. It deals with the study of memory in humans. This theory encompasses many concepts. The idea applied is as follows:

The human being does not store the information with the same detail. Store the information you just received in more detail than the information you received a long time ago. This concept is used to store more data in a memory of limited capacity. The duration of the time during which the data is stored depends on the type of memory technology used, becoming several months in the embodiment described. The system incorporates an expert module of artificial intelligence based on neural networks. The incorporation of artificial intelligence into the device consists of several stages:

- Design / update phase: The design phase has been carried out in the laboratory. The completion of this phase is carried out without customer intervention. In this way the client does not suffer the inconvenience derived from the collection of videos or the study of the problem. In this way the client receives a system that works from the first day of installation without assuming expenses of technicians derived from this stage of defining scenarios. This is possible thanks to the incorporation of the learning mode of the artificial intelligence model. The points that are developed in this phase are the following: o Selection of generic videos: Several generic videos have been taken of object steps (or where appropriate people) that include different complex situations. These videos were not taken from each client's particular scenario but were generic videos. o Signal processing: Video signal processing was performed. As a result, a set of signals and values that are characteristic of the passage of objects (or where appropriate people) were extracted. o Learning: In this process, the signals and values extracted in the signal processing have been associated to three possible situations: • Object (or, where appropriate, person) enters. • Object (or in your case person) leaves. • It is not an object (or in your case person).

In this way the appropriate signal and characteristic is associated to each of the three previous situations. As a result of the design / update phase, an intelligent system capable of distinguishing situations is obtained (object enters, object leaves, cart, vehicle, person enters, person leaves ...)

- Development phase: In this phase, the artificial intelligence model obtained in an autonomous system equipped with the chosen communications is introduced. As a result an embedded expert system is obtained.

- Operation phase:

In the operating phase the system is equipped with the following characteristics: o High level intelligence o In reduced dimensions or At low cost In addition the device has the ability to take a sequence of frames in real time.

The quality of the sequence of frames is sufficient to distinguish the number of objects (or possibly people) that access the evaluation area. The external computer system communicates with the device through a specific control program that has the ability to display in real time this sequence of frames sent by the device. Together with the transmission of the sequence of frames, the device has the ability to send in real time to the control program the number of inputs and outputs of objects (or possibly people) detected. The control program has the ability to present this account of inputs and outputs in real time. In this way, the device-program control unit integrates a method of counting verification without having to resort to parallel account demonstration systems with the consequent saving of time, inconvenience and costs. The solution is therefore an integral solution for detecting and counting objects (or, where appropriate, people):. Efficient counting system . Online real-time account verification system. . Automatic and integrated device alarm control system. The system is self-adaptable to traffic. Given an area of fixed dimensions, it has been found that the effective area occupied by an object varies depending on the number of objects in that area. In this way, two types of scenarios are distinguished: • Medium, low flow of people: In this scenario, the average effective area of an object becomes larger. In the case of people, it represents about 0.25 square meters with a variation of plus or minus 25%. • High flow of people: In this scenario, the average effective area of an object becomes smaller. In the case of people, it represents about 0.15 square meters with a variation of plus or minus 25%.

The device detects the density of object traffic (or, where appropriate, people) by access, working in one of these modes, depending on the density of object flow.

As shown in Figure 5a, a DISP device counts the inputs and outputs that are made in an AC coverage area. To count the inputs and outputs in very large accesses, several systems that work in cooperation are needed. Each device allows to cooperate with one or several additional devices with the aim of covering a wider access without losing or duplicating any person steps, as can be seen in Figure 5b. The cooperation algorithm is based on overlapping the coverage areas of the devices and assigning the overlapping zone to a single device.

The system consists of two blocks: - Algorithmic subsystem: System that counts inputs and outputs and automatically adapts to the stage and the flow of inputs and outputs. This system stores this input and output data in local memory. - Communications subsystem: Its mission is to send the values of inputs and outputs stored in the algorithmic subsystem to an external computer system as well as Send device status information, images or even real-time video.

This subsystem allows communication through different current standards and protocols. These different types of communication between the device and the external computer system are characterized by two factors: transmission medium, communication distance:

The different types of communication available to the device include two groups in terms of communication means: 1. Communication via physical support: the device has the ability to communicate with the external computer system through a physical medium that can be electrical medium (copper, coaxial cable ...) or optical medium (optical fiber, ...). 2. Wireless communication: the device has the ability to communicate with the external computer system wirelessly in different frequency bands (such as 2.4 GHz, 1.8 GHz, 900 MHZ). On the other hand, the different types of communication available to the device are divided into three groups in terms of communication distance: 1. Average distance: the device communicates with an external computer system located hundreds of meters away. 2. Long distance: the device communicates with an external computer system located thousands of meters away. 3. World distance: the device communicates with a computer system located anywhere in the world with coverage of a telephone operator. The algorithm on which the detection is based, the determination of the direction of passage and the counting of the objects (or in their case people) is composed of the following blocks: 1. Digital treatment of the image: The image is processed and They get a set of features and signals. 2. Neural networks A neural network is implemented in which the inputs are the characteristics and the signals obtained from the digital treatment of the image. The outputs of the neural network define the inputs and outputs.

These two blocks are implemented in the microprocessor which is the one that performs all the computing and determines the count of inputs and outputs.

The flowchart of the algorithm is depicted in Figure 4, where DM represents motion detection, FS signal filtering, EC feature extraction, RPIA recognition of artificial intelligence patterns and

NE / NS the number of inputs and the number of outputs.

The device has the ability to communicate with an external computer system in which a control program is executed that is able to detect the non-functioning of the device, as well as alarms, automatically (without user intervention) and periodically (the period is configurable: 15 minutes, 30 minutes, 45 minutes or any time interval desirable by the user).

The device has the ability to incorporate a web page server. In this way it is possible to connect from the Internet to the web page that the device is serving in real time and view the account of inputs and outputs from anywhere in the world that has an Internet connection.

In summary and based on the aforementioned characteristics, the operation is as explained in Figure 6:

People or objects pass under the device (the preferred installation position is zenith), so that they are illuminated (if the application needs it) by the built-in infrared lighting system, thus capturing its Cl image.

The signal obtained by the image captor Cl is sent to the microcontroller, which in a first phase performs a digital processing of the PDI signal, which avoids the changes of stage and lighting, and then that data is evaluated by the artificial intelligence module MINE, which includes an expert system embedded in the device that establishes the unequivocal determination of what has happened as well as its sense of passage, determining if it is an object (or in its case person) OBJ and if it is input or output I / O, storing the data in the corresponding local storage ALE or ALS.

The embedded expert system has been obtained in a previous phase of high-level learning and is likely to be improved or expanded to incorporate more decision sets.

The device includes a memory where the data obtained from the evaluation are stored at predetermined time intervals, which can then be downloaded through a communications port to be exploited in an external information system.

The system evaluates in real time the functioning of the evaluation algorithm that best suits the traffic conditions of the scene you are observing, which is capable of adapting autonomously and independently, also to different traffic conditions for Get the best efficiency.

The resulting system is a completely autonomous device, which detects objects (or people where appropriate), determines their sense of passage and has the ability to automatically adapt to changes in the scene such as natural light changes, or furniture changes. This system communicates with external computer systems in which a control program is executed that has the ability to detect device failures and present alarms in real time associated with possible failures, images captured by the devices and even present in real time a sequence of images (video) of sufficient quality to evaluate the number and meaning of the objects (or persons, if any) present in the scene.

Claims

1- Autonomous influx control system using artificial intelligence, based on the analysis of the signals captured by an image sensor that are sent to a microprocessor, characterized by an expert system based on an algorithm for bidirectional input and output counting, With ability to store data internally for a very long period of time.

2- Autonomous influx control system by artificial intelligence according to the preceding claim, characterized in that the algorithm resident in the microcontroller is an embedded expert system obtained from a high level prior learning.

3- Autonomous influx control system by artificial intelligence according to any of the preceding claims, characterized in that it does not need to make recordings of the scenario in which it will work .. 4- Autonomous influx control system by artificial intelligence according to any of the claims previous, characterized in that it can adapt automatically to any change of scenery or lighting. 5- Autonomous influx control system by artificial intelligence according to any of the preceding claims, characterized in that the system operates with natural light and infrared light for night vision, the infrared module being integrated in the device itself. 6- Autonomous influx control system by artificial intelligence according to any of the preceding claims, characterized in that the Algorithm performs the digital treatment of the image through neural networks, within the microprocessor itself. - Autonomous influx control system by artificial intelligence according to any of the preceding claims, characterized in that the system includes a communications subsystem that communicates by physical or wireless transmission means, at unlimited distances. - Autonomous influx control system using artificial intelligence according to any of the preceding claims, characterized in that the system counts inputs and outputs and stores them in the device's own memory for several months. - Autonomous influx control system by artificial intelligence according to any of the preceding claims, characterized in that the system is connected to a control program capable of reflecting the system information of the device and allows detecting both the non-functioning of the device and its alarms 0- Autonomous influx control system by artificial intelligence according to any of the preceding claims, characterized in that the system detects whether there is a low flow of inputs or not, determining the area occupied by an object (or person, if applicable) and adapting its mode of operation depending on that flow. 1- Autonomous influx control system using artificial intelligence according to any of the preceding claims, characterized in that it has inputs and outputs in a coverage area, being able to count in very large access areas through several systems operating in cooperation, with high reliability . - Autonomous influx control system by artificial intelligence according to any of the preceding claims, characterized in that its small size and consumption allows it to be operative with battery power in mobile systems.

- Autonomous influx control system by artificial intelligence according to any of the preceding claims, characterized in that the system can transmit frames and video to the control program for remote verification and verification - Autonomous influx control system by artificial intelligence according to any of the claims Previous, characterized in that the system has the possibility of incorporating a web page server to check the account of inputs and outputs in real time from anywhere. - Autonomous influx control system by artificial intelligence according to any of the preceding claims, characterized in that the system is configured easily and simply by only 6 parameters.