PT107559B - COMPUTATIONAL VISION SYSTEM OF MODULAR ARCHITECTURE AND RESPECTIVE APPLICATION PROGRAMMING INTERFACE METHOD - Google Patents

Description

DESCRIPTION

Computer vision system with modular architecture and respective application programming interface method

Technical domain of the invention

The present invention refers to a modular and scalable architecture and the respective application programming interface (from English Application Programming InterfaceAPI), which creates an abstraction layer to unify access to the different sensors and / or algorithms used in solutions based on computer vision.

Background of the Invention

The need to measure and characterize the population of a given location exists and has grown more and more these days. The information that can be collected from a specific audience is central to businesses that depend on the knowledge that organizations are able to obtain from their potential customers.

Most of the time, the measurement and characterization of audiences and the profile of potential consumers is carried out by people, in the form of surveys and market studies, which raises two fundamental problems: the introduction of human error and the high costs of human resources. Several efforts have been made to solve these problems, with special emphasis on the recent trend to automate the process of acquiring demographic data, using computer vision solutions.

There are already several patent documents that present new solutions.

US 7,987,111 describes a system for the demographic characterization of a commercial space using various computer vision technologies. It encompasses a huge number of algorithms to extract multiple characteristics from the audience but it does not specify how the sensors and algorithms transfer data between themselves, nor the underlying software architecture. It also does not provide for situations in which a given configuration is required to mutate or climb, to adapt to new business needs.

US 7,921,036 describes a system for displaying content on a screen based on automatic recognition of predefined demographic characteristics, using a sequence of images from a video stream. The contents are selected from a previously defined set and according to the extracted attributes. This system consists of two components: hardware, which captures images, and software, which processes and extracts characteristics from the audience. However, the way in which the two components communicate with each other is not specified.

US 8,165,386 describes a system for measuring audiences also using video streams. This system uses a modular and dynamic hardware architecture, but in a limited space, since it is an embedded system. Modularity is achieved at the level of the components that make up the computational unit.

US 2011/0239245 describes methods and architectures for measuring audiences. The disclosed system, despite being modular, only guarantees scalability and has the main focus on the way the data of the various devices are aggregated.

US 2008/0294475 and WO 2008/134562 describe a method for determining the flow of consumers in a commercial space and relating it to the commercial activity performed in that same space. It does not aim to specify the solution at the hardware and / or software level, but at the conceptual level. The document does not yet specify the methods for monitoring the flow of consumers, who may be instructed to carry out this measurement, or be able to use fully automatic methods.

Finally, the US document

2012/0072939 describes a system for measuring an audience's reaction to multimedia content.

The document focuses exclusively on the continuous measurement of the emotions of the recipient of the content and its changes, during the time of exposure to it.

All the inventions described in the documents listed above seek, in some way, to automate the process of acquiring demographic data. However, its realization involves many challenges, from the point of view of developing solutions. First, the sensors used are very diverse, either from the point of view of their type, that is, Web cameras, analog, digital or 3D cameras, or from their manufacturers and their respective control software. Second, the algorithms are also diverse and for the most part do not use common standards or protocols for entering and leaving your data. Finally, given the complexity of these algorithms, it is sometimes necessary to distribute the processing and separate the acquisition of the images from their processing and presentation. The present invention aims to overcome the limitations of the state of the art described above.

Summary of the invention

The present invention consists of a modular and scalable architecture and the API that implements it, which creates an abstraction layer to unify access to the different peripherals and / or algorithms used in solutions based on computer vision, regardless of the server where the peripherals and / or algorithms are linked and / or running and the platform where the solutions are developed.

The solutions are based on a hardware architecture that comprises only a main server, computer vision peripherals and presentation device or a network of servers connected through a network switch, switch, each with its peripherals and presentation devices.

At the software level, the present invention defines the architecture that consists of a set of configurable modules that communicate with each other, using the same protocol to debit, debit consume or just consume data from peripherals and / or computer vision algorithms.

These modules can be configured and grouped in the metadata required to produce exactly the data and for the development of the solution.

In any adopted hardware architecture, there is always a server, called the main server, which holds a table of module configurations and the connections between the modules of all servers that make up the solution.

The architecture described in this invention includes an input module, called an event generator, which connects to all modules running in a given environment and correlates and stores the audience data for further processing or statistical analysis.

A computer vision system with a modular architecture is described, comprising a main computer server and a set of computer vision modules that publish their ability to send and / or receive data to and / or selected from a set of algorithms hardware and computer vision, characterized in that the set of hardware and computer vision algorithm definitions and said module connections between modules are stored on said main server, and in which a module connection between any of the two referred modules, a source module and a target module, are established peer-to-peer, as soon as the target module definitions are obtained by the source module from the main server.

In one embodiment, the set of computer vision modules runs on the same physical server, using the shared memory of the physical server for said module connections.

In one embodiment, in which the set of computer vision modules is executed on different physical servers, using network sockets for the referred module connections.

An embodiment comprises different computer servers, where any one of the different computer servers referred to has the ability to automatically assume, in a controlled manner, the role of primary server, in the event that the primary server stops responding.

In one embodiment, the different referred servers are referenced in an ordered list stored on said main server, said list being ordered according to the order in which said servers will assume the role of main server, in case the main server stops responding .

In one embodiment, said main server is configured in such a way that, upon receiving a configuration request from any other server of the different referred servers, it stores said other server in the second position in the referred ordered list, making said other server the next candidate for the role of primary server.

In one embodiment, the main server is the same computer server or one of the computer servers, where some or all modules based on computer vision are executed.

One embodiment comprises an application programming interface, API, to instantiate the aforementioned computer vision modules selected from said set of vision algorithms in hardware and computer; to turn on and off said instantiated computer vision modules; and to finish the instantiated computer vision modules.

In an embodiment, in which each of these computer vision modules is selected from input modules, output modules or input-output modules.

In one embodiment, an input module comprises a computer vision input or inputs; wherein an output module comprises a computer vision output or outputs; wherein an input-output module comprises a computer vision input or inputs and also comprises computer vision output or outputs.

In a realization each input referred and each output referred is individually addressable per one identifier. In a realization each input referred and each output referred is individually addressable per one identifier.

A computer vision module for use in a computer vision system with a modular architecture is also described, the system comprising a main computer server and a set of computer vision modules, in which the said modules publish their capabilities send and / or receive data to and / or selected from a set of hardware and computer vision algorithms, characterized in that the set of hardware and computer vision algorithms definitions and said module connections between modules are stored on said main server, and in which a module connection between any of the two referred modules, a source module and a destination module, is established peer-to-peer, once the definitions of the destination module are obtained by the source module from the main server.

A method of operating a computer vision system with a modular architecture is also described, the system comprising a main computer server and a set of computer vision modules that publish its ability to send and / or receive data to and / or selected from a set of hardware and computer vision algorithms, characterized in that said method comprises: storage of the set of hardware and computer vision algorithm definitions and the said module connections between the modules on said server main;

and peer-to-peer establishment of a module connection between any of the two modules mentioned, a source module and a destination module, once the definitions of the destination module are obtained by the source module from the main server; use, for computer vision modules that run on the same physical server, of the shared memory of the physical server for said module connections; use, for computer vision modules that run on different physical servers, of network sockets for said module connections; establishing the connection between any of the two modules mentioned, by automatically switching between shared memory and sockets, depending on whether it is running on the same physical server or on different physical servers; and automatic data compression for different physical servers.

An embodiment comprises any of the different computer servers referred to, automatically assuming, principally, in a controlled manner, the role of server in case the main server fails to respond.

One realization comprises an application programming interface, API, which comprises instantiation of the referred computer vision modules selected from the set of algorithm definitions in hardware and in computer; connection and disconnection of the instantiated computer-based modules referred to; connection and disconnection of the instantiated computer-based modules referred to.

Description of the Figures

For an easier understanding of the invention, the figures are attached, which represent preferred embodiments of the invention which, however, are not intended to limit the object of the present invention.

Fig. 1 represents a diagram of a hardware configuration in accordance with a defined architecture.

Fig. 2 represents fundamental elements of a software architecture.

Fig. 3 represents a detailed view of the main server, more specifically its configuration table.

Fig. 4 represents a sequence of messages exchanged between servers in establishing a connection between two modules.

Detailed description of the invention

The present invention is now described in detail, using the figures attached to this document for this purpose.

Fig.

presents a block diagram where the hardware components of a possible solution that implement the architecture defined in the invention are identified.

Hardware components are typically used in commerce. The solution contains three servers 2, one of which is the main server 1, connected by a network switch 3, a camera 4, which can be IP, analog or USB, a sensor

3D 5 and a display device 6.

There is no maximum limit for the number of servers 2, but they must all belong to the same local network. There is at least one server 2 with the role of primary server 1. There is also no maximum or minimum limit for computer vision peripherals 4, 5 per solution or per server 2, this limit being imposed by the processing capacity of the server 2 in question .

Fig.

represents fundamental elements of a software architecture. There are output modules 7, input modules and / or output modules 8, and input modules.

Output modules 7 have only outputs 10 and acquire data from computer vision peripherals 4, 5, such as web cameras, IP cameras, analog cameras, 3D sensors, and others, and make them available on their outputs 10; the input and / or output modules 8, which have at least one input 11 and at least one output 10, are responsible for processing data from one or more inputs 11, transforming them into a new type of data, using the various algorithms of computer vision, such as the detection of gender, age, height, emotions, look direction, pose, color composition of an object, analysis of the direction of movement and others, and make them available in your outputs 10; and input modules 9, which only consume data from their inputs 11, processed by other modules, with the aim of storing, triggering some action, generating and presenting multimedia content, among other applications, in a presentation device 6.

In the context of the described architecture, a module is a process that runs on a server 2.

Each module has a unique identification key for which it is known in the solution to be inserted.

A module is an instance of a controller, driver, which can be seen as the computer program developed in accordance with the architecture and using the presented programming interface.

Each driver specifies a set of parameters that the modules or instances must configure, and which change their functioning.

A driver also specifies the set of outputs 10 and inputs that the modules, from instantiated, have.

Each module can be configured to turn on or off these outputs 10 or inputs 11. Each output 10 and input 11 has a unique identification key for which it is known in the module to which it belongs. Outputs 10 and inputs have a format, that is, a name assigned to the data structure that must be known by the various modules and that specifies the size and types of data that will be transferred between them.

The data formats are not covered by the invention, and are at the discretion of the programmer who develops the driver, which must be public, so that other programmers can consume them in their modules.

Fig. 3 represents the configuration table 12 of the main server 1. This table contains the configuration of all modules 7, 8 and 9, of the servers 2 where they are being executed, of the inputs 11 and outputs 10, and of the connections between they. A connection can only be established between the output 10 of one module and the input 11 of another, and both have the same format.

The configuration table keeps servers 2 numbered in an increasing order, in the order in which they will assume the role of primary server 1, which is always in first place, in case the server fails to respond to requests.

Primary server 1 propagates its configuration table 12 to all servers 2, whenever it receives a request to change any configuration. This ensures that all servers 2 maintain the most recent configuration and are prepared to replace it if the primary server stops responding.

Whenever the primary server receives a configuration request from a server 2, the primary server 1

Put the in second place in the order of succession 13 before propagate The table in settings 12, making this server 2 in the next candidate for server role main 1 Fig. 4 represents at messages 15, 16, 17, 18, 19, 20, 21, 22, 23 exchanged to establish the connection between two modules, involving always the server main 1 for the

acquisition of the configurations of both modules. This flow is completely transparent to the programmer and is implemented in its entirety by the programming interface presented. From the point of view of the programming interface, all servers 2 are the same, in that they have a configuration table 12 and a sequence order 13 that indicates the network address of the current main server 1. In the case of the main server 1, the order of succession 13 must indicate itself first.

When a module starts on a given server 2, the order of succession 13 of the server 2 itself is consulted, to obtain the address of the main server, 1, 15. Then, the module establishes a connection with the main server 1 and registers itself , or update, if you have already registered, your information on the main server 16. The information sent is a hardware key, the physical and network address. In this way, the main server 1 starts to include in its configuration table current information about the location on the network of any server 2 that has executed a module in the entire solution. If the main server 1 does not respond, the request is sent to the second server 2 in succession order 13, which assumes the role of main server 1, updating a lineage identifier. This identifier guarantees that if the old main server 1 responds again, as it has a different identifier, it will not assume the role of main server 1 again, without first updating its configuration table 12.

After the configurations are updated, the main server changes the order of succession to include server 2 from which it received the registration request in second place. Thus, this server, having demonstrated that it is active, becomes the next candidate for the role of primary server 1, 17. Immediately after this change, primary server 1 propagates the configuration table 12 to all servers 2 of solution 18, making them all able to assume their role, in the event of this fail to respond.

In the next step, the module establishes a new connection with the main server 1 and asks for information about itself 19. In response, the module obtains the configuration parameters, its inputs 11 and outputs 10 and the name of the modules to which it connects 20.

In the fourth step, the module establishes a new connection with the main server 1 and asks for information about the module to be connected 21. In response, it obtains the network address of the server 2 where it is being executed, as well as the parameters of the connection 22. Main server 1 holds this information because the destination module of the connection performed the same procedure, and therefore registered in the configuration table 12. Finally, the source module performs the peer-to-peer connection with the target module 23. The technique used to establish communication between the modules is selected automatically. From the moment the source module holds information about server 2 where it executes the target module

22, select shared memory, if server 2 is the same where the source module is being executed, otherwise it selects the sockets. Automatic data compression works in the same way, being used only in case the modules are executed on 2 different servers.

To ensure a simple, flexible implementation that promotes scalability, the present invention uses some techniques, such as the centralized configuration of the system which, in the case of using a hardware architecture that comprises several servers, is unified in the main server 1 the solution configuration;

automatic propagation of configurations to all servers 2 of the solution automatic naming of a new primary server 1, in case of failure in the server in use; the automatic redirection of configuration requests to the main server 1; automatic module discovery; the automatic choice of the communication protocol between sockets and shared memory, depending on the server where the intervening modules are being executed; automatic compression and decompression of data to ensure optimal use of the network.

The achievements described above are combinable with each other. The following claims further define preferred embodiments of the present invention.

Claims (16)

1. System architecture holds a responsible CLAIMS of computer vision that comprises a modular consisting of: a main server that table of system configuration; a module for the correlation of data acquired / processed by the set of modules that make up the system; a set of modules that publish their capabilities to send and / or receive data, to and / or to a set of hardware and computer vision algorithms; characterized by the definitions and connections between the modules are stored on a main server and the connection between modules is established directly between them, with automatic choice of communication protocol, after the definitions of the destination module are obtained by the source module of the connection, from the main server. 2. Computer vision system, according to the claim 1, in which the set of computer vision modules is executed using the shared memory of the physical server (2) for said module connections. 3. Computer vision system, according to claim 1, in which the set of computer vision modules is executed on different physical servers (2), using network sockets for said module connections. 4. Computer vision system, according to claim 1, comprising different computer servers, in which any of the different computer servers referred to (1, 2) has the ability to automatically assume, in a controlled manner, the role of main server (1), in case the main server (1) stops responding. 5. Computer vision system, according to the previous claim, in which the different servers referenced in an ordered list stored in said one, said list being ordered according to the order in which said servers (1, will assume the role of server main answer. 6. Computer vision system, according to the previous claim, in which said main server (1) is configured in such a way that, upon receiving a configuration request from any other server of the different servers (1, 2) referred to, stores said other server (2) in the second position in said ordered list, making said other server (2) the next candidate for the role of primary server (1). 7. Computer vision system, according to claim 1, in which the main server (1) is the same computer server or one of the computer servers (1,2), where some or all modules based on computer vision are performed. 8. Computer vision system, according to claim 1, comprising an application programming interface, API, to instantiate the referred computer vision modules selected from said set of vision algorithms in hardware and in computer; to turn on and off said instantiated computer vision modules; and to finish the instantiated computer vision modules. 9. Computer vision system, according to claim 1, in which each of these computer vision modules is selected from input modules (6), output modules (4, 5) or input-output modules. 10. Computer vision system, according to the previous claim, in which an input module (6) comprises a computer vision input or inputs; wherein an output module (4, 5) comprises a computer vision output or outputs; wherein an input-output module comprises a computer vision input or inputs and also comprises computer vision output or outputs. 11. Computer vision system, according to the previous claim, in which each referred input and each referred output is individually addressable by an identifier. 12. Computer vision system, according to the previous claim, in which each referred input and each referred output is individually addressable by an identifier. 13. Computer vision module for use in a computer vision system with a modular architecture, said system comprising a main computer server (1) and a set of computer vision modules referred to, in which said modules publish their capabilities of send and / or receive data to and / or selected from a set of hardware and computer vision algorithms, characterized in that the set of hardware and computer vision algorithms definitions and said module connections between modules are stored on said main server (1), and in which a module connection between any of the two referred modules, a source module and a destination module, is established peer-to-peer, as soon as the definitions of the destination module are obtained by the source module from the main server (1). 14. Method of operation of a computer vision system with a modular architecture, the system comprising a main computer server (1) and a set of computer vision modules that publish its capabilities to send and / or receive data to and / or selected from a set of hardware and computer vision algorithms, characterized in that the referred method comprises: - storage of the set of vision algorithm definitions in hardware and in computer and the referred module connections between the modules in said main server (1); and - peer-to-peer establishment of a module connection between any of the two mentioned modules, a source module and a destination module, once the definitions of the destination module are obtained by the source module from the main server ( 1); - use, for computer vision modules that run on the same physical server (2), of the shared memory of the physical server (2) for said module connections; - use, for computer vision modules that run on different physical servers (2), of network sockets for said module connections; - establishing the connection between any of the two modules mentioned, by automatically switching between shared memory and sockets, depending on whether it is running on the same physical server (2) or on different physical servers (2); and - automatic data compression in the case of different physical servers (2). 15. Method according to claim 14, which comprises any one of the different computer servers referred to (1, 2), automatically assuming, in a controlled way, the role of primary server (1), in case the server main (1) stop responding. 16. Method, according to claim 14, for an application programming interface, API, comprising: instantiation of the referred computer vision modules selected from the set of algorithm definitions in hardware and in computer; - connection and disconnection of the instantiated computer-based modules referred to; - connection and disconnection of the instantiated computer-based modules referred to.