WO2021058100A1

WO2021058100A1 - Multisensory learning system for traffic prediction

Info

Publication number: WO2021058100A1
Application number: PCT/EP2019/075909
Authority: WO
Inventors: Cristian AXENIE; Radu TUDORAN; Stefano BORTOLI; Mohamad Al Hajj HASSAN; Goetz BRASCHE
Original assignee: Huawei Technologies Co., Ltd.
Priority date: 2019-09-25
Filing date: 2019-09-25
Publication date: 2021-04-01
Also published as: CN114287023A; CN114287023B

Abstract

A road traffic flow prediction system configured to receive data from a first sensor being configured to acquire traffic flow data in a first data domain and from second and third sensors being configured to acquire data in a second data domain, the first and second sensors being located at a first location and the third sensor being located at a second location spatially remote from the first location, wherein the system is configured to: receive data from the first sensor, the second sensor and the third sensor; and process the data received from the third sensor in dependence on the data received from the first and second sensors to estimate traffic flow at the second location.

Description

MULTISENSORY LEARNING SYSTEM FOR TRAFFIC PREDICTION

FIELD OF THE INVENTION This invention relates to traffic control systems, particularly to a system to optimize traffic prediction by leveraging a multisensory correlation learning and fusion engine that applies statistical and machine learning for big data processing.

BACKGROUND Traffic congestion poses serious challenges for city infrastructure facilities and also affects the socio-economic lives of residents due to time wasted whilst waiting in traffic.

Despite the multiple characteristics describing traffic-flow models, one can follow a general assumption that traffic-flow data is a timeseries, which is a time indexed sequence of values (i.e. tuples containing various types of data: number of cars, speed of cars, etc.). The traffic data may be collected from various sources, such as cameras or street induction loop sensors, pollution sensors, noise sensors and weather sensors in a chronologically ordered fashion. Such data is generated continuously by the sensors measuring the traffic metrics as time progresses. These sensors are typically spatially distributed in urban infrastructures. Various approaches for traffic modelling and control, such as Macro- or Microscopic Model, Filtering Models and other combination models have been developed. Such research is lacking real-world deployment for reasons including low forecast precision, inability to handle correlations, strict demand for data, lack of expressivity or understanding of internal mechanisms and intolerable time cost for prediction. The key reason that state-of-the-art traffic prediction models fail in practice is that they fail to fully exploit the multi-sensory sources available. At the same time, cameras and induction loops which directly measure the number of cars at a particular location are expensive, intrusive, and hard to maintain and need algorithms to process data, increasing the costs for such a solution, which may make it infeasible at scale. Prior models also fail to fully utilize the unique information provided by transportation networks, including dynamic space factors, namely the topological structure and intrinsic temporal flow of vehicles and their correlations, which can be sensed with multiple cheap sensors available in urban agglomerates, such as pollution sensors (for example, NO2, NO, PM10), noise sensors, weather sensors (for example, humidity, rain duration), online data and cellular line data. These are all available and alternative sources of data indirectly characterizing traffic flow information.

Typical statistical and machine learning models for timeseries prediction, such as the Auto Regressive Moving Average family (AR, MA, ARMA, ARIMA), Bayesian Inference and Regression Trees, can only model and predict a single dimensional timeseries. Such predictor models assume that the correlation in data can be adequately described by parameters that are globally fixed temporally. Furthermore, they are not intrinsically extensible to multivariate spatially correlated predictions, making them inadequate for cases in which the correlations among data are dynamic and heterogeneous, such as road traffic data.

In the field of machine learning based timeseries modelling and prediction, a variety of approaches try to compensate for the limitations of typical statistical approaches and supplement them with novel techniques to increase expressivity and flexibility. Prior art techniques generally focus on offline procedures employing Bayesian integration, Support- Vector Regression, Nonlinear Least Squares, Ensemble Methods, and Expert Systems.

The approach described in R. Fonseca et al., "Automatic Model Selection in Ensembles for Time Series Forecasting", IEEE Transactions, vol. 14, no. 8, pp. 3811-3819, Aug. 2016, postulates that diversity and accuracy of the involved models are the most important factors to be considered when selecting them and looked at a new method for timeseries prediction based on a neural network and meta-features. This method automatically adjusts the required balance between diversity and accuracy in the selection of the forecasters and provided good results on highly nonlinear timeseries. However, the offline unsupervised training and the complex model updates make this approach intractable in a real-time scenario. Another approach described in M. Adya, et al., “Development and validation of a rule-based timeseries complexity scoring technique to support design of adaptive forecasting DSS”, Decision Support Systems, Volume 83, 2016, postulates that understanding timeseries complexity can enable design of adaptive forecasting decision support systems to positively support forecasting behaviors and accuracy of outcomes. This system is based on a rule- based complexity scoring technique that generates a complexity score for timeseries using twelve rules that rely on fourteen features of series. Despite the interesting approach that embeds expert system rules in modelling, the high number of rules and features needed for the decision system to select a model makes it hard to fit in the low resource and time budget that real-time processing obeys. Moreover, this Forecasting Decision Support System (FDSS) considers complex features, such as discontinuity levels, direction of basic trend, level discontinuities and domain knowledge, which can indeed increase adaptability, but also the complexity and computation time. These metrics are not computed incrementally, rather offline, such that the computation time depends on the length of the timeseries, but without any real-time prediction requirements.

In the field of traffic prediction, to satisfy the demand of traffic congestion early warning, many studies have developed methods for real-time traffic flow state identification and prediction based on big data-driven theory. Traffic big data holds several characteristics, such as temporal correlation, spatial correlation, historical correlation, and multistate.

The approach described in Hua-pu Lu, Zhi-yuan Sun, and Wen-cong Qu, “Big Data-Driven Based Real-Time Traffic Flow State Identification and Prediction”, Discrete Dynamics in Nature and Society, vol. 2015, Article ID 284906, 11 pages, 2015, quantifies traffic flow state, the basis of traffic flow state identification, through the use of a SAGA-FCM (simulated annealing genetic algorithm based fuzzy c-means) based traffic clustering model. Considering simple calculation and predictive accuracy, a bi-level optimization model for regional traffic flow correlation analysis was established to predict traffic flow parameters based on temporal-spatial-historical correlation. Despite its flexibility, the model assumed many assumptions on the spatio-temporal parametrization of the model for a gain of just 10% in traffic decongestion.

Another method described in T. L. Pan, A. Sumalee, R. X. Zhong and N. Indra-payoong, "Short-Term Traffic State Prediction Based on Temporal-Spatial Correlation", in IEEE Transactions on Intelligent Transportation Systems, vol. 14, no. 3, pp. 1242-1254, Sept. 2013, employs the stochastic cell transmission model (SCTM) incorporating the covariance structure calibrated from the spatial correlation analysis for probabilistic traffic state evaluation. Despite the heavy computations behind the model, the system had overall MAPE (Maximum Absolute Percentage Error) of all prediction around 16.2%, which was unfortunately only obtained in static, empirical studies (and not real-world scenarios).

Using a more real-time oriented approach, the study in Canaud, M., Mihaylova, L, Sau, J. et al. (2013), “Probabilty hypothesis density filtering for real-time traffic state estimation and prediction”, Networks and Heterogeneous Media (NHM), 8 (3), 825-842, investigates the potential of Probability Hypothesis Density (PHD) filters for real-time traffic state estimation. This approach uses a Cell Transmission Model (CTM) coupled with the PHD filter, taking into account the measurement origin uncertainty and showed that this can provide high accuracy in a traffic setting and real-time computational costs. Despite its attractive properties, the model was only used on a high-way setup without a large number of lanes and intersections that usually bring the complexity describing real-world traffic.

US 2002/0067292 A1 describes the use of sensor systems for environmental sensing for an intelligent scene interpretation. To determine the position of a motor vehicle with respect to the traffic lane, the data from a digital road map coupled with a navigation system is fused with the data delivered by a distance-resolving sensor. In this context, the signature and the distance-related variation of the received signal of the distance-resolving sensor are evaluated to determine the distance from the road edge.

It is desirable to develop a system and method for traffic prediction that overcomes these problems. SUMMARY OF THE INVENTION

According to a first aspect there is provided a road traffic flow prediction system configured to receive data from a first sensor being configured to acquire traffic flow data in a first data domain and from second and third sensors being configured to acquire data in a second data domain, the first and second sensors being located at a first location and the third sensor being located at a second location spatially remote from the first location, wherein the system is configured to: receive data from the first sensor, the second sensor and the third sensor; and process the data received from the third sensor in dependence on the data received from the first and second sensors to estimate traffic flow at the second location.

The data received from each of the sensors may comprise a timeseries of values. This may allow data generated continuously by the sensors to measure traffic metrics as time progresses.

The first sensor may be a camera and the first data domain may be one or more images of vehicles at the location. This may allow the relationship between traffic flow and other types of sensor data to be learned.

The second and third sensors may be configured to acquire data relating to the level of an environmental property at the first and second locations respectively. This may allow the relationship between traffic flow and data that can be collected from ubiquitous and/or cheaper sensors to be learned.

Each of the second and third sensors may comprise one of a weather sensor, a pollution sensor, a noise sensor, a CO sensor, and an induction loop. These are convenient sensor implementations for which a relationship between the sensor data and traffic flow may be learned. The third sensor may be configured to acquire data relating to the same environmental property as the second sensor. This may allow a relationship learned at one location to be transferred to a second location such that data from the third sensor may be used to infer traffic flow at the second location.

The system may be configured to perform the said processing by implementing a learned artificial intelligence model. The artificial intelligence model may be a neural network. This may be a convenient implementation.

The system may be configured to learn a mapping from the second data domain to the first data domain. The learned mapping may be applied to other locations to predict traffic flow where a direct measurement of the number of cars is not available. The system may be configured to iteratively update the parameters of the model over time in dependence on the data received from the first and second sensors. This may improve the accuracy for the learned relationship.

The system may be further configured to process the data received from the third sensor in dependence on data received from at least one other sensor to estimate traffic flow at the second location. This may allow further types of sensors to be used to increase robustness of the predictions.

The system may be further configured to: receive data from a fourth sensor being located at the first location and from a fifth sensor being located at the second location, the fourth and fifth sensors being configured to acquire data in a third data domain; and process the data received from the third and fifth sensors in dependence on the data received from the first, second and fourth sensors to estimate traffic flow at the second location.

The first and second locations may be first and second traffic intersections respectively. The system may be further configured generate a time plan for respective sets of traffic lights at the first and second intersections. The system may be therefore implemented to manage traffic in urban environments.

According to a second aspect there is provided a method for implementation at a road traffic flow prediction system configured to receive data from a first sensor being configured to acquire traffic flow data in a first data domain and from second and third sensors being configured to acquire data in a second data domain, the first and second sensors being located at a first location and the third sensor being located at a second location spatially remote from the first location, the method comprising: receiving data from the first sensor, the second sensor and the third sensor; and processing the data received from the third sensor in dependence on the data received from the first and second sensors to estimate traffic flow at the second location.

The said processing may comprise implementing a learned artificial intelligence model. The artificial intelligence model may be a neural network. This may be a convenient implementation. There may also be provided a computer program which, when executed by a computer, causes the computer to perform the methods described above. The computer program may be stored on a non-transitory computer-readable storage medium.

BRIEF DESCRIPTION OF THE FIGURES

The present invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

Fig. 1 shows a graphical depiction of traffic control systems and the typical sensory data available at different locations.

Fig. 2 shows an overview of a generic multisensory processing setup that can be used for traffic prediction.

Fig. 3 shows an example using sensors located at two spatially remote traffic intersections. Fig. 4 shows an overview of the multisensory learning traffic predictor architecture.

Fig. 5 shows the modules of the multisensory learning system.

Fig. 6 illustrates timeseries correlation learning in the system.

Fig. 7 shows a further example of correlation learning in the system.

Fig. 8 illustrates extensibility features of the system. Figs. 9(a)-(c) illustrate inference and prediction in the system. Fig. 9(a) shows the inputs, Fig. 9(b) shows the learnt relations, and Fig. (c) shows the decoded relation.

Fig. 10 illustrates the runtime functionality of the system.

Fig. 11 illustrates traffic prediction using a model learnt at one location to predict traffic flow at other spatially remote locations. Figs. 12(a) and 12(b) illustrate learnt sensory relationships. Fig. 12(a) illustrates a learnt relationship between NO2 level and humidity. Fig. 12(b) illustrates a learnt relationship between NO2 level and vehicle count.

Fig. 13 illustrates inferring data from missing sensors. Fig. 14 shows an example of a method for implementation at a road traffic flow prediction system.

DETAILED DESCRIPTION OF THE INVENTION

The present invention relates to a processing system configured to collect traffic data from different types of sensors located at an intersection and model the data by extracting correlations between the available sensors, and make predictions about the future traffic flow. Such a processing system can be used to control traffic-light sequences for improved road traffic flow. T raffic flow may be defined as the number of vehicles passing a particular point during a particular time period (for example, the number of vehicles per minute), or the number of vehicles passing through an intersection per traffic light cycle (i.e. the number of vehicles passing during the time that a traffic light at the intersection was green). Therefore, the time resolution for assessing the number of vehicles may be a unit time period or the duration of a traffic light cycle.

As shown in Fig. 1 , different road traffic locations may have different types of sensors located at them. For example, at site 1 , indicated generally at 101 , variables including NO level, NO2 level, particle count and humidity are measured by corresponding sensors, or from sensors collecting data that may be processed to determine these quantities. At site 2, indicated generally at 102, variables including NO level, CO level, NO2 level, particle count and humidity are measured by corresponding sensors, or from sensors collecting data that may be processed to determine these quantities. Neither site 1 nor site 2 has a sensor from which the traffic flow (i.e. the number of cars) at the site may be directly measured, such as a camera or an induction loop. Site 3, indicated generally at 103, has a camera located at the site, from which the number of cars may be measured. Additionally, site 3 features other types of sensors to measure variables including the levels of NO2, O3, NO_x, NO, the humidity and rain duration. Therefore, the traffic flow at site 3 may be directly measured via the camera. However, at sites 1 and 2, which do not have a camera or induction loop capable of measuring vehicle count, traffic flow cannot be directly measured.

Another typical scenario illustrating the urban traffic context (i.e. an intersection of two or more roads) is depicted in Fig. 2. At an intersection 201 of roads S1-S4, there is located a camera 202, a noise sensor 203, a humidity sensor 204 and a CO2 sensor 205. Given the measured traffic flow from the camera (i.e. the number of cars passing per lane, shown generally at 206), the system predicts the future flow, shown at 207, and the control timing for the traffic lights, shown at 208, to maximize flow.

As discussed above, employing only camera data at every location or intersection for traffic flow prediction is expensive and requires additional infrastructure, such as specialized image processing software. Therefore, this is not cost efficient to be installed at scale. Moreover, privacy issues are also an impediment. However, a multitude of other available sensors, such as the noise, CO2 and humidity sensors 203, 204, 205, are ubiquitous.

The present invention utilises data from these other types of sensors and exploits all available sources of data describing the traffic situation to improve traffic flow prediction.

As illustrated in Fig. 3, an urban area 300 comprises a series of roads, along which vehicles may travel in at least one direction, and intersections or crossings between intersecting roads. A first sensor 301 is located at a first location at traffic intersection 302. The first sensor is configured to acquire data in a first data domain. The first data domain provides a measure of the number of cars at the intersection, from which traffic flow is determined. Preferably, the first sensor 301 is a camera that collects images of the number of cars at the intersection, from which traffic flow may be determined. Also located at the first location at intersection 302 is a sensor 303 configured to collect data relating to an environmental property. In this example, the sensor 303 is a CO sensor. Located at a second location at intersection 304, spatially remote to the first intersection 302, there is a second CO sensor 305.

As will be described in more detail below, the system learns the relationship between the data collected by the camera 301 and the CO sensor 303. The system then applies the learned relationship to the data collected from the CO sensor 305 at the second location to infer the number of cars, and hence the traffic flow, at the section location.

The system may also learn the relationship between the camera and at least one additional sensor at the first location and use the learned relationship(s) to determine traffic flow at the second location. For example, in Fig. 3, noise sensors 306 and 307 are also present at the first and second locations respectively. The system may receive data from the noise sensor 307 and based on the learned relationship between camera 301 and noise sensor 306, predict traffic flow at the second location (i.e. at intersection 304). The system may also receive data from more than one sensor at the second location and use more than one learned relationship from the first location to form an output of the traffic flow at the second location.

The determined traffic flow can be used to make predictions of traffic flow in the future (for example, hours ahead of the present time) and generate a time plan for respective sets of traffic lights at the first and second intersections, shown at 308 and 309 respectively.

The system therefore aggregates multiple sensory data sources available at different locations to describe and make predictions of the traffic scene in an urban environment. Such a system is a component that interacts with the traffic control system present at each intersection. The system is therefore positioned at the intersection between traffic estimation and modelling and the control component for traffic light sequencing.

Fig. 4 introduces the overall architecture and the data flow of the proposed system. The schema shown in Fig. 4 at 400 comprises several parts that operate in concert on the incoming sensory timeseries data 401.

The first component, shown at 402, is a road traffic flow timeseries representation and modelling module. Being an intrinsically complex process, road traffic flow can be modelled using temporal models that assume data in the form of spatially distributed timeseries describing local variations of a global phenomenon. The module uses 1 D Self-Organizing- Maps (SOMs), which are neural networks capable of encoding sensory timeseries in a distributed activity pattern over a lattice of processing units (i.e. neurons). Each processing unit has a preferred value for which it will issue an output when the value is fed to the network. Such a mechanism allows the system to represent adjacent values in the input sensory space, close to each other in the SOM. Using such a mechanism, the system is able to extract the data distribution from the timeseries by modulating Gaussian functions to encode the distribution (i.e. low-distribution with narrow Gaussian and high-distribution with large Gaussian). The representation and modelling system converges quickly, such that the distribution of the sensory data is recovered for the subsequent correlation learning. The system is therefore lightweight and fast.

The second component, shown at 403, is a multisensory correlation learning system for road traffic prediction. The system uses the distributed encoding of the sensory timeseries, determined by module 402, to extract the temporal co-activation between different types of sensors. This is achieved through a correlation learning mechanism, such as Hebbian Learning. This allows the processing units, or neurons, in each of the input sensory SOMs to strengthen the connection between the sensors based on the co-activation (i.e. the neurons are active at the same time for a certain sample coming from the sensory timeseries). The learning converges fast for each new sensory data sample towards a representation which represents the relationship between the input sensors. The representation is conveniently a matrix of connections, similar to an adjacency matrix. The underlying mechanism is fast and resource-efficient, in terms of the time to converge and memory allocation.

The third component, shown at 404, is a fault tolerant inference and prediction system for road traffic. The learnt relationship between the sensors is subsequently extracted from the matrix representation, taking into account the neuron location in the input encoding system (i.e. the SOM). This process comprises a simple optimization method to find the closest value to the correct value following the mathematical representation of the relation in the learnt matrix representation. This is realized through simple mathematical calculations, for example sum, product, sqrt, which support the overall simple and fast operation. The decoded functional mathematical relation is used to infer the missing sensory data, for example where there is no camera present at the interface, or to predict the correct values when a sensor is faulty, as shown at 405.

The overall architecture is modular and provides an adaptive system suited to model and predict highly heterogeneous timeseries and non-stationary processes. The system can exploit correlations among different non-stationary, deterministic timeseries describing multisensory large-scale (network) phenomena. Using a distributed representation mechanism capable of capturing the shape and temporal distribution of the sensory timeseries, together with a temporal correlation learning mechanism to fuse the data, and a powerful inference mechanism based on simple operations, the system may achieve improved timeseries prediction performance in highly non-stationary problems, such as traffic flow prediction.

The system therefore provides a pipelined processing mechanism that converts timeseries data acquired from the sensors into a representation capable of extracting underlying correlations between sensors in the representation and modelling module, learns correlations among the sensors and fuses them in the multisensory correlation learning module, and infers traffic flow (for example, for locations where there are no cameras, or places where cameras are faulty) or improves the flow estimate (when cameras are present) in the fault tolerant inference and prediction module.

The operation of these elements will now be described in further detail with reference to Fig. 5.

As shown at 501 , the representation and modelling module processes the timeseries data from all sensors, shown at 500, in a distributed encoding process to extract statistics and data distributions from the timeseries.

Modelling of timeseries that are non-stationary and deterministic requires the use of specialized mechanisms that account for their special properties. Moreover, handling multiple such correlated timeseries requires an appropriate substrate to describe their covariance. The module 501 encodes the sensory timeseries 500 using a Distributed Network of Processing Units. The sensory timeseries are encoded in a distributed representation using Self-Organizing Maps (SOMs). This machine learning algorithm is responsible for extracting the statistics of the incoming data and encoding sensory samples in a distributed activity pattern, as shown in Figs. 6(a) and 6(b). This activity pattern is generated such that the neuron closest to the input sample, in terms of its preferred value, will be strongly activated. Activation decays as a function of distance between the input and the preferred value. Using the SOM distributed representation, the model learns the boundaries of the input data, such that, after relaxation, the SOMs provide a topology preserving representation of the input space.

The multisensory correlation learning module 502 extracts sensory correlations from the sensor data using machine learning. In a preferred implementation, the machine learning algorithm used is Hebbian Learning. This algorithm learns correlations between the data collected by the sensors and stores them in an efficient, inexpensive and explainable representation. Such an approach is based on a lightweight neural network-based learning system, with no backpropagation training. It can represent and extract the underlying statistics of the sensory data in the same learning substrate. This allows the system to learn data representation in dependence on extracted statistics and allocate resources accordingly.

The system uses the temporal co-activation pattern to encode correlations among different sensors, as shown for two sensors in Figs. 6(a) and 6(b). In Fig. 6(a), the SOMs of the timeseries data from sensors 1 and 2 are extracted and the relationship between the two sets of data is learned using the machine learning algorithm. In Fig. 6(b), a cross-modal weights matrix is generated to represent the correlation between the two sets of sensory data.

Such an approach has several advantages. For example, it can provide an explainable structure (“no black-box design”) and an easily interpretable output. The approach is able to handle bi-modal, tri-modal, or multi-modal extensions using the same representation and learning substrate. Furthermore, in contrast to many state-of-the-art approaches, in the system described herein there is no need to explicitly encode sensory associations and sensor fusion rules.

In order to illustrate the potential of the system and its underlying mechanisms, the learning of a nonlinear relationship between two sets of sensor data, in this case a third-order power- law relationship, is shown in Figs. 7(a)-(d). Fig. 7(a) shows the input data resembling a third-order power-law relationship and the input data distributions. Fig. 7(b) shows the internal model architecture that determines the cross-modal weights matrix. Fig. 7(c) shows the computation stages. When sensory data is received, the system detects whether the data is from a new sensor that a relationship has not previously been learned for. If the sensor is not new, the mechanism proceeds to the inference phase to predict traffic flow using the sensor data. If the data is from a new sensor, the sensor data is used in training of the model to determine the relationship(s) between the new sensor data and received data from known sensor types. Fig. 7(d) illustrates the learnt correlation between the data from sensors 1 and 2.

The system is able to handle cases where there are arbitrary nonlinear relationships between data from different sensor types and is highly extensible, capable of handling multiple sensors. Fig. 8 illustrates a scenario in a three-dimensional system comprising sensors si, S₂ and S₃ with a tree shaped correlation structure. Fig. 8(a) compares the input data and the decoded learned representation encoded in models m-i, m₂ and m₃, whereas Fig. 8(b) illustrates the learned sensory relations encoded in the neural network weights.

Once the system has learnt the underlying sensory correlations, the system is able to decode the efficient and explainable learnt representation for prediction or inference in the fault tolerant interference and prediction module, shown at 503 in Fig. 5. More precisely, after the learning process, the network stores a stable representation of the relationship between the two sensory inputs considered during training. By considering only one input sensory source (though more than one may be used), the network can infer the corresponding quantity for the missing source using the learned co-activation pattern. The module uses a decoder, which simply computes a term to finely tune the preferred value of the most active (winning) neuron towards a more precise estimate. These computations are simple, for example, sums and products.

Fig. 9 illustrates an example of the inference capabilities of the system. For different sets of input data, Fig. 9(a) shows the input signals and relations, Fig. 9(b) shows the learnt relations, and Fig. (c) shows the decoded relations.

The aforementioned system components act in concert at system runtime and correspond to the functional modules depicted in Fig. 10. The system runtime sequence and the operations performed by the functional modules (blocks) 1001 to 1005 are as follows. In module 1001 , a distributed representation is calculated and statistics of the input sensory timeseries describing the traffic, shown generally at 1000, are extracted. In module 1002, the data statistics of the sensory data 1000 are combined and the underlying sensory relations are learnt using an explainable representation. In module 1003, the sensory correlations from the explainable encoding are decoded. Module 1004 performs predictions for a sensor based on the learnt correlation and other available sensors. Module 1005 performs inference of a missing or faulty sensor based on the learnt relationship and the data from other available sensors. The output of module 1005, shown at 1006, is a timeseries prediction (dotted line), shown compared to the true value (dashed line).

The system can efficiently learn sensory correlations in various traffic scenarios. In the exemplary traffic scenario illustrated in Figs. 11 to 13, the correlation between environmental parameters (for example, NO, O₃, NO₂, NO_x), weather (for example, humidity, rain) and the traffic flow (the number of vehicles, determined by a camera) is learnt at site 3, as shown generally at 1101 in Fig. 11.

Once the correlation has been learned, this learned relationship can be transferred to a different site in order to infer traffic flow in regions where there are no traffic sensors installed that can measure the number of cars directly, but where other types of sensors are present. For example, at sites 1 and 2, as shown generally at 1102 and 1103 respectively, there are no direct traffic flow sensors but there are NO2 and humidity sensors at these locations. Once the system has learned the sensory relations between the different sensors at site 3, it can predict the vehicle count at site 2 using just the NO2 and humidity sensory data collected from site 2, as shown in Fig. 13. The system therefore learns the pair-wise correlations between NO2 and vehicle count (via the camera) and humidity at site 3. The learned relationships between NO2 and humidity and NO2 and vehicle count are illustrated in Figs. 12(a) and (b) respectively. Once learnt at site 3, the correlation between two different sensors types can be used to infer traffic flow in locations such as site 2, where a traffic sensor such as a camera or an induction loop, which directly measures the number of vehicles at the intersection, are not installed. As this location has other types of sensors present, the number of vehicles can be inferred using the leant correlation from site 3.

The output of the system can be applied to the traffic control unit updating the traffic lights at the intersections in the region in order to maximize the traffic flow. The unit can operate using any traffic sensory data available. The system is supported by a flexible instrumentation ensuring updates with low-latency, high incoming event rates and a fixed resource budget. Furthermore, the system can be deployed at any type of location or intersection without pre-training, and agnostic of the road or intersection layout, size, and the available sensors. This offers major advantages in terms of deployment cost reduction, especially because the learnt underlying correlations can be transferred to new infrastructure layouts equipped with different sensors.

Optimization of road traffic may be performed continuously, combining historical data and the incoming stream of current traffic data (for example, the number of cars, speed of cars, occupation at traffic light, noise of a certain street segment, pollution values recorded on a street segment). The system can therefore continuously learn traffic sensory correlations, fuse sensory data describing road traffic and adapt to changes for improved road traffic prediction. Moreover, in traffic flow prediction and control, the control unit can estimate and accommodate changes in the data distribution and provide accurate predictions and judicious control actions, for example control traffic light green colour timings. The system can therefore predict traffic flow given the available sensory data with a fixed resource budget.

As well as predicting traffic flow at locations where a camera or induction loop is not present, alternatively or additionally, the system may also be used to detect when a fault has developed in a sensor. For example, when a camera or induction loop is defective, the system can use data collected from other sensors to predict the correct parameter data based on the learned relationship. The system can provide data to replace the data from a faulty sensor based on data from another sensor, given that the correlation between the other sensors type and the faulty sensor type was previously learnt. The system is therefore capable of inferring missing sensory quantities. This can be utilised where the correction of faulty sensory quantities is required. For example, when pollution sensors have undergone a drift due to humidity.

The system can be implemented using a multitude of available sensors. Examples of sensors which may be used include, but are not limited to, cameras, pollution sensors, noise sensors, CO sensors, NO₂ sensors, NO sensors, O₃ sensors, NO₂ sensors, NO_x sensors, weather sensors (for example, humidity, rain) induction loops, mobility data, GSM user cell switches overlaid on geospatial motion parameters, ranges of high-intensity sound mounted on streets and particle count sensors for high-range exhaust gas. Fig. 14 summarises a method for implementation in a road traffic flow prediction system configured to receive data from a first sensor being configured to acquire traffic flow data in a first data domain and from second and third sensors being configured to acquire data in a second data domain, the first and second sensors being located at a first location and the third sensor being located at a second location spatially remote from the first location. At step 1401 , the method comprises receiving data from the first sensor, the second sensor and the third sensor. At step 1402, the method comprises processing the data received from the third sensor in dependence on the data received from the first and second sensors to estimate traffic flow at the second location.

The system and method described herein may help to minimize the overall cost of a traffic control system by inferring traffic flow from multiple sources. This alleviates the need for expensive sensors and data sources, such as cameras and induction loops, at every location or intersection by inferring traffic flow from other cheaper and readily available correlated sensors, allowing the system to efficiently scale for large urban scenarios.

The system is particularly applicable to traffic prediction, which requires modelling, prediction and fast adaptation for sensory timeseries. As shown by its modular structure, the system employs an efficient timeseries representation and modelling methodology capable of extracting data distributions and, through exploiting temporal co-activation, learning multisensory correlations for fault-tolerant traffic prediction in a unified computation unit.

Regardless of the deployment scenario, the data processing unit is able to model timeseries distributedly based on their temporal structure exploiting the representation of the data distribution they extract automatically. The compute unit has the capability to adapt to any type of sensory timeseries and any number of available sensors describing road traffic.

The system also enables the encoding of various sensory timeseries into a distributed representation capable of capturing the underlying statistics and data distribution model and provides a generic approach for timeseries modeling capable of finding the best fitting model that describes the underlying structure of the sensory data. The system comprises a generic architecture capable of operating as part of a traffic control system, obtaining prediction models for each location. The system can be deployed for multiple scenarios, independent of road geometry, size and configuration and available sensors, for which flexibility and scalability are required. The system may comprise a processor and a non-volatile memory. The system may comprise more than one processor and more than one memory. The memory may store data that is executable by the processor. The processor may be configured to operate in accordance with a computer program stored in non-transitory form on a machine readable storage medium. The computer program may store instructions for causing the processor to perform its methods in the manner described herein. The components may be implemented in physical hardware or may be deployed on various edge or cloud devices.

As the data enters the system, the computations have a limited time span to be handled, bounded by resource allocation and execution time, yielding for simple operations to be executed. The proposed system overcomes the resource greedy, computationally expensive and complex state-of-the-art approaches, such as complex analytical flow models based on differential equations and numerical methods, and empirical methods, by proposing a road traffic prediction compute unit that exploits the temporal correlations among the different traffic sensory data describing the traffic situation.

The road traffic flow timeseries representation and modelling and compute unit is capable of modelling sensory timeseries using an efficient distributed model of data representation and statistics modelling that supports a lightweight learning algorithm. The system provides a multisensory correlation learning system for road traffic prediction which, given pairs of sensory data describing the traffic situation, learns the temporal correlation patterns among them, encoding the underlying functional mathematical relationship between them, without any prior information on the underlying statistics and correlations. The decoding mechanism decodes the learnt correlation between sensors and encoded in an efficient data representation to an interpretable mathematical functional relation. The proposed system therefore provides a flexible infrastructure to analyse arbitrary sensory correlations and extract understandable sensory data associations for traffic prediction. The lightweight learning system represents and extracts the underlying statistics of sensory data in an efficient data representation. Furthermore, the system has an explainable structure and no “black-box design” with an interpretable output applicable to bi-sensory, tri-sensory, or multi-sensory scenarios. The proposed system combines the learning capabilities that the neural networks and SOM exhibit in terms of representing timeseries efficiently and an efficient correlation learning and multisensory fusion mechanism. Such a combination allows for learning and re-learning in real-time. Due to its learning capabilities, the system does not need to explicitly encode sensory associations and sensor models. This has the potential to minimize the cost associated with equipping all crosses with expensive sensors through its capability to transfer the learnt correlations to other locations or intersections with without reconfiguration to predict road traffic flow from other available sensory data. The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

1. A road traffic flow prediction system configured to receive data from a first sensor being configured to acquire traffic flow data in a first data domain and from second and third sensors being configured to acquire data in a second data domain, the first and second sensors being located at a first location and the third sensor being located at a second location spatially remote from the first location, wherein the system is configured to: receive data from the first sensor, the second sensor and the third sensor; and process the data received from the third sensor in dependence on the data received from the first and second sensors to estimate traffic flow at the second location.

2. The system as claimed in claim 1 , wherein the data received from each of the sensors comprises a timeseries of values.

3. The system as claimed in claim 1 or claim 2, wherein the first sensor is a camera and the first data domain is one or more images of vehicles at the location.

4. The system as claimed in any preceding claim, wherein the second and third sensors are configured to acquire data relating to the level of an environmental property at the first and second locations respectively.

5. The system as claimed in claim 4, wherein each of the second and third sensors comprises one of a weather sensor, a pollution sensor, a noise sensor, a CO sensor, and an induction loop.

6. The system as claimed in claim 4 or claim 5, wherein the third sensor is configured to acquire data relating to the same environmental property as the second sensor.

7. The system as claimed in any preceding claim, wherein the system is configured to perform the said processing by implementing a learned artificial intelligence model.

8. The system of claim 7, wherein the artificial intelligence model is a neural network.

9. The system as claimed in claim 7 or claim 8, wherein the system is configured to learn a mapping from the second data domain to the first data domain.

10. The system as claimed in any of claims 7 to 9, wherein the system is configured to iteratively update the parameters of the model overtime in dependence on the data received from the first and second sensors.

11. The system as claimed in any preceding claim, wherein the system is further configured to process the data received from the third sensor in dependence on data received from at least one other sensor to estimate traffic flow at the second location.

12. The system as claimed in any preceding claim, wherein the system is further configured to: receive data from a fourth sensor being located at the first location and from a fifth sensor being located at the second location, the fourth and fifth sensors being configured to acquire data in a third data domain; and process the data received from the third and fifth sensors in dependence on the data received from the first, second and fourth sensors to estimate traffic flow at the second location.

13. The system as claimed in any preceding claim, wherein the first and second locations are first and second traffic intersections respectively.

14. The system as claimed in claim 13, wherein the system is further configured generate a time plan for respective sets of traffic lights at the first and second intersections.

15. A method for implementation at a road traffic flow prediction system configured to receive data from a first sensor being configured to acquire traffic flow data in a first data domain and from second and third sensors being configured to acquire data in a second data domain, the first and second sensors being located at a first location and the third sensor being located at a second location spatially remote from the first location, the method comprising: receiving data from the first sensor, the second sensor and the third sensor; and processing the data received from the third sensor in dependence on the data received from the first and second sensors to estimate traffic flow at the second location.

16. The method of claim 15, wherein the said processing comprises implementing a learned artificial intelligence model.