WO2016123424A1 - Remote accident monitoring and vehcile diagnostic distributed database - Google Patents

Remote accident monitoring and vehcile diagnostic distributed database Download PDF

Info

Publication number
WO2016123424A1
WO2016123424A1 PCT/US2016/015514 US2016015514W WO2016123424A1 WO 2016123424 A1 WO2016123424 A1 WO 2016123424A1 US 2016015514 W US2016015514 W US 2016015514W WO 2016123424 A1 WO2016123424 A1 WO 2016123424A1
Authority
WO
WIPO (PCT)
Prior art keywords
vehicle
accident
surveillance
vehicles
information
Prior art date
Application number
PCT/US2016/015514
Other languages
French (fr)
Inventor
Gil Emanuel FUCHS
Clayton Richard Morlock
Samuel LAVIE
Original Assignee
Scope Technologies Holdings Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Scope Technologies Holdings Limited filed Critical Scope Technologies Holdings Limited
Priority to EP16744154.2A priority Critical patent/EP3251107A4/en
Publication of WO2016123424A1 publication Critical patent/WO2016123424A1/en

Links

Classifications

    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/20—Monitoring the location of vehicles belonging to a group, e.g. fleet of vehicles, countable or determined number of vehicles
    • G08G1/207—Monitoring the location of vehicles belonging to a group, e.g. fleet of vehicles, countable or determined number of vehicles with respect to certain areas, e.g. forbidden or allowed areas with possible alerting when inside or outside boundaries
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions
    • G08G1/0108—Measuring and analyzing of parameters relative to traffic conditions based on the source of data
    • G08G1/0112—Measuring and analyzing of parameters relative to traffic conditions based on the source of data from the vehicle, e.g. floating car data [FCD]
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions
    • G08G1/0108—Measuring and analyzing of parameters relative to traffic conditions based on the source of data
    • G08G1/0116—Measuring and analyzing of parameters relative to traffic conditions based on the source of data from roadside infrastructure, e.g. beacons
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions
    • G08G1/0108—Measuring and analyzing of parameters relative to traffic conditions based on the source of data
    • G08G1/012—Measuring and analyzing of parameters relative to traffic conditions based on the source of data from other sources than vehicle or roadside beacons, e.g. mobile networks
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions
    • G08G1/0125—Traffic data processing
    • G08G1/0129—Traffic data processing for creating historical data or processing based on historical data
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions
    • G08G1/0137—Measuring and analyzing of parameters relative to traffic conditions for specific applications
    • G08G1/0141—Measuring and analyzing of parameters relative to traffic conditions for specific applications for traffic information dissemination
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/017—Detecting movement of traffic to be counted or controlled identifying vehicles
    • G08G1/0175—Detecting movement of traffic to be counted or controlled identifying vehicles by photographing vehicles, e.g. when violating traffic rules
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/04—Detecting movement of traffic to be counted or controlled using optical or ultrasonic detectors
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/052—Detecting movement of traffic to be counted or controlled with provision for determining speed or overspeed
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems
    • G08G1/164—Centralised systems, e.g. external to vehicles
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems
    • G08G1/166—Anti-collision systems for active traffic, e.g. moving vehicles, pedestrians, bikes
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • G01S13/93—Radar or analogous systems specially adapted for specific applications for anti-collision purposes
    • G01S13/931—Radar or analogous systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G01S2013/9316—Radar or analogous systems specially adapted for specific applications for anti-collision purposes of land vehicles combined with communication equipment with other vehicles or with base stations

Definitions

  • This invention generally pertains to vehicle accident surveillance and methods to deal with vehicle accidents. Embodiments of the invention are also generally related to systems and methods to diagnose vehicle issues and related to communication among components of a distributed vehicle database.
  • EP 0466499 A1 is referred to in this document.
  • aspects of this invention are designed to provide as close to real-time surveillance of a vehicle accident as possible to both estimate the amount and extent of damage to the vehicle or vehicles and to determine bodily harm.
  • appropriate emergency response vehicles can be deployed and the repair process can be initiated including relaying, to adjusters, damage estimates and surveillance information to determine causality.
  • Specific aspects of embodiments of this invention include methods and system to detect accidents before they happen or while they happen; methods and systems to anticipate an accident based on measurements acquired of vehicle movements and driving conditions that are historically indicative of an accident.
  • Sensors within a vehicle deployed from a vehicle for aerial surveillance; deployed from a fixed based station for aerial surveillance; long term flight aerial surveillance; and fixed sensors that monitor a transportation network may be deployed.
  • An aim of this invention is both to navigate the uncertain regulatory landscape and also take advantage of the array of sensors; sensor delivery vehicles and methods; and statistical and machine learning analysis techniques for accident prediction and accident scene surveillance.
  • Embodiments of the present system alleviates potential bandwidth deficiencies and needless transfer costs for a vehicle analytics database, storing all the information in a distributed fashion, and only relaying information to a processing unit when it is needed for analysis.
  • an analysis unit selectively querying across a network or networks for only the pertinent data for a relative event or task and by also identifying what type of device and/vehicle the information is desired from, then massive data transfers can be avoided.
  • Typical vehicle diagnostic systems and assisted or autonomous driving systems rely on crowdsourced information that was compiled from collecting data stored in on-board vehicle systems and from external feeds such as traffic and weather. All this information is compiled and sifted through in an effort to, for example, provide a prediction of some type of hazardous conditions or need of repair. Tremendous amounts of data need to be wirelessly transmitted, typically on mobile networks, where two-way communications are established between every vehicle or external feed and the central server or servers.
  • Embodiments of this invention drastically reduce the bandwidth necessary to collect and utilize information.
  • a central server/s broadcasts via radio transmission (for example FM sideband) a selective query which contains specific identifiers with respect to the type of vehicle or vehicle configuration (if necessary) and/or location and time requirements and a request for information, or a notification that information is available for client systems meeting the selection criteria. If a client system receives the transmission and the client system meets the selection criteria, then and only then will a two-way communication be established with a handshake initiated by the client/s (an individual vehicle or subsystem or service).
  • the qualifying clients connect to the server/s and download the information.
  • bandwidth will always be saved.
  • the server/s does not have to be aware of every vehicle on the road. The vehicles simply have to be aware of the server/s.
  • Embodiments of this invention comprise a distributed database and method of use, where the database comprises raw sensor and environmental data related to a vehicle and the driving conditions the vehicle was subjected to. All information is both spatially and temporally referenced. In addition the information is referenced based on the type of vehicle and how the vehicle is equipped/loaded and optionally by the driver of the vehicle.
  • the database is distributed among one or more central servers and clients: satellite servers, individual vehicles, and hand-held units.
  • Each server and clients houses a database of information that is pertinent to one of: one or more vehicles or to an individual driver who drives the one or more vehicles.
  • a vehicle will store raw sensor data from sensors embedded in the vehicle and/or that reside in-vehicle and also information acquired from external feeds - for example traffic (in the vicinity of the vehicle or along a route the vehicle will travel) and weather information.
  • a satellite server will contain, for example, information for vehicles and weather and the transportation network for a given geographic area.
  • Another example is a server hosted by a repair facility that has information on the type and cost of repairs for vehicles they have worked on.
  • FIG. 1 depicts and embodiment using aerial surveillance from a blimp.
  • FIG. 2 Depicts an accident scene being tracked.
  • FIG. 3 is a flowchart showing input into an Accident Prediction Module
  • FIG. 4 is a prior art depiction of rocket propelled surveillance system.
  • FIG. 5 is a prior art depiction of a small quad copter with mounted camera.
  • FIG. 6 is a flowchart of events required to launch an aerial
  • FIG. 7 is a depiction of an embodiment of a system that houses a distributed vehicle diagnostic database.
  • FIG. 8 is a flow chart of an embodiment of a method of using a distributed vehicle diagnostic database.
  • FIG. 9 is a flow chart of an embodiment of a method for follow up instructions after an event has occurred and been reported.
  • FIG. 10 is a flow chart of an embodiment of how patterns and indicators are updated.
  • Maintenance Report a document or report (either hardcopy or online) that results from analysis of information relating to a vehicle operation, that schedules maintenance and repairs that are required to keep a vehicle in peak operating condition.
  • ln-vehicle Refers to anything that is part of the vehicle or within or attached to the vehicle.
  • Sensors measurement devices which measure parameters that are directly or indirectly related to the amount and extent of maintenance and/or repair needed to keep a vehicle in peak operating condition. Sensors could be in-vehicle - either part of the vehicle or an after-market attachment to the vehicle such as a fleet management system or as part of a mobile device within the vehicle such as the sensors in a mobile phone - like accelerometers or gyroscopes. Sensors may also be outside the vehicle such as roadside traffic counters in the vicinity of the vehicle, weather stations, and satellite or airborne based sensor such as LIDAR.
  • Transceiver A device capable of both receiving and sending information to another device whether it be wired or wireless. Examples are two-way radios, mobile phones, wired modems and the like.
  • Transmitter A device capable of sending information over radio waves.
  • Receiver A device capable of receiving information over radio waves.
  • Location where an object is relative to a reference frame.
  • the location of a vehicle is some embodiments is relative to the earth in terms of a coordinate system such as latitude and longitude (and perhaps elevation).
  • Vehicle any object capable of moving material or people. This includes cars, trucks, boats, airplanes, construction equipment and the like.
  • External Observations See the definition of sensors above for examples of observations that can come from outside the vehicle.
  • Source for this information can also be from web services, for example weather data, or traffic information that is a feed coming in from a FM sideband via an FM receiver.
  • Reference (for a database) an index or other attribute that can be used to select database records of interest by querying using the index or attribute.
  • reference for accident information could be: location, time, time of day, time of week; make of vehicle, year of vehicle (or Vehicle Identification Number), weather conditions, location of impact (zone on the car), direction of impact, force of impact and the like.
  • Normalized transforming data from a variety of sources into the same units, in the same frame of reference.
  • Historical Maintenance Database a database or collection of linked databases containing information that is related to individual accident events where all information is cross referenced so that it can be used for statistical analysis of accidents and the cost of repair resulting from the accident.
  • Cross-referenced With respect to a database, one entry can be queried as to its relationship to another if there is some type of relationship between the two. For example, a certain model of water pump produced by General Motors may have been used in a variety of car models over a variety of model years, so the part number for the water pump will be cross referenced to vehicle model number, year, engine type. Also note these parameters may not be sufficient information, because a part used may change mid-model year. For example, a wheel type my not be compatible halfway through a model year because the lug spacing was changed for safety reasons. In this case, the wheel would have to be referenced to the specific Vehicle Identification Number (VIN) which could be further cross referenced to a linked database containing more detailed information.
  • VIN Vehicle Identification Number
  • Confidence Interval One method of expressing the probability that an outcome will be observed to happen within a specific range for a given set of
  • the probability that the water pump will have to be replaced for shortly after 100,000 miles of driving is 95 percent for a Ford Focus and 92 percent for a BMW 928i.
  • Satellite Servers Part of the network that contains the distributed database where a portion of the database is held. Typically, the portion of the database will have information pertaining to a particular geographic area or a particular fleet of vehicles, or may contain only certain types of information, for example snow depths.
  • these databases may contain accident information that identifies damage specific information, and cost of repair with is correlated with make, model, and model year of the vehicle/s involved. Once again this information is spatially and temporally indexed.
  • weather related information may also be stored and indexed to location and time as well as correlated with accidents. This information can come from police reports, private insurance databases, and similar.
  • Patterns Time series or frequency distribution of sequential sensor data of one or more sensors or feeds for a given time period and locale that can be used to identify Driving Events. Patterns are created by analyzing many datasets with known events happening.
  • Patterns are updated by a central server in communication with a vehicle or satellite server system through the process of querying the vehicle fleet or satellite server network and where one of these remote entities has information that match the query, the remote entity will respond with relaying data for the event in question back to the main server. Definition of new patterns are further refined by soliciting data from like vehicles or circumstances, to be relayed to the central server where these data can then be used to refine the existing patterns that define an event. [0051] Patterns typically cannot be determined by human observation as they may be dependent on many variables that do not lend themselves to human observation. A human may be able to observe that the necessity of applying the brake while traveling around a curve is probably indicative of too much speed, however, combining observations of brakes, abs sensors, acceleration, weather feeds and more is beyond the ability of a human to assimilate.
  • Patterns may be based on the output of 1 or more sensor and/or 1 or more observations.
  • the pattern could be based on exceeding (or falling below) a threshold value, or exceeding (or falling below) an average value over time. Patterns may be analyzed in the frequency domain (after a fast Fourier transform is applied to time series data).
  • Indicators Readings from one or more sensors for a given time period and locale that exceed or fall below a specified threshold value indicative or an Event, or Situation. An example of an indicator is exceeding the speed limit.
  • Driving Events Something of interest that happens related to a vehicle, location, or time period which is identifiable by monitoring patterns or indicators. Events generally are categorized by something that is out of the ordinary. Examples of an event are a vehicle accident, a vehicle exceeding the speed limit, a vehicle being driven in an unsafe manner.
  • An Ongoing Driving Event is a subset of an Event where the event occurs over a period of time. For example, an accident may be a momentary event, but may cause an Ongoing Event such as a slowing of traffic on the road where the accident occurs
  • External Data Feeds Servers or services that available via a web interface or that are broadcast over radio frequency that provide information on conditions such as weather and traffic.
  • Mutli-variate analysis A statistical technique to identify or maintain patterns. Examples are artificial neural networks and machine learning.
  • Circumstances Background information related to individual events. For example, location, time, weather conditions, traffic, road condition are all
  • Some objects of this invention are systems and methods to detect vehicle accidents and observe vehicle accident scenes.
  • the tools used for this are sensors within a vehicle or vehicles including: video cameras; sensors that are part of the vehicle; and additional sensors that are part of a portable device within the vehicle.
  • Other sensor systems include stationary sensors that are associated with the vehicle transportation network, for example, traffic counters, and speed cameras. Additional information may be provided by weather stations.
  • Aerial sensors can be mounted, for example, in ROVS, autonomous drones, and manned aircraft. In addition, sensors can be outside the atmosphere mounted on satellites. Individual vehicles may be tracked by GPS or wireless transmitter signal strength triangulation to assess movements prior to an accident.
  • Analysis consists of statistical analysis of sensor data from one or more system types and delivery systems where the analysis is performed by comparison of historical patterns indicative of an accident about to occur or an accident that has happened and further patterns used to assess damage and injury.
  • Novel ROV / autonomous flying vehicles that are deployed from a vehicle are also part of this invention.
  • Aerial surveillance which has an identified area to monitor vehicle movement and activities.
  • the area could be part of a road network defined by geographic borders; it could be an intersection known to have a potential for many collisions.
  • the area could change during different time periods or day of the week based on historical collision or accident rates.
  • On-Vehicle Surveillance which consists of a sensor suite that is part of the vehicle and perhaps sensors that are part of a mobile device within the vehicle.
  • Remote Sensing consists of numerous techniques including such things as weather satellites that can provide background information with respect to weather and road conditions.
  • All of these surveillance method could be used in both a passive or active mode.
  • Passive mode is where general information is recorded and stored for a fixed amount of time, then discarded unless an event such as an accident is identified. If an event such as an accident occurs, pertinent information is retrieved and analyzed and then transmitted to an analysis station or first responders or other surveillance systems.
  • Active mode surveillance is defined as occurring when some sensor pattern indicative of an event of interest occurs and is used to initiate specific
  • the sensor pattern may trigger additional recordation of information and/or direct sensors to monitor at a certain location and perhaps with an increased frequency of measurement than that which happens during passive surveillance.
  • Aerial Surveillance can be from fixed wing aircraft or rotary aircraft or lighter than air vehicles. The surveillance can occur from manned or unmanned vehicles.
  • passive aerial surveillance is used by itself or in tandem with other surveillance methods.
  • FIG. 1 An example of a passive aerial surveillance is shown in FIG. 1.
  • An aerial vehicle 101 continually scans an area filled with roads and vehicles 102. Vehicles coming in and out of the area 102 are identified. An account of individual vehicles entering and leaving the survey area can be maintained over time.
  • An aerial surveillance module (either that is part of the aerial surveillance vehicle or that is in remote communication with the aerial vehicle) is used to observe ground vehicle movement.
  • the aerial surveillance module in addition to vehicle recognition software, also has a digital map of the survey area. By tracking the movement of individual vehicles through the survey area, the aerial surveillance module can detect:
  • the aerial surveillance module can transmit instructions to other surveillance systems (either aerial, fixed or vehicle based) via wireless communications to alert these other systems that active monitoring of a situation may be necessary.
  • FIG. 2 depicts result that could be obtained from aerial surveillance.
  • vehicle 201 and a second vehicle 203 are observed at a first location and are continued to be tracked until a second time where it is observed that Vehicle 203 collides with vehicle 201 at location 205.
  • information may be transmitted to vehicles 201 and 203 or to emergency authorities or others.
  • the information may contain the travel history of the two vehicles including their locations and speeds and driving behavior.
  • communications may be initiated with other surveillance systems when a vehicle moves out of the surveillance area and if there was a reason to continue monitoring it in other quadrants or surveillance areas.
  • a scenario for aerial surveillance is:
  • a. Notify local surveillance assets to start actively monitoring vehicles with risky behavior by transmitting location and trajectory information b. Notify individual vehicle monitoring systems in the vicinity of vehicles that are driving in a risky manner, of the risk, and make sure that monitoring systems are activated c. When vehicles are near the boundary of the aerial surveillance area, notify the adjacent aerial surveillance areas to actively monitor the incoming vehicle.
  • Aerial surveillance at lower altitudes may comprise passive monitoring, for example, at a busy intersection where many accidents are known to happen simply scan the intersection recording a time series of information (for example video) and simultaneously be performing pattern recognition analysis on the information for patterns that would indicate an accident or impending accident. Once an accident is detected or is imminent, the time series data that is pertinent to the accident, is transferred to an analysis station or the authorities or to vehicles involved in the accident.
  • passive monitoring for example, at a busy intersection where many accidents are known to happen simply scan the intersection recording a time series of information (for example video) and simultaneously be performing pattern recognition analysis on the information for patterns that would indicate an accident or impending accident.
  • Surveillance system at an interchange may not be on an air vehicle, but could be attached to a pole or other structure where sensors are high above the interchange, so effectively there is an aerial view of the interchange.
  • Active surveillance may be initiated when any passive surveillance system detects a pattern of concern. Active surveillance would occur when a passive surveillance system deviated from it standard sweep path to monitor a specific vehicle or vehicles or a specific location.
  • Vehicles equipped with sensors that measure vehicle motion, and vehicle behavior and/or motion and behavior of adjacent vehicles fall into this category and are part of embodiments of this invention.
  • On-vehicle sensors are monitored for patterns indicative of an accident occurrence or an impending accident. These patterns, for example, could be rapid changes in acceleration, proximity alerts either from video analysis or other
  • electromagnetic monitoring such as sonar, or infrared.
  • Ground based surveillance can be one of:
  • Remote sensing such as analysis of imagery from satellites can provide general information about driving conditions, for example, weather. Resolution of imagery would typically be on the other of 1 square meter or more, so in most cases, you could not discern an individual vehicle.
  • a variety of flying vehicles can be used for aerial surveillance.
  • aerial vehicles are better suited for different applications.
  • Basic types of aerial vehicles include fixed-wing, traditional helicopters, multi-prop copters such as a hexi-copter, blimps or dirigibles; and variations or combinations of the above.
  • a fixed location implies that the aerial vehicle is normally housed on the ground when not in use, in a single location that is more or less central to area under surveillance. Size of the vehicle will depend on the application.
  • Air vehicle can be designed with electric motors powered by batteries which are in turn charged with solar panels.
  • very light weight slow moving fixed wing or blimp type vehicles can be up in the air for extended periods of time with minimal fuel.
  • a flight vehicle When wishing to capture information about an accident while it happens or shortly thereafter, in an embodiment, a flight vehicle is in communication with sensors within the vehicle such as accelerometers. When either an impending accident or an accident in progress is detected via analysis of patterns, the flight vehicle is launched very rapidly in an attempt to have a vertical launch should the vehicle begin to roll over.
  • the air vehicle could be a rotary type or a type of rocket with a deployable parachute.
  • a rocket or similar device could be deployed much like a torpedo, from a tube, but vertically oriented.
  • Virtually any type of airframe can be made to take off or, land and fly autonomously. This would require location and altitude sensors as well as some frame of reference, for example a digital map or a location beacon either at a fixed location or on a vehicle of interest.
  • the flying vehicle can be piloted remotely 6.2.5.3 Combination
  • a combination of remote piloting and autonomous flight can be used.
  • take-off and landing can be remotely piloted, while in surveillance mode, the flight could be autonomous.
  • an aerial vehicle will contain a human pilot.
  • 6.2.6.1 Quad or other copter There is a variety of remotely operated or semi-autonomous vehicle which achieve lift using one or more propellers. Configuration with 4 or 6 blades usually mounted in the same plain and all oriented with the direction of thrust perpendicular to the mounting plane. These copter or drones as they are often called come in a variety of sizes from less than a kilogram in weight up to 20 kilograms or more
  • Blimps have the advantage that they can stay in flight for extended periods as most of the energy is directed to moving the vehicle rather than keeping it aloft and the helium provides most of the lift.
  • a parachute mounted sensor suite which comprises a camera and perhaps other sensing devices is contained in a cylindrical or other aerodynamic container which in turn is attached to a chemical propellant or compressed gas engine or a kinetic energy device (for example a spring) capable of propelling the sensor suite and parachute at rapid speed above the vehicle.
  • the motor or other propulsion device is actuated by a signal from the vehicle monitoring system (or potentially a remote systems) when it detects an accident about to happen or that is in progress.
  • the vehicle monitoring system is equipped with a sensor or sensors (such as a gyroscope) that can be used to determine if the vehicle is oriented with the top of the car being up (within a threshold angle). If the top of the vehicle is not up and within the threshold angle of being perpendicular to the vertical direction, the apparatus is not launched - to prohibit injury or damage to objects or people on the ground.
  • a sensor or sensors such as a gyroscope
  • FIG 6 depicts a launch scenario in an embodiment of this invention that utilizes a projectile with a parachute.
  • the apparatus is housed in a weatherproof container with either a retractable hatch or cover that is penetrable by the apparatus.
  • the sensor suite is in standby mode 602 and in communication with a pattern detection module in the vehicle. If an accident pattern is identified 604, the launch mechanism is checked to be in a vertical position 606 and if so, the apparatus is launched 608, the hatch is either opened (prior to engine ignition) or penetrated when the apparatus lifts off.
  • An example of a mechanism for launch would be much like a jack-in-the-box where a cover and latch hold into place the projectile which is mounted on a spring. Once the latch is opened, the projectile is free to exit and the spring force is released propelling the projectile into the air.
  • a mechanism for launch would be much like a jack-in-the-box where a cover and latch hold into place the projectile which is mounted on a spring. Once
  • the parachute is deployed by various means known in the art.
  • the vehicle is located and tracked.
  • the camera is mounted on a gimbal and servo motors keep the lens oriented towards the car. There may optionally be a servo to stabilize the compass direction of the view of the camera, as the parachute and apparatus may be rotating.
  • the apparatus is equipped with a propeller or propellant to provide a horizontal and/or vertical forces to either prolong the length of time the apparatus can stay airborne or to be able to circle the vehicle for measurements at various altitudes above the vehicle or angles around the vehicle.
  • the camera may be equipped with a zoom lens to capture more or less detail of the accident scene.
  • Potential triggers (patterns) that would initiate a launch are the same as described in the section on indications of an accident occurring or about to happen.
  • FIG. 4 depicts a similar solution in the art (from European Patent
  • EP 0466499 A1 is a battlefield aerial surveillance device where a rocket is launched from a ground vehicle 49 at time (A). At time (B) near the apex of the flight the aerodynamic casing of the rocket is separated exposing the surveillance apparatus 9 with a parachute 15 comprising a camera with a field of view 7 and configured with a device to prohibit rotation 29. The video is transmitted to a ground vehicle at time (E).
  • the present invention differs from EP0466499 in that the rocket deployment is from the vehicle being surveyed and the deployment is initiated based on sensor output and pattern recognition.
  • the camera may be able to be directed and the parachute may be steerable.
  • image software may be able to detect the vehicle of interest and zoom in on it.
  • an autonomous air vehicle deployed from a ground vehicle.
  • the air vehicle comprises a communication module that is in wireless communication with on-board sensors in the ground vehicle. If a pattern is detected by the surveillance module in the ground vehicle that indicates that an accident in progress or that an accident has happened, this in-turn triggers the launch of the autonomous air vehicle.
  • a quadcopter of this size could be launched from a vehicle in a variety of ways: • A rigid quadcopter could be contained in a spherical container housed in a vertical tube imbedded in the vehicle. A spring loaded propulsion mechanism much like the mechanism used to proper a ball bearing in a pin-ball machine could be held in place by a latch. The latch could be triggered by the recognition of an accident pattern.
  • the cross arms of the quadcopter could be folded at a point where the two arms cross in the center such that two adjacent motors are nearly touching one another on opposite sides.
  • the apparatus in the folded stated could be housed in a bullet or rocket shaped container and launch much like the parachute system of the previous section.
  • a rocket is used to deploy payload of a sensor suite attached to a fixed wing or rotary aircraft.
  • An example of a vehicle that may be suitable for this type of deployment is show, for example in US Patent US 8444082 B1 .
  • sensos there is a variety of sensos that can be used to determine both vehicle movement and behavior and the conditions associated with the vehicle movement and behavior.
  • Various type of sensors may be used with aerial vehilces, at fixe ground locations or within vehicles.
  • GPS Global Position Satellite Receiver
  • This type of device can also be used to determine a low resolution altitude.
  • a GPS requires a line-of-site view of 3 (or more) satellites to determine a position, sometimes is may be necessary to augment a location
  • Bluetooth Low Energy As part of the protocol for a communication standard such as this, there is a parameter that is a measure of signal strength of the radio frequency signal that is received by a receiver from a transmitter. It is well known in the art that by knowing the signal strength from three different transmitters that are geographic spaced, the relative location of the receiver with respect to the three transmitters can be determined. Of course there is a substantial amount of error in the signal strength measurement so this method only provides an approximate relative location.
  • a vehicle is equipped with a radio frequency transmitter and as part of a sensor suite that is deployed using a rocket or a aerial vehicle deployed from the vehicle, there is a directional antenna that receives an indication of signal strength of the transmitted frequency from the radio transmitter, it is possible to determine the relative location of the sensor suite to the vehicle - so that video or other sensors can be directed towards the vehicle.
  • the camera would initially point towards the vehicle, and would further register an image of the vehicle and track the vehicle using conventional image analysis software described elsewhere in this document such that the video can be trained on the vehicle and not stay on the anticipated trajectory of the vehicle.
  • altimeters There are a variety of altimeters known in the art, which include ones based on barometric pressure and/or a combination of barometric pressure and gps measurements and potential gyroscopic measurements. Altitude is important when dealing with position relative to the earth rather than relative to a moving vehicle.
  • a pattern is the term used to describe one or more time-series of sensor readings that can be analyzed to: ⁇ Predict that an accident will happen
  • Patterns can comprise a time series of a specific sensor
  • acceleration could be measured directly by an accelerometer or inferred from location measurements over time from a GPS receiver and/or a combination of these two types of measurements could be used to determine a mean acceleration for a given time interval by a weighted average of the two measurements, with more weight being attributed to the measurement deemed the most accurate.
  • Patterns could also be analyzed in the frequency domain using Fourier analysis
  • Patterns are determined by some form of multivariable analysis such as machine learning where data is collected from sensors for many accidents where the extent of damage and severity of impact are known.
  • Raster image analysis can be considered another form of pattern analysis. In this case vehicles are identified and tracked.
  • Rapid deceleration above a specific threshold that would indicate emergency braking.
  • One method of detection of rapid deceleration would be to monitoring vehicle onboard accelerometers and gyros. Airbag deployment
  • Patterns may be expressed as polynomial equation; they may be a threshold constant or upper and lower range for a specific sensor; they may be based on frequency and/or amplitude analysis of a single type or multiple types of sensors or they could be a statistical mean value for one or more sensor outputs or environmental factors. Patterns will change over time as more data is added, more sophisticated analysis is performed or more sensor types are available for on-board measurement. Patterns for one type of vehicle may be entirely different than for another type of vehicle. This may be due to different sensor suites being available or different physical attributes of the vehicle. 6.2.13.1 Image analysis software to detect ground vehicles
  • one method is to use vehicle recognition software. Patterns in an image that are indicative of a vehicle. There are several methods for analyzing both video, still and infrared imagery to detect vehicles.
  • One example of a method for recognizing vehicles in a image is Real-time People and Vehicle Detection from UAV Imagery by Gaszczak, A, etal (see
  • New data is collected from vehicle on-board sensors and from external feeds such as sensor suites that are part of the road network system or for example from weather satellites.
  • the data for the last time period is stored and analyzed and the older data is thrown out (provided no patterns of interest were detected).
  • the data is stored in a memory stack of a set size where new data is added to top of the stack and the oldest data (at the bottom of the stack) is thrown out.
  • an accident pattern or impending accident pattern is looked for. If a patterns is detected, indicating an accident or impending accident has occurred or will occur, then the sampling rate may be increased to acquire more data per time period, and/or other sensor data, previously not being recorded, may be recorded.
  • the end of the accident event in an embodiment, is defined when the vehicle is stationary. Once the accident is over, the stored data is analyzed to detect damage and injury patterns. If accident and/or injury patterns are detected, then the location and estimated damage and injury associated with these patterns is recorded and transmitted to pertinent individuals or computer servers.
  • FIG 3 illustrates and embodiment using a monitoring system within the vehicle.
  • Real-time time series data is acquired from many sensors on-board the vehicle 302 and transmitted to an Accident Prediction Module 310.
  • the Accident Prediction Module 310 receives external information from other surveillance systems 308 by wireless communication.
  • the Accident Prediction Module 310 performs analysis comparing the sensor data feeds 302, 308 to accident patterns acquired from a historical database 304. If an accident pattern is matched to the sensor feeds, this triggers recording of detailed information and a search for damage and injury patterns within the data. If a damage or injury pattern is detected, then analysis is performed concerning the extent of damage or injury and the location of damage or injury and this information along with the underlying data is transmitted to interested parties.
  • Monitor for a prescribed time or until an accident occurs OCR the license plate; send warning message for continued bad driving; citation if bad behavior does not cease.
  • Sensor systems within the cars themselves are contacted via wireless communication and instructed to record information at a rapid rate.
  • Pre-accident patterns from the vehicle/s are compared with patterns either from macro aerial or local aerial surveillance systems to verify the analysis
  • sensors that can be used to detect an adjacent vehicle.
  • Video cameras for example could be used in conjunction with vehicle detection software to know when an adjacent vehicle is too close.
  • Adjacent vehicles will reflect light and other forms of electromagnetic radiation such as infrared, and / or may be equipped with an active transponder which transmits a signal which can be located and identified.
  • Modern vehicles are generally equipped with a variety of sensors that measure physical parameters associated with the moving vehicle. These sensors can be a part of the vehicle or within the vehicle, for example as part of a mobile device.
  • Vehicle behavior can be inferred based on patterns exhibited in the sensor data overtime -either from observations of a single type of sensor or a sensor suite, for example a gyroscope and also a 3 component accelerometer. Rapid changes in the orientation of the vehicle may be exhibited by changes in the values measured by a gyroscope and/or accelerometers. It is intuitively known, for example, if a car is spinning on wet pavement or on ice, that there is a strong likelihood that the vehicle will sustain damage and/or passengers will be injured. However, this likelihood can be quantified by tracking patterns in the sensor output leading up to previous accidents with known damage and injury - performing statistical analysis on those patterns.
  • Patterns observable from aerial surveillance may indicate: ⁇ A burning vehicle (infrared signature)
  • Hardware for an on-board accident detection and analysis system comprises the following components:
  • an on-board database comprising: o vehicle specific information; o patterns, for the individual vehicle type, used to analyze sensor data to detect accidents and to assess resulting injury and damage and useful to predict driver behavior and driver / insurance risk; o driver information; o emergency contact information;
  • a remote central server in communication with multiple vehicle systems comprising : o one or more computers; o a comprehensive central database located on one or more servers comprising:
  • the system will initiate the following sequence: 1 ) launch drone or rocket a) if the vehicle is equipped with a spring loaded hatch, open it b) check the orientation of the car to make sure that the launch will be relatively vertical - based on vehicle sensor input such as magnetometers or accelerometers.
  • the following tasks comprise one method to determine accident patterns initially based on accident reports:
  • ⁇ Develop transfer functions between observations in historical databases built from accident reports to on-board sensor measurements that are indicative of the observed damage. For example, an accident impact could be inferred when a rapid deceleration is detected either by accelerometer measurements or change in speed measurements. Location, and relative speed of an impact can be inferred based on 3 component acceleration. Alternatively, a side impact can be inferred when a side airbag is deployed. ⁇ Test the transfer function by predicting vehicle damage and resulting cost based on sensor data after an accident. Confirm the prediction based on conventional accident and insurance adjustor reports.
  • collisions may be classified based on relative speed of impact, for example. With more accurate speed data from sensors and vehicle weights, the classification could be changed to an impact momentum in N/m 2 using finer ranges for classification rather than simply an approximate relative speed of collision.
  • Raw data may need to be parameterized in such a way as they can be used into a numeric model.
  • An example of parameterization would be to characterize incidents into a grouping. For example, it may be desirable to collectively refer to impact force based on accelerometer readings in ranges in units of meters/second 2 rather than actual recorded values or as a mean over a time frame.
  • Database maintenance comprises removing older or poorer quality data, continually updating the patterns as newer and better information comes on line.
  • patterns can be broken into smaller subdivisions, for example, an accident pattern could be vehicle type specific as to vehicle class specific.
  • Range of fixed based aerial vehicles and sweep are; length of deployment; weather extremes that operation can occur.
  • FIG. 7 is an example of a distributed vehicle database system.
  • System hardware is distributed between one or more central servers 702 and client systems 706.
  • the client systems can include, for example, passenger vehicles 708, trucks 720, satellite servers 712, external data feeds, such as weather 714 and traffic 718, onboard vehicle monitoring systems and portable devices 710.
  • Information is communicated in the form of a query or a notification from the one or more central servers 702 to clients 706 via a radio broadcast 704. All clients have a radio receiver that conforms to the radio frequency and standard of broadcast as the radio broadcast device connected to the central server/s. All of the clients 706 in range of the broadcast receive the broadcast and digest the query or bulletin.
  • each client 706 establishes two-way communication, for example, over a mobile network 724, with the central server/s 702 and uploads to the central server/s 702 the requested information for a query or downloads the available information to the clients 706 for a bulletin.
  • the data are maintained in the device or system where it was generated and/or in a regional client of some kind. All data does not need to be uploaded to a central server for storage and subsequent analysis unless a central server asked for it.
  • a vehicle diagnostic port onboard diagnostics OBD
  • OBD onboard diagnostics
  • sensors including: mass flow, oxygen, seat belt, air bag, tire pressure, gps, accelerometers, gyroscope, and more.
  • VIN and/or make model and model year, accessories such as larger than normal tires, engine type and size, etc.
  • the driver may be identified either by manually input, or via automatic connection between the vehicle and a mobile device of the driver, or by visual or audible input query by controlling software within the car. These are just examples and any type of identification of the driver could be used.
  • the driver could carry an RFID tag that identifies her, for example.
  • Information relayed from roadside sensors or sensors or devices that monitor road conditions or weather can be stored. These can come from Bluetooth connections, side bands on radio stations (for example traffic); internet feeds and peer- to-peer networks from other vehicles.
  • Other information may concern repair history of warn vehicle parts as related to all the above mentioned information. This information could be stored directly in a server at a repair shop - for example.
  • All information stored in the distributed database is optionally spatially and/or temporally referenced.
  • the information can be referenced based on the type of vehicle and/or how the vehicle is equipped and/or by the driver of the vehicle. Patterns that relate changes in one or more sensor and environmental data overtime to events are also stored in the database, both in vehicles were the patterns are pertinent and in a central database located on a central server or in satellite databases.
  • Patterns are created by analyzing many datasets with known events happening and developing a predictive model of the event based on the available data. Patterns are updated by a central server in communication with client systems through the process of querying the client system (such as a vehicle fleet) and having clients (vehicles) that match the requirements for the pattern, respond with relaying data for the event in question.
  • client system such as a vehicle fleet
  • clients vehicles
  • FIG. 8 explains how an embodiment of the system is used.
  • the system consists of one or more central servers 802 equipped with one or more processors and one or more client devices 804 (see client devices as depicted in FIG. 7).
  • the central server/s is configured with software to predict events and to make assessments.
  • the central server/s is further equipped with a list of events of interest.
  • the central server/s 802 needs to correlate potential contributing factors to particular events or types of events 806. In some embodiments, the prediction or assessment is of regional or local in interest only and this will effect what sources of information are used 810.
  • the central server/s 802 broadcasts, using a radio broadcast transmitter, a request to all potential clients 804 including identifying parameters of type of information and also what type of vehicle or other source for the information that is needed 812. This request is generated by instructions loaded in one or more processors and referred to as a query engine.
  • Clients 804 are typically listening for broadcasts, using a radio broadcast receiver, from the central server/s 802. Once a broadcast is detected by a client, each client 804 parses the query or bulletin that was broadcast and determines whether it is able to comply with all of the
  • the client 804 If the client 804 can comply and the request (or query) is for information, the client will 1 ) relay the requested information to the central server/s 802, using a two-way radio transceiver, and/or 2) start collecting the requested information from sensors or other devices. Once a packet of information is acquired, the client 804 establishes a two-way radio communication link using the transceiver and uploads the information 818 to the central server 802 which receives the information on a radio transceiver that is part of the central server/s.
  • the central server/s 802 uses a form of statistical analysis to establish a relationship between the uploaded information from numerous clients 804 and creates predictive models 820 in the form of patterns and indicators.
  • a broadcast using the transmitter, is made to the clients 804 to let them know the derived patterns, and indicators are available for usage 822.
  • FIG. 9 depicts how certain events trigger other events. A follow-on analysis may be necessary after certain types of events are detected to form a proper assessment of needs to be performed.
  • a client 902 continuously monitors sensors and external feeds 904 and compares to patterns and indicators to detect an event. If an event is detected 906, pre-programmed activities are initiated that are associated with the event 908. One of these activities may be to inform the central server/s of the event and to upload information pertaining to the event.
  • the client 902 establishes two way
  • the central server/s 910 receives the information 912 and determines the type of follow-up information that is needed 914.
  • An example would be, an accident is detected; it is reported to the central server along with particulars about speed, location, severity of impact.
  • the central server may be programmed to determine if the accident caused a traffic slow down; therefore, it would broadcast a bulletin that it is interested in knowing the speed of vehicles that are in the vicinity of the accident so it can determine what, if any, roads were affected by the accident 918.
  • other clients 920 that meet the selection criteria would communicate with the central server and upload their speed 922.
  • the central server could also send out a request for available tow trucks that could be dispatched immediately to the scene. Yet another request would be to the insurance company of the vehicle involved in the accident; to emergency responders; and the like to send out a representative to the scene. Another example would be notifying local auto body shops of a potential client.
  • FIG. 10 describes periodic update of the patterns and indicators stored in the system. Patterns and indicators need to be updated by central server/s. This happens by determining when certain patterns are either too old or new information is available that needs to be incorporated. For example, a new sensor reading may have been determined to be useful in prediction of certain types of events and has previously not been included in the patterns for that event.
  • the central server/s 1002 would send out a query for additional information to be used for this update including the type of sensor reading that are needed and other pertinent information 1006.
  • Potential clients of interest receive the query and determines if they are a client 1004 of interest (have pertinent information) 1014. Clients 1004 of interest establish a link 1016 with the server/s 1002 and then upload pertinent information 1018.
  • the server/s 1002 receives the information 1008, and retires older information 1010 and re-compute updated patterns and indicators 1012.
  • Table 1 is an example of information that might be conveyed in a bulletin using the distributed vehicle database system - regarding a terrorist alert status.
  • the scenario would be as follows: One or more of the central servers received a statement from the department of Homeland Security (a potential client) indicating that there will be an amber alert for a specific geographic area for a specified period.
  • the information in the alert could be represented with fields defined in table 1 and with their XML (extensible markup language) counterpart below that.
  • a single active alert Contains the following attributes start
  • the central server may be programmed to receive the xml above and parse it. Based on the location fields, the type of alert field and the duration, the server may be further programmed to divert members of a fleet of vehicle away from the alert area if they are scheduled to be in the alert area at the specified time. This would happen by broadcasting a request that all members of the fleet of interest to establish a link with the central server and receive further information about the threat level. This would save having to relay the entire message to all vehicles. In addition, the central server would know have a good indication of how many vehicles in its fleet would be impacted by the alert. Of course other scenarios could be treated in the same manner.
  • the prediction of required maintenance and the estimated cost of maintenance is transmitted to the vehicle when service or maintenance is needed.
  • the transmission can occur to either the in-vehicle system or to a mobile device carried by a driver or passenger or directly to a service technician.
  • results can be displayed either graphically and/or in text on a screen in the vehicle - for example, an infotainment system screen.
  • Messaging channels operate. Communication could also be peer-to-peer and/or repeated.
  • the central server/s could broadcast a query or bulletin which is received by a vehicle. The vehicle could then rebroadcast it over a different frequency or using a different protocol - for example Bluetooth.
  • Communications between various sensors and a processor in a client can happen via the system bus in the client or via short range wireless or via fiber optics or wired connections.
  • the client device is an add-on product or consists of software running on a mobile device within a vehicle, then communication with the integral vehicle sensor may be by using an interface that can read on-board diagnostic (OBD II) codes by interfacing with a vehicle portal designed for external communications.
  • OBD II on-board diagnostic
  • DTC diagnostic trouble codes
  • client device that are in a stationary location, for example, road side sensor suites, an autobody repair facility, communication to a certain server or other nodes could happen over the internet and/or other form of wired and/or satellite communication.
  • a client device may communicate with a central server and/or a client device using a per-to-per network where a message is transmitted to one or more other client devices and then the message is repeated for other client that are in communication with the transmitting client device.
  • vehicle maintenance and service requirements are predicted by comparing the observed conditions that occur during vehicle operation over time with similar observed conditions for similarly classed vehicle used in similar conditions stored in a historical vehicle maintenance database.
  • Algorithms are developed to classify each maintenance or service event as succinctly as possible, given the available data, such that when the conditions requiring maintenance or service for a vehicle in use match a classification, this can be used with a degree of certainty, to predict resulting maintenance required and the parts and services necessary to effect the maintenance.
  • the observed conditions of interest during vehicle operation include:
  • Raw data that may be used to predict maintenance and service needed can come from a plurality of sources.
  • Sources include:
  • the database initially will have a mix of more qualitative data, for example from manually entered fleet maintenance records and repair shop invoices and quantitative data, for example, from in-vehicle sensors. As such there is a subjective element in the reporting and the likelihood of human error will reduce the quality of the manually entered data and therefore if the manually entered data makes up the bulk of the available information, the error in prediction of maintenance will be greater.
  • the information For information from disparate sources to be compared, the information must be normalized, i.e. converted to the same units of measure and be relative to the same reference frame.
  • the quality and precision of the data must also be evaluated and represented within the database in a normalized fashion. In other words, if for example, one speed is known to be accurate within +/- 10 mph, then all speeds in the database should have an error of estimate in mph (as opposed to kph for example). 6.3.5 Components of the system
  • One or more processors containing:
  • Time period of Interest o Radio Transceiver used to upload and download information to one or more specific client systems after two-way communication has been established by the client
  • Radio Receiver for information from central servers via the one-way transmitter and/or other vehicles or systems including:
  • Radio Transceiver configured to establish two-way communications with one or more central servers and further used to upload and download information o Sensors (in-vehicle and external to vehicles) ⁇ See listing of sensors elsewhere in this document
  • the central server can query the satellite servers for regional information, when, for example, an insurance carrier wants to adjust rates base on region or a fleet management company wants to perform preventative maintenance on their fleet which is region dependent.
  • An operator of the system may desire to update the geography of a specific road segment. To do this a query may be sent to all vehicle, requesting a download of gps traces for vehicles that have traversed the segment within a specified time period. Vehicles that meet this requirement and that received the query then respond by sending the appropriate information. Once the information is received, then the GPS traces can be processed to revise the geometry or the road segment in the central database. Transportation network information comprises the physical location of roads, the road condition, traffic density throughout the day or week and typical weather conditions for a given time and relative to a road position and more. [0205] Another example of how the distributed database could be used would be, for example, in the current Volkswagen scandal.
  • the central processor/s could send out a query request to all vehicles in the network and request that all vehicles with a specific model and model year and that have the specific engine type of interest, record the above parameters over a period of time and transmit that information back to the central server.
  • the vehicles in question will already be recording and storing this information, and can relay this information to the central server for a former time period once it is requested by the central server.
  • the central server could send the stored relationship (pattern or indicators) with the query, so that the individual vehicle systems can determine emission values based on the stored pattern and/or indicators and send only the computed emission values back to the central server.
  • Another example usage is in comparing vehicle wear as a function of region. Similarly, equipped vehicles will wear out faster or sustain differing levels of damage when involved in an accident depending on where the vehicle is driven. This type of information could be important for determining insurance rates. Corrosion due to salt being used as a de-icer for roads, corrodes vehicle parts significantly faster than when the salt is not applied. Likewise, in an area where there is significant rain, corrosion will be higher than in an arid region.
  • the present invention may be conveniently implemented using one or more conventional general purpose or specialized digital computers or microprocessors programmed according to the teachings of the present disclosure, or a portable device (e.g., a smartphone, tablet computer, computer or other device), equipped with including one or more sensors (e.g., accelerometers, GPS) or where the portable device are connected to the data collection devices that are remote to the portable device, or that are connected via wired or wireless means.
  • a portable device e.g., a smartphone, tablet computer, computer or other device
  • sensors e.g., accelerometers, GPS
  • Appropriate software coding can readily be prepared by skilled programmers based on the teachings of the present disclosure, as will be apparent to those skilled in the software art.
  • the present invention includes a computer program product which is a non-transitory storage medium (media) having instructions stored thereon/in which can be used to program a computer to perform any of the processes of the present invention.
  • the storage medium can include, but is not limited to, any type of disk including floppy disks, optical discs, DVD, CD-ROMs, micro drive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of media or device suitable for storing instructions and/or data.
  • a vehicle accident surveillance network comprises at least one of: a) one or more surveillance systems which in turn comprises:
  • a sensor suite configured to observe ground based vehicles
  • a pattern recognition module configured to interpret the sensor suite readings as vehicle movements, locations, pending accidents, and accident incidents and to identify specific vehicles
  • a wireless transceiver configured to transmit and receive the identity and location of specific vehicles that had pattern identified, to surveillance systems
  • one or more deployable aerial surveillance systems comprising:
  • an airframe configured to launch from one of a ground based vehicle, and fixed base station, and a larger airframe, wherein a launch is triggered by detection of a pattern indicative of an accident occurring or about to occur as detected by one or more of the surveillance systems;
  • a second wireless transceiver configured to receive the identity and location of the vehicle or vehicles which correspond to identified patterns from one or more surveillance systems;
  • a directional sensor suite configured to be directed towards the identified vehicle or vehicles after deployment of the aerial surveillance system
  • an aerial surveillance module configured to:
  • a deployable aerial surveillance system is configured with a receiver that can identify a location beacon (for example attached to a vehicle) and track the location beacon.
  • deployable aerial surveillance systems are deployed by an operator when the system is provided with one or more of: a) coordinates of a vehicle to be surveyed;
  • Vehicle accident surveillance network can be one or more of: a) an airborne surveillance system; b) a ground based vehicle equipped with a surveillance system; and c) a ground based stationary surveillance system.
  • airborne surveillance systems are configured: a) with image detection sensors that observe the earth below in a plurality of
  • the pattern detection module is configured to detect vehicles using image
  • Surveillance systems in an embodiment comprises a memory cache configured to store sensor data from sensors for a predetermined time prior to the present time and further configured to save this data upon detection of a pattern and continue to save incoming sensor data for a predetermined time after the pattern is detected.
  • An embodiment of a vehicle accident surveillance system installed in a ground vehicle comprises: a) an on-vehicle sensor suite configured to observe location and motions of the ground vehicle;
  • a pattern recognition module configured to interpret the sensor suite readings as pending accidents, and accident incidents
  • a deployable aerial surveillance system comprising:
  • an airframe configured to launch from the vehicle when triggered by detection of a pattern indicative of an accident occurring or about to occur.
  • a directional sensor suite configured to be directed towards the vehicle after deployment of the aerial surveillance system
  • an aerial surveillance module configured to:
  • a vehicle accident surveillance system installed in a ground vehicle can optionally be configure with a directional sensor suite that contains one or more cameras.
  • a transceiver in an accident surveillance system is optionally configured to: a) communicate with other accident surveillance systems;
  • a vehicle accident surveillance system installed in a ground vehicle comprises: a) an on-vehicle sensor suite configured to observe location and motions of the ground vehicle;
  • a pattern recognition module configured to interpret the sensor suite readings as pending accidents, and accident incidents
  • a wireless transmitter configured to transmit a request to nearby surveillance systems to deploy and monitor the ground vehicle should the pattern recognition module detect a pattern indicative of a potential accident or accident.
  • a system to create, manage and utilize a vehicle diagnostic distributed database comprises:
  • At least one central server operable on one or more computers configured with at least: i) a radio transmitter configured to broadcast a query; ii) a first radio transceiver configured to perform two-way communications with the one or more client devices; and iii) a query engine configured to generate a query containing:
  • a query is a request to upload information that is specific for a particular vehicle type or component and related to known vehicle events or situations and comprises one or more of:
  • vehicle component replacement and maintenance records a) vehicle component replacement and maintenance records; b) vehicle sensor data referenced in space and time; and c) environmental data referenced in space and time.
  • a system to create, manage and utilize a vehicle diagnostic distributed database has at least one central server that is further configured to develop at least one of patterns and indices to predict vehicle events and identify situations based on information, from the one or more client devices, that was previously uploaded to the central server.
  • a system to create, manage and utilize a vehicle diagnostic distributed database comprises query engine that generates a bulletin apprising clients of interest on the availability for download of, one or more of patterns and indices and instructions for the usage of the one or more patterns and indices and a radio transmitter broadcasts the query.
  • a system to create, manage and utilize a vehicle diagnostic distributed database clients of interest establish communication with central servers using a radio transceiver and download patterns and indices to predict events and identify situations.
  • a selection criteria comprising at least one of make, model, year of manufacturer and optional equipment of a vehicle is used.
  • a selection criteria comprises at least one of a geographic region, a climate zone, and within a political boundary.
  • a request can be made for GPS traces along roads a client vehicle has traversed.
  • a server can receive a requested GPS trace/s and is configured to update road geometry based on the GPS trace/s.
  • identified events and situations can be:
  • client device can be a:
  • a database comprises at least one:
  • the system comprises:
  • one or more computer based servers configured with at least: i) a broadcast radio transmitter; ii) a first radio transceiver; and b) one or more client devices configured with at least: i) a broadcast radio receiver; ii) a second radio transceiver; the one or more computer based servers transmits, using the broadcast radio
  • the transmitter a request to one or more of upload and download information and further containing selection criteria
  • the one or more client devices a) listen for and receive the transmission, utilizing the broadcast radio receiver; b) determines whether the selection criteria are met; and c) if the selection criteria are met, establish communication between the first and second radio transceivers and perform the request.
  • a communication method is used to operate a distributed vehicle diagnostic database which comprises:
  • the broadcast contains a query or bulletin including a client selection criteria
  • b) listening for and receiving the broadcast utilizing the broadcast radio receiver that is part of a client device and determining whether the client device meets the client selection criteria, and if the client device meets the client selection criteria, establish communication between a first radio transceiver that is part of the client device and second radio transceivers that is part of the one or more central servers, and moving information between the one or more computer servers and the client device that meets the client selection criteria.
  • a communication method is used to operate a distributed vehicle diagnostic database and comprises:
  • a communication method is used to operate a distributed vehicle diagnostic database with a client device that is one of a:
  • a communication method is used to operate a distributed vehicle diagnostic database with a client transceiver of the one or more client devices configured to also act as a repeater transmitting information to another client device which in turn can repeat the information and further transmit to yet another client or one or more of the computer based servers.
  • a vehicle accident surveillance network comprises: a) one or more surveillance systems comprising:
  • a sensor suite configured to observe ground based vehicles
  • a pattern recognition module configured to interpret the sensor suite readings as vehicle movements, locations, pending accidents, and accident incidents and to identify specific vehicles
  • a computer processor configured to receive a query from the broadcast radio receiver and evaluate it; and b) at least one central server operable on one or more computers configured with at least:
  • a radio transmitter configured to broadcast a query
  • a second radio transceiver configured to perform two-way communications with the one or more surveillance systems
  • a query engine configured to generate a query wherein the one or more surveillance systems: a) detects patterns, b) identifies a vehicle or vehicles associated with the patterns; c) determines the location of the vehicle or vehicles; d) transmits this information using the first radio transmitter to the second radio transmitter of the at least one central server; and wherein the at least one central server:
  • surveillance systems in proximity to the vehicles or vehicles identified in the query respond by: a) recording information using the sensor suite about the vehicle or vehicles and in the vicinity of the vehicle or vehicles;
  • the surveillance systems can be deployable aerial surveillance systems further comprising: a) an airframe configured to launch from one of a ground based vehicle, and fixed base station, and a larger airframe, wherein a launch is triggered by receiving a query from the at least one central server if the aerial surveillance system is in proximity to the vehicle or vehicles;
  • a directional sensor suite configured to be directed towards the identified vehicle or vehicles after deployment
  • an aerial surveillance module configured to launch the deployable aerial
  • the surveillance system and after being launched, determine the relative location of the identified vehicle or vehicles, and one or more of: approach the identified vehicle or vehicles, circle the vehicle or vehicles at a predetermined circumference and altitude, and point the directional sensors towards the vehicle or vehicles and record the sensor data.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Traffic Control Systems (AREA)

Abstract

Systems and method for vehicle accident monitoring using multiple monitoring systems deployed in the air, in vehicles, and stationary on the ground are described. To support the monitoring systems, in addition to providing a means of communicating pertinent vehicle and transportation information back and forth between servers, nodes, monitoring stations and pertinent databases, a communication network is also described. Monitoring systems sense accidents that have happened or are about to happen which in turn triggers other systems to collect pertinent information concerning the accident scene. Once an accident happens, the communication network can poll nodes connected to the network for information on similar accidents.

Description

REMOTE ACCIDENT MONITORING AND
VEHCILE DIAGNOSTIC DISTRIBUTED DATABASE
1 TECHNICAL FIELD
[0001] This invention generally pertains to vehicle accident surveillance and methods to deal with vehicle accidents. Embodiments of the invention are also generally related to systems and methods to diagnose vehicle issues and related to communication among components of a distributed vehicle database.
2 RELATED APPLICATIONS [0002] This application claims priority to US Provisional Application 62/109434 filed on 29 Jan 2015 and entitled ACCIDENT MONITORING USING REMOTELY OPERATED OR AUTONOMOUS AERIAL VEHICLES which is herein incorporated by reference.
[0003] This application claims priority to provisional patent application US 62/265215 entitled DEVELOPING AND USING A VEHCILE DIAGNOSTIC
DISTRIBUTED DATABASE, filed on 9 Dec 2015 and is herein incorporated by reference.
[0004] U.S. Patent Application titled "SYSTEM AND METHOD FOR USE OF PATTERN RECOGNITION IN ASSESSING OR MONITORING VEHICLE STATUS OR OPERATOR DRIVING BEHAVIOR", Application No. 13/679,722, filed November 16, 2012; which claims the benefit of priority to U.S. Provisional Patent Application No. 61/578,51 1 , filed December 21 , 201 1 ; PCT/US 12/71487 titled "SYSTEMS AND METHODS FOR ASSESSING OR MONITORING VEHICLE STATUS OR OPERATOR STATUS" filed 21 December 2012; and 14/317624 titled "System and Method for Determining Of Vehicle Accident information" file on 27 June 2014; each of which the above applications are herein incorporated by reference.
[0005] European Patent Application EP 0466499 A1 is referred to in this document.
[0006] US Patent US 8444082 B1 is referred to in this document. 3 SUMMARY OF INVENTION
[0007] Aspects of this invention are designed to provide as close to real-time surveillance of a vehicle accident as possible to both estimate the amount and extent of damage to the vehicle or vehicles and to determine bodily harm. In response to the estimates appropriate emergency response vehicles can be deployed and the repair process can be initiated including relaying, to adjusters, damage estimates and surveillance information to determine causality.
[0008] Specific aspects of embodiments of this invention include methods and system to detect accidents before they happen or while they happen; methods and systems to anticipate an accident based on measurements acquired of vehicle movements and driving conditions that are historically indicative of an accident.
[0009] Sensors within a vehicle; deployed from a vehicle for aerial surveillance; deployed from a fixed based station for aerial surveillance; long term flight aerial surveillance; and fixed sensors that monitor a transportation network may be deployed.
[0010] Various techniques for image analysis, statistics and machine learning are utilized to analyze both real-time and historical data concerning accidents.
[0011] In addition to accident detection and analysis the systems and methods described within this document are also useful, for example, for traffic monitoring, and crime prevention and detection.
4 TECHNICAL PROBLEM
4.1 Vehicle Monitoring
[0012] Driving a vehicle continues to be one of the most hazardous activities that a person can participate in. Vehicle accidents are one of the leading cause of death every year. Damage from accidents amounts to billions of dollars a year. To date, most vehicle accidents are assessed after-the-fact by personal arriving on the scene after the incident. The assessments almost always utilize some type of human interface either to estimate damage or transpose information into a machine readable form. This human interface introduces many biases and uncertainties into the process. These biases and uncertainties then translate into litigation when it is necessary to determine cause for insurance purposes or from a safety standard. [0013] Attempts have been made to take sensor information from in-vehicle sensors and associate this information to external information and factors to reconstruct accidents and/or determine when an accident occurs, but this work is in early stages.
[0014] The advent of unmanned aerial vehicles (UAV) with associated sensor arrays has added a new method of monitoring vehicle activity and accident scene surveillance. However, the changing regulatory atmosphere makes reliance on any one type of surveillance method risky from the fact that it may be illegal in the near future. For example, in Dec 2014, the US Congress is considering that UAVs for commercial unlicensed use can only fly 400 feet in the air and must be in view of the handler. Even more restrictive rulings may apply. For this reason and others, data fusion among several sensor arrays is important to any vehicle or accident scene surveillance system as certain methods may not be allowed in the long run.
[0015] An aim of this invention is both to navigate the uncertain regulatory landscape and also take advantage of the array of sensors; sensor delivery vehicles and methods; and statistical and machine learning analysis techniques for accident prediction and accident scene surveillance.
[0016] The amount of data that potentially can be stored in a vehicle analytics database is huge - many, many terabytes. The data is not always useful in all areas or regions or for all types of vehicles and situations and may never be used. Therefore, to centrally locate all the information requires unneeded data transferred over limited infrastructure and bandwidth.
[0017] Embodiments of the present system alleviates potential bandwidth deficiencies and needless transfer costs for a vehicle analytics database, storing all the information in a distributed fashion, and only relaying information to a processing unit when it is needed for analysis. By having an analysis unit selectively querying across a network or networks for only the pertinent data for a relative event or task and by also identifying what type of device and/vehicle the information is desired from, then massive data transfers can be avoided.
4.2 Vehicle Diagnostics
[0018] Typical vehicle diagnostic systems and assisted or autonomous driving systems rely on crowdsourced information that was compiled from collecting data stored in on-board vehicle systems and from external feeds such as traffic and weather. All this information is compiled and sifted through in an effort to, for example, provide a prediction of some type of hazardous conditions or need of repair. Tremendous amounts of data need to be wirelessly transmitted, typically on mobile networks, where two-way communications are established between every vehicle or external feed and the central server or servers.
[0019] Embodiments of this invention drastically reduce the bandwidth necessary to collect and utilize information. When information is desired from specific class of vehicle, or from a web service or radio transmission service, a central server/s broadcasts via radio transmission (for example FM sideband) a selective query which contains specific identifiers with respect to the type of vehicle or vehicle configuration (if necessary) and/or location and time requirements and a request for information, or a notification that information is available for client systems meeting the selection criteria. If a client system receives the transmission and the client system meets the selection criteria, then and only then will a two-way communication be established with a handshake initiated by the client/s (an individual vehicle or subsystem or service).
[0020] If the radio transmission was a notification, then the qualifying clients connect to the server/s and download the information. By not establishing a two-way communication with vehicles on the road that do not meet the criteria of the query, bandwidth will always be saved. Furthermore, the server/s does not have to be aware of every vehicle on the road. The vehicles simply have to be aware of the server/s.
[0021] Embodiments of this invention comprise a distributed database and method of use, where the database comprises raw sensor and environmental data related to a vehicle and the driving conditions the vehicle was subjected to. All information is both spatially and temporally referenced. In addition the information is referenced based on the type of vehicle and how the vehicle is equipped/loaded and optionally by the driver of the vehicle.
[0022] The database is distributed among one or more central servers and clients: satellite servers, individual vehicles, and hand-held units. Each server and clients houses a database of information that is pertinent to one of: one or more vehicles or to an individual driver who drives the one or more vehicles. For example, a vehicle will store raw sensor data from sensors embedded in the vehicle and/or that reside in-vehicle and also information acquired from external feeds - for example traffic (in the vicinity of the vehicle or along a route the vehicle will travel) and weather information. A satellite server will contain, for example, information for vehicles and weather and the transportation network for a given geographic area. Another example is a server hosted by a repair facility that has information on the type and cost of repairs for vehicles they have worked on.
[0023] In addition to raw sensor data being stored, patterns or indicators that relate changes in one or more sensor readings over time and changes in environmental data overtime to events that are also stored.
5 BRIEF DESCRIPTION OF THE DRAWINGS
[0024] FIG. 1 depicts and embodiment using aerial surveillance from a blimp.
[0025] FIG. 2 Depicts an accident scene being tracked. [0026] FIG. 3 is a flowchart showing input into an Accident Prediction Module
[0027] FIG. 4 is a prior art depiction of rocket propelled surveillance system.
[0028] FIG. 5 is a prior art depiction of a small quad copter with mounted camera.
[0029] FIG. 6 is a flowchart of events required to launch an aerial
reconnaissance device from a vehicle.
[0030] FIG. 7 is a depiction of an embodiment of a system that houses a distributed vehicle diagnostic database. [0031] FIG. 8 is a flow chart of an embodiment of a method of using a distributed vehicle diagnostic database.
[0032] FIG. 9 is a flow chart of an embodiment of a method for follow up instructions after an event has occurred and been reported. [0033] FIG. 10 is a flow chart of an embodiment of how patterns and indicators are updated.
6 DESCRIPTION OF THE EMBODIMENTS
6.1 Glossary
[0034] Maintenance Report: a document or report (either hardcopy or online) that results from analysis of information relating to a vehicle operation, that schedules maintenance and repairs that are required to keep a vehicle in peak operating condition.
[0035] ln-vehicle: Refers to anything that is part of the vehicle or within or attached to the vehicle. [0036] Sensors: measurement devices which measure parameters that are directly or indirectly related to the amount and extent of maintenance and/or repair needed to keep a vehicle in peak operating condition. Sensors could be in-vehicle - either part of the vehicle or an after-market attachment to the vehicle such as a fleet management system or as part of a mobile device within the vehicle such as the sensors in a mobile phone - like accelerometers or gyroscopes. Sensors may also be outside the vehicle such as roadside traffic counters in the vicinity of the vehicle, weather stations, and satellite or airborne based sensor such as LIDAR. External sensors that can provide information about the condition of pavement, weather, freeze thaw conditions or the like are included. [0037] Transceiver: A device capable of both receiving and sending information to another device whether it be wired or wireless. Examples are two-way radios, mobile phones, wired modems and the like.
[0038] Transmitter: A device capable of sending information over radio waves. [0039] Receiver: A device capable of receiving information over radio waves.
[0040] Location: where an object is relative to a reference frame. The location of a vehicle is some embodiments is relative to the earth in terms of a coordinate system such as latitude and longitude (and perhaps elevation). [0041] Vehicle: any object capable of moving material or people. This includes cars, trucks, boats, airplanes, construction equipment and the like.
[0042] External Observations: See the definition of sensors above for examples of observations that can come from outside the vehicle. Source for this information can also be from web services, for example weather data, or traffic information that is a feed coming in from a FM sideband via an FM receiver.
[0043] Reference (for a database): an index or other attribute that can be used to select database records of interest by querying using the index or attribute. For example, reference for accident information could be: location, time, time of day, time of week; make of vehicle, year of vehicle (or Vehicle Identification Number), weather conditions, location of impact (zone on the car), direction of impact, force of impact and the like.
[0044] Normalized: transforming data from a variety of sources into the same units, in the same frame of reference.
[0045] Historical Maintenance Database: a database or collection of linked databases containing information that is related to individual accident events where all information is cross referenced so that it can be used for statistical analysis of accidents and the cost of repair resulting from the accident.
[0046] Cross-referenced: With respect to a database, one entry can be queried as to its relationship to another if there is some type of relationship between the two. For example, a certain model of water pump produced by General Motors may have been used in a variety of car models over a variety of model years, so the part number for the water pump will be cross referenced to vehicle model number, year, engine type. Also note these parameters may not be sufficient information, because a part used may change mid-model year. For example, a wheel type my not be compatible halfway through a model year because the lug spacing was changed for safety reasons. In this case, the wheel would have to be referenced to the specific Vehicle Identification Number (VIN) which could be further cross referenced to a linked database containing more detailed information.
[0047] Confidence Interval: One method of expressing the probability that an outcome will be observed to happen within a specific range for a given set of
circumstances. For example, the probability that the water pump will have to be replaced for shortly after 100,000 miles of driving is 95 percent for a Ford Focus and 92 percent for a BMW 928i.
[0048] Satellite Servers: Part of the network that contains the distributed database where a portion of the database is held. Typically, the portion of the database will have information pertaining to a particular geographic area or a particular fleet of vehicles, or may contain only certain types of information, for example snow depths.
[0049] With respect to satellite servers that contain regional information, these databases may contain accident information that identifies damage specific information, and cost of repair with is correlated with make, model, and model year of the vehicle/s involved. Once again this information is spatially and temporally indexed. In addition, weather related information may also be stored and indexed to location and time as well as correlated with accidents. This information can come from police reports, private insurance databases, and similar. [0050] Patterns: Time series or frequency distribution of sequential sensor data of one or more sensors or feeds for a given time period and locale that can be used to identify Driving Events. Patterns are created by analyzing many datasets with known events happening. Patterns are updated by a central server in communication with a vehicle or satellite server system through the process of querying the vehicle fleet or satellite server network and where one of these remote entities has information that match the query, the remote entity will respond with relaying data for the event in question back to the main server. Definition of new patterns are further refined by soliciting data from like vehicles or circumstances, to be relayed to the central server where these data can then be used to refine the existing patterns that define an event. [0051] Patterns typically cannot be determined by human observation as they may be dependent on many variables that do not lend themselves to human observation. A human may be able to observe that the necessity of applying the brake while traveling around a curve is probably indicative of too much speed, however, combining observations of brakes, abs sensors, acceleration, weather feeds and more is beyond the ability of a human to assimilate.
[0052] Patterns may be based on the output of 1 or more sensor and/or 1 or more observations. The pattern could be based on exceeding (or falling below) a threshold value, or exceeding (or falling below) an average value over time. Patterns may be analyzed in the frequency domain (after a fast Fourier transform is applied to time series data).
[0053] Examples of patterns based on time series or frequency analysis of sensor traces:
• Impact severity and direction of impact; · Hazardous driving
• Dangerous Roads Segments or Intersections
• Probability of Hazardous Driving Conditions for a given location
• Excessive Speed
• Patterns based on location and/or time: · Road locations
• Traffic density
• Speed of Travel
[0054] Indicators: Readings from one or more sensors for a given time period and locale that exceed or fall below a specified threshold value indicative or an Event, or Situation. An example of an indicator is exceeding the speed limit.
[0055] Assessment: Given a variety of patterns and indicators as input, predictions are made for the resulting cost or extent of an event associated with the patterns and indicators. [0056] Driving Events: Something of interest that happens related to a vehicle, location, or time period which is identifiable by monitoring patterns or indicators. Events generally are categorized by something that is out of the ordinary. Examples of an event are a vehicle accident, a vehicle exceeding the speed limit, a vehicle being driven in an unsafe manner. An Ongoing Driving Event is a subset of an Event where the event occurs over a period of time. For example, an accident may be a momentary event, but may cause an Ongoing Event such as a slowing of traffic on the road where the accident occurs
[0057] External Data Feeds: Servers or services that available via a web interface or that are broadcast over radio frequency that provide information on conditions such as weather and traffic.
[0058] Situation: Something that is associated with the likelihood that an event will happen for a particular place, time and/or conditions. For example, a particular curve in the road may be more dangerous if a certain safe speed is exceeded or if the road is icy or wet. Therefore, if a specific vehicle is traveling on a specific curve and that curve happens to be icy, and the vehicle weighs a specific amount, then the situation may be that the vehicle is in eminent danger of sliding off the road.
[0059] Mutli-variate analysis: A statistical technique to identify or maintain patterns. Examples are artificial neural networks and machine learning. [0060] Circumstances: Background information related to individual events. For example, location, time, weather conditions, traffic, road condition are all
circumstances.
6.2 Accident Monitoring Using Remotely Operated or Autonomous Aerial Vehicles [0061] Some objects of this invention are systems and methods to detect vehicle accidents and observe vehicle accident scenes.
[0062] The tools used for this are sensors within a vehicle or vehicles including: video cameras; sensors that are part of the vehicle; and additional sensors that are part of a portable device within the vehicle. Other sensor systems include stationary sensors that are associated with the vehicle transportation network, for example, traffic counters, and speed cameras. Additional information may be provided by weather stations. Aerial sensors can be mounted, for example, in ROVS, autonomous drones, and manned aircraft. In addition, sensors can be outside the atmosphere mounted on satellites. Individual vehicles may be tracked by GPS or wireless transmitter signal strength triangulation to assess movements prior to an accident.
[0063] Analysis consists of statistical analysis of sensor data from one or more system types and delivery systems where the analysis is performed by comparison of historical patterns indicative of an accident about to occur or an accident that has happened and further patterns used to assess damage and injury. [0064] Novel ROV / autonomous flying vehicles that are deployed from a vehicle are also part of this invention.
6.2.1 Surveillance Methods
[0065] Surveillance methods for this document are broken into five types:
• Aerial surveillance which has an identified area to monitor vehicle movement and activities. The area could be part of a road network defined by geographic borders; it could be an intersection known to have a potential for many collisions. The area could change during different time periods or day of the week based on historical collision or accident rates.
• On-Vehicle Surveillance which consists of a sensor suite that is part of the vehicle and perhaps sensors that are part of a mobile device within the vehicle.
• Event Surveillance where sensor suites are deployed (usually in the air) from either a vehicle, a fixed base station or where a passive sensors system is trained on an accident scene once an accident or potential accident is detected.
• Ground Based surveillance which typically is performed by stationary
sensors along a roadway or intersection and measures things like traffic counts or vehicle speed (an average speed) and/or weather and road conditions. • Remote Sensing consists of numerous techniques including such things as weather satellites that can provide background information with respect to weather and road conditions.
[0066] All of these surveillance method could be used in both a passive or active mode. Passive mode is where general information is recorded and stored for a fixed amount of time, then discarded unless an event such as an accident is identified. If an event such as an accident occurs, pertinent information is retrieved and analyzed and then transmitted to an analysis station or first responders or other surveillance systems.
[0067] Active mode surveillance is defined as occurring when some sensor pattern indicative of an event of interest occurs and is used to initiate specific
surveillance. The sensor pattern may trigger additional recordation of information and/or direct sensors to monitor at a certain location and perhaps with an increased frequency of measurement than that which happens during passive surveillance.
6.2.1.1 Aerial Surveillance [0068] If is known in the art how to identify and track multiple vehicles and/or pedestrians using high altitude surveillance. This technology is much like facial recognition software used in virtually every digital camera where multiple faces can be identified and tracked. Aerial Surveillance can be from fixed wing aircraft or rotary aircraft or lighter than air vehicles. The surveillance can occur from manned or unmanned vehicles.
[0069] In an embodiment of this invention passive aerial surveillance is used by itself or in tandem with other surveillance methods.
[0070] An example of a passive aerial surveillance is shown in FIG. 1. An aerial vehicle 101 continually scans an area filled with roads and vehicles 102. Vehicles coming in and out of the area 102 are identified. An account of individual vehicles entering and leaving the survey area can be maintained over time.
[0071] An aerial surveillance module (either that is part of the aerial surveillance vehicle or that is in remote communication with the aerial vehicle) is used to observe ground vehicle movement. [0072] The aerial surveillance module, in addition to vehicle recognition software, also has a digital map of the survey area. By tracking the movement of individual vehicles through the survey area, the aerial surveillance module can detect:
If a vehicle is off the road
If a vehicle is exceeding the speed limit
If a vehicle is in very close proximity with another vehicle
If equipped with thermal imaging, if a vehicle is on fire.
A vehicle driving erratically or swerving
Traffic as a whole moving slowly or stopped [0073] During analysis of the passively acquired data, the aerial surveillance module can transmit instructions to other surveillance systems (either aerial, fixed or vehicle based) via wireless communications to alert these other systems that active monitoring of a situation may be necessary.
[0074] FIG. 2 depicts result that could be obtained from aerial surveillance. At a time 1 , vehicle 201 and a second vehicle 203 are observed at a first location and are continued to be tracked until a second time where it is observed that Vehicle 203 collides with vehicle 201 at location 205. Upon detection of the accident pattern indicative of the collision (described elsewhere in this text), information may be transmitted to vehicles 201 and 203 or to emergency authorities or others. The information may contain the travel history of the two vehicles including their locations and speeds and driving behavior.
[0075] In addition to the above triggers, communications may be initiated with other surveillance systems when a vehicle moves out of the surveillance area and if there was a reason to continue monitoring it in other quadrants or surveillance areas. [0076] In an embodiment, a scenario for aerial surveillance is:
1 ) Identify vehicles entering the surveillance area
2) Track each vehicle and determine speed and acceleration through the area 3) Communicate with weather and on-vehicle surveillance systems and identify risky driving behavior patterns associated with the current driving conditions.
4) Identify vehicles exhibiting risky driving behavior in the surveillance area by comparison of movements of each vehicle relative to the road network and speed limits and appropriate driving behavior patterns for the driving conditions.
5) Do one or more of the following: a. Notify local surveillance assets to start actively monitoring vehicles with risky behavior by transmitting location and trajectory information b. Notify individual vehicle monitoring systems in the vicinity of vehicles that are driving in a risky manner, of the risk, and make sure that monitoring systems are activated c. When vehicles are near the boundary of the aerial surveillance area, notify the adjacent aerial surveillance areas to actively monitor the incoming vehicle.
6) Store the above information for a specified period of time until such time as no accident is detected.
7) If an accident is predicted to happen, for example, due to proximity warnings from aerial surveillance, identify the vehicles involved and transmit the stored information of the driving path of the vehicles involved to the on-vehicle monitoring systems, the insurance company and /or the authorities or emergency responders.
[0077] Aerial surveillance at lower altitudes may comprise passive monitoring, for example, at a busy intersection where many accidents are known to happen simply scan the intersection recording a time series of information (for example video) and simultaneously be performing pattern recognition analysis on the information for patterns that would indicate an accident or impending accident. Once an accident is detected or is imminent, the time series data that is pertinent to the accident, is transferred to an analysis station or the authorities or to vehicles involved in the accident.
[0078] Surveillance system at an interchange, for example, may not be on an air vehicle, but could be attached to a pole or other structure where sensors are high above the interchange, so effectively there is an aerial view of the interchange.
[0079] Active surveillance may be initiated when any passive surveillance system detects a pattern of concern. Active surveillance would occur when a passive surveillance system deviated from it standard sweep path to monitor a specific vehicle or vehicles or a specific location.
6.2.1.2 On-vehicle Surveillance
[0080] Vehicles equipped with sensors that measure vehicle motion, and vehicle behavior and/or motion and behavior of adjacent vehicles fall into this category and are part of embodiments of this invention. [0081] On-vehicle sensors are monitored for patterns indicative of an accident occurrence or an impending accident. These patterns, for example, could be rapid changes in acceleration, proximity alerts either from video analysis or other
electromagnetic monitoring such as sonar, or infrared.
6.2.1.3 Event Surveillance [0082] Once an accident pattern is detected by any of one of the surveillance systems, then, provided the various system are in communication, the system that detects a pattern transmits the information about the pattern (when and where and what) and requests that other systems be deployed and/or focused on the event of interest.
6.2.1.4 Ground Based Surveillance
[0083] Ground based surveillance can be one of:
• Sensors embedded in pavement that detect things such as traffic counts or average speed of vehicles;
• Localized weather conditions • Ice or other covering of the road surface
• Images of traffic passing past a fixed position.
[0084] Any of the above information can be included in an accident pattern to predict when an accident will happen or has happened.
6.2.2 Remote Sensing
[0085] Remote sensing such as analysis of imagery from satellites can provide general information about driving conditions, for example, weather. Resolution of imagery would typically be on the other of 1 square meter or more, so in most cases, you could not discern an individual vehicle.
6.2.3 Aerial Vehicles
[0086] A variety of flying vehicles can be used for aerial surveillance.
Depending on the design criteria (the altitude of flight, the time in the air, the area of coverage, the weather conditions anticipated), different aerial vehicles are better suited for different applications. Basic types of aerial vehicles include fixed-wing, traditional helicopters, multi-prop copters such as a hexi-copter, blimps or dirigibles; and variations or combinations of the above.
6.2.4 Deployment
6.2.4.1 Fixed Location
[0087] A fixed location implies that the aerial vehicle is normally housed on the ground when not in use, in a single location that is more or less central to area under surveillance. Size of the vehicle will depend on the application.
6.2.4.2 Police or other moving vehicle
[0088] Police or other emergency vehicles can be equipped with small ROVs or autonomous flying vehicles which can then be deployed when necessary. For real-time applications, such as deployment immediately after sensors indicate that an accident has occurred, it will be necessary to have the flying device attached to the exterior of the vehicle or within a compartment that has a hatch that can open for deployment (Or simply keep in the trunk of the vehicle for manual deployment. If fixed-wing, then some type of catapult may be attached to the flight vehicle; if a rotary type flying vehicle, then some type of secure attachment that can release quickly and/or a spring or chemical propellant that can propel the vehicle vertically.
6.2.4.3 Always in flight
[0089] Air vehicle can be designed with electric motors powered by batteries which are in turn charged with solar panels. Alternatively, very light weight slow moving fixed wing or blimp type vehicles can be up in the air for extended periods of time with minimal fuel.
6.2.4.4 Airport
[0090] In most cases using an airport for deployment would be reserved for large aerial vehicles that are subject to the same flight restrictions as a piloted aircraft.
6.2.4.5 On-Vehicle
[0091] When wishing to capture information about an accident while it happens or shortly thereafter, in an embodiment, a flight vehicle is in communication with sensors within the vehicle such as accelerometers. When either an impending accident or an accident in progress is detected via analysis of patterns, the flight vehicle is launched very rapidly in an attempt to have a vertical launch should the vehicle begin to roll over. The air vehicle could be a rotary type or a type of rocket with a deployable parachute. A rocket or similar device could be deployed much like a torpedo, from a tube, but vertically oriented.
6.2.5 Piloting System
6.2.5.1 Autonomous
[0092] Virtually any type of airframe can be made to take off or, land and fly autonomously. This would require location and altitude sensors as well as some frame of reference, for example a digital map or a location beacon either at a fixed location or on a vehicle of interest.
6.2.5.2 ROV
[0093] If the application for surveillance is for accident site observation or normally scheduled surveillance of a predefined area, then the flying vehicle can be piloted remotely 6.2.5.3 Combination
[0094] In the above stated scenario for an ROV, a combination of remote piloting and autonomous flight can be used. For example, take-off and landing can be remotely piloted, while in surveillance mode, the flight could be autonomous.
6.2.5.4 With a pilot
[0095] For high altitude flight or large vehicle flight, then, in an embodiment, an aerial vehicle will contain a human pilot.
6.2.6 Type of Aerial Vehicles
6.2.6.1 Quad or other copter [0096] There is a variety of remotely operated or semi-autonomous vehicle which achieve lift using one or more propellers. Configuration with 4 or 6 blades usually mounted in the same plain and all oriented with the direction of thrust perpendicular to the mounting plane. These copter or drones as they are often called come in a variety of sizes from less than a kilogram in weight up to 20 kilograms or more
6.2.6.2 Fixed Wing
[0097] Of course fixed wing aircraft can be flown either piloted, autonomous or semi-autonomously.
6.2.6.3 Blimp - for High altitude long term surveillance
[0098] Blimps have the advantage that they can stay in flight for extended periods as most of the energy is directed to moving the vehicle rather than keeping it aloft and the helium provides most of the lift.
6.2.6.4 Projectile with parachute
[0099] For applications where close aerial surveillance is required at an accident scene either while the accident is occurring or immediately thereafter, a parachute mounted sensor suite which comprises a camera and perhaps other sensing devices is contained in a cylindrical or other aerodynamic container which in turn is attached to a chemical propellant or compressed gas engine or a kinetic energy device (for example a spring) capable of propelling the sensor suite and parachute at rapid speed above the vehicle. [0100] The motor or other propulsion device is actuated by a signal from the vehicle monitoring system (or potentially a remote systems) when it detects an accident about to happen or that is in progress. Optionally, the vehicle monitoring system is equipped with a sensor or sensors (such as a gyroscope) that can be used to determine if the vehicle is oriented with the top of the car being up (within a threshold angle). If the top of the vehicle is not up and within the threshold angle of being perpendicular to the vertical direction, the apparatus is not launched - to prohibit injury or damage to objects or people on the ground.
[0101] FIG 6 depicts a launch scenario in an embodiment of this invention that utilizes a projectile with a parachute. The apparatus is housed in a weatherproof container with either a retractable hatch or cover that is penetrable by the apparatus. The sensor suite is in standby mode 602 and in communication with a pattern detection module in the vehicle. If an accident pattern is identified 604, the launch mechanism is checked to be in a vertical position 606 and if so, the apparatus is launched 608, the hatch is either opened (prior to engine ignition) or penetrated when the apparatus lifts off. An example of a mechanism for launch would be much like a jack-in-the-box where a cover and latch hold into place the projectile which is mounted on a spring. Once the latch is opened, the projectile is free to exit and the spring force is released propelling the projectile into the air. [0102] Simultaneously with the launch (or previous to the launch) a
communication link is established between the launched sensor suite and a directional beacon on the vehicle 610.
[0103] At the apex of the flight of the apparatus (or at a specified time or altitude above the vehicle, the parachute is deployed by various means known in the art. By using a direction antenna or other means, the vehicle is located and tracked.
[0104] The camera is mounted on a gimbal and servo motors keep the lens oriented towards the car. There may optionally be a servo to stabilize the compass direction of the view of the camera, as the parachute and apparatus may be rotating.
[0105] Optionally, the apparatus is equipped with a propeller or propellant to provide a horizontal and/or vertical forces to either prolong the length of time the apparatus can stay airborne or to be able to circle the vehicle for measurements at various altitudes above the vehicle or angles around the vehicle.
[0106] The camera may be equipped with a zoom lens to capture more or less detail of the accident scene. [0107] Potential triggers (patterns) that would initiate a launch are the same as described in the section on indications of an accident occurring or about to happen.
[0108] FIG. 4 depicts a similar solution in the art (from European Patent
Application EP 0466499 A1 ). In EP 0466499 this is a battlefield aerial surveillance device where a rocket is launched from a ground vehicle 49 at time (A). At time (B) near the apex of the flight the aerodynamic casing of the rocket is separated exposing the surveillance apparatus 9 with a parachute 15 comprising a camera with a field of view 7 and configured with a device to prohibit rotation 29. The video is transmitted to a ground vehicle at time (E).
[0109] The present invention differs from EP0466499 in that the rocket deployment is from the vehicle being surveyed and the deployment is initiated based on sensor output and pattern recognition. In addition, the camera may be able to be directed and the parachute may be steerable. In addition, image software may be able to detect the vehicle of interest and zoom in on it.
6.2.6.5 Autonomous air vehicle deployed from a ground vehicle [0110] In an embodiment of this invention an autonomous air vehicle is deployable from a ground vehicle. The air vehicle comprises a communication module that is in wireless communication with on-board sensors in the ground vehicle. If a pattern is detected by the surveillance module in the ground vehicle that indicates that an accident in progress or that an accident has happened, this in-turn triggers the launch of the autonomous air vehicle.
[0111] Commercial quadcopters are available in a small form factor, for example as shown in FIG. 5 (from https://www.ajwavsinnovatina.com/products/mecam.htm). A quadcopter of this size could be launched from a vehicle in a variety of ways: • A rigid quadcopter could be contained in a spherical container housed in a vertical tube imbedded in the vehicle. A spring loaded propulsion mechanism much like the mechanism used to proper a ball bearing in a pin-ball machine could be held in place by a latch. The latch could be triggered by the recognition of an accident pattern.
• Alternatively, the cross arms of the quadcopter could be folded at a point where the two arms cross in the center such that two adjacent motors are nearly touching one another on opposite sides. The apparatus in the folded stated could be housed in a bullet or rocket shaped container and launch much like the parachute system of the previous section.
6.2.7 Combination of the above
[0112] In an embodiment, a rocket is used to deploy payload of a sensor suite attached to a fixed wing or rotary aircraft. An example of a vehicle that may be suitable for this type of deployment is show, for example in US Patent US 8444082 B1 .
6.2.8 Fuel for Aerial Vehicle
[0113] Conventional methods for fueling an aerial vehicle include many petroleum products including gasoline, aviation fuel, jet fuel, alcohol and others. In addition, smaller aircraft designed for short flight can use light weight batteries and electric engines. For longer duration flights, for very light aircraft such as a blimp, solar panels or some form of photovoltaic can be affixed to surfaces of the craft to charge batteries while in flight.
6.2.9 Sensors
[0114] There is a variety of sensos that can be used to determine both vehicle movement and behavior and the conditions associated with the vehicle movement and behavior. Various type of sensors may be used with aerial vehilces, at fixe ground locations or within vehicles.
[0115] Numerous sensors can be used in vehicle surveillance. A comprehensive summary can be found in: A Summary of Vehicle Detection and Surveillance
Technologies used in Intelligent Transportation Systems (see https://wwwihwa.dot.aov/ohim/tvtw/vdstits.pdf) although this text is somewhat dated now.
[0116] Examples of sensor that can be deployed from Aerial vehicles:
• Camera - both natural color and infrared · LIDAR
• GPS
• Gyroscope
• Digital Map
[0117] Examples of sensor that can be part of a vehicle or in a vehicle · Accelerometers
• Temperature sensors
• Forward facing camera
• Backup camera
• Air bag deployment · Gyroscope
• GPS
• Engine sensors
• Tire pressure
• Speedometer · Digital map
• Seat belt or seat pressure sensor • ABS braking actuated
[0118] Examples of Road Side Sensors
• Traffic Counts
• Average Speed Limit · Weather and Road Condition Sensors
• Vehicle Tracking
[0119] Part of a worldwide research effort in intelligent transportation systems (ITS), there is a variety of methods using sensor networks of various types to detect the movement of vehicles and track them. See for example: A Study on Vehicle Detection and Tracking Using Wireless Sensor Networks 2010, by G. Padmavathi, D.
Shanmugapriya, M. Kalaivani
(ii ; A^^
[0120] Examples of types of satellite imagery
• Color image · Infrared image
• Radar / Lidar
6.2.10 Location Determination Devices
6.2.10.1 Satellite based
[0121] In virtually all of the surveillance methods, there is a need to know where the surveillance vehicle is with respect to a vehicle or accident location and/or with respect to the earth, for example, latitude and longitude.
[0122] For a location relative to the earth, typically a GPS (Global Position Satellite Receiver) is used. This type of device can also be used to determine a low resolution altitude. As a GPS requires a line-of-site view of 3 (or more) satellites to determine a position, sometimes is may be necessary to augment a location
determination with techniques known in the art such as dead reckoning using a gyroscope, and/or a digital compass or other sensors. There are also other satellite location systems available from both Russia and the European Union.
[0123] In the case of a sensor suite deployed from a vehicle that will imminently be in an accident or was in an accident, it may only be necessary for the sensor suite to orient itself, relative to the plane of the earth and the location of the vehicle. This orientation can occur by using a beacon mounted in the vehicle.
6.2.10.2 Location determination: local area network triangulation
[0124] Most portable electronic devices are equipped with some form of local area networking, for example, Bluetooth Low Energy. As part of the protocol for a communication standard such as this, there is a parameter that is a measure of signal strength of the radio frequency signal that is received by a receiver from a transmitter. It is well known in the art that by knowing the signal strength from three different transmitters that are geographic spaced, the relative location of the receiver with respect to the three transmitters can be determined. Of course there is a substantial amount of error in the signal strength measurement so this method only provides an approximate relative location.
6.2.10.3 Location determination: radio direction finding
[0125] If a vehicle is equipped with a radio frequency transmitter and as part of a sensor suite that is deployed using a rocket or a aerial vehicle deployed from the vehicle, there is a directional antenna that receives an indication of signal strength of the transmitted frequency from the radio transmitter, it is possible to determine the relative location of the sensor suite to the vehicle - so that video or other sensors can be directed towards the vehicle.
6.2.10.4 Location determination: dead reckoning [0126] When a sensor suite is deployed from a moving vehicle, the vehicle and/or the sensor suite are equipped with sensors that can measure the velocity and direction of motion. In an embodiment, at the time of deployment of the sensor suite, the direction of motion and the speed of the vehicle is known. The acceleration profile of the sensor suite based on the propellant system used and the relative direction of deployment with respect to the vehicle motion is known. Assuming the that vehicle will continue to travel in the same direction at the same speed, the relative position of the sensor suite with respect to the vehicle can be calculated through time. This will enable a gimbal mounted camera to be continually pointed in the direction of the vehicle. Of course this assumes that the vehicle continues to move in the same direction and speed which would not necessary be the case if a collision occurs. Therefore, in an embodiment, the camera would initially point towards the vehicle, and would further register an image of the vehicle and track the vehicle using conventional image analysis software described elsewhere in this document such that the video can be trained on the vehicle and not stay on the anticipated trajectory of the vehicle.
6.2.11 Altimeter
[0127] There are a variety of altimeters known in the art, which include ones based on barometric pressure and/or a combination of barometric pressure and gps measurements and potential gyroscopic measurements. Altitude is important when dealing with position relative to the earth rather than relative to a moving vehicle.
6.2.12 Digital Map and Terrain Model
[0128] In scenarios where a pattern is used to identify when a vehicle is driving erratically or when a vehicle is off the road, then once the location of a vehicle is identified, it must be compared with a digital road map in order to determine the above. Of course the accuracy of the measurement of location of the vehicle and the accuracy of the digital map must be sufficient such that there is a high confidence of where the vehicle is relative to the road.
6.2.13 Patterns
[0129] A pattern is the term used to describe one or more time-series of sensor readings that can be analyzed to: · Predict that an accident will happen
• Determine that an accident has happened
• Predict the extent of the damage and injury incurred during an accident
• Simply track a vehicle over time and its relationship to a road network [0130] The sensors involved can be associated with any or all of the surveillance systems described above. Associated with each pattern is a statistical uncertainty in the prediction. Patterns may comprise a time series of a specific sensor
measurements or may comprise a collection of calculated parameters inferred from a variety of sensor measurements. For example, acceleration could be measured directly by an accelerometer or inferred from location measurements over time from a GPS receiver and/or a combination of these two types of measurements could be used to determine a mean acceleration for a given time interval by a weighted average of the two measurements, with more weight being attributed to the measurement deemed the most accurate.
[0131] Patterns could also be analyzed in the frequency domain using Fourier analysis
[0132] Patterns are determined by some form of multivariable analysis such as machine learning where data is collected from sensors for many accidents where the extent of damage and severity of impact are known.
[0133] Raster image analysis can be considered another form of pattern analysis. In this case vehicles are identified and tracked.
[0134] Conventional analysis of measurements from several types of sensors that measure different physical parameters over time may not be able to identify complex patterns associated with an impending accident. Or it might not be obvious how these measurements are related to a known accident hazard (for example ice on the road). Machine learning techniques applied to historical data may identify complex patterns that relate sensor output overtime to accident potential without identifying the underlying cause or causes of the accident potential. [0135] Determination of patterns that can be used to predict or detect accidents and the results of accidents are described in: U.S. Patent Application titled "SYSTEM AND METHOD FOR USE OF PATTERN RECOGNITION IN ASSESSING OR MONITORING VEHICLE STATUS OR OPERATOR DRIVING BEHAVIOR", Application No. 13/679,722, filed November 16, 2012; which claims the benefit of priority to U.S. Provisional Patent Application No. 61/578,51 1 , filed December 21 , 201 1 ; PCT/US12/71487 titled "SYSTEMS AND METHODS FOR ASSESSING OR MONITORING VEHICLE STATUS OR OPERATOR STATUS" filed 21 December 2012; and 14/317624 titled "System and method for Determining Of Vehicle Accident information" file on 27 June 2014; each of which the above applications are herein incorporated by reference.
[0136] Below are examples of physical events that can be used to anticipate an impending accident. These indicators can be measured in a variety of ways with a variety of sensors and one or more of these measurements can be incorporated into patterns. These are just examples and as stated above, patterns may be determined using machine learning that cannot be correlated with a single physical event, but never-the-less have a strong correlation with an impending accident and an accident in progress. The indicators below and other can be used to, for example, to initiate an airborne launch of a sensor suite. Examples of patterns and/or physical events are:
• Rapid deceleration above a specific threshold that would indicate emergency braking. One method of detection of rapid deceleration would be to monitoring vehicle onboard accelerometers and gyros. Airbag deployment
• Rapid change in direction that would indicated spinning on ice or locked brakes [again, augmented by accelerometer and gyro sensing]
· Extremely close proximity to other vehicles as detected by video camera or other proximity sensor such as sonar, radar or infrared
[0137] Patterns may be expressed as polynomial equation; they may be a threshold constant or upper and lower range for a specific sensor; they may be based on frequency and/or amplitude analysis of a single type or multiple types of sensors or they could be a statistical mean value for one or more sensor outputs or environmental factors. Patterns will change over time as more data is added, more sophisticated analysis is performed or more sensor types are available for on-board measurement. Patterns for one type of vehicle may be entirely different than for another type of vehicle. This may be due to different sensor suites being available or different physical attributes of the vehicle. 6.2.13.1 Image analysis software to detect ground vehicles
[0138] In order to detect vehicles from raster images, one method is to use vehicle recognition software. Patterns in an image that are indicative of a vehicle. There are several methods for analyzing both video, still and infrared imagery to detect vehicles. One example of a method for recognizing vehicles in a image is Real-time People and Vehicle Detection from UAV Imagery by Gaszczak, A, etal (see
http://breckon.eu/toby/pubiicatjons/papers/qaszczak1 1 uavpeopie.pdn. By tracking the location of an identified vehicle through an image over time, the acceleration and velocity of the vehicle can also be determined. If the image is orthorectified to align with a digital road network, then the location of a vehicle with respect to the road network can be determined.
6.2.13.2 Patterns from vehicle sensors
[0139] New data is collected from vehicle on-board sensors and from external feeds such as sensor suites that are part of the road network system or for example from weather satellites. At given time intervals the data for the last time period is stored and analyzed and the older data is thrown out (provided no patterns of interest were detected). Alternatively, the data is stored in a memory stack of a set size where new data is added to top of the stack and the oldest data (at the bottom of the stack) is thrown out. At intervals which could correspond to the sample interval or multiples of the sampling rate, an accident pattern or impending accident pattern is looked for. If a patterns is detected, indicating an accident or impending accident has occurred or will occur, then the sampling rate may be increased to acquire more data per time period, and/or other sensor data, previously not being recorded, may be recorded.
[0140] The end of the accident event, in an embodiment, is defined when the vehicle is stationary. Once the accident is over, the stored data is analyzed to detect damage and injury patterns. If accident and/or injury patterns are detected, then the location and estimated damage and injury associated with these patterns is recorded and transmitted to pertinent individuals or computer servers.
[0141] If the severity of injury anticipated by the analysis is sufficient, then an ambulance and/or paramedic is contacted (provided communication is available). If a tow is needed, then a tow vehicle is called, provided the local information for such services is available. An insurance adjuster is contacted. Parts and repair services are queried to check availability. Depending on the configuration, information about the accident is displayed on an infotainment screen in the vehicle or on an authorized portable device. [0142] The raw data and/or the sensor analysis is transferred to a server via the communication network (wireless, or otherwise) for inclusion into the central database and for future determination of accident, damage and injury patterns.
[0143] FIG 3 illustrates and embodiment using a monitoring system within the vehicle. Real-time time series data is acquired from many sensors on-board the vehicle 302 and transmitted to an Accident Prediction Module 310. In addition, the Accident Prediction Module 310 receives external information from other surveillance systems 308 by wireless communication. At intervals, the Accident Prediction Module 310 performs analysis comparing the sensor data feeds 302, 308 to accident patterns acquired from a historical database 304. If an accident pattern is matched to the sensor feeds, this triggers recording of detailed information and a search for damage and injury patterns within the data. If a damage or injury pattern is detected, then analysis is performed concerning the extent of damage or injury and the location of damage or injury and this information along with the underlying data is transmitted to interested parties. [0144] It may be desirable to limit the data/parameters that are utilized and make some simplifying assumptions.
[0145] Accident detection patterns could be inferred simply by knowing the weight of the car and inferring a maximum acceleration or change in momentum that would indicate an accident occurred. Damage and Injury patterns are approximated by relating specific accident descriptions to ranges of acceleration or momentum and the direction of impact. Once accidents are categorized as to the location and severity of damage in terms of anticipated range of acceleration that occurred during an event, then a cross correlation between repairs and injury treatments required for a given vehicle type can be made with the each range of acceleration. [0146] Examples of patterns to record are: • Number of Occupants of the vehicle
• Impact zone
• Roll or skid characteristics
• Deceleration
• Movement of occupants
• Distortion of passenger compartment
• Breaking of glass
• Thermal (indication of fire)
• Rear facing camera - following car too close
• Front facing camera - too close to forward car; cars serving in from other lanes
• Rapid steering changes
• Brakes Locking
6.2.13.3 Pre-accident aerial surveillance
[0147] It is desirable to predict when an accident will occur prior to it happening so the accident can be observed while in progress or alternatively to initiate some action that would avoid the accident. Some examples of patterns that might indicate an accident about to happen would be:
• A vehicle exceeding the speed limit over a certain threshold
• Erratic driving behavior
• Vehicles without a safe distance between
• If questionable behavior found:
Record video; record speed/acceleration profile; record lane changes
Monitor for a prescribed time or until an accident occurs OCR the license plate; send warning message for continued bad driving; citation if bad behavior does not cease.
• If no questionable behavior lock onto another vehicle
6.2.13.4 Aerial monitoring of distance between vehicles [0148] Using image analysis techniques described previously, it is possible to identify specific vehicles in images and in successive images, identify those same vehicles as they move through an area. By knowing the relative time when the images were acquired and the location of the vehicles in the image, parameters or patterns can be determined. [0149] If the relative distance between two vehicles as determined in either macro or local aerial monitoring becomes less than a threshold value - indicating that collision is about to occur - several actions could happen:
• Information acquired about the vehicles of interest for the time leading up to the time period in question is set aside and stored. · Velocity and Accelerations are calculated
• Event surveillance assets are communicated with and deployed
Sensor systems within the cars themselves are contacted via wireless communication and instructed to record information at a rapid rate.
Pre-accident patterns from the vehicle/s are compared with patterns either from macro aerial or local aerial surveillance systems to verify the analysis
If the aerial surveillance is capable, acquire the license number of the vehicles involved
If one or more of the vehicles is moving out of the surveillance area, alert adjacent surveillance areas to be on the look-out.
When in communication with vehicle equipped with deployable surveillance systems, signal that one should be deployed 6.2.13.5 On-vehicle monitoring of adjacent vehicles
[0150] A variety of methods exist in the art to determine the distance between one vehicle and adjacent ones. There are a variety of sensors that can be used to detect an adjacent vehicle. Video cameras for example could be used in conjunction with vehicle detection software to know when an adjacent vehicle is too close. Adjacent vehicles will reflect light and other forms of electromagnetic radiation such as infrared, and / or may be equipped with an active transponder which transmits a signal which can be located and identified.
6.2.13.6 On-vehicle accident detection [0151] The ultimate goal of using in-vehicle accident detection would be to anticipate an accident before it happens to enable deployment of emergency services and also to assess the severity of the impending accident in real-time. Rapid
assessment immediately after an accident is also the goal - should there be no way to detect the accident beforehand. [0152] Modern vehicles are generally equipped with a variety of sensors that measure physical parameters associated with the moving vehicle. These sensors can be a part of the vehicle or within the vehicle, for example as part of a mobile device.
[0153] Vehicle behavior can be inferred based on patterns exhibited in the sensor data overtime -either from observations of a single type of sensor or a sensor suite, for example a gyroscope and also a 3 component accelerometer. Rapid changes in the orientation of the vehicle may be exhibited by changes in the values measured by a gyroscope and/or accelerometers. It is intuitively known, for example, if a car is spinning on wet pavement or on ice, that there is a strong likelihood that the vehicle will sustain damage and/or passengers will be injured. However, this likelihood can be quantified by tracking patterns in the sensor output leading up to previous accidents with known damage and injury - performing statistical analysis on those patterns. It may be determined that if a vehicle spins 360 degrees within 3 seconds, when the initial speed was 90 kph, that there is a 90 percent probability that the vehicle will flip over. If we take into account the type of vehicle, it may be apparent that a vehicle with a high center of gravity will have a higher probability of rolling over than a vehicle with a low center of gravity. It may further be found that if a vehicle flips with the initial speed that there is a 50% probably of severe injury to a passenger in the front seat.
6.2.13.7 Remote accident detection - aerial
[0154] Patterns observable from aerial surveillance may indicate: · A burning vehicle (infrared signature)
• Vehicle off-road or eschew on the lane (based on image recognition of vehicle location when compared to a digital road map)
• Overturned vehicle (as indicated by changes in the image recognition profile of the vehicle)
6.2.13.8 On-vehicle accident detection hardware
[0155] Hardware for an on-board accident detection and analysis system comprises the following components:
• a processor which monitors and analyses onboard sensors used to detect vehicle activity and driver behavior;
· an on-board database comprising: o vehicle specific information; o patterns, for the individual vehicle type, used to analyze sensor data to detect accidents and to assess resulting injury and damage and useful to predict driver behavior and driver / insurance risk; o driver information; o emergency contact information;
• one or more of several data transmission components which can include both short range wireless (for example Bluetooth), long range wireless transmission (for example mobile service) and wired data communication component - which can communicate with external servers to transmit both raw sensor data and damage/injury estimation and to provide software and database updates to the vehicle on-board system. • a remote central server in communication with multiple vehicle systems comprising : o one or more computers; o a comprehensive central database located on one or more servers comprising:
■ historical information from several sources
■ raw sensor data or indices derived from the raw sensor data from individual vehicles. o patterns for all vehicle types and areas o geographic Information o spatial, temporal, and severity Information pertaining to historical accident incidents o metadata
6.2.14 Procedures Concerning Response to Patterns
6.2.14.1 Response after accident detected
[0156] If an accident pattern is detected by any surveillance method, then in an embodiment, the following scenario would occur:
• send wireless transmission to a surveillance dispatch
• look for drones, fixed cameras or moving camera that are in the vicinity of the detected accident
• dispatch and/or point surveillance device at the accident
• once an accident is identified, circle from several angles for a 3-D view
• transmit Data to an Accident Investigator and/or emergency services
6.2.14.2 Response for cars equipped with own drone, once accident is
detected
[0157] Once a pattern from an incident detection system is identified that would indicate with a high degree of certainty that an accident is about to happen, the system will initiate the following sequence: 1 ) launch drone or rocket a) if the vehicle is equipped with a spring loaded hatch, open it b) check the orientation of the car to make sure that the launch will be relatively vertical - based on vehicle sensor input such as magnetometers or accelerometers.
2) simultaneous to launch or prior to the launch, establish wireless communication with the vehicle including one or more location beacons
3) once the drone or rocket reaches its apex, use a gyro in the device to stabilize orientation (down facing down) 4) use a gimbal mount and servo motors to orient a video camera or other sensor in the direction of the vehicle
5) if equipped, use vehicle recognition to zoom the camera or other sensors in such that the vehicle or vehicle that are in the accident fill the field of view (in other embodiment both a near and fare range image are taken. 6) do several passes of vehicle at various altitudes with respect to the vehicle
altitude and several view angles
6.2.15 Historical Analysis of Collected Data
6.2.15.1 Patterns for sensor records and accident reports
[0158] The following tasks comprise one method to determine accident patterns initially based on accident reports:
■ Develop transfer functions between observations in historical databases built from accident reports to on-board sensor measurements that are indicative of the observed damage. For example, an accident impact could be inferred when a rapid deceleration is detected either by accelerometer measurements or change in speed measurements. Location, and relative speed of an impact can be inferred based on 3 component acceleration. Alternatively, a side impact can be inferred when a side airbag is deployed. ■ Test the transfer function by predicting vehicle damage and resulting cost based on sensor data after an accident. Confirm the prediction based on conventional accident and insurance adjustor reports.
■ Refine the transfer functions as necessary to increase statistical reliability. ■ Gradually incorporate sensor measurements and create a more granular
predictive models based solely on sensor measurements (without inference from historical data not from sensors). In the initial database collisions may be classified based on relative speed of impact, for example. With more accurate speed data from sensors and vehicle weights, the classification could be changed to an impact momentum in N/m2 using finer ranges for classification rather than simply an approximate relative speed of collision.
6.2.15.2 Comparison of traffic, weather and sensor data and the likelihood of an accident
[0159] After performing analysis of many sensors reading from many vehicle and correlating this information with weather information and road condition information, patterns may emerge that can be used to identify area and timeframes where accident are very likely to occur. These predictions can then be used to deploy aerial surveillance system for increased surveillance in accident prone zone or at accident prone timeframes.
6.2.16 Combining Datasets Measuring the Same Parameter From
Different Types of Sensors or Sensors with Differing Resolution
[0160] Ideally, it would be easier to identify specific patterns indicative of an imminent accident, an accident event or damage and injury related to an accident if the database comprised identical measurements, for example if the sampling rate and resolution and accuracy of an accelerometer was always the same. In practice, this would never happen as sensor technology continues to advance. Therefor predictions based on low resolution, accuracy or a slow sampling rate must have an indication that the prediction is less certain than a prediction based on better quality information. As new, better quality information is stored in the database, older, poorer quality data should be removed from the database and patterns adjusted accordingly.
[0161] Raw data may need to be parameterized in such a way as they can be used into a numeric model. An example of parameterization would be to characterize incidents into a grouping. For example, it may be desirable to collectively refer to impact force based on accelerometer readings in ranges in units of meters/second2 rather than actual recorded values or as a mean over a time frame.
6.2.17 Database Maintenance [0162] Database maintenance comprises removing older or poorer quality data, continually updating the patterns as newer and better information comes on line. In addition, as the database increases in size, patterns can be broken into smaller subdivisions, for example, an accident pattern could be vehicle type specific as to vehicle class specific.
6.2.18 Database Content
[0163] In addition to raw sensor output, patterns, currency of data and the resolution and accuracy of the data must be stored. Other pertinent information is:
• Location of fixed sensors systems including range of operation
• Range of fixed based aerial vehicles and sweep are; length of deployment; weather extremes that operation can occur.
• Standard sensor suites in a stock vehicles - including access protocol and
frequency, procession and accuracy of measurements
• Video resolution and color perception and/or frequency range
6.3 Distributed Vehicle Database System [0164] FIG. 7 is an example of a distributed vehicle database system. System hardware is distributed between one or more central servers 702 and client systems 706. The client systems can include, for example, passenger vehicles 708, trucks 720, satellite servers 712, external data feeds, such as weather 714 and traffic 718, onboard vehicle monitoring systems and portable devices 710. [0165] Information is communicated in the form of a query or a notification from the one or more central servers 702 to clients 706 via a radio broadcast 704. All clients have a radio receiver that conforms to the radio frequency and standard of broadcast as the radio broadcast device connected to the central server/s. All of the clients 706 in range of the broadcast receive the broadcast and digest the query or bulletin. If the query or bulletin pertains to the particular client 706, each client 706 establishes two-way communication, for example, over a mobile network 724, with the central server/s 702 and uploads to the central server/s 702 the requested information for a query or downloads the available information to the clients 706 for a bulletin. [0166] As not all data are needed for every situation, the data are maintained in the device or system where it was generated and/or in a regional client of some kind. All data does not need to be uploaded to a central server for storage and subsequent analysis unless a central server asked for it.
[0167] Below are examples of data that could be stored:
· raw sensor data recorded as time histories
• accelerometer readings over time
• all sensors that can be queried via a vehicle diagnostic port (onboard diagnostics OBD) including: mass flow, oxygen, seat belt, air bag, tire pressure, gps, accelerometers, gyroscope, and more.
· sensor data derived from mobile or 3rd party devices within the vehicle - including accelerometers, gyroscope, air pressure, gps.
• VIN, and/or make model and model year, accessories such as larger than normal tires, engine type and size, etc.
• Wheel diameter
· Tire tread pattern / age of tires/ inflation pressure
• Characteristic about the load in the vehicle (could for example be provided by RFID tags on cargo or passengers)
• Number of passengers
• Gross and Net Vehicle Weight
· Load distribution
• Environmental sensors and feeds either from on-board vehicle sensors or sensors external to the vehicle
[0168] The driver may be identified either by manually input, or via automatic connection between the vehicle and a mobile device of the driver, or by visual or audible input query by controlling software within the car. These are just examples and any type of identification of the driver could be used. The driver could carry an RFID tag that identifies her, for example.
[0169] Information relayed from roadside sensors or sensors or devices that monitor road conditions or weather can be stored. These can come from Bluetooth connections, side bands on radio stations (for example traffic); internet feeds and peer- to-peer networks from other vehicles.
[0170] Other information may concern repair history of warn vehicle parts as related to all the above mentioned information. This information could be stored directly in a server at a repair shop - for example.
6.3.1 Method of Use
[0171] All information stored in the distributed database is optionally spatially and/or temporally referenced. In addition, the information can be referenced based on the type of vehicle and/or how the vehicle is equipped and/or by the driver of the vehicle. Patterns that relate changes in one or more sensor and environmental data overtime to events are also stored in the database, both in vehicles were the patterns are pertinent and in a central database located on a central server or in satellite databases.
[0172] Patterns are created by analyzing many datasets with known events happening and developing a predictive model of the event based on the available data. Patterns are updated by a central server in communication with client systems through the process of querying the client system (such as a vehicle fleet) and having clients (vehicles) that match the requirements for the pattern, respond with relaying data for the event in question.
[0173] FIG. 8 explains how an embodiment of the system is used. The system consists of one or more central servers 802 equipped with one or more processors and one or more client devices 804 (see client devices as depicted in FIG. 7). The central server/s is configured with software to predict events and to make assessments. The central server/s is further equipped with a list of events of interest.
[0174] To predict an event, the central server/s 802 needs to correlate potential contributing factors to particular events or types of events 806. In some embodiments, the prediction or assessment is of regional or local in interest only and this will effect what sources of information are used 810. The central server/s 802 broadcasts, using a radio broadcast transmitter, a request to all potential clients 804 including identifying parameters of type of information and also what type of vehicle or other source for the information that is needed 812. This request is generated by instructions loaded in one or more processors and referred to as a query engine. Clients 804 are typically listening for broadcasts, using a radio broadcast receiver, from the central server/s 802. Once a broadcast is detected by a client, each client 804 parses the query or bulletin that was broadcast and determines whether it is able to comply with all of the
requirements 816. If the client 804 can comply and the request (or query) is for information, the client will 1 ) relay the requested information to the central server/s 802, using a two-way radio transceiver, and/or 2) start collecting the requested information from sensors or other devices. Once a packet of information is acquired, the client 804 establishes a two-way radio communication link using the transceiver and uploads the information 818 to the central server 802 which receives the information on a radio transceiver that is part of the central server/s.
[0175] The central server/s 802 then uses a form of statistical analysis to establish a relationship between the uploaded information from numerous clients 804 and creates predictive models 820 in the form of patterns and indicators. A broadcast, using the transmitter, is made to the clients 804 to let them know the derived patterns, and indicators are available for usage 822. Clients 804 that have need of the patterns and indicators 824, then download the information 826, then monitor sensors and other incoming information to determine if patterns or indicators occur that are indicate an event has happened 828. [0176] FIG. 9 depicts how certain events trigger other events. A follow-on analysis may be necessary after certain types of events are detected to form a proper assessment of needs to be performed. An example of an assessment would be to indicate how much damage occurs when a vehicle is in an accident. Sensor output could be used to determine the type and severity of impact, and averages of repair bills for similar accidents in the vicinity of the current accident could be used to estimate the cost of repair. [0177] In FIG 9, a client 902 continuously monitors sensors and external feeds 904 and compares to patterns and indicators to detect an event. If an event is detected 906, pre-programmed activities are initiated that are associated with the event 908. One of these activities may be to inform the central server/s of the event and to upload information pertaining to the event. The client 902 establishes two way
communications with the central server/s and then uploads the information and data associated with the event. The central server/s 910 in turn receives the information 912 and determines the type of follow-up information that is needed 914. An example would be, an accident is detected; it is reported to the central server along with particulars about speed, location, severity of impact. The central server may be programmed to determine if the accident caused a traffic slow down; therefore, it would broadcast a bulletin that it is interested in knowing the speed of vehicles that are in the vicinity of the accident so it can determine what, if any, roads were affected by the accident 918. In response other clients 920 that meet the selection criteria would communicate with the central server and upload their speed 922.
[0178] In the same scenario above, the central server could also send out a request for available tow trucks that could be dispatched immediately to the scene. Yet another request would be to the insurance company of the vehicle involved in the accident; to emergency responders; and the like to send out a representative to the scene. Another example would be notifying local auto body shops of a potential client.
[0179] FIG. 10 describes periodic update of the patterns and indicators stored in the system. Patterns and indicators need to be updated by central server/s. This happens by determining when certain patterns are either too old or new information is available that needs to be incorporated. For example, a new sensor reading may have been determined to be useful in prediction of certain types of events and has previously not been included in the patterns for that event. The central server/s 1002 would send out a query for additional information to be used for this update including the type of sensor reading that are needed and other pertinent information 1006. Potential clients of interest receive the query and determines if they are a client 1004 of interest (have pertinent information) 1014. Clients 1004 of interest establish a link 1016 with the server/s 1002 and then upload pertinent information 1018. The server/s 1002 receives the information 1008, and retires older information 1010 and re-compute updated patterns and indicators 1012.
6.3.2 Normal mode of operation
[0180] Numerous databases exist that have valuable information for vehicle diagnostics and analysis and for warning vehicles of impending events about to happen. One problem is that all of these databases may have different means of access (different query language, different security, and different accessibility on a network). For a distributed database to work, then all the databases that are a part of the whole, must either be structured in a similar fashion (have the same schema and identifying elements with the same nomenclature) and have a common query language or there must be an intermediate step to translate one schema and one query language to another.
[0181] For example, in a vehicle collision database, identification of where damage was sustained on the vehicle and the severity of the damage must be codified so that a central server understands the information provided by a client.
[0182] In Table 1 , below, is an example of information that might be conveyed in a bulletin using the distributed vehicle database system - regarding a terrorist alert status. The scenario would be as follows: One or more of the central servers received a statement from the department of Homeland Security (a potential client) indicating that there will be an amber alert for a specific geographic area for a specified period.
[0183] The information in the alert could be represented with fields defined in table 1 and with their XML (extensible markup language) counterpart below that.
Alert Parameters
alert
A single active alert. Contains the following attributes start
The effective start date/time of the alert, in the form YYYY/MM/DD HH:MM, expressed in GMT end
The effective end date/time of the alert, in the form YYYY/MM/DD HH:MM, expressed in GMT type
The type of the alert. Possible choices are Imminent Threat Elevated Threat link
a URL to a PDF document which provides further details about the alert summary
A short plain text summary of the alert, details
A longer explanation of the alert. May contain HTML sectors
A possibly empty list of sectors which are impacted by this alert sector
An individual sector impacted by an alert. Represented as plain text locations
A possibly empty list of locations which are impacted by this alert location
An individual location impacted by an alert. Represented as plain text duration
A plain text description of the expected duration of this alert May be blank detail sections
A list of information specific to the threat detailed by the alert detail section
An individual detail section containing a specific piece of information for the alert, detail title
An individual detail section title. May contain HTML detail
An individual piece of detail related to the threat. May contain HTML
Alert in XML format
<?xml version="1 .0" encoding="UTF-8"?>
<alert start="201 1/04/14 14:39" end="2012/04/14 14:38" type="Elevated Threat" link="http://www.dhs.Qov
<summary><![CDATA[This is a summary of the alert]]></summary>
<details><![CDATA[<p>This is a more detailed description of the
alert</p>]]></details>
<locations>
<location><![CDATA[A location or region]]></location>
<location><![CDATA[Another location or region]]></location>
</locations>
<sectors>
<sector><![CDATA[An impacted sector]]></sector>
<sector><![CDATA[Another impacted sector]]></sector>
</sectors>
<duration><![CDATA[Freeform text description of the duration of the
alert]]></duration>
<detail_sections>
<detail_section> <detail_title><![CDATA[Freeform detail title]]></detail_title>
<detail><![CDATA[<p>Freeform text description of threat
details</p>]]></detail>
</detail_section>
<detail_section>
<detail_title><![CDATA[Freeform detail title]]></detail_title>
<detail><![CDATA[<p>Freeform text description of threat
details</p>]]></detail>
</detail_section>
<detail_section>
<detail_title><![CDATA[Freeform detail title]]></detail_title>
<detail><![CDATA[<p>Freeform text description of threat
details</p>]]></detail>
</detail_section>
<detail_section>
<detail_title><![CDATA[Freeform detail title]]></detail_title>
<detail><![CDATA[<p>Freeform text description of threat
details</p>]]></detail>
</detail_section>
</detail_sections>
</alert>
Table 1
[0185] The central server may be programmed to receive the xml above and parse it. Based on the location fields, the type of alert field and the duration, the server may be further programmed to divert members of a fleet of vehicle away from the alert area if they are scheduled to be in the alert area at the specified time. This would happen by broadcasting a request that all members of the fleet of interest to establish a link with the central server and receive further information about the threat level. This would save having to relay the entire message to all vehicles. In addition, the central server would know have a good indication of how many vehicles in its fleet would be impacted by the alert. Of course other scenarios could be treated in the same manner.
[0186] In an embodiment of the system and method, the prediction of required maintenance and the estimated cost of maintenance is transmitted to the vehicle when service or maintenance is needed. The transmission can occur to either the in-vehicle system or to a mobile device carried by a driver or passenger or directly to a service technician.
[0187] If the analysis is transmitted to the car, results can be displayed either graphically and/or in text on a screen in the vehicle - for example, an infotainment system screen.
6.3.3 Communication Protocols
[0188] For radio broadcast from the central server to client devices
communications could for example be using FM side-bands much like Traffic
Messaging channels operate. Communication could also be peer-to-peer and/or repeated. For example, the central server/s could broadcast a query or bulletin which is received by a vehicle. The vehicle could then rebroadcast it over a different frequency or using a different protocol - for example Bluetooth.
[0189] Communications between various sensors and a processor in a client can happen via the system bus in the client or via short range wireless or via fiber optics or wired connections. If the client device is an add-on product or consists of software running on a mobile device within a vehicle, then communication with the integral vehicle sensor may be by using an interface that can read on-board diagnostic (OBD II) codes by interfacing with a vehicle portal designed for external communications.
Another type of code that is somewhat standardized for vehicle diagnostics is the diagnostic trouble codes (DTC).
[0190] Many vehicles have Bluetooth or similar short range wireless protocol communication modules and can transmit information such as DTC codes to nearby devices. Longer range telematics devices that use, for example, mobile phone communication methods, also exist that can transmit DTC codes or similar code to a central location [0191] If the vehicle data collection module has software running on a general purpose computing device such as a mobile phone, the phone or other device could be plugged into the vehicle using a wired means such as a Universal Serial Bus (USB) or short range wireless such as Bluetooth. [0192] Sensor that are part of the mobile device can also be considered in- vehicle sensors provided the device is in or attached to the vehicle. These types of sensors can include gyroscopes, accelerometers, altimeters and GPS, for example. Communication with these sensors would be over the data bus of the portable device.
[0193] For client device that are in a stationary location, for example, road side sensor suites, an autobody repair facility, communication to a certain server or other nodes could happen over the internet and/or other form of wired and/or satellite communication.
[0194] In addition, a client device may communicate with a central server and/or a client device using a per-to-per network where a message is transmitted to one or more other client devices and then the message is repeated for other client that are in communication with the transmitting client device.
6.3.4 Database design and input normalization
[0195] In embodiments of this invention, vehicle maintenance and service requirements are predicted by comparing the observed conditions that occur during vehicle operation over time with similar observed conditions for similarly classed vehicle used in similar conditions stored in a historical vehicle maintenance database.
Algorithms are developed to classify each maintenance or service event as succinctly as possible, given the available data, such that when the conditions requiring maintenance or service for a vehicle in use match a classification, this can be used with a degree of certainty, to predict resulting maintenance required and the parts and services necessary to effect the maintenance.
[0196] In an embodiment, the observed conditions of interest during vehicle operation include:
Specific type of vehicle, including make, year, model, weight and options · Condition of the vehicle (prior damage, corrosion, state of repair) • Maintenance and Repair History
• Accident History
• Locale of vehicle operation (for determination of regional variable costs)
• Environmental factors (weather, road conditions) during operation
[0197] Raw data that may be used to predict maintenance and service needed can come from a plurality of sources. Sources include:
• ln-vehicle Sensors o Accelerometer to measure starting and stopping, rapid turning o ABS sensors to detect when slippery conditions occur o Gyroscope to erratic driving patterns o GPS for speed and direction of travel o Seatbelt sensors o Engine sensors such as oxygen, rpm, pollution control, temperature, etc
• External Sensors o Weather from web services o Traffic information from web services or FM sideband (Traffic Messaging Channel) o Road Condition Information from web-sources such as highway departments o Other means of collecting information o Fleet Maintenance Reports (subsequently manually entered into the system database by manual entry using a computer interface application o Repair Shop Invoices o GIS information (speed limits where traveled and roads traveled) o A listing of parts replaced o Price of labor (preferably broken out by part installed or service
performed) o Price of parts o Time between suggested maintenance and it actually occurring o Amount of lost time failures when maintenance suggestions are followed vs when maintenance lags
[0198] The database initially will have a mix of more qualitative data, for example from manually entered fleet maintenance records and repair shop invoices and quantitative data, for example, from in-vehicle sensors. As such there is a subjective element in the reporting and the likelihood of human error will reduce the quality of the manually entered data and therefore if the manually entered data makes up the bulk of the available information, the error in prediction of maintenance will be greater.
[0199] In addition, since much of qualitative information would have initially have been manually entered on a piece of paper, there will also be transcription errors regardless of whether the information is manually input into the database by a human or if the information is machine input using optical character recognition and algorithmic processing of the text.
[0200] Available information to input into the database will change with time. As more information of a quantitative nature or more precise, accurate and with less bias information becomes available, older more qualitative data will be replaced and the resulting predictive model or associated statistics will be updated to reflect the new data.
[0201] For information from disparate sources to be compared, the information must be normalized, i.e. converted to the same units of measure and be relative to the same reference frame. In addition, the quality and precision of the data must also be evaluated and represented within the database in a normalized fashion. In other words, if for example, one speed is known to be accurate within +/- 10 mph, then all speeds in the database should have an error of estimate in mph (as opposed to kph for example). 6.3.5 Components of the system
[0202] The following components are parts of embodiments of the system described in this application:
• One or more central servers that contain: o Computer memory loaded with:
■ a comprehensive database and patterns and indicators that correlate to events and situations o One or more processors containing:
■ Instructions for when and how to transmit requests for information and/or bulletins
■ Instructions for receiving and analyzing information from remote devices o Communications Devices
■ One-way transmitter of binary digital data containing:
• Coded requests for information including:
• Identification of the information requested
• Optional: Identification of the type of vehicle and/or
components required by for a respondent
• Optional: Geographic Area of Interest
• Optional: Time period of Interest o Radio Transceiver used to upload and download information to one or more specific client systems after two-way communication has been established by the client
• Vehicle Systems and / or Portable Devices o Radio Receiver for information from central servers via the one-way transmitter and/or other vehicles or systems including:
■ Query for information ■ Updates of patterns and indicators
■ Updates on codes that are used to identify fields in a query
■ Updates on sensor information and/or external feeds that are
available on the network ■ Updates on parts inventory that are part of each car o Radio Transceiver configured to establish two-way communications with one or more central servers and further used to upload and download information o Sensors (in-vehicle and external to vehicles) ■ See listing of sensors elsewhere in this document
6.3.6 Examples of Use
[0203] An example of how to use the above information is that the central server can query the satellite servers for regional information, when, for example, an insurance carrier wants to adjust rates base on region or a fleet management company wants to perform preventative maintenance on their fleet which is region dependent.
[0204] An operator of the system may desire to update the geography of a specific road segment. To do this a query may be sent to all vehicle, requesting a download of gps traces for vehicles that have traversed the segment within a specified time period. Vehicles that meet this requirement and that received the query then respond by sending the appropriate information. Once the information is received, then the GPS traces can be processed to revise the geometry or the road segment in the central database. Transportation network information comprises the physical location of roads, the road condition, traffic density throughout the day or week and typical weather conditions for a given time and relative to a road position and more. [0205] Another example of how the distributed database could be used would be, for example, in the current Volkswagen scandal. Suppose it is desired to measure fuel emissions from diesel engines while on the road, yet it is very difficult and expensive to do this directly. By using the relationship between the observed emissions values and the vehicle weight, slope angle the vehicle is driving on, country of origin of the vehicle (depending on country regulations, the vehicle is equipped with different engine features for example), octane level of the fuel, engine rpms, and tire type and inflation a pattern can be determined so that emissions can be measured indirectly. Then it can be determined how often vehicles on the road have emission values that exceed the emissions for the similarly equipped vehicle when run on a dynamometer.
[0206] The central processor/s could send out a query request to all vehicles in the network and request that all vehicles with a specific model and model year and that have the specific engine type of interest, record the above parameters over a period of time and transmit that information back to the central server. Alternatively, the vehicles in question will already be recording and storing this information, and can relay this information to the central server for a former time period once it is requested by the central server. Yet another possibility is for the central server to send the stored relationship (pattern or indicators) with the query, so that the individual vehicle systems can determine emission values based on the stored pattern and/or indicators and send only the computed emission values back to the central server.
[0207] Another example usage is in comparing vehicle wear as a function of region. Similarly, equipped vehicles will wear out faster or sustain differing levels of damage when involved in an accident depending on where the vehicle is driven. This type of information could be important for determining insurance rates. Corrosion due to salt being used as a de-icer for roads, corrodes vehicle parts significantly faster than when the salt is not applied. Likewise, in an area where there is significant rain, corrosion will be higher than in an arid region.
6.3.7 Other Use Scenarios
[0208] Determining if a detour and/or road construction is still in effect for a particular street segment.
• Request issued from Central Server/s: Rate of speed for vehicles traveling a specific road segment within the last day.
• Analysis: If no replies: road most likely closed:
If replies from client devices indicate vehicles are moving slow over the whole day, construction most likely active [0209] Determining why a particular vehicle part broke or wore out.
• Request for a response from any repair facility; record of repairs or replacement of the part of interest that has worn out; supply ail vehicle information, such as mileage on the part, location, vehicle type, vehicle model, subcomponents related to the part of interest.
« Analysis: Cross correlate subcomponents and mileage and vehicle type with the failure. Determine how to avoid the failure and/or determine what set of variable most likely predict the failure.
• Broadcast a repair bulletin for the identified vehicle characteristics of interest for when to perform scheduled maintenance based on when the part is anticipated to fail. Send a request to update the database in in-vehicle systems for this particular anticipated failure.
[0210] Update the geometry of a road
• Broadcast a request for GPS traces of a road from all vehicles driving the road.
• Analysis: Merge the traces from the responding vehicles into an average and compare with the existing segment in a navigation database. If the new segment is substantially different, replace the old one, or merge the new one with the old one.
• Broadcast that a new update in road geometry is available for download
[0211] Update traffic light timing
• Request: For all vehicles that are traveling within a defined geographic area, upload the gps trace and traveltime while in the defined area.
• Analysis: Determine how long each vehicle had to wait for a traffic signal (i.e. more than once) in all four directions of travel at times throughout the day.
Compare with other traffic lights that are in the area. Using traffic modeling determine if changing the timing of the traffic lights would minimize congestion and improve transit speed.
• Broadcast to the traffic lights in question that a new timing scenario is available for download. 6.3.8 Implementations
[0212] The present invention may be conveniently implemented using one or more conventional general purpose or specialized digital computers or microprocessors programmed according to the teachings of the present disclosure, or a portable device (e.g., a smartphone, tablet computer, computer or other device), equipped with including one or more sensors (e.g., accelerometers, GPS) or where the portable device are connected to the data collection devices that are remote to the portable device, or that are connected via wired or wireless means. Appropriate software coding can readily be prepared by skilled programmers based on the teachings of the present disclosure, as will be apparent to those skilled in the software art.
[0213] In some embodiments, the present invention includes a computer program product which is a non-transitory storage medium (media) having instructions stored thereon/in which can be used to program a computer to perform any of the processes of the present invention. The storage medium can include, but is not limited to, any type of disk including floppy disks, optical discs, DVD, CD-ROMs, micro drive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of media or device suitable for storing instructions and/or data.
6.4 Example Embodiments
[0214] In an embodiment, a vehicle accident surveillance network comprises at least one of: a) one or more surveillance systems which in turn comprises:
i) a sensor suite configured to observe ground based vehicles;
ii) a pattern recognition module configured to interpret the sensor suite readings as vehicle movements, locations, pending accidents, and accident incidents and to identify specific vehicles; and iii) a wireless transceiver configured to transmit and receive the identity and location of specific vehicles that had pattern identified, to surveillance systems;
b) one or more deployable aerial surveillance systems comprising:
i) an airframe configured to launch from one of a ground based vehicle, and fixed base station, and a larger airframe, wherein a launch is triggered by detection of a pattern indicative of an accident occurring or about to occur as detected by one or more of the surveillance systems;
ii) a second wireless transceiver configured to receive the identity and location of the vehicle or vehicles which correspond to identified patterns from one or more surveillance systems;
iii) a directional sensor suite configured to be directed towards the identified vehicle or vehicles after deployment of the aerial surveillance system;
iv) an aerial surveillance module configured to:
1 . receive the location of the vehicle or vehicles identified by one or more surveillance systems;
2. launch the one or more deployable aerial surveillance systems;
3. after being launched, determine the relative location of the identified vehicle or vehicles, and one or more of: approach the identified vehicle, circle the accident scene at a predetermined circumference and altitude, and point the directional sensors towards the vehicle or vehicles and record the sensor data.
[0215] In an embodiment, a deployable aerial surveillance system is configured with a receiver that can identify a location beacon (for example attached to a vehicle) and track the location beacon.
[0216] In an embodiment, deployable aerial surveillance systems are deployed by an operator when the system is provided with one or more of: a) coordinates of a vehicle to be surveyed;
b) a trajectory of a vehicle to be surveyed; and
c) specification of a location beacon that resides in the vehicle to be surveyed that can be tracked by the system. [0217] Vehicle accident surveillance network can be one or more of: a) an airborne surveillance system; b) a ground based vehicle equipped with a surveillance system; and c) a ground based stationary surveillance system. [0218] In an embodiment, airborne surveillance systems are configured: a) with image detection sensors that observe the earth below in a plurality of
spectral bands; and
b) the pattern detection module is configured to detect vehicles using image
analysis techniques. [0219] Surveillance systems, in an embodiment comprises a memory cache configured to store sensor data from sensors for a predetermined time prior to the present time and further configured to save this data upon detection of a pattern and continue to save incoming sensor data for a predetermined time after the pattern is detected. [0220] An embodiment of a vehicle accident surveillance system installed in a ground vehicle comprises: a) an on-vehicle sensor suite configured to observe location and motions of the ground vehicle;
b) a pattern recognition module configured to interpret the sensor suite readings as pending accidents, and accident incidents; and
c) a deployable aerial surveillance system comprising:
i) an airframe configured to launch from the vehicle when triggered by detection of a pattern indicative of an accident occurring or about to occur.; ii) a directional sensor suite configured to be directed towards the vehicle after deployment of the aerial surveillance system;
iii) an aerial surveillance module configured to:
(1 ) launch the deployable aerial surveillance system; (2) after being launched, determine the relative location of the vehicle, and one or more of: approach the vehicle, circle the vehicle at a predetermined altitude and circumference from the vehicle and point the directional sensors towards the vehicle and record the sensor data. [0221] A vehicle accident surveillance system installed in a ground vehicle can optionally be configure with a directional sensor suite that contains one or more cameras.
[0222] A transceiver in an accident surveillance system is optionally configured to: a) communicate with other accident surveillance systems; and
b) launch a deployable aerial surveillance system to observe the ground vehicle, the vehicle containing the requesting surveillance system or other ground vehicles.
[0223] In an embodiment, a vehicle accident surveillance system installed in a ground vehicle comprises: a) an on-vehicle sensor suite configured to observe location and motions of the ground vehicle;
b) a pattern recognition module configured to interpret the sensor suite readings as pending accidents, and accident incidents; and
c) a wireless transmitter configured to transmit a request to nearby surveillance systems to deploy and monitor the ground vehicle should the pattern recognition module detect a pattern indicative of a potential accident or accident.
[0224] In an embodiment a system to create, manage and utilize a vehicle diagnostic distributed database comprises:
a) at least one central server operable on one or more computers configured with at least: i) a radio transmitter configured to broadcast a query; ii) a first radio transceiver configured to perform two-way communications with the one or more client devices; and iii) a query engine configured to generate a query containing:
(1 ) a selection criteria for client devices of interest; and
(2) a request to at least one of upload information to the based station and download information from the based station; and b) the one or more client device configured with at least: i) a broadcast radio receiver configured to receive broadcasts from the at least one central server; ii) a second radio transceiver configured to perform two-way communications with the first radio transceiver in the at least one central server; and iii) a computer processor configured to receive a query from the broadcast radio transmitter and evaluate if the client device meets the selection criteria and if so, establish two-way communication, using the second radio transceiver, with the at least one central server, and perform the request. [0225] In an embodiment of a system to create, manage and utilize a vehicle diagnostic distributed database a query is a request to upload information that is specific for a particular vehicle type or component and related to known vehicle events or situations and comprises one or more of:
a) vehicle component replacement and maintenance records; b) vehicle sensor data referenced in space and time; and c) environmental data referenced in space and time.
[0226] In embodiments, a system to create, manage and utilize a vehicle diagnostic distributed database has at least one central server that is further configured to develop at least one of patterns and indices to predict vehicle events and identify situations based on information, from the one or more client devices, that was previously uploaded to the central server. [0227] In embodiments, a system to create, manage and utilize a vehicle diagnostic distributed database comprises query engine that generates a bulletin apprising clients of interest on the availability for download of, one or more of patterns and indices and instructions for the usage of the one or more patterns and indices and a radio transmitter broadcasts the query.
[0228] In embodiments of a system to create, manage and utilize a vehicle diagnostic distributed database, clients of interest establish communication with central servers using a radio transceiver and download patterns and indices to predict events and identify situations. [0229] In embodiments of a system to create, manage and utilize a vehicle diagnostic distributed database, a selection criteria comprising at least one of make, model, year of manufacturer and optional equipment of a vehicle is used.
[0230] In embodiments of a system to create, manage and utilize a vehicle diagnostic distributed database, a selection criteria comprises at least one of a geographic region, a climate zone, and within a political boundary.
[0231] In embodiments of a system to create, manage and utilize a vehicle diagnostic distributed database, a request can be made for GPS traces along roads a client vehicle has traversed.
[0232] In embodiments of a system to create, manage and utilize a vehicle diagnostic distributed database, a server can receive a requested GPS trace/s and is configured to update road geometry based on the GPS trace/s.
[0233] In embodiments of a system to create, manage and utilize a vehicle diagnostic distributed database, identified events and situations can be:
a) a vehicle accident occurring or having occurred; b) hazardous driving conditions; c) hazardous or illegal driving by a current driver of a vehicle; and d) required vehicle maintenance. [0234] In embodiments of a system to create, manage and utilize a vehicle diagnostic distributed database, client device can be a:
a) vehicle; b) satellite server or service; and c) mobile device.
[0235] In embodiments of a system to create, manage and utilize a vehicle diagnostic distributed database, an assessment for damage or repair costs is made for and identified event or situation.
[0236] In embodiments of a system to create, manage and utilize a vehicle diagnostic distributed database, a database comprises at least one:
a) vehicle driving situations; b) vehicle driving patterns and indicators; c) vehicle components and maintenance records; d) sensor data referenced in space and time; and e) remote environmental data referenced in space and time.
[0237] In embodiments of a communication system used to operate a distributed vehicle diagnostic database, the system comprises:
a) one or more computer based servers configured with at least: i) a broadcast radio transmitter; ii) a first radio transceiver; and b) one or more client devices configured with at least: i) a broadcast radio receiver; ii) a second radio transceiver; the one or more computer based servers transmits, using the broadcast radio
transmitter, a request to one or more of upload and download information and further containing selection criteria, and the one or more client devices: a) listen for and receive the transmission, utilizing the broadcast radio receiver; b) determines whether the selection criteria are met; and c) if the selection criteria are met, establish communication between the first and second radio transceivers and perform the request.
[0238] In an embodiment, a communication method is used to operate a distributed vehicle diagnostic database which comprises:
a) broadcasting a radio transmission, using a radio transmitter that is part of a
central server wherein the broadcast contains a query or bulletin including a client selection criteria, and b) listening for and receiving the broadcast, utilizing the broadcast radio receiver that is part of a client device and determining whether the client device meets the client selection criteria, and if the client device meets the client selection criteria, establish communication between a first radio transceiver that is part of the client device and second radio transceivers that is part of the one or more central servers, and moving information between the one or more computer servers and the client device that meets the client selection criteria.
[0239] In an embodiment, a communication method is used to operate a distributed vehicle diagnostic database and comprises:
a) driving situations; b) driving patterns and indicators; c) vehicle components and maintenance records; d) sensor data referenced in space and time; and e) remote environmental data referenced in space and time. [0240] In an embodiment, a communication method is used to operate a distributed vehicle diagnostic database with a client device that is one of a:
a) vehicle; b) satellite server or service; and c) mobile device.
[0241] In an embodiment, a communication method is used to operate a distributed vehicle diagnostic database with a client transceiver of the one or more client devices configured to also act as a repeater transmitting information to another client device which in turn can repeat the information and further transmit to yet another client or one or more of the computer based servers.
[0242] In an embodiment, a vehicle accident surveillance network comprises: a) one or more surveillance systems comprising:
i) a sensor suite configured to observe ground based vehicles;
ii) a pattern recognition module configured to interpret the sensor suite readings as vehicle movements, locations, pending accidents, and accident incidents and to identify specific vehicles; and
iii) a first radio transceiver;
iv) a broadcast radio receiver; and
v) a computer processor configured to receive a query from the broadcast radio receiver and evaluate it; and b) at least one central server operable on one or more computers configured with at least:
i) a radio transmitter configured to broadcast a query; ii) a second radio transceiver configured to perform two-way communications with the one or more surveillance systems; and iii) a query engine configured to generate a query wherein the one or more surveillance systems: a) detects patterns, b) identifies a vehicle or vehicles associated with the patterns; c) determines the location of the vehicle or vehicles; d) transmits this information using the first radio transmitter to the second radio transmitter of the at least one central server; and wherein the at least one central server:
a) receives the transmitted information;
b) configures a query requesting further information be gathered by surveillance systems that are in proximity to the vehicle or vehicles identified in the patterns; and
c) broadcasts, using the radio transmitter, the query; and
wherein surveillance systems in proximity to the vehicles or vehicles identified in the query respond by: a) recording information using the sensor suite about the vehicle or vehicles and in the vicinity of the vehicle or vehicles; and
b) transmit, using the first radio transmitter, the gathered information to the central server.
[0243] In embodiments of the vehicle accident surveillance network, the surveillance systems can be deployable aerial surveillance systems further comprising: a) an airframe configured to launch from one of a ground based vehicle, and fixed base station, and a larger airframe, wherein a launch is triggered by receiving a query from the at least one central server if the aerial surveillance system is in proximity to the vehicle or vehicles;
b) a directional sensor suite configured to be directed towards the identified vehicle or vehicles after deployment; and
c) an aerial surveillance module configured to launch the deployable aerial
surveillance system and after being launched, determine the relative location of the identified vehicle or vehicles, and one or more of: approach the identified vehicle or vehicles, circle the vehicle or vehicles at a predetermined circumference and altitude, and point the directional sensors towards the vehicle or vehicles and record the sensor data.

Claims

CLAIM 1. A vehicle accident surveillance network comprising at least one of:
a) one or more surveillance systems comprising:
i) a sensor suite configured to observe ground based vehicles;
ii) a pattern recognition module configured to interpret the sensor suite readings as vehicle movements, locations, pending accidents, and accident incidents and to identify specific vehicles; and
iii) a wireless transceiver configured to transmit and receive the identity and location of specific vehicles that had pattern identified, to surveillance systems;
b) one or more deployable aerial surveillance systems comprising:
i) an airframe configured to launch from one of a ground based vehicle, and fixed base station, and a larger airframe, wherein a launch is triggered by detection of a pattern indicative of an accident occurring or about to occur as detected by one or more of the surveillance systems;
ii) a second wireless transceiver configured to receive the identity and location of the vehicle or vehicles which correspond to identified patterns from one or more surveillance systems;
iii) a directional sensor suite configured to be directed towards the identified vehicle or vehicles after deployment of the aerial surveillance system;
iv) an aerial surveillance module configured to:
(1 ) receive the location of the vehicle or vehicles identified by one or more surveillance systems;
(2) launch the one or more deployable aerial surveillance systems;
(3) after being launched, determine the relative location of the identified
vehicle or vehicles, and one or more of: approach the identified vehicle, circle the accident scene at a predetermined circumference and altitude, and point the directional sensors towards the vehicle or vehicles and record the sensor data.
CLAIM 2. The vehicle accident surveillance network of Claim 1 wherein the one or more deployable aerial surveillance systems is further configured with a receiver that can identify a location beacon and track the location beacon.
CLAIM 3. The vehicle accident surveillance network of Claim 2 wherein the one or more deployable aerial surveillance systems are further configured to be deployed by an operator when the system is provided with one or more of:
a) coordinates of a vehicle to be surveyed;
b) a trajectory of a vehicle to be surveyed; and
c) specification of a location beacon that resides in the vehicle to be surveyed that can be tracked by the system.
CLAIM 4. The vehicle accident surveillance network of Claim 1 wherein the one or more surveillance systems are one or more of:
a) an airborne surveillance system;
b) a ground based vehicle equipped with a surveillance system; and
c) a ground based stationary surveillance system.
CLAIM 5. The vehicle accident surveillance network of Claim 2 wherein the one or more deployable airborne surveillance systems comprises one or more of an airframe, either manned or remotely operated or autonomous configured as one of:
a) a fixed wing aircraft;
b) a rotary aircraft with one or more rotors;
c) a lighter than air craft;
d) a satellite; and
e) a rocket propelled projectile.
CLAIM 6. The vehicle accident surveillance network of Claim 2 wherein the one or more airborne surveillance systems is configured:
a) with image detection sensors that observe the earth below in a plurality of
spectral bands; and
b) the pattern detection module is configured to detect vehicles using image
analysis techniques.
CLAIM 7. The vehicle accident surveillance network of Claim 1 wherein one or both of the first and second transceivers are configured to transmit sensor data and analysis to interested parties comprising one or more of: a) first responders;
b) insurance adjusters; and
c) vehicle owners.
CLAIM 8. The vehicle accident surveillance network of Claim 1 wherein the one or more surveillance systems further comprises a memory cache configured to store sensor data from sensors for a predetermined time prior to the present time and further configured to save this data upon detection of a pattern and continue to save incoming sensor data for a predetermined time after the pattern is detected.
CLAIM 9. The vehicle accident surveillance network of Claim 1 wherein one or more surveillance systems in the network are additionally configured to monitor driving conditions.
CLAIM 10. A vehicle accident surveillance system installed in a ground vehicle comprising:
a) an on-vehicle sensor suite configured to observe location and motions of the ground vehicle;
b) a pattern recognition module configured to interpret the sensor suite readings as pending accidents, and accident incidents; and
c) a deployable aerial surveillance system comprising:
i) an airframe configured to launch from the vehicle when triggered by detection of a pattern indicative of an accident occurring or about to occur.; ii) a directional sensor suite configured to be directed towards the vehicle after deployment of the aerial surveillance system;
iii) an aerial surveillance module configured to:
(1 ) launch the deployable aerial surveillance system;
(2) after being launched, determine the relative location of the vehicle, and one or more of: approach the vehicle, circle the vehicle at a predetermined altitude and circumference from the vehicle and point the directional sensors towards the vehicle and record the sensor data.
CLAIM 11. The vehicle accident surveillance system installed in a ground vehicle of Claim 10 wherein the directional sensor suite comprises one or more cameras.
CLAIM 12. The vehicle accident surveillance system installed in a ground vehicle of Claim 10 further configured with a transceiver configured to: a) communicate with other accident surveillance systems; and
b) launch the deployable aerial surveillance system to observe the ground vehicle, the vehicle containing the requesting surveillance system or other ground vehicles.
CLAIM 13. A vehicle accident surveillance system installed in a ground vehicle comprising:
a) an on-vehicle sensor suite configured to observe location and motions of the ground vehicle;
b) a pattern recognition module configured to interpret the sensor suite readings as pending accidents, and accident incidents; and
c) a wireless transmitter configured to transmit a request to nearby surveillance systems to deploy and monitor the ground vehicle should the pattern recognition module detect a pattern indicative of a potential accident or accident.
CLAIM 14. A system to create, manage and utilize a vehicle diagnostic distributed database comprising:
e) at least one central server operable on one or more computers configured with at least: i) a radio transmitter configured to broadcast a query; a first radio transceiver configured to perform two-way communications with the one or more client devices; and a query engine configured to generate a query containing:
(1 ) a selection criteria for client devices of interest; and
(2) a request to at least one of upload information to the based station and download information from the based station; and the one or more client device configured with at least: i) a broadcast radio receiver configured to receive broadcasts from the at least one central server; a second radio transceiver configured to perform two-way communications with the first radio transceiver in the at least one central server; and iii) a computer processor configured to receive a query from the broadcast radio transmitter and evaluate if the client device meets the selection criteria and if so, establish two-way communication, using the second radio transceiver, with the at least one central server, and perform the request.
CLAIM 15. The system of claim 14 wherein the query is a request to upload information that is specific for a particular vehicle type or component and related to known vehicle events or situations and comprises one or more of: a) vehicle component replacement and maintenance records; b) vehicle sensor data referenced in space and time; and c) environmental data referenced in space and time.
CLAIM 16. The system of claim 14 wherein the at least one central server is further configured to develop at least one of patterns and indices to predict vehicle events and identify situations based on information, from the one or more client devices, that was previously uploaded to the central server.
CLAIM 17. The system of claim 16 wherein the query engine generates a bulletin apprising clients of interest on the availability for download of, one or more of patterns and indices and instructions for the usage of the one or more patterns and indices and the radio transmitter broadcasts the query.
CLAIM 18. The system of claim 17 wherein the clients of interest establish
communication with the at least one central servers using the second radio transceiver and download the patterns and indices to predict events and identify situations.
CLAIM 19. The system of claim 14 wherein the selection criteria comprises at least one of make, model, year of manufacturer and optional equipment of a vehicle.
CLAIM 20. The system of claim 14 wherein the selection criteria comprises at least one of a geographic region, a climate zone, and within a political boundary.
CLAIM 21. The system of claim 14 wherein the request is for GPS traces along roads a client vehicle has traversed.
CLAIM 22. The system of claim 21 wherein the at least one server receives the requested GPS traces and is configured to update road geometry based on the GPS traces.
CLAIM 23. The system of claim 21 wherein the identified events and situations are one or more of: a) a vehicle accident occurring or having occurred; b) hazardous driving conditions; g) hazardous or illegal driving by a current driver of a vehicle; and h) required vehicle maintenance.
CLAIM 24. The system of claim 14 wherein the at least one client device is one or more of a: a) vehicle; b) satellite server or service; and c) mobile device.
CLAIM 25. The system of claim 23 wherein an assessment for damage or repair costs is made for the identified event or situation.
CLAIM 26. The system of claim 14 wherein the database comprises at least one: a) vehicle driving situations; b) vehicle driving patterns and indicators; c) vehicle components and maintenance records; d) sensor data referenced in space and time; and e) remote environmental data referenced in space and time.
CLAIM 27. A communication system used to operate a distributed vehicle diagnostic database comprising: a) one or more computer based servers configured with at least: iii) a broadcast radio transmitter; iv) a first radio transceiver; and c) one or more client devices configured with at least: i) a broadcast radio receiver; ii) a second radio transceiver; wherein the one or more computer based servers transmits, using the broadcast radio transmitter, a request to one or more of upload and download information and further containing selection criteria, and the one or more client devices: d) listen for and receive the transmission, utilizing the broadcast radio receiver; e) determines whether the selection criteria are met; and f) if the selection criteria are met, establish communication between the first and second radio transceivers and perform the request.
CLAIM 28. A communication method used to operate a distributed vehicle diagnostic database comprising: a) broadcasting a radio transmission, using a radio transmitter that is part of a
central server wherein the broadcast contains a query or bulletin including a client selection criteria, b) listening for and receiving the broadcast, utilizing the broadcast radio receiver that is part of a client device and determining whether the client device meets the client selection criteria, and if the client device meets the client selection criteria, establish communication between a first radio transceiver that is part of the client device and second radio transceivers that is part of the one or more central servers, and moving information between the one or more computer servers and the client device that meets the client selection criteria.
CLAIM 29. Then communication method of claim 28 wherein the information to upload or download comprises one or more of: a) driving situations; b) driving patterns and indicators; c) vehicle components and maintenance records; d) sensor data referenced in space and time; and e) remote environmental data referenced in space and time.
CLAIM 30. The communication method of claim 28 wherein the client device is one of a: d) vehicle; e) satellite server or service; and f) mobile device.
CLAIM 31. The communication method of claim 28 wherein the client transceiver of the one or more client devices is configured to also act as a repeater transmitting information to another client device which in turn can repeat the information and further transmit to yet another client or one or more of the computer based servers.
CLAIM 32. A vehicle accident surveillance network comprising:
a) one or more surveillance systems comprising:
i) a sensor suite configured to observe ground based vehicles; ii) a pattern recognition module configured to interpret the sensor suite readings as vehicle movements, locations, pending accidents, and accident incidents and to identify specific vehicles; and
iii) a first radio transceiver;
iv) a broadcast radio receiver; and
v) a computer processor configured to receive a query from the broadcast radio receiver and evaluate it; and b) at least one central server operable on one or more computers configured with at least:
i) a radio transmitter configured to broadcast a query; ii) a second radio transceiver configured to perform two-way communications with the one or more surveillance systems; and iii) a query engine configured to generate a query wherein the one or more surveillance systems: a) detects patterns, b) identifies a vehicle or vehicles associated with the patterns; c) determines the location of the vehicle or vehicles; d) transmits this information using the first radio transmitter to the second radio transmitter of the at least one central server; and wherein the at least one central server:
d) receives the transmitted information;
e) configures a query requesting further information be gathered by surveillance systems that are in proximity to the vehicle or vehicles identified in the patterns; and
f) broadcasts, using the radio transmitter, the query; and
wherein surveillance systems in proximity to the vehicles or vehicles identified in the query respond by: c) recording information using the sensor suite about the vehicle or vehicles and in the vicinity of the vehicle or vehicles; and
d) transmit, using the first radio transmitter, the gathered information to the central server.
CLAIM 33. The vehicle accident surveillance network of claim 32 wherein the one or more surveillance systems is a deployable aerial surveillance system further comprising:
a) an airframe configured to launch from one of a ground based vehicle, and fixed base station, and a larger airframe, wherein a launch is triggered by receiving a query from the at least one central server if the aerial surveillance system is in proximity to the vehicle or vehicles;
b) a directional sensor suite configured to be directed towards the identified vehicle or vehicles after deployment; and
c) an aerial surveillance module configured to launch the deployable aerial
surveillance system and after being launched, determine the relative location of the identified vehicle or vehicles, and one or more of: approach the identified vehicle or vehicles, circle the vehicle or vehicles at a predetermined
circumference and altitude, and point the directional sensors towards the vehicle or vehicles and record the sensor data.
PCT/US2016/015514 2015-01-29 2016-01-29 Remote accident monitoring and vehcile diagnostic distributed database WO2016123424A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP16744154.2A EP3251107A4 (en) 2015-01-29 2016-01-29 Remote accident monitoring and vehcile diagnostic distributed database

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US201562109434P 2015-01-29 2015-01-29
US62/109,434 2015-01-29
US201562265215P 2015-12-09 2015-12-09
US62/265,215 2015-12-09

Publications (1)

Publication Number Publication Date
WO2016123424A1 true WO2016123424A1 (en) 2016-08-04

Family

ID=56544366

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2016/015514 WO2016123424A1 (en) 2015-01-29 2016-01-29 Remote accident monitoring and vehcile diagnostic distributed database

Country Status (2)

Country Link
EP (1) EP3251107A4 (en)
WO (1) WO2016123424A1 (en)

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106846796A (en) * 2017-01-06 2017-06-13 四川克瑞斯航空科技有限公司 A kind of small-sized detecting devices of real-time detection road conditions
CN106920293A (en) * 2016-10-27 2017-07-04 蔚来汽车有限公司 The automatic log analysis methodology of car accident
WO2018153563A1 (en) * 2017-02-27 2018-08-30 Robert Bosch Gmbh Artificial neural network and unmanned aerial vehicle for recognizing a road accident
CN108604284A (en) * 2016-08-16 2018-09-28 华为技术有限公司 Scene of the accident restoring method, device and movement monitoring equipment
CN109360137A (en) * 2018-09-25 2019-02-19 平安科技（深圳）有限公司 A kind of car accident appraisal procedure, computer readable storage medium and server
CN110602664A (en) * 2018-06-12 2019-12-20 通用汽车有限责任公司 Method and system for distributed ledger technology communication for vehicles
US10657739B2 (en) 2016-10-05 2020-05-19 Solera Holdings, Inc. Vehicle tire monitoring systems and methods
CN111785011A (en) * 2019-04-04 2020-10-16 长沙智能驾驶研究院有限公司 Road vehicle monitoring and regulating method, device and system and computer equipment
DE102019210513A1 (en) * 2019-07-17 2021-01-21 Audi Ag Procedure for accident assistance, unmanned aircraft and motor vehicle
US10901423B2 (en) 2017-09-01 2021-01-26 International Business Machines Corporation Generating driving behavior models
AU2018394476B2 (en) * 2017-12-27 2021-05-13 Komatsu Ltd. Management system of work site and management method of work site
CN114724373A (en) * 2022-04-15 2022-07-08 地平线征程(杭州)人工智能科技有限公司 Traffic site information acquisition method and device, electronic device and storage medium
US20220292956A1 (en) * 2017-10-20 2022-09-15 Zendrive, Inc. Method and system for vehicular-related communications
US11450099B2 (en) 2020-04-14 2022-09-20 Toyota Motor North America, Inc. Video accident reporting
US11508189B2 (en) 2020-04-14 2022-11-22 Toyota Motor North America, Inc. Processing of accident report
US11615200B2 (en) 2020-04-14 2023-03-28 Toyota Motor North America, Inc. Providing video evidence
US11871313B2 (en) 2017-11-27 2024-01-09 Zendrive, Inc. System and method for vehicle sensing and analysis

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030095038A1 (en) * 2001-10-05 2003-05-22 Case Corporation Remote vehicle diagnostic system
US20060092890A1 (en) * 2004-11-01 2006-05-04 Gupta Vivek G Global network neighborhood: scheme for providing information about available networks in a geographical location
US20060092043A1 (en) * 2004-11-03 2006-05-04 Lagassey Paul J Advanced automobile accident detection, data recordation and reporting system
US20060241853A1 (en) * 2005-04-25 2006-10-26 The Boeing Company AGTM airborne surveillance
US7761544B2 (en) * 2002-03-07 2010-07-20 Nice Systems, Ltd. Method and apparatus for internal and external monitoring of a transportation vehicle
US20110046842A1 (en) * 2009-08-21 2011-02-24 Honeywell International Inc. Satellite enabled vehicle prognostic and diagnostic system
US20110147513A1 (en) * 2009-01-21 2011-06-23 John Steven Surmont Aerial payload deployment system
CN102436738A (en) * 2011-09-26 2012-05-02 同济大学 Traffic monitoring device based on unmanned aerial vehicle (UAV)
WO2014080388A2 (en) * 2014-03-25 2014-05-30 Alshdaifat, Wasfi Police drone
US20140279707A1 (en) * 2013-03-15 2014-09-18 CAA South Central Ontario System and method for vehicle data analysis
US8930044B1 (en) * 2012-12-28 2015-01-06 Google Inc. Multi-part navigation process by an unmanned aerial vehicle for navigating to a medical situatiion

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030095038A1 (en) * 2001-10-05 2003-05-22 Case Corporation Remote vehicle diagnostic system
US7761544B2 (en) * 2002-03-07 2010-07-20 Nice Systems, Ltd. Method and apparatus for internal and external monitoring of a transportation vehicle
US20060092890A1 (en) * 2004-11-01 2006-05-04 Gupta Vivek G Global network neighborhood: scheme for providing information about available networks in a geographical location
US20060092043A1 (en) * 2004-11-03 2006-05-04 Lagassey Paul J Advanced automobile accident detection, data recordation and reporting system
US20060241853A1 (en) * 2005-04-25 2006-10-26 The Boeing Company AGTM airborne surveillance
US20110147513A1 (en) * 2009-01-21 2011-06-23 John Steven Surmont Aerial payload deployment system
US20110046842A1 (en) * 2009-08-21 2011-02-24 Honeywell International Inc. Satellite enabled vehicle prognostic and diagnostic system
CN102436738A (en) * 2011-09-26 2012-05-02 同济大学 Traffic monitoring device based on unmanned aerial vehicle (UAV)
US8930044B1 (en) * 2012-12-28 2015-01-06 Google Inc. Multi-part navigation process by an unmanned aerial vehicle for navigating to a medical situatiion
US20140279707A1 (en) * 2013-03-15 2014-09-18 CAA South Central Ontario System and method for vehicle data analysis
WO2014080388A2 (en) * 2014-03-25 2014-05-30 Alshdaifat, Wasfi Police drone

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3251107A4 *

Cited By (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108604284A (en) * 2016-08-16 2018-09-28 华为技术有限公司 Scene of the accident restoring method, device and movement monitoring equipment
CN108604284B (en) * 2016-08-16 2021-02-12 华为技术有限公司 Accident site restoration method and device and motion monitoring equipment
US10657739B2 (en) 2016-10-05 2020-05-19 Solera Holdings, Inc. Vehicle tire monitoring systems and methods
CN106920293A (en) * 2016-10-27 2017-07-04 蔚来汽车有限公司 The automatic log analysis methodology of car accident
CN106846796A (en) * 2017-01-06 2017-06-13 四川克瑞斯航空科技有限公司 A kind of small-sized detecting devices of real-time detection road conditions
WO2018153563A1 (en) * 2017-02-27 2018-08-30 Robert Bosch Gmbh Artificial neural network and unmanned aerial vehicle for recognizing a road accident
US10901423B2 (en) 2017-09-01 2021-01-26 International Business Machines Corporation Generating driving behavior models
US20220292956A1 (en) * 2017-10-20 2022-09-15 Zendrive, Inc. Method and system for vehicular-related communications
US11871313B2 (en) 2017-11-27 2024-01-09 Zendrive, Inc. System and method for vehicle sensing and analysis
AU2018394476B2 (en) * 2017-12-27 2021-05-13 Komatsu Ltd. Management system of work site and management method of work site
US11535374B2 (en) 2017-12-27 2022-12-27 Komatsu Ltd. Management system of work site and management method of work site
CN110602664A (en) * 2018-06-12 2019-12-20 通用汽车有限责任公司 Method and system for distributed ledger technology communication for vehicles
CN109360137B (en) * 2018-09-25 2023-04-18 平安科技（深圳）有限公司 Vehicle accident assessment method, computer readable storage medium and server
CN109360137A (en) * 2018-09-25 2019-02-19 平安科技（深圳）有限公司 A kind of car accident appraisal procedure, computer readable storage medium and server
CN111785011A (en) * 2019-04-04 2020-10-16 长沙智能驾驶研究院有限公司 Road vehicle monitoring and regulating method, device and system and computer equipment
DE102019210513A1 (en) * 2019-07-17 2021-01-21 Audi Ag Procedure for accident assistance, unmanned aircraft and motor vehicle
US11450099B2 (en) 2020-04-14 2022-09-20 Toyota Motor North America, Inc. Video accident reporting
US11508189B2 (en) 2020-04-14 2022-11-22 Toyota Motor North America, Inc. Processing of accident report
US11615200B2 (en) 2020-04-14 2023-03-28 Toyota Motor North America, Inc. Providing video evidence
US11853358B2 (en) 2020-04-14 2023-12-26 Toyota Motor North America, Inc. Video accident reporting
US11954952B2 (en) 2020-04-14 2024-04-09 Toyota Motor North America, Inc. Processing of accident report
CN114724373A (en) * 2022-04-15 2022-07-08 地平线征程(杭州)人工智能科技有限公司 Traffic site information acquisition method and device, electronic device and storage medium

Also Published As

Publication number Publication date
EP3251107A4 (en) 2018-09-26
EP3251107A1 (en) 2017-12-06

Similar Documents

Publication Publication Date Title
US11370541B2 (en) Accident monitoring using remotely operated or autonomous aerial vehicles
WO2016123424A1 (en) Remote accident monitoring and vehcile diagnostic distributed database
CN110036423B (en) Method and control unit for adjusting the inter-vehicle distance between vehicles in a vehicle train
US10417897B1 (en) Systems and methods for reconstruction of a vehicular crash
US11816981B2 (en) Traffic monitoring and management systems and methods
US11294398B2 (en) Personal security robotic vehicle
US10445954B2 (en) Drive event capturing based on geolocation
US9875657B2 (en) Automated un-manned air traffic control system
CN109478063B (en) Automatic autonomous driving of a vehicle
EP3793227A1 (en) Systems and methods for detecting vehicle or environmental changes from data from automated vehicles
CN110582802A (en) Vehicle behavior monitoring system and method
EP3866138A1 (en) Systems and methods for automated cross-vehicle navigation using sensor data fusion
CN107406079A (en) For the system and method for the weather performance for predicting vehicle
US20210314753A1 (en) Systems and methods for automatic breakdown detection and roadside assistance
US20200209895A1 (en) System and method for supporting safe operation of an operating object
CA3159551C (en) Methods and systems for remote identification, messaging, and tolling of aerial vehicles
US11688278B1 (en) Traffic drone system
SE1650346A1 (en) Method and control unit for vehicle self-diagnosis
EP3438702B1 (en) Position reporting for vehicles
EP4063987A1 (en) Systems and methods for identifying landing zones for unmanned aircraft
US20220309934A1 (en) Systems and methods for detect and avoid system for beyond visual line of sight operations of urban air mobility in airspace
US20230252888A1 (en) Systems and Methods for Interactive Vehicle Transport Networks
WO2023154134A1 (en) Vehicle-originated wireless safety alert
EP4064245A1 (en) Systems and methods for detect and avoid system for beyond visual line of sight operations of urban air mobility in airspace
EP3819839A1 (en) Vehicle benchmarking method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16744154

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

REEP Request for entry into the european phase

Ref document number: 2016744154

Country of ref document: EP