US20210216795A1 - System of vehicles equipped with imaging equipment for high-definition near real-time map generation - Google Patents
System of vehicles equipped with imaging equipment for high-definition near real-time map generation Download PDFInfo
- Publication number
- US20210216795A1 US20210216795A1 US17/216,079 US202117216079A US2021216795A1 US 20210216795 A1 US20210216795 A1 US 20210216795A1 US 202117216079 A US202117216079 A US 202117216079A US 2021216795 A1 US2021216795 A1 US 2021216795A1
- Authority
- US
- United States
- Prior art keywords
- vehicle
- video
- data
- swarm
- capture
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000003384 imaging method Methods 0.000 title abstract description 73
- 230000001413 cellular effect Effects 0.000 claims description 16
- 238000012546 transfer Methods 0.000 claims description 8
- 230000010354 integration Effects 0.000 claims description 4
- 238000000034 method Methods 0.000 abstract description 48
- 238000012544 monitoring process Methods 0.000 abstract description 8
- 238000013481 data capture Methods 0.000 description 69
- 238000000605 extraction Methods 0.000 description 31
- 238000012545 processing Methods 0.000 description 30
- 230000008569 process Effects 0.000 description 22
- 238000004590 computer program Methods 0.000 description 14
- 238000010801 machine learning Methods 0.000 description 10
- 238000013507 mapping Methods 0.000 description 10
- 238000013480 data collection Methods 0.000 description 7
- 230000033001 locomotion Effects 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 230000003068 static effect Effects 0.000 description 5
- 241000255749 Coccinellidae Species 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 238000001514 detection method Methods 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 4
- 230000003442 weekly effect Effects 0.000 description 4
- 102100026278 Cysteine sulfinic acid decarboxylase Human genes 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 230000000295 complement effect Effects 0.000 description 3
- 238000010276 construction Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 108010064775 protein C activator peptide Proteins 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000004883 computer application Methods 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 230000005670 electromagnetic radiation Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- JMANVNJQNLATNU-UHFFFAOYSA-N oxalonitrile Chemical compound N#CC#N JMANVNJQNLATNU-UHFFFAOYSA-N 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 235000009434 Actinidia chinensis Nutrition 0.000 description 1
- 244000298697 Actinidia deliciosa Species 0.000 description 1
- 235000009436 Actinidia deliciosa Nutrition 0.000 description 1
- 240000001436 Antirrhinum majus Species 0.000 description 1
- 241000282376 Panthera tigris Species 0.000 description 1
- 241000238370 Sepia Species 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000004888 barrier function Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 239000000969 carrier Substances 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000010304 firing Methods 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000003116 impacting effect Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 230000005389 magnetism Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 238000002310 reflectometry Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000001931 thermography Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W60/00—Drive control systems specially adapted for autonomous road vehicles
- B60W60/001—Planning or execution of driving tasks
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
-
- G06K9/00805—
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
- B60R1/20—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R11/00—Arrangements for holding or mounting articles, not otherwise provided for
- B60R11/04—Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
- B60W40/02—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to ambient conditions
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
- B60W50/08—Interaction between the driver and the control system
- B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
- G01C21/28—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network with correlation of data from several navigational instruments
- G01C21/30—Map- or contour-matching
- G01C21/32—Structuring or formatting of map data
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/38—Electronic maps specially adapted for navigation; Updating thereof
- G01C21/3804—Creation or updating of map data
- G01C21/3833—Creation or updating of map data characterised by the source of data
- G01C21/3841—Data obtained from two or more sources, e.g. probe vehicles
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/38—Electronic maps specially adapted for navigation; Updating thereof
- G01C21/3885—Transmission of map data to client devices; Reception of map data by client devices
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/38—Electronic maps specially adapted for navigation; Updating thereof
- G01C21/3885—Transmission of map data to client devices; Reception of map data by client devices
- G01C21/3893—Transmission of map data from distributed sources, e.g. from roadside stations
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
-
- G06K9/00825—
-
- G06K9/22—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/17—Image acquisition using hand-held instruments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V20/584—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of vehicle lights or traffic lights
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/01—Detecting movement of traffic to be counted or controlled
- G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions
- G08G1/0108—Measuring and analyzing of parameters relative to traffic conditions based on the source of data
- G08G1/0112—Measuring and analyzing of parameters relative to traffic conditions based on the source of data from the vehicle, e.g. floating car data [FCD]
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/01—Detecting movement of traffic to be counted or controlled
- G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions
- G08G1/0125—Traffic data processing
- G08G1/0129—Traffic data processing for creating historical data or processing based on historical data
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/01—Detecting movement of traffic to be counted or controlled
- G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions
- G08G1/0125—Traffic data processing
- G08G1/0133—Traffic data processing for classifying traffic situation
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/01—Detecting movement of traffic to be counted or controlled
- G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions
- G08G1/0137—Measuring and analyzing of parameters relative to traffic conditions for specific applications
- G08G1/0141—Measuring and analyzing of parameters relative to traffic conditions for specific applications for traffic information dissemination
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/20—Monitoring the location of vehicles belonging to a group, e.g. fleet of vehicles, countable or determined number of vehicles
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/30—Services specially adapted for particular environments, situations or purposes
- H04W4/40—Services specially adapted for particular environments, situations or purposes for vehicles, e.g. vehicle-to-pedestrians [V2P]
- H04W4/44—Services specially adapted for particular environments, situations or purposes for vehicles, e.g. vehicle-to-pedestrians [V2P] for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H]
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/20—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of display used
- B60R2300/207—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of display used using multi-purpose displays, e.g. camera image and navigation or video on same display
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
- B60R2300/301—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
- B60R2300/302—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/40—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the details of the power supply or the coupling to vehicle components
- B60R2300/406—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the details of the power supply or the coupling to vehicle components using wireless transmission
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/50—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the display information being shared, e.g. external display, data transfer to other traffic participants or centralised traffic controller
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
- B60W2050/0001—Details of the control system
- B60W2050/0002—Automatic control, details of type of controller or control system architecture
- B60W2050/0004—In digital systems, e.g. discrete-time systems involving sampling
- B60W2050/0005—Processor details or data handling, e.g. memory registers or chip architecture
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
- B60W2050/0062—Adapting control system settings
- B60W2050/0075—Automatic parameter input, automatic initialising or calibrating means
- B60W2050/009—Priority selection
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
- B60W50/08—Interaction between the driver and the control system
- B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
- B60W2050/146—Display means
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2420/00—Indexing codes relating to the type of sensors based on the principle of their operation
- B60W2420/40—Photo, light or radio wave sensitive means, e.g. infrared sensors
- B60W2420/403—Image sensing, e.g. optical camera
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2420/00—Indexing codes relating to the type of sensors based on the principle of their operation
- B60W2420/40—Photo, light or radio wave sensitive means, e.g. infrared sensors
- B60W2420/408—Radar; Laser, e.g. lidar
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2554/00—Input parameters relating to objects
- B60W2554/40—Dynamic objects, e.g. animals, windblown objects
- B60W2554/402—Type
- B60W2554/4029—Pedestrians
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2555/00—Input parameters relating to exterior conditions, not covered by groups B60W2552/00, B60W2554/00
- B60W2555/60—Traffic rules, e.g. speed limits or right of way
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2556/00—Input parameters relating to data
- B60W2556/45—External transmission of data to or from the vehicle
- B60W2556/50—External transmission of data to or from the vehicle of positioning data, e.g. GPS [Global Positioning System] data
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/42—Simultaneous measurement of distance and other co-ordinates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30236—Traffic on road, railway or crossing
Definitions
- Street level intelligence has applications in a wide variety of fields including autonomous driving, architecture, construction, engineering, real estate, advertising, city planning, research and others.
- a fleet of vehicles equipped with imaging equipment can produce and/or supplement a map with high-definition and/or near-real time data to provide enhanced street level intelligence.
- the street level intelligence may comprise telematics systems on which commercial fleet managers may rely to keep track of vehicles and drivers when they are out in the field. Some fleets may incorporate cameras to provide liability protection from accidents, moving violations or parking violations.
- street level intelligence platforms comprising: at least one mapper vehicle, each mapper vehicle comprising an active data capture system comprising a location determination device, a LiDAR device, and at least one imaging device configured to actively capture data pertaining to an environment surrounding the mapper vehicle within a territory; a fleet of swarm vehicles, each swarm vehicle comprising a passive data capture system comprising a location determination device and a plurality of imaging devices configured to passively capture images of an environment surrounding the swarm vehicle within the territory; and a computing system comprising at least one processor and instructions that when executed by the at least one processor cause the at least one processor to create a street level intelligence application comprising: a data processing pipeline configured to: receive data from the active data capture system, combine data from the LiDAR, the imaging device, and the location determination device, perform feature extraction on the combined data from the active data capture system; receive data from the passive data capture systems, combine data from the imaging devices and the location determination devices, perform feature extraction on the combined data from the passive data capture system; and merge
- the at least one mapper vehicle comprises at least 2, at least 5, at least 10, at least 50, or at least 100 mapper vehicles.
- the active data capture system comprises at least one color panoramic camera.
- the at least one mapper vehicle refreshes data at least bi-yearly.
- the at least one mapper is a dedicated vehicle.
- the fleet of swarm vehicles comprises at least 10, at least 50, at least 100, at least 1000, or at least 5000 swarm vehicles.
- each swarm vehicle in the fleet of swarm vehicles refreshes data at least monthly.
- the fleet of swarm vehicles comprises a third-party fleet of vehicles to which passive data capture systems are affixed.
- the passive data capture systems are affixed magnetically. In some embodiments, wherein the passive data capture system comprises a plurality of externally-powered smartphones. In further embodiments, at least one smartphone faces outward from the front of the vehicle to capture video and at least one smartphone faces outward from each side of the vehicle to capture static images. In various embodiments, the territory has an average radius of less than 10 miles, less than 50 miles, or less than 100 miles. In some embodiments, the territory is a neighborhood, a city, a country, or a state.
- the merging comprises: performing feature extraction on mapper data to detect landmarks; performing feature extraction on swarm data to detect landmarks; comparing swarm landmarks within a predefine threshold of each mapper landmark to find the same landmarks; and updating the swarm location data when matching features are found.
- the street level intelligence interface comprises one or more of: a map overlaid with defined street segments for which pertinent data has been collected; tools allowing a user to view one or more of the street segments; tools allowing the user to select one or more of the street segments and add them to a cart; tools allowing the user to pay for access to the one or more street segments added to the cart; tools allowing the user to download a payload for street segments, for which access has been paid for, containing either 3D or 2D site intelligence or pedestrian analytics; tools allowing the user access payloads previously paid for; and tools allowing the user share accessed payloads with partners or other customers.
- the fleet manager interface comprises one or more of: tools allowing a user to see the current location of each fleet vehicle; tools allowing the user to download video offloaded from a fleet vehicle; tools allowing the user request priority offload of video from a fleet vehicle; and tools allowing the user view driver quality data.
- a digital processing device comprising: at least one processor, an operating system configured to perform executable instructions, a memory, and a computer program including instructions executable by the digital processing device to create a street level intelligence application comprising: a software module for receiving data from at least one active data capture system associated with a mapper vehicle, each active data capture system comprising a location determination device, a LiDAR device, and at least one imaging device configured to actively capture data pertaining to an environment surrounding the mapper vehicle within a territory; a software module for combining data from the LiDAR, the imaging device, and the location determination device; a software module for performing feature extraction on the combined data from the active data capture system; a software module for receiving data from a plurality of passive data capture systems associated with a fleet of swarm vehicles, each passive data capture system comprising a location determination device and a plurality of imaging devices configured to passively capture images of an environment surrounding a swarm vehicle within the territory; a software module for
- each active data capture system comprising a location determination device, a LiDAR device, and at least one imaging device configured to actively capture data pertaining to an environment surrounding the mapper vehicle within a territory; combining data from the LiDAR, the imaging device, and the location determination device; performing feature extraction on the combined data from the active data capture system; receiving data from a plurality of passive data capture systems associated with a fleet of swarm vehicles, each passive data capture system comprising a location determination device and a plurality of imaging devices configured to passively capture images of an environment surrounding a swarm vehicle within the territory; combining data from the imaging devices and the location determination devices; performing feature extraction on the combined data from the passive data capture system; merging the combined data from the active data capture system with the combined data from the passive data capture system based on the extracted features; providing a street level intelligence interface; and providing a fleet
- FIG. 1 shows an exemplary overview of a street level intelligence system including a Mapper system and a Swarm system in accordance with some embodiments of the present disclosure
- FIG. 2 shows a front left view of a Swarm capture device with a housing in accordance with some embodiments of the present disclosure
- FIG. 3 shows a front right view of a Swarm capture device with a housing in accordance with some embodiments of the present disclosure
- FIG. 4 shows a bottom view of a Swarm capture device with a housing in accordance with some embodiments of the present disclosure
- FIG. 5 shows a non-limiting example of a wiring diagram of Swarm capture devices deployed in a vehicle in accordance with some embodiments of the present disclosure
- FIG. 6 shows a non-limiting example of Swarm selective capture logic in accordance with some embodiments of the present disclosure
- FIG. 7 shows a non-limiting example of Swarm download logic in accordance with some embodiments of the present disclosure
- FIG. 8 is a flow chart of a method of processing video to identify high-priority events, according to an embodiment.
- FIG. 9 is a screenshot of a Real-Time Events Service UI, according to an embodiment.
- FIG. 10 shows a non-limiting example of a processing pipeline for a Swarm system, according to an embodiment
- FIG. 11 shows a non-limiting example of a wiring diagram of a Mapper system in accordance with some embodiments of the present disclosure
- FIG. 12 shows a non-limiting example of a processing pipeline for a mapper system, according to an embodiment
- FIG. 13 shows a non-limiting example of a Graphic User Interface (GUI) for a fleet manager in accordance with some embodiments of the present disclosure
- FIG. 14 shows a non-limiting example of a GUI for a street levelmobilence platform including a selection tool to examine pedestrian counts, according to an embodiment
- FIG. 15 shows an example of a pedestrian report for the area as selected according to the GUI of FIG. 14 ;
- FIG. 16 shows a non-limiting example of a GUI for a street level intelligence platform in accordance with some embodiments of the present disclosure.
- street level intelligence platforms comprising: at least one mapper vehicle, each mapper vehicle comprising an active data capture system comprising a location determination device, a LiDAR device, and at least one imaging device configured to actively capture data pertaining to an environment surrounding the mapper vehicle within a territory; a fleet of swarm vehicles, each swarm vehicle comprising a passive data capture system comprising a location determination device and a plurality of imaging devices configured to passively capture images of an environment surrounding the swarm vehicle within the territory; and a computing system comprising at least one processor and instructions that when executed by the at least one processor cause the at least one processor to create a street level intelligence application comprising: a data processing pipeline configured to: receive data from the active data capture system, combine data from the LiDAR, the imaging device, and the location determination device, perform feature extraction on the combined data from the active data capture system; receive data from the passive data capture systems, combine data from the imaging devices and the location determination devices, perform feature extraction on the combined data from the passive data capture
- a digital processing device comprising: at least one processor, an operating system configured to perform executable instructions, a memory, and a computer program including instructions executable by the digital processing device to create a street level intelligence application comprising: a software module for receiving data from at least one active data capture system associated with a mapper vehicle, each active data capture system comprising a location determination device, a LiDAR device, and at least one imaging device configured to actively capture data pertaining to an environment surrounding the mapper vehicle within a territory; a software module for combining data from the LiDAR, the imaging device, and the location determination device; a software module for performing feature extraction on the combined data from the active data capture system; a software module for receiving data from a plurality of passive data capture systems associated with a fleet of swarm vehicles, each passive data capture system comprising a location determination device and a plurality of imaging devices configured to passively capture images of an environment surrounding a swarm vehicle within the territory; a software module for receiving data from a plurality of passive data capture systems associated with a fleet
- Some embodiments described herein relate to a software-implemented method (e.g., a non-transitory processor readable medium storing code configured to be executed by a processor to perform a method) that includes capturing a video of a streetscape from a camera of a smartphone.
- a first pass of computer vision analysis can be performed on the video of the streetscape to identify candidate high-priority events.
- a second pass of computer vision analysis can be performed on candidate high-priority events to identify a high-priority event. In this way, a more detailed analysis of the video of the streetscape can be performed than would otherwise be possible with available computational resources.
- available resources may be unable to process the video of the streetscape in real time to identify high-priority events, but may be adequate to identify candidate high-priority events in real time and screen candidate high profile events to identify (actual) high profile events.
- An indication of high-profile events can be sent to a remote analysis service such that high-priority events can be integrated into a map of the streetscape.
- the remote analysis service may have greater computational resources than are available at a vehicle associated with the smartphone.
- each active data capture system comprising a location determination device, a LiDAR device, and at least one imaging device configured to actively capture data pertaining to an environment surrounding the mapper vehicle within a territory; combining data from the LiDAR, the imaging device, and the location determination device; performing feature extraction on the combined data from the active data capture system; receiving data from a plurality of passive data capture systems associated with a fleet of swarm vehicles, each passive data capture system comprising a location determination device and a plurality of imaging devices configured to passively capture images of an environment surrounding a swarm vehicle within the territory; combining data from the imaging devices and the location determination devices; performing feature extraction on the combined data from the passive data capture system; merging the combined data from the active data capture system with the combined data from the passive data capture system based on the extracted features; providing a street level intelligence interface; and providing
- Vehicle-mounted smartphones can continuously capture streetscape video.
- the vehicle-mounted smartphones may be unable to transfer raw streetscape video to the video-and-map integration device, for example, because average (e.g., daily) bandwidth for a cellular data network is less than what would be required to transfer video.
- the vehicle-mounted smartphones can therefore locally store streetscape video (e.g., using internal memory and/or peripheral local storage).
- vehicle-mounted smartphones can be operable to store at least eight hours of streetscape footage.
- Vehicle-mounted smartphones can also be operable to identify high-priority features in streetscape video and send a portion of the video containing the high-priority feature to the video-and-map-integration device, e.g., via the smartphone's cellular data radio and/or a cellular data network.
- the video-and-map integration device can be operable to integrate high-priority events into a map and send updated maps to the fleet of vehicle-mounted smartphones over the cellular data network. In this way, the fleet can be alerted to high-priority events in near real-time (e.g., within 3 hours).
- vehicle-mounted smartphones can connect to a WiFi network, which may have higher bandwidth than the cellular data network and/or at which the vehicles may spend a greater amount of time, and transfer locally stored streetscape video to the video-and-map-integration device via the vehicle-mounted smartphone's WiFi radios.
- the video-and-map-integration device can be operable to integrate video received via WiFi into the map so that the map is updated at least daily. Updated maps can be sent to map-viewer devices, vehicle-mounted smartphones, and/or navigation devices.
- Swarm refers to a passive capture system capable of passively capturing images and/or videos of a surrounding environment and depicts traffic impacting events in real-time.
- a Swarm system can be provided by a third-party fleet which may have multiple vehicles outfitted with Swarm systems for data collecting and processing.
- Intelligence products such as pedestrian foot traffic for each segment of the map, can be derived at the block or neighborhood level with a relatively short refresh rate, for example, nightly, weekly, or a monthly.
- Mapper refers to an active capture system capable of actively capturing images and videos of a surrounding environment and generating high-resolution graphical representations, such as a 2D or three-dimensional (3D) base map.
- the Mapper system can be arranged on a dedicated vehicle with multiple sensors for sensing the surrounding environment and collecting data. The data can be used for deriving the high-resolution 2D and 3D map at an address level with a relatively long refresh rate, for example, a bi-yearly refresh rate.
- the Mapper system and Swarm system as discussed above can be configured to operate in a combined manner, thereby dataset provided by the Mapper system being more accurate and dataset provided by the Swarm system being updated more regularly.
- the combination of these two types of datasets can be carried out via feature extraction and feature matching based on machine learning.
- a street level intelligence system 100 may comprise two sub-systems, i.e., a Mapper system and a Swarm system.
- the Swarm system as shown may comprise a Collect module 101 , a Fuse module 102 , and an Analyze module 103 .
- the Mapper system as shown may comprise a Collect module 104 , a Fuse module 105 , and an Analyze module 106 .
- a Merge module 107 and a Transform module 108 which may be shared by the Mapper system and Swarm system and are capable of merging the data from the analyze modules and transforming the merged data into a desired format for utilization.
- the Collect module 101 of the Swarm system may comprise one or more imaging devices for capturing still images or video.
- the imaging device can be configured to detect electromagnetic radiation (e.g., visible, infrared, and/or ultraviolet light, etc.) and generate image data based on the detected electromagnetic radiation.
- the imaging device may include a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) sensor that generates electrical signals in response to wavelengths of light.
- CMOS complementary metal-oxide-semiconductor
- the image data generated by the imaging device can include one or more images, which may be static images (e.g., photographs), dynamic images (e.g., video), or suitable combinations thereof.
- the image data can be polychromatic (e.g., RGB, CMYK, HSV, etc.) or monochromatic (e.g., grayscale, black-and-white, sepia, etc.).
- the imaging device may include a lens configured to direct light onto an image sensor.
- the imaging device can be embodied as a smartphone which may include one or more cameras capable of capturing still or dynamic image data (e.g., video).
- the camera can be a still camera that captures static images (e.g., photographs).
- the camera may capture both dynamic image and static images.
- the camera may switch between capturing dynamic images and static images.
- a camera can be used to generate 2D images of a 3D scene (e.g., an environment, one or more objects, etc.).
- the images generated by the camera can represent the projection of the 3D scene onto a 2D image plane. Accordingly, each point in the 2D image corresponds to a 3D spatial coordinate in the scene.
- the camera may comprise optical elements (e.g., lens, mirrors, filters, etc.).
- the camera may capture color images, greyscale image, infrared images, and the like.
- the camera may be a thermal imaging device when it is configured to capture infrared images.
- the imaging device e.g., a smartphone
- the imaging device may include moderate processing and/or memory (e.g., partially or completely disposed within the housing of the smartphone).
- the imaging device when in the field, may have a limited bandwidth, including periods of no connectivity to more robust processing capabilities, such as cloud computing resources.
- a cellular data network accessible to the imaging device may not provide sufficient bandwidth to transfer continuously collected video and/or rates charged by cellular data network carriers for transferring data may make transfer of continuously collected video cost-prohibitive.
- some embodiments relate to systems and methods for processing and analyzing data captured by the imaging device to provide better results than would otherwise be possible with available computational resources and/or for managing data transfers from the imaging device to remote storage and/or analysis facilities.
- the Collect module 101 of the Swarm system can be started automatically when the Swarm system boots.
- the collect module 101 can be configured to commence capturing the images or videos once the smartphone is disconnected from a Wi-Fi network, for example, when the smartphone is mounted on a vehicle of the fleet and the vehicle is moving out of the fleet garage and in the street.
- a Wi-Fi network for example, when the smartphone is mounted on a vehicle of the fleet and the vehicle is moving out of the fleet garage and in the street.
- motion blur may be encountered and should be taken into account through a Swarm selective capture logic as illustrated in FIG. 6 , as is discussed herein.
- the Collect module 101 may also comprise one or more sensors, for example, vision sensors, positioning sensors such as Global Position System (GPS) sensors, or inertial sensors such as accelerometers, gyroscopes, and/or gravity detection sensors.
- GPS Global Position System
- additional data such as time, frame numbers, locations, speeds, accelerations, light levels and counts nearby of Bluetooth/Wi-Fi devices can be captured or collected by one or more of these sensors and then can be fused together with the image or video data by the Fuse module 102 of the Swarm system.
- real-time or near real-time imagery with approximate or changing location can be achieved for Video-on-Demand (VOD) services, for example, requested by a fleet manager or a potential third-party for driver analysis.
- VOD Video-on-Demand
- the Collect module 101 of the Swarm system may additionally transmit location and status reports to a remote server on a regular basis, for example, via a Hyper Text Transfer Protocol (HTTPS) over a wireless communication system.
- HTTPS Hyper Text Transfer Protocol
- the remote server herein may be a cloud server at a remote site or a server located or owned by a third party.
- the Analyze module 103 of the Swarm system may analyze the captured imagery and apply processing algorithms to determine the pedestrian traffic, which will be discussed in detail later with reference to FIG. 10 .
- the Collect module 104 of the Mapper system may comprise hardware with high precision and definition.
- the Collect module 104 of the Mapper system may comprise an advanced GPS/Global Navigation Satellite System (GNSS), or Inertial Measurement Unit (IMU) sensors for location and orientation determination.
- the Collect module 104 may comprise one or more LiDAR sensors for forming point cloud data and a spherical digital video camera system, such as PointGray Ladybug 5, for filming or capturing panoramic images.
- GNSS GPS/Global Navigation Satellite System
- IMU Inertial Measurement Unit
- LiDAR sensors for forming point cloud data
- a spherical digital video camera system such as PointGray Ladybug 5, for filming or capturing panoramic images.
- the hardware mentioned herein is only for an illustrative purpose and an example of the Mapper system including suitable hardware is described with reference to FIG. 10 .
- the various types of data can be collected into files with corresponding formats.
- the files may include, but are not limited to, WMV files, ASF files, ASX files, RM files, RMVB files, MPG files, MPEG files, MPE files, 3GP files, MOV files, MP4 files, M4V files, AVI files, DAT files, MKV files, FLU files, VOB files, JPG files, TIFF files, RAM files, BMP files, GIF files, PNG files, PCX files, WMF files, PCX files, SWF files, GDF files, KIWI files, NavTech files, raw network PACP files, Ladybug PGR files, and GPS files.
- some of these files are fused with one another as appropriate by the Fuse module 105 of the Mapper system.
- the LiDAR data from PCAP files can be combined with GNSS data to create point cloud slices in CSV files and position/orientation information in POSE files.
- the Analyze module 106 can generate high definition or resolution map and site intelligence can be achieved by performing feature extraction and other necessary data processing. It should be understood that the descriptions of the operations of the Fuse module 105 and the Analyze module 106 are only illustrative of some aspects of the processing performed thereby and detailed discussions will be made later with reference to FIGS. 10 and 12 , which illustrate further details of the processing flow of the Mapper system and Swarm systems.
- the resultant data from the Mapper system and Swarm system can be merged and transformed by the Merge module 107 and Transform module 108 , respectively, in accordance with some embodiments of the present disclosure.
- one or more machine learning models can be established and trained to detect objects or features, such as windows, signage (e.g., road signs or advertising signs) and other landmarks. For example, when the features are related to the windows, feature extraction can be performed on the Mapper data to detect windows and likewise, feature extraction can be performed on the Swarm data to detect windows.
- the windows detected in the Mapper data can be compared to those detected in the Swarm data so as to determine whether they are the same. If this is the case, it means that the Swarm data is correct in this regard. However, if this is not the case, then the Swarm data with respect to the windows can be updated, for example, location data, thereby improving the accuracy of the Swarm data.
- the image can be localized (i.e., image localization) and therefore real-time high definition map can be obtained at the Transform module 108 .
- predictions regarding the pedestrian foot traffic can be changed and mapping can be prioritized at the Transform module 108 .
- the street level intelligence system in accordance with the embodiments of the present disclosure can be both high-resolution and kept current and insights can be derived from two combined datasets using machine learning technique. Further, with the aid of the machine learning based feature extraction and feature matching, the Swarm data can be made more accurate and the Mapper data can be updated more regularly.
- a Swarm capture device 200 may comprise a housing for accommodating an imaging device.
- the housing may have two parts, i.e., a first main component 201 and a second main component 202 , which can be designed as complementary for fastening and tightening the imaging device in a vehicle.
- the first main component and the second main component can be movably connected to one another with a connecting mechanism, such as a locking or clamping mechanism.
- the first main component and the second main component can be a single piece with a pull-out design such that the imaging device can be inserted therein when the two main components are pulled in different directions.
- the housing may further comprise two protrusion components 204 and 205 , which are complementary to one another and arranged respectively on the first main component and second main component.
- Both protrusion components can be attached to one another by, for example, a connecting component, such as a bolt, a belt, a screw, a nut or the like, such that the imaging device can be tightly clasped in the housing.
- the protrusion components herein can also be attached to other connecting mechanisms or components in the vehicle such that the imaging device can be fastened to the vehicle for image collection.
- the imaging device 203 as shown is embodied as a smartphone with a camera 206 for exemplary purposes; other imaging device, such as a camera, is also possible.
- the smartphone can be placed or inserted in the housing for image capture.
- the Swarm capture device may optionally have one or more power units, such as battery units for powering the smartphone, thereby extending the shooting time of the smartphone when the power level of the smartphone is lower than a predetermined threshold.
- the Swarm capture device may further comprise one or more slots or ports for connecting the smartphone to other devices, for example, other imaging devices arranged on the same vehicle for imaging capture, which is exemplarily shown in FIG. 5 .
- some functions of the smartphone can be disabled and thereby it is dedicated to capture the image and video surrounding the vehicle.
- the call function of the smartphone can be disabled.
- other functions such as receiving short messages, can also be disabled.
- the smartphone can be set into a flight mode in which no communication can be made and thus power of the smartphone can be further saved for environmental data collection.
- FIG. 3 a front right view of a Swarm capture device 300 is shown. It should be understood that the Swarm capture device 300 can be similar or equal to the Swarm capture device 200 as shown in FIG. 2 . Therefore, any descriptions of the Swarm capture device with respect to FIG. 2 can be equally applied to the Swarm capture device 300 herein.
- the Swarm capture device 300 has a housing comprising a first main component 301 and a second main component 302 which can accommodate an imaging device 303 inserted therein. Further, the housing may further comprise two protrusion components for fastening the Swarm capture device 300 in place when capturing the images.
- the imaging device 303 herein is depicted as a smartphone, which can be the same as shown in FIG. 2 . Similar to the descriptions made with respect to FIG. 2 , the smartphone herein can be configured to be exclusively used for image collection. Additionally or alternatively, the smartphone can also be connected with a Bluetooth connection; therefore, a call function can still be enabled even if the smartphone is operating to capture the images and videos.
- the smartphone as shown in FIGS. 2 and 3 can be configured to run on any suitable operation system (OS), for example, an Android OS or a Mac OS.
- OS operation system
- the smartphones in accordance with some embodiments of the present disclosure can be configured to run a customized version of a LinageOS (formerly Cyanogen) Android distribution. This customization may enable the smartphone of boot-on-power, remote access, and additional logging.
- a smartphone can be connected to vehicle power in such a manner that the smartphone can detect when the vehicle engine has been turned on and/or off.
- the smartphones can be configured to operate in a “headless” manner, for example, disabling call functionality, some and/or all input modes, configured to provide custom output, and/or so forth.
- the smartphones can be configured to power on when the vehicle is turned on, automatically enter a data capture mode (e.g., capturing video and/or pictures of a streetscape), disable call functionality, and/or power off or enter a power saving state when the vehicle is turned off.
- the smartphones can be configured to automatically capture data when certain conditions are met, such as when the smartphone detects it has left a home base or garage (e.g., via GPS and/or inertial sensors), when it is in an area that has not previously and/or recently been imaged (e.g., based on data received from a remote analysis, monitoring, and/or coordination device (e.g., a Real-Time Events Service, as described in further detail herein).
- a remote analysis, monitoring, and/or coordination device e.g., a Real-Time Events Service, as described in further detail herein.
- the smartphones can also be configured to automatically cease capturing data when certain such as when the smartphone detects it has entered a home base or garage. Furthermore, the smartphones can be configured to automatically send data to a remote analysis, monitoring, and/or coordination device and/or manage modes through with data is sent. For example, smartphones can be configured to use a relatively low-bandwidth and/or high cost network (e.g., cellular data networks, such as an LTE network) to send indications of high-priority events, while storing low-priority data (e.g., video data) locally until a relatively high-bandwidth and/or low cost network is available (such as WiFi or a wired local area network).
- a relatively low-bandwidth and/or high cost network e.g., cellular data networks, such as an LTE network
- low-priority data e.g., video data
- a relatively high-bandwidth and/or low cost network such as WiFi or a wired local area network
- the smartphones can be configured to connect to WiFi when they detect that they have entered a home base or garage and upload video data via the WiFi network.
- the smartphones can be configured to periodically contacts a remote analysis, monitoring, and/or coordination device for instructions, map updates, indications of high-priority events detected by other smartphones, and/or the like.
- the Swarm capture device 400 herein may be similar or equal to the one respectively shown in FIGS. 2 and 3 . Therefore, any descriptions of the Swarm capture device made before with respect to FIGS. 2 and 3 can be equally applied to the Swarm capture device 400 .
- the Swarm capture device 400 has a housing comprising a first main component 401 and a second main component 402 . Also shown at the bottom of the Swarm capture device are two mounting components, such as a first magnetic component 403 and a second magnetic component 404 .
- the first magnetic component 403 and second magnetic component 404 can provide sufficient magnetism to fix the Swarm capture device 400 to a metallic surface, for example, at the front sides, left and right sides of the vehicle. Therefore, it would be easy for the user to mount the Swarm capture device on the vehicle or remove the Swarm capture device from the vehicle.
- magnetic components herein are only for illustrative purposes and any other coupling mechanism, such as fasteners, actuation elements, joints, hinges, bolts, screws, etc. can also be used for movably coupling the Swarm capture device to the vehicle.
- the vehicle herein may be an autonomous vehicle or unmanned vehicle.
- a wiring diagram 500 of Swarm capture devices deployed in a vehicle 501 is shown.
- four Swarm capture devices 504 , 505 , 506 , and 507 are arranged on the vehicle 501 .
- the Swarm capture devices 505 and 507 are arranged at the top of the cab of the vehicle and the lens of the cameras are forward-facing.
- the Swarm capture devices 504 and 506 are arranged at the two sides of the cab and the lens of the cameras are sideways-facing. With the aid of this arrangement of the Swarm capture devices, the view around the front part of the vehicle can be captured.
- the Swarm capture devices can be configured to capture video or still images.
- two front-facing Swarm capture devices e.g., 505 and 507
- two side facing Swarm capture devices e.g., 504 and 506
- the arrangement of the Swarm capture devices herein is only for exemplary purposes and a person skilled in the art can envisage that the Swarm capture devices can be arranged at anywhere of the vehicle as appropriate.
- the Swarm capture devices can also be disposed at the rear of the vehicle, for example, near the back mirror.
- a microcontroller 502 can be optionally applied to control the Swarm capture devices, for example, for powering the Swarm captured devices with. e.g., 5V power, and for synchronizing them via Universal Serial Bus (USB) ports.
- a DC-DC converter may be wired into a 12V power system of the vehicle using a fuse tap that is powered on when the vehicle's motor is running and powered off when the vehicle's motor stops running.
- custom software can be used for allowing the Swarm capture devices to detect when the vehicle's motor is running, power up and/or begin capturing when the vehicle is turned on and/or power down (e.g., enter a standby or low energy state) when the vehicle's motor is turned off. While the vehicle's motor is running, the Swarm capture device can receive power from the DC-DC converter.
- a Swarm selective capture logic 600 is used for selectively taking and keeping an image, thereby ensuring the quality and effectiveness of the still images.
- the still capture logic can determine whether the present time is daylight. If the present time is not daylight but at night unsuitable for taking a picture, then the logic may decide not to take a picture at block 606 . However, if the present time is daylight suitable for taking a picture, then the flow proceeds to block 602 , at which it is determined whether the current location is near the garage where the vehicle is parked. If it is determined that the vehicle is near the garage, for example, the vehicle not moving out of the garage, then the logic can decide not to take a picture at block 606 .
- the logic can further determine whether the vehicle travels equal to or more than 3 meters. If this is the case, then the Swarm capture device may take a picture at block 605 . Otherwise, the flow proceeds to block 604 , at which the logic can determine whether the last picture was taken 10 second ago. If this is the case, then the Swarm capture device can take a picture at block 605 . If this is not the case, then the flow returns back to block 606 , at which it decides not to take a picture. This may be due to the fact that frequently taking still images is not desirable in some situations.
- the flow chart of the process for selectively taking a picture as discussed above is only for illustrative purposes and a person skilled in the art can understand that other judging conditions, in addition to time or location, can also be applied for deciding whether or not to take a still image.
- the judging condition may be a power level or a storage level of the Swarm capture device, a given time slot for taking pictures as specified by the user. Additionally, in some instances, once the Swarm capture device is disconnected from a Wi-Fi connection and its GPS reports it has moved sufficiently far from the fleet garage, then imaging capture will begin.
- a still picture is taken by the Swarm capture device and divided into nine grids or regions, wherein a middle grid and four corner grids are denoted by 3 , 1 , 2 , 4 , 5 , respectively.
- a process for deciding whether or not to keep the captured still picture may be performed as illustrated in the flow chart at a lower right side of FIG. 6 .
- the Swarm selective capture logic may calculate a Laplacian variance for each pixel in each of five regions 1 , 2 , 3 , 4 , and 5 .
- the logic may determine whether the Laplacian variances calculated for all five regions are less than a predetermined threshold. If this is the case, then at block 609 , the logic may decide to keep the picture. Otherwise, if the Laplacian variances calculated for all five regions are equal to or greater than the predetermined threshold, then the logic may decide to delete the captured picture at block 610 . In this way, motion blur that may occur in the captured image can also be eliminated or alleviated. In an instance where some regions have Laplacian variances greater than the predetermined threshold, while other regions have Laplacian variances below the predetermined threshold, the image may be kept.
- the image may be kept if the number of regions having Laplacian variances greater than the predetermined threshold is less than another predetermined threshold (e.g., if Laplacian variances for a majority of regions is less the predetermined threshold).
- another predetermined threshold e.g., if Laplacian variances for a majority of regions is less the predetermined threshold.
- the process for deciding whether or not to keep the picture as discussed above is only for illustrative purposes and a person skilled in the art can envisage other solutions or alternatives for deciding whether or not to keep the picture.
- the picture is divided into nine grids and five of them are selected, it can also be divided into more or less grids so as to meet the precision requirements.
- the Laplacian variance is used herein, other algorithms suitable for determining the quality of the image can also be applied as envisaged by those skilled in the art.
- watchdog software can be installed on a Swarm capture device to monitor and ensure that the image and video capture logic (or software) are working properly.
- the watchdog software may also be responsible for downloading and installing software or firmware updates.
- the Swarm capture device can be configured to collect and generate other types of data, including but not limited to time, frame numbers, locations, accelerometer states, light levels, and counts nearby of Bluetooth/Wi-Fi devices in a separate metadata file.
- synchronization pulse data can also be included in the metadata file.
- Swarm download logic is used for controlling video download, wherein a process 701 for downloading recent video is shown in the flow chart at the upper part of FIG. 7 and a process 708 for downloading historical video is shown in the flow chart at the lower part of FIG. 7 .
- a customer or a client may transmit to a fleet manager a request for video onboard the vehicle 713 which can be autonomous vehicle in some instances.
- the request can be transmitted via a smartphone of the user, for example, with the aid of a client application installed on the smartphone.
- the fleet manager may forward the request at step 703 to a Command and Control Center (CNC) where the requests from different fleet managers may be collected and processed.
- the CNC may transmit the request to the vehicle 713 via a wireless communication network, such as a Long Term Evolution (LTE) system.
- LTE Long Term Evolution
- the vehicle 713 can transmit its status to the CNC at step 704 or poll the CNC for potential requests.
- the vehicle 713 can transmit or upload the video collected during the travel via the LTE network to a cloud storage system such as the Amazon Simple Storage Service (S3), whereby the requested or demanded video can be transmitted to the customer.
- S3 Amazon Simple Storage Service
- the customer can download the video from the database S3 and view it locally on his or her smartphone.
- the above process can be performed when the vehicle is moving in the street or crossing blocks. Further, the steps as shown in the process 701 are only for exemplary purposes and a person skilled in the art can understand that some of the steps can be omitted or combined as appropriate.
- the CNC and the fleet manager may be co-located and therefore the request can be directly handled by the CNC without any involvement of the fleet manager.
- the CNC and database S3 may be also co-located and therefore the CNC may directly forward the video from the database S3 to the client without establishing a new connection between the database S3 and client.
- the Swarm capture devices can send location and status reports regularly via HTTPS over the LTE network to the database S3 for records and statistical analysis. Alternatively, the Swarm capture devices can also send the location and status reports regularly or in real-time to the fleet manager such that the fleet manager can monitor the vehicle at any given time.
- a Wi-Fi connection can be established with the Swarm capture devices and capturing operations performed by the Swarm capture devices may stop. Then, Swarm capture devices can offload all the image data to the database S3. Once image data has been completely offloaded and the vehicle has been turned off, the Swarm capture devices will shut down immediately. Alternatively, the offloading operations can be carried out at any suitable location upon request of the Swarm capture devices or a remote server, such as the remote server controlled by the fleet manager. Additionally, the Swarm capture devices can request offloading operations to the remote server and can begin offloading operations upon receipt of confirmation information from the remote server.
- the fleet manager can forward the request to the database S3 through the Wi-Fi network at step 712 .
- the database S3 may transmit the requested video to the user at step 711 .
- the user herein can be a person or a company, such as a third party (e.g., Zendrive), and therefore, the requested video including priority data can be used for driver analysis.
- the process 708 can be performed when the vehicle returns to the garage or is parked in the garage. Further, the steps as shown in the process 708 are only for exemplary purposes and a person skilled in the art can understand that some of the steps can be omitted or combined as appropriate.
- the database S3 and the fleet manager may be co-located and therefore the request can be directly handled by the database S3 without any involvement of the fleet manager. Additionally or alternatively, the fleet manager can communicate with the database S3 via a separate connection, for example, via a LTE connection.
- FIG. 8 is a flow chart of a method of processing video to identify high-priority events, according to an embodiment. At least portions of the method shown and described with reference to FIG. 8 can be performed by computing entities with relatively low computing power, such as a smartphone having a Qualcomm 820 processor. Thus, the method described with reference to FIG. 8 allows relatively modest computing resources to produce results that they would be unable to achieve using known techniques. For example, the method described with reference to FIG. 8 can identify high priority events from real-time video using a processor incapable of performing known machine vision techniques on video in real time.
- a Swarm capture device can be operable to continuously record video.
- the video may be of high quality (e.g., having a resolution of at least 1280 ⁇ 720 and a frame rate of at least 30 frames/second).
- captured video can be stored locally, for example on memory within a smartphone housing and/or on a local hard drive or flash memory device.
- locally stored video can be transferred to an analysis, monitoring, and/or coordination device (e.g., a Real-Time Events Service, a video-and-map-integration device, or other suitable device and/or service) via a WiFi radio/network, for example when the vehicle returns to a home base or garage.
- the analysis, monitoring, and/or coordination device can integrate video received from the Swarm capture devices via WiFi into the map such that the map is updated with video data received via WiFi, at least every 4 hours, every 12 hours, daily, weekly, monthly, or at any other suitable time interval.
- the analysis, monitoring, and/or coordination device can be operable send a signal representing a map updated with content received from the Swarm capture devices to a map-viewer device (e.g., a personal computer, smartphone, tablet, etc. running a web browser or map-viewer application), a Swarm capture device, and/or a navigation device.
- a map-viewer device e.g., a personal computer, smartphone, tablet, etc. running a web browser or map-viewer application
- the Swarm capture device can be communicatively coupled to one or more sensors, such as GPS, LiDAR, other image capture devices and/or so forth, and associate data received from such sensors with captured video.
- GPS coordinates, LiDAR point clouds, and/or so forth can be associated with captured video and/or appended as metadata to the video.
- the Swarm capture device can be operable to perform a first pass computer vision analysis of the video in real time.
- the first pass can identify candidate high-priority features. For example, a bandpass filtering technique can be performed on the video to identify predefined characteristic signals associated with high-priority features. For example, if a safety-orange or safety-yellow color is identified, these frames can be identified as potentially containing traffic cones or barrels. Similarly, if flashing blue or red lights are detected, the video can be identified as potentially containing emergency vehicles.
- the first pass can be performed in real-time, potentially at a lower frame rate than the video is captured (e.g., at 1 frame/second, 10 frames per second, 30 frames per second, etc.).
- a second pass can of machine vision/machine learning analysis can be performed on frames and/or video clips identified as candidate high-priority feature.
- the second pass can be more computationally intensive, and the processor of the Swarm capture device may have insufficient resources to perform the analysis performed during the second pass in real time on the continuously recorded video.
- Swarm capture device can perform higher-quality image recognition tasks than would otherwise be possible.
- the second pass can, for example, identify cones, jersey barriers, roadblocks and/or barrels to identify a construction area.
- the second pass can identify emergency light bars, police logos/markings, and police roadblocks.
- the second pass can be performed in near-real time (e.g., within five minutes of the video being captured).
- High-priority events identified during the second pass can be sent to a Real-Time Event Service for further processing and/or analysis.
- a portion of the video captured by the Swarm capture device can be uploaded with metadata (e.g., location data, time stamp, indications of high-priority features, etc.) to the Real-Time Event Service, for example over a cellular data network (e.g., LTE).
- the Real-Time Event Service may possess computational and/or human resources beyond those available in the Swarm vehicle.
- the Swarm capture device may be unable to send full raw video to the Real-Time Event Service.
- the cellular data network may have insufficient average daily bandwidth to allow the Swarm capture device to upload continuously captured video.
- the bandwidth at which video is captured by the Swarm capture device may, on average, exceed the bandwidth of a cellular data network available to the Swarm capture device.
- the Real-Time Event Service can perform additional machine vision/machine learning and/or human evaluation to determine the impact of the high-priority event. For example, the Real-Time Event Service can apply machine learning models to verify the detection of event objects identified in the second pass with additional accuracy.
- a notification can appear in a Real-Time Events Service UI, which can be used to determine and/or record impact of the high-priority event.
- FIG. 9 is a screenshot of a Real-Time Events Service UI, according to an embodiment.
- a GIS analyst using the Real-Time Events Service UI can watch video along with a synchronized map. The analyst can determine if the high-priority event impacts traffic.
- a polygon can be drawn (by the GIS analyst and/or machine learning processes) and the type of impact elected (e.g., from a drop-down menu).
- Swarm vehicles and/or subscribed parties can be notified of the possible event, confirmation status, and impacted area.
- Swarm capture devices can periodically poll a Real-Time Events API for updates on high-priority events (e.g., occurring within a predetermined range of the Swarm vehicle) and/or the Real-Time Event Service can push updates on high-priority events to Swarm capture devices.
- the Real-Time Events Service can be operable to determine when a high priority event occurred. For example, upon receiving an indication of a high-priority event from a Swarm device, the Real-Time Events Service determine if any other Swarm devices passed by the location of the high-priority event (e.g., within the last hour, last 6 hours, last 24 hours, last 48 hours, last week, etc.) and request video footage of the location of the high-priority event from any such Swarm devices. The Real-Time Event Service can analyze video received from such Swarm devices and determine a window in which the high-priority event appeared.
- Swarm capture devices passing an identified high-priority events can be configured to identify the absence of the event (e.g., using a process similar to the two pass analysis described above).
- the Real-Time Event Service can be configured to update the map to remove the high-priority event.
- a Mapper 800 in accordance with some embodiments of the present disclosure can comprise an antennas module 801 (e.g., Novatel 702GG Antenna), a GPS/GNSS module 802 (e.g., Novatel SPAN-IGM-S1 GNSS), a data collection module 803 , a LiDAR module 804 (e.g., Velodyne VLP-16 “Puck”), a LiDAR interface module 805 (e.g., Velodyne VLP-16 Interface), a spherical imaging module 806 (e.g., PointGrey Ladybug 5), a battery module 807 and an inverter 808 , which are connected with one another via a variety of communication interfaces or cables, such as a USB 2.0, a USB 3.0, or an Ethernet port, as shown.
- the Mapper 800 can be installed on a vehicle that is being driven around in some areas to collect high resolution images including pictures and videos.
- the antenna module 801 can be an L1/L2 GNSS antenna, offering combined GPS+GLONASS signal reception. In some cases, users can use the same antenna for GPS only or GPS+GLONASS applications to increase integration flexibility and reduce equipment costs.
- the antenna unit 801 can be connected to the GPS/GNSS module 802 via a coaxial cable and transmit the received signals thereto.
- the GPS/GNSS module 802 can comprise a receiver for receiving the location signals from the antenna unit 801 and compute position and velocity of the vehicle with high accuracy.
- the GPS/GNSS module 802 can also be embedded with an Inertial Measurement Unit (IMU) to further improve the accuracy of the measurements.
- IMU Inertial Measurement Unit
- the GPS/GNSS module may additionally comprise one or more accelerometers (e.g., three), one or more gyroscopes (e.g., three), thereby allowing the receiver to operate in environments where very high dynamic and frequent interruption of signals can be expected.
- accelerometers e.g., three
- gyroscopes e.g., three
- the LiDAR module 804 can be configured to perform laser scanning, for example, by using one or more (e.g., 16) laser/detector pairs mounted in a compact housing, which can rapidly spin to scan the surrounding environment.
- the LiDAR module 804 can be mounted on the rear of a vehicle with the cable pointing downward and tilted 90 degrees in order to scan the road.
- the lasers fire thousands of times per second, thereby providing a rich, 3D point cloud in real-time.
- the LiDAR module 804 can be configured to perform advanced digital signal processing and waveform analysis to provide high accuracy, extended distance sensing, and calibrated reflectivity data.
- the LiDAR module 804 is capable of horizontal Field of View (FOV) of 360°, adjustable rotational speed of 5-20 rotations per second, vertical FOV of 30°, and returns of up to 100 meters dependent on application.
- the LiDAR module 804 can also synchronize its data with precision, GPS-supplied time pulses, enabling the users to determine the exact firing time of each laser.
- the LiDAR module 804 can be connected to the LiDAR interface module 805 , which may serve as an interface box for the LiDAR module 804 .
- the LiDAR interface module 805 can receive a once-a-second synchronization pulse (PPS) issued by the GPS/GNSS module 802 and forward it to the LiDAR module 804 .
- PPS once-a-second synchronization pulse
- the LiDAR module 804 can set its time stamp to the number of microsecond past the hour per coordinated universal time (UTC) time. Thereby, it would be easy to geo-reference the LiDAR data into a point cloud.
- UTC coordinated universal time
- the spherical imaging module 806 can receive a trigger pulse from the GPS/GNSS module 802 and capture location-based visualizations for display in geographical mapping applications.
- the user can use the spherical vision camera to capture video in a mobile environment, and view the transmission of images as they are captured using the data collection module 803 , which can be embodied as a laptop or desktop computer.
- the spherical imaging module 806 is capable of 360° video streaming and covering 90% of the visual sphere. Further, the spherical imaging module 806 can be pre-calibrated and come with its own Software Development Kit (SDK), thereby allowing creation of dedicated applications for imaging capture.
- SDK Software Development Kit
- the GPS/GNSS module 802 , the LiDAR interface module 805 and the spherical imaging module 806 each can connect to the data collection module 803 for data exchange, storage, synchronization and control using various interfaces, such as USB or Ethernet interfaces.
- the data collection module 803 herein can be a laptop computer or a desktop computer with mass storage, which may serve as a database for storage of image data.
- the battery module 807 is provided as a power supply.
- the battery module 807 herein may be an auxiliary battery on the vehicle, which is able to provide a 12V direct current (DC).
- this 12V DC can be directly applied to the GPS/GNSS module 802 , the LiDAR interface module 805 and the spherical imaging module 806 .
- the inverter 808 can be introduced to convert 12V DC into 120V alternating current (AC).
- a wheel encoder can be used for converting the angular position or motion of a shaft or axle of the vehicle to an analog or digital code.
- the analog or digital code can be further processed into information such as speed, distance and position, thereby providing further details of the vehicle in motion.
- a processing pipeline 900 will start to process it. All images data is stored in file storage or database S3 and other kinds of data are stored in a database PG. All data is associated to edges roughly equal to street segments.
- the pipeline 900 of the Swarm system may perform the following steps at corresponding blocks while exchanging data with associated databases:
- Known methods of estimating pedestrian counts typically involve examining video of fixed cameras. When using fixed cameras, a fairly accurate count of pedestrians can be determined for particular locations, but little to no pedestrian information is available for areas for which camera coverage does not exist.
- mobile cameras such as Swarm capture devices described above, can increase the area for which pedestrian counts can be determined with a given number of cameras, but introduces several new challenges. In particular, mobile cameras coverage area and coverage gaps can change in unpredictable manners and video depicting a particular street segment may be received only on an irregular basis. The following method describes a technique(s) that can be used to extrapolate pedestrian counts based on inconsistent coverage:
- mapping inventory for continuously updated based on newly received information (e.g., from Swarm system elements and/or Mapper system elements). For example, permanent and/or temporary changes to roads and/or traffic conditions can be detected and integrated into available maps. In some instances mapping inventory can be tracked for particular locations, such as points of interest, intersections, addresses, and the like. In such an instance, video collected from in the vicinity of the particular location (e.g. an intersection) can be analyzed based, for example, on metadata associated with the video. Images likely to depict the particular location (e.g., images likely to be showing crossing though an intersection) can be analyzed. For each image, the set of mapping objects that is most commonly detected can be identified. For the winning set of objects, the object types and relative positions can be associated with the intersection.
- mapping inventory for continuously updated based on newly received information (e.g., from Swarm system elements and/or Mapper system elements). For example, permanent and/or temporary changes to roads and/or traffic conditions can be detected and integrated into available maps. In some instances mapping inventory can be tracked for
- Changes to the particular locations can be identified by identifying changes in the objects associated with the intersection. For example, during periodic mapping inventory updates (e.g., nightly, weekly, monthly, etc.) images of the particular location received during that period can be analyzed. If the set of images received during the period includes objects that match the set of objects associated with the intersection, the intersection can be considered unchanged. If, however, the set of images received during the period does not include objects associated with the intersection, for example, if objects received during the period includes objects that deviate from objects associated with the intersection by more than a threshold amount, then the images, video, and/or associated metadata can be passed to an Events Service UI, such as shown and discussed above with reference to FIGS. 8 and 9 .
- a GIS analyst and/or machine learning process can determine if the inventory has changed and/or update the mapping inventory (e.g., indicate that a lane is closed due to construction, changes in traffic patterns, lane markings, and so forth).
- a processing pipeline 906 will start to process it. All images data is stored in file storage or database S3 and other kinds of data are stored in a database PG.
- the data herein may include raw network PCAP files, PGR files, and GPS files. All data is associated to edges roughly equal to street segments.
- the pipeline 906 of the Mapper system may perform the following steps at corresponding blocks while exchanging data with associated databases:
- the feature extraction can be performed in different stages as follows:
- the Mapper system can also process the GPS data at block 910 , and perform Pedestrian detection and blur faces at blocks 912 and 913 using the same or similar algorithms as the Swarm system.
- the data from the Mapper and Swarm systems can be merged, thereby improving the efficacy of each system.
- the system can compare the features in Mapper data to features in Swarm data to improve the Swarm accuracy.
- the exemplary steps are as follows:
- a web application which can be used by a fleet manager and referred to as “fleet manager software,” is capable of controlling and monitoring the vehicles using the street level intelligence system.
- the application can be configured to allow the fleet manager to carry out multiple actions, including but not limited to:
- FIG. 13 there are six vehicles on the road and shown in real-time on the map.
- the “Vehicle” column lists these six vehicles with their respective name, traveled distances, and grades.
- the vehicle named “Grimlock” is shown grey in the list as it is offline and in maintenance.
- the vehicles herein may be outfitted with the Swarm systems as discussed above and elsewhere in the specification. It should be understood that the user interface shown in FIG. 12 can be a homepage of the fleet manager software.
- a web application which can be used by a customer and referred to as “street level intelligence software,” is capable of providing site intelligence payload to the customer.
- the web application herein can be configured to allow the customers to carry out the actions, including but not limited to:
- the site intelligence payload may comprise the following:
- the steps for stitching point cloud data are as follows:
- the pedestrian analytics payload may contain the following information:
- the customer can input areas or segments of interest in the Search field. Further, the customer can select the cart icon as shown and place the segments selected from the map into it. In some cases, the customer can also see the discounts shown in the cart. Finally, the customer can select coin icon as shown to pay the search fees using for example, credit cards.
- the street level intelligence software can provide items, such as Search miles, Search groups, or Search cost for customer's viewing. Additionally, the customer can order other data that is interested to him or her, such as the PNG map of the searched segments or CSV files.
- the platforms, systems, media, and methods disclosed herein include one or more non-transitory computer readable storage media encoded with a program including instructions executable by the operating system of an optionally networked digital processing device.
- a computer readable storage medium is a tangible component of a digital processing device.
- a computer readable storage medium is optionally removable from a digital processing device.
- a computer readable storage medium includes, by way of non-limiting examples, CD-ROMs, DVDs, flash memory devices, solid state memory, magnetic disk drives, magnetic tape drives, optical disk drives, cloud computing systems and services, and the like.
- the program and instructions are permanently, substantially permanently, semi-permanently, or non-transitorily encoded on the media.
- the platforms, systems, media, and methods disclosed herein include processor executable instructions, or use of the same.
- a computer program includes a sequence of instructions, executable in the digital processing device's CPU, written to perform a specified task.
- Computer readable instructions may be implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), data structures, and the like, that perform particular tasks or implement particular abstract data types.
- APIs Application Programming Interfaces
- a computer program may be written in various versions of various languages.
- a computer program comprises one sequence of instructions. In some embodiments, a computer program comprises a plurality of sequences of instructions. In some embodiments, a computer program is provided from one location. In other embodiments, a computer program is provided from a plurality of locations. In various embodiments, a computer program includes one or more software modules. In various embodiments, a computer program includes, in part or in whole, one or more web applications, one or more mobile applications, one or more standalone applications, one or more web browser plug-ins, extensions, add-ins, or add-ons, or combinations thereof.
- the platforms, systems, media, and methods disclosed herein include software, server, and/or database modules, or use of the same.
- software modules are created by techniques known to those of skill in the art using machines, software, and languages known to the art.
- the software modules disclosed herein are implemented in a multitude of ways.
- a software module comprises a file, a section of code, a programming object, a programming structure, or combinations thereof.
- a software module comprises a plurality of files, a plurality of sections of code, a plurality of programming objects, a plurality of programming structures, or combinations thereof.
- the one or more software modules comprise, by way of non-limiting examples, a web application, a mobile application, and a standalone application.
- software modules are in one computer program or application. In other embodiments, software modules are in more than one computer program or application. In some embodiments, software modules are hosted on one machine. In other embodiments, software modules are hosted on more than one machine. In further embodiments, software modules are hosted on cloud computing platforms. In some embodiments, software modules are hosted on one or more machines in one location. In other embodiments, software modules are hosted on one or more machines in more than one location.
- the platforms, systems, media, and methods disclosed herein include one or more databases, or use of the same.
- suitable databases include, by way of non-limiting examples, relational databases, non-relational databases, object oriented databases, object databases, entity-relationship model databases, associative databases, and XML databases. Further non-limiting examples include SQL, PostgreSQL, MySQL, Oracle, DB2, and Sybase.
- a database is internet-based.
- a database is web-based.
- a database is cloud computing-based.
- a database is based on one or more local computer storage devices.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Electromagnetism (AREA)
- Automation & Control Theory (AREA)
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- Mechanical Engineering (AREA)
- Transportation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Human Computer Interaction (AREA)
- Signal Processing (AREA)
- Mathematical Physics (AREA)
- Traffic Control Systems (AREA)
- Aviation & Aerospace Engineering (AREA)
- Image Analysis (AREA)
Abstract
Description
- This application is a divisional of U.S. patent application Ser. No. 15/995,041, filed May 31, 2018, which is a non-provisional of, and claims the benefit of priority of U.S. Provisional Patent Application No. 62/513,056, filed May 31, 2017, entitled “Near Real-Time Street Level Intelligence Platform,” the disclosure of each of which is hereby incorporated by reference in its entirety
- Street level intelligence has applications in a wide variety of fields including autonomous driving, architecture, construction, engineering, real estate, advertising, city planning, research and others. A fleet of vehicles equipped with imaging equipment can produce and/or supplement a map with high-definition and/or near-real time data to provide enhanced street level intelligence. In general, the street level intelligence may comprise telematics systems on which commercial fleet managers may rely to keep track of vehicles and drivers when they are out in the field. Some fleets may incorporate cameras to provide liability protection from accidents, moving violations or parking violations.
- Prior methods for capturing street level intelligence at city scale are either expensive or ineffective. Systems like Google Street View involve expensive equipment mounted on modified vehicles, which are driven for the express purpose of collecting data for creating maps and a visual database. Solutions that use volunteers to “crowd source” the data never reach useful scale. In both cases, the data collected becomes stale quickly.
- In one aspect, disclosed herein are street level intelligence platforms comprising: at least one mapper vehicle, each mapper vehicle comprising an active data capture system comprising a location determination device, a LiDAR device, and at least one imaging device configured to actively capture data pertaining to an environment surrounding the mapper vehicle within a territory; a fleet of swarm vehicles, each swarm vehicle comprising a passive data capture system comprising a location determination device and a plurality of imaging devices configured to passively capture images of an environment surrounding the swarm vehicle within the territory; and a computing system comprising at least one processor and instructions that when executed by the at least one processor cause the at least one processor to create a street level intelligence application comprising: a data processing pipeline configured to: receive data from the active data capture system, combine data from the LiDAR, the imaging device, and the location determination device, perform feature extraction on the combined data from the active data capture system; receive data from the passive data capture systems, combine data from the imaging devices and the location determination devices, perform feature extraction on the combined data from the passive data capture system; and merge the combined data from the active data capture system with the combined data from the passive data capture system based on the extracted features; a street level intelligence interface; and a fleet manager interface. In various embodiments, the at least one mapper vehicle comprises at least 2, at least 5, at least 10, at least 50, or at least 100 mapper vehicles. In some embodiments, the active data capture system comprises at least one color panoramic camera. In some embodiments, the at least one mapper vehicle refreshes data at least bi-yearly. In some embodiments, the at least one mapper is a dedicated vehicle. In various embodiments, the fleet of swarm vehicles comprises at least 10, at least 50, at least 100, at least 1000, or at least 5000 swarm vehicles. In some embodiments, each swarm vehicle in the fleet of swarm vehicles refreshes data at least monthly. In some embodiments, the fleet of swarm vehicles comprises a third-party fleet of vehicles to which passive data capture systems are affixed. In some embodiments, the passive data capture systems are affixed magnetically. In some embodiments, wherein the passive data capture system comprises a plurality of externally-powered smartphones. In further embodiments, at least one smartphone faces outward from the front of the vehicle to capture video and at least one smartphone faces outward from each side of the vehicle to capture static images. In various embodiments, the territory has an average radius of less than 10 miles, less than 50 miles, or less than 100 miles. In some embodiments, the territory is a neighborhood, a city, a country, or a state. In some embodiments, the merging comprises: performing feature extraction on mapper data to detect landmarks; performing feature extraction on swarm data to detect landmarks; comparing swarm landmarks within a predefine threshold of each mapper landmark to find the same landmarks; and updating the swarm location data when matching features are found. In some embodiments, the street level intelligence interface comprises one or more of: a map overlaid with defined street segments for which pertinent data has been collected; tools allowing a user to view one or more of the street segments; tools allowing the user to select one or more of the street segments and add them to a cart; tools allowing the user to pay for access to the one or more street segments added to the cart; tools allowing the user to download a payload for street segments, for which access has been paid for, containing either 3D or 2D site intelligence or pedestrian analytics; tools allowing the user access payloads previously paid for; and tools allowing the user share accessed payloads with partners or other customers. In some embodiments, the fleet manager interface comprises one or more of: tools allowing a user to see the current location of each fleet vehicle; tools allowing the user to download video offloaded from a fleet vehicle; tools allowing the user request priority offload of video from a fleet vehicle; and tools allowing the user view driver quality data.
- In another aspect, disclosed herein are computer-implemented systems comprising: a digital processing device comprising: at least one processor, an operating system configured to perform executable instructions, a memory, and a computer program including instructions executable by the digital processing device to create a street level intelligence application comprising: a software module for receiving data from at least one active data capture system associated with a mapper vehicle, each active data capture system comprising a location determination device, a LiDAR device, and at least one imaging device configured to actively capture data pertaining to an environment surrounding the mapper vehicle within a territory; a software module for combining data from the LiDAR, the imaging device, and the location determination device; a software module for performing feature extraction on the combined data from the active data capture system; a software module for receiving data from a plurality of passive data capture systems associated with a fleet of swarm vehicles, each passive data capture system comprising a location determination device and a plurality of imaging devices configured to passively capture images of an environment surrounding a swarm vehicle within the territory; a software module for combining data from the imaging devices and the location determination devices; a software module for performing feature extraction on the combined data from the passive data capture system; a software module for merging the combined data from the active data capture system with the combined data from the passive data capture system based on the extracted features; a software module for providing a street level intelligence interface; and a software module for providing a fleet manager interface.
- In yet another aspect, disclosed herein are non-transitory computer-readable storage media encoded with a computer program including instructions executable by at least one processor to create an application comprising: a software module for receiving data from at least one active data capture system associated with a mapper vehicle, each active data capture system comprising a location determination device, a LiDAR device, and at least one imaging device configured to actively capture data pertaining to an environment surrounding the mapper vehicle within a territory; a software module for combining data from the LiDAR, the imaging device, and the location determination device; a software module for performing feature extraction on the combined data from the active data capture system; a software module for receiving data from a plurality of passive data capture systems associated with a fleet of swarm vehicles, each passive data capture system comprising a location determination device and a plurality of imaging devices configured to passively capture images of an environment surrounding a swarm vehicle within the territory; a software module for combining data from the imaging devices and the location determination devices; a software module for performing feature extraction on the combined data from the passive data capture system; a software module for merging the combined data from the active data capture system with the combined data from the passive data capture system based on the extracted features; a software module for providing a street level intelligence interface; and a software module for providing a fleet manager interface.
- In yet another aspect, disclosed herein are computer-implemented methods of generating real-time street level intelligence comprising: receiving data from at least one active data capture system associated with a mapper vehicle, each active data capture system comprising a location determination device, a LiDAR device, and at least one imaging device configured to actively capture data pertaining to an environment surrounding the mapper vehicle within a territory; combining data from the LiDAR, the imaging device, and the location determination device; performing feature extraction on the combined data from the active data capture system; receiving data from a plurality of passive data capture systems associated with a fleet of swarm vehicles, each passive data capture system comprising a location determination device and a plurality of imaging devices configured to passively capture images of an environment surrounding a swarm vehicle within the territory; combining data from the imaging devices and the location determination devices; performing feature extraction on the combined data from the passive data capture system; merging the combined data from the active data capture system with the combined data from the passive data capture system based on the extracted features; providing a street level intelligence interface; and providing a fleet manager interface.
- An understanding of the features and advantages of the described subject matter will be obtained by reference to the following detailed description that sets forth illustrative embodiments and the accompanying drawings of which:
-
FIG. 1 shows an exemplary overview of a street level intelligence system including a Mapper system and a Swarm system in accordance with some embodiments of the present disclosure; -
FIG. 2 shows a front left view of a Swarm capture device with a housing in accordance with some embodiments of the present disclosure; -
FIG. 3 shows a front right view of a Swarm capture device with a housing in accordance with some embodiments of the present disclosure; -
FIG. 4 shows a bottom view of a Swarm capture device with a housing in accordance with some embodiments of the present disclosure; -
FIG. 5 shows a non-limiting example of a wiring diagram of Swarm capture devices deployed in a vehicle in accordance with some embodiments of the present disclosure; -
FIG. 6 shows a non-limiting example of Swarm selective capture logic in accordance with some embodiments of the present disclosure; -
FIG. 7 shows a non-limiting example of Swarm download logic in accordance with some embodiments of the present disclosure; -
FIG. 8 is a flow chart of a method of processing video to identify high-priority events, according to an embodiment. -
FIG. 9 is a screenshot of a Real-Time Events Service UI, according to an embodiment. -
FIG. 10 shows a non-limiting example of a processing pipeline for a Swarm system, according to an embodiment; -
FIG. 11 shows a non-limiting example of a wiring diagram of a Mapper system in accordance with some embodiments of the present disclosure; -
FIG. 12 shows a non-limiting example of a processing pipeline for a mapper system, according to an embodiment; -
FIG. 13 shows a non-limiting example of a Graphic User Interface (GUI) for a fleet manager in accordance with some embodiments of the present disclosure; -
FIG. 14 shows a non-limiting example of a GUI for a street level intelegence platform including a selection tool to examine pedestrian counts, according to an embodiment; -
FIG. 15 shows an example of a pedestrian report for the area as selected according to the GUI ofFIG. 14 ; and -
FIG. 16 shows a non-limiting example of a GUI for a street level intelligence platform in accordance with some embodiments of the present disclosure. - Described herein, in certain embodiments, are street level intelligence platforms comprising: at least one mapper vehicle, each mapper vehicle comprising an active data capture system comprising a location determination device, a LiDAR device, and at least one imaging device configured to actively capture data pertaining to an environment surrounding the mapper vehicle within a territory; a fleet of swarm vehicles, each swarm vehicle comprising a passive data capture system comprising a location determination device and a plurality of imaging devices configured to passively capture images of an environment surrounding the swarm vehicle within the territory; and a computing system comprising at least one processor and instructions that when executed by the at least one processor cause the at least one processor to create a street level intelligence application comprising: a data processing pipeline configured to: receive data from the active data capture system, combine data from the LiDAR, the imaging device, and the location determination device, perform feature extraction on the combined data from the active data capture system; receive data from the passive data capture systems, combine data from the imaging devices and the location determination devices, perform feature extraction on the combined data from the passive data capture system; and merge the combined data from the active data capture system with the combined data from the passive data capture system based on the extracted features; a street level intelligence interface; and a fleet manager interface.
- Also described herein, in certain embodiments, are computer-implemented systems comprising: a digital processing device comprising: at least one processor, an operating system configured to perform executable instructions, a memory, and a computer program including instructions executable by the digital processing device to create a street level intelligence application comprising: a software module for receiving data from at least one active data capture system associated with a mapper vehicle, each active data capture system comprising a location determination device, a LiDAR device, and at least one imaging device configured to actively capture data pertaining to an environment surrounding the mapper vehicle within a territory; a software module for combining data from the LiDAR, the imaging device, and the location determination device; a software module for performing feature extraction on the combined data from the active data capture system; a software module for receiving data from a plurality of passive data capture systems associated with a fleet of swarm vehicles, each passive data capture system comprising a location determination device and a plurality of imaging devices configured to passively capture images of an environment surrounding a swarm vehicle within the territory; a software module for combining data from the imaging devices and the location determination devices; a software module for performing feature extraction on the combined data from the passive data capture system; a software module for merging the combined data from the active data capture system with the combined data from the passive data capture system based on the extracted features; a software module for providing a street level intelligence interface; and a software module for providing a fleet manager interface.
- Some embodiments described herein relate to a software-implemented method (e.g., a non-transitory processor readable medium storing code configured to be executed by a processor to perform a method) that includes capturing a video of a streetscape from a camera of a smartphone. A first pass of computer vision analysis can be performed on the video of the streetscape to identify candidate high-priority events. A second pass of computer vision analysis can be performed on candidate high-priority events to identify a high-priority event. In this way, a more detailed analysis of the video of the streetscape can be performed than would otherwise be possible with available computational resources. For example, available resources may be unable to process the video of the streetscape in real time to identify high-priority events, but may be adequate to identify candidate high-priority events in real time and screen candidate high profile events to identify (actual) high profile events. An indication of high-profile events can be sent to a remote analysis service such that high-priority events can be integrated into a map of the streetscape. The remote analysis service may have greater computational resources than are available at a vehicle associated with the smartphone.
- Also described herein, in certain embodiments, are non-transitory computer-readable storage media encoded with a computer program including instructions executable by at least one processor to create an application comprising: a software module for receiving data from at least one active data capture system associated with a mapper vehicle, each active data capture system comprising a location determination device, a LiDAR device, and at least one imaging device configured to actively capture data pertaining to an environment surrounding the mapper vehicle within a territory; a software module for combining data from the LiDAR, the imaging device, and the location determination device; a software module for performing feature extraction on the combined data from the active data capture system; a software module for receiving data from a plurality of passive data capture systems associated with a fleet of swarm vehicles, each passive data capture system comprising a location determination device and a plurality of imaging devices configured to passively capture images of an environment surrounding a swarm vehicle within the territory; a software module for combining data from the imaging devices and the location determination devices; a software module for performing feature extraction on the combined data from the passive data capture system; a software module for merging the combined data from the active data capture system with the combined data from the passive data capture system based on the extracted features; a software module for providing a street level intelligence interface; and a software module for providing a fleet manager interface.
- Also described herein, in certain embodiments, are computer-implemented methods of generating real-time street level intelligence comprising: receiving data from at least one active data capture system associated with a mapper vehicle, each active data capture system comprising a location determination device, a LiDAR device, and at least one imaging device configured to actively capture data pertaining to an environment surrounding the mapper vehicle within a territory; combining data from the LiDAR, the imaging device, and the location determination device; performing feature extraction on the combined data from the active data capture system; receiving data from a plurality of passive data capture systems associated with a fleet of swarm vehicles, each passive data capture system comprising a location determination device and a plurality of imaging devices configured to passively capture images of an environment surrounding a swarm vehicle within the territory; combining data from the imaging devices and the location determination devices; performing feature extraction on the combined data from the passive data capture system; merging the combined data from the active data capture system with the combined data from the passive data capture system based on the extracted features; providing a street level intelligence interface; and providing a fleet manager interface.
- Some embodiments described herein relate to a system that includes multiple vehicles each equipped with a vehicle-mounted smartphone and a video-and-map-integration device. Vehicle-mounted smartphones can continuously capture streetscape video. The vehicle-mounted smartphones may be unable to transfer raw streetscape video to the video-and-map integration device, for example, because average (e.g., daily) bandwidth for a cellular data network is less than what would be required to transfer video. The vehicle-mounted smartphones can therefore locally store streetscape video (e.g., using internal memory and/or peripheral local storage). In some instances, vehicle-mounted smartphones can be operable to store at least eight hours of streetscape footage. Vehicle-mounted smartphones can also be operable to identify high-priority features in streetscape video and send a portion of the video containing the high-priority feature to the video-and-map-integration device, e.g., via the smartphone's cellular data radio and/or a cellular data network. The video-and-map integration device can be operable to integrate high-priority events into a map and send updated maps to the fleet of vehicle-mounted smartphones over the cellular data network. In this way, the fleet can be alerted to high-priority events in near real-time (e.g., within 3 hours). When vehicles returns to a home location (e.g., a garage), vehicle-mounted smartphones can connect to a WiFi network, which may have higher bandwidth than the cellular data network and/or at which the vehicles may spend a greater amount of time, and transfer locally stored streetscape video to the video-and-map-integration device via the vehicle-mounted smartphone's WiFi radios. The video-and-map-integration device can be operable to integrate video received via WiFi into the map so that the map is updated at least daily. Updated maps can be sent to map-viewer devices, vehicle-mounted smartphones, and/or navigation devices.
- Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. Any reference to “or” herein is intended to encompass “and/or” unless otherwise stated.
- As used herein, “Swarm” refers to a passive capture system capable of passively capturing images and/or videos of a surrounding environment and depicts traffic impacting events in real-time. In some instances, a Swarm system can be provided by a third-party fleet which may have multiple vehicles outfitted with Swarm systems for data collecting and processing. Intelligence products, such as pedestrian foot traffic for each segment of the map, can be derived at the block or neighborhood level with a relatively short refresh rate, for example, nightly, weekly, or a monthly.
- As used herein, “Mapper” refers to an active capture system capable of actively capturing images and videos of a surrounding environment and generating high-resolution graphical representations, such as a 2D or three-dimensional (3D) base map. In some instances, the Mapper system can be arranged on a dedicated vehicle with multiple sensors for sensing the surrounding environment and collecting data. The data can be used for deriving the high-resolution 2D and 3D map at an address level with a relatively long refresh rate, for example, a bi-yearly refresh rate.
- In accordance with some embodiments of the present disclosure, the Mapper system and Swarm system as discussed above can be configured to operate in a combined manner, thereby dataset provided by the Mapper system being more accurate and dataset provided by the Swarm system being updated more regularly. In some instances, the combination of these two types of datasets can be carried out via feature extraction and feature matching based on machine learning.
- Referring to
FIG. 1 , in a particular embodiment, a streetlevel intelligence system 100 may comprise two sub-systems, i.e., a Mapper system and a Swarm system. The Swarm system as shown may comprise aCollect module 101, aFuse module 102, and anAnalyze module 103. Similarly and correspondingly, the Mapper system as shown may comprise aCollect module 104, aFuse module 105, and anAnalyze module 106. Further illustrated are aMerge module 107 and aTransform module 108, which may be shared by the Mapper system and Swarm system and are capable of merging the data from the analyze modules and transforming the merged data into a desired format for utilization. - In some instances, the
Collect module 101 of the Swarm system may comprise one or more imaging devices for capturing still images or video. The imaging device can be configured to detect electromagnetic radiation (e.g., visible, infrared, and/or ultraviolet light, etc.) and generate image data based on the detected electromagnetic radiation. The imaging device may include a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) sensor that generates electrical signals in response to wavelengths of light. The resultant electrical signals can be processed to produce image data. The image data generated by the imaging device can include one or more images, which may be static images (e.g., photographs), dynamic images (e.g., video), or suitable combinations thereof. The image data can be polychromatic (e.g., RGB, CMYK, HSV, etc.) or monochromatic (e.g., grayscale, black-and-white, sepia, etc.). The imaging device may include a lens configured to direct light onto an image sensor. - In some embodiments, the imaging device can be embodied as a smartphone which may include one or more cameras capable of capturing still or dynamic image data (e.g., video). The camera can be a still camera that captures static images (e.g., photographs). The camera may capture both dynamic image and static images. The camera may switch between capturing dynamic images and static images. Although certain embodiments provided herein are described in the context of smartphones, it shall be understood that the present disclosure can be applied to any suitable imaging device, and any description herein relating to a cameras can also be applied to any suitable imaging device, and any description herein relating to cameras can also be applied to other types of imaging devices. A camera can be used to generate 2D images of a 3D scene (e.g., an environment, one or more objects, etc.). The images generated by the camera can represent the projection of the 3D scene onto a 2D image plane. Accordingly, each point in the 2D image corresponds to a 3D spatial coordinate in the scene. The camera may comprise optical elements (e.g., lens, mirrors, filters, etc.). The camera may capture color images, greyscale image, infrared images, and the like. The camera may be a thermal imaging device when it is configured to capture infrared images.
- In some embodiments, the imaging device (e.g., a smartphone) and/or other local (e.g., on-vehicle) computational resources may be limited. For example, the imaging device may include moderate processing and/or memory (e.g., partially or completely disposed within the housing of the smartphone). Furthermore, the imaging device, when in the field, may have a limited bandwidth, including periods of no connectivity to more robust processing capabilities, such as cloud computing resources. For example, a cellular data network accessible to the imaging device may not provide sufficient bandwidth to transfer continuously collected video and/or rates charged by cellular data network carriers for transferring data may make transfer of continuously collected video cost-prohibitive. Accordingly, as described in further detail here, some embodiments relate to systems and methods for processing and analyzing data captured by the imaging device to provide better results than would otherwise be possible with available computational resources and/or for managing data transfers from the imaging device to remote storage and/or analysis facilities.
- The
Collect module 101 of the Swarm system can be started automatically when the Swarm system boots. In some instances, when the Swarm system is installed on a smartphone which may be used as an imaging device for data collection, thecollect module 101 can be configured to commence capturing the images or videos once the smartphone is disconnected from a Wi-Fi network, for example, when the smartphone is mounted on a vehicle of the fleet and the vehicle is moving out of the fleet garage and in the street. During the capture processing, if high-resolution still images are captured, motion blur may be encountered and should be taken into account through a Swarm selective capture logic as illustrated inFIG. 6 , as is discussed herein. - In addition to using the imaging device to capture the images and videos, the
Collect module 101 may also comprise one or more sensors, for example, vision sensors, positioning sensors such as Global Position System (GPS) sensors, or inertial sensors such as accelerometers, gyroscopes, and/or gravity detection sensors. Thereby, additional data such as time, frame numbers, locations, speeds, accelerations, light levels and counts nearby of Bluetooth/Wi-Fi devices can be captured or collected by one or more of these sensors and then can be fused together with the image or video data by theFuse module 102 of the Swarm system. Upon fusing operations via theFuse module 102, real-time or near real-time imagery with approximate or changing location can be achieved for Video-on-Demand (VOD) services, for example, requested by a fleet manager or a potential third-party for driver analysis. - In some instances, the
Collect module 101 of the Swarm system may additionally transmit location and status reports to a remote server on a regular basis, for example, via a Hyper Text Transfer Protocol (HTTPS) over a wireless communication system. The remote server herein may be a cloud server at a remote site or a server located or owned by a third party. - In some instances, the
Analyze module 103 of the Swarm system may analyze the captured imagery and apply processing algorithms to determine the pedestrian traffic, which will be discussed in detail later with reference toFIG. 10 . - In some instances, for collecting and obtaining high-resolution images and videos, the
Collect module 104 of the Mapper system may comprise hardware with high precision and definition. For example, theCollect module 104 of the Mapper system may comprise an advanced GPS/Global Navigation Satellite System (GNSS), or Inertial Measurement Unit (IMU) sensors for location and orientation determination. Further, theCollect module 104 may comprise one or more LiDAR sensors for forming point cloud data and a spherical digital video camera system, such asPointGray Ladybug 5, for filming or capturing panoramic images. The hardware mentioned herein is only for an illustrative purpose and an example of the Mapper system including suitable hardware is described with reference toFIG. 10 . - The various types of data, such as high-resolution images and videos, time data, location data can be collected into files with corresponding formats. The files may include, but are not limited to, WMV files, ASF files, ASX files, RM files, RMVB files, MPG files, MPEG files, MPE files, 3GP files, MOV files, MP4 files, M4V files, AVI files, DAT files, MKV files, FLU files, VOB files, JPG files, TIFF files, RAM files, BMP files, GIF files, PNG files, PCX files, WMF files, PCX files, SWF files, GDF files, KIWI files, NavTech files, raw network PACP files, Ladybug PGR files, and GPS files. In some embodiments, some of these files are fused with one another as appropriate by the
Fuse module 105 of the Mapper system. For example, the LiDAR data from PCAP files can be combined with GNSS data to create point cloud slices in CSV files and position/orientation information in POSE files. By a further combination of SCAN, POSE, and rectified JPG files, registered and colorized point cloud slices can be created. After that, theAnalyze module 106 can generate high definition or resolution map and site intelligence can be achieved by performing feature extraction and other necessary data processing. It should be understood that the descriptions of the operations of theFuse module 105 and theAnalyze module 106 are only illustrative of some aspects of the processing performed thereby and detailed discussions will be made later with reference toFIGS. 10 and 12 , which illustrate further details of the processing flow of the Mapper system and Swarm systems. - As shown in
FIG. 1 , subsequent to analyzing the data respectively by theAnalyze module 103 andAnalyze module 106, the resultant data from the Mapper system and Swarm system can be merged and transformed by theMerge module 107 andTransform module 108, respectively, in accordance with some embodiments of the present disclosure. In some instances, with the aid of a deep learning technique, one or more machine learning models can be established and trained to detect objects or features, such as windows, signage (e.g., road signs or advertising signs) and other landmarks. For example, when the features are related to the windows, feature extraction can be performed on the Mapper data to detect windows and likewise, feature extraction can be performed on the Swarm data to detect windows. Then, the windows detected in the Mapper data can be compared to those detected in the Swarm data so as to determine whether they are the same. If this is the case, it means that the Swarm data is correct in this regard. However, if this is not the case, then the Swarm data with respect to the windows can be updated, for example, location data, thereby improving the accuracy of the Swarm data. Further, in some instances, with the aid of theMerge module 107, the image can be localized (i.e., image localization) and therefore real-time high definition map can be obtained at theTransform module 108. In addition, based on the merging operations at theMerge module 107, predictions regarding the pedestrian foot traffic can be changed and mapping can be prioritized at theTransform module 108. - The above has discussed modules or blocks of the street level intelligence system and main functionality thereof. It should be understood that the descriptions herein of these modules and their operations are only for illustrative purposes and a person skilled in the art, based on the teaching of the present disclosure, can add, remove, omit, or combine one or more of the modules without departing from the spirit and scope of the present disclosure. Further, by an efficient and effective combination of Mapper data from the Mapper system and Swarm data from the Swarm system, the street level intelligence system in accordance with the embodiments of the present disclosure can be both high-resolution and kept current and insights can be derived from two combined datasets using machine learning technique. Further, with the aid of the machine learning based feature extraction and feature matching, the Swarm data can be made more accurate and the Mapper data can be updated more regularly.
- Referring to
FIG. 2 , in a particular embodiment, a Swarm capture device 200 may comprise a housing for accommodating an imaging device. The housing may have two parts, i.e., a firstmain component 201 and a secondmain component 202, which can be designed as complementary for fastening and tightening the imaging device in a vehicle. In some instances, the first main component and the second main component can be movably connected to one another with a connecting mechanism, such as a locking or clamping mechanism. In some instances, the first main component and the second main component can be a single piece with a pull-out design such that the imaging device can be inserted therein when the two main components are pulled in different directions. The housing may further comprise twoprotrusion components - The
imaging device 203 as shown is embodied as a smartphone with acamera 206 for exemplary purposes; other imaging device, such as a camera, is also possible. The smartphone can be placed or inserted in the housing for image capture. In some instances, the Swarm capture device may optionally have one or more power units, such as battery units for powering the smartphone, thereby extending the shooting time of the smartphone when the power level of the smartphone is lower than a predetermined threshold. Further, the Swarm capture device may further comprise one or more slots or ports for connecting the smartphone to other devices, for example, other imaging devices arranged on the same vehicle for imaging capture, which is exemplarily shown inFIG. 5 . - In some embodiments, some functions of the smartphone can be disabled and thereby it is dedicated to capture the image and video surrounding the vehicle. For example, when placed in the housing and prepared for shooting, the call function of the smartphone can be disabled. In some instances, to save the power, other functions, such as receiving short messages, can also be disabled. Further, the smartphone can be set into a flight mode in which no communication can be made and thus power of the smartphone can be further saved for environmental data collection.
- Referring to
FIG. 3 , a front right view of a Swarm capture device 300 is shown. It should be understood that the Swarm capture device 300 can be similar or equal to the Swarm capture device 200 as shown inFIG. 2 . Therefore, any descriptions of the Swarm capture device with respect toFIG. 2 can be equally applied to the Swarm capture device 300 herein. - As shown in
FIG. 3 , the Swarm capture device 300 has a housing comprising a firstmain component 301 and a secondmain component 302 which can accommodate animaging device 303 inserted therein. Further, the housing may further comprise two protrusion components for fastening the Swarm capture device 300 in place when capturing the images. Theimaging device 303 herein is depicted as a smartphone, which can be the same as shown inFIG. 2 . Similar to the descriptions made with respect toFIG. 2 , the smartphone herein can be configured to be exclusively used for image collection. Additionally or alternatively, the smartphone can also be connected with a Bluetooth connection; therefore, a call function can still be enabled even if the smartphone is operating to capture the images and videos. - The smartphone as shown in
FIGS. 2 and 3 can be configured to run on any suitable operation system (OS), for example, an Android OS or a Mac OS. In some instances, the smartphones in accordance with some embodiments of the present disclosure can be configured to run a customized version of a LinageOS (formerly Cyanogen) Android distribution. This customization may enable the smartphone of boot-on-power, remote access, and additional logging. For example, as discussed in further detail herein, a smartphone can be connected to vehicle power in such a manner that the smartphone can detect when the vehicle engine has been turned on and/or off. The smartphones can be configured to operate in a “headless” manner, for example, disabling call functionality, some and/or all input modes, configured to provide custom output, and/or so forth. For example, the smartphones can be configured to power on when the vehicle is turned on, automatically enter a data capture mode (e.g., capturing video and/or pictures of a streetscape), disable call functionality, and/or power off or enter a power saving state when the vehicle is turned off. Moreover, the smartphones can be configured to automatically capture data when certain conditions are met, such as when the smartphone detects it has left a home base or garage (e.g., via GPS and/or inertial sensors), when it is in an area that has not previously and/or recently been imaged (e.g., based on data received from a remote analysis, monitoring, and/or coordination device (e.g., a Real-Time Events Service, as described in further detail herein). The smartphones can also be configured to automatically cease capturing data when certain such as when the smartphone detects it has entered a home base or garage. Furthermore, the smartphones can be configured to automatically send data to a remote analysis, monitoring, and/or coordination device and/or manage modes through with data is sent. For example, smartphones can be configured to use a relatively low-bandwidth and/or high cost network (e.g., cellular data networks, such as an LTE network) to send indications of high-priority events, while storing low-priority data (e.g., video data) locally until a relatively high-bandwidth and/or low cost network is available (such as WiFi or a wired local area network). Similarly stated, the smartphones can be configured to connect to WiFi when they detect that they have entered a home base or garage and upload video data via the WiFi network. In addition or alternatively, the smartphones can be configured to periodically contacts a remote analysis, monitoring, and/or coordination device for instructions, map updates, indications of high-priority events detected by other smartphones, and/or the like. - Referring to
FIG. 4 , a bottom view of a Swarm capture device 400 is shown. The Swarm capture device 400 herein may be similar or equal to the one respectively shown inFIGS. 2 and 3 . Therefore, any descriptions of the Swarm capture device made before with respect toFIGS. 2 and 3 can be equally applied to the Swarm capture device 400. The Swarm capture device 400 has a housing comprising a firstmain component 401 and a secondmain component 402. Also shown at the bottom of the Swarm capture device are two mounting components, such as a firstmagnetic component 403 and a secondmagnetic component 404. The firstmagnetic component 403 and secondmagnetic component 404 can provide sufficient magnetism to fix the Swarm capture device 400 to a metallic surface, for example, at the front sides, left and right sides of the vehicle. Therefore, it would be easy for the user to mount the Swarm capture device on the vehicle or remove the Swarm capture device from the vehicle. It should be understood that magnetic components herein are only for illustrative purposes and any other coupling mechanism, such as fasteners, actuation elements, joints, hinges, bolts, screws, etc. can also be used for movably coupling the Swarm capture device to the vehicle. The vehicle herein may be an autonomous vehicle or unmanned vehicle. - Referring to
FIG. 5 , a wiring diagram 500 of Swarm capture devices deployed in avehicle 501 is shown. In a particular embodiment, fourSwarm capture devices FIGS. 2-4 are arranged on thevehicle 501. As shown, theSwarm capture devices Swarm capture devices - It is to be understood that the arrangement of the Swarm capture devices herein is only for exemplary purposes and a person skilled in the art can envisage that the Swarm capture devices can be arranged at anywhere of the vehicle as appropriate. For example, the Swarm capture devices can also be disposed at the rear of the vehicle, for example, near the back mirror.
- For centralized control and data synchronization, a
microcontroller 502 can be optionally applied to control the Swarm capture devices, for example, for powering the Swarm captured devices with. e.g., 5V power, and for synchronizing them via Universal Serial Bus (USB) ports. To better power-up the Swarm capture devices, a DC-DC converter may be wired into a 12V power system of the vehicle using a fuse tap that is powered on when the vehicle's motor is running and powered off when the vehicle's motor stops running. In this case, custom software can be used for allowing the Swarm capture devices to detect when the vehicle's motor is running, power up and/or begin capturing when the vehicle is turned on and/or power down (e.g., enter a standby or low energy state) when the vehicle's motor is turned off. While the vehicle's motor is running, the Swarm capture device can receive power from the DC-DC converter. - Referring to
FIG. 6 , in a particular embodiment, a Swarm selective capture logic 600 is used for selectively taking and keeping an image, thereby ensuring the quality and effectiveness of the still images. - As shown in the flow chart of a process for taking a picture at the upper part of
FIG. 6 , atblock 601, the still capture logic can determine whether the present time is daylight. If the present time is not daylight but at night unsuitable for taking a picture, then the logic may decide not to take a picture atblock 606. However, if the present time is daylight suitable for taking a picture, then the flow proceeds to block 602, at which it is determined whether the current location is near the garage where the vehicle is parked. If it is determined that the vehicle is near the garage, for example, the vehicle not moving out of the garage, then the logic can decide not to take a picture atblock 606. If the current location is not near the garage, for example, the vehicle moving out of the garage, then atblock 603, the logic can further determine whether the vehicle travels equal to or more than 3 meters. If this is the case, then the Swarm capture device may take a picture atblock 605. Otherwise, the flow proceeds to block 604, at which the logic can determine whether the last picture was taken 10 second ago. If this is the case, then the Swarm capture device can take a picture atblock 605. If this is not the case, then the flow returns back to block 606, at which it decides not to take a picture. This may be due to the fact that frequently taking still images is not desirable in some situations. - It should be understood that the flow chart of the process for selectively taking a picture as discussed above is only for illustrative purposes and a person skilled in the art can understand that other judging conditions, in addition to time or location, can also be applied for deciding whether or not to take a still image. For example, the judging condition may be a power level or a storage level of the Swarm capture device, a given time slot for taking pictures as specified by the user. Additionally, in some instances, once the Swarm capture device is disconnected from a Wi-Fi connection and its GPS reports it has moved sufficiently far from the fleet garage, then imaging capture will begin.
- As shown at a lower left side of
FIG. 6 , a still picture is taken by the Swarm capture device and divided into nine grids or regions, wherein a middle grid and four corner grids are denoted by 3, 1, 2, 4, 5, respectively. Based on the still picture as pre-processed in this way, a process for deciding whether or not to keep the captured still picture may be performed as illustrated in the flow chart at a lower right side ofFIG. 6 . Atblock 607, the Swarm selective capture logic may calculate a Laplacian variance for each pixel in each of fiveregions block 608, the logic may determine whether the Laplacian variances calculated for all five regions are less than a predetermined threshold. If this is the case, then atblock 609, the logic may decide to keep the picture. Otherwise, if the Laplacian variances calculated for all five regions are equal to or greater than the predetermined threshold, then the logic may decide to delete the captured picture atblock 610. In this way, motion blur that may occur in the captured image can also be eliminated or alleviated. In an instance where some regions have Laplacian variances greater than the predetermined threshold, while other regions have Laplacian variances below the predetermined threshold, the image may be kept. Alternatively, the image may be kept if the number of regions having Laplacian variances greater than the predetermined threshold is less than another predetermined threshold (e.g., if Laplacian variances for a majority of regions is less the predetermined threshold). In this way, partially blurred images, such as may occur when imaging a streetscape having a fast-moving object (e.g., a car) may be retained. - It should be noted that the process for deciding whether or not to keep the picture as discussed above is only for illustrative purposes and a person skilled in the art can envisage other solutions or alternatives for deciding whether or not to keep the picture. For example, although the picture is divided into nine grids and five of them are selected, it can also be divided into more or less grids so as to meet the precision requirements. Further, although the Laplacian variance is used herein, other algorithms suitable for determining the quality of the image can also be applied as envisaged by those skilled in the art.
- In some instances, watchdog software can be installed on a Swarm capture device to monitor and ensure that the image and video capture logic (or software) are working properly. In addition, the watchdog software may also be responsible for downloading and installing software or firmware updates. In some instances, in addition to taking pictures and collecting image data, the Swarm capture device can be configured to collect and generate other types of data, including but not limited to time, frame numbers, locations, accelerometer states, light levels, and counts nearby of Bluetooth/Wi-Fi devices in a separate metadata file. In case the microcontroller is being used, synchronization pulse data can also be included in the metadata file.
- Referring to
FIG. 7 , in a particular embodiment, Swarm download logic is used for controlling video download, wherein aprocess 701 for downloading recent video is shown in the flow chart at the upper part ofFIG. 7 and aprocess 708 for downloading historical video is shown in the flow chart at the lower part ofFIG. 7 . - First referring to the
process 701, atstep 702, a customer or a client may transmit to a fleet manager a request for video onboard thevehicle 713 which can be autonomous vehicle in some instances. The request can be transmitted via a smartphone of the user, for example, with the aid of a client application installed on the smartphone. Upon receipt of the request by the fleet manager, for example, via a remote server, the fleet manager may forward the request atstep 703 to a Command and Control Center (CNC) where the requests from different fleet managers may be collected and processed. Then, the CNC may transmit the request to thevehicle 713 via a wireless communication network, such as a Long Term Evolution (LTE) system. Through the LTE network, thevehicle 713 can transmit its status to the CNC atstep 704 or poll the CNC for potential requests. In case thevehicle 713 receives the request atstep 705, it can transmit or upload the video collected during the travel via the LTE network to a cloud storage system such as the Amazon Simple Storage Service (S3), whereby the requested or demanded video can be transmitted to the customer. In other words, the customer can download the video from the database S3 and view it locally on his or her smartphone. - It can be understood the above process can be performed when the vehicle is moving in the street or crossing blocks. Further, the steps as shown in the
process 701 are only for exemplary purposes and a person skilled in the art can understand that some of the steps can be omitted or combined as appropriate. For example, in some instances, the CNC and the fleet manager may be co-located and therefore the request can be directly handled by the CNC without any involvement of the fleet manager. Additionally or alternatively, the CNC and database S3 may be also co-located and therefore the CNC may directly forward the video from the database S3 to the client without establishing a new connection between the database S3 and client. In some cases, the Swarm capture devices can send location and status reports regularly via HTTPS over the LTE network to the database S3 for records and statistical analysis. Alternatively, the Swarm capture devices can also send the location and status reports regularly or in real-time to the fleet manager such that the fleet manager can monitor the vehicle at any given time. - Now referring to the
process 708, atstep 702, when thevehicle 713 is driven into the garage, a Wi-Fi connection can be established with the Swarm capture devices and capturing operations performed by the Swarm capture devices may stop. Then, Swarm capture devices can offload all the image data to the database S3. Once image data has been completely offloaded and the vehicle has been turned off, the Swarm capture devices will shut down immediately. Alternatively, the offloading operations can be carried out at any suitable location upon request of the Swarm capture devices or a remote server, such as the remote server controlled by the fleet manager. Additionally, the Swarm capture devices can request offloading operations to the remote server and can begin offloading operations upon receipt of confirmation information from the remote server. - In some instances, when the user sends to the fleet manager a request for video at
step 710, the fleet manager can forward the request to the database S3 through the Wi-Fi network atstep 712. Upon receipt of the request from the fleet manager, the database S3 may transmit the requested video to the user atstep 711. The user herein can be a person or a company, such as a third party (e.g., Zendrive), and therefore, the requested video including priority data can be used for driver analysis. - It can be understood that the
process 708 can be performed when the vehicle returns to the garage or is parked in the garage. Further, the steps as shown in theprocess 708 are only for exemplary purposes and a person skilled in the art can understand that some of the steps can be omitted or combined as appropriate. For example, in some instances, the database S3 and the fleet manager may be co-located and therefore the request can be directly handled by the database S3 without any involvement of the fleet manager. Additionally or alternatively, the fleet manager can communicate with the database S3 via a separate connection, for example, via a LTE connection. -
FIG. 8 is a flow chart of a method of processing video to identify high-priority events, according to an embodiment. At least portions of the method shown and described with reference toFIG. 8 can be performed by computing entities with relatively low computing power, such as a smartphone having a Snapdragon 820 processor. Thus, the method described with reference toFIG. 8 allows relatively modest computing resources to produce results that they would be unable to achieve using known techniques. For example, the method described with reference toFIG. 8 can identify high priority events from real-time video using a processor incapable of performing known machine vision techniques on video in real time. - A Swarm capture device can be operable to continuously record video. The video may be of high quality (e.g., having a resolution of at least 1280×720 and a frame rate of at least 30 frames/second). As discussed in further detail herein, it may not be feasible to wireless transmit continuously captured video over cellular data networks. Therefore, captured video can be stored locally, for example on memory within a smartphone housing and/or on a local hard drive or flash memory device. In some embodiments, there can be sufficient local (e.g., on-vehicle) storage capacity to store at least 4 hours, at least 8 hours, at least 12 hours, or at least 24 hours of video. In such an instance, locally stored video can be transferred to an analysis, monitoring, and/or coordination device (e.g., a Real-Time Events Service, a video-and-map-integration device, or other suitable device and/or service) via a WiFi radio/network, for example when the vehicle returns to a home base or garage. The analysis, monitoring, and/or coordination device can integrate video received from the Swarm capture devices via WiFi into the map such that the map is updated with video data received via WiFi, at least every 4 hours, every 12 hours, daily, weekly, monthly, or at any other suitable time interval. Furthermore, the analysis, monitoring, and/or coordination device can be operable send a signal representing a map updated with content received from the Swarm capture devices to a map-viewer device (e.g., a personal computer, smartphone, tablet, etc. running a web browser or map-viewer application), a Swarm capture device, and/or a navigation device. In some embodiments, the Swarm capture device can be communicatively coupled to one or more sensors, such as GPS, LiDAR, other image capture devices and/or so forth, and associate data received from such sensors with captured video. For example, GPS coordinates, LiDAR point clouds, and/or so forth can be associated with captured video and/or appended as metadata to the video.
- The Swarm capture device can be operable to perform a first pass computer vision analysis of the video in real time. The first pass can identify candidate high-priority features. For example, a bandpass filtering technique can be performed on the video to identify predefined characteristic signals associated with high-priority features. For example, if a safety-orange or safety-yellow color is identified, these frames can be identified as potentially containing traffic cones or barrels. Similarly, if flashing blue or red lights are detected, the video can be identified as potentially containing emergency vehicles. The first pass can be performed in real-time, potentially at a lower frame rate than the video is captured (e.g., at 1 frame/second, 10 frames per second, 30 frames per second, etc.).
- A second pass can of machine vision/machine learning analysis can be performed on frames and/or video clips identified as candidate high-priority feature. The second pass can be more computationally intensive, and the processor of the Swarm capture device may have insufficient resources to perform the analysis performed during the second pass in real time on the continuously recorded video. By selectively performing the second pass on candidate high-priority features, Swarm capture device can perform higher-quality image recognition tasks than would otherwise be possible. The second pass can, for example, identify cones, jersey barriers, roadblocks and/or barrels to identify a construction area. In addition or alternatively, the second pass can identify emergency light bars, police logos/markings, and police roadblocks. The second pass can be performed in near-real time (e.g., within five minutes of the video being captured).
- High-priority events identified during the second pass can be sent to a Real-Time Event Service for further processing and/or analysis. For example, a portion of the video captured by the Swarm capture device can be uploaded with metadata (e.g., location data, time stamp, indications of high-priority features, etc.) to the Real-Time Event Service, for example over a cellular data network (e.g., LTE). In some instances, the Real-Time Event Service may possess computational and/or human resources beyond those available in the Swarm vehicle. Furthermore, the Swarm capture device may be unable to send full raw video to the Real-Time Event Service. For example, the cellular data network may have insufficient average daily bandwidth to allow the Swarm capture device to upload continuously captured video. Similarly stated, the bandwidth at which video is captured by the Swarm capture device may, on average, exceed the bandwidth of a cellular data network available to the Swarm capture device.
- The Real-Time Event Service can perform additional machine vision/machine learning and/or human evaluation to determine the impact of the high-priority event. For example, the Real-Time Event Service can apply machine learning models to verify the detection of event objects identified in the second pass with additional accuracy. A notification can appear in a Real-Time Events Service UI, which can be used to determine and/or record impact of the high-priority event.
FIG. 9 is a screenshot of a Real-Time Events Service UI, according to an embodiment. In some embodiments, a GIS analyst, using the Real-Time Events Service UI can watch video along with a synchronized map. The analyst can determine if the high-priority event impacts traffic. A polygon can be drawn (by the GIS analyst and/or machine learning processes) and the type of impact elected (e.g., from a drop-down menu). Swarm vehicles and/or subscribed parties can be notified of the possible event, confirmation status, and impacted area. Additionally or alternatively, Swarm capture devices can periodically poll a Real-Time Events API for updates on high-priority events (e.g., occurring within a predetermined range of the Swarm vehicle) and/or the Real-Time Event Service can push updates on high-priority events to Swarm capture devices. - The Real-Time Events Service can be operable to determine when a high priority event occurred. For example, upon receiving an indication of a high-priority event from a Swarm device, the Real-Time Events Service determine if any other Swarm devices passed by the location of the high-priority event (e.g., within the last hour, last 6 hours, last 24 hours, last 48 hours, last week, etc.) and request video footage of the location of the high-priority event from any such Swarm devices. The Real-Time Event Service can analyze video received from such Swarm devices and determine a window in which the high-priority event appeared.
- Swarm capture devices passing an identified high-priority events can be configured to identify the absence of the event (e.g., using a process similar to the two pass analysis described above). In the event is not detected by a subsequent Swarm capture device, the Real-Time Event Service can be configured to update the map to remove the high-priority event.
- Referring to
FIG. 11 , in a particular embodiment, aMapper 800 in accordance with some embodiments of the present disclosure can comprise an antennas module 801 (e.g., Novatel 702GG Antenna), a GPS/GNSS module 802 (e.g., Novatel SPAN-IGM-S1 GNSS), adata collection module 803, a LiDAR module 804 (e.g., Velodyne VLP-16 “Puck”), a LiDAR interface module 805 (e.g., Velodyne VLP-16 Interface), a spherical imaging module 806 (e.g., PointGrey Ladybug 5), abattery module 807 and an inverter 808, which are connected with one another via a variety of communication interfaces or cables, such as a USB 2.0, a USB 3.0, or an Ethernet port, as shown. TheMapper 800 can be installed on a vehicle that is being driven around in some areas to collect high resolution images including pictures and videos. - In some instances, the
antenna module 801 can be an L1/L2 GNSS antenna, offering combined GPS+GLONASS signal reception. In some cases, users can use the same antenna for GPS only or GPS+GLONASS applications to increase integration flexibility and reduce equipment costs. Theantenna unit 801 can be connected to the GPS/GNSS module 802 via a coaxial cable and transmit the received signals thereto. The GPS/GNSS module 802 can comprise a receiver for receiving the location signals from theantenna unit 801 and compute position and velocity of the vehicle with high accuracy. The GPS/GNSS module 802 can also be embedded with an Inertial Measurement Unit (IMU) to further improve the accuracy of the measurements. To this end, the GPS/GNSS module may additionally comprise one or more accelerometers (e.g., three), one or more gyroscopes (e.g., three), thereby allowing the receiver to operate in environments where very high dynamic and frequent interruption of signals can be expected. - In some instances, the
LiDAR module 804 can be configured to perform laser scanning, for example, by using one or more (e.g., 16) laser/detector pairs mounted in a compact housing, which can rapidly spin to scan the surrounding environment. For example, theLiDAR module 804 can be mounted on the rear of a vehicle with the cable pointing downward and tilted 90 degrees in order to scan the road. - In some cases, the lasers fire thousands of times per second, thereby providing a rich, 3D point cloud in real-time. Further, the
LiDAR module 804 can be configured to perform advanced digital signal processing and waveform analysis to provide high accuracy, extended distance sensing, and calibrated reflectivity data. In some instances, theLiDAR module 804 is capable of horizontal Field of View (FOV) of 360°, adjustable rotational speed of 5-20 rotations per second, vertical FOV of 30°, and returns of up to 100 meters dependent on application. TheLiDAR module 804 can also synchronize its data with precision, GPS-supplied time pulses, enabling the users to determine the exact firing time of each laser. - In some instances, the
LiDAR module 804 can be connected to theLiDAR interface module 805, which may serve as an interface box for theLiDAR module 804. As shown, theLiDAR interface module 805 can receive a once-a-second synchronization pulse (PPS) issued by the GPS/GNSS module 802 and forward it to theLiDAR module 804. Thereby, upon synchronization, theLiDAR module 804 can set its time stamp to the number of microsecond past the hour per coordinated universal time (UTC) time. Thereby, it would be easy to geo-reference the LiDAR data into a point cloud. - In some instances, the
spherical imaging module 806 can receive a trigger pulse from the GPS/GNSS module 802 and capture location-based visualizations for display in geographical mapping applications. For example, the user can use the spherical vision camera to capture video in a mobile environment, and view the transmission of images as they are captured using thedata collection module 803, which can be embodied as a laptop or desktop computer. In some cases, thespherical imaging module 806 is capable of 360° video streaming and covering 90% of the visual sphere. Further, thespherical imaging module 806 can be pre-calibrated and come with its own Software Development Kit (SDK), thereby allowing creation of dedicated applications for imaging capture. - In some instances, the GPS/
GNSS module 802, theLiDAR interface module 805 and thespherical imaging module 806 each can connect to thedata collection module 803 for data exchange, storage, synchronization and control using various interfaces, such as USB or Ethernet interfaces. Thedata collection module 803 herein can be a laptop computer or a desktop computer with mass storage, which may serve as a database for storage of image data. - In some instances, in order to power up one or more of the above modules, the
battery module 807 is provided as a power supply. Thebattery module 807 herein may be an auxiliary battery on the vehicle, which is able to provide a 12V direct current (DC). As shown, this 12V DC can be directly applied to the GPS/GNSS module 802, theLiDAR interface module 805 and thespherical imaging module 806. However, in order to properly power thedata collection module 803, the inverter 808 can be introduced to convert 12V DC into 120V alternating current (AC). - In some instances, a wheel encoder can be used for converting the angular position or motion of a shaft or axle of the vehicle to an analog or digital code. The analog or digital code can be further processed into information such as speed, distance and position, thereby providing further details of the vehicle in motion.
- Referring to
FIG. 10 , in a particular embodiment, once image data is uploaded to the Swarm system as shown inFIG. 1 , aprocessing pipeline 900 will start to process it. All images data is stored in file storage or database S3 and other kinds of data are stored in a database PG. All data is associated to edges roughly equal to street segments. In some instances, as shown in the upper part ofFIG. 11 , thepipeline 900 of the Swarm system may perform the following steps at corresponding blocks while exchanging data with associated databases: -
- a) Extract frames from video using FFMPEG and associate frames with metadata at
block 901; - b) Correct location data using heuristics and mapping data at block 902 (OSRM+Tiger);
- c) Locate objects, such as pedestrians, cars, street signs, street lights, etc., within imagery using trained machine learning models, such as YOLO detection at
block 903; - d) If necessary, blur images of pedestrians using OPENCV at
block 904; - e) Extrapolate raw pedestrian counts into foot traffic data at
block 905; and - f) Detects changes in mapping inventory, including, for example, new high-priority events, the removal of old high-priority events, changes in vehicle or pedestrian traffic, and/or so forth.
- a) Extract frames from video using FFMPEG and associate frames with metadata at
- Known methods of estimating pedestrian counts typically involve examining video of fixed cameras. When using fixed cameras, a fairly accurate count of pedestrians can be determined for particular locations, but little to no pedestrian information is available for areas for which camera coverage does not exist. Using mobile cameras, such as Swarm capture devices described above, can increase the area for which pedestrian counts can be determined with a given number of cameras, but introduces several new challenges. In particular, mobile cameras coverage area and coverage gaps can change in unpredictable manners and video depicting a particular street segment may be received only on an irregular basis. The following method describes a technique(s) that can be used to extrapolate pedestrian counts based on inconsistent coverage:
-
- e1) Swarm images can be queried and congregated into single time slots that represent a single vehicle pass on a street segment. This can be performed on a periodic basis, such as daily, weekly, etc. Similarly stated, video data from multiple Swarm capture devices can be received and bucketized. For example, video for each Swarm capture device can be subdivided into a bucket for each street segment (e.g., intersection to intersection) captured by that Swarm capture device.
- e2) Swarm images (e.g, bucketized Swarm images) can be processed by Darknet YOLO or any other suitable image processing technique to attribute tags to images. For example, each still image from a video segment (e.g., corresponding to a bucket associated with a street segment) can be analyzed to identify the number of pedestrians in that image. The total number of people seen for the video segment can be divided by the number of frames in that video to determine an average pedestrian count for the video segment, which may be associated with a street segment.
- e3) As discussed in further detail below, to determine a foot-traffic count for a selected street segment, pedestrian data for a time period and for the selected street segment can be combined with pedestrian data from immediately adjacent street segments. Similarly stated, video of any street segments that intersect the selected street segment can be analyzed to determine the number of pedestrians for those adjacent street segments. Video of adjacent street segments can be analyzed over any suitable time period, such as a week, a month, a year, etc.
- e4) If multiple videos have been collected for the selected street segment or adjacent street segments, pedestrian counts can be de-duplicated by any suitable means (such as aggregating passes and dividing by number of passes).
- e5) An hourly histogram (or a histogram of any other suitable interval) can be calculated based on historic observations of pedestrians, such as past observations made by Swarm devices and/or based on publicly available pedestrian count data. Hourly histogram(s) can be used to extrapolate the pedestrian count from individual vehicle passes to a total day count. Hourly histograms can be used to estimate pedestrian counts for the selected or adjacent street segments during periods (e.g., hourly periods) for which no video is available. Daily projections for the selected street segment and/or adjacent street segment can be determined by summing hourly histograms over a 24 hour period.
- e6) An inter quartile range algorithm can be applied to the approximate day projections to remove outliers.
- e7) If the multiple vehicle passes exist for the adjacent street segments, those adjacent street segment counts can be used to estimate pedestrian for the selected street segment. For example, an average (or other measure of central tendency) pedestrian count of adjacent street segment values can be calculated, outliers can be removed, and an average (or other measure of central tendency) between adjacent street segments and the selected segment can be computed.
- e8) If selected street segment counts are missing (e.g., if no video of the selected street segment is available to calculate pedestrian counts for a particular period), the average (or other measure of central tendency) of the adjacent street segments can be used to estimate the average (or other measure of central tendency) pedestrian count for the selected street segment and the period. For example, for each adjacent street segment for which video was received during the time period, image recognition of such video can be performed to produce a pedestrian count for that adjacent street segment. An average (or other measure of central tendency) of pedestrian counts of adjacent street segments can be used to estimate the pedestrian count of the selected street segment during that time period.
- e9) The daily projection of foot traffic for a selected segment can be used to approximate a monthly foot traffic projection by multiplying by 365/12.
- Mapping inventory for continuously updated based on newly received information (e.g., from Swarm system elements and/or Mapper system elements). For example, permanent and/or temporary changes to roads and/or traffic conditions can be detected and integrated into available maps. In some instances mapping inventory can be tracked for particular locations, such as points of interest, intersections, addresses, and the like. In such an instance, video collected from in the vicinity of the particular location (e.g. an intersection) can be analyzed based, for example, on metadata associated with the video. Images likely to depict the particular location (e.g., images likely to be showing crossing though an intersection) can be analyzed. For each image, the set of mapping objects that is most commonly detected can be identified. For the winning set of objects, the object types and relative positions can be associated with the intersection. Changes to the particular locations can be identified by identifying changes in the objects associated with the intersection. For example, during periodic mapping inventory updates (e.g., nightly, weekly, monthly, etc.) images of the particular location received during that period can be analyzed. If the set of images received during the period includes objects that match the set of objects associated with the intersection, the intersection can be considered unchanged. If, however, the set of images received during the period does not include objects associated with the intersection, for example, if objects received during the period includes objects that deviate from objects associated with the intersection by more than a threshold amount, then the images, video, and/or associated metadata can be passed to an Events Service UI, such as shown and discussed above with reference to
FIGS. 8 and 9 . A GIS analyst and/or machine learning process can determine if the inventory has changed and/or update the mapping inventory (e.g., indicate that a lane is closed due to construction, changes in traffic patterns, lane markings, and so forth). - Further referring to
FIG. 12 , in a particular embodiment, once image data is uploaded to the Mapper system as shown inFIG. 1 , a processing pipeline 906 will start to process it. All images data is stored in file storage or database S3 and other kinds of data are stored in a database PG. The data herein may include raw network PCAP files, PGR files, and GPS files. All data is associated to edges roughly equal to street segments. In some instances, the pipeline 906 of the Mapper system may perform the following steps at corresponding blocks while exchanging data with associated databases: -
- a) Automate Novatel Inertial Explorer to process GPS files into GNSS files that have corrected location data using the IMU data, wheel encoder data, and public base station data at
block 907; - b) Extract LiDAR data from PCAP and combine with GNSS data to create point cloud slices in CSV files and position/orientation information in POSE files, and further extract rectified and panoramic imagery from Ladybug PGR files into JPG files at
block 908; - c) Combine SCAN, POSE and rectified JPG files to create registered and colorized point cloud slices at
block 909, wherein the Iterative Closest Point (ICP) is applied herein to correct for poor GPS data, and Colorized point clouds and updated POSE information are saved to CSV and POSE files; and - d) Perform feature extraction (described below) to identify streets, sidewalks, poles, and building facades at
block 911.
- a) Automate Novatel Inertial Explorer to process GPS files into GNSS files that have corrected location data using the IMU data, wheel encoder data, and public base station data at
- In some instances, the feature extraction can be performed in different stages as follows:
-
Stage 1 -
- a) Compute normals at each 3D point by local plane fitting;
- b) Use the 3D points and normals of the scan perform a RANSAC plane fitting algorithm, detect planes with vertical normals (i.e. horizontal planes), keep the biggest two such horizontal planes and refer the set of points in these horizontal planes as Points_Ransac;
- c) Perform a Euclidean clustering (e.g., PCL Euclidean cluster extraction) on the points in Points_Ransac;
- d) Keep the largest of the clusters computed in step (c) and refer the set of these points as Points_Clustered;
- e) For each 3D point in the scan, compute its maximum distance from the horizontal planes computed in step (b), wherein the distance is an approximation of the height from the ground for each 3D point;
- f) Add points that are not in Points_Clustered, with a height from the ground below a given threshold, in Points_Clustered;
- g) Label each 3D point with a label “ground” if it belongs to Points_Clustered and a label “ransac” if it belongs to Points_Ransac and label each 3D point not in Points_Clustered with a label “non-ground,” wherein each point is assigned a distance from the ground.
-
Stage 2 -
- a) Build a K-D tree index of points in Points_Clustered (i.e. ground points) for fast search.
- b) For each 3D point P in Points_Clustered:
- i) Reconstruct a scanline neighborhood around P, by computing a neighborhood of points using Euclidean clustering, and by sorting the points based on the angle of the sensing direction;
- ii) Move forward and backward in the scanline neighborhood to detect sudden changes in depth using a cumulative statistic. If such a change is detected, the point of change is marked as a possible curb point; and
- c) Output: points in scan that could be on curbs are labeled as possible-curb-point.
-
Stage 3 -
- a) Place all possible-curb-points from all scans into a set referred to as Points_Possible_Curbs;
- b) Place all ground points from all scans into a set referred to as Points_Ground;
- c) Place the sequence of scanner positions (from global poses) into a list Scanner_Positions;
- d) Cluster the points Points_Possible_Curbs into components using Euclidean clustering, and discard small clusters;
- e) For each cluster from step (d), order points in sequence following scanner's movement by finding closest scanner positions in Scanner_Positions;
- f) Subsample ordered points of step (e) and smooth them, thereby generating a set of polylines or curves;
- g) Organize the curves of step (f) into left and right curves based on whether each curve is to the left or the right of the moving scanner (represented by Scanner_Positions);
- h) Process the left curves in sequence, and decide whether to connect with the next curve. The connection is decided based on various criteria, such as: 1) whether the connection between the curves traverses empty space wrt Points_Ground (i.e. in the case a parked car is occluding the curb), 2) the length of the connection, 3) the angle between the line connecting the curve endpoints and the curves themselves, and 4) the distance between the connecting curve and the scanner trajectory.
- i) Perform step h) for the right curves; and
- j) Output: a sequence of curb curves on the left and right of the sensor trajectory.
-
Stage 4 -
- a) Re-label as road the Points that are labeled ground or non-ground (but at a height from the ground smaller than a threshold) and being in between a left and right curb curve;
- b) Label as sidewalk the points that are labeled ground, or non-ground (but at a height from the ground smaller than a threshold) that are to the left of the left curb curve;
- c) Label as sidewalk the points that are labeled ground, or non-ground (but at a height from the ground smaller than a threshold) that are to the right of the right curb curve;
- d) Perform a RANSAC plane clustering to the non-road and non-sidewalk points into non-horizontal planes, wherein the points on such RANSAC planes that are to the left or to the right of the curb curves are labeled as facades.
- e) Output: points of each Velodyne scan are labeled either as a road, a sidewalk, facade, or others.
- The above has described in detail the operations of the feature extraction using multiple stages. It is to be understood that the processing stages herein are only for illustrative purposes and a person skilled in the art can envisage other stages or steps to perform the feature extraction based on the teaching herein. In addition, the Mapper system can also process the GPS data at
block 910, and perform Pedestrian detection and blur faces atblocks - As previously noted, according to the embodiments of the present disclosure, the data from the Mapper and Swarm systems can be merged, thereby improving the efficacy of each system. For example, the system can compare the features in Mapper data to features in Swarm data to improve the Swarm accuracy. The exemplary steps are as follows:
-
- a) Perform feature extraction on Mapper 2D data to detect landmarks;
- b) Perform feature extraction on Swarm data to detect landmarks;
- c) Compare Swarm features within 100 m of each Mapper feature to find the same landmarks; and
- d) Update the Swarm location data when matching features are found.
- Referring to
FIG. 13 , in a particular embodiment, a web application, which can be used by a fleet manager and referred to as “fleet manager software,” is capable of controlling and monitoring the vehicles using the street level intelligence system. The application can be configured to allow the fleet manager to carry out multiple actions, including but not limited to: -
- a) see the current location of each fleet vehicle;
- b) download video already offloaded;
- c) request priority offload over LTE; and
- d) view driver quality data from Zendrive.
- As illustrated in
FIG. 13 , there are six vehicles on the road and shown in real-time on the map. The “Vehicle” column lists these six vehicles with their respective name, traveled distances, and grades. The vehicle named “Grimlock” is shown grey in the list as it is offline and in maintenance. The vehicles herein may be outfitted with the Swarm systems as discussed above and elsewhere in the specification. It should be understood that the user interface shown inFIG. 12 can be a homepage of the fleet manager software. - Referring to
FIG. 16 , in a particular embodiment, a web application, which can be used by a customer and referred to as “street level intelligence software,” is capable of providing site intelligence payload to the customer. The web application herein can be configured to allow the customers to carry out the actions, including but not limited to: -
- a) view a map overlaid with segments for which pertinent data has been collected;
- b) select one or more segments and add them to a cart;
- c) pay for access to segments using a credit card;
- d) download a payload for those segments containing either 3D/2D site intelligence or pedestrian analytics;
- e) access previously-paid payloads; and
- f) share payloads with partners or other customers.
- In some instances, the site intelligence payload may comprise the following:
-
- a) stitched color 3D point cloud (see Stitching Process as below);
- b) panoramic images;
- c) panoramic viewing software; and
- d) metadata and instructions.
- As an example, the steps for stitching point cloud data are as follows:
-
- a) Colorized scans are gathered for segment runs, ordered by capture date;
- b) Scans are filtered to eliminate stalled traffic redundancy (movement checked);
- c) Ensure that the scans have an acceptable count based on segment length and cleaned scan count;
- d) Concatenate the filtered scans into a single xyz file; and
- e) Resultant file is passed to a las converter binary which exports a las file.
- In some instances, the pedestrian analytics payload may contain the following information:
-
- a) a PNG map of the requested segments overlaid with a infographics representing estimated pedestrian foot traffic for each segment; and
- b) a CSV containing raw data that can be imported into other software.
- As illustrated in
FIG. 16 , the customer can input areas or segments of interest in the Search field. Further, the customer can select the cart icon as shown and place the segments selected from the map into it. In some cases, the customer can also see the discounts shown in the cart. Finally, the customer can select coin icon as shown to pay the search fees using for example, credit cards. In response, the street level intelligence software according to the embodiments of the present disclosure can provide items, such as Search miles, Search groups, or Search cost for customer's viewing. Additionally, the customer can order other data that is interested to him or her, such as the PNG map of the searched segments or CSV files. - In some embodiments, the platforms, systems, media, and methods disclosed herein include one or more non-transitory computer readable storage media encoded with a program including instructions executable by the operating system of an optionally networked digital processing device. In further embodiments, a computer readable storage medium is a tangible component of a digital processing device. In still further embodiments, a computer readable storage medium is optionally removable from a digital processing device. In some embodiments, a computer readable storage medium includes, by way of non-limiting examples, CD-ROMs, DVDs, flash memory devices, solid state memory, magnetic disk drives, magnetic tape drives, optical disk drives, cloud computing systems and services, and the like. In some cases, the program and instructions are permanently, substantially permanently, semi-permanently, or non-transitorily encoded on the media.
- In some embodiments, the platforms, systems, media, and methods disclosed herein include processor executable instructions, or use of the same. A computer program includes a sequence of instructions, executable in the digital processing device's CPU, written to perform a specified task. Computer readable instructions may be implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), data structures, and the like, that perform particular tasks or implement particular abstract data types. In light of the disclosure provided herein, those of skill in the art will recognize that a computer program may be written in various versions of various languages.
- The functionality of the computer readable instructions may be combined or distributed as desired in various environments. In some embodiments, a computer program comprises one sequence of instructions. In some embodiments, a computer program comprises a plurality of sequences of instructions. In some embodiments, a computer program is provided from one location. In other embodiments, a computer program is provided from a plurality of locations. In various embodiments, a computer program includes one or more software modules. In various embodiments, a computer program includes, in part or in whole, one or more web applications, one or more mobile applications, one or more standalone applications, one or more web browser plug-ins, extensions, add-ins, or add-ons, or combinations thereof.
- In some embodiments, the platforms, systems, media, and methods disclosed herein include software, server, and/or database modules, or use of the same. In view of the disclosure provided herein, software modules are created by techniques known to those of skill in the art using machines, software, and languages known to the art. The software modules disclosed herein are implemented in a multitude of ways. In various embodiments, a software module comprises a file, a section of code, a programming object, a programming structure, or combinations thereof. In further various embodiments, a software module comprises a plurality of files, a plurality of sections of code, a plurality of programming objects, a plurality of programming structures, or combinations thereof. In various embodiments, the one or more software modules comprise, by way of non-limiting examples, a web application, a mobile application, and a standalone application. In some embodiments, software modules are in one computer program or application. In other embodiments, software modules are in more than one computer program or application. In some embodiments, software modules are hosted on one machine. In other embodiments, software modules are hosted on more than one machine. In further embodiments, software modules are hosted on cloud computing platforms. In some embodiments, software modules are hosted on one or more machines in one location. In other embodiments, software modules are hosted on one or more machines in more than one location.
- In some embodiments, the platforms, systems, media, and methods disclosed herein include one or more databases, or use of the same. In view of the disclosure provided herein, those of skill in the art will recognize that many databases are suitable for storage and retrieval of vehicle, location, image, feature, and street level intelligence information. In various embodiments, suitable databases include, by way of non-limiting examples, relational databases, non-relational databases, object oriented databases, object databases, entity-relationship model databases, associative databases, and XML databases. Further non-limiting examples include SQL, PostgreSQL, MySQL, Oracle, DB2, and Sybase. In some embodiments, a database is internet-based. In further embodiments, a database is web-based. In still further embodiments, a database is cloud computing-based. In other embodiments, a database is based on one or more local computer storage devices.
Claims (6)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/216,079 US20210216795A1 (en) | 2017-05-31 | 2021-03-29 | System of vehicles equipped with imaging equipment for high-definition near real-time map generation |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201762513056P | 2017-05-31 | 2017-05-31 | |
US15/995,041 US11802968B2 (en) | 2017-05-31 | 2018-05-31 | System of vehicles equipped with imaging equipment for high-definition near real-time map generation |
US17/216,079 US20210216795A1 (en) | 2017-05-31 | 2021-03-29 | System of vehicles equipped with imaging equipment for high-definition near real-time map generation |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/995,041 Division US11802968B2 (en) | 2017-05-31 | 2018-05-31 | System of vehicles equipped with imaging equipment for high-definition near real-time map generation |
Publications (1)
Publication Number | Publication Date |
---|---|
US20210216795A1 true US20210216795A1 (en) | 2021-07-15 |
Family
ID=64455156
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/995,041 Active 2038-11-27 US11802968B2 (en) | 2017-05-31 | 2018-05-31 | System of vehicles equipped with imaging equipment for high-definition near real-time map generation |
US17/216,079 Abandoned US20210216795A1 (en) | 2017-05-31 | 2021-03-29 | System of vehicles equipped with imaging equipment for high-definition near real-time map generation |
US17/216,043 Active 2039-01-22 US11879977B2 (en) | 2017-05-31 | 2021-03-29 | System of vehicles equipped with imaging equipment for high-definition near real-time map generation |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/995,041 Active 2038-11-27 US11802968B2 (en) | 2017-05-31 | 2018-05-31 | System of vehicles equipped with imaging equipment for high-definition near real-time map generation |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/216,043 Active 2039-01-22 US11879977B2 (en) | 2017-05-31 | 2021-03-29 | System of vehicles equipped with imaging equipment for high-definition near real-time map generation |
Country Status (5)
Country | Link |
---|---|
US (3) | US11802968B2 (en) |
EP (1) | EP3631682A4 (en) |
JP (1) | JP7199627B2 (en) |
KR (1) | KR20200016886A (en) |
WO (1) | WO2018222937A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11260864B2 (en) * | 2018-10-08 | 2022-03-01 | Mando Mobility Solutions Corporation | Path generation apparatus at intersection, and method and apparatus for controlling vehicle at intersection |
Families Citing this family (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11802968B2 (en) | 2017-05-31 | 2023-10-31 | Woven By Toyota, U.S., Inc. | System of vehicles equipped with imaging equipment for high-definition near real-time map generation |
WO2019000417A1 (en) * | 2017-06-30 | 2019-01-03 | SZ DJI Technology Co., Ltd. | Map generation systems and methods |
US11349903B2 (en) * | 2018-10-30 | 2022-05-31 | Toyota Motor North America, Inc. | Vehicle data offloading systems and methods |
US10823855B2 (en) * | 2018-11-19 | 2020-11-03 | Fca Us Llc | Traffic recognition and adaptive ground removal based on LIDAR point cloud statistics |
US10915762B1 (en) * | 2018-12-07 | 2021-02-09 | Waymo Llc | Sidewalk detection for pedestrian behavior modeling |
JP1671702S (en) * | 2018-12-24 | 2020-11-02 | passenger car | |
US11520331B2 (en) * | 2018-12-28 | 2022-12-06 | Intel Corporation | Methods and apparatus to update autonomous vehicle perspectives |
JP7215231B2 (en) * | 2019-03-04 | 2023-01-31 | トヨタ自動車株式会社 | Information processing device, detection method and program |
US20220172492A1 (en) * | 2019-04-05 | 2022-06-02 | Volvo Truck Corporation | A method and a control unit for determining a parameter indicative of a road capability of a road segment supporting a vehicle |
US11245909B2 (en) * | 2019-04-29 | 2022-02-08 | Baidu Usa Llc | Timestamp and metadata processing for video compression in autonomous driving vehicles |
JPWO2020230710A1 (en) * | 2019-05-10 | 2020-11-19 | ||
US11619724B2 (en) * | 2019-06-26 | 2023-04-04 | Nvidia Corporation | Calibration of multiple lidars mounted on a vehicle using localization based on a high definition map |
US10848920B1 (en) * | 2019-09-17 | 2020-11-24 | Microsoft Technology Licensing, Llc | Generation of precise geospatial coordinates |
CN111259767B (en) * | 2020-01-13 | 2023-04-18 | 厦门大学 | Traffic illegal behavior identification method and system based on traffic data and street view data |
US20210282338A1 (en) * | 2020-03-11 | 2021-09-16 | Aerobotics (Pty) Ltd | Systems and methods for predicting crop size and yield |
CN111623787A (en) * | 2020-03-16 | 2020-09-04 | 蘑菇车联信息科技有限公司 | Distributed data acquisition method and device based on Internet vehicle machine |
CN111935644B (en) * | 2020-08-10 | 2021-08-24 | 腾讯科技(深圳)有限公司 | Positioning method and device based on fusion information and terminal equipment |
JP2023045313A (en) | 2021-09-21 | 2023-04-03 | トヨタ自動車株式会社 | Information processing device and information processing method |
US11938963B1 (en) * | 2022-12-28 | 2024-03-26 | Aurora Operations, Inc. | Remote live map system for autonomous vehicles |
Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070154067A1 (en) * | 1998-10-23 | 2007-07-05 | Facet Technology Corp. | Method and apparatus for identifying objects depicted in a videostream |
US20140111647A1 (en) * | 2011-05-03 | 2014-04-24 | Alon Atsmon | Automatic image content analysis method and system |
US20150312341A1 (en) * | 2014-04-24 | 2015-10-29 | Vivint, Inc. | Saving video clips on a storage of limited size based on priority |
US20160035096A1 (en) * | 2012-12-28 | 2016-02-04 | Trimble Navigation Limited | Scene documentation |
US20160050114A1 (en) * | 2014-08-18 | 2016-02-18 | Qualcomm Incorporated | Multi-device sensor subsystem joint optimization |
US9418554B2 (en) * | 2014-08-07 | 2016-08-16 | Verizon Patent And Licensing Inc. | Method and system for determining road conditions based on driver data |
US20160236621A1 (en) * | 2005-09-16 | 2016-08-18 | Digital Ally, Inc. | Vehicle-mounted video system with distributed processing |
US9620168B1 (en) * | 2015-12-21 | 2017-04-11 | Amazon Technologies, Inc. | Cataloging video and creating video summaries |
US20170124476A1 (en) * | 2015-11-04 | 2017-05-04 | Zoox, Inc. | Automated extraction of semantic information to enhance incremental mapping modifications for robotic vehicles |
US9996750B2 (en) * | 2016-06-01 | 2018-06-12 | Gopro, Inc. | On-camera video capture, classification, and processing |
Family Cites Families (36)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP0807352A1 (en) | 1995-01-31 | 1997-11-19 | Transcenic, Inc | Spatial referenced photography |
US9183351B2 (en) | 2000-05-30 | 2015-11-10 | Vladimir Shusterman | Mobile system with network-distributed data processing for biomedical applications |
US6895126B2 (en) | 2000-10-06 | 2005-05-17 | Enrico Di Bernardo | System and method for creating, storing, and utilizing composite images of a geographic location |
JP2002319091A (en) | 2001-04-20 | 2002-10-31 | Fuji Heavy Ind Ltd | Device for recognizing following vehicle |
US8207964B1 (en) | 2008-02-22 | 2012-06-26 | Meadow William D | Methods and apparatus for generating three-dimensional image data models |
US20070132773A1 (en) * | 2005-12-08 | 2007-06-14 | Smartdrive Systems Inc | Multi-stage memory buffer and automatic transfers in vehicle event recording systems |
US20070217761A1 (en) * | 2006-03-07 | 2007-09-20 | Coban Research And Technologies, Inc. | Method for video/audio recording using unrestricted pre-event/post-event buffering with multiple bit and frame rates buffer files |
EP2084629A1 (en) * | 2006-11-14 | 2009-08-05 | Johnson Controls Technology Company | System and method of synchronizing an in-vehicle control system with a remote source |
US8156320B2 (en) * | 2008-08-27 | 2012-04-10 | Wireless Silicon Group, Llc | Method and apparatus for fast booting a portable computing device allowing for immediate operation |
US8817094B1 (en) | 2010-02-25 | 2014-08-26 | Target Brands, Inc. | Video storage optimization |
EP2378465A1 (en) | 2010-04-13 | 2011-10-19 | Autoliv Development AB | Driver assisting system and method for a motor vehicle |
US8526973B2 (en) * | 2011-03-21 | 2013-09-03 | John Douglas Thomson | Safety system to disable and activate functions in smartphone's, mobile phones and portable communication devices while driving |
US8907968B2 (en) | 2011-03-31 | 2014-12-09 | Panasonic Corporation | Image rendering device, image rendering method, and image rendering program for rendering stereoscopic panoramic images |
CN102834849B (en) | 2011-03-31 | 2016-08-31 | 松下知识产权经营株式会社 | Carry out the image displaying device of the description of three-dimensional view picture, image drawing method, image depiction program |
US9108640B2 (en) | 2012-01-31 | 2015-08-18 | Google Inc. | Systems and methods for monitoring and reporting road quality |
US9080882B2 (en) | 2012-03-02 | 2015-07-14 | Qualcomm Incorporated | Visual OCR for positioning |
WO2014028672A1 (en) * | 2012-08-14 | 2014-02-20 | Inmobly, Inc. | System and method for efficient use of network bandwidth |
US8996228B1 (en) | 2012-09-05 | 2015-03-31 | Google Inc. | Construction zone object detection using light detection and ranging |
RU2569123C1 (en) | 2012-09-12 | 2015-11-20 | Омрон Корпорейшн | Device to generate command of data flow control and sensor control device |
US9488492B2 (en) | 2014-03-18 | 2016-11-08 | Sri International | Real-time system for multi-modal 3D geospatial mapping, object recognition, scene annotation and analytics |
WO2014060786A1 (en) * | 2012-10-15 | 2014-04-24 | Freescale Semiconductor, Inc. | Method and apparatus for reporting traffic information |
AU2014214545B2 (en) * | 2013-02-07 | 2018-06-28 | Iomniscient Pty Ltd | A surveillance system |
US9633272B2 (en) * | 2013-02-15 | 2017-04-25 | Yahoo! Inc. | Real time object scanning using a mobile phone and cloud-based visual search engine |
US9141107B2 (en) | 2013-04-10 | 2015-09-22 | Google Inc. | Mapping active and inactive construction zones for autonomous driving |
US9070344B2 (en) | 2013-05-24 | 2015-06-30 | Lenovo Enterprise Solutions (Singapore) Pte. Ltd. | Integrating street view with live video data |
TWI552602B (en) | 2014-08-19 | 2016-10-01 | 晶睿通訊股份有限公司 | Blur detection method of images, monitoring device, and monitoring system |
US20160075282A1 (en) * | 2014-09-11 | 2016-03-17 | Christian Johnson | Vehicle Monitoring, Safety, and Tracking System |
GB2532948B (en) | 2014-12-02 | 2021-04-14 | Vivo Mobile Communication Co Ltd | Object Recognition in a 3D scene |
JP6453645B2 (en) | 2014-12-25 | 2019-01-16 | 京セラ株式会社 | Stereo camera device, moving object, stereo camera system, server device, and control method |
JP2017021745A (en) | 2015-07-15 | 2017-01-26 | パイオニア株式会社 | Information collection device, information collection server, and information collection system |
JP6676147B2 (en) | 2015-08-20 | 2020-04-08 | ゼンドライヴ, インコーポレイテッドZendrive, Inc. | A method for accelerometer-assisted navigation |
US9776500B2 (en) | 2015-08-24 | 2017-10-03 | Kawasaki Jukogyo Kabushiki Kaisha | Side cover body for motorcycle |
US9805276B2 (en) * | 2016-03-31 | 2017-10-31 | Toyota Jidosha Kabushiki Kaisha | Generating real-time driver familiarity index for fine-grained dynamic road scenes |
US9460613B1 (en) * | 2016-05-09 | 2016-10-04 | Iteris, Inc. | Pedestrian counting and detection at a traffic intersection based on object movement within a field of view |
US10764713B2 (en) * | 2016-05-11 | 2020-09-01 | Here Global B.V. | Map based feedback loop for vehicle observation |
US11802968B2 (en) | 2017-05-31 | 2023-10-31 | Woven By Toyota, U.S., Inc. | System of vehicles equipped with imaging equipment for high-definition near real-time map generation |
-
2018
- 2018-05-31 US US15/995,041 patent/US11802968B2/en active Active
- 2018-05-31 WO PCT/US2018/035494 patent/WO2018222937A1/en active Application Filing
- 2018-05-31 KR KR1020197038274A patent/KR20200016886A/en not_active Application Discontinuation
- 2018-05-31 EP EP18809691.1A patent/EP3631682A4/en active Pending
- 2018-05-31 JP JP2019564035A patent/JP7199627B2/en active Active
-
2021
- 2021-03-29 US US17/216,079 patent/US20210216795A1/en not_active Abandoned
- 2021-03-29 US US17/216,043 patent/US11879977B2/en active Active
Patent Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070154067A1 (en) * | 1998-10-23 | 2007-07-05 | Facet Technology Corp. | Method and apparatus for identifying objects depicted in a videostream |
US20160236621A1 (en) * | 2005-09-16 | 2016-08-18 | Digital Ally, Inc. | Vehicle-mounted video system with distributed processing |
US20140111647A1 (en) * | 2011-05-03 | 2014-04-24 | Alon Atsmon | Automatic image content analysis method and system |
US20160035096A1 (en) * | 2012-12-28 | 2016-02-04 | Trimble Navigation Limited | Scene documentation |
US20150312341A1 (en) * | 2014-04-24 | 2015-10-29 | Vivint, Inc. | Saving video clips on a storage of limited size based on priority |
US9418554B2 (en) * | 2014-08-07 | 2016-08-16 | Verizon Patent And Licensing Inc. | Method and system for determining road conditions based on driver data |
US20160050114A1 (en) * | 2014-08-18 | 2016-02-18 | Qualcomm Incorporated | Multi-device sensor subsystem joint optimization |
US20170124476A1 (en) * | 2015-11-04 | 2017-05-04 | Zoox, Inc. | Automated extraction of semantic information to enhance incremental mapping modifications for robotic vehicles |
US9620168B1 (en) * | 2015-12-21 | 2017-04-11 | Amazon Technologies, Inc. | Cataloging video and creating video summaries |
US9996750B2 (en) * | 2016-06-01 | 2018-06-12 | Gopro, Inc. | On-camera video capture, classification, and processing |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11260864B2 (en) * | 2018-10-08 | 2022-03-01 | Mando Mobility Solutions Corporation | Path generation apparatus at intersection, and method and apparatus for controlling vehicle at intersection |
Also Published As
Publication number | Publication date |
---|---|
KR20200016886A (en) | 2020-02-17 |
US11802968B2 (en) | 2023-10-31 |
JP2020522783A (en) | 2020-07-30 |
WO2018222937A1 (en) | 2018-12-06 |
EP3631682A1 (en) | 2020-04-08 |
EP3631682A4 (en) | 2021-02-24 |
US20180349715A1 (en) | 2018-12-06 |
JP7199627B2 (en) | 2023-01-06 |
US20210216794A1 (en) | 2021-07-15 |
US11879977B2 (en) | 2024-01-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11879977B2 (en) | System of vehicles equipped with imaging equipment for high-definition near real-time map generation | |
US10520593B2 (en) | Installation and use of vehicle light ranging system | |
US11003919B1 (en) | Systems and methods for detecting traffic violations using mobile detection devices | |
US11127162B2 (en) | Method and apparatus for improved location decisions based on surroundings | |
KR20190094405A (en) | Method and system for video-based positioning and mapping | |
US11481991B2 (en) | System and method for detecting and transmitting incidents of interest of a roadway to a remote server | |
JP6006641B2 (en) | Traffic signal map creation and detection | |
US11175156B2 (en) | Method and apparatus for improved location decisions based on surroundings | |
US11475766B1 (en) | Systems and methods for user reporting of traffic violations using a mobile application | |
CN107851125A (en) | The processing of two step object datas is carried out by vehicle and server database to generate, update and transmit the system and method in accurate road characteristic data storehouse | |
CN110945320B (en) | Vehicle positioning method and system | |
US10223598B2 (en) | Method of generating segmented vehicle image data, corresponding system, and vehicle | |
CN107850453A (en) | Road data object is matched to generate and update the system and method for accurate transportation database | |
CN107850672A (en) | System and method for accurate vehicle positioning | |
US10984275B1 (en) | Determining location coordinates of a vehicle based on license plate metadata and video analytics | |
US20220351525A1 (en) | Systems and methods for enforcing traffic congestion pricing | |
WO2021086884A1 (en) | System, apparatus and method of provisioning allotments utilizing machine visioning | |
Pordel et al. | dmap: A low-cost distributed mapping concept for future road asset management | |
Jackson et al. | DATE OF SUBMISSION (REVISED VERSION): NOV 15TH, 2012 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED |
|
AS | Assignment |
Owner name: CARMERA, INC., NEW YORK Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GUPTA, RO;DAY, JUSTIN;NICHOLL, CHASE;AND OTHERS;SIGNING DATES FROM 20170810 TO 20170911;REEL/FRAME:056166/0677 Owner name: CARMERA, INC., NEW YORK Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SORRELGREEN, ETHAN;DINH, HUONG;REEL/FRAME:056166/0645 Effective date: 20180625 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: CARMERA, LLC, NEW YORK Free format text: ENTITY CONVERSION;ASSIGNOR:CARMERA, INC.;REEL/FRAME:061296/0032 Effective date: 20220527 Owner name: WOVEN PLANET NORTH AMERICA, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:CARMERA, LLC;REEL/FRAME:060857/0207 Effective date: 20220628 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
AS | Assignment |
Owner name: WOVEN BY TOYOTA, U.S., INC., CALIFORNIA Free format text: CHANGE OF NAME;ASSIGNOR:WOVEN PLANET NORTH AMERICA, INC.;REEL/FRAME:064065/0601 Effective date: 20230322 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |