CN113748315A

CN113748315A - System for automatic lane marking

Info

Publication number: CN113748315A
Application number: CN201880100697.6A
Authority: CN
Inventors: 侯庭波; 张妍
Original assignee: Beijing Didi Infinity Technology and Development Co Ltd
Current assignee: Beijing Didi Infinity Technology and Development Co Ltd
Priority date: 2018-12-27
Filing date: 2018-12-27
Publication date: 2021-12-03
Also published as: WO2020139355A1

Abstract

Systems and processes may automatically identify lane markers within an image by using a machine learning model. The machine learning model may use the reduced data set and output an improved estimate of the lane markings by applying normalized data or images to the machine learning model. Each image applied to the model may be normalized by, for example, rotating each of the images so that the depicted roads are horizontal or otherwise share the same angle. By aligning different road images, the amount of data applied to the model or model generation can be reduced and the accuracy of the machine learning model can be improved. Furthermore, the use of standardized images by the machine learning model enables the computational resources for applying data to the machine learning model, for example, to identify lane markers within the images, to be reduced.

Description

System for automatic lane marking

Incorporation by reference of priority applications

Any and all applications, if any, for which a foreign or domestic priority claim is identified in the application data sheet of the present application are herein incorporated by reference in their entirety in accordance with 37CFR 1.57. Again, the present disclosure is related to the following disclosures filed on 12/27/2018 (same date as the present disclosure), which are all incorporated herein by reference for any purpose: U.S. application No. 16/234,130 entitled "image preprocessing in a lane marker determination system"; U.S. application No. 16/233,960 entitled "generating machine learning models using image preprocessing"; and U.S. application No. 16/233,989 entitled "system for automatic lane marking".

Copyright notice

A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the U.S. patent and trademark office files and/or records, but otherwise reserves all copyright rights whatsoever.

Background

Vehicles, such as vehicles for shared travel purposes, vehicles providing driver assistance functionality, and/or automated or autonomously driven vehicles (AV), may use an onboard data processing system to acquire and process sensor data to perform a wide variety of functions. For example, functions may include determining and/or displaying navigation routes, identifying road signs, detecting objects and/or road obstacles, controlling vehicle operation, and so forth.

Some of the various functions performed by the onboard data processing system involve the use of maps and/or road images. For example, providing a navigational route may rely on providing a map to an onboard data processing system. These maps may be obtained using data provided by the imaging system and/or the user. Sometimes maps are out-of-date or provide less than optimal information for certain functions, such as navigation routing, for example, changes in road networks, accidents, weather conditions, or accidents. Furthermore, even if the maps are up-to-date, some maps are not granular enough for some functions. For example, a map illustrating the existence of roads may be sufficient to provide a route, but may not be sufficient for autonomous driving, which may require additional information (e.g., speed zone information).

Disclosure of Invention

As previously mentioned, certain functions performed by the onboard data processing system rely on maps and/or images of roads. Alternatively, the onboard data processing system may rely on data obtained from or otherwise corresponding to map data. Some functions that rely on maps, road images, and/or map data include navigation routing, autopilot, driving assistance functions, or shared travel routing, among others. Many of these functions require access to the lane marking data or otherwise improve performance by having access to the lane marking data. For example, in order to keep an autonomous vehicle within a particular lane, it is beneficial, if not necessary, to identify lane markings on a road or road network.

One method of providing lane marking data to a mapping or routing system is to have a user manually identify lane markings within the mapping or image data provided to the routing or automatic driving system. However, manually identifying lane markings may be inadequate and problematic for a variety of reasons. For example, the number of roads and the number of road markings in a road network may hinder manual marking, human users are prone to errors, and road and lane markings change from time to time, requiring constant updating. Therefore, artificial lane marking is not feasible for large or real-world road networks.

Another method of providing lane marking data is to use an automatic lane marking system. An automatic lane marking system may analyze images of a map to determine the presence and location of lane markings on a roadway. Because the road network and/or lane markings change over time, the automatic lane marking system may analyze the updated images to determine the location of the lane. Using an automatic system makes lane marking more robust than manual marking. However, because roads are not consistent in direction and shape, and because road markings may vary greatly (e.g., different colors, different shapes, solid and dashed lines, etc.) among different roads in a road network, a large amount of training data may be required to train a machine learning system that can be used to generate machine learning models. These machine learning models may be used in an automatic lane marking process. Furthermore, significant computational resources, both memory and processing speed, may be required, as well as processing time to perform the automatic lane marking process.

Moreover, in some cases, the image of the map may contain significantly more cumbersome and unrelated data than the road-related data. For example, the image of the map may contain buildings, trees, and other non-road related environmental features. Thus, in some cases, the percentage of an image that includes road-related pixels compared to background-related pixels or non-road-related images may be relatively low. For example, in some cases, the road may be 25%, 15%, 10% of the above examples of images or any value therebetween or less. Significantly more non-road-related pixels than road-related pixels may result in inaccurate processing of the machine learning model or may result in a relatively large amount of training data used to generate the machine learning model, which in turn may require more computing resources to process.

Embodiments disclosed herein present systems and processes that enable reduction or division of images of a road network into sub-images that depict straight or substantially straight roads in the road network. Still further, embodiments disclosed herein can normalize each sub-image by, for example, rotating the sub-image so that the depicted road is level or otherwise shares the same angle. Advantageously, in certain embodiments, the systems and processes disclosed herein reduce the amount of training data used in machine learning systems to create machine learning models for automatic lane marking systems and processes. Also, embodiments disclosed herein may be as accurate or more accurate as compared to systems that use larger amounts of training data. In addition, embodiments disclosed herein reduce the amount of computing resources used to perform the automatic lane marking process while maintaining or improving the accuracy of the lane marking process.

Additionally, certain embodiments disclosed herein are capable of identifying non-road portions of a divided image and cropping the divided image to increase the percentage of the divided image that contains roads while reducing the percentage of the divided image that includes background material or pixels. By emphasizing foreground pixels in the image that are associated with the road and deemphasizing background pixels in the image that are relevant to non-road aspects, the accuracy of the machine learning model can be improved while reducing the amount of training data, and thus the amount of computing resources used to generate the machine learning model.

Drawings

FIG. 1A illustrates a block diagram of a networked vehicle environment in which one or more vehicles and/or one or more user devices interact with a server via a network, according to one embodiment.

FIG. 1B illustrates a block diagram showing the vehicle of FIG. 1A communicating with one or more other vehicles and/or servers of FIG. 1A, according to one embodiment.

FIG. 2A illustrates a block diagram of a map generation system, according to some embodiments.

FIG. 2B illustrates an embodiment of the model generation system of FIG. 2A.

FIG. 3 presents a flow diagram of an example road extraction and normalization process, in accordance with certain embodiments.

FIG. 4 presents a flowchart of an example machine learning model generation process, in accordance with certain embodiments.

Fig. 5 presents a flow chart of an example automatic lane marking process, in accordance with certain embodiments.

FIG. 6 illustrates an example image of a road network with a segmented sketch in accordance with some embodiments.

Fig. 7 shows an enlarged portion of the example image of fig. 6.

FIGS. 8A and 8B illustrate examples of image segments pre-processed to identify lane markers within the image shown in FIG. 8C, according to some embodiments.

Fig. 9A and 9B illustrate additional examples of pre-processed image segments depicting road segments, according to some embodiments.

Fig. 9C illustrates a portion of an image segment corresponding to the image segment of fig. 9A and 9B after applying lane markings determined by a machine learning model to the image, in accordance with certain embodiments.

FIG. 9D illustrates the image segment shown in FIG. 9C, but with a bounding box label corresponding to a marked lane, in accordance with certain embodiments.

Fig. 10 shows an image comparison of the lane marking process.

Detailed Description

For a detailed description and examples of systems and methods according to one or more illustrative embodiments of the present disclosure, reference may be made to the accompanying drawings, which are entitled "systems and methodsLane marking system based on machine learningIs and is entitled asExample embodimentsAnd figures 2A-10. Still further, the network vehicle environment 100 described in FIGS. 1A-1B herein may be configuredLane based on machine learning Marking systemAnd/or incorporated into the networked vehicle environment described herein in fig. 1A-1B.

The various embodiments described herein are closely related to, enabled by, and dependent on vehicle and/or computer technology. For example, the machine learning-based automatic lane marking systems described herein in connection with various embodiments cannot reasonably be performed by humans alone without the vehicle and/or computer technology employed to implement the embodiments.

■Networked vehicle environment

FIG. 1A illustrates a block diagram of a networked vehicle environment 100 in which one or more vehicles 120 and/or one or more user devices 102 interact with a server 130 via a network 110, according to one embodiment. For example, vehicle 120 may be equipped to provide travel sharing and/or other location-based services to help the driver control vehicle operation (e.g., through various driver assistance features such as adaptive and/or conventional cruise control, adaptive headlamp control, anti-lock braking, auto parking, night vision, blind spot monitoring, collision avoidance, crosswind stabilization, driver fatigue detection, driver monitoring systems, emergency driver assistance, intersection assistance, hill descent control, smart speed adaptation, lane centering, lane departure warning, forward, rear, and/or side parking sensors, pedestrian detection, rain sensors, look-around systems, tire pressure monitors, traffic sign identification, steering assistance, reverse driving warnings, traffic condition alerts, etc.) and/or to fully control vehicle operation. Thus, the vehicle 120 may be conventional gasoline, natural gas, biofuel, electric, hydrogen, etc., configured to provide shared egress and/or other location-based services, provide driver-assisted functionality (e.g., one or more of the driver-assist features described herein), and/or an automated or autonomous driving vehicle (AV). Vehicle 120 may be an automobile, truck, van, bus, motorcycle, scooter, bicycle, and/or any other motorized vehicle.

Server 130 may communicate with vehicle 120 to obtain vehicle data, such as route data, sensor data, perception data, vehicle 120 control data, vehicle 120 component failure and/or fault data, and so forth. Server 130 may process and store these vehicle data for use in other operations performed by server 130 and/or another computing system (not shown). Such operations may include running a diagnostic model for identifying vehicle 120 operational issues (e.g., causes of vehicle 120 navigation errors, abnormal sensor readings, unidentified objects, vehicle 120 component failures, etc.); running a model that simulates the performance of the vehicle 120 given a set of variables; identify objects that the vehicle 120 cannot identify, generate control instructions that, when executed by the vehicle 120, cause the vehicle 120 to drive and/or maneuver in a certain manner along the specified path; and/or the like.

Server 130 may also transmit data to vehicle 120. For example, server 130 may transmit map data, firmware and/or software updates, vehicle 120 control instructions, identification of objects that are not otherwise recognized by vehicle 120, passenger access information, traffic data, and/or the like.

In addition to communicating with one or more vehicles 120, server 130 may also be capable of communicating with one or more user devices 102. In particular, server 130 may provide web services to enable users to request location-based services (e.g., a shipping service, such as a shared travel service) through an application running on user device 102. For example, the user device 102 may correspond to a computing device, such as a smart phone, tablet, laptop, smart watch, or any other device that communicates with the server 130 over the network 110. In this embodiment, the user device 102 executes an application, such as a mobile application, that the user operating the user device 102 may use to interact with the server 130. For example, the user device 102 may communicate with the server 130 to provide location data and/or queries to the server 130, receive map-related data and/or directions from the server 130, and/or the like.

Server 130 may process the request and/or other data received from user device 102 to identify a service provider (e.g., a driver of vehicle 120) to provide the requested service to the user. Further, server 130 may receive data, such as user trip access or destination data, user location query data, and the like, based on which server 130 identifies areas, addresses, and/or other locations associated with various users. The server 130 may then use the identified location to provide directions to the service provider and/or user to the determined access location.

Applications running on user device 102 may be created and/or manufactured by the same entity responsible for server 130. Alternatively, the application running on the user device 102 may be a third party application that includes features (e.g., an application programming interface or software development suite) that enable communication with the server 130.

For simplicity and ease of explanation, one server 130 is illustrated in FIG. 1A. However, it should be appreciated that server 130 may be a single computing device, or may include a plurality of different computing devices logically or physically grouped to operate collectively as a server system. The components of the server 130 may be implemented in dedicated hardware (e.g., a server computing device with one or more ASICs) without software, or as a combination of hardware and software. Additionally, the modules and components of server 130 may be combined on one server computing device, or provided individually or in groups on several server computing devices. In some embodiments, server 130 may include additional or fewer components than shown in FIG. 1A.

The network 110 includes any wired network, wireless network, or combination thereof. For example, the network 110 may be a personal area network, a local area network, a wide area network, an over-the-air broadcast network (e.g., a network for broadcast or television), a cable network, a satellite network, a cellular telephone network, or a combination thereof. As another example, the network 110 may be a publicly accessible network linking networks, possibly operated by a variety of different mechanisms, such as the Internet. In some embodiments, the network 110 may be a private or semi-private network, such as a corporate or university intranet. Network 110 may include one or more wireless networks, such as for a global system for mobile communications (GSM) network, a Code Division Multiple Access (CDMA) network, a Long Term Evolution (LTE) network, or any other type of wireless network. The network 110 may communicate via the internet or any other of the types of networks described above using protocols and components. For example, the protocols used by the network 110 may include hypertext transfer protocol (HTTP), hypertext transfer security protocol (HTTPs), Message Queue Telemetry Transport (MQTT), restricted application protocol (CAAP), and the like. Protocols and components for communicating via the internet or any other type of communication network as previously described are well known to those skilled in the art and are therefore not described in detail herein.

The server 130 may include a navigation unit 140, a vehicle data processing unit 145, and a data store 150. The navigation unit 140 may assist in location-based services. For example, the navigation unit 140 may assist a user (also referred to herein as a "driver") in transporting another user (also referred to herein as a "lift") and/or object (e.g., food, packages, etc.) from a first location (also referred to herein as an "pickup location") to a second location (also referred to herein as a "destination location"). The navigation unit 140 may assist in enabling user and/or object transport by providing maps and/or navigation instructions to applications running on the user device 102 of the lift, to applications running on the user device 102 of the driver, and/or to a navigation system running on the vehicle 120.

As an example, the navigation unit 140 may include a matching service (not shown) that pairs a lift requesting a trip from an pickup location to a destination location with a driver who is able to complete the trip. The matching service may interact with an application running on the lift's user device 102 and/or an application running on the driver's user device 102 to establish the lift's itinerary and/or to process payment from the lift to the driver.

The navigation unit 140 may also communicate with an application running on the driver's user device 102 during the trip to obtain trip location information from the user device 102 (e.g., via Global Positioning System (GPS) components coupled to and/or embedded in the user device 102) and provide navigation directions to the application that assist the driver in traveling from the current location to the destination location. The navigation unit 140 may also indicate a number of different geographical locations or points of interest to the driver, whether or not the driver is carrying a lift.

The vehicle data processing unit 145 may be configured to support driver assistance features of the vehicle 120 and/or to support autonomous driving. For example, the vehicle data processing unit 145 may generate and/or transmit map data to the vehicle 120, run a diagnostic model to identify operational issues with the vehicle 120, run a model to simulate performance of the vehicle 120 given a set of variables, use vehicle data provided by the vehicle 120 to identify objects and transmit the identification of the objects to the vehicle 120, generate and/or transmit vehicle 120 control instructions to the vehicle 120, and/or type operations.

The data storage 150 may store various types of data used by the navigation unit 140, the vehicle data processing unit 145, the user device 102, and/or the vehicle 120. For example, the data store 150 may store user data 152, map data 154, search data 156, and log data 158.

The user data 152 may include information about some or all users registered with the location-based service, such as drivers and lift riders. The information may include, for example, a username, password, name, address, billing information, data associated with previous trips taken or taken by the user, user rating information, user loyalty program information, and/or the like.

Map data 154 may include high-definition maps generated from sensors (e.g., light detection and ranging (LiDAR) sensors, radio detection and ranging (RADAR) sensors, infrared cameras, visible light cameras, stereo cameras, Inertial Measurement Units (IMU), etc.), satellite images, optical character identification (OCR) performed on captured street images (e.g., identifying street names, identifying street sign text, identifying point of interest names, etc.), and so forth; information for calculating a route; information for rendering a two-dimensional and/or three-dimensional graphical map; and/or the like. For example, the map data 154 may include a number of elements: such as the layout of streets and intersections, bridges (e.g., including information about the height and/or width of a overpass), exit ramps, buildings, parking structure entrances and exits (e.g., including information about the height and/or width of a vehicle entrance and/or exit), the location of signboards and stop lights, emergency crossings, points of interest (e.g., parks, restaurants, gas stations, attractions, landmarks, etc., and associated names), road markings (e.g., centerline markings separating opposing lanes, lane markings, stop lines, left turn guide lines, right turn guide lines, pedestrian crossings, bus lane markings, bicycle lane markings, safety island markings, road surface text, highway exit and entrance markings, etc.), curb, railway lines, waterways, turn radii and/or angles for left and right turns, distances and sizes of road features, road length, and size of road features, road length, and/or distance to a road intersection, and/or to a road intersection, a road or a vehicle, The location of the partition between two-way traffic and/or the like, along with the geographic location (e.g., geographic coordinates) to which these elements are associated. The map data 154 may also include reference data such as real-time and/or historical traffic information, current and/or predicted weather conditions, road work information, information regarding laws and regulations (e.g., speed limits, whether to allow or disallow a right turn at a red light, whether to allow or disallow a turn around, allowed travel directions, and/or the like), news events, and/or the like.

Although the map data 154 is shown as being stored in the data store 150 of the server 130, this is not meant to be limiting. For example, server 130 may transmit map data 154 to vehicle 120 for storage therein (e.g., in data store 129, as described below).

The search data 156 may include searches that were entered by a number of different users in the past. For example, the search data 156 may include a text search for access and/or destination locations. The search may be for a particular address, geographic location, name associated with the geographic location (e.g., name of park, restaurant, gas station, attraction, landmark, etc.), and so forth.

The log data 158 may include vehicle data provided by one or more vehicles 120. For example, the vehicle data may include route data, sensor data, perception data, vehicle 120 control data, vehicle 120 component failure and/or fault data, and the like.

FIG. 1B illustrates a block diagram showing the vehicle 120 of FIG. 1A communicating with one or more other vehicles 170A-N and/or the server 130 of FIG. 1A, according to one embodiment. As shown in fig. 1B, vehicle 120 may include various components and/or data storage. For example, the vehicle 120 may include a sensor array 121, a communication array 122, a data processing system 123, a communication system 124, an internal interface system 125, a vehicle control system 126, an operating system 127, a map engine 128, and/or a data store 129.

Communications 180 may be sent and/or received between vehicle 120, one or more vehicles 170A-N, and/or server 130. Server 130 may transmit and/or receive data from vehicle 120, as described above in connection with fig. 1A. For example, server 130 may transmit vehicle control instructions or commands to vehicle 120 (e.g., as communication 180). The vehicle control instructions may be received by a communication array 122 (e.g., an array of one or more antennas configured to transmit and/or receive wireless signals) operated by a communication system 124 (e.g., a transceiver). The communication system 124 may communicate vehicle control commands to a vehicle control system 126 that may operate an acceleration, steering, braking, lights, signals, and other operating systems 127 of the vehicle 120 to drive and/or steer the vehicle 120 and/or assist the driver in driving and/or steer the vehicle 120 through road traffic toward a destination location specified by the vehicle control commands.

As an example, the vehicle control instructions may include route data 163 that may be processed by the vehicle control system 126 to maneuver the vehicle 120 and/or assist a driver in maneuvering the vehicle 120 along a given route (e.g., an optimized route calculated by the server 130 and/or the map engine 128) toward a specified destination location. In processing the route data 163, the vehicle control system 126 may generate control commands 164 for execution by the operating system 127 (e.g., acceleration, steering, braking, steering, reversing, etc.) to cause the vehicle 120 to travel along the route to the destination location, and/or to assist the driver in steering the vehicle 120 along the route to the destination location.

Destination location 166 may be specified by server 130 based on a user request (e.g., an access request, a delivery request, etc.) transmitted by an application running on user device 102. Alternatively or additionally, the lift and/or driver of the vehicle 120 may provide the destination location 166 by providing user input 169 through the internal interface system 125 (e.g., a vehicle navigation system). In some embodiments, vehicle control system 126 may transmit the input destination location 166 and/or the current location of vehicle 120 (e.g., as a GPS data packet) as communication 180 to server 130 via communication system 124 and communication array 122. The server 130 (e.g., navigation unit 140) may perform an optimization operation using the current location of the vehicle 120 and/or the input destination location 166 to determine an optimal route for the vehicle 120 to travel to the destination location 166. Route data 163, including the optimal route, may be communicated from server 130 to vehicle control system 126 via communication array 122 and communication system 124. As a result of receiving the route data 163, the vehicle control system 126 can cause the operating system 127 to steer the vehicle 120 along the optimal route to the destination location 166 via traffic, assist the driver in steering the vehicle 120 along the optimal route to the destination location 166 via traffic, and/or cause the internal interface system 125 to display and/or present instructions for steering the vehicle 120 along the optimal route to the destination location 166 via traffic.

Alternatively or additionally, the route data 163 includes an optimal route, and the vehicle control system 126 automatically inputs the route data 163 into the map engine 128. The map engine 128 may generate map data 165 using the optimal route (e.g., generate a map to display the optimal route and/or take instructions for the optimal route) and provide the map data 165 to the internal interface system 125 (e.g., via the vehicle control system 126) for display. The map data 165 may include information derived from the map data 154 stored in the data store 150 on the server 130. The displayed map data 165 may indicate an estimated time of arrival and/or display the progress of the journey of the vehicle 120 along the optimal route. The displayed map data 165 may also include indicators such as diversion commands, emergency notifications, road work information, real-time traffic data, current weather conditions, information about laws and regulations (e.g., speed limits, whether or not to allow or prohibit a right turn at a red light, where to allow or prohibit a turn around, allowed directions of travel, etc.), news events, and/or the like.

User input 169 may also be a request to access a network (e.g., network 110). In response to such a request, the internal interface system 125 can generate an access request 168, which can be processed by the communication system 124 to configure the communication array 122 to transmit and/or receive data corresponding to user interaction with the internal interface system 125 and/or user device 102 interaction with the internal interface system 125 (e.g., user device 102 connected to the internal interface system 125 via a wireless connection). For example, the vehicle 120 may include an onboard Wi-Fi that passengers and/or drivers may access to send and/or receive email and/or text messages, audio streaming and/or video content, browse content pages (e.g., web pages, website pages, etc.), and/or access applications accessed using a network. Based on the user interaction, internal interface system 125 can receive content 167 via network 110, communication array 122, and/or communication system 124. Communication system 124 may dynamically manage network access to avoid or minimize disruption of the transmission of content 167.

The sensor array 121 may include any number of one or more types of sensors, such as a satellite radio navigation system (e.g., GPS), light detection and ranging sensors, landscape sensors (e.g., radio detection and ranging sensors), inertial measurement units, cameras (e.g., infrared cameras, visible light cameras, stereo cameras, etc.), Wi-Fi detection systems, cellular communication systems, inter-vehicle communication systems, road sensor communication systems, feature sensors, proximity sensors (e.g., infrared, electromagnetic, photoelectric, etc.), distance sensors, depth sensors, and/or the like. The satellite radio navigation system may calculate the current position of vehicle 120 (e.g., within a range of 1-10 meters) based on an analysis of signals received from a constellation of satellites.

Light detection and ranging sensors, radio detection and ranging sensors, and/or any other similar type of sensor may be used to detect the environment surrounding vehicle 120 when vehicle 120 is in motion or is about to begin moving. For example, light detection and ranging sensors may be used to reflect multiple laser beams from an approaching object to assess their distance and provide accurate three-dimensional information about the surrounding environment. Data obtained from the light detection and ranging sensors may be used to perform object identification, motion vector determination, collision prediction, and/or implement accident avoidance procedures. Alternatively, the light detection and ranging sensor may use a rotating scanning mirror assembly to provide a 360 ° viewing angle. The light detection and ranging sensors may optionally be mounted on the roof of the vehicle 120.

The inertial measurement unit may include an X, Y, Z-oriented gyroscope and/or accelerometer. The inertial measurement unit provides data about the rotational and linear motion of the vehicle 120, which can be used to calculate the motion and position of the vehicle 120.

The camera may be used to capture a visual image of the environment surrounding the vehicle 120. Depending on the configuration and number of cameras, the cameras may provide a 360 ° view around the vehicle 120. The images from the camera may be used to read road markings (e.g., lane markings), read street signs, detect objects, and/or the like.

A Wi-Fi detection system and/or a cellular communication system may be used to triangulate Wi-Fi hotspots or cell towers, respectively, to determine the location of the vehicle 120 (optionally in conjunction with a satellite radio navigation system).

The inter-vehicle communication system (which may include a Wi-Fi detection system, a cellular communication system, and/or the communication array 122) may be used to receive and/or transmit data to other vehicles 170A-N, such as current speed and/or position coordinates of the vehicle 120, time and/or position coordinates corresponding to a planned deceleration time and a planned deceleration rate, time and/or position coordinates when a stopping operation is planned, time and/or position coordinates when a lane change is planned and a lane change direction, time and/or position coordinates when a turning operation is planned, time and/or position coordinates when a stopping operation is planned, and/or the like.

A road sensor communication system (which may include a Wi-Fi detection system and/or a cellular communication system) may be used to read information from road sensors (e.g., indicating traffic speed and/or traffic congestion) and/or to read information from traffic control devices (e.g., traffic lights).

When a user requests pickup (e.g., through an application running on the user device 102), the user may specify a particular destination location. The initial position may be a current position of the vehicle 120, which may be determined using a satellite radio navigation system (e.g., GPS, Galileo, COMPASS, DORIS, GLONASS, and/or other satellite radio navigation systems) installed in the vehicle, a Wi-Fi positioning system, cellular tower triangulation, and/or the like. Alternatively, the initial position may be specified by the user through a user interface (e.g., internal interface system 125) provided by vehicle 120 or through user device 102 running the application. Alternatively, the initial location may be automatically determined from location information obtained from the user device 102. In addition to the initial location and the destination location, one or more waypoints may be specified, enabling multiple destination locations.

Raw sensor data 161 from sensor array 121 may be processed by an on-board data processing system 123. The processed data 162 may then be transmitted by the data processing system 123 to the vehicle control system 126 and optionally to the server 130 via the communication system 124 and the communication array 122.

Data store 129 may store map data (e.g., map data 154) and/or a subset of map data 154 (e.g., a portion of map data 154 corresponding to an approximate area in which vehicle 120 is currently located). In some embodiments, the vehicle 120 may record updated map data along the travel route using the sensor array 121 and transmit the updated map data to the server 130 via the communication system 124 and the communication array 122. The server 130 may then transmit the updated map data to one or more of the vehicles 170A-N and/or further process the updated map data.

The data processing system 123 may provide continuously or near continuously processed data 162 to the vehicle control system 126 in response to point-to-point activity in the environment surrounding the vehicle 120. The processed data 162 may include a comparison between raw sensor data 161, representing the operating environment of the vehicle 120 and continuously collected by the sensor array 121, and map data stored in the data store 129. In one example, the data processing system 123 is programmed with machine learning or other artificial intelligence capabilities to enable the vehicle 120 to identify and respond to conditions, events, and/or potential hazards. In a variation, the data processing system 123 may continuously or near continuously compare the raw sensor data 161 to stored map data to perform positioning to continuously or near continuously determine the position and/or orientation of the vehicle 120. The positioning of vehicle 120 may enable vehicle 120 to know the immediate location and/or direction of vehicle 120 as compared to stored map data to steer vehicle 120 through traffic on a block road and/or to assist a driver in steering vehicle 120 through traffic on a block road and to identify and respond to potentially dangerous (e.g., pedestrian) or local conditions, such as weather or traffic conditions.

Further still, positioning may enable vehicle 120 to tune or beam steer communication array 122 to maximize communication link quality and/or minimize interference from other communications of other vehicles 170A-N. For example, communication system 124 may beam steer the radiation pattern of communication array 122 in response to network configuration commands received from server 130. Data store 129 may store current network resource map data that identifies network base stations and/or other network sources that provide network connectivity. The network resource map data may indicate the location of the base stations and/or available network types (e.g., 3G, 4G, LTE, Wi-Fi, etc.) within the area in which the vehicle 120 is located.

Although fig. 1B describes certain operations as being performed by the vehicle 120 or the server 130, this is not meant to be limiting. The operations performed by the vehicle 120 and the server 130 as described herein may be performed by any entity. For example, certain operations typically performed by the server 130 (e.g., transmitting update map data to the vehicles 170A-N) may be performed by the vehicle 120 for load balancing purposes (e.g., reducing the processing load of the server 130, utilizing idle processing power on the vehicle 120, etc.).

Still further, any of the vehicles 170A-N may include some or all of the components of the vehicle 120 described herein. For example, vehicles 170A-N may include communication array 122 to communicate with vehicle 120 and/or server 130.

■Lane marking system based on machine learning

As previously described, the use of lane markings helps to enable or improve various functions or features performed by an onboard data processing system, such as data processing system 123. Lane markers may be included as part of and/or generated from a map or map image, such as those that may be stored in the map data store 154. These lane markings may be determined or generated using a machine learning model. The machine learning model may be generated based on training data including road images containing lane markers. Still further, the road images may be normalized by preprocessing the training data to improve and/or generate the machine learning model using the reduced training data. Normalizing the road image may include cropping the image to reduce the amount of background data or pixels that are not related to the road. Still further, normalizing the images may include rotating each image such that the traced road within the image is aligned with the target axis. The target axis may be a horizontal axis.

In some embodiments, the lane markers contained on the image may not be identifiable or available to certain processing units, or the amount of computing resources used to identify the lane markers is such that the lane cannot be identified in real-time. For example, a vehicle routing system, driver assistance system, or autonomous vehicle system may not be able to identify lane markings from an image, or may not be able to identify lane markings quickly enough to enable vehicle routing or autonomous driving. Also, the vehicle routing system, driver assistance system, or autonomous vehicle system may be configured to process data formats other than images when determining a route or when performing autonomous driving functions. However, the map drawing information is generally obtained from a map image. It is therefore desirable to identify lane markers within an image and modify or otherwise render (announce) the image in a manner that can be processed by a particular processing unit, such as a vehicle routing system, driver assistance system, or autonomous driving system. Embodiments of the system disclosed herein are capable of preprocessing an image to identify lane markers within the image. These systems can apply labels or markers to identify the location of marked lanes within an image to facilitate a variety of navigation and/or autopilot processes.

To simplify the discussion, many of the examples and numerous disclosures discussed herein describe aligning the depicted road with a horizontal axis. However, it should be understood that the target axis may be any axis with which the image of the road is aligned. For example, the target axis may be the longitudinal axis or a 30 °, 45 °, 195 °, or any other steering axis. Again, to simplify the discussion, much of the discussion herein relates to identifying lane markings on a roadway. However, embodiments disclosed herein may be applied to markers or other callouts that identify other elongated objects. For example, embodiments disclosed herein may be applied to railways, sidewalks, painted road stop lines, or let-down lines, or architectural layouts.

In some embodiments, lane markers may be applied to an image or map to create a sketched image or map that is sketched or otherwise modified or updated with lane marker or lane marker data. These sketched or modified images or maps may be used by a vehicle routing system, a driver assistance system, or an autonomous vehicle system. For example, the sketch map may be used by the navigation unit 140, the map engine 128, or the data processing system 123 to generate a navigation route or to assist in autonomous or semi-autonomous driving.

In some embodiments, an image or map containing lane markers may be generated by a map generation system. FIG. 2A illustrates a block diagram of a map generation system 200, according to some embodiments. The map generation system 200 may include any system capable of generating a map or image containing lane markers. Still further, the map generation system 200 may pre-process the images to reduce the amount of overhead and computational resources used to identify the lane markers within the images, and render the images containing the lane markers in a form that can be processed by a slave system, such as a vehicle routing system, driver assistance system, or autonomous vehicle system. In some embodiments, the map generation system 200 may include any type of computing device, such as desktop computers, laptops, servers, network or cloud computing devices, automotive console devices, computerized appliances, wearable devices (e.g., smart watches and glasses with computing functionality), and wireless mobile devices (e.g., smartphones, PDAs, tablets, etc.), among others.

The map generation system 200 may include a number of systems or subsystems that can facilitate pre-processing map images or road network images. Still further, these systems may help generate machine learning models, such as parametric or predictive models, that may be used to identify lanes within an image. The map generation system 200 may include a map image processing system 202, a model generation system 204, and a lane detection system 206. The map generation system 200 may include more or fewer systems. Still further, one or more systems of map generation system 200 may be implemented separately from map generation system 200.

The map image processing system 202 may include any system capable of receiving a map or image and capable of preprocessing the map or image to help identify lane markings within the map or image. In some embodiments, the map image processing system 202 may pre-process the map or image for use as training data for a model generation system to generate a machine learning model. The pre-processed map or image may comprise a portion of a larger map or image. For example, the map image processing system 202 may divide the map into segments to aid in processing. The segments may be equal or unequal in size. Still further, the segments may be determined based on content within the image segments, such as road portions in a road network.

The map image processing system 202 may pre-process the map or image by dividing the image into a plurality of segments, including straight or substantially straight portions of the road. Still further, the map image processing system 202 may identify roads within the image segment and may rotate the image segment to align the roads with a desired target axis. Advantageously, in certain embodiments, by aligning road segments with desired target axes, the amount of training data used to generate machine learning models, and thus the amount of computing resources (e.g., processor usage, memory usage, cache usage, network bandwidth usage, etc.), can be reduced. Moreover, the accuracy of the machine learning model can be improved.

The model generation system 204 may include any machine learning system capable of generating one or more machine learning models for identifying lane markings within an image. Further, model generation system 204 may obtain or use the pre-processed images as training data. Using preprocessed images can reduce the amount of training data required to generate the machine learning model while maintaining or increasing the accuracy of the machine learning model generated by model generation system 204.

Model generation system 204 may use one or more machine learning algorithms to generate one or more machine learning models, predictive models, or parametric functions. One or more of these machine learning models may be used to determine an expected value or expected occurrence based on a set of inputs. For example, a machine learning model may be used to determine a lane position or a probability that a portion of an image contains lane markers, or a particular type of lane marker (e.g., double line, solid line, dashed line, etc.) based on one or more inputs to a predictive model, such as training data containing pre-labeled or pre-labeled lane marker information. In some cases, the machine learning model may be referred to as a predictive model because, for example, its output may be a prediction or probability of, or may be related to, an identified location in an image or content for which an identified set of pixels includes a particular object (e.g., lane marker). The model generation system 204 can use a variety of different types of algorithms to help generate the machine learning model 260. For example, certain embodiments herein may use logistic regression models. However, other models are also possible, such as a linear regression model, a discrete choice model or a generalized linear model.

The machine learning algorithm may be configured to adaptively develop and update the model based on new inputs received by the model generation system 204. For example, the model may be generated repeatedly on a periodic basis, as new training data is available to help keep predictions in the model more accurate. Model generation system 204 is described in more detail herein. After the model is generated, it may be provided to the lane detection system 206 to assist in implementing automatic identification of lane markers in the additional images without user input or direction in the lane marker identification process.

Some non-limiting examples of machine learning algorithms that may be used to generate and update machine learning models, parametric functions, or predictive models may include supervised and unsupervised machine learning algorithms: including regression algorithms (e.g., conventional least squares regression), instance-based algorithms (e.g., learning vector quantization), decision tree algorithms (e.g., classification and regression trees), bayesian algorithms (e.g., na iotave bayes), clustering algorithms (e.g., K-means clustering), association rule learning algorithms (e.g., a priori algorithms), artificial neural network algorithms (e.g., a perceptron), deep learning algorithms (e.g., a deep boltzmann machine), dimension reduction algorithms (e.g., principal component analysis), integration algorithms (e.g., stack generalization), and/or other machine learning algorithms.

The lane detection system 206 may include any system capable of inputting or applying images to a machine learning model to automatically identify, or without user input, lane markings within the images. Still further, the lane detection system 206 may include any system capable of labeling or otherwise marking lane markings identified within an image. By labeling the lanes within the image, a routing or navigation system (e.g., a vehicle routing system, a driver assistance system, an autonomous vehicle system, or other processing system) can identify the lanes without separately processing the image to identify the lanes within the image.

In certain embodiments, the server 130 may include a mapping system 200. Still further, in some embodiments, the vehicle data processing unit 145 may include the map generation system 200 or functionality thereof. The map generation system 200 may be implemented in hardware, software, or a combination of hardware and software. Still further, the map generation system 200 may be a single computing system or a distributed computing system. In other embodiments, the mapping system 200 may be implemented by an application running on the user device 102 and/or another computing system separate from the server 130 or the user device 102.

■Example model Generation System

FIG. 2B illustrates an embodiment of the model generation system 204 of FIG. 2A. Model generation system 204 may be used to generate or determine one or more machine learning models 260 based on training data 252. The training data 252 may be historical data associated with previously processed images and/or may be generated training data. The training data may be generated by a user and/or by the map image processing system 202. Typically, although not necessarily, training data 252 includes data associated with a large number of pre-processed images, e.g., hundreds, thousands, hundreds of thousands, or more images. However, the present disclosure enables the use of preprocessing systems and processes on the graph to reduce the amount of training data, which can reduce the amount of training data 252 applied to the model generation system 204 by one or more orders of magnitude, while not reducing the accuracy of the subsequently generated machine learning model 260. Still further, the training data 252 may include data received from one or more data sources, such as images previously processed by the machine learning model and images manually processed by the user. Further, the training data 252 may include any data from different data sources, different data types, or images, and generated by one or more sensors (e.g., the sensor array 121 or a light detection and ranging system). In some cases, training data 252 may be accessed from a repository, such as data store 150.

In some cases, model generation system 204 may also receive feedback data 254. This receipt of data may be part of a supervised model generation process that enables a system user, such as an administrator, to provide additional input to model generation system 204 that may be used to assist in the generation of machine learning model 260. For example, if there are anomalies in the training data 252, the system user may label the anomalous data, thereby enabling the model generation system 204 to process the labeled data differently, such as applying different weights to the data or culling the data from the model generation process.

Further, model generation system 204 can receive control data 256. The control data 256 may identify one or more features or characteristics required by the model generation system 204 to determine the model. Still further, in some cases, the control data 256 may indicate values for one or more features identified in the control data 256. For example, assume that the control data 256 indicates that a machine learning model is to be generated using the training data 252 to determine marked lanes, and in some cases unmarked lanes, in images of a road network. The control data 256 may include manually marked lanes in the training data 252. The control data 256 may be provided as separate metadata or data layers or may be included as part of the training data 252.

The model generation system 204 may generally include a set of model generation rules 270 for generating the machine learning model 260. Rule set 270 may include one or more parameters 262. Each set of parameters 262 may be combined using one or more mathematical functions to obtain a parameter function, which may or may not be used to create the machine learning model 260. Again, one or more particular parameters may be weighted with a weight 264. In some cases, the parameter function may be obtained by combining a set of parameters with a corresponding set of weights 264. Based on specific input data during the training process: such as training data 252, feedback data 254 and control data 256 and may include defined output criteria therein for training purposes along with control data 256, from which respective parameters 262 of machine learning model 260 and/or machine learning model 260 are derived. The set of model generation rules 270 may define specific machine learning rules and/or algorithms that the model generation system 204 uses to generate a model based on a defined objective function, such as determining lane markings or other guidance markings (e.g., painted arrows on a road surface, stop or yield lines, or speed limits). In some embodiments, the initial parameters 262 and weights 264 may be provided manually during the start-up of the model generation process. Parameters 262 and weights 264 may be updated and modified during the model generation phase to generate predictive model 260.

The model generation system 204 may filter and categorize the training data set according to various features and parameters of the data. For example, the data may be categorized by data source (e.g., images from a user or images generated by other automated systems), information type (e.g., images containing lane markers, images without lane markers, images containing lane markers of a particular type, etc.), or other categories that may be associated with training data. Model generation system 204 can filter the information to identify information for subsequent processing. In some embodiments, model generation system 204 is configured to filter and separate training data 252 into multiple data types or categories, which are then subsequently processed. Further, in some cases, some of training data 252 may be filtered or removed from training data 252 based on the data being associated with a correlation determined by model generation system 204 to not satisfy a threshold correlation.

Optionally, one or more of the predictive models 260 may be associated with a penalty (penalty) 266. These penalty points 266 can be used to facilitate the generation or selection of a particular machine learning model 260 based on one or more factors used to derive the penalty points. For example, the number of mathematical complexities or parameters included in a particular machine learning model 260 may be used to generate a penalty for the particular machine learning model 260, which may affect the generation of the model and/or selection algorithm or the probability of enrollment at which the particular machine learning model 260 is selected. After generating the machine learning model 260, the model may be used during operation of the lane detection system 206 to determine lane markings in the image.

■Example road extraction and normalization Process

Fig. 3 presents a flow diagram of an example road extraction and normalization process 300, in accordance with certain embodiments. Process 300 may be implemented by any system capable of preprocessing an image by identifying a road within the image, and normalizing the image portion containing the road by rotating the portion of the image of the road to align with the identified target axis. The process 300 may be wholly or partially performed by, for example, the vehicle data processing unit 145, the map generation system 200, a computer system, or the like,

Map image processing system 202, model generation system 204, lane detection system 206, or the like. Although any number of systems may implement process 300 in whole or in part, process 300 will be described in connection with a particular system for simplicity of discussion.

In certain embodiments, process 300 may be used as part of a preprocessing method for generating training data for model generation system 204. Alternatively or additionally, the process 300 may be used to pre-process images to help edit maps or images with lane markers, or lane marker labels, or other desired features to be annotated within the images.

The process 300 begins at block 302, for example, where the map image processing system 202 receives a high definition image of a road network map. The resolution of the high definition image may be 10 cm. For example, the high definition image may be a satellite image having a resolution of 10cm per pixel. The resolution of the high definition image may vary. However, generally a high definition image is an image having a resolution of at least 1280 × 720 pixels. Although the embodiments described herein are described in connection with an image having a resolution that is up-to-standard as a high definition image, in certain embodiments, the embodiments disclosed herein may also be used in connection with a lower resolution image. In some embodiments, the image of the map may have a lower or higher resolution than the high-definition image. Still further, in certain embodiments, the high-definition images may be generated by sensors, such as light detection and ranging sensors, radio detection and ranging sensors, infrared cameras, visible light cameras, stereo cameras, inertial measurement units, and the like. Alternatively or additionally, a high-definition image or map may be generated from satellite images or images generated by other imaging systems.

Although the high-definition image received at block 302 depicts a road network, in some embodiments, the image may depict other features visible near the road network. For example, the high-definition images may depict flora (e.g., grass, shrubs, trees, etc.), fauna (e.g., dogs, cats, birds, deer, etc.), buildings or other structures, signs, sound barriers, central road isolation structures, and so forth. Further, as previously noted, the embodiments disclosed herein may be applied to structures or networks other than highway networks, and may include any type of relatively elongated structure that may contain markings or other features within an image that require markings within the image. As such, the image may depict other networks, such as a railway network, a waterway, a walking path (e.g., a sidewalk or a mountain lane), or a building layout, among others.

Optionally, the map image processing system 202 converts the high definition image to a binary image. Converting a high definition image to a binary image may include converting the image to a black and white image (also sometimes referred to as a monochrome or monochrome image) or other bi-color image. In some embodiments, the binary image may comprise a grayscale image with pixels limited to one sample per pixel. Since binary images typically contain only two possible values per pixel, in some cases, a certain number of features depicted within the image may be erased or no longer depicted. Also, in some cases, some features depicted within the image may still be depicted, but may not be readily discernable as compared to the original high-definition image. In some embodiments, converting a high-definition image to a binary image may cause the road to be emphasized within the image compared to other features. In some embodiments, converting the image to a binary image can help obtain an estimate of the shape of the road or road network contained in the image, since, for example, most non-road pixels may be empty or associated with a value or color designated as empty or no content image data. For example, empty pixels may occur because non-road areas may not be scanned by the sensor used to generate the image.

In some embodiments, the processes disclosed herein may be performed using grayscale images instead of binary images or both. Thus, in some embodiments, a high definition image may be converted to a grayscale image. In other embodiments, the high definition image is generated or received as a grayscale image. In such embodiments, the grayscale conversion process may be omitted. Since some or all of the features described herein can be performed on grayscale images, in some embodiments, the step of converting a high definition image to a binary image may be omitted or optional. Fig. 8A shows an example of a grayscale image. As can be seen in fig. 8A, a road 802 and some flora 804 bordering the road 802 are visible within the image. Also, portions of the building 806 may also be visible within the image. However, most buildings and other additional features within the image are weakened or not depicted in the grayscale image. In some embodiments, colored pixels in the high definition image may be converted to white (or one color) and the remaining pixels may be converted to black or another color. In other embodiments, pixels associated with a particular luminance or set of colors may be converted to one color and the remaining pixels may be converted to another color selected in the binary image. In yet another embodiment, colored pixels can be converted to various forms of gray on a gray palette.

At block 306, the map image processing system 202 identifies roads within the road network from or using the high definition image or a grayscale or binary image obtained from the high definition image. The map image processing system 202 may use contrast and gradient information obtained from the image to identify roads in the image. In some embodiments, the map image processing system 202 may select several points as keypoints along the road contour. The contour may be determined based on one of a plurality of colors in the image. For example, in the case of processing a grayscale or binary image, the outline may be rendered or identified as a surrounding pixel of white.

Still further, the map image processing system 202 may identify inflection points within the image that have a greater gradient (e.g., a color gradient greater than a threshold difference within the image, such as greater than 50, 100, etc.). These points or portions with greater gradients may be determined as starting and/or ending points of the road depicted within the image. Accordingly, roads may be identified or otherwise extracted from the image based on the starting and ending points of the roads within the image. In some embodiments, multiple roads within an image may be identified or depicted.

At block 308, the map image processing system 202 divides the image of the road into a plurality of segments or image segments. Dividing the image into a plurality of segments may include dividing the high definition image into a plurality of segments. In some embodiments, the divided image is a binary or grayscale image. In other embodiments, the map image processing system 202 uses a binary or grayscale image as a reference to identify segments to divide a high definition image. In other words, the map image processing system 202 may identify segments using binary or grayscale images and then divide the high definition image based on the identified segments. The image segments may all be the same size. However, typically the image segments are of different sizes. In some embodiments, the image may be divided into a plurality of segments based on user input. In other embodiments, the image may be divided without user input. In some implementations, the image can be automatically divided into segments using, for example, a machine learning algorithm. Alternatively or additionally, the image may be divided into a plurality of segments using rules that distinguish different portions of the road based at least in part on the image identification data. In some cases, different segments of an image may be determined based at least in part on the identification of intersections or other features that intersect the road. Still further, different segments of the image may be determined based at least in part on a difference comparison between a portion of the image that includes a road and a portion of the image that does not include the road.

The map image processing system 202 may determine to divide the plurality of segments of the image based on the curvature of the road. The straighter the road, the longer the selected image segment. The more curved the image road, the smaller the image segment. The size of the image segment may be selected to reduce the curvature of the road depicted within the image segment. Thus, each image segment may contain a portion of the road with a degree of curvature no greater than a threshold degree or curvature. Fig. 6 shows an example image of a road network 600. It should be understood that the road network 600 may be part of a larger road network. Again, there may be additional intersections in the road network, which are omitted from fig. 6 for ease of illustration. For example, some or all of the gaps between

segments

602, 604, and 606 may omit intersections.

As shown, the image may be divided into a plurality of segments including road portions within a road network. The segment associated with a straight portion of a road (e.g., segment 602) may be longer than the segment associated with a curved road portion (e.g., segment 604), thereby reducing curvature within a particular segment. In some embodiments, the segment 604 containing a curved road may be broken down into smaller segments to further reduce the magnitude of the road curvature within one segment. In some embodiments, a segment may be associated with or limited to a maximum size. For example, although the curvatures of the roads included in the sequential segments 602 and 606 are relatively equal, the portion of the road is divided into two segments 602 and 606. In some embodiments, by placing an upper limit on the size of the segment, the machine learning algorithm may be simplified and/or the computational resources required to perform lane identification and labeling may be reduced. In some embodiments, segments 602 and 606 may be combined into one segment.

In certain embodiments, the segments may be split based on identifying intersections, crosswalks, waterways, speed bumps, railroad tracks, or other features that may traverse the roadway. In some embodiments, the segments are continuous. In other embodiments, as shown in FIG. 6, gaps may be left between the segments. The gaps may identify the location of cross streets, pedestrian crossings, gutters, speed bumps, railroad tracks, or other features that may cross the roadway. Alternatively, there may be gaps to facilitate processing and recombination of multiple individual road segments.

At block 310, for each image segment or at least some image segments, map image processing system 202 crops portions of the image segment along the road. Cropping the portions of the image segment along the road may include removing at least some portions of the image segment that do not include a road segment. Further, cropping the portion of the image segment that does not include the road segment may include omitting the portion of the image that is at least a certain number of pixels away from the identified road within the road segment. In some embodiments, image segments are cropped such that adjacent image segments retain at least some overlapping information or pixels. Advantageously, in some embodiments, by maintaining overlapping buffers, the map image processing system 202 can ensure that information related to roads is not lost during the cropping process. In some embodiments, some image segments may be cropped, while other image segments may not be cropped. Again, the amount of cropping each image segment may vary. For example, an image segment containing more road portions may be cropped less than an image segment containing more background material other than roads. Further, whether or not an image segment is cropped or the amount of image segment cropped may depend on the confidence level that no road portion is lost during the cropping process. For example, if the map image processing system 202 determines that the portion cropped from the image segment does not contain a road with a certainty level of 95% or greater (or any other threshold certainty level), the map image processing system 202 may crop the image segment. On the other hand, if the certainty confidence that the portion of the map image processing system 202 having an image segment does not contain a road is less than the threshold, the map image processing system 202 may omit the image segment from the cropping process or do not crop. In some embodiments, block 310 may be optional or may be omitted.

At block 312, the map image processing system 202 determines a target axis. The target axis may be a horizontal axis, a vertical axis, or any other axis. In some embodiments, the target axis is selected from within a two-dimensional plane. The target axis may be specified by the user, or may be automatically selected using one or more algorithms. For example, the target axis may be selected based on the number or percentage of image segments that are aligned with the target axis or that are within a threshold number of degrees of rotation relative to the target axis.

At block 314, the map image processing system 202 rotates each image segment to align the road segment depicted in the image segment with the target axis. For example, the image segment 602 may be rotated such that the road contained within the image segment 602 becomes horizontal, vertical, or aligned with any other selected target axis. Advantageously, in certain embodiments, rotating each image segment to align with a target axis reduces the amount of training data required to train a machine learning algorithm or generate a machine learning model, while maintaining or in some cases improving the accuracy of the machine learning model. In some embodiments, the rotated images may be referred to as normalized images, where the road segments depicted in each rotated image are aligned with the same target axis. In some embodiments, multiple target axes may be used. In some such cases, at least some of the image segments may be at least replicated and rotated to align with different target axes.

At block 316, the map image processing system 202 stores the rotated image segments and degrees of rotation at a repository. The repository may be a repository of training data used by a machine learning process to generate a machine learning model or a parametric model. In some cases, the repository may be a data store 150. For example, the rotated image segments and degrees of rotation may be stored as part of the map data 154. In some embodiments, block 316 may be optional or omitted. For example, the rotated image segment and the degree of rotation of the image segment may be provided to model generation system 204 without storing this data at a repository.

If the rotated image segment is used as part of the training data, the process 300 further includes adding lane markers or labels to the rotated image segment to assist in generating the machine learning or parametric model. The machine learning model generation process is described in further detail below in connection with process 400. On the other hand, if the rotated image segment is used as an input to a machine learning model or parametric model to identify lane markers to be labeled, the rotated image segment may be provided to the machine learning model without adding lane markers.

■Example machine learning model Generation Process

Fig. 4 presents a flow diagram of an example machine learning model generation process 400, in accordance with certain embodiments. The process 400 may be implemented by any system capable of generating a machine learning model using refined training data using preprocessed images. The process 400 may be implemented in whole or in part by, for example, the vehicle data processing unit 145, the map generation system 200, the map image processing system 202, the model generation system 204, or the lane detection system 206, among others. Although any number of systems may implement process 400 in whole or in part, process 400 will be described in connection with a particular system for simplicity of discussion.

Process 400 begins at block 402, for example, where model generation system 204 receives a training data set that includes normalized images of road segments. These normalized images of road segments may include graphics or image segments of the image or image segments that are rotated to align the road depicted within the image segment with a particular target axis. For example, the normalized images may include images that are each rotated to align the depicted road segment with a horizontal axis. In some embodiments, the training data set may be received as output from process 300. Still further, in some cases, the training data set may be obtained from a repository, such as repository 150.

At block 404, a lane marker overlay for the normalized image corresponding to the lane marker of the road depicted in the normalized image may be generated. In some embodiments, the lane marker overlay may be generated by a user or generated at least in part by user input. For example, the user may identify multiple locations within each standardized image that includes or depicts lane markers. Based on the location of the lane markers within the user-identified image, a lane marker overlay may be created to identify the location of the lane markers to the model generation system 204. In some embodiments, a user may generate a lane marker overlay using a computing system, such as the map generation system 200. The lane markings provided at block 404 may be used as or included in control data of the model generation system 204 to assist in generating the machine learning model.

At block 406, the model generation system 204 performs a data enhancement process on the normalized image containing the lane marker overlay. The data enhancement process may include creating one or more additional sets of images by rotating copies of at least some of the normalized images. The rotated copy of the normalized image may be a rotational random quantity, a pseudo random quantity or a specific fixed quantity. Further, each rotated copy of the normalized image may be rotated by the same amount or a different amount. Advantageously, in certain embodiments, the data enhancement process reduces or eliminates the over-fitting problem when generating the machine learning model.

The overfitting problem may occur when the generated machine learning model is too accurate for or builds too well into the training data. When a machine learning model is too accurate for training data or models training data too well, the machine learning model may not be universal enough to process subsequently provided input data. For example, the machine learning model may learn noise or random fluctuations in the training data as the concept to be modeled. An overly accurate machine learning model can be problematic because the new data may not contain some noise or random fluctuations, and the results may be less accurate than the machine learning model or the new data may not be able to be processed in some cases.

One solution to the overfitting problem is to provide additional training data. However, as previously described, the greater the amount of training data, the greater the load on computing resources, and in some cases may require human resources to prepare or manually process the training data before it is provided to the model generation system (e.g., manually labeling lane markers, for example). Advantageously, in certain embodiments, the use of a data enhancement process enables model generation system 204 to reduce or eliminate overfitting problems using a reduced training data set as described herein.

At block 408, the model generation system 204 generates one or more machine learning models based on the training data set, the output of the data enhancement process, and the lane marker overlay. The one or more machine learning models may be parametric or predictive models. When input data, such as road images, is provided to one or more machine learning models, the one or more machine learning models may output a sketch image identifying road and/or location information for where road markings are present within the image. Also, in some embodiments, the one or more machine learning models may output the type of lane marking. For example, the machine learning models may output lane markers as solid lines, dashed lines, double lines, yellow lines, white lines, arrows, and so forth. In some cases, the output of these machine learning models may contain a likelihood associated with some or all of the points (e.g., a pixel or a group of pixels) within the image that indicates a likelihood that the point includes a lane marker. Based at least in part on the likelihood that a particular point contains a lane marker, the lane detection system 206 may label or otherwise mark a corresponding location on the image to indicate the location of the lane marker and/or the type of lane marker. In some embodiments, the images themselves may not be marked with labels or lane markings. In some such embodiments, data indicative of the location and/or type of lane markings associated with the images may be stored to the vehicle data processing unit 145, the navigation unit 140, the map engine 128, the vehicle control system 126, the data processing system 123, or any other path navigation or autopilot system.

In some embodiments, a weight or penalty may be associated with each machine learning or parametric model generated by model generation system 204. The weight or penalty associated with each machine learning model may be based at least in part on the accuracy of the machine learning model, the complexity of the machine learning model, the relative amount of computing resources required to execute the machine learning model, or other factors that may affect the selection or execution of the machine learning model. Based at least in part on the weights or penalties associated with each machine learning model, the model generation system 204 may select one or a subset of the generated machine learning models to store or use for lane marking processing, such as the process 500 described below.

■Example automatic Lane marking Process

Fig. 5 presents a flow diagram of an example automatic lane marking process 500, in accordance with certain embodiments. Process 500 may be implemented by any system capable of automatically marking or annotating a lane within an image without user input. The process 500 may be implemented in whole or in part by, for example, the vehicle data processing unit 145, the map generation system 200, the map image processing system 202, the model generation system 204, or the lane detection system 206, among others. Although any number of systems may implement process 500 in whole or in part, process 500 will be described in connection with a particular system for simplicity of discussion.

The process 500 begins at block 502, for example, where the lane detection system 206 accesses one or more rotated image segments of a road portion depicted in a high-definition map. In certain embodiments, the rotated image segments may be accessed from a storage repository, such as data storage 150. Also, in some embodiments, the rotated image segments may be obtained using the process 300 previously described above. In certain embodiments, block 502 may include one or more embodiments previously described in connection with block 402.

At block 504, the lane detection system 206 applies the rotated image segment as input to a machine learning model to obtain a rotated image segment of the lane marker. The rotated image segments of the lane markings may include rotated image segments containing labels that identify where the lane markings are located within the image. These labels may be added to the rotated image segment or may be part of an overlay associated with the rotated image segment.

Alternatively or additionally, block 504 may include obtaining data indicative of a location of a lane marker within the rotated image segment. For example, as described above, the machine learning model may output a likelihood that each pixel or a group of pixels within an image segment contains a lane marker or a portion of a lane marker. In some cases, the probability or confidence that a group of pixels contains a lane marker may be increased if a threshold degree or probability of inclusion of lane markers by a surrounding plurality of groups of pixels is determined. For example, if it is determined that a particular set of pixels may contain lane markings, there is an increased likelihood that neighboring sets of pixels may contain lane markings, as the lane markings may extend over multiple sets of pixels. Also, as previously described, in certain embodiments, the use of rotated image segments enables the machine learning model to more accurately identify lane markers using reduced computational resources.

At block 506, the lane detection system 206 determines the degree of rotation of each rotated image segment relative to the target axis. The degree of rotation may be determined from data stored in a repository containing rotated image segments. Alternatively, the degree of rotation may be metadata inserted into the rotated image segment or metadata in a media file including the rotated image segment. In some embodiments, the rotated image segments may be recast with the degree of rotation that the original segment was rotated to obtain the image segment. In such cases, the lane detection system 206 may determine the degree of rotation from the callout.

At block 508, the lane detection system 206 counter-rotates the rotated image segments of each lane marker by the degree of rotation determined for the corresponding rotated image segment. In some embodiments, counter-rotating the rotated image segment comprises: the rotated image segments of the rotated lane markers are obtained in opposite directions and by the same number of degrees as the image segments are rotated. Once the rotated image segment is rotated in the reverse direction, the resulting image segment should match or substantially match the original image segment from which the rotated image segment was derived. Thus, in some embodiments, the road portion depicted in the resulting image segment may be matched with the original alignment of the road portion in the map originally received, for example, at block 302.

At block 510, the lane detection system 206 applies lane markings from the image segments that are counter-rotated and mark the lane to corresponding segments within the high-definition map. Applying lane markings to corresponding segments within a high definition map may include: the corresponding portion of the high definition map is replaced with the image segment that is counter-rotated and marks the lane. Alternatively, applying lane markers to corresponding segments in a high definition map may comprise: the high-definition map is marked or mapped with lane markers and/or labels based on the position of the labels in the image section that is rotated in the opposite direction and marks the lane. In some embodiments, block 510 may include storing the data or metadata with a high definition map that identifies the location of lane markers determined based at least in part on the image segments that counter-rotate and mark the lane. .

At block 512, the lane detection system 206 stores the modified map in a repository, such as the data store 150 and/or the data store 129. The modified map may be included in map data 154 of data store 150. The modified map may be processed by the vehicle or navigation processing system: such as vehicle routing systems, driver assistance systems, or autonomous vehicle systems, to help identify lanes within a route, change from one lane to another, and/or hold a vehicle within a lane.

■Example image

6-10 illustrate several example images that include

processes

300, 400, and 500 previously described, according to some embodiments described herein. As previously described, FIG. 6 illustrates an example image of a road network having a segmented sketch in accordance with some embodiments. This road network may be part of a larger road network. Again, the image segments identified by the white bounding box may be identified using any of the processes described herein, including process 300 and the operations associated with block 308. As shown, for example, by segments 602 and 608, some segments may overlap. Also, as previously described, the image portion depicting a curved road may be broken down into smaller segments to reduce the curvature of the depicted road. For ease of illustration, the curved segment 604 is relatively large. It should be appreciated that the segment 604 may be divided into smaller segments to reduce the magnitude of the road curvature shown in any one segment.

Fig. 7 shows an enlarged portion of the example image of fig. 6. As more clearly seen in fig. 7, the segments may be separated at certain portions to reduce the curvature of the roadway depicted in a single segment, as shown by gap 702. Again, as shown by the gap 704, the segments may be divided by the presence of an intersection or crosswalk.

FIGS. 8A and 8B illustrate examples of image segments pre-processed to identify lane markers within the image shown in FIG. 8C, according to some embodiments. Fig. 8A shows an example of an image segment 800 that may be obtained from an image of, for example, a road network 600. Although the image of the road network 600 does not include any roads aligned with the horizontal axis, the image segment 800 has been rotated to be horizontal as part of, for example, the process 300. The image segment 800 may be rotated using one or more of the operations described in connection with block 314.

Further, the image segment 800 may be cropped to reduce the amount of background pixels contained in the image segment. As previously described, the gradient and/or contrast information may be used to automatically crop a binary or grayscale image of an image or road and focus on the road within the image foreground. Cropping typically occurs prior to rotation to more accurately identify and align the road with the target axis. However, in some embodiments, the rotation may occur first or substantially in parallel with the cropping process. As previously described, the image segment 800 may depict a road 802 along with boundary objects, such as a vegetation group 804 and a building 806. In some alternative embodiments, map image processing system 202 weakens additional objects through a binary or grayscale image conversion process and/or a cropping process.

Fig. 8B illustrates an image segment 810. The image segment 810 corresponds to the image segment 800, but includes a lane marker or label 812 that marks or identifies a lane marker contained in the image segment from the original image. The user may interact with the map generation system 200 to sketch or add lane markers 812 to the image segments to help prepare training data for the model generation system 204. Lane markings or labels may be applied using one or more of the operations described in connection with block 404. The image segments 810 containing the lane markers may be provided to the model generation system 204 as training data.

FIG. 8C shows an image segment 820 generated using embodiments disclosed herein. Similar to image segment 810, image segment 820 corresponds to image segment 800. However, the image segment 820 is processed by a machine learning model generated by the model generation system 204. The image segment 820 includes a mask or overlay of lane markings that identify the roadway 802. As shown in fig. 8C, the mask or overlay with lane markings or labels generated by the machine learning model using embodiments disclosed herein is substantially more accurate than the original image segment 800. The lane markings include solid lane markings 822 and dashed line markings 824. In some embodiments, the lane markings or labels may have different intensities or brightnesses, confidence levels or probabilities that lane markings are present corresponding to locations identified by the machine learning model. This confidence level is shown by the mask or lane marker overlay depicted in fig. 8C, specifically illustrated by the darker lane marker 822 and the lighter lane marker 826. The lane markers 822 and 824 may be used by the lane detection system 206, for example, where the process 500 is applied to an image segment or an image containing an image segment. For example, the operation of block 510 may be used to apply the mask from FIG. 8C to the image segment 800 in FIG. 8A.

Fig. 9A and 9B illustrate additional examples of pre-processed image segments depicting road segments, according to some embodiments. Fig. 9A shows an image segment 900 of a road section. As with image segment 800, image segment 900 has been rotated from its original axis to align with the horizontal axis as part of pre-processing the image for use as training data to generate or for provision to a generated machine learning algorithm to identify lane markings.

Fig. 9B illustrates an image segment 910 of a road portion. The roads depicted in image segment 910 correspond to the roads depicted in image segment 900. However, the image segment 910 includes a mask or overlay that identifies or otherwise marks lane markings on the roadway. The mask is used by a computing system to mask lane markings in a description of a road with labels, such as the mapping system 200. As with the image segment 810, the user may interact with the map generation system 200 to create a mask or overlay. Map generation system 200 may provide image segments 810 to model generation system 204 as training data.

Fig. 9C illustrates a portion of an image segment corresponding to the image segment of fig. 9A and 9B after application of the lane markers determined by the machine learning model to the image, according to embodiments disclosed herein. The lane detection system 206 generates lane markings or labels 922 for the image segment 920 using a machine learning model. As can be seen in the enlarged portion 924 in fig. 9C, the generated label approximately coincides with the actual lane marker contained in the original image. For ease of illustration, not all lane markings are labeled in fig. 9C. However, it should be understood that each element of the lane markings 922 that match the markings is included as part of the lane markings 922.

Fig. 9D shows the image segment shown in fig. 9C, but with a bounding box marker 932 corresponding to the lane marker. The lane detection system 206 may add the boundary box marker 932 to the lane marker, for example, as part of the operations associated with the box 510. Advantageously, in some embodiments, the bounding box tags may help enable the autonomous vehicle to identify the lane in which the vehicle is to remain. In some embodiments, the bounding box markers may be used to help enable the map generation system 200 or the lane detection system 206 to identify lane markers. The lane detection system 206, using, for example, a machine learning model, may represent the location of the bounding box based on the probability of the presence of a lane marker in the image, crossing, or touching a point, pixel, or group of pixels in the image. A bounding box may be placed at this point or around a threshold number of pixels centered at this point. For example, using a polyline process, lane markers may be identified by connecting at least some of the bounding boxes. Determining whether to connect a set of bounding boxes to form a line associated with a lane marker may be based at least in part on probabilities output by one or more machine learning models. For ease of illustration, not all bounding box labels are labeled in FIG. 9D. However, it should be understood that each element of the bounding box label 932 of the match flag is included as part of the bounding box label 932.

Fig. 10 shows an image comparison 1000 of the lane marking process. Image comparison 1000 includes three

columns

1002, 1004, and 1006. Column 1002 represents a variety of image segments of a road that have been divided into segments in order to minimize the amount of curvature depicted in the image segments, and that have been rotated to align each road segment with the horizontal axis.

Column 1004 corresponds to column 1002, but the illustrated lane markings are determined by a machine learning system that does not implement the features disclosed herein. As can be seen from the lane markings in column 1004, it is difficult for existing systems to distinguish between different types of lane markings. For example, the outputs included in column 1004 do not distinguish between solid and dashed line markers. Further, as shown at block 1008, some existing systems are unable to process or identify lane markings when the lane has been divided or split. Additionally, existing systems do not predict lanes where no lane markers exist, as shown in block 1010.

Column 1006 corresponds to

columns

1002 and 1004, but the illustrated lane markings are determined by the lane detection system 206 implementing embodiments disclosed herein. Comparing column 1006 to column 1004 may determine that lane detection system 206 includes several improvements over existing systems. For example, the lane detection system 206 can distinguish between different types of lane markings, such as dashed lines and solid lines. Again, as indicated by the absence of a blank in column 1004 corresponding to block 1008, the lane detection system 206 is able to process lane divisions and includes a marker or tag that identifies or corresponds to a lane diversion. For example, as can be seen in the top road segment of column 1002, one of these lanes branches to allow a separate turn lane. No lane diversion is captured in column 1004 corresponding to the existing system. However, lane diversion is shown in column 1006, which implements the embodiments described herein.

Further, as shown in column 1006, the lane detection system 206 is able to predict the continuation of the lane when there is no lane marker. While some prior systems are confused by the traversed railway track 1012 and the missing lane markers (as indicated by the blank boxes 1010), the lane detection system 206 implementing embodiments disclosed herein is able to determine the continuity of the lane without lane markers (as indicated by the markers in the boxes 1014 of the column 1006 corresponding to the blank boxes 1010 of the column 1004) and to distinguish lane markers from image noise, such as the railway track 1012.

Still further, while the comparison system used to generate the lane markings in column 1004 does not identify additional street markings, as shown in column 1006, the lane detection system 206 is able to detect additional street markings, such as turn arrows and follow-up arrows. In the third set of exemplary drawings in fig. 10, the ability to detect the marked arrow in the road image is shown.

Thus, in certain embodiments, the map generation system 200, which uses less training data than existing systems, is able to generate more accurate lane markings than existing systems, as may be determined by the image comparison 1000. Further, in certain embodiments, the pre-processing of the images can reduce computational resources used to train the machine learning system and to execute the machine learning model.

Other embodiments are possible within the scope of the invention, such that the above-described components, steps, blocks, operations, and/or messages/requests/queries/instructions are arranged, ordered, subdivided, organized, and/or combined differently with respect to the figures described herein. In some embodiments, different components may initiate or perform a given operation. For example, in some embodiments, map image processing system 202 may be divided into several systems, including a system for image segmentation, a system for image rotation, a system for noise filtering, and a system for binary and/or grayscale image generation. As another example, in some embodiments, the rotation and cropping of the image segment may first perform either process sequentially, or each process may be performed at least partially in parallel.

■Example embodiments

Some exemplary enumerated embodiments of the present invention are cited in this paragraph section in the form of a method, system, and non-transitory computer-readable medium, and not a limitation.

One aspect of the present disclosure sets forth a computer-implemented method of performing an automatic lane marking process on an image of a road network using a machine learning model. The computer-implemented method may be implemented by a lane detection system comprising one or more hardware processors and configured with certain computer-executable instructions, and further, the computer-implemented method may comprise: accessing a rotated image segment depicting a road segment, wherein the rotated image segment comprises a rotated portion of a road network image, and wherein the rotated image segment is rotated to align the road segment with a target axis; applying the rotated image segments to a machine learning model to obtain rotated image segments that mark lanes; determining a degree of rotation of the rotated image segment relative to a target axis; reversely rotating the rotated image segment of the marking lane according to the rotation degree to obtain an image segment of the marking lane; and updating the image of the road network based at least in part on the image segment of the marked lane to obtain an updated image.

The method of the preceding paragraph may include any subcombination of the following features: wherein lane markings on the rotated image segment of the marker lane or on the image segment of the marker lane are included as part of an overlay that provides lane marking data for one or more of the following systems: a vehicle routing system, a driver assistance system, or an autonomous automotive system; wherein the degree of rotation comprises an amount by which a portion of the image of the road network is rotated to obtain the rotated image segment; wherein the degree of rotation is determined from metadata associated with the rotated image segment; wherein the image of the road network comprises a map of the road network; wherein updating the image of the road network comprises replacing a corresponding portion of the image with an image segment of the marked lane; wherein updating the image of the road network comprises applying lane markers from image segments of the marked lane to corresponding portions of the image; wherein the method further comprises storing the updated map in a map repository configured to store map data for one or more of: a vehicle routing system, a driver assistance system, or an autonomous vehicle system; and wherein the target axis comprises a horizontal axis.

Another aspect of the present disclosure is directed to a system for performing an automatic lane marking process on an image of a road network using a machine learning model. The system may include a storage device configured to store rotated image segments depicting portions of a roadway and rotated to be aligned with a target axis; and a lane detection system including one or more hardware processors. Still further, the lane detection system may be configured to: accessing, from the storage device, a rotated image segment depicting a road segment, wherein the rotated image segment is a rotated portion of a road network image, and wherein the rotated image segment is rotated to align the road segment with the target axis; inputting the rotated image segments into a machine learning model to obtain rotated image segments that mark lanes; determining a degree of rotation of the rotated image segment relative to a target axis; reversely rotating the rotated image segment of the marking lane according to the rotation degree to obtain an image segment of the marking lane; and modifying an image of the road network based at least in part on the image segment of the marked lane to obtain a modified image.

The system of the previous paragraph may include any subcombination of the following features: wherein the lane detection system is further configured to generate an overlay comprising lane markings from the rotated image segment of the marked lane or the image segment of the marked lane, and wherein the overlay is provided to one or more of: a vehicle routing system, a driver assistance system, or an autonomous vehicle system; wherein the degree of rotation comprises an amount by which a portion of the image of the road network is rotated to obtain the rotated image segment; wherein the lane detection system is further configured to modify the image of the road network by replacing image segments of the marked lanes with corresponding portions of the image; wherein the lane detection system is further configured to modify the image of the road network by applying lane markers from image segments of the marked lane to corresponding portions of the image; wherein the lane detection system is further configured to store the modified map in a map store configured to store map data for one or more of: a vehicle routing system, a driver assistance system, or an autonomous vehicle system; and wherein the storage device comprises a map repository.

Yet another aspect of the disclosure is directed to a non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more computing devices, configure the one or more computing devices to perform a set of operations. These operations may include: accessing a rotated image segment depicting a road segment, wherein the rotated image segment is a rotated portion of a road network image, and wherein the rotated image segment is rotated to align the road segment with a target axis; applying the rotated image segments as input to a machine learning model to obtain rotated image segments that mark lanes; determining a degree of rotation of the rotated image segment relative to a target axis; reversely rotating the rotated image segment of the marking lane according to the rotation degree to obtain an image segment of the marking lane; and annotating the image of the road network based at least in part on the image segments of the marked lanes to obtain an annotated image.

The non-transitory computer-readable storage medium of the preceding paragraph can include any subcombination of the following features: wherein the operations further comprise: generating an overlay including lane markings from the rotated image segments of the marked lane or the image segments of the marked lane; and providing the superimposed layers to one or more of the following systems: a vehicle routing system, a driver assistance system, or an autonomous vehicle system; wherein recalling the image of the road network comprises replacing a corresponding portion of the image with an image segment of the marked lane; and wherein profiling the image of the road network comprises applying lane markers from image segments of the marked lanes to corresponding portions of the image.

Another aspect of the present disclosure proposes a computer-implemented method of pre-processing images from map images to assist in implementing an automatic process of determining lane markings. The computer-implemented method may be implemented by a map image processing system that includes one or more hardware processors and that is configured with certain computer-executable instructions. Further, the computer-implemented method may include: receiving an image of a road network; converting the image into a binary image; identifying a road within a road network using the binary image; dividing a portion of an image including a depiction of a road into a plurality of image segments, each image segment containing a depiction of a road segment of the road; determining a target axis for a road segment of the road; rotating at least one image segment to align a road segment depicted in the at least one image segment with a target axis; and using the rotated at least one image segment to identify lane markers within the image of the road network.

The method of the preceding paragraph may include any subcombination of the following features: wherein the image comprises a map of the road network; wherein the image comprises a top-down two-dimensional view of the road network; wherein the image comprises a high definition image; wherein the method further comprises cropping a portion of at least one image segment to exclude from the at least one image segment content not depicting a portion of the road segment; wherein the target axis comprises a horizontal axis; wherein the method further comprises storing the rotated at least one image segment in a map repository, wherein the map repository stores training data for a model generation system that generates a machine learning model to automatically identify lane markers in road network images; wherein the method further comprises storing in a map repository an indication of a degree of rotation of the rotated at least one image segment; wherein identifying a road within the road network using the binary image comprises: determining a contour of the road based on a switching between black and white pixels in the binary image; selecting a plurality of inflection points within the binary image; identifying a starting location and an ending location of the roadway based at least in part on a comparison of gradients between the plurality of inflection points; and identifying a road based on the identified start location and the identified end location; and wherein the method further comprises providing the rotated at least one image segment and the overlay containing the identified lane markers as training data to a machine learning model generation system for generating a machine learning model identifying the lane markers in the image of the road network.

Another aspect of the present disclosure is directed to a system that pre-processes images from map images to facilitate an automatic process of determining lane markings. The system may include a non-volatile storage device configured to store an image of a road network; and a map image processing system comprising one or more hardware processors. The map image processing system may be configured to: accessing an image of a road network from a non-volatile storage device; converting the image into a grayscale image; identifying a road within the road network using the grayscale image; generating image segments based on a portion of an image including a depiction of a road, each image segment containing a depiction of a road segment of the road; determining a target axis for a road segment of the road; rotating at least one image segment to align a road segment depicted in the at least one image segment with a target axis; and using the rotated at least one image segment to identify lane markers within the image of the road network.

The system of the previous paragraph may include any subcombination of the following features: wherein the map image processing system is further configured to crop a portion of at least one image segment to exclude from the at least one image segment content that does not depict a portion of a road segment; wherein the map image processing system is further configured to: storing the rotated at least one image segment in a non-volatile memory, wherein the non-volatile storage is further configured to: storing training data for a model generation system that generates a machine learning model to automatically identify lane markers in road network partial images; wherein the map image processing system is further configured to store in the non-volatile memory a measurement of a degree of rotation of the rotated at least one image segment; wherein the map image processing system is further configured to use the rotated at least one image segment to identify lane markers within the road network image by at least: determining a contour of the road based on a transition between at least two color gradients in the grayscale image; selecting a plurality of inflection points in the gray-scale image; identifying a starting location and an ending location of the roadway based at least in part on a comparison of gradients between the plurality of inflection points; and identifying a road based on the identified start location and the identified end location; and wherein the map image processing system is further configured to provide the rotated at least one image segment and the identified lane markers as training data to a machine learning model generation system for generating a machine learning model identifying the lane markers in the images of the road network.

Yet another aspect of the disclosure is directed to a non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more computing devices, configure the one or more computing devices to perform a set of operations. These operations may include: receiving an image of a road network; converting the image into a binary image; identifying a road within a road network using the binary image; generating image segments based on a portion of an image including a depiction of a road, each image segment containing a depiction of a road segment of the road; determining a target axis for a road segment of the road; rotating at least one image segment to align a road segment depicted in the at least one image segment with a target axis; and using the rotated at least one image segment to identify lane markers within the image of the road network.

The non-transitory computer-readable storage medium of the preceding paragraph can include any subcombination of the following features: wherein the operations further comprise: removing a portion of at least one image segment to exclude from the at least one image segment content not depicting a road segment portion; and the operations further comprise: identifying lane markings within an image of a road network using the rotated at least one image segment by: determining a contour of the road based on a switching between black and white pixels in the binary image; selecting a plurality of inflection points within the binary image; identifying a starting location and an ending location of the roadway based at least in part on a comparison of gradients between the plurality of inflection points; and identifying a road based on the identified start location and the identified end location; and wherein the operations further comprise: the rotated at least one image segment and the identified lane markers are provided as training data to a machine learning model generation system for generating a machine learning model identifying the lane markers in the images of the road network.

Another aspect of the present disclosure presents a computer-implemented method of generating a lane marker learning model using a pre-processed image dataset. The computer-implemented method may be implemented by a model generation system comprising one or more hardware processors and configured with certain computer-executable instructions, and further, the computer-implemented method may comprise: receiving a set of training data comprising standardized images of road segments; generating a lane marker overlay for each of the standardized images, the lane marker overlay corresponding to a lane marker of a road segment depicted in the standardized images; performing a data enhancement process on each normalized image and the corresponding lane marker overlay of each normalized image to obtain a set of enhanced images; and generating one or more machine learning models based at least in part on the training dataset, the lane-marking overlay for each normalized image, and the set of enhanced images.

The method of the preceding paragraph may include any subcombination of the following features: wherein each standardized image of the road segment comprises a binary image derived from a segment of the high definition image; wherein the training data set comprises at least one standardized image free of lane markers; wherein said generating a lane marker overlay does not include generating a lane marker overlay for the at least one standardized image without lane markers; wherein said generating a lane marker overlay comprises generating a lane marker overlay for the at least one normalized image without lane markers and indicative of a missing lane marker; wherein performing the data enhancement process comprises rotating each normalized image and the corresponding lane marker overlay of each normalized image to obtain the set of enhanced images; wherein at least some of the normalized images are rotated by a different amount than at least some other of the normalized images; wherein at least some of the normalized images are rotated by a random or pseudo-random degree; wherein the training data set comprises a second set of normalized images of road segments, and wherein the second set of normalized images is excluded from said performing the data enhancement process; wherein each normalized image of a road segment is normalized such that the corresponding road segment depicted in the normalized image is rotated to be aligned along an axis, and wherein the axis is the same for each normalized image; wherein the generating one or more machine learning models comprises generating a plurality of machine learning models, and wherein the method further comprises selecting a machine learning model from the plurality of machine learning models based on one or more selection factors; and wherein the one or more selection factors include: complexity of each machine learning model; the accuracy of each machine learning model; or the amount of time to execute each machine learning model.

Another aspect of the present disclosure proposes a system for generating a lane marker learning model using a pre-processed image dataset. The system may include a non-volatile memory configured to store training data for generating one or more machine learning models that identify lane markers within a road network image; and a model generation system comprising one or more hardware processors. The model generation system may be configured to: accessing, from a non-volatile storage device, a set of training data comprising a standardized image of a road segment; generating a lane marker overlay for each of the standardized images, the lane marker overlay corresponding to a lane marker of a road segment depicted in the standardized images; obtaining a set of enhanced images using a data enhancement process applied to the superposition of the normalized images and the corresponding lane markers; and generating one or more machine learning models based at least in part on the training dataset, the lane-marking overlay for each normalized image, and the set of enhanced images.

The system of the preceding paragraph may include any subcombination of the following features: wherein the training data set comprises at least one normalized image free of lane markers, and wherein the one or more hardware processors are further configured to exclude the at least one normalized image free of lane markers when generating a lane marker overlay for each normalized image; wherein the training data set comprises at least one normalized image free of lane markers, and wherein the model generation system is further configured to generate a lane marker overlay of the at least one normalized image free of lane markers and indicative of a missing lane marker, when generating a lane marker overlay of each normalized image; wherein the model generation system is further configured to perform the data enhancement process by at least rotating each normalized image and the corresponding lane marker overlay of each normalized image to obtain the set of enhanced images; wherein at least some of the normalized images are rotated by a different amount than at least some other of the normalized images; wherein the training data set includes a second set of normalized images of road segments, and wherein the model generation system is further configured to exclude the second set of normalized images from the data enhancement process; and wherein each normalized image of a road segment is normalized such that the corresponding road segment depicted in the normalized image is rotated to be aligned along an axis, and wherein the axis is the same for each normalized image.

Yet another aspect of the disclosure is directed to a non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more computing devices, configure the one or more computing devices to perform a set of operations. These operations may include: receiving a set of training data comprising standardized images of road segments; generating a lane marker overlay for each of the standardized images, the lane marker overlay corresponding to a lane marker of a road segment depicted in the standardized images; obtaining a set of enhanced images based on the normalized images and the respective lane marker overlays; and generating one or more machine learning models based at least in part on the training dataset, the lane-marking overlay for each normalized image, and the set of enhanced images.

In other embodiments, the system or systems may operate according to one or more of the methods and/or computer-readable media described in the preceding paragraphs. In still other embodiments, one or more methods may operate in accordance with one or more of the systems and/or computer-readable media recited in the preceding paragraphs. In further embodiments, one or more computer-readable media, excluding transitory propagating signals, may cause one or more computing devices having one or more processors and non-transitory computer-readable memory to operate in accordance with one or more of the systems and/or methods recited in the preceding paragraphs.

■Term(s) for

Conditional language, such as "can," "might," "may," or "may," unless specifically stated otherwise, or otherwise understood in the context of usage, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements, and/or steps. Thus, such conditional language is not generally intended to imply that features, elements, and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without user input or prompting, whether to include such features, elements, and/or steps in or to be performed by any particular embodiment.

Unless the context clearly requires otherwise, throughout the description and the claims, the words "comprise", "comprising", and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense, i.e., "including but not limited to". As used herein, the terms "connected," "coupled," or any variant thereof, refer to any direct or indirect connection or coupling between two or more elements; the coupling or connection between the elements may be physical, logical, or a combination thereof. Moreover, as used in this application, the words "herein," "above," "below," and words of similar import refer to this application as a whole and not to any particular portions of this application. Where the context permits, words using the singular or plural number may also include the plural or singular number, respectively. When taken in conjunction with a list of two or more items, the word "or" encompasses all of the following interpretations of the word: any one item in the list, all items in the list, and any combination of items in the list. Likewise, the term "and/or," when used in conjunction with a list of two or more items, encompasses all of the following interpretations of the word: any one item in the list, all items in the list, and any combination of items in the list.

In some embodiments, certain operations, actions, events, or functions of any of the algorithms described herein may be performed in a different order, may be added, merged, or eliminated altogether (e.g., not all are essential to performing the algorithms). In some embodiments, operations, actions, functions, or events may be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors or processor cores, or on other parallel architectures, rather than sequentially.

The systems and modules described herein may include software, firmware, hardware or any combination of software, firmware or hardware suitable for the purposes described. The software and other modules may reside on and execute on servers, workstations, personal computers, computerized tablets, PDAs, and other computing devices suitable for the purposes described herein. Software and other modules may be accessed through local computer memory, a network, a browser, or other means suitable for the purposes described herein. The data structures described herein may include computer files, variables, programming arrays, programming structures, or any electronic information storage schemes or methods, or any combination thereof, suitable for the purposes described herein. User interface elements described herein may include elements from graphical user interfaces, interactive voice responses, command line interfaces, and other suitable interfaces.

Moreover, the processing of the various components of the illustrated system may be distributed across multiple machines, networks, and other computing resources. Two or more components of a system may be combined into fewer components. The various components of the illustrated system may be implemented in one or more virtual machines, rather than in dedicated computer hardware systems and/or computing devices. Likewise, the illustrated data stores may represent physical and/or logical data stores, including for example, storage area networks or other distributed storage systems. Furthermore, in some embodiments, the connections between the illustrated components represent possible paths of data flow, rather than actual connections between hardware. Although a few examples of possible connections are shown, in various implementations any subset of the components shown are capable of communicating with each other.

Embodiments are also described above in connection with flowchart illustrations and/or block diagrams for methods, apparatus (systems) and computer program products. Each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. Such instructions may be provided to a processor of a general purpose computer, special purpose computer (e.g., including a high performance database server, graphics subsystem, etc.), or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the actions specified in the flowchart and/or block diagram block or blocks. These computer program instructions may also be stored in a non-transitory computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the action specified in the flowchart and/or block diagram block or blocks. These computer program instructions may also be loaded onto a computing device or other programmable data processing apparatus to cause a series of operational steps to be performed on the computing device or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computing device or other programmable apparatus provide steps for implementing the actions specified in the flowchart and/or block diagram block or blocks.

Any patents and applications and other references mentioned above, including any that may be listed in the accompanying filing documents, are incorporated herein by reference. Aspects of the invention can be modified, if necessary, to employ the systems, functions and concepts described above to provide further embodiments of the invention. These and other changes can be made to the invention in light of the above detailed description. While certain examples of the invention have been described above, and while the best mode contemplated, the invention may be practiced in a variety of ways, no matter how detailed the text appears. The details of the system vary considerably in its specific embodiments, while still encompassing the invention disclosed herein. As noted above, certain terms used in describing certain features of aspects of the invention should not be construed to imply that the terms are redefined herein to be restricted to any specific characteristics, features, or aspects of the invention with which the terms are associated. In general, the terms used in the following claims should not be construed to limit the invention to the scope of the specific examples disclosed in the specification, unless such terms are explicitly defined in the above detailed description. Accordingly, the actual scope of the invention encompasses not only the disclosed examples, but also all equivalent ways of practicing or implementing the invention under the claims.

Certain aspects of the invention are presented below in certain claim forms in order to reduce the number of claims, but applicants may contemplate other aspects of the invention in any number of claim forms. For example, while only one aspect of the invention is recited in the means-plus-function claims under 35u.s.csec.112(f) (AIA), other aspects can equally be embodied as the means-plus-function claims, or in other forms, such as in a computer-readable medium. Any claim intended to be treated pursuant to 35u.s.c. § 112(f) will begin with a "for" statement, but use of "for" in any other context does not imply that treatment pursuant to 35u.s.c. § 112(f) is referenced. Accordingly, the applicant reserves the right to add additional claims to this or a later application after filing the present application.

Claims

1. A computer-implemented method of performing an automatic lane marking process on an image of a road network using a machine learning model, the computer-implemented method comprising:

implemented by a lane detection system comprising one or more hardware processors and configured with specific computer-executable instructions,

accessing a rotated image segment depicting a road segment, wherein the rotated image segment comprises a rotated portion of a road network image, and wherein the rotated image segment is rotated to align the road segment with a target axis;

applying the rotated image segments to a machine learning model to obtain rotated image segments that mark lanes;

determining a degree of rotation of the rotated image segment relative to the target axis;

reversely rotating the rotated image segment of the marking lane according to the rotation degree to obtain an image segment of the marking lane; and is

Updating an image of the road network based at least in part on the image segment of the marked lane to obtain an updated image.

2. The computer-implemented method of claim 1, wherein lane markings on the rotated image segment of the marked lane or on the image segment of the marked lane are made part of an overlay that provides lane marking data for one or more of: a vehicle routing system, a driver assistance system, or an autonomous automotive system.

3. The computer-implemented method of claim 1, wherein the degree of rotation comprises an amount by which a portion of the image of the road network is rotated to obtain the rotated image segment.

4. The computer-implemented method of claim 1, wherein the degree of rotation is determined from metadata associated with the rotated image segment.

5. The computer-implemented method of claim 1, wherein the image of the road network comprises a map of the road network.

6. The computer-implemented method of claim 1, wherein updating the image of the road network comprises replacing a corresponding portion of the image with an image segment of the marked lane.

7. The computer-implemented method of claim 1, wherein updating the image of the road network comprises applying lane markers from image segments of the marked lane to corresponding portions of the image.

8. The computer-implemented method of claim 1, further comprising storing the updated map in a map store configured to store map data for one or more of: a vehicle routing system, a driver assistance system, or an autonomous vehicle system.

9. The computer-implemented method of claim 1, wherein the target axis comprises a horizontal axis.

10. A system for performing an automatic lane marking process on an image of a road network using a machine learning model, the system comprising:

a storage device configured to store rotated image segments depicting portions of a road and rotated to be aligned with a target axis; and is

A lane detection system comprising one or more hardware processors configured to:

accessing, from the storage device, a rotated image segment depicting a road segment, wherein the rotated image segment is a rotated portion of a road network image, and wherein the rotated image segment is rotated to align the road segment with the target axis;

inputting the rotated image segments into a machine learning model to obtain rotated image segments that mark lanes;

Modifying an image of the road network based at least in part on the image segment of the marked lane to obtain a modified image.

11. The system of claim 10, wherein the lane detection system is further configured to generate an overlay comprising lane markings from the rotated image segment of the marker lane or the image segment of the marker lane, and wherein the overlay is provided to one or more of: a vehicle routing system, a driver assistance system, or an autonomous vehicle system.

12. The system of claim 10, wherein the degree of rotation comprises an amount by which a portion of the image of the road network is rotated to obtain the rotated image segment.

13. The system of claim 10, wherein the lane detection system is further configured to modify the image of the road network by replacing image segments of the marked lane with corresponding portions of the image.

14. The system of claim 10, wherein the lane detection system is further configured to modify the image of the road network by applying lane markers from image segments of the marked lane to corresponding portions of the image.

15. The system of claim 10, wherein the lane detection system is further configured to store the modified map in a map store configured to store map data for one or more of: a vehicle routing system, a driver assistance system, or an autonomous vehicle system.

16. The system of claim 15, wherein the storage device comprises the map store.

17. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more computing devices, configure the one or more computing devices to perform operations comprising:

obtaining a rotated image segment depicting a road segment, wherein the rotated image segment is a rotated portion of a road network image, and wherein the rotated image segment is rotated to align the road segment with a target axis;

applying the rotated image segments as input to a machine learning model to obtain rotated image segments that mark lanes;

Annotating the image of the road network based at least in part on image segments of the marked lanes to obtain an annotated image.

18. The non-transitory computer-readable storage medium of claim 17, wherein the operations further comprise:

generating an overlay comprising a marker lane from the rotated image segment of the marker lane or the image segment of the marker lane; and is

Providing the superimposed layers to one or more of: a vehicle routing system, a driver assistance system, or an autonomous vehicle system.

19. The non-transitory computer-readable storage medium of claim 17, wherein annotating the image of the road network comprises replacing a corresponding portion of the image with an image segment of the marked lane.

20. The non-transitory computer-readable storage medium of claim 17, wherein annotating the image of the road network comprises applying lane markers from image segments of the marked lane to corresponding portions of the image.