DE102019124419A1 - SCENE CLASSIFICATION - Google Patents

Abstract

In one aspect, scene classification can be provided. An image capturing device can capture a series of individual images of an environment from a moving vehicle. A time classifier can classify individual images with temporal predictions and generate a series of individual images that are assigned to the respective temporal predictions based on a scene classification model. The time classifier can classify frames based on a neural convolution network (CNN), a long short term memory (LSTM) network and a fully connected layer. The scene classifier can classify individual images based on a CNN, a global average pooling and a completely connected layer and generate an associated scene prediction based on the scene classification model and corresponding temporal predictions. A controller of a vehicle can activate or deactivate vehicle sensors or vehicle systems of the vehicle based on the scene prediction.

Description

BACKGROUND

In driving scenarios, a person's understanding of a scene involves answering questions about a location, environmental conditions and the behavior of road users. Interestingly, people are able to perform dynamic scene recognition quickly and precisely, paying little attention to objects in the scene. Human drivers have the remarkable ability to classify complex traffic scenes and adapt their driving behavior to their environment. In this context, automated dynamic scene recognition on a human level can be an attractive target.

SHORT DESCRIPTION

In one aspect, a scene classification system may include an image capture device, an image segmentation module, an image masker, a time classifier, and a scene classifier. The image capturing device can capture a first series of individual images of an environment from a moving vehicle. The image segmentation module can identify one or more road users in the environment based on a first neural convolution network (CNN). The image masker can generate a second series of individual images by masking one or more of the road users from the surroundings. The time classifier can classify one or more frames of the second series of frames with one of two or more temporal predictions and generate a third series of frames that are associated with the respective temporal predictions based on a scene classification model. The time classifier can perform the classification based on a second CNN, a long short term memory (LSTM) network and a first fully connected layer. The scene classifier can classify one or more individual images of the third series of individual images based on a third CNN, a global average pooling and a second completely connected layer and generate an associated scene prediction based on the scene classification model and corresponding temporal predictions.

The two or more temporal predictions can include an approaching annotation, a retracting annotation, and a passing annotation. The first CNN, the second CNN or the third CNN can be a Deepnet-CNN or a ResNet 50-CNN. The scene classification system may be implemented in a vehicle and the vehicle may include a controller that enables or disables one or more sensors or one or more vehicle systems of the vehicle based on the scene prediction.

The scene classifier may classify one or more frames of the third series of frames with a weather classification including clear, sunny, snow, rainy, cloudy, or foggy, and the controller may classify one or more of the sensors or one or more of the vehicle's vehicle systems based on the weather classification activate or deactivate. The scene classifier may classify one or more frames of the third series of frames with a road surface classification including dry, wet, or snow, and the controller may activate or deactivate one or more of the sensors or one or more of the vehicle's vehicle systems based on the road surface classification. The scene classifier may classify one or more frames of the third series of frames with an environmental classification including urban, up / down, highway or local, and the controller may activate one or more of the sensors or one or more of the vehicle's vehicle systems based on the environmental classification or deactivate.

One or more of the vehicle systems can be a LIDAR system or a radar system. The controller can disable the LIDAR system or radar system based on the scene prediction being a tunnel. The controller can prioritize the search for traffic lights, stop signs, stop lines based on the fact that the scene prediction is an intersection.

In one aspect, a vehicle equipped with a scene classification system may include an image capture device, an image segmentation module, an image masker, a time classifier, a scene classifier, and a controller. The image capturing device can capture a first series of individual images of an environment from a moving vehicle. The image segmentation module can identify one or more road users in the environment based on a first neural convolution network (CNN). The image masker can generate a second series of individual images by masking one or more of the road users from the surroundings. The time classifier can classify one or more frames of the second series of frames with one of two or more temporal predictions and generate a third series of frames that are associated with the respective temporal predictions based on a scene classification model. The Time classifier can perform the classification based on a second CNN, a Long Short Term Memory (LSTM) network and a first fully connected layer. The scene classifier can classify one or more individual images of the third series of individual images based on a third CNN, a global average pooling and a second completely connected layer and generate an associated scene prediction based on the scene classification model and corresponding temporal predictions. The controller may activate or deactivate one or more sensors or one or more vehicle systems of the vehicle based on the scene prediction.

The two or more temporal predictions can include an approaching annotation, a retracting annotation, and a passing annotation. The first CNN, the second CNN or the third CNN can be a Deepnet-CNN or a ResNet 50-CNN. One or more of the vehicle systems may be a LIDAR system or a radar system, and the controller may deactivate the LIDAR system or radar system based on the scene prediction being a tunnel.

In one aspect, a scene classification system may include an image capture device, a time classifier, and a scene classifier. The image capturing device can capture a first series of individual images of an environment from a moving vehicle. The time classifier may classify one or more frames of the first series of frames with one of two or more temporal predictions and generate a second series of frames that are associated with the respective temporal predictions based on a scene classification model. The time classifier can perform a classification based on a neural convolution network (CNN), a long short term memory (LSTM) network and a first fully connected layer. The scene classifier can classify one or more individual images of the second series of individual images based on a second CNN, a global average pooling and a second completely connected layer and generate an associated scene prediction based on the scene classification model and corresponding temporal predictions.

The two or more temporal predictions can include an approaching annotation, a retracting annotation, and a passing annotation. The CNN or the second CNN can be a ResNet 50-CNN. The scene classification system may be implemented in a vehicle and the vehicle may include a controller that enables or disables one or more sensors or one or more vehicle systems of the vehicle based on the scene prediction.

The scene classifier can classify one or more frames of the third series of frames with a weather classification including clear, sunny, snow, rainy, overcast or foggy. The controller may activate or deactivate one or more of the sensors or one or more of the vehicle systems of the vehicle based on the weather classification. The scene classifier can classify one or more frames of the third series of frames with a road surface classification including dry, wet, or snow. The controller may activate or deactivate one or more of the sensors or one or more of the vehicle systems of the vehicle based on the road surface classification.

Figure list

• 1 10 is a component diagram of a scene classification system according to one aspect.
• 2nd 10 is a flowchart of a scene classification method according to one aspect.
• 3rd FIG. 10 is an exemplary diagram of temporal predictions or predictions associated with a scene classification, according to one aspect.
• The 4A - 4B are exemplary diagrams of temporal predictions or predictions associated with different scene classifications, according to one aspect.
• 5 FIG. 4 is an exemplary diagram of an architecture used to train the scene classification system of 1 assigned.
• 6 FIG. 4 is an illustration of an exemplary computer readable medium or device including processor executable instructions configured to embody one or more of the provisions set forth herein in one aspect.
• 7 FIG. 4 is an illustration of an exemplary computing environment in which one or more of the provisions set forth herein are implemented in accordance with one aspect of the invention.

DETAILED DESCRIPTION

The following terms are used throughout the disclosure; Definitions are included herein to facilitate understanding of one or more aspects of the disclosure.

A "processor", as used herein, processes signals and performs general calculations and arithmetic functions. Signals processed by the processor may include digital signals, data signals, computer instructions, processor instructions, messages, a bit, a bit stream, or other means that can be received, transmitted, and / or recognized. In general, the processor can be a variety of different processors, including multiple single and multi-core processors and coprocessors, and other architectures with multiple single and multi-core processors and coprocessors. The processor can include different modules to perform different functions.

"Memory" as used herein can include volatile memory and / or non-volatile memory. Non-volatile memories can include, for example, ROM (read only memory), PROM (programmable read only memory), EPROM (erasable PROM) and EEPROM (electrically erasable PROM). Volatile memory may include, for example, RAM (Random Access Memory), synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM) and direct RAM bus RAM (DRRAM). The memory can store an operating system that controls or allocates resources to a computing device.

A "disk" or "drive" as used herein can be a magnetic disk drive, a solid state drive, a floppy disk drive, a tape drive, a zip drive, a flash memory card and / or a memory stick. Furthermore, the disc can be a CD-ROM (Compact Disk ROM), a writable CD drive (CD-R drive), a rewritable CD drive (CD-RW drive) and / or a digital video ROM drive ( DVD-ROM). The disk can store an operating system that controls or allocates resources to a computing device.

A "bus" as used herein refers to an interconnected architecture that is operatively connected to other computer components within or between computers. The bus can transfer data between the computer components. The bus can be, inter alia, a memory bus, a memory controller, a peripheral bus, an external bus, a crossbar switch and / or a local bus. The bus can also be a vehicle bus that connects components within a vehicle using protocols such as Media Oriented Systems Transport (MOST), Controller Area Network (CAN) and Local Interconnect Network (LIN).

A "database" as used herein may refer to a table, a set of tables, and a set of data stores (e.g., disks) and / or methods of accessing and / or manipulating these data stores.

An "operational link" or a link through which units are "operationally connected" is a link in which signals, physical communications and / or logical communications can be sent and / or received. An operational connection can include a wireless interface, a physical interface, a data interface, and / or an electrical interface.

"Computer communication," as used herein, refers to communication between two or more computing devices (e.g., computer, personal digital assistant, cell phone, network device) and may include, for example, network transfer, file transfer, applet transfer, E -Mail, a Hypertext Transfer Protocol (HTTP) transfer and so on. For example, computer communication may be via a wireless system (e.g., IEEE 802.11), an Ethernet system (e.g., IEEE 802.3), a token ring system (e.g., IEEE) 802.5), a local area network (LAN), a wide area network (WAN), a point-to-point system, a system for switching circuits, a packet switching system.

A "vehicle" as used herein means any moving vehicle that is capable of carrying one or more human occupants and that is powered by any form of energy. The term "vehicle" includes cars, trucks, vans, minivans, SUVs, motorcycles, scooters, boats, water scooters and planes. In some scenarios, an automobile includes one or more engines. The term “vehicle” may also refer to an electric vehicle (EV) that is wholly or partially powered by one or more electric motors that are powered by an electric battery. The EV can include battery-powered electric vehicles (BEV) and plug-in hybrid electric vehicles (PHEV). In addition, the term “vehicle” may refer to an autonomous vehicle and / or self-driving vehicle that is powered by any form of energy. The autonomous vehicle may or may not carry one or more human occupants.

A "vehicle system" as used herein can be any automatic or manual system that can be used to improve the vehicle, driving, and / or safety. Exemplary vehicle systems include, among other things, an autonomous drive system, an electronic stability control system, an anti-lock braking system, a brake assistance system, an automatic brake priming system, a low-speed following system, a cruise control system, a collision warning system, a collision reduction braking system, an automatic cruise control system, a lane departure warning system, a blind spot, an angle display system Lane departure warning systems, a navigation system, a transmission system, brake pedal systems, an electronic power steering system, visual devices (e.g. camera systems, proximity sensor systems), a climate control system, an electronic preload system, a monitoring system, a passenger detection system, a vehicle suspension system, a vehicle seat configuration system, a vehicle cabin lighting system, an audio cabin lighting system and a sensor system.

The issues discussed herein can be described and implemented in the context of a non-volatile computer readable storage medium that stores computer executable instructions. Non-volatile computer readable storage media include computer storage media and communication media. For example, flash memory drives, DVDs (Digital Versatile Discs), CDs (Compact Discs), floppy disks and tape cartridges. Non-volatile computer readable storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, modules, or other data.

1 Figure 3 is a component diagram of a scene classification system 100 according to one point of view. A vehicle 10th can with a scene classification system 100 be equipped. The scene classification system 100 can be an image capture device 102 , a processor 104 , a memory 106 , a storage drive 108 , a communication interface 110 , an image segmentation module 112 , an image masker 114 , a convolutor 116 , a time classifier 118 and a scene classifier 120 lock in. The vehicle 10th can be a controller, one or more vehicle sensors and one or more vehicle systems 190 lock in. The communication interface 110 can with a server 130 stay in contact. The server 130 may include a scene classification database that contains a ground truth image sequence 132 and a scene classification model 134 or may include a scene classification policy network. In one aspect, one or more of the image segmentation module 112 , Image masker 114 , Convolutor 116 , Time classifier 118 and / or scene classifiers 120 via the processor 104 , the store 106 , the storage drive 108 etc. are implemented.

Ground truth

In one aspect, the Ground Truth image sequence 132 include a series of one or more still images associated with a moving vehicle that can be collected during a training phase. Note that some scenes can be static while other scenes or locations can be dynamic. For example, an intersection can be a static scene or location, while a construction site can be dynamic, in that the construction site can be defined by traffic cones, the size, shape, appearance and / or location between the construction sites and between different days or times can change.

Each of the one or more individual images of the Ground Truth image sequence 132 can be provided with one or more labels, such as a time classification label, a weather classification label, a road surface classification label, an environmental classification label and a scene classification label (e.g. with manual annotations and indication of ground truth) . Examples of time classification labels can include background, approaching, entering, passing, etc. In other words, the individual images are temporally provided with fine-grained labels such as “approaching” (A), “entering” (E) and “passing” (P), depending on the viewing angle and / or the position of the training vehicle in relation to the position at Place of interest or the scene. The classification labels can be organized hierarchically and causally. For example, the surroundings can be annotated above, followed by the scene classes on the middle level and the fine-grained annotations such as "Approaching, entering and passing" on the lower level.

Examples of weather classification labels can include clear, sunny, snow, rainy, overcast, cloudy, foggy, light, dark, etc. Examples of road surface classification labels can include dry, wet, snow, rainy, covered (e.g. some traffic markings not visible), mud, etc. Examples of environmental classification labels can Include environment types such as urban, rural, suburban, on / off, highway, local (e.g. neighborhood, neighborhood, school), etc. Driveways / exits can be, for example, a connection between two motorways or between a motorway and another type of road. Examples of scene classification labels can be locations on streets, a construction site, an intersection (e.g. an x-way intersection, such as a three, four, five-way intersection, etc.), a bridge, a Include overpass, level crossing, tunnel, lane junction, lane junction, zebra crossing, etc. Some scene classifications can only be assigned to the "Approach" and "Pass" time classification labels, while others can be assigned to the "Approach, enter and pass" labels. The road surface classification and the weather classifications can be mutually exclusive. In other words, it can be wet on the street, but the weather can be sunny, for example.

This annotated Ground Truth image sequence 132 can be used to train a model that is in the scene classification database, for example as a scene classification model 134 or saved as a scene classification policy network. Because the Ground Truth image sequence 132 the scene classification model can be annotated as desired (e.g. this can be done manually by humans) 134 be trained through machine learning, in-depth learning or another type of artificial intelligence technology. In this context, the scene classification system 100 be trained for this (e.g. via the processor 104 ), Results from the Ground Truth image sequence 132 by minimizing losses and simulating through feedback.

Image capture

The image capture device 102 can capture a first series of individual images (e.g. video) of an environment (e.g. operating environment) from the perspective of a moving vehicle. In one aspect, this first series of still images or ambient videos can be input to the scene classification system 100 be used.

segmentation

The image segmentation module 112 can identify one or more road users in the environment from the individual images based on a first neural convolution network (CNN) and the first series of individual images. In one aspect, the image segmentation module 112 implement a deeplab CNN. Regardless of the implementation, the image segmentation module 112 provide the semantic segmentation as output when the input of the series of frames is provided. The image segmentation module 112 can classify objects within each frame of the first row of frames. For example, the image segmentation module 112 identify one or more pedestrians, one or more vehicles (e.g. in traffic), one or more motorists, one or more bystanders, one or more cyclists, one or more moving objects, etc.

Masking

The image masker 114 can generate a second series of individual images by masking one or more of the road users from the surroundings. Since road users generally have no influence on how a scene is defined (e.g. whether the surroundings are an intersection, a motorway, etc.), the image masker can 114 mask all road users from the surrounding area from the second row of single images. In one aspect, the image masker can 114 mask one or more of the road users from the single image sequence by semantic segmentation. In one aspect, the image masker can 114 also mask other unnecessary objects from the surroundings, such as birds in the sky, etc. In this way, the image masker can 114 the scene classification system 100 provide greater spatial attention by using the system's neural networks for scene classification 100 Focusing on the unmasked portions of the individual images enables and thus provides greater accuracy in the classification. Thus, the image masker 114 and the image segmentation module 112 semantic context are provided.

Time classification

The time classifier 118 may be one or more frames of the second series of frames (e.g., or from the original set of frames created by the image capture device 102 classified) with one of two or more temporal predictions and generate a third series of individual images that correspond to the respective temporal predictions based on a scene classification model 134 assigned. Examples of temporal predictions can include background, approaching, entering, passing a scene or location, etc. The time classifier 118 can learn that approaching is generally followed by entering and then passing.

In one aspect, the time classifier 118 perform a classification based on a second CNN, a Long Short Term Memory (LSTM) network and a first fully connected layer on an input set of frames that concatenate the original input frames (RGB) with semantic segmentation (RGBS) Single images, single images with road users that are masked with semantic segmentation (RGB masked), or only using a semantic single-channel segmentation image (S). In this way, the time classifier 118 used to determine where the vehicle is 10th located within a scene (e.g. on a frame-by-frame basis). In one aspect, the second CNN may be implemented as ResNet 50, for example. The time classifier 118 can be determined by the scene classifier 120 determine one or more of the temporal predictions with respect to the type of scene or location and assign them to one or more corresponding individual images of the first row of individual images or one or more corresponding individual images of the second series of individual images.

The time classifier 118 can, when performing a classification based on a CNN, the convolutor 116 implement and route input through one or more of the convolutor's CNNs, such as a CNN, a depth CNN, a pose CNN, etc. to produce an output.

In other words, the time classifier 118 determine the beginning, middle and / or end of a scene before determining what type of scene the scene actually is or before determining the associated scene prediction for the scene.

In other words, it can be the time classifier 118 the scene classification system 100 allow to differentiate between different phases of an event, such as when the vehicle 10th a crossing or a construction site happens. In particular, the time classifier 118 identify, assign, or annotate one or more frames of one or more frames of one or more of the series of images with a temporal prediction from a set of temporal predictions. As previously mentioned, examples of these temporal predictions may include background, approaching, entering, or passing through a scene or location. In this way, the time classifier 118 Provide fine-grained or finely tuned time classification (e.g. around the vehicle 10th localize at a specific, unknown scene or location). It should be noted that other time predictions can be used from a different point of view. For example, the time prediction can be numeric and indicate the progression through a scene (e.g., through the scene classifier) 120 is to be defined). Regardless, the Ground Truth image sequence 132 to be used a classifier such as the time classifier 118 to train to recognize when the vehicle is moving 10th Approaching, entering, or happening to a scene regardless of whether the type of scene is known.

Weather classification

The scene classifier 120 can use the scene classification model 134 use that can be trained on a CNN, such as ResNet 50 or Deepnet CNN, to make the weather classification for the vehicle 10th to determine. Similar to scene classification, weather, road surface and surroundings can be classified using an input that masked road users (e.g. using the image masker) 114 generated series of individual images that mask one or more of the road users from the area). However, other inputs can also be provided, such as the original input frames (RGB), frames linked with semantic segmentation (RGBS), frames with road users masked with semantic segmentation (RGB masked), or simply using a semantic single channel -Segmentation picture (S). The scene classification model 134 can be based on the annotated Ground Truth image sequence 132 be trained. Examples of weather classification labels can be lighting conditions, visibility conditions such as clear, sunny, snowfall, rainy, overcast, cloudy, foggy, light, dark, etc.

Classification of the road surface

The scene classifier 120 can use the scene classification model 134 use that may have been trained on a CNN, such as ResNet 50, for road surface classification for the vehicle 10th to determine. The scene classification model 134 can based on the ground truth image sequence 132 are trained, which can be annotated with one or more labels for each of the assigned individual images as described above. Examples of road surface classification labels can be dry, wet, snow, covered (e.g. some traffic markings not visible), mud, etc.

Environment classification

The scene classifier 120 can act similarly to the other types of classifications. Examples of environmental classification labels may include environmental types such as urban, rural, suburban, up / down, highway, local (e.g. neighborhood, neighborhood, school), etc.

Scene or location classification

The scene classifier 120 may classify one or more frames of the third series of frames based on a third CNN, Global Average Pooling and a second fully connected layer and an associated scene prediction based on the scene classification model 134 and generate corresponding temporal forecasts. The scene classifier 120 can generate a fourth series of individual images that correspond to the respective time predictions based on the scene classification model 134 and are assigned to the respective time predictions. In this way, the time classifier 118 are used to cut individual images from the video or from the image sequences in order to enable efficient scene classification. In other words, the scene classifier 120 only consider single images that are marked as approaching, entering and passing a certain surrounding location, while it ignores annotated individual images as the background and thus enables, for example, a dynamic classification of street scenes. In this way, this two-stage architecture reduces the unnecessary use of computing power by excluding background individual images from the examination and / or scene classification. So the time classifier works 118 as a rough separator for the scene classifier 120 , which mitigates the amount of computing power and resources used to classify scenes and only the candidate images for approaching, entering or passing the scene classifier 120 as an event window to the prediction network.

The scene classifier 120 can, similar to the time classifier 118 when performing a classification based on a CNN, the convolutor 116 implement and route input through one or more of the convolutor's CNNs, such as a CNN, a depth CNN, a pose CNN, ResNet 50-CNN, etc. to produce an output.

In one aspect, the third CNN may be implemented as ResNet 50, for example. Therefore, the scene classifier 120 Use one or more of the temporal predictions from one or more of the corresponding frames to help determine what type of scene or location is associated with approaching, entering, and passing a scene. For example, the time classifier 118 have classified one or more individual images from the series of individual images with temporal predictions. Based on these temporal predictions, the scene classifier can 120 determine that a set of frames associated with approaching, entering, and passing a scene from the series of frames is, for example, a construction site. Thus the time classifier 118 determine that the vehicle 10th passes through a beginning, a middle and an end of an unknown type of scene, and the scene classifier 120 can determine what type of scene the scene is after the time classifier 118 made or determined his temporal predictions of the individual images.

Examples of scene or location classifications can be locations on roads, such as a construction site, an intersection (e.g. an x-way intersection, such as a three, four, five way intersection, etc.), a Include a bridge, an overpass, a railroad crossing, a tunnel, a track junction, a track junction, a zebra crossing, etc. In this way, the scene prediction can be a scene classification that indicates a type of location that the vehicle is located 10th e.g. B. approaches, into which it enters or which it happens.

In one aspect, the scene classifier can 120 generate the scene prediction based on the input of the first series of single images in real time, in such a way that a complete series of individual images that are annotated from the background with “approach, drive in, pass” in the background is not necessarily required for the scene prediction to create. In other words, only a subset of individual images may be provided with time predictions (e.g. background, approximation, ..., etc.) before the scene classifier 120 generates the assigned scene prediction based on the CNN, the global average pooling and corresponding temporal predictions. Thus, the development of machine learning that uses the semantic context and temporal nature of the Ground Truth dataset can result in the classification results for the scene classification system 100 improve.

Vehicle application

The controller can have one or more sensors or one or more vehicle systems 190 of the vehicle 10th based on the scene prediction and / or one or more of the classifications, such as the weather classification, the lane classification, the Enable or disable environmental classification, etc. For example, since scene context features can serve as a preliminary stage for other downstream tasks such as recognizing objects, behavior, action, intent, navigation, localization, etc., the control of the vehicle can 10th based on that from the scene classifier 120 certain scene prediction, as well as other classifications, including weather classification, road surface classification, and environmental classification.

For example, if the scene classifier 120 determines the scene prediction as a pedestrian crossing, the control of the vehicle 10th Activate additional sensors to detect pedestrians. At other times, such as when the vehicle is moving 10th located on the highway, pedestrian sensors can be prioritized lower. As another example, if the scene classifier 120 determines the scene prediction as an intersection, can control the vehicle 10th Activate additional sensors or operate certain modules to recognize traffic lights, stop signs, stop lines or other intersection-related information. In other words, the controller can re-prioritize or prioritize the search for traffic lights, stop signs, and stop lines based on the scene prediction being an intersection. Conversely, the controller can disable a lidar or radar system based on the scene prediction being a tunnel.

In one aspect, the scene classifier determines 120 scene prediction as a construction site, control of the vehicle (e.g. via the processor 104 implemented) can warn or issue notifications and / or deactivate autonomous driving, based on the fact that scene prediction is the construction site, since autonomous vehicles can use pre-made, high-resolution maps of a roadway. If the scene classifier 120 determines that it is foggy or rainy outside, the processor can 104 the LIDAR of one or more of the vehicle systems 190 disable to mitigate ghosting effects. If the scene classifier 120 The vehicle scene prediction determined as "in a tunnel" or that there is an overpass may be the priority of the GPS of the vehicle systems 190 be downgraded because the GPS may lose tracking from the tunnel or overpass. In addition, cameras can be prepared for extreme loads when passing through the tunnel or overpass area. Likewise, a lane departure warning system with larger tolerances can be implemented or deactivated if the scene classifier 120 For example, the scene prediction is determined as a branch area or in the vicinity of a departure. Therefore, the scene classifier 120 used to use one or more of the vehicle systems 190 to improve, such as by activating, deactivating, prioritizing, downgrading the priority, etc. of one or more of the respective vehicle systems 190 . In this way, the scene classifier 120 contextual information for other vehicle systems 190 of the vehicle 10th deliver to work efficiently.

2nd is a flowchart of a method 200 for scene classification according to one aspect. The procedure 200 for scene classification can be recorded 202 identifying a first series of individual images of an environment from a moving vehicle 204 of road users within the area based on a first CNN that is generating 206 a second series of individual images by masking road users from the area, the classification 208 of frames of the second series of frames with temporal predictions based on a second CNN, a long short term memory (LSTM) network and a first fully connected layer, classifying 210 of individual images based on a third CNN, a global average pooling and a second completely connected layer and the generation 212 an assigned scene prediction based on the scene classification model 134 and include corresponding time predictions.

3rd FIG. 10 is an exemplary diagram of temporal predictions associated with a scene classification according to one aspect. In 3rd can be different from the image capture device 102 captured individual images in connection with the ground truth image sequence 132 be marked. For example, a first frame 310 as a single picture "approaching", a second single picture 320 as a single picture "drive in" and a third single picture 330 be referred to as a single image "Passing". This approaching, entering and passing can correspond to the fact that the vehicle is moving 10th approaches an intersection 312, enters it 322 and passes 332 as in 3rd shown.

While 3rd representing approaching, entering, and passing for the intersection scene type, other scene types may be annotated in a similar manner (e.g., including temporal predictions of approaching, entering, and passing, as well as other annotations, such as annotations of the scene type of an intersection, a bridge, a tunnel etc.). Note that the Ground Truth image sequence 132 and that from the image capture device 102 captured series of single images from the perspective of a moving Can be vehicle and thus the individual images are not from the perspective of a static or stationary camera. In other words, the Ground Truth image sequence 132 and the acquired series of individual images include space-time variations in terms of viewing angle and / or scene appearance. As in 3rd Visibility variations can be caused by the changing distance to the intersection when the vehicle is moving 10th approaches the scene of interest (ie the intersection as it passes 332 ). The 4A - 4B are exemplary diagrams of temporal predictions, which are assigned to different scene classifications, from one point of view. In the 4A - 4B are various examples of a variety of annotations. In one aspect, one or more CNNs or other networks may be implemented to address the architecture of the 4A - 4B make carried out parameters understandable.

5 Fig. 3 is an exemplary diagram of an architecture used to train the system for scene classification 100 of 1 assigned. The Ground Truth image sequence 132 can be annotated to include the "site" scene classification label, and each of the frames in the entrance row of frames of the site environment can be annotated with temporal predictions indicating where the moving vehicle is within the site. In other words, the temporal predictions of the ground truth image sequence 132 for example marked as "approaching, entering or passing".

The image capture device 102 can capture an input row of still images. The image segmentation module 112 can segment or identify one or more road users using semantic segmentation, such as via a CNN 510 (e.g. a deeplab CNN). The image masker 114 can mask one or more of the road users from the images so that the scene classification system 100 only focus on the environment and can make a correspondingly more precise scene classification.

As in 5 can be seen, the time classifier 118 used to cut uncut video and aggregate the features to classify the entire cut segment. For example, it may be useful to analyze or determine a class as a 4-way intersection by looking at or examining a segment (e.g. approaching, entering and passing) in its entirety and not on a frame-by-frame basis. Here the time classifier 118 the series of individual images are supplied in which the road users are masked (e.g. the RGB masked individual images). According to other aspects or architectures, the time classifier can 118 receive other rows of frames, such as RGB, RGBS or S pictures. In any case, the time classifier 118 receive the input set of frames and these through a CNN 520 , such as the ResNet 50 CNN, a set of features 522 extract this set of characteristics through an LSTM 526 and a fully bonded layer 528 pass through and thus create a series of individual images, each annotated with temporal predictions.

The series of individual images annotated with temporal predictions can be used by the scene classifier 120 supplied to the one or more CNNs 530 , such as the ResNet 50 CNN, may include a set of features 532 extract a global average pooling 536 and perform the results through a fully connected layer 538 pass through to generate a scene prediction for the scene (e.g., which may be unknown up to this point), including still images annotated as they approach, enter, and pass. This model can be based on the Ground Truth image sequence 132 be trained. In other words, the time classifier 118 and the scene classifier 120 be trained through machine learning or deep learning to the annotations of the Ground Truth image sequence 132 to replicate or simulate e.g. B. when the scene classification system 100 a similar non-annotated series of frames is provided, creating a scene classification model 134 or a scene classification policy network is built up in the scene classification database on the server 130 is saved.

The scene classifier 120 can aggregate individual images within this window using global average pooling and create a unique class label for the entire event, location or scene. In one aspect, one or more of the CNNs described herein can be on the ground truth image sequence 132 or another database from the scene classification database. In order to reduce the overfitting, data augmentation can be carried out. Random tilt, resize, and trim can be used. As indicated, the processor can 104 or the control of the vehicle 10th Adaptations for one or more vehicle systems 190 based on the generated scene prediction.

Another aspect relates to a computer readable medium that includes processor executable instructions configured to implement one aspect of the techniques presented herein. One aspect of a computer readable medium or device developed in this manner is shown in 6 illustrates one implementation 600 a computer readable medium 608 includes, such as a CD-R, a DVD-R, a flash drive, a hard disk drive, etc., on which computer readable data 606 are encoded. This encoded computer readable data 606 , such as binary data including a plurality of zeros and ones, as shown in 606, in turn include a set of processor-executable computer instructions 604 which are configured to operate in accordance with one or more of the principles set forth herein. In this implementation 600 can the processor-executable computer instructions 604 to be configured a procedure 602 to perform, such as the procedure 200 of 2nd . In another aspect, the processor-executable computer instructions 604 be set up to implement a system, such as the scene classification system 100 of 1 . Many such computer readable media can be developed by those of ordinary skill in the art and can be arranged to operate in accordance with the techniques presented herein.

As used in this application, the terms "component", "module", "system", "interface" and the like are intended to refer generally to a computer-related entity, either hardware, a combination of hardware and software, software or software in execution. For example, a component can be, but is not limited to, a process running on a processor, processing unit, object, executable program, execution thread, program, or computer. As an illustration, both an application that is executed on a control device and the control device can be a component. One or more components that are in a process or execution thread and a component can be located on one computer or distributed across two or more computers.

Furthermore, the claimed subject matter is implemented as a method, apparatus, or article of manufacture using standard programming or development techniques to create software, firmware, hardware, or any combination thereof, to control a computer to implement the disclosed subject matter. As used herein, the term "article of manufacture" is intended to encompass a computer program that can be accessed by any computer-readable device, medium or medium. Of course, many modifications can be made to this configuration without departing from the scope or spirit of the claimed subject matter.

7 and the following discussion provides a description of a suitable computing environment to implement aspects of one or more of the measures set forth herein. The operating environment of 7 is only an example of a suitable operating environment and is not intended to suggest any restriction with regard to the area of application or the functionality of the operating environment. Exemplary computing devices include, but are not limited to, personal computers, server computers, portable devices or laptops, mobile devices such as cell phones, personal digital assistants (PDAs), media players, and the like, multiprocessor systems, consumer electronics, minicomputers, mainframes, distributed computing environments, any of the above systems or devices include, etc.

In general, aspects are described in the general context of "computer readable instructions" that are executed by one or more computing devices. Computer readable instructions can be distributed over computer readable media, as discussed below. Computer-readable instructions can be implemented as program modules such as functions, objects, application programming interfaces (APIs), data structures and the like, which perform one or more tasks or implement one or more abstract data types. Typically, the functionality of computer readable instructions is combined or distributed in different environments as desired.

7 illustrates a system 700 which is a computing device 712 that is configured to implement an aspect provided herein. In one configuration, the computing device closes 712 at least one processing unit 716 and a memory 718 on. Depending on the exact configuration and the type of computing device, the memory 718 volatile, such as RAM, non-volatile, such as ROM, flash memory, etc., or a combination of the two. This configuration is in 7 through the dashed line 714 illustrated.

In other respects, the computing device closes 712 additional features or Functions. For example, the computing device 712 include additional storage such as removable storage or non-removable storage including, but not limited to, magnetic storage, optical storage, etc. Such additional storage is disclosed in US Pat 7 through the mass storage 720 illustrated. In one aspect, computer readable instructions for implementing a point of view provided herein are in mass storage 720 . The mass storage 720 may store other computer readable instructions for implementing an operating system, application program, etc. Computer readable instructions can, for example, be executed by the processing unit 716 in the store 718 Loading.

The term "computer readable medium" as used herein includes computer storage media. Computer storage media include volatile and non-volatile, removable and non-removable media that are implemented in any method or technology for storing information such as computer readable instructions or other data. The memory 718 and the mass storage 720 are examples of computer storage media. Computer storage media include RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disks (DVDs) or other optical storage, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices or any other medium that can be used to store the save the desired information and on that from the computing device 712 can be accessed, but are not limited to this. Each such computer storage medium is part of the computing device 712 .

The term "computer-readable medium" includes communication media. Communication media typically embodies computer readable instructions or other data in a "modulated data signal", such as a carrier wave or other transport mechanism, and includes any information delivery media. The term “modulated data signal” includes a signal in which one or more of its properties are set or changed so that information is encoded in the signal.

The computing device 712 closes input device (s) 724 such as a keyboard, mouse, pen, voice input device, touch input device, infrared cameras, video input devices, or any other input device. The output device (s) 722 , such as one or more displays, speakers, printers, or any other output device, can be in the computing device 712 be included. The input device (s) 724 and the output device (s) 722 via a wired connection, a wireless connection or a combination thereof with the computing device 712 connected. In one aspect, an input device or an output device may be from a computing device other than the input device (s) 724 or dispenser (s) 722 for the computing device 712 be used. The computing device 712 can (a) communication link (s) 726 include to establish communications with one or more other devices 730 to facilitate, such as over the network 728 .

Although the subject matter has been described in a language specific to structural features or methodological processes, it should be understood that the subject matter of the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as exemplary points of view.

Various operations from an aspect are provided herein. The order in which one or more or all of the operations are described should not be construed to imply that these operations are necessarily order-dependent. Alternative orders are appreciated based on this description. Furthermore, not all operations are necessarily provided in every aspect.

As used in this application, "or" is intended to mean an inclusive "or" rather than an exclusive "or". Furthermore, an enclosing "or" can be any combination thereof (e.g. A , B or any combination thereof). In addition, "a" and "a" as used in this application should generally be understood to mean "one or more" unless otherwise stated or clearly related to a singular form from the context. In addition, at least one of A and B and / or the like in general A or B or both A as well as B . Furthermore, in that "includes", "has", "has", "with" or variants thereof in either the detailed description or in the claims, such a term is to be understood as such that it does so similar to the term "inclusive" is inclusive.

Furthermore, unless stated otherwise, “first”, “second” or the like should not be understood from a temporal point of view, a spatial point of view, an order, etc. Rather, such terms are only to be understood as identifiers, designations, etc. for features, elements, articles, etc. For example, a first channel and a second channel generally correspond to channel A and channel B or two different or two identical channels or the same channel. In addition, “comprehensive,” “includes,” “including,” “including,” or the like generally means, but is not limited to, inclusive or inclusive.

It is understood that various of the features and functions disclosed and other, or alternatives or variants thereof, disclosed above and other, preferably may be combined into many other different systems or applications. Also, that various currently unforeseen or unexpected alternatives, modifications, variations or improvements thereto can subsequently be made by those skilled in the art, which are also covered by the following claims.

Claims (20)

Scene classification system, comprising: an image capturing device that captures a first series of individual images of an environment from a moving vehicle; an image segmentation module that identifies one or more road users in the environment based on a first neural convolution network (CNN); an image masker that generates a second series of individual images by masking one or more of the road users from the surroundings; a time classifier that classifies one or more frames of the second series of frames with one of two or more temporal predictions and generates a third series of frames that are associated with respective temporal predictions based on a scene classification model, the classification being based on a second CNN, one Long Short-Term Memory (LSTM) network and a first fully connected layer based; and a scene classifier that classifies one or more individual images of the third series of individual images based on a third CNN, a global average pooling and a second completely connected layer and generates an associated scene prediction based on the scene classification model and corresponding temporal predictions. System for scene classification according to Claim 1 , where the two or more temporal predictions include an "approach" annotation, an "entry" annotation, and a "pass" annotation. System for scene classification according to Claim 1 , wherein the first CNN, the second CNN or the third CNN is a Deepnet CNN or a ResNet 50 CNN. System for scene classification according to Claim 1 , wherein the scene classification system is implemented in a vehicle and the vehicle includes a controller that enables or disables one or more sensors or one or more vehicle systems of the vehicle based on the scene prediction. System for scene classification according to Claim 4 , wherein the scene classifier classifies one or more frames of the third series of frames with a weather classification that includes clear, sunny, snow, rainy, overcast, or foggy; and wherein the controller enables or disables one or more of the sensors or one or more of the vehicle systems of the vehicle based on the weather classification. System for scene classification according to Claim 4 wherein the scene classifier classifies one or more frames of the third series of frames with a road surface classification including dry, wet or snow; and wherein the controller enables or disables one or more of the sensors or one or more of the vehicle systems of the vehicle based on the road surface classification. System for scene classification according to Claim 4 wherein the scene classifier classifies one or more frames of the third series of frames with an environmental classification including urban, up or down, highway or local; and wherein the controller enables or disables one or more of the sensors or one or more of the vehicle systems of the vehicle based on the environmental classification. System for scene classification according to Claim 4 , wherein one or more of the vehicle systems are a LIDAR system or radar system. System for scene classification according to Claim 8 The controller disables the lidar or radar system based on the scene prediction being a tunnel. System for scene classification according to Claim 4 , with the controller prioritizing the search for traffic lights, stop signs, stop lines based on the scene prediction being an intersection. A vehicle equipped with a scene classification system, comprising: an image capturing device that captures a first series of individual images of an environment from a moving vehicle; an image segmentation module that identifies one or more road users in the environment based on a first neural convolution network (CNN); an image masker that generates a second series of individual images by masking one or more of the road users from the surroundings; a time classifier that classifies one or more frames of the second series of frames with one of two or more temporal predictions and generates a third series of frames that are associated with respective temporal predictions based on a scene classification model, the classification being based on a second CNN, one Long Short-Term Memory (LSTM) network and a first fully connected layer based; and a scene classifier that classifies one or more frames of the third series of frames based on a third CNN, a global average pooling and a second fully connected layer and generates an associated scene prediction based on the scene classification model and corresponding temporal predictions; and a controller that enables or disables one or more sensors or one or more vehicle systems of the vehicle based on the scene prediction. Vehicle after Claim 11 , where the two or more temporal predictions may include an "approach" annotation, a "retract" annotation, and a "pass" annotation. Vehicle after Claim 11 , wherein the first CNN, the second CNN or the third CNN is a Deepnet CNN or a ResNet 50 CNN. Vehicle after Claim 11 wherein one or more of the vehicle systems are a LIDAR system or radar system and wherein the controller deactivates the LIDAR system or radar system based on the scene prediction being a tunnel. Scene classification system, comprising: an image capturing device that captures a first series of individual images of an environment from a moving vehicle; a time classifier that classifies one or more frames of the first series of frames with one of two or more temporal predictions and generates a second series of frames that are associated with respective temporal predictions based on a scene classification model, the classification being based on a convolution network CNN, one Long Short-Term Memory (LSTM) network and a first fully connected layer based; and a scene classifier that classifies one or more individual images of the second series of individual images based on a second CNN, a global average pooling and a second completely connected layer and generates an associated scene prediction based on the scene classification model and corresponding temporal predictions. System for scene classification according to Claim 15 , where the two or more temporal predictions include an "approach" annotation, a "retract" annotation, and a "pass" annotation. System for scene classification according to Claim 15 , wherein the CNN or the second CNN is a ResNet 50-CNN. System for scene classification according to Claim 15 , wherein the scene classification system is implemented in a vehicle and the vehicle includes a controller that enables or disables one or more sensors or one or more vehicle systems of the vehicle based on the scene prediction. System for scene classification according to Claim 18 , wherein the scene classifier classifies one or more frames of the third series of frames with a weather classification that includes clear, sunny, snow, rainy, overcast, or foggy; and wherein the controller enables or disables one or more of the sensors or one or more of the vehicle systems of the vehicle based on the weather classification. System for scene classification according to Claim 18 wherein the scene classifier classifies one or more frames of the third series of frames with a road surface classification including dry, wet or snow; and wherein the controller enables or disables one or more of the sensors or one or more of the vehicle systems of the vehicle based on the road surface classification.

Images