WO2019184825A1 - 车灯检测方法、实现智能驾驶的方法、装置、介质及设备 - Google Patents

车灯检测方法、实现智能驾驶的方法、装置、介质及设备 Download PDF

Info

Publication number
WO2019184825A1
WO2019184825A1 PCT/CN2019/079300 CN2019079300W WO2019184825A1 WO 2019184825 A1 WO2019184825 A1 WO 2019184825A1 CN 2019079300 W CN2019079300 W CN 2019079300W WO 2019184825 A1 WO2019184825 A1 WO 2019184825A1
Authority
WO
WIPO (PCT)
Prior art keywords
vehicle
lamp
image
image block
information
Prior art date
Application number
PCT/CN2019/079300
Other languages
English (en)
French (fr)
Inventor
刘诗男
曾星宇
闫俊杰
王晓刚
Original Assignee
北京市商汤科技开发有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京市商汤科技开发有限公司 filed Critical 北京市商汤科技开发有限公司
Priority to SG11201909146W priority Critical patent/SG11201909146WA/en
Priority to US16/588,126 priority patent/US10984266B2/en
Publication of WO2019184825A1 publication Critical patent/WO2019184825A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/08—Interaction between the driver and the control system
    • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/34—Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G06V20/584—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of vehicle lights or traffic lights
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/09—Arrangements for giving variable traffic instructions
    • G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
    • G08G1/0967—Systems involving transmission of highway information, e.g. weather, speed limits
    • G08G1/096733—Systems involving transmission of highway information, e.g. weather, speed limits where a selection of the information might take place
    • G08G1/09675—Systems involving transmission of highway information, e.g. weather, speed limits where a selection of the information might take place where a selection from the received information takes place in the vehicle
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/09—Arrangements for giving variable traffic instructions
    • G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
    • G08G1/0967—Systems involving transmission of highway information, e.g. weather, speed limits
    • G08G1/096766—Systems involving transmission of highway information, e.g. weather, speed limits where the system is characterised by the origin of the information transmission
    • G08G1/096791—Systems involving transmission of highway information, e.g. weather, speed limits where the system is characterised by the origin of the information transmission where the origin of the information is another vehicle
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems
    • G08G1/161—Decentralised systems, e.g. inter-vehicle communication
    • G08G1/162—Decentralised systems, e.g. inter-vehicle communication event-triggered
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems
    • G08G1/161—Decentralised systems, e.g. inter-vehicle communication
    • G08G1/163—Decentralised systems, e.g. inter-vehicle communication involving continuous checking
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks

Definitions

  • the present application relates to computer vision technology, and in particular to a vehicle lamp detection method, a vehicle lamp detection device, a neural network training method, a neural network training device, a method for realizing intelligent driving, a device for realizing intelligent driving, Electronic devices, computer readable storage media, and computer program products.
  • the lights on the vehicle can often play the role of other information (such as left turn, right turn or brake) to other vehicles and pedestrians. Accurately identifying the information conveyed by the lights of other vehicles in the driving environment is of certain significance for the decision-making of intelligent driving.
  • Embodiments of the present application provide a vehicle lamp detection, a training neural network, and a technical solution for implementing smart driving.
  • a vehicle lamp detecting method comprising: acquiring an image block including a vehicle; performing vehicle lamp detection on the image block via a depth neural network to obtain a vehicle light detection result .
  • a training method of a neural network comprising: acquiring a sample image block including a vehicle; and performing a car on the sample image block via a deep neural network to be trained The light is detected to obtain a vehicle light detection result; and the difference between the vehicle light detection result and the vehicle image labeling information of the sample image block is used as guiding information, and the deep neural network to be trained is supervised and learned.
  • a method for implementing smart driving comprising: acquiring an image block including a vehicle; performing vehicle light detection on the image block via a deep neural network to obtain a vehicle light detection result; determining vehicle light indication information according to a vehicle light detection result of the plurality of image blocks having a time series relationship; and generating driving control information or driving warning prompt information according to the vehicle light instruction information.
  • a vehicle lamp detecting apparatus comprising: an image block module for acquiring an image block including a vehicle; and a first car light detecting module for transmitting a deep nerve The network performs vehicle light detection on the image block to obtain a vehicle light detection result.
  • a training apparatus for a neural network comprising: acquiring a sample image block module for acquiring a sample image block including a vehicle; and a second vehicle light detection module, Performing vehicle lamp detection on the sample image block to obtain a vehicle lamp detection result via a deep neural network to be trained; and a supervision module for using the vehicle lamp detection result and the car light labeling information of the sample image block The difference between the two is the guidance information, and the supervised learning of the deep neural network to be trained is performed.
  • an apparatus for implementing smart driving comprising: an image block module for acquiring an image block including a vehicle; and a first vehicle light detecting module, Performing vehicle light detection on the image block via a deep neural network to obtain a vehicle light detection result; and determining a vehicle light indication module, configured to determine the vehicle light indication information according to the vehicle light detection result of the plurality of image blocks having a timing relationship And an intelligent control module, configured to generate driving control information or driving warning prompt information according to the vehicle light indication information.
  • an electronic device includes: a memory for storing a computer program; a processor for executing a computer program stored in the memory, and when the computer program is executed, Any method embodiment of the present application.
  • an electronic device including: a processor and any device implementation of the present application; when the processor runs the device, the unit in any of the device embodiments of the present application is executed.
  • a computer readable storage medium having stored thereon a computer program, which when executed by a processor, implements any of the method embodiments of the present application.
  • a computer program comprising computer instructions that, when executed in a processor of a device, implement any of the method embodiments of the present application.
  • a computer program product comprising computer readable code, the processor in the device executing the implementation of the present application when the computer readable code is run on a device A method implementation.
  • the above-described vehicle lamp detecting method, vehicle lamp detecting device, neural network training method, neural network training device, method for realizing intelligent driving, device for realizing intelligent driving, electronic device, and computer readable are provided based on the present application.
  • the storage medium and the computer program, the present invention uses the deep neural network to perform the vehicle light detection on the image block including the vehicle, which is convenient for obtaining the vehicle light detection result of the image block quickly and accurately, thereby facilitating the detection of the lamp of the present application.
  • the technology is applied in the real-time environment of intelligent driving such as automatic driving and assisted driving, which is beneficial to improve the decision accuracy or early warning accuracy of intelligent driving.
  • FIG. 1 is a flowchart of a method for detecting a vehicle lamp according to an embodiment of the present application.
  • FIG. 2 is a schematic diagram of an implementation of an image to be processed according to an embodiment of the present application.
  • FIG. 3 is a flowchart of a training method of a neural network according to an embodiment of the present application.
  • FIG. 4 is a flowchart of a method for implementing smart driving according to an embodiment of the present application.
  • FIG. 5 is a schematic diagram of an embodiment of a vehicle lamp detection result according to an embodiment of the present application.
  • FIG. 6 is a schematic diagram of another embodiment of a vehicle lamp detection result according to an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of a vehicle lamp detecting device according to an embodiment of the present application.
  • FIG. 8 is a schematic structural diagram of a training apparatus for a neural network according to an embodiment of the present application.
  • FIG. 9 is a schematic structural diagram of an apparatus for implementing smart driving according to an embodiment of the present application.
  • FIG. 10 is a block diagram of an exemplary apparatus that implements an embodiment of the present application.
  • Embodiments of the present application can be applied to computer systems/servers that can operate with numerous other general purpose or special purpose computing system environments or configurations.
  • Examples of well-known computing systems, environments, and/or configurations suitable for use with computer systems/servers include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, based on Microprocessor systems, set-top boxes, programmable consumer electronics, networked personal computers, small computer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above, and the like.
  • the computer system/server can be described in the general context of computer system executable instructions (such as program modules) being executed by a computer system.
  • program modules may include routines, programs, target programs, components, logic, data structures, and the like that perform particular tasks or implement particular abstract data types.
  • the computer system/server can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communication network.
  • program modules may be located on a local or remote computing system storage medium including storage devices.
  • FIG. 1 is a flowchart of a method for detecting a vehicle lamp according to an embodiment of the present application.
  • the method can be performed by any electronic device, such as a terminal device, a server, a mobile device, an in-vehicle device, or the like.
  • the method of this embodiment includes:
  • the step S100 may be performed by a processor invoking a corresponding instruction stored in the memory, or may be performed by the acquired image block module 700 executed by the processor.
  • S110 Perform a vehicle light detection on the image block via a deep neural network to obtain a vehicle light detection result.
  • the step S110 may be performed by a processor invoking a corresponding instruction stored in the memory, or may be performed by the first lamp detection module 710 operated by the processor.
  • the embodiment of the present invention facilitates the vehicle lamp detection result of the image block including the vehicle by using the deep neural network, which is convenient for obtaining the vehicle lamp detection result of the image block quickly and accurately, thereby facilitating the lamp detection technology provided by the embodiment of the present application. It is applied to the real-time environment of intelligent driving such as automatic driving and assisted driving, which is beneficial to improve the decision accuracy or early warning accuracy of intelligent driving.
  • the image block including the vehicle in the embodiment of the present application may be the entire image to be processed, or may be a partial image including the vehicle in the image to be processed.
  • the image block including the vehicle in the present application may also be an image block obtained by processing a partial image including the vehicle in the image to be processed.
  • the image to be processed in the embodiment of the present application may be an image that presents a static picture or a photo, or may be a video frame in a dynamic video, for example, captured by a camera set in a vehicle.
  • Video frames in the video may be a vehicle that is taken by the camera device at the rear of the vehicle (including the rear side and the rear side, etc.), or may be in front of the vehicle (including the front side of the vehicle) that the camera device picks up.
  • the vehicle in the front side or the like may also be a vehicle that is taken by the imaging device on the side of the vehicle on which it is located.
  • the present application does not limit the specific positional relationship of the vehicle in the image to be processed with respect to the vehicle in which the camera device is located.
  • the image block is the image to be processed
  • the image block is a partial image of the vehicle included in the image to be processed, or
  • the image block is an image block obtained based on partial image processing including a vehicle in the image to be processed.
  • the vehicle included in the image block of the embodiment of the present application may be a complete vehicle (such as the rightmost vehicle in FIG. 2 ), or may be a partial partial vehicle (ie, a vehicle) due to occlusion or the like. Partially, as shown in the left side of Figure 2, multiple vehicles, etc.). This application does not limit the specific manifestation of the vehicle in the image to be processed.
  • obtaining an image block containing a vehicle includes:
  • the image to be processed is segmented to obtain an image block containing the vehicle.
  • the embodiment of the present application can perform vehicle detection on the image to be processed, thereby obtaining a vehicle detection result, and further performing segmentation processing on the image to be processed according to the vehicle detection result, so that an image block including the vehicle can be obtained, thereby facilitating improvement.
  • the accuracy of the lamp detection In the case where a plurality of vehicles are included in the image to be processed, the present application can segment a plurality of image blocks containing the vehicle from the image to be processed.
  • the above vehicle detection result may be vehicle cradle information, for example, coordinates of two vertices located on the diagonal of the circumference frame of the vehicle, and the like.
  • vehicle detection of the image to be processed includes:
  • Vehicle detection is performed on the image to be processed using the vehicle detection model to obtain vehicle enclosure information in the image to be processed.
  • the embodiment of the present application can use the neural network to perform vehicle detection on the image to be processed, so that the vehicle detection result can be obtained according to the information output by the neural network.
  • the embodiment of the present application may perform vehicle detection processing on a to-be-processed image using a vehicle detection model (which may be a neural network for vehicle detection) or the like.
  • the vehicle detection model in the embodiment of the present application may be a Regions Convolutional Neural Network (R-CNN), a Fast Region Convolutional Neural Network (Fast R-CNN), or a faster one.
  • Regional Convolutional Neural Network Faster R-CNN
  • the present application does not limit the vehicle detection process to be processed to obtain a specific implementation of an image block containing the vehicle.
  • the present application also does not limit the specific structure of the vehicle detection model (such as a neural network for vehicle detection) and the like.
  • the embodiment of the present application detects the vehicle from the image to be processed by using the vehicle detection model, and cuts out the image block including the vehicle from the image to be processed, which is beneficial for reducing the detection of the non-vehicle information in the image to be processed. Interference, which is beneficial to reduce the false detection rate of the lamp detection, and thus is beneficial to improve the accuracy of the detection of the lamp.
  • the size of the image block in the embodiment of the present application is generally related to the size requirement of the input image by the deep neural network.
  • the size of the image block may be 256 ⁇ 256 or the like.
  • the deep neural network in this application is a deep neural network for vehicle light detection.
  • the embodiment of the present application may first perform a scaling process on the image to be processed, and then segment the image block containing the vehicle from the image to be processed after the scaling process according to a predetermined size.
  • the present application can perform scaling processing on the image to be processed based on the vehicle detection result (such as vehicle rim information) so that the cut image block containing the vehicle has a predetermined size.
  • the embodiment of the present application may separately perform scaling processing on the image to be processed for at least one vehicle circumscribing frame information, thereby The at least one image block obtained by the image to be processed has a predetermined size, respectively.
  • the segmentation process may be performed first, and then the scaling process may be performed.
  • the embodiment of the present application does not limit the size of the image block and the specific implementation of the scaling process.
  • vehicle detection of the image to be processed further includes:
  • a vehicle outer frame miss detection and/or a vehicle outer frame smoothing process is performed to obtain vehicle outer frame information in the corrected plurality of to-be-processed images.
  • the embodiment of the present application may perform at least one of a vehicle outer frame miss detection and a vehicle outer frame smoothing process for a vehicle detection result output by a neural network (such as a vehicle detection model) (eg, performing a vehicle before performing segmentation processing on the image to be processed)
  • a neural network such as a vehicle detection model
  • the missed detection of the external frame and the smooth processing of the external frame of the vehicle can improve the accuracy of the vehicle detection result and further improve the accuracy of the detection result of the vehicle lamp.
  • multiple video frames usually have a timing relationship (such as consecutively arranged video frames in the video, and then framed for video, based on framed frames)
  • a timing relationship such as consecutively arranged video frames in the video, and then framed for video, based on framed frames
  • vehicle outer frame miss detection and vehicle outer frame smoothing processing for a plurality of video frames having a time series relationship (ie, an image to be processed), thereby Obtaining vehicle rim information of the vehicle detection model missed detection, and correcting the position of the vehicle cradle information output by the vehicle detection model.
  • the vehicle circumstance that is not detected in the n2th video frame is complemented according to the vehicle circumsole in the plurality of video frames before the n2th video frame; and, for example, according to the plurality of frames before the n2th video frame
  • the vehicle cradle in the video frame corrects the position of the corresponding vehicle cradle in the n2th video frame.
  • the corresponding to-be-processed image may be separately segmented according to the vehicle outer frame missed detection and the plurality of vehicle outer frame information after the vehicle outer frame smoothing process, thereby obtaining a plurality of vehicles including the time series relationship. Image block.
  • performing vehicle rim miss detection and/or smoothing processing for a plurality of the to-be-processed images having a timing relationship includes:
  • the embodiment of the present application may first obtain image blocks (ie, vehicle enclosure information) of a plurality of to-be-processed images in a time-series relationship, and then perform vehicle external connection for a plurality of to-be-processed images and vehicle enclosure information in which a timing relationship exists.
  • Image blocks ie, vehicle enclosure information
  • vehicle external connection for a plurality of to-be-processed images and vehicle enclosure information in which a timing relationship exists.
  • an optional example of obtaining an image block in a plurality of to-be-processed images having a timing relationship in the embodiment of the present application is: at least a plurality of to-be-processed images and a plurality of to-be-processed images having a timing relationship
  • the vehicle circumscribing frame information in an image to be processed is respectively provided to the vehicle tracking model (the vehicle tracking model may be an object tracking model of the vehicle), and the embodiment of the present application may obtain at least one according to the information output by the vehicle tracking model.
  • the vehicle identification corresponding to the vehicle's external frame Therefore, the embodiment of the present application can successfully establish an association relationship between at least one vehicle and a corresponding image to be processed.
  • the vehicle identifier corresponding to the vehicle enclosure may also be referred to as an image block identifier, including the same vehicle.
  • the image block identification of an image block is generally the same, while the image block identification of image blocks containing different vehicles is typically different, thereby facilitating vehicle light detection of a vehicle in a multi-vehicle situation.
  • the vehicle tracking model in the embodiment of the present application may be a convolutional neural network (CNN) or a recurrent neural network (RNN) or the like based on two video frames that are adjacent to each other.
  • CNN convolutional neural network
  • RNN recurrent neural network
  • performing vehicle outer frame miss detection and/or vehicle outer frame smoothing processing on at least one of the plurality of to-be-processed images according to the vehicle identifier corresponding to the at least one vehicle enclosure comprises:
  • a vehicle identification For a vehicle identification, predicting a vehicle enclosure having a vehicle identification in the m2th to-be-processed image according to a location of a vehicle cradle having a vehicle identification among n2 to-be-processed images preceding the m2th image to be processed s position;
  • the mth to-be-processed image is subjected to vehicle fascia addition or vehicle rim position correction processing.
  • the embodiment of the present application may perform the vehicle outer frame miss detection and the vehicle outer frame smoothing processing according to the plurality of to-be-processed images and the image block identifiers of the plurality of to-be-processed images, for example, the following formula (1) may be adopted.
  • the current video frame performs vehicle miss frame inspection and vehicle outer frame smoothing processing:
  • a 1 , a 2 , a 3 , a 4 , b 1 , b 2 , b 3 , b 4 , c 1 , c 2 , c 3 and c 4 are all linear equations (ie, formulas) (1))
  • ⁇ x represents the difference between the center point abscissa of the vehicle cradle in the previous video frame and the center point abscissa of the vehicle cradle having the same vehicle identification in the next video frame
  • ⁇ y indicating the previous video frame
  • performing vehicle cradle addition or vehicle cradle position correction processing on the m2th to-be-processed image includes:
  • weighted averaging processing is performed on the predicted position of the vehicle circumscribing frame and the position of the vehicle circumscribing frame having the vehicle identification in the m2th to-be-processed image.
  • the present application may predict that the current video frame has the location according to a location of a vehicle cradle having the vehicle identity in a previous video frame of the current video frame and ⁇ x, ⁇ y, ⁇ w, and ⁇ h corresponding to the calculated current video frame.
  • the embodiment of the present application may add a vehicle cradle to the current video frame according to the predicted location of the vehicle circumscribing frame.
  • the vehicle outer frame miss detection processing of the current video frame is implemented; if the current video frame has a vehicle outer frame having the vehicle identifier, the embodiment of the present application may be configured for the predicted location of the vehicle outer frame and the current video frame. The location of the vehicle cradle of the vehicle identification is weighted averaged to achieve the vehicle identification in the current video frame Smoothing the outer frame of the vehicle.
  • other missed detection processing and smoothing processing techniques can be used to perform the missed detection processing of the vehicle circumscribing frame and the smoothing processing of the vehicle circling frame for a plurality of video frames.
  • the embodiment of the present application does not limit the specific implementation manner of the vehicle outer frame miss detection processing and the vehicle outer frame smoothing processing for the image to be processed.
  • the embodiment of the present application also does not limit the specific structure and the like of the vehicle tracking model.
  • the deep neural network in the embodiment of the present application may use a deep neural network based on region detection.
  • a Regions Convolutional Neural Network RCNN
  • a Fast Region Convolutional Neural Network may be used.
  • Network Faster RCNN and so on.
  • the network structure of the deep neural network may be flexibly designed according to the actual requirements for extracting the detection result of the vehicle lamp.
  • the embodiment of the present application does not limit the specific network structure of the deep neural network; for example, the deep neural network in the embodiment of the present application may include It is not limited to a convolutional layer, a non-linear activation (Relu) layer, a pooled layer, a fully connected layer, etc., and the deeper the number of layers included in the deep neural network, the deeper the network; for example, the depth of the embodiment of the present application
  • the network structure of the neural network may adopt, but is not limited to, a network structure adopted by a neural network such as an ALexNet, a Deep Residual Network (ResNet), or a Visual Geometry Group Network (VGGnet).
  • the deep neural network in the embodiment of the present application is obtained by training the sample image block with the information of the headlights, so that the trained deep neural network has the ability to accurately detect the state of the light.
  • the vehicle light labeling information of the sample image block may include one or more of the vehicle light strip frame information labeling information, the vehicle light on and off status labeling information, and the headlight labeling information.
  • the vehicle light detection result output by the deep neural network of the embodiment of the present application may include, but is not limited to, at least one of a vehicle light enclosure information, a vehicle light on state, and a vehicle light orientation.
  • the deep neural network outputs the car's external frame information, the lights on and off, and the headlights.
  • the information of the outer frame of the lamp is generally referred to as information indicating the position of the outer frame of the lamp in the image block (such as the coordinates of the two vertices on the diagonal of the outer frame of the lamp) on the image block, and further
  • the application embodiment may determine the position of the vehicle light in the image to be processed (such as the coordinates of the two vertices on the diagonal of the outer frame of the vehicle light on the image to be processed) according to the position of the outer frame of the lamp light in the image block.
  • the information of the outer cover of the vehicle output from the deep neural network can also directly be the information of the position of the outer frame of the lamp in the image to be processed (for example, the two vertices on the diagonal of the outer frame of the lamp are on the image block to be processed. coordinate of).
  • the state in which the lights are off can usually indicate whether the lights are on or off.
  • the orientation of the lamp is usually used to indicate the orientation of the lamp in the vehicle.
  • the orientation of the lamp can be: left front light, right front light, left rear light or right rear light.
  • the method before determining the vehicle light indication information according to the vehicle light detection result of the plurality of image blocks in which the timing relationship exists, the method further includes:
  • the vehicle lamp outer frame miss detection and/or the vehicle light outer frame smoothing process are performed according to the vehicle lamp detection result to obtain the lamp outer frame information in the corrected plurality of image blocks.
  • the embodiment of the present application can perform at least one of a vehicle lamp external frame miss detection and a vehicle light external frame smoothing process for the vehicle light detection result output by the deep neural network, thereby facilitating improvement of the accuracy of the vehicle light detection result.
  • a plurality of image blocks generally have a timing relationship (eg, at least one of a plurality of video frames consecutively arranged in the video, for example, for The image is subjected to frame drawing, at least one image block of the plurality of consecutively extracted video frames formed based on the result of the frame drawing, and the like, and the embodiment of the present application can perform the car light external frame for the image blocks having the timing relationship.
  • the missed detection and the external frame of the lamp are smoothed, so that the information of the outer frame of the vehicle for the deep neural network missed detection can be obtained, and the position information of the outer frame of the lamp output by the deep neural network can be corrected.
  • the unmanned headlight enclosure in the n1th image block is complemented according to the headlight enclosure in the plurality of image blocks preceding the n1th image block; for example, based on the n1th image block
  • the lamp circumscribing frame in the plurality of image blocks corrects the position of the corresponding lamp circumscribing frame in the n1th image block.
  • the vehicle light indication information analysis may be performed based on the missed detection of the outer frame of the vehicle lamp and the at least one outer frame of the lamp after the smooth processing of the outer frame of the lamp and other information such as the state of the light on and off.
  • the vehicle lamp enclosure miss detection and/or the vehicle lamp enclosure smoothing processing according to the vehicle lamp detection result includes:
  • the embodiment of the present application may first obtain the information of the outer cover of the vehicle in the plurality of image blocks having the time series relationship (such as the coordinates of the two vertices on the diagonal of the outer frame of the lamp), and then, in the presence of The plurality of image blocks of the time series relationship and the at least one headlight frame information of the vehicle are subjected to a missed detection process of the lamp outer frame and a smoothing process of the lamp outer frame.
  • the time series relationship such as the coordinates of the two vertices on the diagonal of the outer frame of the lamp
  • the embodiment of the present application may provide a plurality of image blocks having a time-series relationship and the car-outlight frame information in the image block to the vehicle lamp tracking model respectively (the vehicle lamp tracking model may be the target object for the vehicle)
  • the object tracking model of the lamp can be obtained according to the information output by the vehicle lamp tracking model, and the vehicle lamp identifier corresponding to the at least one lamp outer frame is obtained. Therefore, the embodiment of the present application can successfully establish an association relationship between the vehicle lamp and the vehicle.
  • Different headlight enclosures containing the same headlight logo generally correspond to the same headlight of the same vehicle.
  • the vehicle light enclosure miss detection and/or the vehicle exterior enclosure smoothing process for at least one of the plurality of image blocks according to the vehicle light identifier corresponding to the at least one vehicle light enclosure comprises:
  • predicting a headlight with a headlight identifier in the m1th image block according to the position of the headlight enclosure of the n1 image block located before the m1th image block The location of the external frame;
  • the m1 image block is subjected to a lamp circumscribing frame addition or a lamp circumscribing frame position correction process.
  • the embodiment of the present application may perform the missed detection processing of the external bracket of the vehicle lamp and the smoothing processing of the external frame of the vehicle lamp according to the plurality of image blocks having the time series relationship and the lamp indicator of the at least one outer frame of the lamp included in the plurality of image blocks.
  • the vehicle light outer frame miss detection processing and the vehicle light outer frame smoothing processing may be performed for the current image block by using the above formula (1).
  • ⁇ x represents the center point abscissa of the headlight enclosure in the previous image block and the vehicle with the same headlight identifier in the next image block.
  • ⁇ y indicates the center point ordinate of the lamp circumscribing frame in the previous image block and the center of the lamp circumscribing frame with the same lamp logo in the next image block
  • ⁇ w represents the difference between the width of the circumscribing rim of the previous image block and the width of the circumscribing fascia of the vehicle with the same illuminator in the next image block
  • ⁇ h represents The height of the headlight enclosure in the previous image block has the same headlight ID as in the next image block The difference between the height of the lights external box.
  • the addition of the illuminating frame to the m1th image block or the position correction process of the illuminating circumscribing frame includes:
  • the illuminating outer frame is added to the m1th image block according to the predicted position of the illuminating frame of the illuminating lamp;
  • the position of the predicted lamp circumscribing frame and the position of the lamp lamp circumscribing frame having the lamp lamp identifier in the m1th image block are performed. Weighted average processing.
  • the linear equation parameters a 1 , a 2 , a 3 , a 4 , b 1 , b 2 , b 3 , b 4 , c 1 , c 2 , c 3 , and c 4 in the above formula (1) are calculated. Therefore, in the present application, the ⁇ x, ⁇ y, ⁇ w, and ⁇ h of the current image block can be calculated by using the formula (1) in the case where the linear equation parameter of the above formula (1) is known.
  • the present application is implemented.
  • the current image block may be predicted according to the position of the headlight circumspores of the previous image block of the current image block and the ⁇ x, ⁇ y, ⁇ w, and ⁇ h corresponding to the calculated current image block.
  • the position of the lamp circumscribing frame of the lamp light if there is no illuminating frame of the lamp lamp having the lamp lamp identifier in the current image block, the application can add the current image block according to the predicted position of the lamp circumscribing frame The vehicle lamp circumscribing frame, so as to realize the missed detection processing of the headlight external frame of the current image block; if the current image block has the lamp lamp circumscribing frame having the lamp lamp identifier, the present application can be directed to the predicted position of the lamp lamp circling frame Performing weighted averaging with the position of the circumscribing frame of the vehicle with the illuminating sign in the current image block, thereby achieving smoothing of the circumscribing frame of the vehicle with the illuminating sign in the current image block Management.
  • the existing missed detection processing and the smoothing processing technology may be used to perform the missed detection processing of the vehicle light enclosure and the smoothing processing of the vehicle light enclosure for at least one image block of at least one frame of the video frame.
  • the embodiment of the present application does not limit the specific implementation manner of the vehicle light outer frame miss detection processing and the vehicle light outer frame smoothing processing for the image block in the image to be processed. This application also does not limit the specific structure of the vehicle head tracking model.
  • the embodiment of the present application may determine the vehicle light indication information according to the vehicle light detection result of the plurality of image blocks in which the timing relationship exists.
  • the vehicle light indication information includes:
  • the present application can perform statistics on the on and off state of the same vehicle of the same vehicle in a period of time, and judge and identify the statistical result, thereby It is known whether the same vehicle light of the same vehicle is in a blinking state, a long-light state or a long-off state, and the embodiment of the present application can determine the vehicle of the vehicle in combination with the orientation of the vehicle lamp and the state of other lights of the same vehicle.
  • the light indication information is: one-side left car light flashes, one-side right car light flashes, two-side car lights flash, both side lights are completely off, or both side lights are fully lit.
  • One-sided left-handlights flashing usually indicate that the vehicle is moving to the left (such as left-hand or left-turn).
  • One-sided right lights flashing (left lights are off) usually indicate that the vehicle is moving to the right (such as right-hand or right-turn).
  • Blinking of the two-sided lights usually indicates that the vehicle has special conditions such as emergency or temporary parking. When the two side lights are completely off, it means that the vehicle does not change direction normally. The full illumination of the two side lights indicates that the brakes are slowing down.
  • determining the vehicle light indication information according to the vehicle light detection result of the plurality of image blocks having the timing relationship includes:
  • the vehicle lights are off and on, and the vehicle light indication information is determined according to the statistical result.
  • the state of the vehicle lamp can be classified more carefully by using the timing information of the instantaneous state of the same vehicle lamp.
  • the state of the two lights of the same car such as two rear lights
  • the embodiment of the present application can use the following formulas (2) to (5) to implement statistics on the on and off states of the same vehicle of the same vehicle over a period of time:
  • i denotes the i-th stage, one stage usually includes a plurality of times, s n represents the on-off state of the lamp at the nth time, and the value of s n is 0, indicating that the car The lamp is in the off state at the nth time, and the value of s n is 1, indicating that the lamp is in the bright state at the nth time; C i is the number of times the i-th phase is in the light state.
  • a i is used to represent different values of C i by different values.
  • D m represents the number of A i having a value of m.
  • S t represents the on-off state of the vehicle lamp for a period of time; if the value of S t is 2, it indicates that the vehicle lamp is in a long-on state, and if the value of S t is 1, It means that the lamp is in a flashing state. If the value of S t is 0, it means that the lamp is in the long-off state.
  • FIG. 3 is a flowchart of a training method of a neural network according to an embodiment of the present application.
  • the method can be performed by any electronic device, such as a terminal device, a server, a mobile device, an in-vehicle device, etc., as shown in FIG. 3, the method of this embodiment includes: step S300, step S310, and step S320. The steps in Fig. 3 will be described in detail below.
  • the step S300 may be performed by a processor invoking a corresponding instruction stored in the memory, or may be performed by the acquisition sample image block module 800 executed by the processor.
  • S310 Perform a vehicle light detection on the sample image block via the deep neural network to be trained to obtain a vehicle light detection result.
  • the step S310 may be performed by a processor invoking a corresponding instruction stored in the memory, or may be performed by a second headlight detection module 810 that is executed by the processor.
  • S320 taking the difference between the detection result of the lamp and the marking information of the sample image block as the guidance information, and supervising and learning the deep neural network to be trained.
  • step S320 may be performed by the processor invoking a corresponding instruction stored in the memory, or may be performed by the supervisory module 820 executed by the processor.
  • the sample image block including the vehicle in the embodiment of the present application may be the entire image sample, or may be a partial image of the image sample including the vehicle.
  • the sample image block including the vehicle in the embodiment of the present application may also be a sample image block obtained by processing a partial image including the vehicle in the image sample.
  • an image sample is taken from a training data set and a sample image block containing the vehicle in the image sample is acquired.
  • the training data set in the embodiment of the present application includes a plurality of image samples for training a deep neural network.
  • each image sample is provided with vehicle lamp labeling information, and the vehicle lamp labeling information may include but is not limited to: a lamp At least one of the external frame information labeling information, the vehicle light-on state labeling information, and the vehicle heading labeling information.
  • the embodiment of the present application may read one or more image samples from the training data set at a time according to a random reading manner or a sequential reading manner according to an image sample arrangement order.
  • the embodiment of the present application may acquire a sample image block including a vehicle in a plurality of manners.
  • the embodiment of the present application may use a neural network to acquire an image block including a vehicle in an image sample.
  • the embodiment of the present application can obtain the vehicle detection result by performing vehicle detection on the image sample, and then, according to the vehicle detection result, the image sample is subjected to the segmentation process, and the sample image block including the vehicle can be obtained.
  • embodiments of the present application may slice a plurality of sample image blocks containing the vehicle from the image samples.
  • the above vehicle detection result may be vehicle cradle information, for example, coordinates of two vertices located on the diagonal of the circumference frame of the vehicle, and the like.
  • the embodiment of the present application may perform vehicle detection on the read image samples by using a vehicle detection model (which may also be referred to as an object detection model of the vehicle).
  • vehicle detection model which may also be referred to as an object detection model of the vehicle.
  • Embodiments of the present application do not limit vehicle detection of image samples to obtain a specific implementation of a sample image block containing a vehicle.
  • the size of the sample image block in the embodiment of the present application is generally related to the size requirement of the input image by the deep neural network.
  • the size of the sample image block may be 256 ⁇ 256 or the like.
  • the embodiment of the present application may perform scaling processing on the image sample according to the vehicle detection result after performing vehicle detection, and then, according to the predetermined size and the vehicle enclosure information (such as the center position of the vehicle enclosure) ), the sample image block including the vehicle is cut out from the scaled image sample.
  • the embodiment of the present application may separately perform scaling processing on the image samples for at least one vehicle circling frame information, so that the obtained at least one sample image block has respectively Scheduled size.
  • the embodiment of the present application does not limit the size of the sample image block and the specific implementation of the scaling process.
  • the deep neural network to be trained in the embodiment of the present application performs vehicle light detection on at least one sample image block that is input, and outputs a vehicle light detection result, for example, outputting a light bulb external frame information. At least one of a light-off state and a headlight orientation. Under normal circumstances, the deep neural network to be trained outputs the lamp enclosure information, the lamp on and off state, and the headlight orientation for at least one sample image block.
  • the information of the outer frame of the lamp light generally refers to information capable of indicating the position of the outer frame of the lamp light in the sample image block (such as the coordinates of the two vertices on the diagonal of the outer frame of the lamp light on the sample image block).
  • the embodiment of the present application can determine the position of the lamp in the image sample according to the position of the lamp circumference frame in the image block (such as the coordinates of the two vertices on the diagonal of the illuminating frame of the illuminator on the image sample).
  • the information of the lamp enclosure of the deep neural network to be trained can also directly be the information of the location of the car circumscribing frame in the image sample.
  • the state in which the lights are off can usually indicate whether the lights are on or off.
  • the orientation of the lamp is usually used to indicate the orientation of the lamp in the vehicle.
  • the orientation of the lamp can be: left front light, right front light, left rear light or right rear light.
  • the embodiment of the present application may use the difference between the headlight detection result of the at least one sample image block output by the deep neural network to be trained and the headlight labeling information of the corresponding sample image block as the guidance information,
  • the corresponding loss function is used to supervise the deep neural network to be trained.
  • the training process ends when the training for the deep neural network to be trained reaches a predetermined iteration condition.
  • the predetermined iteration condition in the embodiment of the present application may include that a difference between the vehicle light detection result output by the deep neural network to be trained and the vehicle light labeling information of the image sample satisfies a predetermined difference requirement. In the case that the difference satisfies the predetermined difference requirement, the successful training of the deep neural network to be trained this time is completed.
  • the predetermined iteration condition in the embodiment of the present application may also include: training the deep neural network to be trained, the number of used image samples reaches a predetermined number requirement, and the like.
  • the deep neural network to be trained this time is not successfully trained.
  • the deep neural network that is successfully trained can be used for the vehicle light detection processing of the image block containing the vehicle in the image to be processed.
  • FIG. 4 is a flowchart of a method for implementing smart driving according to an embodiment of the present application.
  • the method can be performed by any electronic device, such as a terminal device, a server, a mobile device, an in-vehicle device, etc., as shown in FIG. 4, the method of this embodiment includes: step S400, step S410, step S420, and step S430.
  • step S400 the method of this embodiment includes: step S400, step S410, step S420, and step S430.
  • the step S400 may be performed by a processor invoking a corresponding instruction stored in the memory, or may be performed by the acquired image block module 700 executed by the processor.
  • the step S410 may be performed by the processor invoking a corresponding instruction stored in the memory, or may be performed by the first lamp detection module 710 operated by the processor.
  • S420 determines the vehicle light indication information according to the vehicle light detection result of the plurality of image blocks in which the timing relationship exists.
  • the step S420 may be performed by a processor invoking a corresponding instruction stored in the memory, or may be performed by the determined headlight indication module 720 operated by the processor.
  • step S430 may be performed by the processor invoking a corresponding instruction stored in the memory or by the intelligent control module 730 being executed by the processor.
  • the vehicle light detection result may include, but is not limited to, at least one of a vehicle light enclosure information, a vehicle light on state, and a vehicle light orientation.
  • the deep neural network outputs the car's external frame information, the lights on and off, and the headlights.
  • the vehicle light indication information in the embodiment of the present application may include: a single-side left vehicle light flashes, a single-side right vehicle light flashes, a double-side vehicle light flashes, a double-side vehicle light is completely off, or a double-sided vehicle The lights are all on and off.
  • the embodiment of the present application may further distinguish whether the left headlight is a left front light or a left rear light.
  • the light indication information may include: one-side left front light flashes, one-side left rear light flashes, one-side right front light flashes, one-side right rear light blinks, two-side front light blinks, two sides
  • the rear lights flash the front lights on both sides are completely off, the rear lights on both sides are completely off, the front lights on both sides are fully illuminated, and the rear lights on both sides are fully illuminated.
  • FIG 5 An alternative example of full illumination of the double-sided rear lights is shown in Figure 5.
  • the "on” on the left side of Figure 5 indicates that the left rear light is on, and the “on” on the right side of Figure 5 indicates that the right rear light is on.
  • An optional example of single-sided right rear light flashing is shown in Figure 6.
  • the "off” on the left side of Figure 6 indicates that the left rear light is off, and the "flash" on the right side of Figure 6 indicates the right rear light. flicker.
  • the driving control information generated by the embodiment of the present application may include: deceleration driving control information, left parallel line control information, right parallel line control information, maintaining current speed driving control information, or acceleration. Driving control information, etc.
  • the generated driving warning prompt information may include: front vehicle collocation prompt information, front vehicle deceleration prompt information, front vehicle left/right steering prompt information, and the like. The embodiment of the present application does not limit the specific expression form of the driving control information and the driving warning prompt information.
  • the foregoing program may be stored in a computer readable storage medium, and the program is executed when executed.
  • the foregoing steps include the steps of the foregoing method embodiments; and the foregoing storage medium includes: a medium that can store program codes, such as a ROM, a RAM, a magnetic disk, or an optical disk.
  • FIG. 7 is a schematic structural diagram of a vehicle lamp detecting device according to an embodiment of the present application.
  • the device of this embodiment can be used to implement the vehicle lamp detecting method provided by any of the above embodiments of the present application.
  • the apparatus of this embodiment includes an acquisition image block module 700 and a first vehicle light detection module 710.
  • the apparatus may further include, but is not limited to: determining one or more of the vehicle light indication module 720, the intelligent control module 730, the first correction module 740, and the training device 750 of the neural network.
  • the acquisition image block module 700 is for acquiring an image block containing a vehicle.
  • the first vehicle light detection module 710 is configured to perform vehicle light detection on the image block via the deep neural network to obtain the vehicle light detection result.
  • the determined vehicle light indication module 720 is configured to determine the vehicle light indication information according to the vehicle light detection result of the plurality of image blocks in which the timing relationship exists.
  • the intelligent control module 730 is configured to generate driving control information or driving warning prompt information according to the vehicle light instruction information.
  • the first correction module 740 is configured to perform, for the plurality of image blocks having a timing relationship, the vehicle lamp outer frame miss detection and/or the vehicle light outer frame smoothing process according to the vehicle lamp detection result, to obtain at least the corrected plurality of image blocks. Headlight cover information in an image block.
  • the neural network training device 750 is used to train a deep neural network to be trained using a sample image block with vehicle heading information.
  • the acquired image block module 700 in the embodiment of the present application may include: a detection module and a segmentation module.
  • the acquired image block module 700 may further include: a second correction module.
  • the detection module can be used for vehicle detection of the image to be processed.
  • the detection module performs vehicle detection on the image to be processed using the vehicle detection model to obtain vehicle enclosure information in the image to be processed.
  • the segmentation module can be used to segment the image to be processed according to the vehicle detection result to obtain an image block containing the vehicle.
  • the second correction module may be configured to perform vehicle outer frame miss detection and/or vehicle outer frame smoothing processing on the plurality of to-be-processed images having a timing relationship to obtain at least one of the plurality of to-be-processed images to be processed. Vehicle enclosure information in the vehicle.
  • the second modification module in the embodiment of the present application may include: a third module and a fourth module.
  • the third module may be configured to acquire the vehicle identifier corresponding to the at least one vehicle circumference box according to the plurality of to-be-processed images and the vehicle enclosure information thereof having a timing relationship.
  • the fourth module may be configured to perform vehicle outer frame miss detection and/or vehicle outer frame smoothing processing on at least one to-be-processed image according to the vehicle identifier corresponding to the at least one vehicle outer frame.
  • the fourth module may be based on n2 (eg, 8) pending images preceding the m2th pending image (such as the current pending image in the foregoing description) Position of the vehicle cradle having the vehicle identification, predicting a position of the vehicle circumscribing box having the vehicle identification in the m2th to-be-processed image; and based on the predicted position of the vehicle circumscribed frame, the fourth module performs the m2th to-be-processed image Vehicle RC add or vehicle rim position correction processing.
  • the fourth module adds a vehicle circumscribing frame to the m2th to-be-processed image according to the predicted position of the vehicle circumscribing frame, thereby realizing Missing detection processing; for example, in the case where there is a vehicle circumscribing box having the vehicle identification in the m2th to-be-processed image, the fourth module may have a position of the predicted vehicle circumscribing frame and the m2th to-be-processed image The position of the vehicle circumscribing frame of the vehicle identification is subjected to weighted averaging processing, thereby achieving smoothing processing.
  • the vehicle light detection result in the embodiment of the present application includes, but is not limited to, at least one of a vehicle light enclosure information, a vehicle light on state, and a vehicle light orientation.
  • the first modification module 740 in the embodiment of the present application may include: a first unit and a second unit.
  • the first unit is configured to acquire the vehicle light identifier corresponding to the at least one vehicle light enclosure according to the plurality of image blocks having the timing relationship and the vehicle lamp enclosure information of the plurality of image blocks.
  • the second unit is configured to perform a vehicle lamp external frame miss detection and/or a vehicle light external frame smoothing process on the at least one image block according to the vehicle light identifier corresponding to the at least one vehicle light external frame.
  • the second unit may be further configured to, for a vehicle light identification, have the n1 (eg, 8) image blocks located before the m1th image block (such as the current image block in the foregoing description)
  • the position of the lamp circumscribing frame of the lamp indicator, predicting the position of the illuminating frame of the lamp having the lamp illuminator in the m1th image block; and performing the car on the m1th image block based on the predicted position of the circumscribing frame of the lamp The lamp external frame is added or the lamp outer frame position correction processing is performed.
  • the second unit may perform the vehicle lamp circumscribing frame addition or the vehicular lamp rim frame position correction process on the m1th image block based on the predicted position of the illuminating frame of the illuminating unit, and the method may include: In the case of the lamp outer frame, a lamp outer frame is added to the m1th image block based on the predicted position of the lamp outer frame.
  • the second unit may further include: adding, to the m1 image block, the vehicle lamp circumscribing frame or the illuminating device circumscribing frame position correction processing based on the predicted location of the illuminating frame of the illuminating device, the second unit may further include: having the illuminating lamp identifier in the m1th image block In the case of the headlight circling frame, the position of the predicted headlight circling frame and the position of the headlight circumscribing frame having the headlight flag in the m1th image block are subjected to weighted averaging processing.
  • the vehicle light indication information in the embodiment of the present application may include, but is not limited to, a single-side left vehicle blinking, a single-side right vehicle blinking, a double-side vehicle blinking, and a double-side vehicle light extinguishing, and At least one of the double side lights is fully illuminated.
  • the determined vehicle light indication module 720 in the embodiment of the present application may be further configured to perform, for a plurality of vehicle light enclosures corresponding to the same vehicle of the same vehicle in a time series relationship, performing vehicle light on and off status statistics, According to the statistical results, the headlight indication information is determined.
  • the image block in the embodiment of the present application may be an image to be processed.
  • the image block in the embodiment of the present application may also be a partial image of the vehicle to be processed in the image to be processed.
  • the image block in the embodiment of the present application may also be an image block obtained based on partial image processing including a vehicle in the image to be processed.
  • the deep neural network in the embodiment of the present application includes: a fast regional convolutional neural network (Faster RCNN).
  • Faster RCNN fast regional convolutional neural network
  • the training device 750 of the neural network in the embodiment of the present application may include: a sample image block module 800, a second vehicle light detection module 810, and a supervision module 820.
  • the acquisition sample image block module 800 can be used to acquire a sample image block including a vehicle.
  • the second vehicle light detecting module 810 can be configured to perform vehicle light detection on the sample image block via the deep neural network to be trained to obtain the vehicle light detection result.
  • the monitoring module 820 can be used to guide the learning of the deep neural network to be trained by using the difference between the headlight detection result and the headlight labeling information of the sample image block as the guiding information.
  • the vehicle lamp labeling information in the embodiment of the present application includes at least one of the vehicle lamp outer frame information labeling information, the vehicle light-on state labeling information, and the vehicle heading labeling information.
  • the operations performed by at least one of the training devices 750 of the neural network may be referred to the description of FIG. 3 in the above method embodiments.
  • the structure of the training device 750 of the neural network can be seen in the description of FIG. 8 in the following embodiments. The description will not be repeated here.
  • the operations performed by the image block module 700, the first lamp detection module 710, the determination lamp indicator module 720, the intelligent control module 730, and the first correction module 740, and the technical effects thereof, may be referred to above.
  • the description of the method embodiments is directed to FIGS. 1, 2, 4, 5, and 6. The description will not be repeated here.
  • FIG. 8 is a schematic structural diagram of a training apparatus for a neural network according to an embodiment of the present application.
  • the apparatus of this embodiment may be used to implement the training method of the neural network provided by any of the above embodiments of the present application.
  • the apparatus of this embodiment mainly includes: a sample image block module 800, a second vehicle light detection module 810, and a supervision module 820.
  • the acquisition sample image block module 800 can be used to acquire a sample image block that includes a vehicle.
  • the second vehicle light detection module 810 can be configured to perform vehicle light detection on the sample image block via the deep neural network to be trained to obtain the vehicle light detection result.
  • the monitoring module 820 can be used to supervise and learn the deep neural network to be trained by using the difference between the headlight detection result and the headlight labeling information of the sample image block as the guiding information.
  • the vehicle lamp labeling information in the embodiment of the present application may include, but is not limited to, at least one of the vehicle lamp enclosure information information, the vehicle light-off state labeling information, and the vehicle heading labeling information.
  • FIG. 9 is a schematic structural diagram of an apparatus for implementing smart driving according to an embodiment of the present application.
  • the apparatus of this embodiment can be used to implement the method for implementing smart driving provided by any of the above embodiments of the present application.
  • the device in FIG. 9 mainly includes: an image block module 700, a first lamp detection module 710, a determination lamp indicator module 720, and an intelligent control module 730.
  • the device may further include: a first correction module 740.
  • the acquisition image block module 700 is mainly used to acquire an image block containing a vehicle.
  • the first vehicle light detection module 710 is mainly used for performing vehicle light detection on the image block via the deep neural network to obtain the vehicle light detection result.
  • the determination of the vehicle light indication module 720 is mainly for determining the vehicle light indication information according to the vehicle light detection result of the plurality of image blocks in which the timing relationship exists.
  • the intelligent control module 730 is mainly configured to generate driving control information or driving warning prompt information according to the vehicle light instruction information.
  • the first correction module 740 is mainly configured to perform a vehicle light enclosure miss detection and/or a vehicle exterior enclosure smoothing process according to the vehicle light detection result for the plurality of image blocks having a timing relationship to obtain the corrected plurality of image blocks.
  • the car light enclosure information in at least one image block.
  • the first vehicle light detection module 710 the determination of the vehicle light indication module 720, the intelligent control module 730, and the first correction module 740, the technical effects, and the structure of the module, refer to the above method implementation manner.
  • the relevant descriptions in FIGS. 1 and 7 are described. The description will not be repeated here.
  • FIG. 10 illustrates an exemplary device 1000 suitable for implementing the present application, which may be a control system/electronic system configured in a car, a mobile terminal (eg, a smart mobile phone, etc.), a personal computer (PC, eg, a desktop computer Or a notebook computer, etc.), a tablet computer, a server, and the like.
  • a mobile terminal eg, a smart mobile phone, etc.
  • PC personal computer
  • tablet computer eg, a tablet computer, a server, and the like.
  • device 1000 includes one or more processors, communication units, etc., which may be: one or more central processing units (CPUs) 1001, and/or one or more utilized
  • the image processor (GPU) 1013 or the like that performs neural light detection by the neural network, the processor may be loaded into the random access memory (RAM) 1003 according to executable instructions stored in the read only memory (ROM) 1002 or from the storage portion 1008.
  • Various appropriate actions and processes are performed by executing the instructions.
  • the communication unit 1012 may include, but is not limited to, a network card, which may include, but is not limited to, an IB (Infiniband) network card.
  • the processor can communicate with the read only memory 1002 and/or the random access memory 1003 to execute executable instructions, connect to the communication portion 1012 via the bus 1004, and communicate with other target devices via the communication portion 1012, thereby completing the corresponding in the present application. step.
  • the RAM 1003 various programs and data required for the operation of the device can be stored.
  • the CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004.
  • ROM 1002 is an optional module.
  • the RAM 1003 stores executable instructions or writes executable instructions to the ROM 1002 at runtime, the executable instructions causing the central processing unit 1001 to perform the steps included in the object segmentation method described above.
  • An input/output (I/O) interface 1005 is also coupled to bus 1004.
  • the communication unit 1012 may be integrated, or may be configured to have a plurality of sub-modules (for example, a plurality of IB network cards) and be respectively connected to the bus.
  • the following components are connected to the I/O interface 1005: an input portion 1006 including a keyboard, a mouse, etc.; an output portion 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker; a storage portion 1008 including a hard disk or the like And a communication portion 1009 including a network interface card such as a LAN card, a modem, or the like.
  • the communication section 1009 performs communication processing via a network such as the Internet.
  • Driver 1010 is also coupled to I/O interface 1005 as needed.
  • a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory or the like is mounted on the drive 1010 as needed so that a computer program read therefrom is installed in the storage portion 1008 as needed.
  • FIG. 10 is only an optional implementation manner.
  • the number and type of components in the foregoing FIG. 10 may be selected, deleted, added, or Replacement; in different function component settings, you can also use separate settings or integrated settings, such as GPU and CPU detachable settings, and then, the GPU can be integrated on the CPU, the communication can be separated, or Integration settings on the CPU or GPU, etc.
  • These alternative embodiments are all within the scope of the present application.
  • embodiments of the present application include a computer program product comprising tangibly embodied on a machine readable medium.
  • a computer program comprising program code for performing the steps shown in the flowcharts, the program code comprising instructions corresponding to the steps in the method of the present invention.
  • the computer program can be downloaded and installed from the network via the communication portion 1009, and/or installed from the removable medium 1011.
  • the computer program is executed by the central processing unit (CPU) 1001
  • the instructions for realizing the above-described respective steps described in the present application are executed.
  • the embodiment of the present application further provides a computer program product for storing computer readable instructions, when executed, causing a computer to execute the lamp provided in any of the above embodiments.
  • the computer program product can be implemented by means of hardware, software or a combination thereof.
  • the computer program product is embodied as a computer storage medium.
  • the computer program product is embodied as a software product, such as a Software Development Kit (SDK) or the like.
  • the embodiment of the present application further provides another vehicle lamp detecting method and a neural network training method, and corresponding devices and electronic devices, computer storage media, computer programs, and computer program products.
  • the method includes: the first device transmitting a vehicle light detection indication or training a neural network indication or an indication for implementing smart driving to the second device, the indication causing the second device to perform the light in any of the above possible embodiments a detection method or a training neural network method or a method for implementing intelligent driving; the first device receives a vehicle light detection result transmitted by the second device or a training result of the neural network or driving control information or driving warning prompt information for implementing intelligent driving .
  • the vehicle light detection indication or the training neural network indication may be a call instruction
  • the first device may instruct the second device to perform a vehicle light detection operation or train a neural network operation or to implement smart driving by means of a call.
  • the second device may perform the steps and/or procedures in any of the above-described vehicle light detection methods or methods of training a neural network or methods for implementing smart driving.
  • the methods and apparatus of the present application may be implemented in a number of ways.
  • the methods and apparatus of the present application can be implemented in software, hardware, firmware, or any combination of software, hardware, and firmware.
  • the above-described sequence of steps for the method is for illustrative purposes only, and the steps of the method of the present application are not limited to the order specifically described above unless otherwise specifically stated.
  • the present application can also be implemented as a program recorded in a recording medium, the programs including machine readable instructions for implementing the method according to the present application.
  • the present application also covers a recording medium storing a program for executing the method according to the present application.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • General Engineering & Computer Science (AREA)
  • Automation & Control Theory (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Atmospheric Sciences (AREA)
  • Transportation (AREA)
  • Human Computer Interaction (AREA)
  • Mechanical Engineering (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Image Analysis (AREA)
  • Traffic Control Systems (AREA)
  • Image Processing (AREA)

Abstract

一种车灯检测方法、神经网络的训练方法、用于实现智能驾驶的方法、装置、电子设备、计算机可读存储介质以及计算机程序，其中的车灯检测方法包括：获取包含有车辆的图像块（S100）；经由深度神经网络对所述图像块进行车灯检测，以获取车灯检测结果（S110）。

Description

车灯检测方法、实现智能驾驶的方法、装置、介质及设备
本申请要求在2018年3月30日提交中国专利局、申请号为CN201810278351.7、发明名称为“车灯检测方法、实现智能驾驶的方法、装置、介质及设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机视觉技术，尤其是涉及一种车灯检测方法、车灯检测装置、神经网络的训练方法、神经网络的训练装置、用于实现智能驾驶的方法、用于实现智能驾驶的装置、电子设备、计算机可读存储介质以及计算机程序产品。
背景技术
在智能驾驶技术中，准确的对车辆周围信息进行检测和判别，对于智能驾驶车辆行驶路线的选择以及车辆规避制动等决策，是非常重要的。
车辆上的车灯往往可以起到，向其他车辆以及行人等，传达相应信息(如左转弯、右转弯或者刹车等)的作用。准确识别行驶环境中的其他车辆的车灯所传达的信息，对于智能驾驶的决策而言，是具有一定的意义的。
发明内容
本申请实施方式提供一种车灯检测、训练神经网络以及用于实现智能驾驶的技术方案。
根据本申请实施方式其中一方面，提供一种车灯检测方法，所述方法包括：获取包含有车辆的图像块；经由深度神经网络对所述图像块进行车灯检测，以获取车灯检测结果。
根据本申请实施方式的其中再一方面，提供一种神经网络的训练方法，所述方法包括：获取包括有车辆的样本图像块；经由待训练的深度神经网络，对所述样本图像块进行车灯检测，以获取车灯检测结果；以所述车灯检测结果与所述样本图像块的车灯标注信息之间的差异为指导信息，对所述待训练的深度神经网络进行监督学习。
根据本申请实施方式其中再一方面，提供一种用于实现智能驾驶的方法，所述方法包括：获取包含有车辆的图像块；经由深度神经网络对所述图像块进行车灯检测，以获取车灯检测结果；根据存在时序关系的多个图像块的车灯检测结果，确定车灯指示信息；根据所述车灯指示信息生成驾驶控制信息或驾驶预警提示信息。
根据本申请实施方式其中再一方面，提供一种车灯检测装置，所述装置包括：获取图像块模块，用于获取包含有车辆的图像块；第一车灯检测模块，用于经由深度神经网络对所述图像块进行车灯检测，以获取车灯检测结果。
根据本申请实施方式的其中再一方面，提供一种神经网络的训练装置，所述装置包括：获取样本图像块模块，用于获取包括有车辆的样本图像块；第二车灯检测模块，用于经由待训练的深度神经网络，对所述样本图像块进行车灯检测，以获取车灯检测结果；监督模块，用于以所述车灯检测结果与所述样本图像块的车灯标注信息之间的差异为指导信息，对所述待训练的深度神经网络进行监督学习。
根据本申请实施方式的其中再一方面，提供一种用于实现智能驾驶的装置，所述装置包括：获取图像块模块，用于获取包含有车辆的图像块；第一车灯检测模块，用于经由深度神经网络对所述图像块进行车灯检测，以获取车灯检测结果；确定车灯指示模块，用于根据存在时序关系的多个图像块的车灯检测结果，确定车灯指示信息；智能控制模块，用于根据所述车灯指示信息生成驾驶控制信息或驾驶预警提示信息。
根据本申请实施方式再一方面，提供一种电子设备，包括：存储器，用于存储计算机程序；处理器，用于执行所述存储器中存储的计算机程序，且所述计算机程序被执行时，实现本申请任一方法实施方式。
根据本申请实施方式再一方面，提供一种电子设备，包括：处理器和本申请任一装置实施方式；在处理器运行所述装置时，本申请任一装置实施方式中的单元被运行。
根据本申请实施方式再一个方面，提供一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时，实现本申请任一方法实施方式。
根据本申请实施方式的再一个方面，提供一种计算机程序，包括计算机指令，当所述计算机指令在设备的处理器中运行时，实现本申请任一方法实施方式。
根据本申请实施方式的再一个方面，提供一种计算机程序产品，包括计算机可读代码，当所述计算机可读代码在设备上运行时，所述设备中的处理器执行用于实现本申请任一方法实施方式。
基于本申请提供的上述车灯检测方法、车灯检测装置、神经网络的训练方法、神经网络的训练装置、用于实现智能驾驶的方法、用于实现智能驾驶的装置、电子设备、计算机可读存储介质及计算机程序，本申请通过利用深度神经网络对包含有车辆的图像块进行车灯检测，有利于快速且准确的获得图像块的车灯检测结果，从而有利于使本申请的车灯检测技术应用于自动驾驶、辅助驾驶等智能驾驶的实时环境中，进而有利于提高智能驾驶的决策准确性或预警准确性。
下面通过附图和实施例，对本申请的技术方案做进一步的详细描述。
附图说明
构成说明书的一部分的附图描述了本申请的实施例，并且连同描述一起用于解释本申请的原理。
参照附图，根据下面的详细描述，可以更加清楚地理解本申请，其中：
图1为本申请实施例提供的车灯检测方法的一个流程图。
图2为本申请实施例提供的待处理图像的一个实施示意图。
图3为本申请实施例提供的神经网络的训练方法的一个流程图。
图4为本申请实施例提供的用于实现智能驾驶的方法的一个流程图。
图5为本申请实施例提供的车灯检测结果的一个实施方式示意图。
图6为本申请实施例提供的车灯检测结果的另一个实施方式示意图。
图7为本申请实施例提供的车灯检测装置的一个结构示意图。
图8为本申请实施例提供的神经网络的训练装置的一个结构示意图。
图9为本申请实施例提供的用于实现智能驾驶的装置的一个结构示意图。
图10为实现本申请实施方式的一示例性设备的框图。
具体实施方式
现在将参照附图来详细描述本申请的各种示例性实施例。应注意到：除非另外具体说明，否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值不限制本申请的范围。
同时，应当明白，为了便于描述，附图中所示出的各个部分的尺寸并不是按照实际的比例关系绘制的。
以下对至少一个示例性实施例的描述实际上仅仅是说明性的，决不作为对本申请及其应用或使用的任何限制。
对于相关领域普通技术人员已知的技术、方法和设备可能不作详细讨论，但在适当情况下，所述技术、方法和设备应当被视为说明书的一部分。
应注意到：相似的标号和字母在下面的附图中表示类似项，因此，一旦某一项在一个附图中被定义，则在随后的附图中不需要对其进行进一步讨论。
本申请实施例可以应用于计算机系统/服务器，其可与众多其它通用或专用计算系统环境或配置一起操作。适于与计算机系统/服务器一起使用的众所周知的计算系统、环境和/或配置的例子包括但不限于：个人计算机系统、服务器计算机系统、瘦客户机、厚客户机、手持或膝上设备、基于微处理器的系统、机顶盒、可编程消费电子产品、网络个人电脑、小型计算机系统﹑大型计算机系统和包括上述任何系统的分布式云计算技术环境，等等。
计算机系统/服务器可以在由计算机系统执行的计算机系统可执行指令(诸如程序模块)的一般语境下描述。通常，程序模块可以包括例程、程序、目标程序、组件、逻辑、数据结构等等，它们执行特定的任务或者实现特定的抽象数据类型。计算机系统/服务器可以在分布式云计算环境中实施，分布式云计算环境中，任务是由通过通信网络链接的远程处理设备执行的。在分布式云计算环境中，程序模块可以位于包括存储设备的本地或远程计算系统存储介质上。
图1为本申请实施例提供的车灯检测方法的一个流程图。该方法可以由任意电子设备执行，例如终端设备、服务器、移动设备、车载设备等等，如图1所示，该实施例方法包括：
S100、获取包含有车辆的图像块。
在一个可选示例中，该步骤S100可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的获取图像块模块700执行。
S110、经由深度神经网络对图像块进行车灯检测，以获取车灯检测结果。
在一个可选示例中，该步骤S110可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的第一车灯检测模块710执行。
本申请实施例通过利用深度神经网络对包含有车辆的图像块进行车灯检测，有利于快速且准确的获得图像块的车灯检测结果，从而有利于使本申请实施例提供的车灯检测技术应用于自动驾驶、辅助驾驶等智能驾驶的实时环境中，进而有利于提高智能驾驶的决策准确性或预警准确性。
在一个可选示例中，本申请实施例中的包含有车辆的图像块可以为整个待处理图像，也可以为待处理图像中的包含有车辆的局部图像。另外，本申请中的包含有车辆的图像块还可以为通过针对待处理图像中的包含有车辆的局部图像进行处理，而得到的图像块。
在一个可选示例中，本申请实施例中的待处理图像可以为呈现静态的图片或照片等图像，也可以为呈现动态的视频中的视频帧，例如，车辆中设置的摄像装置所摄取到的视频中的视频帧。该待处理图像中的车辆可以为摄像装置摄取到的位于其所在车辆的后方(包括正后方以及侧后方等)的车辆，也可以为摄像装置摄取到的位于其所在车辆的前方(包括正前方以及侧前方等)的车辆，还可以为摄像装置摄取到的位于其所在车辆的正侧方的车辆。本申请不限制待处理图像中的车辆相对于摄像装置所在车辆的具体位置关系。
在一个可选示例中，图像块为待处理图像；或者，
图像块为待处理图像中包含车辆的局部图像，或者，
图像块为基于待处理图像中包含车辆的局部图像处理而得到的图像块。
可选地，本申请实施例的图像块中所包含的车辆可以是完整的车辆(如图2中最右侧的车辆)，也可以是由于遮挡等原因而造成的局部部分车辆(即车辆的局部，如图2中左侧的多个车辆等)。本申请不限制待处理图像中的车辆的具体表现形态。
在一个可选示例中，获取包含有车辆的图像块包括：
对待处理图像进行车辆检测；
根据车辆检测结果，对待处理图像进行切分，以获得包含有车辆的图像块。
可选地，本申请实施例可以通过对待处理图像进行车辆检测，从而获得车辆检测结果，进而根据车辆检测结果，对待处理图像进行切分处理，可以获得包含有车辆的图像块，从而有利于提高车灯检测的准确性。在待处理图像中包含有多个车辆的情况下，本申请可以从待处理图像中切分出多个包含有车辆 的图像块。上述车辆检测结果可以为车辆外接框信息，例如，位于车辆外接框对角线上的两个顶点坐标等。
在一个可选示例中，对待处理图像进行车辆检测包括：
利用车辆检测模型对待处理图像进行车辆检测，以获得待处理图像中的车辆外接框信息。
本申请实施例可以利用神经网络，对待处理图像进行车辆检测，从而可以根据神经网络输出的信息获得车辆检测结果。例如，本申请实施例可以使用车辆检测模型(该车辆检测模型可以为用于车辆检测的神经网络)等，对待处理图像进行车辆检测处理。在一些可选示例中，本申请实施例中的车辆检测模型可以为区域卷积神经网络(Regions with Convolutional Neural Network，R-CNN)、快速区域卷积神经网络(Fast R-CNN)或者更快速区域卷积神经网络(Faster R-CNN)等。本申请不限制对待处理图像进行车辆检测处理，以获得包含有车辆的图像块的具体实现方式。本申请也不限制车辆检测模型(如用于车辆检测的神经网络)的具体结构等。本申请实施例通过使用车辆检测模型从待处理图像中检测出车辆，并从待处理图像中切分出包含有车辆的图像块，有利于减少待处理图像中的非车辆信息对车灯检测的干扰，从而有利于降低车灯检测的虚检率，进而有利于提高车灯检测的准确性。
在一个可选示例中，本申请实施例中的图像块的大小通常与深度神经网络对输入图像的尺寸要求相关，例如，图像块的大小可以为256×256等。本申请中的深度神经网络即用于车灯检测的深度神经网络。为了获得具有预定大小的图像块，本申请实施例可以先对待处理图像进行缩放处理，然后，按照预定大小从缩放处理后的待处理图像中切分出包含有车辆的图像块。本申请可以根据车辆检测结果(如车辆外接框信息)对待处理图像进行缩放处理，以便于使剪切出的包含有车辆的图像块具有预定大小。在针对一个待处理图像的车辆检测结果包括多个车辆外接框信息的情况下，本申请实施例可以针对至少一个车辆外接框信息，分别对该待处理图像进行相应的缩放处理，从而使基于该待处理图像而获得的至少一个图像块分别具有预定大小。另外，本申请实施例也可以先进行切分处理，再进行缩放处理。本申请实施例对图像块的大小以及缩放处理的具体实现方式不作限制。
在一个可选示例中，对待处理图像进行车辆检测还包括：
针对存在时序关系的多个待处理图像，进行车辆外接框漏检和/或车辆外接框平滑处理，以获得修正的多个待处理图像中的车辆外接框信息。
本申请实施例可以针对神经网络(如车辆检测模型)输出的车辆检测结果，进行车辆外接框漏检以及车辆外接框平滑处理中的至少一个(如在对待处理图像进行切分处理前，执行车辆外接框漏检以及车辆外接框平滑处理)，从而可以有利于提高车辆检测结果的准确性，进而有利于提高车灯检测结果的准确性。一个可选例子，在针对视频中的多个视频帧进行车辆检测的情况下，多个视频帧通常存在时序关系(如视频中连续排列的视频帧，再如针对视频进行抽帧，基于抽帧的结果而形成的多个连续抽取出的视频帧)，本申请实施例可以针对多个存在时序关系的视频帧(即待处理图像)进行车辆外接框漏检以及车辆外接框平滑处理，从而可以获得车辆检测模型漏检的车辆外接框信息，并可以对车辆检测模型输出的车辆外接框信息进行位置校正。例如，根据位于第n2视频帧之前的多个视频帧中的车辆外接框，来补全第n2视频帧中未被检测出的车辆外接框；再例如，根据位于第n2视频帧之前的多个视频帧中的车辆外接框，对第n2视频帧中的相应车辆外接框的位置进行校正。本申请实施例可以根据车辆外接框漏检以及车辆外接框平滑处理后的多个车辆外接框信息，分别对相应的待处理图像进行切分处理，从而获得存在时序关系的多个包含有车辆的图像块。
在一个可选示例中，针对存在时序关系的多个所述待处理图像，进行车辆外接框漏检和/或平滑处理包括：
根据存在时序关系的多个待处理图像及其车辆外接框信息，获取至少一个车辆外接框所对应的车辆标识；
根据至少一个车辆外接框所对应的车辆标识，对多个待处理图像中的至少一个待处理图像进行车辆 外接框漏检和/或车辆外接框平滑处理。
本申请实施例可以先获得存在时序关系的多个待处理图像中的图像块(即车辆外接框信息)，然后，再针对存在时序关系的多个待处理图像以及车辆外接框信息，进行车辆外接框漏检处理以及车辆外接框平滑处理。
在一个可选示例中，本申请实施例获得存在时序关系的多个待处理图像中的图像块的一个可选例子为：将存在时序关系的多个待处理图像和多个待处理图像中至少一个待处理图像中的车辆外接框信息，分别提供给车辆跟踪模型(该车辆跟踪模型可以为目标对象为车辆的物体跟踪模型)，本申请实施例可以根据车辆跟踪模型输出的信息，获得至少一个车辆外接框所对应的车辆标识。由此，本申请实施例可以成功建立至少一个车辆分别与相应的待处理图像之间的关联关系。由于本申请实施例中的图像块可以是基于车辆外接框信息对待处理图像进行切分而产生的，因此，车辆外接框所对应的车辆标识，也可以称为图像块标识，包含有相同车辆的图像块的图像块标识通常相同，而包含不同车辆的图像块的图像块标识通常不相同，由此有利于实现多车情形下车辆的车灯检测。
在一个可选示例中，本申请实施例中的车辆跟踪模型可以为基于前后相邻的两视频帧对比的卷积神经网络(CNN)或者循环神经网络(Recurrent Neural Networks，RNN)等。
在一个可选示例中，根据至少一个车辆外接框所对应的车辆标识，对多个待处理图像中的至少一个待处理图像进行车辆外接框漏检和/或车辆外接框平滑处理包括：
针对一车辆标识而言，根据位于第m2个待处理图像之前的n2个待处理图像中的具有车辆标识的车辆外接框的位置，预测第m2个待处理图像中的具有车辆标识的车辆外接框的位置；
基于预测的车辆外接框的位置，对第m2个待处理图像进行车辆外接框添加或车辆外接框位置校正处理。
本申请实施例可以根据存在时序关系的多个待处理图像和多个待处理图像的图像块标识，进行车辆外接框漏检以及车辆外接框平滑处理，例如，可以采用下述公式(1)针对当前视频帧进行车辆外接框漏检以及车辆外接框平滑处理：
Δx＝a 1t 2+b 1t+c 1
Δy＝a 2t 2+b 2t+c 2
Δw＝a 3t 2+b 3t+c 3
Δh＝a 4t 2+b 4t+c 4            公式(1)
在上述公式(1)中，a 1、a 2、a 3、a 4、b 1、b 2、b 3、b 4、c 1、c 2、c 3以及c 4均为线性方程(即公式(1))的参数，t表示当前时刻之前的第t个时刻(如，t＝0、……、t＝8，每一个时刻对应一个视频帧，即每一个时刻对应一个待处理图像)，Δx表示前一视频帧中的车辆外接框的中心点横坐标与下一视频帧中的具有相同车辆标识的车辆外接框的中心点横坐标之间的差值，Δy表示前一视频帧中的车辆外接框的中心点纵坐标与下一视频帧中的具有相同车辆标识的车辆外接框的中心点纵坐标之间的差值，Δw表示前一视频帧中的车辆外接框的宽度与下一视频帧中的具有相同车辆标识的车辆外接框的宽度之间的差值，Δh表示前一视频帧中的车辆外接框的高度与下一视频帧中的具有相同车辆标识的车辆外接框的高度之间的差值。
可选地，基于预测的车辆外接框的位置，对第m2个待处理图像进行车辆外接框添加或车辆外接框位置校正处理包括：
在第m2个待处理图像中不存在具有车辆标识的车辆外接框的情况下，根据预测的车辆外接框的位置，为第m2个待处理图像添加车辆外接框；或者
在第m2个待处理图像中存在具有车辆标识的车辆外接框的情况下，对预测的车辆外接框的位置以及第m2个待处理图像中的具有车辆标识的车辆外接框的位置进行加权平均处理。
本申请实施例可以根据当前视频帧(即当前时刻)的前N(如N＝8)个视频帧(即前N个时刻) 中的具有相同车辆标识的车辆外接框信息，计算出上述公式(1)中的线性方程参数a 1、a 2、a 3、a 4、b 1、b 2、b 3、b 4、c 1、c 2、c 3以及c 4。从而本申请实施例可以在已知上述公式(1)的参数的情况下，利用上述公式(1)计算出当前视频帧对应的Δx、Δy、Δw以及Δh，针对一车辆标识而言，本申请实施例可以根据当前视频帧的上一视频帧中具有该车辆标识的车辆外接框的位置以及上述计算出的当前视频帧对应的Δx、Δy、Δw以及Δh，预测出当前视频帧中的具有该车辆标识的车辆外接框的位置，如果当前视频帧不存在具有该车辆标识的车辆外接框，则本申请实施例可以根据预测出的车辆外接框的位置，为当前视频帧中添加车辆外接框，从而实现对当前视频帧的车辆外接框漏检处理；如果当前视频帧存在具有该车辆标识的车辆外接框，则本申请实施例可以针对预测出的车辆外接框的位置与当前视频帧中的具有该车辆标识的车辆外接框的位置进行加权平均，从而实现对当前视频帧中的具有该车辆标识的车辆外接框的平滑处理。
另外，本申请实施例可以采用现有的其他漏检处理以及平滑处理技术，针对多个视频帧进行车辆外接框的漏检处理以及车辆外接框的平滑处理。本申请实施例不限制对待处理图像进行车辆外接框漏检处理以及车辆外接框平滑处理的具体实现方式。本申请实施例也不限制车辆跟踪模型的具体结构等。
在一个可选示例中，本申请实施例中的深度神经网络可以采用基于区域检测的深度神经网络，例如，可以采用区域卷积神经网络(Regions with Convolutional Neural Network，RCNN)或者快速区域卷积神经网络(Faster RCNN)等。该深度神经网络的网络结构可以根据提取车灯检测结果的实际需求灵活设计，本申请实施例并不限制该深度神经网络的具体网络结构；例如，本申请实施例的该深度神经网络可以包括但不限于卷积层、非线性激活(Relu)层、池化层以及全连接层等，该深度神经网络所包含的层数越多，则网络越深；再例如，本申请实施例的该深度神经网络的网络结构可以采用但不限于ALexNet、深度残差网络(Deep Residual Network，ResNet)或视觉几何组网络(Visual Geometry Group Network，VGGnet)等神经网络所采用的网络结构。
可选地，本申请实施例中的深度神经网络是利用带有车灯标注信息的样本图像块训练获得的，使得训练后的深度神经网络具有准确检测车灯状态的能力。为了提高神经网络的训练效率，样本图像块的车灯标注信息可以包括：车灯外接框信息标注信息、车灯亮灭状态标注信息以及车灯方位标注信息中的一个或者多个。该深度神经网络的训练过程可以参见下述针对图3的描述，在此不再重复说明。
在一个可选示例中，本申请实施例的深度神经网络所输出的车灯检测结果可以包括但不限于：车灯外接框信息、车灯亮灭状态以及车灯方位中的至少一个。例如，深度神经网络输出车灯外接框信息、车灯亮灭状态以及车灯方位。其中的车灯外接框信息通常是指能够表示出车灯外接框在图像块中的位置的信息(如位于车灯外接框对角线上的两个顶点在图像块上的坐标)，进而本申请实施例可以根据车灯外接框在图像块中的位置确定车灯在待处理图像中的位置(如位于车灯外接框对角线上的两个顶点在待处理图像上的坐标)。当然，深度神经网络输出的车灯外接框信息也可以直接为车灯外接框在待处理图像中的位置的信息(如位于车灯外接框对角线上的两个顶点在待处理图像块上的坐标)。其中的车灯亮灭状态通常可以表示出车灯处于亮状态，还是处于灭状态。其中的车灯方位通常用于指示车灯在车辆中的方位，例如，车灯方位可以为：左前车灯、右前车灯、左后车灯或者右后车灯等。
在一个可选示例中，在根据存在时序关系的多个图像块的车灯检测结果，确定车灯指示信息之前，还包括：
针对存在时序关系的多个图像块，根据车灯检测结果，进行车灯外接框漏检和/或车灯外接框平滑处理，以获得修正的多个图像块中的车灯外接框信息。
本申请实施例可以针对深度神经网络输出的车灯检测结果进行车灯外接框漏检以及车灯外接框平滑处理中的至少一个，从而可以有利于提高车灯检测结果的准确性。一个可选例子，在针对多个图像块进行车灯检测的情况下，多个图像块通常存在时序关系(例如，视频中连续排列的多个视频帧中的至少一个图像块，再例如，针对视频进行抽帧，基于抽帧的结果而形成的多个连续抽取出的视频帧中的至少 一个图像块等)，本申请实施例可以针对多个存在时序关系的图像块，进行车灯外接框漏检以及车灯外接框平滑处理，从而可以获得深度神经网络漏检的车灯外接框信息，并可以对深度神经网络输出的车灯外接框信息进行位置校正。例如，根据位于第n1图像块之前的多个图像块中的车灯外接框，来补全第n1图像块中未被检测出的车灯外接框；再例如，根据位于第n1图像块之前的多个图像块中的车灯外接框，对第n1图像块中的相应车灯外接框的位置进行校正。本申请实施例可以基于车灯外接框漏检以及车灯外接框平滑处理后的至少一个车灯外接框信息以及车灯亮灭状态等其他信息，进行车灯指示信息分析。
在一个可选示例中，针对存在时序关系的多个图像块，根据车灯检测结果，进行车灯外接框漏检和/或车灯外接框平滑处理包括：
根据存在时序关系的多个图像块和多个图像块中车灯外接框信息，获取至少一个车灯外接框所对应的车灯标识；
根据至少一个车灯外接框所对应的车灯标识，对多个图像块中至少一个图像块进行车灯外接框漏检和/或车灯外接框平滑处理。
可选地，本申请实施例可以先获得存在时序关系的多个图像块中的车灯外接框信息(如车灯外接框的对角线上的两个顶点的坐标)，然后，在针对存在时序关系的多个图像块以及至少一个车灯外接框信息，进行车灯外接框漏检处理以及车灯外接框平滑处理。
在一个可选示例中，本申请实施例可以将存在时序关系的多个图像块以及图像块中的车灯外接框信息分别提供给车灯跟踪模型(该车灯跟踪模型可以为目标对象为车灯的物体跟踪模型)，本申请实施例可以根据车灯跟踪模型输出的信息，获得至少一个车灯外接框所对应的车灯标识。由此，本申请实施例可以成功建立车灯与车辆之间的关联关系。包含有相同车灯标识的不同车灯外接框通常对应同一车辆的同一车灯。
在一个可选示例中，根据至少一个车灯外接框所对应的车灯标识，对多个图像块中至少一个图像块进行车灯外接框漏检和/或车灯外接框平滑处理包括：
针对一车灯标识而言，根据位于第m1个图像块之前的n1个图像块中的具有车灯标识的车灯外接框的位置，预测第m1个图像块中的具有车灯标识的车灯外接框的位置；
基于预测的车灯外接框的位置，对第m1个图像块进行车灯外接框添加或车灯外接框位置校正处理。
本申请实施例可以根据存在时序关系的多个图像块和该多个图像块所包括的至少一个车灯外接框的车灯标识，进行车灯外接框漏检处理以及车灯外接框平滑处理，例如，可以采用上述公式(1)针对当前图像块进行车灯外接框漏检处理以及车灯外接框平滑处理。在利用公式(1)进行车灯外接框漏检处理以及车灯外接框平滑处理的过程中，上述公式(1)中的a 1、a 2、a 3、a 4、b 1、b 2、b 3、b 4、c 1、c 2、c 3以及c 4仍然为线性方程的参数，t表示当前时刻之前的第t个时刻(如t＝0、……、t＝8，每一个时刻对应一个图像块，即每一个时刻对应一个视频帧或者待处理图像)，Δx表示前一图像块中的车灯外接框的中心点横坐标与下一图像块中的具有相同车灯标识的车灯外接框的中心点横坐标之间的差值，Δy表示前一图像块中的车灯外接框的中心点纵坐标与下一图像块中的具有相同车灯标识的车灯外接框的中心点纵坐标之间的差值，Δw表示前一图像块中的车灯外接框的宽度与下一图像块中的具有相同车灯标识的车灯外接框的宽度之间的差值，Δh表示前一图像块中的车灯外接框的高度与下一图像块中的具有相同车灯标识的车灯外接框的高度之间的差值。
可选地，基于预测的车灯外接框的位置，对第m1个图像块进行车灯外接框添加或车灯外接框位置校正处理包括：
在第m1个图像块中不存在具有车灯标识的车灯外接框的情况下，根据预测的车灯外接框的位置，为第m1个图像块添加车灯外接框；或者
在第m1个图像块中存在具有车灯标识的车灯外接框的情况下，对预测的车灯外接框的位置以及第 m1个图像块中的具有车灯标识的车灯外接框的位置进行加权平均处理。
本申请实施例可以根据当前图像块(即当前时刻)的前N(如N＝8)个图像块(即前N个时刻)中的至少一个图像块的具有相同车灯标识的车灯外接框信息，计算出上述公式(1)中的线性方程参数a 1、a 2、a 3、a 4、b 1、b 2、b 3、b 4、c 1、c 2、c 3以及c 4。从而本申请可以在已知上述公式(1)的线性方程参数的情况下，利用公式(1)计算出当前图像块的Δx、Δy、Δw以及Δh，针对一车灯标识而言，本申请实施例可以根据当前图像块的上一图像块中具有该车灯标识的车灯外接框的位置以及上述计算出的当前图像块对应的Δx、Δy、Δw以及Δh，预测出当前图像块中的具有该车灯标识的车灯外接框的位置，如果当前图像块中不存在具有该车灯标识的车灯外接框，则本申请可以根据预测出的车灯外接框的位置，为当前图像块添加车灯外接框，从而实现对当前图像块的车灯外接框漏检处理；如果当前图像块存在具有该车灯标识的车灯外接框，则本申请可以针对预测出的车灯外接框的位置与当前图像块中的具有该车灯标识的车灯外接框的位置进行加权平均，从而实现对当前图像块中的具有该车灯标识的车灯外接框的平滑处理。
另外，本申请实施例可以采用现有的其他漏检处理以及平滑处理技术，针对至少一帧视频帧中的至少一个图像块进行车灯外接框的漏检处理以及车灯外接框的平滑处理。本申请实施例不限制对待处理图像中的图像块进行车灯外接框漏检处理以及车灯外接框平滑处理的具体实现方式。本申请也不限制车灯跟踪模型的具体结构等。
在一个可选示例中，本申请实施例可以根据存在时序关系的多个图像块的车灯检测结果，确定车灯指示信息。
可选地，车灯指示信息包括：
单侧左车灯闪烁、单侧右车灯闪烁、双侧车灯闪烁、双侧车灯全灭以及双侧车灯全亮中的至少一个。
可选的，本申请可以根据同一车辆的多个图像块的车灯检测结果，对同一车辆的同一个车灯在一段时间内的亮灭状态进行统计，并对统计结果进行判断识别，从而可以获知同一车辆的同一个车灯是否处于闪烁状态、长亮状态或者长灭状态等，进而本申请实施例可以结合车灯方位以及同一车辆的其他车灯所处的状态，判断出该车辆的车灯指示信息为：单侧左车灯闪烁、单侧右车灯闪烁、双侧车灯闪烁、双侧车灯全灭或者双侧车灯全亮等。单侧左车灯闪烁(右车灯不亮)通常表示车辆向左侧移动(如左并线或者左转弯等)。单侧右车灯闪烁(左车灯不亮)通常表示车辆向右侧移动(如右并线或者右转弯等)。双侧车灯闪烁通常表示车辆存在应急或者临时停车等特殊状态。双侧车灯全灭表示正常不改变方向行驶。双侧车灯全亮表示刹车减速行驶。
在一个可选示例中，根据存在时序关系的多个图像块的车灯检测结果，确定车灯指示信息包括：
针对存在时序关系的对应同一车辆的同一车灯的多个车灯外接框，进行车灯亮灭状态统计，并根据统计结果确定车灯指示信息。
本申请实施例通过运用同一个车灯的瞬时状态的时序信息，可以对车灯的状态进行更加细致的分类。通过结合同一辆车的两个车灯(如两个后车灯)的状态对整个车的趋向进行判定，从而可以对智能驾驶系统中的车辆环境感知系统提供准确快速且直观的信息，有利于帮助智能驾驶系统进行决策，从而有利于使车辆更加安全快速的行驶。
在一个可选示例中，本申请实施例可以利用下述公式(2)至公式(5)实现对同一车辆的同一个车灯在一段时间内的亮灭状态的统计：
Figure PCTCN2019079300-appb-000001
Figure PCTCN2019079300-appb-000002
D m＝count(A i＝m)     公式(4)
Figure PCTCN2019079300-appb-000003
在上述公式(2)中，i表示第i个阶段，一个阶段通常包括多个时刻，s n表示车灯在第n个时刻的亮灭状态，s n的取值为0，则表示该车灯在第n个时刻处于灭状态，s n的取值为1，则表示该车灯在第n个时刻处于亮状态；C i表示第i个阶段，车灯处于亮状态的次数。
上述公式(3)中，A i用于通过不同取值对C i的不同取值范围进行表示。
上述公式(4)中，D m表示对取值为m的A i的数量进行统计。
在上述公式(5)中，S t表示车灯在一段时间内的亮灭状态；如果S t的取值为2，则表示该车灯处于长亮状态，如果S t的取值为1，则表示该车灯处于闪烁状态，如果S t的取值为0，则表示该车灯处于长灭状态。
需要特别说明的是，上述公式(2)至公式(5)仅为一种可选的统计方式，本申请实施例可以采用其他统计方式来确定车灯指示信息，且统计方式可以根据实际需求灵活设计。本申请实施例不限制确定车灯指示信息的具体统计方式。
图3为本申请实施例提供的神经网络的训练方法的一个流程图。该方法可以由任意电子设备执行，例如终端设备、服务器、移动设备、车载设备等等，如图3所示，该实施例方法包括：步骤S300、步骤S310以及步骤S320。下面对图3中的步骤进行详细说明。
S300、获取包括有车辆的样本图像块。
在一个可选示例中，该步骤S300可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的获取样本图像块模块800执行。
S310、经由待训练的深度神经网络，对样本图像块进行车灯检测，以获取车灯检测结果。
在一个可选示例中，该步骤S310可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的第二车灯检测模块810执行。
S320、以车灯检测结果与样本图像块的车灯标注信息之间的差异为指导信息，对待训练的深度神经网络进行监督学习。
在一个可选示例中，该步骤S320可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的监督模块820执行。
在一个可选示例中，本申请实施例中的包含有车辆的样本图像块可以为整个图像样本，也可以为图像样本中的包含有车辆的局部图像。另外，本申请实施例中的包含有车辆的样本图像块还可以为通过针对图像样本中的包含有车辆的局部图像进行处理，而得到的样本图像块。
在一个可选示例中，从训练数据集中获取图像样本，并获取图像样本中的包含有车辆的样本图像块。本申请实施例中的训练数据集中包括多条用于训练深度神经网络的图像样本，通常情况下，每一个图像样本均设置有车灯标注信息，车灯标注信息可以包括但不限于：车灯外接框信息标注信息、车灯亮灭状态标注信息以及车灯方位标注信息中的至少一个。本申请实施例可以按照随机读取方式或者按照图像样本排列次序顺序读取方式，一次从训练数据集中读取一个或者多个图像样本。
在一个可选示例中，本申请实施例可以采用多种方式获取包含有车辆的样本图像块，例如，本申请实施例可以利用神经网络的方式，来获取图像样本中的包含有车辆的图像块。一个可选的例子，本申请实施例可以通过对图像样本进行车辆检测，从而获得车辆检测结果，进而根据车辆检测结果，对图像样本进行切分处理，可以获得包含有车辆的样本图像块。在图像样本中包含有多个车辆的情况下，本申请实施例可以从图像样本中切分出多个包含有车辆的样本图像块。上述车辆检测结果可以为车辆外接框信息，例如，位于车辆外接框对角线上的两个顶点坐标等。
在一个可选示例中，本申请实施例可以车辆检测模型(也可以称为目标对象为车辆的物体检测模型) 等，对读取出的图像样本进行车辆检测。本申请实施例不限制对图像样本进行车辆检测，以获得包含有车辆的样本图像块的具体实现方式。
在一个可选示例中，本申请实施例中的样本图像块的大小通常与深度神经网络对输入图像的尺寸要求相关，例如，样本图像块的大小可以为256×256等。为了获得具有预定大小的样本图像块，本申请实施例可以在进行车辆检测后，根据车辆检测结果对图像样本进行缩放处理，然后，按照预定大小以及车辆外接框信息(如车辆外接框的中心位置)，从缩放处理后的图像样本中切分出包含有车辆的样本图像块。在车辆检测结果包括多个车辆外接框信息的情况下，本申请实施例可以针对至少一个车辆外接框信息，分别对图像样本进行相应倍数的缩放处理，从而使获得的至少一个样本图像块分别具有预定大小。本申请实施例对样本图像块的大小以及缩放处理的具体实现方式不作限制。
在一个可选示例中，本申请实施例中的待训练的深度神经网络，会针对输入的至少一个样本图像块，进行车灯检测，并输出车灯检测结果，例如，输出车灯外接框信息、车灯亮灭状态以及车灯方位中的至少一个。在通常情况下，待训练的深度神经网络针对至少一个样本图像块均输出车灯外接框信息、车灯亮灭状态以及车灯方位。其中的车灯外接框信息通常是指能够表示出车灯外接框在样本图像块中的位置的信息(如位于车灯外接框对角线上的两个顶点在样本图像块上的坐标)，进而本申请实施例可以根据车灯外接框在图像块中的位置确定车灯在图像样本中的位置(如位于车灯外接框对角线上的两个顶点在图像样本上的坐标)。
当然，待训练的深度神经网络输出的车灯外接框信息也可以直接为车灯外接框在图像样本中的位置的信息。其中的车灯亮灭状态通常可以表示出车灯处于亮状态，还是处于灭状态。其中的车灯方位通常用于指示车灯在车辆中的方位，例如，车灯方位可以为：左前车灯、右前车灯、左后车灯或者右后车灯等。
在一个可选示例中，本申请实施例可以以待训练的深度神经网络输出的至少一个样本图像块的车灯检测结果与相应样本图像块的车灯标注信息之间的差异为指导信息，以减小差异为目的，利用相应的损失函数，对待训练的深度神经网络进行监督学习。
在一个可选示例中，在针对待训练的深度神经网络的训练达到预定迭代条件时，本次训练过程结束。本申请实施例中的预定迭代条件可以包括：待训练的深度神经网络输出的车灯检测结果与图像样本的车灯标注信息之间的差异满足预定差异要求。在差异满足该预定差异要求的情况下，本次对待训练的深度神经网络成功训练完成。本申请实施例中的预定迭代条件也可以包括：对该待训练的深度神经网络进行训练，所使用的图像样本的数量达到预定数量要求等。在使用的图像样本的数量达到预定数量要求，然而，差异并未满足预定差异要求的情况下，本次对待训练的深度神经网络并未训练成功。成功训练完成的深度神经网络可以用于对待处理图像中的包含有车辆的图像块进行车灯检测处理。
图4为本申请实施例提供的用于实现智能驾驶的方法的一个流程图。该方法可以由任意电子设备执行，例如终端设备、服务器、移动设备、车载设备等等，如图4所示，该实施例方法包括：步骤S400、步骤S410、步骤S420以及步骤S430。下面对图4中的步骤进行详细说明。
S400、获取包含有车辆的图像块。
在一个可选示例中，该步骤S400可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的获取图像块模块700执行。
S410、经由深度神经网络对图像块进行车灯检测，以获取车灯检测结果。
在一个可选示例中，该步骤S410可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的第一车灯检测模块710执行。
S420根据存在时序关系的多个图像块的车灯检测结果，确定车灯指示信息。
在一个可选示例中，该步骤S420可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的确定车灯指示模块720执行。
S430、根据车灯指示信息生成驾驶控制信息或驾驶预警提示信息。
在一个可选示例中，该步骤S430可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的智能控制模块730执行。
在一个可选示例中，车灯检测结果可以包括但不限于：车灯外接框信息、车灯亮灭状态以及车灯方位中的至少一个。在通常情况下，深度神经网络输出车灯外接框信息、车灯亮灭状态以及车灯方位。在一个可选示例中，本申请实施例中的车灯指示信息可以包括：单侧左车灯闪烁、单侧右车灯闪烁、双侧车灯闪烁、双侧车灯全灭或双侧车灯全亮等。另外，本申请实施例还可以进一步区分出上述左车灯为左前车灯还是左后车灯。也就是说，车灯指示信息可以包括：单侧左前车灯闪烁、单侧左后车灯闪烁、单侧右前车灯闪烁、单侧右后车灯闪烁、双侧前车灯闪烁、双侧后车灯闪烁、双侧前车灯全灭、双侧后车灯全灭、双侧前车灯全亮以及双侧后车灯全亮等。
双侧后车灯全亮的一个可选例子如图5所示，图5中左侧的“on”表示左侧后车灯亮，图5中右侧的“on”表示右侧后车灯亮。单侧右后车灯闪烁的一个可选例子如图6所示，图6中左侧的“off”表示左侧后车灯灭，图6中右侧的“flash”表示右侧后车灯闪烁。
在一个可选示例中，本申请实施例根据车灯指示信息，生成的驾驶控制信息可以包括：减速行驶控制信息、左并线控制信息、右并线控制信息、保持当前速度行驶控制信息或者加速行驶控制信息等。本申请根据车灯指示信息，生成的驾驶预警提示信息可以包括：前方车辆并线提示信息、前方车辆减速提示信息、前方车辆左/右转向提示信息等。本申请实施例不限制驾驶控制信息及驾驶预警提示信息的具体表现形式。
上述步骤S400、步骤S410以及步骤S420的可选实现方式，可以参见上述方法实施方式中针对图1的描述，在此不再详细说明。
本领域普通技术人员可以理解：实现上述方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成，前述的程序可以存储于一计算机可读取存储介质中，该程序在执行时，执行包括上述方法实施例的步骤；而前述的存储介质包括：ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
图7为本申请实施例提供的车灯检测装置的一个结构示意图。该实施例的装置可用于实现本申请上述任意实施例提供的车灯检测方法。如图7所示，该实施例的装置包括：获取图像块模块700以及第一车灯检测模块710。
可选的，该装置还可以包括但不限于：确定车灯指示模块720、智能控制模块730、第一修正模块740以及神经网络的训练装置750中的一个或多个。
获取图像块模块700用于获取包含有车辆的图像块。
第一车灯检测模块710用于经由深度神经网络对图像块进行车灯检测，以获取车灯检测结果。
确定车灯指示模块720用于根据存在时序关系的多个图像块的车灯检测结果，确定车灯指示信息。
智能控制模块730用于根据车灯指示信息生成驾驶控制信息或驾驶预警提示信息。
第一修正模块740用于针对存在时序关系的多个图像块，根据车灯检测结果，进行车灯外接框漏检和/或车灯外接框平滑处理，以获得修正的多个图像块中至少一个图像块中的车灯外接框信息。
神经网络的训练装置750用于利用带有车灯标注信息的样本图像块对待训练的深度神经网络进行训练。
在一个可选示例中，本申请实施例中的获取图像块模块700可以包括：检测模块以及切分模块。获取图像块模块700还可以包括：第二修正模块。其中的检测模块可以用于对待处理图像进行车辆检测。例如，检测模块利用车辆检测模型对待处理图像进行车辆检测，以获得待处理图像中的车辆外接框信息。其中的切分模块可以用于根据车辆检测结果，对待处理图像进行切分，以获得包含有车辆的图像块。其中的第二修正模块可以用于针对存在时序关系的多个待处理图像，进行车辆外接框漏检和/或车辆外接框平滑处理，以获得修正的多个待处理图像中至少一个待处理图像中的车辆外接框信息。
在一个可选示例中，本申请实施例中的第二修正模块可以包括：第三模块以及第四模块。其中的第三模块可以用于根据存在时序关系的多个待处理图像及其车辆外接框信息，获取至少一个车辆外接框所对应的车辆标识。其中的第四模块可以用于根据至少一个车辆外接框所对应的车辆标识，对至少一个待处理图像进行车辆外接框漏检和/或车辆外接框平滑处理。
在一个可选示例中，针对一个车辆标识而言，第四模块可以根据位于第m2个待处理图像(如前述描述中的当前待处理图像)之前的n2个(如8个)待处理图像中的具有车辆标识的车辆外接框的位置，预测第m2个待处理图像中的具有车辆标识的车辆外接框的位置；基于预测的车辆外接框的位置，第四模块对第m2个待处理图像进行车辆外接框添加或车辆外接框位置校正处理。例如，在第m2个待处理图像中不存在具有该车辆标识的车辆外接框的情况下，第四模块根据预测的车辆外接框的位置，为第m2个待处理图像添加车辆外接框，从而实现漏检处理；再例如，在第m2个待处理图像中存在具有该车辆标识的车辆外接框的情况下，第四模块可以对预测的车辆外接框的位置以及第m2个待处理图像中的具有车辆标识的车辆外接框的位置进行加权平均处理，从而实现平滑处理。
在一个可选示例中，本申请实施例中的车灯检测结果包括但不限于：车灯外接框信息、车灯亮灭状态以及车灯方位中的至少一个。
在一个可选示例中，本申请实施例中的第一修正模块740可以包括：第一单元和第二单元。其中的第一单元用于根据存在时序关系的多个图像块和多个图像块的车灯外接框信息，获取至少一个车灯外接框所对应的车灯标识。其中的第二单元用于根据至少一个车灯外接框所对应的车灯标识，对至少一个图像块进行车灯外接框漏检和/或车灯外接框平滑处理。例如，第二单元可以进一步用于，针对一车灯标识而言，根据位于第m1个图像块(如前述描述中的当前图像块)之前的n1个(如8个)图像块中的具有该车灯标识的车灯外接框的位置，预测第m1个图像块中的具有该车灯标识的车灯外接框的位置；基于预测的车灯外接框的位置，对第m1个图像块进行车灯外接框添加或车灯外接框位置校正处理。第二单元基于预测的车灯外接框的位置，对第m1个图像块进行车灯外接框添加或车灯外接框位置校正处理可以包括：在第m1个图像块中不存在具有车灯标识的车灯外接框的情况下，根据预测的车灯外接框的位置，为第m1个图像块添加车灯外接框。第二单元基于预测的车灯外接框的位置，对第m1个图像块进行车灯外接框添加或车灯外接框位置校正处理也可以包括：在第m1个图像块中存在具有车灯标识的车灯外接框的情况下，对预测的车灯外接框的位置以及第m1个图像块中的具有车灯标识的车灯外接框的位置进行加权平均处理。
在一个可选示例中，本申请实施例中的车灯指示信息可以包括但不限于：单侧左车灯闪烁、单侧右车灯闪烁、双侧车灯闪烁、双侧车灯全灭以及双侧车灯全亮中的至少一个。
在一个可选示例中，本申请实施例中的确定车灯指示模块720可以进一步用于针对存在时序关系的对应同一车辆的同一车灯的多个车灯外接框，进行车灯亮灭状态统计，并根据统计结果确定车灯指示信息。
在一个可选示例中，本申请实施例中的图像块可以为待处理图像。本申请实施例中的图像块也可以为待处理图像中包含车辆的局部图像。本申请实施例中的图像块还可以为基于待处理图像中包含车辆的局部图像处理而得到的图像块。
在一个可选示例中，本申请实施例中的深度神经网络包括：快速区域卷积神经网络(Faster RCNN)。
在一个可选示例中，本申请实施例中的神经网络的训练装置750可以包括：获取样本图像块模块800、第二车灯检测模块810以及监督模块820。其中的获取样本图像块模块800可以用于获取包括有车辆的样本图像块。其中的第二车灯检测模块810可以用于经由待训练的深度神经网络，对样本图像块进行车灯检测，以获取车灯检测结果。其中的监督模块820可以用于以车灯检测结果与样本图像块的车灯标注信息之间的差异为指导信息，对待训练的深度神经网络进行监督学习。本申请实施例中的车灯标注信息包括：车灯外接框信息标注信息、车灯亮灭状态标注信息以及车灯方位标注信息中的至少一个。 神经网络的训练装置750中的至少一个模块所执行的操作可以参见上述方法实施方式中针对图3的描述。神经网络的训练装置750的结构可以参见下述实施方式中针对图8的描述。在此不再重复说明。
在一个可选示例中，获取图像块模块700、第一车灯检测模块710、确定车灯指示模块720、智能控制模块730以及第一修正模块740所执行的操作及其技术效果，可以参见上述方法实施方式中针对图1、图2、图4、图5以及图6的描述。在此不再重复说明。
图8为本申请实施例提供的神经网络的训练装置的一个结构示意图。该实施例的装置可用于实现本申请上述任意实施例提供的神经网络的训练方法。如图8所示，该实施例的装置主要包括：获取样本图像块模块800、第二车灯检测模块810以及监督模块820。
获取样本图像块模块800可以用于获取包括有车辆的样本图像块。
第二车灯检测模块810可以用于经由待训练的深度神经网络，对样本图像块进行车灯检测，以获取车灯检测结果。
监督模块820可以用于以车灯检测结果与样本图像块的车灯标注信息之间的差异为指导信息，对待训练的深度神经网络进行监督学习。
本申请实施例中的车灯标注信息可以包括但不限于：车灯外接框信息标注信息、车灯亮灭状态标注信息以及车灯方位标注信息中的至少一个。
神经网络的训练装置750中的模块所执行的操作及其技术效果可以参见上述方法实施方式中针对图3的描述。在此不再重复说明。
图9为本申请实施例提供的用于实现智能驾驶的装置的一个结构示意图。该实施例的装置可用于实现本申请上述任意实施例提供的用于实现智能驾驶的方法。图9中的装置主要包括：获取图像块模块700、第一车灯检测模块710、确定车灯指示模块720以及智能控制模块730。可选的，该装置还可以包括：第一修正模块740。
获取图像块模块700主要用于获取包含有车辆的图像块。
第一车灯检测模块710主要用于经由深度神经网络对图像块进行车灯检测，以获取车灯检测结果。
确定车灯指示模块720主要用于根据存在时序关系的多个图像块的车灯检测结果，确定车灯指示信息。
智能控制模块730主要用于根据车灯指示信息生成驾驶控制信息或驾驶预警提示信息。
第一修正模块740主要用于针对存在时序关系的多个图像块，根据车灯检测结果，进行车灯外接框漏检和/或车灯外接框平滑处理，以获得修正的多个图像块中至少一个图像块中的车灯外接框信息。
获取图像块模块700、及第一车灯检测模块710、确定车灯指示模块720、智能控制模块730及第一修正模块740所执行的操作、技术效果以及模块的结构，可以参见上述方法实施方式中针对图1及图7中的相关描述。在此不再重复说明。
图10示出了适于实现本申请的示例性设备1000，设备1000可以是汽车中配置的控制系统/电子系统、移动终端(例如，智能移动电话等)、个人计算机(PC，例如，台式计算机或者笔记型计算机等)、平板电脑以及服务器等。图10中，设备1000包括一个或者多个处理器、通信部等，所述一个或者多个处理器可以为：一个或者多个中央处理单元(CPU)1001，和/或，一个或者多个利用神经网络进行车灯检测的图像处理器(GPU)1013等，处理器可以根据存储在只读存储器(ROM)1002中的可执行指令或者从存储部分1008加载到随机访问存储器(RAM)1003中的可执行指令而执行各种适当的动作和处理。通信部1012可以包括但不限于网卡，所述网卡可以包括但不限于IB(Infiniband)网卡。处理器可与只读存储器1002和/或随机访问存储器1003中通信以执行可执行指令，通过总线1004与通信部1012相连、并经通信部1012与其他目标设备通信，从而完成本申请中的相应步骤。
上述各指令所执行的操作可以参见上述方法实施例中的相关描述，在此不再详细说明。此外，在RAM 1003中，还可以存储有装置操作所需的各种程序以及数据。CPU1001、ROM1002以及RAM1003 通过总线1004彼此相连。
在有RAM1003的情况下，ROM1002为可选模块。RAM1003存储可执行指令，或在运行时向ROM1002中写入可执行指令，可执行指令使中央处理单元1001执行上述物体分割方法所包括的步骤。输入/输出(I/O)接口1005也连接至总线1004。通信部1012可以集成设置，也可以设置为具有多个子模块(例如，多个IB网卡)，并分别与总线连接。
以下部件连接至I/O接口1005：包括键盘、鼠标等的输入部分1006；包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出部分1007；包括硬盘等的存储部分1008；以及包括诸如LAN卡、调制解调器等的网络接口卡的通信部分1009。通信部分1009经由诸如因特网的网络执行通信处理。驱动器1010也根据需要连接至I/O接口1005。可拆卸介质1011，诸如磁盘、光盘、磁光盘、半导体存储器等等，根据需要安装在驱动器1010上，以便于从其上读出的计算机程序根据需要被安装在存储部分1008中。
需要特别说明的是，如图10所示的架构仅为一种可选实现方式，在可选实践过程中，可根据实际需要对上述图10的部件数量和类型进行选择、删减、增加或替换；在不同功能部件设置上，也可采用分离设置或集成设置等实现方式，例如，GPU和CPU可分离设置，再如理，可将GPU集成在CPU上，通信部可分离设置，也可集成设置在CPU或GPU上等。这些可替换的实施方式均落入本申请的保护范围。
特别地，根据本申请的实施方式，下文参考流程图描述的过程可以被实现为计算机软件程序，例如，本申请实施方式包括一种计算机程序产品，其包含有形地包含在机器可读介质上的计算机程序，计算机程序包含用于执行流程图所示的步骤的程序代码，程序代码可包括对应执行本申请提供的方法中的步骤对应的指令。
在这样的实施方式中，该计算机程序可以通过通信部分1009从网络上被下载及安装，和/或从可拆卸介质1011被安装。在该计算机程序被中央处理单元(CPU)1001执行时，执行本申请中记载的实现上述相应步骤的指令。
在一个或多个可选实施方式中，本申请实施例还提供了一种计算机程序程序产品，用于存储计算机可读指令，该指令被执行时使得计算机执行上述任意实施例中提供的车灯检测方法或者神经网络的训练方法或者用于实现智能驾驶的方法。
该计算机程序产品可以通过硬件、软件或其结合的方式实现。在一个可选例子中，该计算机程序产品体现为计算机存储介质，在另一个可选例子中，该计算机程序产品体现为软件产品，例如，软件开发包(Software Development Kit，SDK)等等。
在一个或多个可选实施方式中，本申请实施例还提供了另一种车灯检测方法和神经网络的训练方法及其对应的装置和电子设备、计算机存储介质、计算机程序以及计算机程序产品，其中的方法包括：第一装置向第二装置发送车灯检测指示或者训练神经网络指示或者用于实现智能驾驶的指示，该指示使得第二装置执行上述任一可能的实施例中的车灯检测方法或者训练神经网络方法或者用于实现智能驾驶的方法；第一装置接收第二装置发送的车灯检测结果或者神经网络的训练结果或者用于实现智能驾驶的驾驶控制信息或驾驶预警提示信息。
在一些实施例中，该车灯检测指示或者训练神经网络指示可以为调用指令，第一装置可以通过调用的方式指示第二装置执行车灯检测操作或者训练神经网络操作或者用于实现智能驾驶的操作，相应地，响应于接收到调用指令，第二装置可以执行上述车灯检测方法或者训练神经网络的方法或者用于实现智能驾驶的方法中的任意实施例中的步骤和/或流程。
应理解，本申请实施例中的“第一”、“第二”等术语仅仅是为了区分，而不应理解成对本申请实施例的限定。还应理解，在本申请中，“多个”可以指两个或两个以上，“至少一个”可以指一个、两个或两个以上。还应理解，对于本申请中提及的任一部件、数据或结构，在没有明确限定或者在前后文给出相反 启示的情况下，一般可以理解为一个或多个。还应理解，本申请对各个实施例的描述着重强调各个实施例之间的不同之处，其相同或相似之处可以相互参考，为了简洁，不再一一赘述。
本说明书中各个实施例均采用递进的方式描述，每个实施例重点说明的都是与其它实施例的不同之处，各个实施例之间相同或相似的部分相互参见即可。对于系统实施例而言，由于其与方法实施例基本对应，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
可能以许多方式来实现本申请的方法和装置。例如，可通过软件、硬件、固件或者软件、硬件、固件的任何组合来实现本申请的方法和装置。用于所述方法的步骤的上述顺序仅是为了进行说明，本申请的方法的步骤不限于以上具体描述的顺序，除非以其它方式特别说明。此外，在一些实施例中，还可将本申请实施为记录在记录介质中的程序，这些程序包括用于实现根据本申请的方法的机器可读指令。因而，本申请还覆盖存储用于执行根据本申请的方法的程序的记录介质。
本申请的描述是为了示例和描述起见而给出的，而并不是无遗漏的或者将本申请限于所公开的形式。很多修改和变化对于本领域的普通技术人员而言是显然的。选择和描述实施例是为了更好说明本申请的原理和实际应用，并且使本领域的普通技术人员能够理解本申请从而设计适于特定用途的带有各种修改的各种实施例。

Claims (52)

  1. 一种车灯检测方法，其特征在于，包括：
    获取包含有车辆的图像块；
    经由深度神经网络对所述图像块进行车灯检测，以获取车灯检测结果。
  2. 根据权利要求1所述的方法，其特征在于，所述车灯检测结果包括：车灯外接框信息、车灯亮灭状态以及车灯方位中的至少一个。
  3. 根据权利要求1至2中任一项所述的方法，其特征在于，所述方法还包括：
    根据存在时序关系的多个图像块的车灯检测结果，确定车灯指示信息。
  4. 根据权利要求3所述的方法，其特征在于，所述方法还包括：
    根据所述车灯指示信息生成驾驶控制信息或驾驶预警提示信息。
  5. 根据权利要求3至4中任一项所述的方法，其特征在于，所述方法在根据存在时序关系的多个图像块的车灯检测结果，确定车灯指示信息之前，还包括：
    针对存在时序关系的多个图像块，根据车灯检测结果，进行车灯外接框漏检和/或车灯外接框平滑处理，以获得修正的多个图像块中的车灯外接框信息。
  6. 根据权利要求5所述的方法，其特征在于，所述针对存在时序关系的多个图像块，根据车灯检测结果，进行车灯外接框漏检和/或车灯外接框平滑处理包括：
    根据存在时序关系的多个图像块和所述多个图像块中车灯外接框信息，获取至少一个车灯外接框所对应的车灯标识；
    根据所述至少一个车灯外接框所对应的车灯标识，对所述多个图像块中至少一个图像块进行车灯外接框漏检和/或车灯外接框平滑处理。
  7. 根据权利要求6所述的方法，其特征在于，所述根据所述至少一个车灯外接框所对应的车灯标识，对所述多个图像块中至少一个图像块进行车灯外接框漏检和/或车灯外接框平滑处理包括：
    针对一车灯标识而言，根据位于第m1个图像块之前的n1个图像块中的具有所述车灯标识的车灯外接框的位置，预测所述第m1个图像块中的具有所述车灯标识的车灯外接框的位置；
    基于所述预测的车灯外接框的位置，对所述第m1个图像块进行车灯外接框添加或车灯外接框位置校正处理。
  8. 根据权利要求7所述的方法，其特征在于，所述基于所述预测的车灯外接框的位置，对所述第m1个图像块进行车灯外接框添加或车灯外接框位置校正处理包括：
    在所述第m1个图像块中不存在具有所述车灯标识的车灯外接框的情况下，根据所述预测的车灯外接框的位置，为所述第m1个图像块添加车灯外接框；或者
    在所述第m1个图像块中存在具有所述车灯标识的车灯外接框的情况下，对所述预测的车灯外接框的位置以及所述第m1个图像块中的具有所述车灯标识的车灯外接框的位置进行加权平均处理。
  9. 根据权利要求3至8中任一项所述的方法，其特征在于，所述车灯指示信息包括：
    单侧左车灯闪烁、单侧右车灯闪烁、双侧车灯闪烁、双侧车灯全灭以及双侧车灯全亮中的至少一个。
  10. 根据权利要求9所述的方法，其特征在于，所述根据存在时序关系的多个图像块的车灯检测结果，确定车灯指示信息包括：
    针对存在时序关系的对应同一车辆的同一车灯的多个车灯外接框，进行车灯亮灭状态统计，并根据统计结果确定所述车灯指示信息。
  11. 根据权利要求1至10中任一项所述的方法，其特征在于，
    所述图像块为待处理图像；或者，
    所述图像块为待处理图像中包含车辆的局部图像，或者，
    所述图像块为基于待处理图像中包含车辆的局部图像处理而得到的图像块。
  12. 根据权利要求11所述的方法，其特征在于，所述获取包含有车辆的图像块包括：
    对所述待处理图像进行车辆检测；
    根据车辆检测结果，对所述待处理图像进行切分，以获得包含有车辆的所述图像块。
  13. 根据权利要求12所述的方法，其特征在于，所述对待处理图像进行车辆检测包括：
    利用车辆检测模型对所述待处理图像进行车辆检测，以获得所述待处理图像中的车辆外接框信息。
  14. 根据权利要求13所述的方法，其特征在于，所述对待处理图像进行车辆检测还包括：
    针对存在时序关系的多个所述待处理图像，进行车辆外接框漏检和/或车辆外接框平滑处理，以获得修正的多个所述待处理图像中的车辆外接框信息。
  15. 根据权利要求14所述的方法，其特征在于，所述针对存在时序关系的多个所述待处理图像，进行车辆外接框漏检和/或平滑处理包括：
    根据存在时序关系的多个所述待处理图像及其车辆外接框信息，获取至少一个车辆外接框所对应的车辆标识；
    根据所述至少一个车辆外接框所对应的车辆标识，对所述多个待处理图像中的至少一个待处理图像进行车辆外接框漏检和/或车辆外接框平滑处理。
  16. 根据权利要求15所述的方法，其特征在于，所述根据所述至少一个车辆外接框所对应的车辆标识，对所述多个待处理图像中的至少一个待处理图像进行车辆外接框漏检和/或车辆外接框平滑处理包括：
    针对一车辆标识而言，根据位于第m2个待处理图像之前的n2个待处理图像中的具有所述车辆标识的车辆外接框的位置，预测所述第m2个待处理图像中的具有所述车辆标识的车辆外接框的位置；
    基于所述预测的车辆外接框的位置，对所述第m2个待处理图像进行车辆外接框添加或车辆外接框位置校正处理。
  17. 根据权利要求16所述的方法，其特征在于，所述基于所述预测的车辆外接框的位置，对所述第m2个待处理图像进行车辆外接框添加或车辆外接框位置校正处理包括：
    在所述第m2个待处理图像中不存在具有所述车辆标识的车辆外接框的情况下，根据所述预测的车辆外接框的位置，为所述第m2个待处理图像添加车辆外接框；或者
    在所述第m2个待处理图像中存在具有所述车辆标识的车辆外接框的情况下，对所述预测的车辆外接框的位置以及所述第m2个待处理图像中的具有所述车辆标识的车辆外接框的位置进行加权平均处理。
  18. 根据权利要求1至17中任一项所述的方法，其特征在于，所述深度神经网络包括：快速区域卷积神经网络。
  19. 根据权利要求1至18中任一项所述的方法，其特征在于，所述深度神经网络是利用带有车灯标注信息的样本图像块训练获得的。
  20. 根据权利要求19所述的方法，其特征在于，所述训练所述深度神经网络的过程包括：
    获取包括有车辆的样本图像块；
    经由待训练的深度神经网络，对所述样本图像块进行车灯检测，以获取车灯检测结果；
    以所述车灯检测结果与所述样本图像块的车灯标注信息之间的差异为指导信息，对所述待训练的深度神经网络进行监督学习。
  21. 根据权利要求19至20中任一项所述的方法，其特征在于，所述车灯标注信息包括：车灯外接框信息标注信息、车灯亮灭状态标注信息以及车灯方位标注信息中的至少一个。
  22. 一种神经网络的训练方法，其特征在于，包括：
    获取包括有车辆的样本图像块；
    经由待训练的深度神经网络，对所述样本图像块进行车灯检测，以获取车灯检测结果；
    以所述车灯检测结果与所述样本图像块的车灯标注信息之间的差异为指导信息，对所述待训练的深度神经网络进行监督学习。
  23. 根据权利要求22所述的方法，其特征在于，所述车灯标注信息包括：车灯外接框信息标注信息、车灯亮灭状态标注信息以及车灯方位标注信息中的至少一个。
  24. 一种用于实现智能驾驶的方法，其特征在于，包括：
    获取包含有车辆的图像块；
    经由深度神经网络对所述图像块进行车灯检测，以获取车灯检测结果；
    根据存在时序关系的多个图像块的车灯检测结果，确定车灯指示信息；
    根据所述车灯指示信息生成驾驶控制信息或驾驶预警提示信息。
  25. 一种车灯检测装置，其特征在于，包括：
    获取图像块模块，用于获取包含有车辆的图像块；
    第一车灯检测模块，用于经由深度神经网络对所述图像块进行车灯检测，以获取车灯检测结果。
  26. 根据权利要求25所述的装置，其特征在于，所述车灯检测结果包括：车灯外接框信息、车灯亮灭状态以及车灯方位中的至少一个。
  27. 根据权利要求25至26中任一项所述的装置，其特征在于，所述装置还包括：
    确定车灯指示模块，用于根据存在时序关系的多个图像块的车灯检测结果，确定车灯指示信息。
  28. 根据权利要求27所述的装置，其特征在于，所述装置还包括：
    智能控制模块，用于根据所述车灯指示信息生成驾驶控制信息或驾驶预警提示信息。
  29. 根据权利要求27至28中任一项所述的装置，其特征在于，所述装置还包括：
    第一修正模块，用于针对存在时序关系的多个图像块，根据车灯检测结果，进行车灯外接框漏检和/或车灯外接框平滑处理，以获得修正的多个图像块中的车灯外接框信息。
  30. 根据权利要求29所述的装置，其特征在于，所述第一修正模块包括：
    第一单元，用于根据存在时序关系的多个图像块和所述多个图像块中车灯外接框信息，获取至少一个车灯外接框所对应的车灯标识；
    第二单元，用于根据所述至少一个车灯外接框所对应的车灯标识，对所述多个图像块中至少一个图像块进行车灯外接框漏检和/或车灯外接框平滑处理。
  31. 根据权利要求30所述的装置，其特征在于，所述第二单元进一步用于：
    针对一车灯标识而言，根据位于第m1个图像块之前的n1个图像块中的具有所述车灯标识的车灯外接框的位置，预测所述第m1个图像块中的具有所述车灯标识的车灯外接框的位置；
    基于所述预测的车灯外接框的位置，对所述第m1个图像块进行车灯外接框添加或车灯外接框位置校正处理。
  32. 根据权利要求31所述的装置，其特征在于，所述第二单元进一步用于：
    在所述第m1个图像块中不存在具有所述车灯标识的车灯外接框的情况下，根据所述预测的车灯外接框的位置，为所述第m1个图像块添加车灯外接框；或者
    在所述第m1个图像块中存在具有所述车灯标识的车灯外接框的情况下，对所述预测的车灯外接框的位置以及所述第m1个图像块中的具有所述车灯标识的车灯外接框的位置进行加权平均处理。
  33. 根据权利要求27至32中任一项所述的装置，其特征在于，所述车灯指示信息包括：
    单侧左车灯闪烁、单侧右车灯闪烁、双侧车灯闪烁、双侧车灯全灭以及双侧车灯全亮中的至少一个。
  34. 根据权利要求33所述的装置，其特征在于，所述确定车灯指示模块进一步用于：
    针对存在时序关系的对应同一车辆的同一车灯的多个车灯外接框，进行车灯亮灭状态统计，并根据统计结果确定所述车灯指示信息。
  35. 根据权利要求25至34中任一项所述的装置，其特征在于，
    所述图像块为待处理图像；或者，
    所述图像块为待处理图像中包含车辆的局部图像，或者，
    所述图像块为基于待处理图像中包含车辆的局部图像处理而得到的图像块。
  36. 根据权利要求35所述的装置，其特征在于，所述获取图像块模块包括：
    检测模块，用于对所述待处理图像进行车辆检测；
    切分模块，用于根据车辆检测结果，对所述待处理图像进行切分，以获得包含有车辆的所述图像块。
  37. 根据权利要求36所述的装置，其特征在于，所述检测模块进一步用于：
    利用车辆检测模型对所述待处理图像进行车辆检测，以获得所述待处理图像中的车辆外接框信息。
  38. 根据权利要求37所述的装置，其特征在于，所述获取图像块模块还包括：
    第二修正模块，用于针对存在时序关系的多个所述待处理图像，进行车辆外接框漏检和/或车辆外接框平滑处理，以获得修正的多个所述待处理图像中的车辆外接框信息。
  39. 根据权利要求38所述的装置，其特征在于，所述第二修正模块包括：
    第三模块，用于根据存在时序关系的多个所述待处理图像及其车辆外接框信息，获取至少一个车辆外接框所对应的车辆标识；
    第四模块，用于根据所述至少一个车辆外接框所对应的车辆标识，对所述多个待处理图像中的至少一个待处理图像进行车辆外接框漏检和/或车辆外接框平滑处理。
  40. 根据权利要求39所述的装置，其特征在于，所述第四模块进一步用于：
    针对一车辆标识而言，根据位于第m2个待处理图像之前的n2个待处理图像中的具有所述车辆标识的车辆外接框的位置，预测所述第m2个待处理图像中的具有所述车辆标识的车辆外接框的位置；
    基于所述预测的车辆外接框的位置，对所述第m2个待处理图像进行车辆外接框添加或车辆外接框位置校正处理。
  41. 根据权利要求40所述的装置，其特征在于，所述第四模块进一步用于：
    在所述第m2个待处理图像中不存在具有所述车辆标识的车辆外接框的情况下，根据所述预测的车辆外接框的位置，为所述第m2个待处理图像添加车辆外接框；或者
    在所述第m2个待处理图像中存在具有所述车辆标识的车辆外接框的情况下，对所述预测的车辆外接框的位置以及所述第m2个待处理图像中的具有所述车辆标识的车辆外接框的位置进行加权平均处理。
  42. 根据权利要求25至41中任一项所述的装置，其特征在于，所述深度神经网络包括：快速区域卷积神经网络。
  43. 根据权利要求25至42中任一项所述的装置，其特征在于，所述深度神经网络是，神经网络的训练装置利用带有车灯标注信息的样本图像块训练获得的。
  44. 根据权利要求43所述的装置，其特征在于，所述神经网络的训练装置包括：
    获取样本图像块模块，用于获取包括有车辆的样本图像块；
    第二车灯检测模块，用于经由待训练的深度神经网络，对所述样本图像块进行车灯检测，以获取车灯检测结果；
    监督模块，用于以所述车灯检测结果与所述样本图像块的车灯标注信息之间的差异为指导信息，对所述待训练的深度神经网络进行监督学习。
  45. 根据权利要求43至44中任一项所述的装置，其特征在于，所述车灯标注信息包括：车灯外接框信息标注信息、车灯亮灭状态标注信息以及车灯方位标注信息中的至少一个。
  46. 一种神经网络的训练装置，其特征在于，包括：
    获取样本图像块模块，用于获取包括有车辆的样本图像块；
    第二车灯检测模块，用于经由待训练的深度神经网络，对所述样本图像块进行车灯检测，以获取车灯检测结果；
    监督模块，用于以所述车灯检测结果与所述样本图像块的车灯标注信息之间的差异为指导信息，对所述待训练的深度神经网络进行监督学习。
  47. 根据权利要求46所述的装置，其特征在于，所述车灯标注信息包括：车灯外接框信息标注信息、车灯亮灭状态标注信息以及车灯方位标注信息中的至少一个。
  48. 一种用于实现智能驾驶的装置，其特征在于，包括：
    获取图像块模块，用于获取包含有车辆的图像块；
    第一车灯检测模块，用于经由深度神经网络对所述图像块进行车灯检测，以获取车灯检测结果；
    确定车灯指示模块，用于根据存在时序关系的多个图像块的车灯检测结果，确定车灯指示信息；
    智能控制模块，用于根据所述车灯指示信息生成驾驶控制信息或驾驶预警提示信息。
  49. 一种电子设备，其特征在于，包括：
    存储器，用于存储计算机程序；
    处理器，用于执行所述存储器中存储的计算机程序，且所述计算机程序被执行时，实现上述权利要求1至24任意任一项所述的方法。
  50. 一种电子设备，其特征在于，包括：
    处理器和权利要求25至48任意一项所述的装置；在处理器运行所述装置时，权利要求25至48任意一项所述的装置中的单元被运行。
  51. 一种计算机可读存储介质，其特征在于，其上存储有计算机程序，该计算机程序被处理器执行时，实现上述权利要求1至24任意一项所述的方法。
  52. 一种计算机程序产品，包括计算机可读代码，其特征在于，当所述计算机可读代码在设备上运行时，所述设备中的处理器执行用于实现权利要求1至24任意一项所述的方法。
PCT/CN2019/079300 2018-03-30 2019-03-22 车灯检测方法、实现智能驾驶的方法、装置、介质及设备 WO2019184825A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
SG11201909146W SG11201909146WA (en) 2018-03-30 2019-03-22 Vehicle lamp detection methods and apparatuses, methods and apparatuses for implementing inteliggent driving, media and devices
US16/588,126 US10984266B2 (en) 2018-03-30 2019-09-30 Vehicle lamp detection methods and apparatuses, methods and apparatuses for implementing intelligent driving, media and devices

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810278351.7 2018-03-30
CN201810278351.7A CN108509907B (zh) 2018-03-30 2018-03-30 车灯检测方法、实现智能驾驶的方法、装置、介质及设备

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/588,126 Continuation US10984266B2 (en) 2018-03-30 2019-09-30 Vehicle lamp detection methods and apparatuses, methods and apparatuses for implementing intelligent driving, media and devices

Publications (1)

Publication Number Publication Date
WO2019184825A1 true WO2019184825A1 (zh) 2019-10-03

Family

ID=63379362

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/079300 WO2019184825A1 (zh) 2018-03-30 2019-03-22 车灯检测方法、实现智能驾驶的方法、装置、介质及设备

Country Status (4)

Country Link
US (1) US10984266B2 (zh)
CN (1) CN108509907B (zh)
SG (1) SG11201909146WA (zh)
WO (1) WO2019184825A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11415985B2 (en) * 2018-10-22 2022-08-16 Robert Bosch Gmbh Method and device for ascertaining a state of a vehicle light of a vehicle

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108509907B (zh) 2018-03-30 2022-03-15 北京市商汤科技开发有限公司 车灯检测方法、实现智能驾驶的方法、装置、介质及设备
CN109389064B (zh) * 2018-09-27 2021-02-23 东软睿驰汽车技术（沈阳）有限公司 一种车辆特征获取方法及装置
US10831189B2 (en) * 2019-01-31 2020-11-10 StradVision, Inc. Learning method and learning device for providing functional safety by warning driver about potential dangerous situation by using explainable AI which verifies detection processes of autonomous driving network, and testing method and testing device using the same
CN110532990A (zh) * 2019-09-04 2019-12-03 上海眼控科技股份有限公司 转向灯使用状态识别方法、装置、计算机设备及存储介质
CN110647863B (zh) * 2019-09-30 2024-02-02 重庆元韩汽车技术设计研究院有限公司 用于智能驾驶的视觉信号采集分析系统
CN110969647B (zh) * 2019-12-04 2023-06-30 苏州智加科技有限公司 一种整合车辆的识别跟踪与车灯检测的方法
US11741722B2 (en) * 2020-09-04 2023-08-29 International Business Machines Corporation Coarse-to-fine attention networks for light signal detection and recognition
CN112699781A (zh) * 2020-12-29 2021-04-23 上海眼控科技股份有限公司 车灯状态检测方法、装置、计算机设备和可读存储介质
CN113537003B (zh) * 2021-07-02 2022-10-21 安阳工学院 一种用于辅助驾驶的车外环境视觉检测方法及装置
CN114323583A (zh) * 2021-12-21 2022-04-12 广汽本田汽车有限公司 一种车辆灯光检测方法、装置、设备及系统
CN114677500B (zh) * 2022-05-25 2022-08-23 松立控股集团股份有限公司 一种基于眼动仪点标注信息的弱监督视频车牌识别方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103927548A (zh) * 2014-04-18 2014-07-16 北京航空航天大学 一种新的避免车辆碰撞的刹车行为检测方法
CN103984950A (zh) * 2014-04-22 2014-08-13 北京联合大学 一种适应白天检测的运动车辆刹车灯状态识别方法
US20150304539A1 (en) * 2013-01-28 2015-10-22 Conti Temic Microelectronic Gmbh Method for Detecting Light Sources Operated in Pulsed Mode
CN105975915A (zh) * 2016-04-28 2016-09-28 大连理工大学 一种基于多任务卷积神经网络的前方车辆参数识别方法
CN107316010A (zh) * 2017-06-13 2017-11-03 武汉理工大学 一种识别前方车辆尾灯及判断其状态的方法
CN108509907A (zh) * 2018-03-30 2018-09-07 北京市商汤科技开发有限公司 车灯检测方法、实现智能驾驶的方法、装置、介质及设备

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6382721B2 (ja) * 2012-11-27 2018-08-29 サターン ライセンシング エルエルシーＳａｔｕｒｎ Ｌｉｃｅｎｓｉｎｇ ＬＬＣ 表示装置及び表示方法、並びにコンピューター・プログラム
CN103607554B (zh) * 2013-10-21 2017-10-20 易视腾科技股份有限公司 一种基于全自动人脸无缝合成的视频合成方法
US20170116488A1 (en) * 2015-10-23 2017-04-27 MAGNETI MARELLI S.p.A. Method for identifying an incoming vehicle and corresponding system
JP6310899B2 (ja) * 2015-11-25 2018-04-11 株式会社Ｓｕｂａｒｕ 車外環境認識装置
US10387736B2 (en) * 2017-09-20 2019-08-20 TuSimple System and method for detecting taillight signals of a vehicle

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150304539A1 (en) * 2013-01-28 2015-10-22 Conti Temic Microelectronic Gmbh Method for Detecting Light Sources Operated in Pulsed Mode
CN103927548A (zh) * 2014-04-18 2014-07-16 北京航空航天大学 一种新的避免车辆碰撞的刹车行为检测方法
CN103984950A (zh) * 2014-04-22 2014-08-13 北京联合大学 一种适应白天检测的运动车辆刹车灯状态识别方法
CN105975915A (zh) * 2016-04-28 2016-09-28 大连理工大学 一种基于多任务卷积神经网络的前方车辆参数识别方法
CN107316010A (zh) * 2017-06-13 2017-11-03 武汉理工大学 一种识别前方车辆尾灯及判断其状态的方法
CN108509907A (zh) * 2018-03-30 2018-09-07 北京市商汤科技开发有限公司 车灯检测方法、实现智能驾驶的方法、装置、介质及设备

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11415985B2 (en) * 2018-10-22 2022-08-16 Robert Bosch Gmbh Method and device for ascertaining a state of a vehicle light of a vehicle

Also Published As

Publication number Publication date
SG11201909146WA (en) 2019-11-28
US20200026936A1 (en) 2020-01-23
US10984266B2 (en) 2021-04-20
CN108509907B (zh) 2022-03-15
CN108509907A (zh) 2018-09-07

Similar Documents

Publication Publication Date Title
WO2019184825A1 (zh) 车灯检测方法、实现智能驾驶的方法、装置、介质及设备
US11643076B2 (en) Forward collision control method and apparatus, electronic device, program, and medium
CN107871129B (zh) 用于处理点云数据的方法和装置
WO2019228211A1 (zh) 基于车道线的智能驾驶控制方法和装置、电子设备
WO2019179464A1 (zh) 用于预测目标对象运动朝向的方法、车辆控制方法及装置
US20220130153A1 (en) Vehicle control method, apparatus, electronic device and vehicle
EP3859708B1 (en) Traffic light image processing method and device, and roadside device
CN109035831B (zh) 交通指示灯的识别方法、装置、设备、存储介质及车辆
US11818493B2 (en) Fire source detection method and device under condition of small sample size and storage medium
US10373316B2 (en) Images background subtraction for dynamic lighting scenarios
GB2496266A (en) Improved abandoned object recognition using pedestrian detection
WO2023075863A1 (en) Adversarial contrastive learning and active adversarial contrastive mix (adversemix) for semi-supervised semantic segmentation
CN111950523A (zh) 基于航拍的船只检测优化方法、装置、电子设备及介质
CN111079621A (zh) 检测对象的方法、装置、电子设备和存储介质
EP3842996A1 (en) Method of and system for determining traffic signal state
CN108509876B (zh) 针对视频的物体检测方法、装置、设备、存储介质及程序
CN110817674B (zh) 一种扶梯的缺级检测方法、装置、设备及存储介质
EP3315949B1 (en) Detection system, detection method, and program
US20170256166A1 (en) Information processing apparatus, information processing method, and computer program product
CN113353072A (zh) 一种行车控制方法、装置、服务器及介质
CN113963322B (zh) 一种检测模型训练方法、装置及电子设备
CN115375987B (zh) 一种数据标注方法、装置、电子设备及存储介质
JP2014092964A (ja) 比較的低品質な観測画像から、画像内のオブジェクトの数を推定する方法、コンピュータ・プログラム、コンピュータ。
US20230267749A1 (en) System and method of segmenting free space based on electromagnetic waves
CN113627284A (zh) 基于改进CenterNet的实时火焰检测方法及检测装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19777247

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS (EPO FORM 1205A DATED 25.01.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19777247

Country of ref document: EP

Kind code of ref document: A1