CN115797469A

CN115797469A - Signal lamp image processing method, device, equipment and storage medium

Info

Publication number: CN115797469A
Application number: CN202111044071.8A
Authority: CN
Inventors: 郭一民; 戴开恒
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-09-07
Filing date: 2021-09-07
Publication date: 2023-03-14

Abstract

A signal lamp image processing technology belongs to the technical field of intelligent traffic. In the signal lamp image processing technology, a first image containing a signal lamp is processed through a signal lamp detection model, and state information and an outline of the signal lamp with high accuracy are obtained, so that color enhancement can be performed on an area where the signal lamp is located in the first image according to the color of the signal lamp indicated by the state information and the display shape of the signal lamp indicated by the outline, and a second image is obtained. Since the accuracy of the color of the traffic light and the display shape of the traffic light in the second image is high, the state change of the traffic light is detected based on the second image, and a highly accurate detection result can be obtained.

Description

Signal lamp image processing method, device and equipment and storage medium

Technical Field

The invention relates to the technical field of intelligent traffic, in particular to a signal lamp image processing method, a signal lamp image processing device, signal lamp image processing equipment and a storage medium.

Background

In daily traffic management, the states of signal lamps at intersections are regularly changed to ensure the orderly progress of traffic order, thereby ensuring traffic safety and smooth roads. In recent years, with the development of the intelligent transportation industry, the traffic management department has a strong desire to automatically detect the state change of the signal lamp through the video monitoring system and automatically detect the traffic violation behaviors by combining the current driving behaviors of the vehicle. In a video monitoring system, the imaging effect of a signal lamp can directly influence the detection result of the state change of the signal lamp.

However, since the signal lamp is photographed outdoors, the imaging effect is easily affected by external conditions such as shading, illumination, weather, day and night, and the like, which causes problems such as white and yellow images of the signal lamp, and deformed display shape, for example, a yellow signal lamp is displayed by color cast after the red signal lamp is imaged, and a round signal lamp is displayed by thickening after the arrow signal lamp is imaged, and therefore, in some scenes affected by the external conditions, the color recognition of the signal lamp is inaccurate, the accuracy of the contour recognition is poor, and the accuracy of the detection result of the state change of the signal lamp is low.

Disclosure of Invention

The embodiment of the invention provides a signal lamp image processing method, a signal lamp image processing device, signal lamp image processing equipment and a signal lamp image processing storage medium, which can improve the accuracy of a detection result of signal lamp state change. The technical scheme is as follows:

in a first aspect, a signal lamp image processing method is provided, and the method includes:

processing a first image containing a signal lamp based on a signal lamp detection model to obtain state information and a contour of the signal lamp, wherein the signal lamp detection model is obtained based on artificial intelligence model training, the state information indicates the color of the signal lamp, and the contour indicates the display shape of the signal lamp;

and based on the state information and the outline, performing color enhancement on the area where the signal lamp is located in the first image to obtain a second image.

Through the signal lamp detection model, a first image containing a signal lamp is processed to obtain state information and a contour of the signal lamp, wherein the signal lamp detection model is obtained through training of an artificial intelligence model, so that the signal lamp detection model can output accurate state information and contour of the signal lamp according to extracted semantic information of the first image, and accuracy of a detection result of state change of the signal lamp is effectively improved. Further, according to the color of the signal lamp indicated by the state information and the display shape of the signal lamp indicated by the outline, color enhancement is performed on the area where the signal lamp is located in the first image, and a second image is obtained. Since the accuracy of the color of the traffic light and the display shape of the traffic light in the second image is high, the state change of the traffic light is detected based on the second image, and a highly accurate detection result can be obtained.

In some implementations, the color of the area in the second image where the signal lamp is located matches the color of the signal lamp in the real world; the display shape of the area where the signal lamp is located in the second image is matched with the display shape of the signal lamp in the real world. The state change of the signal lamp is detected based on the second image, and a detection result with accurate color identification and high contour identification accuracy can be obtained.

In some implementations, the color enhancement of the area where the signal lamp is located in the first image based on the state information and the contour to obtain a second image includes: when the state information indicates that the color of the signal lamp is red, performing color enhancement on the area where the signal lamp is located in the first image based on the state information and the outline to obtain a second image; and when the state information indicates that the color of the signal lamp is non-red, the color of the area where the signal lamp is located in the first image is not enhanced.

Under the condition that the color of the state information indicating signal lamp is red, the color of the area where the signal lamp is located in the first image is continuously enhanced, and redundant signal lamp images can be avoided from being processed, so that the data processing amount of the server is reduced, the load consumption of the server is reduced, and the signal lamp image processing efficiency is improved.

In some implementation manners, the color enhancement of the area where the signal lamp is located in the first image based on the state information and the contour to obtain a second image includes: and when the signal lamp basic outline matched with the outline exists in the signal lamp basic outline library, based on the state information and the outline, performing color enhancement on the area where the signal lamp is located in the first image to obtain the second image.

The contour output by the signal lamp detection model is further judged, and after the signal lamp basic contour matched with the contour exists in the signal lamp basic contour library, image processing is carried out to obtain a second image so as to ensure the accuracy of the contour of the signal lamp and improve the accuracy of the detection result of the state change of the signal lamp.

In some implementations, the method further includes: when the signal lamp basic outline matched with the outline does not exist in the signal lamp basic outline library, correcting the outline to obtain the corrected outline; and based on the state information and the corrected outline, performing color enhancement on the area where the signal lamp is located in the first image to obtain the second image.

After the state information and the outline of the signal lamp are obtained through the signal lamp detection model, the outline is corrected under the condition that the accuracy of the outline is not in accordance with the requirement, so that a second image is obtained according to the corrected outline and the state information, the state change of the signal lamp is detected based on the second image, and the detection result with accurate color identification and high outline identification accuracy can be obtained.

In some implementations, the modifying the contour to obtain a modified contour includes: determining the outline type of the signal lamp based on the outline; determining at least one signal lamp basic contour corresponding to the contour type from the signal lamp basic contour library; and correcting the contour based on the difference between the basic contour and the contour of the at least one signal lamp to obtain the corrected contour.

The contour output by the model is corrected through the signal lamp basic contour in the signal lamp basic contour library, so that the corrected contour can meet the requirement of the signal lamp basic contour, and the accuracy of the corrected contour is improved.

In some implementations, the method further includes: and storing the corrected outline as a signal lamp basic outline in the signal lamp basic outline library.

The corrected outline is stored, and the signal lamp basic outline library is updated in time, so that signal lamp basic outlines in various scenes such as different time periods, different weather conditions, different illumination and the like are contained in the signal lamp basic outline library, and the server can obtain the accurate outline of the signal lamp according to the image containing the signal lamp aiming at various complex scenes.

In some implementations, the storing the modified outline as a signal light basic outline in the signal light basic outline library includes: when the corrected outline meets the target condition, the corrected outline is used as a signal lamp basic outline and stored in the signal lamp basic outline library; the target condition means that the modified contour is a continuous contour meeting a target rule, and the contour similarity between the modified contour and the basic contour of the target signal lamp is greater than or equal to a target threshold value.

The corrected outline meeting the target condition is stored, and on the basis of updating the signal lamp basic outline library in time, the accuracy of the signal lamp basic outline in the signal lamp basic outline library is further ensured, so that the server can obtain the accurate outline of the signal lamp according to the image containing the signal lamp aiming at various complex scenes.

In some implementations, the method further includes:

acquiring an original image which is shot by shooting equipment and contains the signal lamp;

and performing image clipping on the original image to obtain the first image, wherein the size of the area where the signal lamp is located in the first image conforms to the target size.

The original image containing the signal lamp is subjected to image cutting, so that the size of the area where the signal lamp is located in the first image accords with the target size, the signal lamp detection model is convenient to perform feature extraction based on the area where the signal lamp is located in the same size at each time, and the stability of the model and the accuracy of a model output result are ensured.

In a second aspect, a method for training a signal lamp detection model is provided, where the signal lamp detection model is obtained by training based on an artificial intelligence model, and the method includes:

in the ith iteration of training the artificial intelligence model, inputting a sample image containing a sample signal lamp into the artificial intelligence model to obtain first state information and a first outline of the sample signal lamp, wherein the sample image carries the labeled state information and the labeled outline of the sample signal lamp, and i is a positive integer;

calculating a loss value based on the first state information, the first contour, the labeled state information, and the labeled contour;

and when the loss value or the iteration meets the iteration cutoff condition, outputting the artificial intelligence model, if the loss value or the iteration meets the iteration cutoff condition, adjusting the network parameters of the artificial intelligence model, and performing the (i + 1) th iteration based on the adjusted artificial intelligence model.

Iterative training is carried out on the artificial intelligence model through a large number of sample images, so that the signal lamp detection model obtained through final training has good universality and robustness.

In a third aspect, a signal lamp image processing apparatus is provided, the apparatus including:

the processing module is used for processing a first image containing a signal lamp based on a signal lamp detection model to obtain state information and a contour of the signal lamp, the signal lamp detection model is obtained based on artificial intelligence model training, the state information indicates the color of the signal lamp, and the contour indicates the display shape of the signal lamp;

and the enhancement module is used for carrying out color enhancement on the area where the signal lamp is located in the first image based on the state information and the outline to obtain a second image.

In some implementations, the color of the area in the second image where the signal lamp is located matches the color of the signal lamp in the real world; the display shape of the area of the signal lamp in the second image is matched with the display shape of the signal lamp in the real world.

In some implementations, the enhancement module is to: when the state information indicates that the color of the signal lamp is red, performing color enhancement on the area where the signal lamp is located in the first image based on the state information and the outline to obtain a second image; and when the state information indicates that the color of the signal lamp is not red, not performing color enhancement on the area where the signal lamp is located in the first image.

In some implementations, the enhancement module is to: and when the signal lamp basic outline matched with the outline exists in the signal lamp basic outline library, based on the state information and the outline, performing color enhancement on the area where the signal lamp is located in the first image to obtain the second image.

In some implementations, the apparatus further includes:

the correction module is used for correcting the contour when the signal lamp basic contour matched with the contour does not exist in the signal lamp basic contour library to obtain the corrected contour;

the enhancement module is used for enhancing the color of the area where the signal lamp is located in the first image based on the state information and the corrected outline to obtain the second image.

In some implementations, the correction module is to: determining the profile type of the signal lamp based on the profile; determining at least one signal lamp basic contour corresponding to the contour type from the signal lamp basic contour library; and correcting the contour based on the difference between the basic contour and the contour of the at least one signal lamp to obtain the corrected contour.

In some implementations, the apparatus further includes: and the storage module is used for storing the corrected outline serving as the basic outline of the signal lamp in the basic outline library of the signal lamp.

In some implementations, the storage module is to: when the corrected outline meets the target condition, the corrected outline is used as a signal lamp basic outline and stored in the signal lamp basic outline library; the target condition means that the modified contour is a continuous contour meeting a target rule, and the contour similarity between the modified contour and the basic contour of the target signal lamp is greater than or equal to a target threshold value.

In some implementations, the apparatus further includes:

the acquisition module is used for acquiring an original image which is shot by the shooting equipment and contains the signal lamp;

and the image cutting module is used for cutting the original image to obtain the first image, and the size of the area where the signal lamp is located in the first image accords with the target size.

In a fourth aspect, a training device for a signal lamp detection model is provided, the signal lamp detection model is obtained based on artificial intelligence model training, the device comprises a training module, and the training module is used for:

In a fifth aspect, there is provided a computing device comprising a processor and a memory for storing at least one piece of program code, the at least one piece of program code being loaded into and executed by the processor, to cause the computing device to perform the signal light image processing method provided in the first aspect or any one of the alternatives of the first aspect, or to perform the training method of the signal light detection model provided in the second aspect.

In a sixth aspect, a computer-readable storage medium is provided, which is used for storing at least one program code, which is loaded and executed by a processor, so as to make a computer execute the signal lamp image processing method provided in the first aspect or any one of the alternatives of the first aspect, or execute the training method of the signal lamp detection model provided in the second aspect.

In a seventh aspect, a computer program product or a computer program is provided, which comprises program code, which, when run on a computing device, causes the computing device to perform the signal light image processing method as provided in the first aspect or the various alternative implementations of the first aspect, or to perform the training method of the signal light detection model as provided in the second aspect.

Drawings

Fig. 1 is a schematic diagram of an implementation environment of a signal lamp image processing method according to an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of a computing device 200 according to an embodiment of the present invention;

fig. 3 is a flowchart of a signal lamp image processing method according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a signal lamp detection model according to an embodiment of the present invention;

FIG. 5 is a flowchart illustrating a signal detection model according to an embodiment of the present invention;

fig. 6 is a schematic diagram of a signal lamp image processing method according to an embodiment of the present invention;

fig. 7 is a schematic structural diagram of a signal lamp image processing apparatus according to an embodiment of the present invention;

fig. 8 is a schematic structural diagram of a training device for a signal lamp detection model according to an embodiment of the present invention.

Detailed Description

To make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Before describing the technical solutions provided by the embodiments of the present invention, the following description will be made on key terms related to the present invention.

Signal lamps are lamps which emit various signals by using changes in light, and are often used in traffic facilities, electronic equipment, and the like. In the embodiment of the present invention, the signal lamp is a traffic signal lamp, which is used to ensure the orderly progress of traffic, and generally comprises a red lamp, a green lamp and a yellow lamp, wherein the red lamp indicates no traffic, the green lamp indicates permission, and the yellow lamp indicates warning. In some implementations, types of signal lights include, but are not limited to: the signal light of the present invention is not limited to a signal light of a motor vehicle, a signal light of a non-motor vehicle, a signal light of a crosswalk, a direction indicator light (also referred to as an arrow signal light), a signal light of a lane, a flashing warning signal light, and a signal light of a crossing between a road and a railway plane.

The RAW image format (RAW image format) is RAW image data in which an image sensor converts a captured optical signal into an electronic signal, that is, grayscale data of an image.

A Red Green Blue (RGB) color model is a color standard in the industry, which obtains various colors by changing three color channels of red R, green G, and blue B and superimposing them, where RGB represents the colors of the three channels of red, green, and blue, and the standard almost includes all colors that can be perceived by human vision, and is one of the most widely used color systems at present. RAW images in various formats can be converted into RGB images by Image Signal Processing (ISP).

Binary Image (binary Image) refers to a digital Image with only two possible values per pixel. For example, the gray value of any pixel in the binary image is 0 or 255, which respectively represents black and white, and this is not limited in the embodiment of the present invention.

Machine Learning (ML) is a multi-domain cross discipline, which relates to multiple disciplines such as probability theory, statistics, approximation theory, graph analysis, algorithm complexity theory, etc., and studies how a computer simulates or realizes human learning behaviors to acquire new knowledge or skills, and reorganizes an existing knowledge structure to continuously improve the performance of the computer.

An Artificial Intelligence (AI) model is a mathematical algorithm model for solving practical problems by using a machine learning idea, and includes a large number of parameters and calculation formulas (or calculation rules), wherein the parameters in the AI model are values obtained by training an initial AI model through a training data set, and for example, the parameters of the AI model are weights of the calculation formulas or calculation factors in the AI model. The AI model also contains some hyper (hyper) parameters, the hyper parameters are parameters which can not be obtained by training the AI model through a training data set, the hyper parameters can be used for guiding the construction of the AI model or the training of the AI model, and the hyper parameters are various. For example, the number of iterations (iteration) of AI model training, learning rate (learning rate), batch size (batch size), number of layers of AI model, number of neurons per layer. In other words, the hyper-parameters of the AI model differ from the parameters in that: the values of the hyper-parameters of the AI model cannot be obtained by analyzing the training data in the training data set, and the values of the parameters of the AI model can be modified and determined by analyzing the training data in the training data set during the training process.

The neural network model is a mathematical algorithm AI model simulating the structure and function of biological neural network (animal central nervous system). A neural network model may include a plurality of different functional neural network layers, each layer including parameters and computational formulas. Different layers in the neural network model have different names according to different calculation formulas or different functions. For example, the layers that perform convolution calculations are called convolutional layers, which are often used to perform feature extraction on input data. One neural network model may also be composed of a combination of a plurality of existing neural network models. Neural network models of different structures may be used in different scenarios (e.g., detecting a change in the state of a signal, extracting the contour of a signal, etc.) or to provide different effects when used in the same scenario. The neural network model structure specifically includes one or more of the following: the neural network model has different network layers, different sequences of the network layers, and different weights, parameters or calculation formulas in each network layer.

The following briefly introduces an application scenario of the signal lamp image processing method provided by the present invention.

The signal lamp image processing method provided by the embodiment of the invention can be applied to scenes such as traffic violation penalty, automatic driving of vehicles and the like which need to process the signal lamp image so as to obtain an accurate detection result of the state change of the signal lamp. Schematically, the scenes to which the signal lamp image processing method provided by the embodiment of the present invention can be applied include, but are not limited to:

and a first scene is a traffic violation penalty.

At present, traffic video monitored control system can monitor vehicle pedestrian on the road all the weather, wherein, at the road junction of installing the signal lamp, traffic video monitored control system can be according to the image that includes the signal lamp that shoots, whether target such as pedestrian or vehicle appears in the state change of signal lamp in the detection image and the image, when signal lamp state change's testing result instructs the signal lamp to be red light and appear in the image when, combine the target position, judge whether there is illegal action such as rushing the red light at current road junction, if there is illegal action such as rushing the red light, then carry out the colour reinforcing to the signal lamp place region in the current image, obtain the illegal snapshot image that includes target and signal lamp, regard this illegal snapshot image as the foundation, support traffic violation penalty.

And a second scene is that the vehicle is automatically driven.

Along with the rapid development of artificial intelligence technology, vehicle automatic driving becomes one of the scenes that artificial intelligence falls to the ground and promotes in a large number at present, wherein, in order to realize vehicle automatic driving, need to dispose shooting function and signal lamp recognition function for the vehicle, when the vehicle traveles to the road crossing of installing the signal lamp, this signal lamp recognition function is used for including the image of signal lamp according to what the shooting obtained, detect the state change of signal lamp in the image, when the detection result of signal lamp state change instructs the signal lamp to be red light, control the vehicle and travel to stop traveling after the corresponding position, and carry out the colour enhancement to the region that signal lamp place in the current image, it archives to obtain the image that includes the signal lamp. Of course, when the detection result of the state change of the signal lamp indicates that the signal lamp is green, the color of the area where the signal lamp is located in the current image may be enhanced, and the corresponding image is archived, which is not limited by the present invention.

It should be noted that the above-mentioned scenes are only exemplary descriptions, and the signal lamp image processing method provided in the embodiment of the present invention can also be applied to other scenes that need to process a signal lamp image to obtain an accurate detection result of a signal lamp state change, which is not limited in the embodiment of the present invention.

The following describes an implementation environment of the signal lamp image processing method provided by the present invention.

Fig. 1 is a schematic diagram of an implementation environment of a signal lamp image processing method according to an embodiment of the present invention. As shown in fig. 1, the implementation environment includes: a camera device 101 and a computing device 102. The photographing apparatus 101 and the computing apparatus 102 are directly or indirectly connected through a wired network or a wireless network, which is not limited herein.

The shooting device 101 is configured to shoot an area where the signal lamp is located, obtain an image including the signal lamp, and send the image to the computing device 102. In some implementations, the image is a RAW image, i.e., the capture device 101 sends RAW image data acquired by the image sensor directly to the computing device 102. In some implementations, the image is an RGB image, that is, the capturing device 101 has an image processing function, and converts the RAW image into an RGB image and sends the RGB image to the computing device 102, which is not limited in this embodiment of the present invention. In some implementations, the shooting device 101 is a civil camera, an industrial camera, a video camera, or a smartphone with a shooting function, which is not limited in this embodiment of the present invention. The photographing apparatus 101 has a communication function and can access the internet, the photographing apparatus 101 may be generally referred to as one of a plurality of photographing apparatuses, and the embodiment of the present invention is illustrated only by the photographing apparatus 101.

The computing device 102 is configured to process the received image containing the signal light based on the image. The computing device 102 may be an independent physical server, a server cluster or a distributed file system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a network service, cloud communication, a middleware service, a domain name service, a security service, a Content Delivery Network (CDN), a big data and artificial intelligence platform, and the like. The number of computing devices 102 may be greater or fewer, and embodiments of the invention are not limited in this respect. In some implementations, a system with AI computing power is deployed on the computing device 102, and can implement the signal light image processing method provided by the method embodiments described below. In some implementations, the computing device 102 includes one or more chips that, when run on the computing device 102, cause the computing device 102 to perform the signal light image processing methods provided by the method embodiments described below. Embodiments of the invention are not limited with respect to the specific form of the computing device 102.

In some implementations, the wireless or wired networks described above use standard communication technologies and/or protocols. The network is typically the internet, but can be any network including, but not limited to, a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), a mobile, wired or wireless network, a private network, or any combination of virtual private networks. In some implementations, data exchanged over the network is represented using techniques and/or formats including hypertext markup language (HTML), extensible markup language (XML), and so forth. In addition, all or some of the links can be encrypted using conventional encryption techniques such as Secure Socket Layer (SSL), transport Layer Security (TLS), virtual Private Network (VPN), internet protocol security (IPsec), and the like. In other embodiments, custom and/or dedicated data communication techniques can also be used in place of or in addition to the data communication techniques described above.

The hardware architecture of the computing device 102 in the above-described implementation environment is described below.

The embodiment of the invention provides computing equipment. Referring to fig. 2, fig. 2 is a schematic diagram of a hardware structure of a computing device according to an embodiment of the present invention. As shown in fig. 2, the computing device 200 includes a memory 201, a processor 202, a communication interface 203, and a bus 204. The memory 201, the processor 202 and the communication interface 203 are connected to each other through a bus 204.

The memory 201 may be, but is not limited to, a read-only memory (ROM) or other type of static storage device that may store static information and instructions, a Random Access Memory (RAM) or other type of dynamic storage device that may store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disk read-only memory (CD-ROM) or other optical disk storage, optical disk storage (including compact disk, laser disk, optical disk, digital versatile disk, blu-ray disk, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. The memory 201 may store at least one piece of program code, and when the program code stored in the memory 201 is executed by the processor 202, the processor 202 and the communication interface 203 are used to execute the signal lamp image processing method shown in the embodiments described below. The memory 201 may further store a signal light base profile, an initial AI model, a signal light detection model, and the like, which are not limited in the embodiment of the present invention.

The processor 202 may be a Network Processor (NP), a Central Processing Unit (CPU), an application-specific integrated circuit (ASIC), or an integrated circuit for controlling the execution of programs according to the present invention. The processor 202 may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. The number of the processors 202 may be one or more. The communication interface 203 enables communication between the computing device 200 and other devices or communication networks using transceiver modules, such as transceivers. For example, the data may be acquired through the communication interface 203.

The memory 201 and the processor 202 may be separately provided or may be integrated together.

Bus 204 may include a pathway to transfer information between various components of computing device 200 (e.g., memory 201, processor 202, communication interface 203).

It should be noted that, in some implementations, the signal light image processing method provided by the present invention is implemented by a plurality of computing devices distributed and deployed in different environments, and the embodiments of the present invention do not limit this.

The following describes an exemplary signal lamp image processing method according to an embodiment of the present invention.

Fig. 3 is a flowchart of a signal light image processing method according to an embodiment of the present invention, and schematically illustrates an interaction between the photographing apparatus 101 and the computing apparatus 102 shown in fig. 1 as an example. The signal lamp image processing method comprises the following steps.

301. The shooting device sends the original image containing the signal lamp to the computing device.

In the embodiment of the present invention, the original image is an image including objects such as signal lights, roads, and vehicles/pedestrians. Optionally, the original image is a RAW image, or the original image is an RGB image, which is not limited in this embodiment of the present invention.

In some implementations, the capture device sends each captured frame of the original image to the computing device. For example, the shooting device is erected on a bayonet device installed at a road intersection and used for realizing real-time traffic monitoring, the shooting device can shoot targets such as a signal lamp area, a road, vehicles/pedestrians appearing at the road intersection and the like, and in this scene, the shooting device sends each frame of image obtained by shooting to the computing device, so that the computing device can detect the state change of the signal lamp in each frame of image and judge whether traffic violation behaviors exist at the current road intersection or not. By the method, omission is avoided, and the accuracy of the detection result of the state change of the signal lamp is improved. In other embodiments, the shooting device sends the original image including the signal lamp to the computing device at intervals of preset frame numbers, and since the image content is the same within the preset frame numbers and the detection result of the state change of the signal lamp obtained by the computing device is also the same, the data transmission can be reduced and the image processing efficiency can be improved on the premise of ensuring the accuracy of the final detection result by using the method.

In some implementations, the capture device has an intelligent detection function for detecting the captured original image and sending the original image including the signal lamp to the computing device. For example, the photographing device is installed in an intelligent driving vehicle for implementing an automatic driving function of the vehicle, and the photographing device can photograph road conditions in front of, at the side of, and at the back of the vehicle, in this scene, the photographing device intelligently detects an original image obtained by photographing, and when it is detected that the original image includes a signal lamp, the original image is sent to the computing device, so that the computing device detects a state change of the signal lamp and controls the vehicle to run.

302. The computing device processes the original image to obtain a first image containing the signal lamp.

In the embodiment of the invention, the computing equipment carries out image preprocessing on the original image to obtain a first image, and the size of the area where the signal lamp is located in the first image accords with the target size. In some implementations, the first image is an RGB image including the signal light. In some implementations, image pre-processing includes, but is not limited to: clipping, scaling, and edge filling. Illustratively, after acquiring an original image which is shot by the shooting device and contains the signal lamp, the computing device performs image cropping on the original image to obtain the first image. For example, in the case where the original image includes regions such as signal lights, signal light mounts, and roads, the computing device crops the original image to obtain a first image including the signal lights. By means of image cutting on the original image, the size of the area where the signal lamp is located in the first image is made to accord with the target size, the signal lamp detection model can conveniently extract features based on the signal lamps with the same size at each time, and stability of the model and accuracy of a model output result are guaranteed. In addition, the size of the first image conforms to the preset size of the input image of the signal lamp detection model, that is, for any one original image received by the computing device, the image size obtained after the computing device processes the original image conforms to the preset size, and the image size is convenient to be directly input into the signal lamp detection model.

In some implementation manners, a ratio of a size of a region where a signal lamp in the first image is located to a size of the first image is smaller than or equal to a preset threshold, for example, the preset threshold is 80%, that is, the size of the first image is larger than the size of the region where the signal lamp in the first image by a certain ratio, so that when a subsequent state change of the signal lamp is detected, information around the signal lamp, for example, halo information and the like, is fully utilized, and thus accuracy of a detection result is improved.

303. The computing device processes the first image based on a signal lamp detection model to obtain state information and a contour of the signal lamp, the signal lamp detection model is obtained based on artificial intelligence model training, the state information indicates the color of the signal lamp, and the contour indicates the display shape of the signal lamp.

In the embodiment of the invention, the computing equipment calls a signal lamp detection model, inputs the first image into the signal lamp detection model, and extracts semantic information of the first image through the signal lamp detection model so as to obtain the state information and the outline of the signal lamp. Wherein the status information indicates the color of the signal lamp, e.g. the signal lamp is red, green or yellow, etc. The outline indicates a display shape of the signal lamp, wherein the display shape of the signal lamp refers to a shape displayed when the signal lamp is lit. For example, the display shape of the signal lamp is a number, an arrow, a disc, a non-motor vehicle, a pedestrian, and the like, which is not limited in the embodiment of the present invention.

In some implementations, the signal light detection model is a neural network model. For example, the signal lamp detection model is a Convolutional Neural Network (CNN) model or a Recurrent Neural Network (RNN) model, and the specific type of the signal lamp detection model is not limited in the embodiments of the present invention.

Through the signal lamp detection model, accurate state information and outline of the signal lamp can be obtained, and especially, under the condition that some scenes are influenced by external conditions, even if the first image has the problems that the signal lamp is white and yellow or the display shape is deformed, the signal lamp detection model still can output accurate state information and outline of the signal lamp according to extracted semantic information of the first image, so that the accuracy of the detection result of the state change of the signal lamp is effectively improved. For example, if the signal lamp is red when the shooting device shoots the first image, but the first image has a white bias problem due to the influence of external conditions, that is, the center position of the area where the signal lamp is located in the first image is white bias, and red halo is displayed on the periphery of the area where the signal lamp is located, after semantic information of the first image is extracted through the signal lamp detection model, accurate state information and outline of the signal lamp can be obtained.

In the following, the process of obtaining the state information and the contour of the signal lamp by the computing device is described by taking the signal lamp detection model as the CNN model as an example. Referring to fig. 4 and 5, fig. 4 is a schematic diagram of a signal lamp detection model according to an embodiment of the present invention; fig. 5 is a processing flow chart of a signal lamp detection model according to an embodiment of the present invention.

As shown in fig. 4, the signal lamp detection model is based on a U-net network optimized model. The signal lamp detection model includes two parts, an encoder (encoder) part and a decoder (decoder) part. Wherein, the encoder part is composed of CNN characteristic extraction layers continuously downsampled and comprises 5 convolution blocks; the decoder part consists of a continuously up-sampled CNN feature extraction layer and also comprises 5 volume blocks. As shown in fig. 5, the computing device inputs the first image into the traffic light detection model, extracts semantic information of the first image through the encoder portion, and distinguishes colors of the traffic light according to the semantic information to obtain status information of the traffic light. Further, the semantic information is reconstructed into a signal lamp outline graph with the size consistent with that of the first image through a decoder part, and the outline of the signal lamp is obtained. In some implementation manners, the signal light contour map is a binary image, for example, values of pixel points in the signal light contour map are divided into 0 and 255, which represent white and black, respectively, where the black pixel point indicates a contour of the signal light, which is not limited in this embodiment of the present invention.

It should be noted that the signal lamp detection model shown in fig. 4 is an encoder-decoder structure, and feature fusion is realized in a splicing manner, so that the structure is simple and stable, and moreover, the signal lamp detection model is a model based on a binary task, so that two types of results can be output according to an input image, the load consumption of computing equipment is reduced, and the signal lamp image processing efficiency is effectively improved.

Next, the training process of the signal lamp detection model is introduced by taking the signal lamp detection model as the CNN model as an example. Illustratively, the training process is performed by a computing device, taking the ith iteration in the training process as an example (i is a positive integer), and the training process includes the following steps a through C.

Step A, inputting a sample image containing a sample signal lamp into a constructed AI model to obtain first state information and a first outline of the sample signal lamp.

The AI model may also be referred to as an initial AI model, among others. The computing equipment extracts semantic information of the sample image based on the network parameters of the AI model, and distinguishes colors of the sample signal lamp according to the semantic information to obtain first state information and a first contour of the sample signal lamp. The sample image carries the labeling state information and the labeling outline of the sample signal lamp.

The number of sample images in the training data set is usually multiple, and the AI model is trained through a large amount of training data, so that the finally trained signal lamp detection model has good universality and robustness. In addition, the embodiment of the present invention does not limit the network structure of the AI model.

And B, calculating a loss value based on the first state information, the first contour, the labeled state information and the labeled contour of the sample signal lamp.

Wherein the computing device employs a difference between the first state information and the annotation state information to construct a first loss function; and constructing a second loss function by adopting the difference value between the first contour and the labeled contour, taking the sum of the first loss function and the second loss function as a target loss function, and calculating to obtain a loss value corresponding to the sample signal lamp based on the target loss function. It should be noted that the manner of constructing the loss function by the computing device is not limited to the above manner, and the loss function in the embodiment of the present invention may be various loss functions commonly used in the training process of the neural network model, such as an absolute value loss function, a cosine similarity loss function, a square loss function, a cross entropy loss function, and the like, which is not limited in the embodiment of the present invention.

And C, if the loss value or the iteration meets the iteration cutoff condition, outputting the AI model, if the loss value or the iteration does not meet the iteration cutoff condition, adjusting the network parameters of the AI model, and performing the (i + 1) th iteration based on the adjusted AI model.

The iteration cutoff condition is that a loss value (also called an error value) is smaller than a set threshold, and the set threshold may be set according to actual requirements, for example, according to the accuracy of the signal lamp detection model. In other embodiments, the iteration stop condition is that the iteration number reaches the target number, or the training duration reaches the target duration, and the content of the iteration stop condition is not limited by the present invention. In the iteration, if the loss value or the iteration meets the iteration cutoff condition, the AI model of the iteration is shown to meet the requirements, and the computing equipment outputs the AI model, namely, the trained signal lamp detection model is obtained. If not, the computing equipment adjusts the network parameters of the current AI model, then performs the (i + 1) th iteration based on the adjusted AI model, namely, the step A is performed again, and the training is stopped until the iteration ending condition is met, so as to obtain the trained signal lamp detection model.

In addition, in some implementations, in an iteration process, a plurality of sample images are input into the AI model, a plurality of loss values (or a total loss value is obtained by processing the plurality of loss values) are obtained, and whether the iteration satisfies an iteration cutoff condition is determined based on the plurality of loss values (or the total loss value). In other implementation manners, in one iteration process, one sample image is input into the AI model to obtain one loss value, and whether the iteration meets the iteration cutoff condition is determined based on the loss value.

It should be noted that the training process of the signal light detection model may further include other steps or other optional implementations, which is not limited in the present invention. In addition, the signal lamp detection model according to the embodiment of the present invention is not limited to the above type, and any other network based on machine learning or deep learning and used for obtaining the state information and the contour of the signal lamp may be used as the signal lamp detection model according to the embodiment of the present invention. In some implementation manners, the signal lamp detection model may further include two submodels, which are respectively used to acquire the status information and the contour of the signal lamp, and this is not limited in the embodiment of the present invention.

304. The computing device determines whether a signal light base contour matching the contour exists in the signal light base contour library, and if so, the computing device performs the following step 305, and if not, the computing device performs the following steps 306 and 307.

In an embodiment of the present invention, the signal lamp base profile library includes a plurality of signal lamp base profiles. Wherein the basic outline of the signal lamp indicates the basic display shape of the signal lamp, and the basic display shape of the signal lamp similarly includes but is not limited to the following steps, similar to the step 303: numbers, arrows, discs, non-motor vehicles, pedestrians, etc., which are not limited in the embodiments of the present invention. Whether the signal lamp basic outline matched with the outline exists in the signal lamp basic outline library or not means whether a first signal lamp basic outline exists in the signal lamp basic outline library or not, and the outline similarity between the first signal lamp basic outline and the outline is larger than or equal to a first threshold value. For example, the first threshold is 0.5, which is not limited in the embodiment of the present invention.

In some implementations, the computing device implements this step 304 by steps 3041 through 3044 described below.

3041. The computing device determines a contour class to which the signal lamp belongs based on the contour of the signal lamp.

The profile categories include, but are not limited to, numbers, arrows, discs, non-motor vehicles, pedestrians, and the like.

3042. The computing device determines at least one signal light base profile corresponding to the profile class from the library of signal light base profiles.

And in the signal lamp basic outline library, each outline category corresponds to at least one signal lamp basic outline. For example, the signal lamp base profile library includes 5 profile categories, and each profile category corresponds to 10 signal lamp base profiles, which is not limited in the embodiment of the present invention.

3043. The computing device derives at least one contour similarity based on the contour and the at least one signal light base contour.

Wherein the computing device compares the contour with each of the at least one signal light base contour one-to-one to obtain at least one contour similarity.

In some implementations, the computing device determines, from the contour, a contour feature of the signal lamp (e.g., taking the signal lamp detection model as an optimized model based on a U-net network as an example, the computing device stores a contour feature map of the signal lamp while obtaining the contour of the signal lamp according to the signal lamp detection model), then the computing device determines a contour feature of at least one signal lamp base contour corresponding to the contour category, and obtains at least one contour similarity based on the contour feature of the signal lamp and the contour feature of the at least one signal lamp base contour. The contour similarity obtained by the characteristic comparison method is more accurate and more robust, and the accuracy of the detection result of the state change of the signal lamp can be further improved.

3044. If a first contour similarity exists in the at least one contour similarity and is greater than or equal to a first threshold, the computing device determines that the signal lamp basic contour corresponding to the first contour similarity is a first signal lamp basic contour, and executes the following step 305, and if the first contour similarity does not exist, the computing device executes the following steps 306 and 307.

Wherein the computing device compares the maximum value of the at least one contour similarity with a first threshold, if the maximum value is greater than or equal to the first threshold, the computing device performs the following step 305, and if the maximum value is less than the first threshold, the computing device performs the following steps 306 and 307. For example, the computing device determines that the contour class to which the signal lamp belongs is an arrow class according to the contour of the signal lamp, determines the signal lamp base contours of 10 arrow classes from a signal lamp base contour library to obtain 10 contour similarities, then the computing device compares the maximum value of the 10 contour similarities with a first threshold value, if the maximum value is greater than or equal to the first threshold value, the computing device performs the following step 305, and if the maximum value is less than the first threshold value, the computing device performs the following step 306 and step 307. Of course, in some implementations, the computing device traverses each of the at least one contour similarity until a contour similarity greater than or equal to the first threshold is obtained, or finds that any contour similarity is smaller than the first threshold after traversing, which is not limited by the embodiment of the present invention.

305. And the computing equipment performs color enhancement on the area where the signal lamp is located in the first image based on the state information and the outline to obtain a second image.

In this embodiment of the present invention, after the step 304, the computing device determines that the signal lamp basic contour matched with the contour exists in the signal lamp basic contour library, which indicates that the accuracy of the contour of the signal lamp obtained by the signal lamp detection model meets the requirement, and then the computing device determines the area where the signal lamp is located in the first image according to the display shape of the signal lamp indicated by the contour, and performs color enhancement on the area where the signal lamp is located based on the color of the signal lamp indicated by the status information to obtain the second image. The color of the area where the signal lamp is located in the second image is matched with the color of the signal lamp in the real world; the display shape of the area of the signal lamp in the second image is matched with the display shape of the signal lamp in the real world.

In some implementations, the performing, by the computing device, color enhancement on the area where the signal lamp is located in the first image means that, according to the display shape of the signal lamp indicated by the outline and the color of the signal lamp indicated by the state information, the RGB components of the pixels corresponding to the area where the signal lamp is located in the first image are corrected, so that the color of the signal lamp in the obtained second image matches the color of the signal lamp indicated by the state information. By the correction method, the color accuracy of the signal lamp in the second image is improved, and the color matching degree with the color in the real world is higher.

In other embodiments, the computing device performs color enhancement on the area where the signal lamp is located in the first image, that is, re-coloring the area where the signal lamp is located in the first image according to the display shape of the signal lamp indicated by the outline and the color of the signal lamp indicated by the status information, so that the color of the signal lamp in the obtained second image matches the color of the signal lamp indicated by the status information. Through the re-coloring mode, the data processing amount of the computing equipment can be reduced, and the signal lamp image processing efficiency is improved on the premise that the signal lamp color in the second image is matched with the color in the real world.

Through the steps 302 to 305, the computing device obtains the state information and the contour of the signal lamp through the signal lamp detection model according to the first image including the signal lamp, so as to obtain the second image, and detects the state change of the signal lamp based on the second image, so that the detection result with accurate color identification and high contour identification accuracy can be obtained. For example, the color of the signal lamp in the real world is red, the display shape is a straight arrow, but the shooting device is affected by external conditions when shooting the signal lamp, and the problem that the color of the signal lamp is white and the arrow is deformed occurs in the obtained first image, then through the above steps 302 to 305, the computing device accurately coats the region where the signal lamp is located in the first image with red to obtain a second image, the color of the signal lamp in the second image is red, and is matched with the color of the signal lamp in the real world, and the arrow display shape of the signal lamp in the second image is clear and recognizable, and is matched with the display shape of the signal lamp in the real world.

306. And the computing equipment corrects the contour to obtain a corrected contour.

In the embodiment of the present invention, after the step 304, the computing device determines that the signal lamp basic contour matched with the contour does not exist in the signal lamp basic contour library, which indicates that the accuracy of the contour of the signal lamp obtained through the signal lamp detection model does not meet the requirement, and then the computing device needs to correct the contour to obtain the corrected contour.

In some implementations, the computing device implements this step 306 by steps 3061 through 3063, described below.

3061. The computing device determines a contour class to which the signal lamp belongs based on the contour.

3062. The computing device determines at least one signal light base profile corresponding to the profile class from a library of signal light base profiles.

Step 3061 and step 3062 are similar to step 3041 and step 3042, and are not described herein.

3063. The computing device corrects the at least one signal lamp base contour based on a difference between the contour and the contour, resulting in a corrected contour.

The computing equipment compares each signal lamp basic contour with at least one signal lamp basic contour respectively to obtain the difference between the signal lamp basic contour and the contour, and then corrects the contour to obtain the corrected contour. For example, the differences between the signal light base profile and the profile include, but are not limited to: noise points, bumps, non-continuous edges, irregular false edges, etc. of the contour, and accordingly, the computing device corrects the contour in a manner including, but not limited to: deleting noise points, correcting bulges, connecting discontinuous edges, smoothing irregular false edges, and the like, which are not limited in the embodiment of the present invention. The contour output by the model is corrected through the signal lamp basic contour in the signal lamp basic contour library, so that the corrected contour can be ensured to meet the requirement of the signal lamp basic contour, and the accuracy of the corrected contour is improved.

In some implementations, the binary image of the at least one signal lamp base contour and the binary image of the contour are calculated to obtain a difference between the at least one signal lamp base contour and the contour, which is not limited by the embodiment of the present invention.

307. And the computing equipment performs color enhancement on the area where the signal lamp is located in the first image based on the state information and the corrected outline to obtain a second image.

In the embodiment of the present invention, after the step 306, the computing device obtains the corrected outline, the accuracy of the corrected outline is higher than that of the signal lamp obtained in the step 303, and the computing device determines the area where the signal lamp is located in the first image according to the display shape of the signal lamp indicated by the corrected outline, and performs color enhancement on the area where the signal lamp is located based on the color of the signal lamp indicated by the state information to obtain the second image. It should be noted that, the optional implementation manner of the computing device performing color enhancement on the area where the signal lamp is located in the first image is the same as that in step 305, and therefore, the description is omitted here.

In some implementations, the computing device stores the modified contour as a signal light base contour in a signal light base contour library. The corrected outline is stored, and the signal lamp basic outline library is updated in time, so that signal lamp basic outlines in various scenes such as different time periods, different weather conditions, different illumination and the like are contained in the signal lamp basic outline library, and the accurate outline of the signal lamp can be obtained by the computing equipment according to the image containing the signal lamp aiming at various complex scenes.

In some implementations, when the modified contour meets the target condition, storing the modified contour as a signal lamp base contour in a signal lamp base contour library; the target condition means that the corrected outline is a continuous outline which accords with a target rule, and the outline similarity between the corrected outline and the basic outline of the target signal lamp is greater than or equal to a target threshold value. The target rule is a preset rule, for example, the target rule means that the display shape indicated by the modified outline conforms to the signal lamp display shape of the traffic field. The target signal lamp base contour is any signal lamp base contour matched with the contour category to which the corrected contour belongs, or the target signal lamp base contour is a specified signal lamp base contour matched with the contour category to which the corrected contour belongs, which is not limited in the embodiment of the present invention. In some implementations, the computing device performs feature extraction on the modified contour to obtain a contour feature of the modified contour, then obtains a contour similarity between the modified contour and a basic contour of a target signal lamp based on the contour feature of the modified contour and the contour feature of the basic contour of the target signal lamp, compares the contour similarity with a target threshold, and if the contour similarity is greater than the target threshold, it indicates that the accuracy of the modified contour meets the requirement, and the modified contour can be stored as the basic contour of the signal lamp. The corrected outline meeting the target condition is stored, and on the basis of updating the signal lamp basic outline library in time, the accuracy of the signal lamp basic outline in the signal lamp basic outline library is further ensured, so that the computing equipment can obtain the accurate outline of the signal lamp according to the image containing the signal lamp aiming at various complex scenes.

Through the above steps 302 to 304, 306 and 307, the computing device obtains the state information and the contour of the signal lamp through the signal lamp detection model according to the first image including the signal lamp, corrects the contour if the accuracy of the contour does not meet the requirement, obtains the second image according to the corrected contour, detects the state change of the signal lamp based on the second image, and can obtain the detection result with accurate color recognition and high contour recognition accuracy. Furthermore, the computing equipment can store the corrected outline as a signal lamp basic outline, and the signal lamp basic outline library can be updated in a self-adaptive mode through the outline self-updating and self-correcting mode, so that the computing equipment can obtain the accurate outline of the signal lamp according to the image containing the signal lamp aiming at various complex scenes.

In some implementations, before performing step 304, the computing device determines whether the color of the signal light indicated by the status information of the signal light is red, and when the status information indicates that the color of the signal light is red, the computing device performs steps 304 and 305, or the computing device performs steps 304, 306, and 307, so as to perform color enhancement on the area where the signal light is located in the first image, thereby obtaining a second image; when the status information indicates that the color of the signal lamp is non-red, the computing device does not perform color enhancement on the area where the signal lamp is located in the first image. For example, taking a traffic violation penalty scene as an example, the computing device needs to determine whether a traffic violation occurs at a current road intersection according to an image including a signal lamp, in this scene, after obtaining state information and a contour of the signal lamp according to the first image, when the state information indicates that the color of the signal lamp is red, that is, the current road intersection is red, the computing device performs subsequent processing according to the contour. By the method, redundant signal lamp images can be avoided from being processed, so that the data processing amount of the computing equipment is reduced, the load consumption of the computing equipment is reduced, and the signal lamp image processing efficiency is improved.

In some implementations, after performing step 303, the computing device performs color enhancement on the area where the signal lamp is located in the first image based on the status information and the contour of the signal lamp directly, to obtain a second image. That is, the steps 304 to 307 are optional implementation steps of the embodiment of the present invention. The computing equipment can directly perform image processing according to the result output by the signal lamp detection model to obtain a second image so as to improve the signal lamp image processing efficiency; the computing device may further determine the state information and the contour output by the signal lamp detection model, and then perform image processing to obtain a second image, so as to ensure accuracy of the contour of the signal lamp, thereby improving accuracy of the detection result of the state change of the signal lamp.

In addition, in some implementations, the computing device performs color enhancement on the area where the signal lamp is located in the original image based on the state information and the contour to obtain a third image, where the third image is an image including the color-enhanced signal lamp, the road, and the vehicle/pedestrian. Because the color of the signal lamp in the third image is accurately identified, and the contour accuracy of the signal lamp is higher, the third image can be used as the basis of traffic violation penalty to support the traffic violation penalty under the condition that the violation behaviors such as red light running appear in the original image.

In summary, in the signal lamp image processing method provided in the embodiment of the present invention, the first image including the signal lamp is processed through the signal lamp detection model, so as to obtain the state information and the outline of the signal lamp with higher accuracy, so that the color of the signal lamp in the first image can be enhanced according to the color of the signal lamp indicated by the state information and the display shape of the signal lamp indicated by the outline, and the second image is obtained. Since the accuracy of the color of the traffic light and the display shape of the traffic light in the second image is high, the state change of the traffic light is detected based on the second image, and a highly accurate detection result can be obtained.

Next, referring to fig. 6, a flow of a traffic light image processing method according to an embodiment of the present invention is illustrated on the basis of the traffic light image processing method shown in fig. 3.

Fig. 6 is a schematic diagram of a signal lamp image processing method according to an embodiment of the present invention. As shown in fig. 6, the signal lamp image processing method is executed by a computing device. The method includes the steps that after a shooting device sends an original image containing a signal lamp to a computing device, the computing device conducts image preprocessing on the original image to obtain a first image, the first image is input into a signal lamp detection model, and state information and outline of the signal lamp are obtained through the signal lamp detection model.

For the contour of the signal lamp, the computing device determines whether a signal lamp basic contour matching the contour exists in the signal lamp basic contour library, which may also be understood as comparing the contour with the signal lamp basic contour. If the signal lamp is located in the first image, the computing equipment takes the contour as an accurate contour of the signal lamp, and color enhancement is performed on the region where the signal lamp is located in the first image based on the contour and the state information, and the region where the signal lamp is located in the first image can be also accurately colored to obtain a second image. If the contour does not exist, the computing device corrects the contour (for example, deleting noise points, correcting bulges, connecting discontinuous edges, smoothing irregular false edges and the like) to obtain a corrected contour, uses the corrected contour as an accurate contour of a signal lamp, and performs color enhancement on a region where the signal lamp is located in the first image based on the corrected contour and state information to obtain a second image. And when the corrected outline meets the target condition, storing the corrected outline as a signal lamp basic outline in a signal lamp basic outline library, and when the corrected outline does not meet the target condition, ending the current process. It should be noted that the content of this part is the same as that of steps 304 to 307 in the embodiment shown in fig. 3, and therefore, the description thereof is omitted here.

For the status information of the signal lamp, when the status information indicates that the color of the signal lamp is red, the computing device performs color enhancement on the area where the signal lamp is located in the first image based on the status information and the outline (or the corrected outline) to obtain a second image. And when the status information indicates that the color of the signal lamp is not red, ending the current process.

In summary, in the signal lamp image processing method provided in the embodiment of the present invention, the signal lamp detection model is used to process the first image including the signal lamp, so as to obtain the state information and the outline of the signal lamp with higher accuracy, and thus, the color enhancement of the area where the signal lamp is located in the first image can be performed according to the color of the signal lamp indicated by the state information and the display shape of the signal lamp indicated by the outline, so as to obtain the second image. Since the accuracy of the color of the traffic light and the display shape of the traffic light in the second image is high, the state change of the traffic light is detected based on the second image, and a highly accurate detection result can be obtained. Meanwhile, under the condition that the color of the signal lamp indicated by the state information is red, the first image is subjected to color enhancement, and redundant signal lamp images can be avoided from being processed, so that the data processing amount of the computing equipment is reduced, the load consumption of the computing equipment is reduced, and the signal lamp image processing efficiency is improved. Furthermore, the corrected outline is stored as the basic outline of the signal lamp, and the basic outline library of the signal lamp can be updated in a self-adaptive mode through the outline self-updating and self-correcting mode, so that the computing equipment can obtain the accurate outline of the signal lamp according to the image containing the signal lamp aiming at various complex scenes.

Fig. 7 is a schematic structural diagram of a signal lamp image processing apparatus according to an embodiment of the present invention. As shown in fig. 7, the traffic light image processing apparatus 700 is configured to execute steps executed by a computing device in the traffic light image processing method. Schematically, the signal lamp image processing apparatus 700 includes, but is not limited to: a processing module 701 and an enhancement module 702.

A processing module 701, configured to process a first image including a signal lamp based on a signal lamp detection model to obtain state information and a contour of the signal lamp, where the signal lamp detection model is obtained based on artificial intelligence model training, the state information indicates a color of the signal lamp, and the contour indicates a display shape of the signal lamp;

the enhancing module 702 is configured to perform color enhancement on the area where the signal lamp is located in the first image based on the status information and the contour, so as to obtain a second image.

In some implementations, the enhancement module 702 is to: when the state information indicates that the color of the signal lamp is red, performing color enhancement on the area where the signal lamp is located in the first image based on the state information and the outline to obtain a second image; and when the state information indicates that the color of the signal lamp is non-red, the color of the area where the signal lamp is located in the first image is not enhanced.

In some implementations, the enhancement module 702 is to: and when the signal lamp basic outline matched with the outline exists in the signal lamp basic outline library, based on the state information and the outline, performing color enhancement on the area where the signal lamp is located in the first image to obtain the second image.

In some implementations, the apparatus further includes:

the enhancing module 702 is configured to perform color enhancement on the area where the signal lamp is located in the first image based on the status information and the corrected outline, so as to obtain the second image.

In some implementations, the correction module is to: determining the outline type of the signal lamp based on the outline; determining at least one signal lamp basic contour corresponding to the contour type from the signal lamp basic contour library; and correcting the contour based on the difference between the basic contour and the contour of the at least one signal lamp to obtain the corrected contour.

In some implementations, the apparatus further includes:

It should be noted that: in the signal light image processing apparatus provided in the above embodiment, when processing the signal light image, only the division of the above functional modules is exemplified, and in practical applications, the above function allocation may be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the above described functions. In addition, the signal lamp image processing apparatus and the signal lamp image processing method provided in the above embodiments belong to the same concept, and specific implementation processes thereof are described in detail in the method embodiments, and are not described again here.

Fig. 8 is a schematic structural diagram of a training device for a signal lamp detection model according to an embodiment of the present invention. As shown in fig. 8, the signal light detection model training device 800 is used to perform the steps of training the signal light detection model performed by the computing device in the embodiment shown in fig. 3. Illustratively, the signal light detection model is trained based on an artificial intelligence model, and the training device 800 of the signal light detection model includes but is not limited to: a training module 801.

The training module 801 is configured to:

if the loss value or the iteration meets the iteration cutoff condition, outputting the artificial intelligence model, if the loss value or the iteration does not meet the iteration cutoff condition, adjusting the network parameters of the artificial intelligence model, and performing the (i + 1) th iteration based on the adjusted artificial intelligence model.

It should be noted that: in the training apparatus for a signal lamp detection model provided in the foregoing embodiment, when the artificial intelligence model is trained to obtain the signal lamp detection model, only the division of the above functional modules is used for illustration, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules, so as to complete all or part of the above described functions. In addition, the training device for the signal lamp detection model provided by the above embodiment belongs to the same concept as the above method embodiment, and the specific implementation process is detailed in the method embodiment and is not described herein again.

The terms "first," "second," and the like in the present invention are used for distinguishing identical items or similar items having substantially the same functions, and it should be understood that the terms "first," "second," and "n" have no logical or temporal dependency, and do not limit the number or execution order. It will be further understood that, although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first image may be referred to as a second image, and similarly, a second image may be referred to as a first image, without departing from the scope of the various described examples. Both the first image and the second image may be images, and in some cases, may be separate and distinct images.

The term "at least one" in the present invention means one or more, and the term "a plurality" in the present invention means two or more, for example, a plurality of images means two or more images.

The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present invention, and these modifications or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a program product. The program product includes one or more program instructions. When loaded and executed on a computing device, cause the flow or functionality in accordance with embodiments of the invention, in whole or in part.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and these modifications or substitutions do not depart from the spirit of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A signal lamp image processing method, characterized in that the method comprises:

2. The method of claim 1,

the color of the area where the signal lamp is located in the second image is matched with the color of the signal lamp in the real world;

and the display shape of the area where the signal lamp is located in the second image is matched with the display shape of the signal lamp in the real world.

3. The method according to claim 1 or 2, wherein the performing color enhancement on the area where the signal lamp is located in the first image based on the status information and the contour to obtain a second image comprises:

when the status information indicates that the color of the signal lamp is red, performing color enhancement on the area where the signal lamp is located in the first image based on the status information and the outline to obtain a second image;

and when the state information indicates that the color of the signal lamp is non-red, not performing color enhancement on the area where the signal lamp is located in the first image.

4. The method according to any one of claims 1 to 3, wherein the performing color enhancement on the area where the signal lamp is located in the first image based on the state information and the outline to obtain a second image comprises:

and when the signal lamp basic outline matched with the outline exists in the signal lamp basic outline library, based on the state information and the outline, performing color enhancement on the area where the signal lamp is located in the first image to obtain the second image.

5. The method of claim 4, further comprising:

when the signal lamp basic outline matched with the outline does not exist in the signal lamp basic outline library, correcting the outline to obtain the corrected outline;

and based on the state information and the corrected outline, performing color enhancement on the area where the signal lamp is located in the first image to obtain the second image.

6. The method of claim 5, wherein the modifying the contour to obtain a modified contour comprises:

determining the contour category to which the signal lamp belongs based on the contour;

determining at least one signal lamp basic contour corresponding to the contour type from the signal lamp basic contour library;

and correcting the contour based on the difference between the at least one signal lamp basic contour and the contour to obtain the corrected contour.

7. The method of claim 5 or 6, further comprising:

and taking the corrected outline as a signal lamp basic outline and storing the signal lamp basic outline in the signal lamp basic outline library.

8. The method according to claim 7, wherein said storing the modified contour as a signal light base contour in the signal light base contour library comprises:

when the corrected outline meets the target condition, the corrected outline is taken as a signal lamp basic outline and stored in the signal lamp basic outline library;

the target condition means that the corrected contour is a continuous contour meeting a target rule, and the contour similarity between the corrected contour and the basic contour of the target signal lamp is greater than or equal to a target threshold value.

9. The method according to any one of claims 1 to 8, further comprising:

and performing image cutting on the original image to obtain the first image, wherein the size of the area where the signal lamp is located in the first image accords with the target size.

10. A training method for a signal lamp detection model is characterized in that the signal lamp detection model is obtained based on artificial intelligence model training, and the method comprises the following steps:

in the ith iteration of training the artificial intelligence model, inputting a sample image containing a sample signal lamp into the artificial intelligence model to obtain first state information and a first contour of the sample signal lamp, wherein the sample image carries the labeled state information and the labeled contour of the sample signal lamp, and i is a positive integer;

and outputting the artificial intelligence model when the loss value or the iteration meets an iteration cutoff condition, if not, adjusting the network parameters of the artificial intelligence model, and performing the (i + 1) th iteration based on the adjusted artificial intelligence model.

11. A signal lamp image processing apparatus, characterized in that the apparatus comprises:

the system comprises a processing module, a signal lamp detection module and a display module, wherein the processing module is used for processing a first image containing a signal lamp based on a signal lamp detection model to obtain state information and a contour of the signal lamp, the signal lamp detection model is obtained based on artificial intelligence model training, the state information indicates the color of the signal lamp, and the contour indicates the display shape of the signal lamp;

and the enhancement module is used for enhancing the color of the area where the signal lamp is located in the first image based on the state information and the outline to obtain a second image.

12. The apparatus of claim 11,

and the display shape of the region where the signal lamp is located in the second image is matched with the display shape of the signal lamp in the real world.

13. The apparatus of claim 11 or 12, wherein the enhancement module is configured to:

when the state information indicates that the color of the signal lamp is red, performing color enhancement on the area where the signal lamp is located in the first image based on the state information and the outline to obtain a second image;

14. The apparatus of any one of claims 11 to 13, wherein the enhancement module is configured to:

15. The apparatus of claim 14, further comprising:

and the enhancement module is used for enhancing the color of the area where the signal lamp is located in the first image based on the state information and the corrected outline to obtain the second image.

16. The apparatus of claim 15, wherein the modification module is configured to:

determining at least one signal lamp basic contour corresponding to the contour category from the signal lamp basic contour library;

and correcting the contour based on the difference between the basic contour and the contour of the at least one signal lamp to obtain the corrected contour.

17. The apparatus of claim 15 or 16, further comprising:

and the storage module is used for storing the corrected outline serving as the basic outline of the signal lamp in the signal lamp basic outline library.

18. The apparatus of claim 17, wherein the storage module is configured to:

when the corrected outline meets the target condition, the corrected outline is used as a signal lamp basic outline and stored in the signal lamp basic outline library;

the target condition means that the modified contour is a continuous contour meeting a target rule, and the contour similarity between the modified contour and the basic contour of the target signal lamp is greater than or equal to a target threshold value.

19. The apparatus of any one of claims 11 to 18, further comprising:

the acquisition module is used for acquiring an original image which is shot by shooting equipment and contains the signal lamp;

20. The utility model provides a training device of signal lamp detection model, its characterized in that, signal lamp detection model obtains based on artificial intelligence model training, the device includes the training module, the training module is used for:

21. A computing device, characterized in that it comprises a processor and a memory for storing at least one piece of program code, which is loaded by the processor and which executes the signal light image processing method according to any one of claims 1 to 9, or the training method of the signal light detection model according to claim 10.

22. A computer-readable storage medium, characterized in that the computer-readable storage medium is configured to store at least one program code for executing the signal light image processing method according to any one of claims 1 to 9, or the training method of the signal light detection model according to claim 10.