CN111178200B - Method for identifying instrument panel indicator lamp and computing equipment - Google Patents

Method for identifying instrument panel indicator lamp and computing equipment Download PDF

Info

Publication number
CN111178200B
CN111178200B CN201911322879.0A CN201911322879A CN111178200B CN 111178200 B CN111178200 B CN 111178200B CN 201911322879 A CN201911322879 A CN 201911322879A CN 111178200 B CN111178200 B CN 111178200B
Authority
CN
China
Prior art keywords
image
instrument panel
indicator
indicator lamp
category
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201911322879.0A
Other languages
Chinese (zh)
Other versions
CN111178200A (en
Inventor
刘华
张培
战立涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hainan Chezhiyi Communication Information Technology Co ltd
Original Assignee
Hainan Chezhiyi Communication Information Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hainan Chezhiyi Communication Information Technology Co ltd filed Critical Hainan Chezhiyi Communication Information Technology Co ltd
Priority to CN201911322879.0A priority Critical patent/CN111178200B/en
Publication of CN111178200A publication Critical patent/CN111178200A/en
Application granted granted Critical
Publication of CN111178200B publication Critical patent/CN111178200B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/02—Recognising information on displays, dials, clocks

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a method for identifying an indicator lamp of an instrument panel, which is executed in a computing device and is suitable for generating an indicator lamp identification model, wherein the indicator lamp identification model is suitable for outputting position information and category information of the indicator lamp in an image of the instrument panel, and the method comprises the following steps: acquiring a plurality of instrument panel images, wherein each instrument panel image comprises at least one indicator lamp; marking the position and the category of each indicator light in the instrument panel image; and training the indicator lamp recognition model by taking the instrument panel image marked with the position and the category of the indicator lamp as a training image. The invention also discloses corresponding computing equipment. The method for identifying the indicator lamp of the instrument panel can accurately identify the position information and the category information of the indicator lamp in the image of the instrument panel, is convenient for a user to understand the meaning of the indicator lamp, and makes proper judgment and driving behavior for the indicator lamp.

Description

Method for identifying instrument panel indicator lamp and computing equipment
Technical Field
The invention relates to the technical field of image processing, in particular to a method for identifying a vehicle instrument panel indicator lamp and computing equipment.
Background
The dashboard is the primary interactive interface of the vehicle with the driver. The indicator lights on the instrument panel, such as a seat belt unbuckled indicator light, a door non-closed indicator light, a brake failure indicator light, and the like, provide important information of the vehicle for a driver. Along with the continuous perfection of vehicle functions, the quantity of pilot lamps is more and more increased, and the figure and the category of certain pilot lamps have the similarity for the driver is difficult to understand the meaning of pilot lamps, can't in time make the judgement to the pilot lamp.
The existing indicator lamp identification method comprises the steps of firstly, shooting an image of a vehicle instrument panel by adopting terminal equipment such as a mobile phone and the like, inquiring a corresponding vehicle type according to the image, and loading an indicator lamp library of the vehicle type; then shooting an image of the indicator lamp, comparing the shot image with the image of the indicator lamp in the indicator lamp library, and determining the type of the indicator lamp. The method needs to shoot two images, namely, one instrument panel and one indicator lamp, is complex in operation and low in real-time performance. In addition, the identification accuracy of the method depends on the quality of images shot by users, and the problems of blurring, tilting, overturning, overexposure/underexposure and the like possibly occur due to the fact that the sizes of the images shot by the users are different and the quality of the images shot by the users is difficult to ensure, so that the false identification of the indicator lamp is caused.
Accordingly, there is a need to provide a method that can efficiently and accurately identify a vehicle dashboard indicator.
Disclosure of Invention
Accordingly, the present invention provides a method and computing device for identifying instrument panel indicator lights in an attempt to solve or at least alleviate the above-identified problems.
According to a first aspect of the present invention, there is provided a method of identifying a dashboard indicator, executed in a computing device, adapted to generate an indicator identification model adapted to output location information and category information of the indicator in a dashboard image, the method comprising the steps of: acquiring a plurality of instrument panel images, wherein the instrument panel images comprise at least one indicator lamp; marking the position and the category of each indicator light in the instrument panel image; training an indicator lamp identification model by taking an instrument panel image marked with the position and the category of an indicator lamp as a training image, wherein the indicator lamp identification model comprises a multi-scale feature extraction module, a region generation module, a target detection module and a judgment module, and the multi-scale feature extraction module takes the instrument panel image as input and outputs a multi-scale feature map of the instrument panel image; the region generation module comprises a plurality of region generation units, wherein each region generation unit takes a scale feature map output by the multi-scale feature extraction module as input and outputs candidate positions of the indicator lamps in the instrument panel image; the target detection module comprises a plurality of cascaded target detection units, wherein the first target detection unit takes a plurality of candidate positions output by the region generation module as input, and the other target detection units take recommended positions output by the previous target detection units as input, so as to output recommended positions and recommended categories of the indicator lamps in the instrument panel image; and the judging module takes the recommended positions and recommended categories output by the target detection units as inputs, and outputs the position information and the category information of the indicator lamps in the instrument panel image.
Optionally, in the method for identifying dashboard indicator according to the present invention, before the step of labeling the position and the category of each indicator in the dashboard image, the method further includes the steps of: and performing de-duplication processing on the plurality of instrument panel images.
Optionally, in the method for identifying a dashboard indicator according to the present invention, the deduplication process includes the steps of: extracting feature vectors of the instrument panel images respectively; clustering the plurality of instrument panel images according to the feature vector; and performing de-duplication treatment on each class respectively.
Optionally, in the method for identifying an instrument panel indicator according to the present invention, the step of performing the de-duplication process on each class includes: and if the similarity of the two instrument panel images in one class is larger than a first threshold value, deleting any one of the two instrument panel images, wherein the similarity is calculated according to the feature vector.
Optionally, in the method for identifying dashboard indicator according to the present invention, the step of marking the positions of the indicator lamps in the dashboard image includes: marking the area range of each indicator lamp in the instrument panel image respectively; for each area range in which the indicator light is located: identifying connected domains in the region range; calculating the area of each connected domain respectively, and taking the connected domain with the area larger than a second threshold value as an effective connected domain; and taking the circumscribed rectangle of each effective communication domain as the position of the indicator lamp.
Optionally, in the method for identifying dashboard indicator according to the present invention, after the step of labeling the position and the category of each indicator in the dashboard image, the method further includes the steps of: and mirroring the instrument panel image, changing the color space, blurring and rotating, and recording the position and the category of each processed indicator lamp.
Optionally, in the method for identifying dashboard indicator according to the present invention, before the step of labeling the position and the category of each indicator in the dashboard image, the method further includes the steps of: dividing an instrument panel image into a plurality of recognition difficulty levels according to the definition of the image, wherein the recognition difficulty levels comprise simple, medium and difficult; the step of taking the instrument panel image marked with the position and the category of the indicator light as the training image comprises the following steps: and respectively taking out a plurality of instrument panel images from each recognition difficulty level as training images, wherein the number of the instrument panel images taken out from each recognition difficulty level accords with a preset proportion.
Optionally, in the method for identifying a dashboard indicator according to the present invention, in the preset ratio, the number of dashboard images with medium identification difficulty is greater than the number of dashboard images with simple identification difficulty, and the number of dashboard images with difficult identification difficulty is greater than the number of dashboard images with difficult identification difficulty.
Optionally, in the method for identifying a dashboard indicator according to the present invention, the multi-scale feature extraction module is a Feature Pyramid Network (FPN), the region generation module is a region candidate network (RPN), and the target detection unit is a region-based convolutional neural network (R-CNN).
Optionally, in the method for identifying a dashboard indicator according to the present invention, in the process of training the indicator identification model, the determining module uses the recommended position and the recommended category output by the last target detecting unit as the position information and the category information of the indicator in the dashboard image.
Optionally, in the method for identifying an instrument panel indicator according to the present invention, after the step of training the indicator identification model, the method further includes the following model testing step: taking an instrument panel image which marks the position and the category of the indicator lamp and does not participate in model training as a test image, and inputting a trained indicator lamp identification model so that the indicator lamp identification model outputs the position information and the category information of the indicator lamp in each test image; calculating an evaluation index value of the indicator lamp recognition model according to the position information and the category information output by the indicator lamp recognition model and the real position and the real category of the indicator lamp in each test image; and when the evaluation index value is larger than a preset threshold value, judging that the indicator lamp identification model is successfully generated.
Optionally, in the method for identifying a dashboard indicator according to the present invention, during testing of the trained indicator identification model, the determining module is adapted to determine the location information and the category information of the indicator in the dashboard image according to the recommended location and the recommended category output by each target detecting unit.
According to a second aspect of the present invention there is provided a computing device comprising: at least one processor; and a memory storing program instructions that, when read and executed by the processor, cause the computing device to perform the method of identifying a dashboard indicator as described above.
According to a third aspect of the present invention, there is provided a readable storage medium storing program instructions that, when read and executed by a computing device, cause the computing device to perform a method of identifying a dashboard indicator as described above.
The method for identifying the indicator lamp of the instrument panel is suitable for generating the identification model of the indicator lamp, and the model can accurately identify the position information and the category information of the indicator lamp in the image of the instrument panel, so that a user can conveniently understand the meaning of the indicator lamp, and can make proper judgment and driving behavior for the indicator lamp.
The indicator light recognition model comprises a multi-scale feature extraction module, a region generation module, a target detection module and a judgment module. The multi-scale feature extraction module can extract features of different scales of the instrument panel image, so that the model has good recognition effect on indicator lamps of different sizes. The target detection module comprises a plurality of cascaded target detection units and is used for outputting recommended positions and recommended categories of the indicator lights. Each target detection unit (except the first target detection unit) can correct the recommended position and the recommended category output by the last target detection unit, so that false detection or missing detection caused by the problem of the image quality of an instrument panel (such as smaller size and fuzzy details of an indicator lamp) is avoided, and the accuracy of the identification of the indicator lamp is improved.
In addition, in the instrument panel indicator lamp identification method, the training image and the test image are subjected to operations such as mirroring, color space changing, blurring, rotation and the like on the basis of the original image, so that the diversity of a data set is increased, the robustness of a model is improved, and the model can accurately identify indicator lamps in instrument panel images with different scenes and different qualities.
In addition, in the instrument panel indicator lamp identification method, the instrument panel image is divided into a plurality of identification difficulty levels according to the definition of the image, the training image and the test image are obtained from the identification difficulty levels according to the preset proportion, the diversity of a data set is increased, the generalization capability of a model is improved, and the model can accurately identify indicator lamps in instrument panel images with different scenes and different qualities.
The foregoing description is only an overview of the present invention, and is intended to be implemented in accordance with the teachings of the present invention in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present invention more readily apparent.
Drawings
To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings, which set forth the various ways in which the principles disclosed herein may be practiced, and all aspects and equivalents thereof are intended to fall within the scope of the claimed subject matter. The above, as well as additional objects, features, and advantages of the present disclosure will become more apparent from the following detailed description when read in conjunction with the accompanying drawings. Like reference numerals generally refer to like parts or elements throughout the present disclosure.
FIG. 1 illustrates a schematic diagram of an indicator light recognition system 100 according to one embodiment of the present invention;
FIG. 2 shows a schematic diagram of a model training apparatus 200 according to one embodiment of the invention;
FIG. 3 illustrates a flow chart of a method 300 of identifying a dashboard indicator in accordance with one embodiment of the present invention;
FIGS. 4A-4C are schematic diagrams illustrating a process of marking a position of a dashboard indicator light according to one embodiment of the invention;
FIG. 5 shows a schematic diagram of a data preprocessing process according to one implementation of the present invention;
FIG. 6 illustrates a block diagram of an indicator light recognition model according to one embodiment of the present invention;
FIG. 7 illustrates a block diagram of an indicator light recognition model according to another embodiment of the present invention;
fig. 8 shows a schematic diagram of a calculation manner of the overlap (IoU) according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Fig. 1 shows a schematic diagram of an indicator light recognition system 100 according to one embodiment of the invention. As shown in fig. 1, the indicator light recognition system 100 includes an application server 110, a terminal device 120, and a model training device 200. It should be noted that the indicator light recognition system 100 shown in fig. 1 is only exemplary, and although only one application server, one terminal device, and one model training device are shown therein, in a specific practical case, different numbers of application servers, terminal devices, and model training devices may be included in the indicator light recognition system, and the present invention does not limit the number of application servers, terminal devices, and model training devices included in the indicator light recognition system.
The terminal device 120 is a device located at the user side, and the user may interact with the terminal device 120 through input/output devices such as a touch screen and a touch pen in the terminal device 120, or a somatosensory device and a remote control device connected to the terminal device 120. The terminal device 120 may be, for example, a mobile phone, a tablet computer, an intelligent wearable device, an on-board intelligent device, an internet of things device, etc., but is not limited thereto.
The terminal device 120 typically has a plurality of applications (apps) installed therein, such as an information application, a photographing application, a shopping application, an instant messaging application, etc., to provide corresponding functions to a user.
In an embodiment of the present invention, in order to implement intelligent recognition of the dashboard indicator, the terminal device 120 is installed with an indicator recognition application. It should be noted that the indicator light recognition application may be implemented as a separate application installed in the terminal device 120, or may be implemented as a functional module installed in a certain application of the terminal device 120, for example, as a functional module in an automotive information application, which does not limit the existence form of the indicator light recognition application in the terminal device 120.
The application server 110 is a server corresponding to the indicator light identification application, and is configured to provide methods and data calls for the indicator light identification application client deployed in the terminal device 120.
Model training device 200 may be any computing device such as, but not limited to, a desktop computer, a notebook computer, a server, a workstation, and the like. The model training apparatus 200 is used to perform the method 300 for identifying instrument panel indicator lamps of the present invention to train and generate the indicator lamp identification model of the present invention. The generated indicator light recognition model is deployed to the application server 110, so that the application server 110 can provide the indicator light recognition service to the terminal device 120.
Specifically, after deploying the indicator light recognition model to the application server 110, the user may access the indicator light recognition application on the terminal device 120, take an image of the dashboard through a specific interface in the application, and upload the taken image to the application server 110. Alternatively, the user may take a dashboard image using the default photographing application of the terminal device 120, and the taken dashboard image will be stored in the gallery of the terminal device 120. Subsequently, the indicator light recognition application is started, and an instrument panel image is selected from the gallery of the terminal device 120 through a specific interface in the application and uploaded to the application server 110.
The application server 110 receives the dashboard image uploaded by the terminal device 120, adopts the trained indicator light recognition model to determine the position information and the category information of the indicator light in the dashboard image (for example, the position of the indicator light can be marked in the dashboard image in a rectangular frame or the like, and the meaning of the indicator light is marked near the rectangular frame), and returns the position information and the category information of the indicator light to the terminal device 120, so that a user can conveniently and clearly determine the meaning of the indicator light, and make appropriate judgment and driving behavior.
FIG. 2 shows a schematic diagram of a model training apparatus 200 according to one embodiment of the invention. It should be noted that the model training apparatus 200 shown in fig. 2 is only an example, and in practice, the model training apparatus for implementing the method for identifying an instrument panel indicator of the present invention may be any type of apparatus, and the hardware configuration may be the same as the model training apparatus 200 shown in fig. 2 or may be different from the model training apparatus 200 shown in fig. 2. In practice, the device for implementing the method for identifying the instrument panel indicator lamp of the present invention may add or delete hardware components of the model training device 200 shown in fig. 2, and the specific hardware configuration of the model training device is not limited by the present invention.
As shown in FIG. 2, in a basic configuration 202, model training apparatus 200 typically includes a system memory 206 and one or more processors 204. A memory bus 208 may be used for communication between the processor 204 and the system memory 206.
Depending on the desired configuration, the processor 204 may be any type of processing including, but not limited to: a microprocessor (μp), a microcontroller (μc), a digital information processor (DSP), or any combination thereof. Processor 204 may include one or more levels of cache, such as a first level cache 210 and a second level cache 212, a processor core 214, and registers 216. The example processor core 214 may include an Arithmetic Logic Unit (ALU), a Floating Point Unit (FPU), a digital signal processing core (DSP core), or any combination thereof. The example memory controller 218 may be used with the processor 204, or in some implementations, the memory controller 218 may be an internal part of the processor 204.
Depending on the desired configuration, system memory 206 may be any type of memory including, but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. Physical memory in a computing device is often referred to as volatile memory, RAM, and data in disk needs to be loaded into physical memory in order to be read by processor 204. The system memory 206 may include an operating system 220, one or more applications 222, and program data 224. In some implementations, the application 222 may be arranged to execute instructions on an operating system by the one or more processors 204 using the program data 224. The operating system 220 may be, for example, linux, windows or the like, which includes program instructions for handling basic system services and performing hardware-dependent tasks. The application 222 includes program instructions for implementing various user desired functions, and the application 222 may be, for example, a browser, instant messaging software, a software development tool (e.g., integrated development environment IDE, compiler, etc.), or the like, but is not limited thereto. When the application 222 is installed into the model training apparatus 200, a driver module may be added to the operating system 220.
When model training apparatus 200 starts up running, processor 204 reads from memory 206 and executes program instructions of operating system 220. Applications 222 run on top of operating system 220, utilizing interfaces provided by operating system 220 and underlying hardware, to implement various user-desired functions. When the user launches the application 222, the application 222 is loaded into the memory 206, and the processor 204 reads and executes the program instructions of the application 222 from the memory 206.
Model training device 200 may also include an interface bus 240 that facilitates communication from various interface devices (e.g., output devices 242, peripheral interfaces 244, and communication devices 246) to basic configuration 202 via bus/interface controller 230. The example output device 242 includes a graphics processing unit 248 and an audio processing unit 250. They may be configured to facilitate communication with various external devices, such as a display or speakers, via one or more a/V ports 252. The example peripheral interface 244 may include a serial interface controller 254 and a parallel interface controller 256, which may be configured to facilitate communication via one or more I/O ports 258 and external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device) or other peripherals (e.g., printer, scanner, etc.). The example communication device 246 may include a network controller 260 that may be arranged to facilitate communication with one or more other computing devices 262 over a network communication link via one or more communication ports 264.
The network communication link may be one example of a communication medium. Communication media may typically be embodied by computer readable instructions, data structures, program modules, and may include any information delivery media in a modulated data signal, such as a carrier wave or other transport mechanism. A "modulated data signal" may be a signal that has one or more of its data set or changed in such a manner as to encode information in the signal. By way of non-limiting example, communication media may include wired media such as a wired network or special purpose network, and wireless media such as acoustic, radio Frequency (RF), microwave, infrared (IR) or other wireless media. The term computer readable media as used herein may include both storage media and communication media.
In the model training apparatus 200 according to the present invention, the application 222 includes instructions for performing the method 300 of identifying a dashboard light of the present invention, which may instruct the processor 204 to perform the method 300 of identifying a dashboard light of the present invention to train to generate a dashboard light identification model. The generated indicator light recognition model is to be deployed to the application server 110, and the application server 110 provides the terminal device 120 with the indicator light smart recognition service based on the indicator light recognition model.
FIG. 3 illustrates a flow chart of a method 300 of identifying a dashboard indicator in accordance with one embodiment of the present invention. The method 300 is performed in a computing device (e.g., the model training device 200 described above) for generating a pilot lamp identification model adapted to take a dashboard image as input, and to output position information and category information for the pilot lamp in the dashboard image. As shown in fig. 3, the method 300 begins at step S310.
In step S310, a plurality of dashboard images are acquired, wherein the dashboard images include at least one indicator light.
It should be noted that, the dashboard image obtained in step S310 may be any dashboard image, and the source, size, definition, etc. of the dashboard image obtained in step S310 are not limited in the present invention. For example, the dashboard image may be a plurality of dashboard high definition images of different angles photographed by a professional camera, an image uploaded by a user, or a dashboard image crawled from the internet, etc., but is not limited thereto.
There may be duplicate images in the plurality of dashboard images acquired in step S310. The duplicate images refer to the two dashboard images being identical or very similar, with only subtle differences. In order to avoid wasting computational resources by labeling repeated images in the subsequent step S320 and to avoid reducing the model generalization capability due to training the indicator light recognition model with repeated images in step S330, according to one embodiment, after a plurality of dashboard images are acquired in step S310, step S312 is performed (step S312 is not shown in fig. 3), and the plurality of dashboard images are deduplicated.
According to one embodiment, in step S312, the dashboard image may be deduplicated as follows: first, feature vectors of respective dashboard images are extracted. The feature vector may be, for example, but not limited to, HOG (Histogram of Oriented Gradient, directional gradient histogram) feature, hash feature, or the like of the image. And then, clustering the plurality of instrument panel images according to the feature vectors to obtain a plurality of image classes. The result of the clustering is to coarsely put more similar images together. The clustering algorithm may employ, for example, but not limited to, k-means, DBScan, hierarchical clustering algorithm, and the like. Then, each class is subjected to a deduplication process separately. For example, if the similarity of two dashboard images in a class is greater than a first threshold, deleting any one of the two dashboard images, wherein the similarity of the dashboard images is calculated according to the feature vector thereof. It should be noted that, the first threshold may be set by a person skilled in the art, and the value of the first threshold is not limited in the present invention.
For example, for the n dashboard images obtained in step S310, HOG features of each image are extracted to obtain n feature vectors f 1 ，f 2 ，…，f n . Subsequently, the n images are clustered into m classes, denoted k, using the k-means algorithm 1 ，k 2 ，…，k m . Subsequently, for each class k i I=1, 2, …, m for image deduplication. At k 1 Class is taken as an example, assuming k 1 There are p images in the class, and the corresponding feature vector is f 1 ，f 2 ，…，f p . For k 1 And calculating the similarity of the feature vectors in the class in pairs, for example, calculating the Euclidean distance of each pair of feature vectors in the class, and if the distance is larger than a first threshold value, indicating that the corresponding two images are repeated images, and deleting any one of the two images. And by the pushing, carrying out de-duplication treatment on each class, and finally obtaining a plurality of de-duplicated instrument panel images.
It should be noted that, for the sake of brevity, only one embodiment of image deduplication is given here. Other methods may be used by those skilled in the art to de-duplicate the dashboard image, for example, by comparing the information entropy, color distribution, hash values, etc. of the two images to identify duplicate images, and then delete the duplicate images. The invention is not limited to a specific method for removing the weight of the instrument panel image, and any weight removing method is within the protection scope of the invention.
According to one embodiment, in order to train and test the indicator recognition model by using the effective information, after performing the de-duplication processing on the dashboard image in step S312, step S314 is further included (step S314 is not shown in fig. 3), and the de-duplicated dashboard image is subjected to difficulty classification.
According to one embodiment, the instrument panel image is divided into a plurality of recognition difficulty levels according to the image definition, the recognition difficulty levels comprise, for example, simple, medium, difficult and extremely difficult, and the image definition corresponding to each recognition difficulty level is as follows:
the method is simple: the outline of the indicator light is basically clear and complete, and the indicator light is slightly shielded. For example, more than 90% of the area is clearly visible. Is easy to be identified by naked eyes.
Medium: the indicator light is partially blurred and partially missing, but the outline is basically complete. Indicator lights or presence of shadows, for example, over 70% of the area is substantially clear. Can be identified by naked eyes.
Difficulty in: most of the indicator lamps are missing and the outline is incomplete. But is visually identifiable, but requires experience.
It is extremely difficult to: the indicator light is severely missing or overly vague. Is difficult to identify by the naked eye.
It should be noted that, in some dashboard images, there may be multiple indicator lights. If a plurality of indicator lamps exist in the instrument panel image, the least clear indicator lamp is used for determining the recognition difficulty level of the image. Because the images with extremely difficult recognition difficulty level have no practical significance for training and testing the recognition model of the indicator lamp, the images are regarded as invalid data to be discarded, and the subsequent model training and testing process is not participated.
In addition, it should be noted that the above only gives an example of the recognition difficulty rating, and the recognition difficulty rating scheme of the present invention is not limited to the above manner. The number of the recognition difficulty grades and the division standard can be set by a person skilled in the art according to actual needs, and the invention does not limit the specific division condition of the recognition difficulty grades.
Subsequently, in step S320, the position and the category to which each indicator lamp in the dashboard image belongs are noted.
The position of the indicator light can be marked by a rectangular frame, and the indicator light is arranged in the rectangular frame. Rectangular boxes contain, for example, 4 coordinate information x 0 、y 0 W, h, where x 0 、y 0 Representing the left side of a rectangular frameThe upper corners have their horizontal and vertical coordinates, w and h representing the width and height of the rectangular frame. The category of the indicator lamp is, for example, a seat belt unbuckled indicator lamp, a door open indicator lamp, a brake failure indicator lamp, or the like, but is not limited thereto.
According to one embodiment, the position of each indicator light in the dashboard image may be determined according to the following steps S322, S324:
in step S322, the area range in which each indicator lamp is located is marked in the dashboard image.
The area where the indicator light is located can be marked manually. For example, for the indicator light shown in fig. 4A, the area range is manually marked as a rectangular box 510.
The range of the indicator light region marked in step S322 may be larger, so according to one embodiment, the following step S324 needs to be performed to adjust the range of the indicator light marked in step S322, i.e. reduce the range of the indicator light region, so as to obtain the accurate position of the indicator light.
In step S324, for each of the area ranges in which the indicator lamps are located: connected domains in the region range are identified. According to one embodiment, before the connected domain is identified, the dashboard image is converted into a gray scale image, then the gray scale image is Gaussian blurred, and then the blurred image is binarized.
Subsequently, connected regions in the binary image are identified, and the connected regions can be four connected regions or eight connected regions. The areas of the connected domains (namely the number of pixels included in each connected domain) are calculated respectively, the connected domains with the areas larger than the second threshold value are used as effective connected domains, the connected domains are screened through the areas, noise areas in images can be removed, and the accuracy of the marked positions of the indicator lamps is improved. It should be noted that, the second threshold may be set by a person skilled in the art, and the value of the second threshold is not limited in the present invention.
Then, the circumscribed rectangle of each effective communication area is taken as the position of the indicator lamp. For example, edge detection may be performed on each effective connected domain to obtain edge pixels of each effective connected domain, and then a minimum value min (x) of an abscissa, a maximum value max (x) of an abscissa, a minimum value min (y) of an ordinate, and a maximum value max (y) of an ordinate in all edge pixels are determined, and then a rectangle with an upper left corner coordinate (min (x), min (y)) and a width (max (x) -min (x) and a height (max (y) -min (y) of each effective connected domain is determined.
For example, as shown in fig. 4B, the area of the manually marked indicator light is a rectangular box 510. In step S324, connected regions in the rectangular frame 510 are identified, and 5 connected regions, that is, connected region 512, connected region 514, connected region 516, connected region 518, and connected region 520 are identified. Subsequently, the area of each connected domain (i.e., the number of pixels included in each connected domain) is calculated as S 512 、S 514 、S 516 、S 518 、S 520 . Wherein S is 514 、S 520 Less than the second threshold, the effective connected domains are connected domains 512, 516, 518. Finally, edge detection is performed on the connected domains 512, 516, 518, respectively, to obtain an edge pixel set thereof.
Determining the minimum value of the abscissa of the edge pixels as x according to the edge pixel set 2 The corresponding edge pixel is (x 2 ，y 2 ) The method comprises the steps of carrying out a first treatment on the surface of the Maximum value of abscissa x 4 The corresponding edge pixel is (x 4 ，y 4 ) The method comprises the steps of carrying out a first treatment on the surface of the The minimum value of the ordinate is y 1 The corresponding edge pixel is (x 1 ，y 1 ) The method comprises the steps of carrying out a first treatment on the surface of the Maximum value of ordinate y 3 The corresponding edge pixel is (x 3 ，y 3 ) As shown in fig. 4C. Therefore, the bounding rectangle of the connected domains 512, 516, 518 is the upper left corner coordinate (x 2 ，y 1 ) Width x 4 -x 2 Height is y 3 -y 1 I.e., rectangular box 530 shown in dashed lines in fig. 4C. Rectangular frame 530 is the exact position of the indicator light.
The category to which the dashboard image belongs can be noted manually. For example, the types of the marked dashboard images are a seatbelt unbuckled indicator, a door open indicator, a brake failure indicator, and the like.
In the actual application scenario, the shooting angle of the user when shooting the dashboard image has uncertainty, so that the conditions of inclination, overturn, blurring, overexposure and the like of the indicator light are easy to occur, but the dashboard image acquired in the step S310 is not necessarily uniformly distributed in various conditions. In order to ensure that the generated indicator light recognition model can recognize indicator lights under various conditions, a part of image data needs to be artificially created to balance various conditions in an actual application scene.
According to one embodiment, after labeling the positions and the categories of the indicator lights in the dashboard image in step S320, step S326 is performed (step S326 is not shown in fig. 3), at least one of mirroring, changing the color space, blurring, and rotating the dashboard image, and recording the positions and the categories of the respective indicator lights after the processing. Step S326 increases the diversity of the data set, thereby improving the robustness of the subsequently generated indicator lamp recognition model, and enabling the model to accurately recognize indicator lamps in instrument panel images with different scenes and different qualities.
It should be noted that, the foregoing steps S312, S314, S320, S322, S324, and S326 are all data preprocessing steps, and the purpose of the foregoing steps is to preprocess the dashboard image obtained in step S310, so as to obtain a training image set for training the indicator recognition model and a test image set for testing the recognition effect of the trained indicator recognition model. Fig. 5 shows a schematic diagram of such a data preprocessing process.
As shown in fig. 5, for the obtained original instrument panel image, image deduplication is performed first, then recognition difficulty division is performed, the instrument panel image is divided into four recognition difficulty levels of simple, medium, difficult and extremely difficult, and all images in the extremely difficult level are discarded, so that the images do not participate in the training and testing process of the subsequent indicator lamp recognition model. And then, labeling the area range and the category of the indicator lamp in the left instrument panel image with the recognition difficulty level of simple, medium and difficult, and correcting the labeled area range to obtain the accurate position of the indicator lamp. And then, carrying out image mirroring, color space changing, blurring, rotation and other enhancement processing on the marked image, and recording the positions and the belonging categories of the indicator lights after the enhancement processing. Finally, the processed image is taken as a usable dataset, which is further divided into a training set and a test set, for example, 70% of the dataset may be taken as the training set and 30% of the dataset may be taken as the test set. The images in the training set are training images and are used for training the identification model of the indicator lamp; the image in the test set is a test image used for testing the identification effect of the generated indicator lamp identification model.
After the location and the category of the indicator light in the dashboard image are noted, an available data set is generated, and step S330 is performed.
In step S330, the instrument panel image to which the position and the category of the indicator lamp are labeled is used as a training image to train the indicator lamp recognition model.
According to one embodiment, a plurality of instrument panel images are respectively taken out of the recognition difficulty levels as training images, wherein the number of the instrument panel images taken out of the recognition difficulty levels accords with a preset proportion. According to one embodiment, in the preset ratio, the number of instrument panel images with medium recognition difficulty is greater than the number of instrument panel images with simple recognition difficulty is greater than the number of instrument panel images with difficult recognition difficulty. For example, the number ratio of the three images identifying the difficulty level may be simple: medium: difficulty = 2:7:1. it will be appreciated by those skilled in the art that the above only gives one example of the image number ratios of three recognition difficulty levels, and in other embodiments, the image number ratios of different recognition difficulty levels may be set to other values, which is not limited by the present invention.
Fig. 6 shows a block diagram of an indicator light recognition model according to an embodiment of the present invention. As shown in fig. 6, the indicator light recognition model of the present invention includes a multi-scale feature extraction module 610, a region generation module 620, a target detection module 630, and a determination module 640.
The multi-scale feature extraction module 610 takes the dashboard image as input and outputs a multi-scale feature map of the dashboard image.
For example, as shown in fig. 6, the multi-scale feature extraction module 610 takes a dashboard image as an input, and outputs feature maps of four different scales from the first scale to the fourth scale.
The multi-scale feature extraction module 610 may extract features of different scales of the input image, i.e., convert a single-size input image into feature images of multiple sizes. The module 610 enables the indicator light recognition model to simultaneously contain features of targets of different sizes, so as to achieve the purpose of detecting objects of different sizes, and thus, the indicator light recognition model has good recognition effects on indicator lights of different sizes.
The region generation module 620 includes a plurality of region generation units, each of which takes a scale feature map output by the multi-scale feature extraction module as input, and outputs candidate positions of the indicator lights in the dashboard image. There are typically a plurality of candidate locations for output.
As shown in fig. 6, the region generation module 620 includes four region generation units 622, 624, 626, 628. The region generating unit 622 takes the first scale feature map output by the multi-scale feature extraction module 610 as an input, and outputs a candidate position (proposal) of the indicator light in the dashboard image by processing the first scale feature map, where the candidate position is represented by a rectangular frame. Similarly, the region generating units 624, 626, 628 respectively take the second to fourth scale feature maps output by the multi-scale feature extraction module 610 as input and process the second to fourth scale feature maps, and output a plurality of candidate positions of the indicator lamp in the dashboard image.
The target detection module 630 includes a plurality of cascaded target detection units, where a first target detection unit takes as input a plurality of candidate positions output by the area generation module, and other target detection units take as input a recommended position output by a previous target detection unit, and output a recommended position and a recommended category of the indicator light in the dashboard image.
The recommended positions are represented by rectangular boxes. The recommended class is determined by a probability vector output by the target detection unit (the probability vector is typically determined using a softmax processing layer). The probability vector is a one-dimensional vector, the number of elements in the vector is the same as the total number of the categories of the indicator lights, the value of each element is the probability that the recommended position belongs to the corresponding category, and the category with the highest probability is the recommended category. For example, the indicator lamps have 156 types in total, the recommended position is box, the probability vector corresponding to the recommended position is p, the probability vector p is a one-dimensional vector containing 156 elements, the value of the ith element in the vector is the probability that the indicator lamp in the box at the recommended position belongs to the ith class, and if the value of the 50 th element in the vector is the largest (namely, the probability that the recommended position belongs to the 50 th class is the largest), the 50 th class is the recommended class.
As shown in fig. 6, the object detection module 630 includes three cascaded object detection units 632, 634, 636. The target detection unit 632 receives as input a plurality of candidate positions output from the region generation module 620, that is, a plurality of candidate positions output from the respective region generation units 622 to 628, corrects the candidate positions, and outputs the recommended positions and the recommended categories of the indicator lamps in the dashboard image. The target detection unit 634 takes as input the recommended position output by the last target detection unit connected thereto, that is, the target detection unit 632, corrects the recommended position output by the target detection unit 632, and outputs the corrected recommended position and recommended category. The target detection unit 636 takes as input the recommended position output by the last target detection unit connected thereto, that is, the target detection unit 634, corrects the recommended position output by the target detection unit 634, and outputs the corrected recommended position and recommended category.
In the embodiment of the invention, each target detection unit (except the first target detection unit) can correct the recommended position and the recommended category output by the last target detection unit, so that false detection or missing detection caused by the problem of the image quality of an instrument panel (such as smaller size and fuzzy detail of the indicator lamp) is avoided, and the identification accuracy of the indicator lamp is improved.
The determination module 640 takes the recommended positions and recommended categories output by the plurality of target detection units as inputs, and outputs position information and category information of the indicator lamps in the dashboard image.
As shown in fig. 6, the determination module 640 takes as input the recommended position and recommended category output by the target detection units 632, 634, 636, and outputs the position information and category information of the indicator lamp in the dashboard image. The position information and the category information output by the determining module 640 are the identification result of the indicator light in the dashboard image.
According to one embodiment, during the model training phase, the determining module 640 takes as input the output of the last target detection unit, i.e., the target detection unit 636, and takes as the location information and the category information of the indicator light in the dashboard image the recommended location and the recommended category output by the last target detection unit.
Then, a loss function value (loss) of the model is calculated according to the position information and the category information output by the determination module 640 and the actual position and the category to which the indicator lamp labeled in step S320 belongs, and parameters of the indicator lamp identification model are updated according to the loss function value, and the parameters of the model include weights, offsets, and the like of the convolution kernels adopted by the respective processing modules. After updating the parameters, the training image is adopted to carry out the next iteration, and the model parameters are updated again until the loss function value is converged within a preset threshold value. At this time, the pilot lamp recognition model training is completed.
According to one embodiment, the method 300 further comprises a model test step S340 (step S340 is not shown in fig. 3). Step S340 is performed after step S330, and the trained indicator light recognition model is tested to detect the recognition effect of the model.
In step S340, taking the dashboard image marked with the position and the category of the indicator light and not participating in the model training as a test image, and inputting a trained indicator light identification model so that the indicator light identification model outputs the position information and the category information of the indicator light in each test image; calculating an evaluation index value (the evaluation index value may be, for example, but not limited to, accuracy, recall, etc.) of the indicator recognition model based on the position information and the category information output by the indicator recognition model, and the actual position and the actual category of the indicator in each test image; and when the evaluation index value is larger than a preset threshold value, judging that the indicator lamp identification model is successfully generated.
The model test procedure of step S340 is performed using the test image. The test image is an image of the instrument panel that has been labeled with the location and category to which it belongs in step S320, and that has not participated in the model training process. According to one embodiment, the method can be used for simplifying the recognition difficulty level from the position of the marked indicator lamp and the instrument panel image of the category to which the marked indicator lamp belongs: medium: difficulty = 2:7:1, and extracting a plurality of instrument panel images. Then, 70% of the extracted images were used as training images, and 30% were used as test images.
In the model test stage, the determining module 640 takes the recommended positions and recommended categories output by all the target detection units 632, 634 and 636 as inputs, and performs comprehensive analysis and determination on the recommended positions and recommended categories output by the target detection units to determine and output the position information and the category information of the indicator lamp in the test image.
Subsequently, an evaluation index value of the indicator recognition model is calculated based on the position information and the category information output from the determination module 640, and the actual position and the actual category of the indicator in each test image. The evaluation index value may be, for example, precision (precision), recall (recall), or the like, but is not limited thereto. And when the evaluation index value is larger than a preset threshold value, judging that the indicator lamp identification model is successfully generated.
Fig. 7 shows a specific structural diagram of an indicator light recognition model according to an embodiment of the present invention. As shown in fig. 7, the multi-scale feature module 610 is implemented as a feature pyramid network (Feature Pyramid Networks, FPN). As shown in FIG. 7, the FPN includes two parts, bottom-up (bottom-up) and top-down (top-down). The bottom-up part is located at the left side of the FPN, and processes the input image by using a trained depth network (e.g., a res net network, a VGG network, etc.), so as to sequentially generate feature maps C1 to C5 with gradually decreasing sizes. The top-down part is located on the right side of the FPN, which generates enhanced feature maps F2-F5 of different scales from the feature maps C2-C5 of different scales extracted by the depth network on the left side. Wherein the F5 layer is obtained by convoluting the C5 layer with 1*1, the F4 layer is obtained by overlapping 1*1 convoluting result of the C4 layer with upsampling result of the F5 layer, the F3 layer is obtained by overlapping 1*1 convoluting result of the C3 layer with upsampling result of the F4 layer, and the F2 layer is obtained by overlapping 1*1 convoluting result of the C2 layer with upsampling result of the F3 layer.
The region generation module 620 is implemented as a plurality of region candidate networks (Region Proposal Network, RPN). Characteristic diagrams F2 to F5 with different scales output by the FPN are input into each RPN after convolution of a convolution kernel 3*3 respectively. Each RPN processes the convolved feature map and outputs a candidate position (proposal) of the indicator light in the dashboard image. The candidate locations for each RPN output are then aggregated.
The object detection module 630 includes three cascaded object detection units, each of which is implemented as a Region-based convolutional neural network (Region-Convolutional Neural Networks, R-CNN), i.e., the object detection module 630 includes three cascaded R-CNNs. The overlap (Intersection over Union, ioU) thresholds for the three R-CNNs are sequentially incremented.
The candidate position output by each RPN is denoted as B0, and the actual positions of B0 and the marked indicator lamp are calculated IoU. The ratio of the area of the overlapping area of the two rectangular frames IoU to the total area covered by the two rectangular frames A and IoU of the rectangular frame B is the ratio of the area A n B (the hatched area in FIG. 8) of the overlapping portion of the two rectangular frames A and B to the total area A n B covered by the two rectangular frames A and B as shown in FIG. 8. The IoU threshold is set to 0.5, i.e., B0 with IoU less than 0.5 is discarded. Subsequently, the remaining B0 is subjected to Pooling treatment by an ROI Pooling layer (ROI Pooling), and then is input into a first R-CNN. The first R-CNN corrects B0 (i.e. adjusts the position of B0), and outputs a corrected recommended position B1 and a recommended category C1.
The IoU threshold of the second R-CNN is set to 0.6, i.e. the recommended position B1 of the first R-CNN output and the actual position of the marked indicator light are calculated IoU, and the recommended position B1 of which IoU is smaller than 0.6 is discarded. And after the rest B1 is subjected to pooling treatment of the ROI pooling layer, inputting a second R-CNN. The second R-CNN corrects B1 (i.e. adjusts the position of B1), and outputs the corrected recommended position B2 and recommended category C2.
The IoU threshold for the third R-CNN is set to 0.7, i.e. the recommended position B2 of the first R-CNN output and the actual position of the marked indicator light are calculated IoU, and the recommended position B2 of which IoU is smaller than 0.7 is discarded. And after the rest B2 is subjected to pooling treatment of the ROI pooling layer, inputting a third R-CNN. The third R-CNN corrects B2 (i.e. adjusts the position of B2), and outputs the corrected recommended position B3 and recommended category C3.
The determination module 640 takes the recommended position and the recommended category output by the R-CNN as input, and outputs the position information and the category information of the dashboard indicator light.
In the model training stage, the judging module receives the recommended position B3 and the recommended category C3 output by the last R-CNN module, namely the third R-CNN module, and outputs the recommended position B3 and the recommended category C3 as the position information and the category information of the final indicator lamp.
Subsequently, the model parameters are updated according to the position information B3 and the category information C3 output from the determination module 640. Specifically, the loss function value loss_3 of the third R-CNN is calculated according to the position information B3, the category information C3, the actual position and the category of the marked indicator lamp, and the parameter of the third R-CNN is updated according to loss_3. And calculating a loss function value loss_2 of the second R-CNN according to the position information B2, the category information C2 and the actual position and the category of the marked indicator lamp, and updating the parameter of the second R-CNN according to the loss_2. According to the position information B1, the category information C1, the actual position and the category of the marked indicator lamp, a loss function value loss_1 of the first R-CNN is calculated, and the parameter of the first R-CNN is updated according to loss_1. According to the recommended position B0, the recommended category C0, the actual position and the category of the marked indicator lamp, calculating a loss function value loss_0 of the RPN, and updating parameters of the RPN module according to the loss_0, wherein the weight parameters of the four RPNs are shared. And then, continuing the model training process by adopting the updated parameters, and carrying out the next iteration. And when the loss function value is converged to be within a preset threshold, finishing training of the identification model of the indicator lamp.
And then, testing the trained indicator lamp recognition model to detect the recognition effect of the model.
In the model test stage, the computation processes of the multi-scale feature extraction module 610, the region generation module 620, and the target detection module 630 are the same as those of the model training stage, and will not be described here again.
In the model test phase, the decision module 640 determines the location information and the category information of the finally output indicator lamp according to the recommended locations B1, B3 output by the three R-CNNs.
Because the overlapping degree threshold values of the three R-CNN modules are sequentially increased, the number of recommended positions output by the three R-CNN modules is sequentially decreased, that is, the number of B1 is greater than the number of B2 and greater than the number of B3, and B2 is a correction to some recommended positions in B1 and B3 is a correction to some recommended positions in B2. For example, the first R-CNN, the second R-CNN, and the third R-CNN output 20, 15, and 10 recommended positions, respectively, and the 15 recommended positions output by the second R-CNN are corrections to the first 15 recommended positions output by the first R-CNN, and the 10 recommended positions output by the third R-CNN are corrections to the first 10 recommended positions output by the second R-CNN.
One embodiment of the determination module 640 determining the location information and the category information of the finally outputted indicator lamps according to the recommended locations B1, B2, B3 outputted by the three R-CNNs is given below.
The 20 recommended positions outputted by the first R-CNN are recorded as B 1-1 、B 1-2 、…、B 1-20 The corresponding probability vectors are P respectively 1-1 、P 1-2 、…、P 1-20 。
The 15 recommended positions output by the second R-CNN are marked as B 2-1 、B 2-2 、…、B 2-15 The corresponding probability vectors are P respectively 2-1 、P 2-2 、…、P 2-15 . The 15 recommended positions are for B 1-1 、B 1-2 、…、B 1-15 Is further corrected.
The 10 recommended positions outputted by the third R-CNN are marked as B 3-1 、B 3-2 、…、B 3-10 The corresponding probability vectors are P respectively 3-1 、P 3-2 、…、P 3-10 . The 10 recommended positions are for B 2-1 、B 2-2 、…、B 2-10 Is further corrected.
Recommended position B 1-16 、B 1-17 、…、B 1-20 Only the first R-CNN is output without correction by the next two R-CNNs. The five recommended positions correspond to the roughThe rate vector is P 1-16 、P 1-17 、…、P 1-20 The category with the largest value in the probability vector is the category of the recommended position. For example, recommended position B 1-16 Probability vector P of (2) 1-16 If the value of the 5 th element is maximum, recommending the position B 1-16 The category of (2) is a fifth category. The decision module 640 recommends position B 1-16 、B 1-17 、…、B 1-20 As position information, probability vector P is set 1-16 、P 1-17 、…、P 1-20 The class with the largest median is used as the corresponding class information to be output.
Recommended position B 1-11 、B 1-12 、…、B 1-15 Output by a first R-CNN and corrected to B by a second R-CNN 2-11 、B 2-12 、…、B 2-15 . The corresponding probability vector of the first R-CNN output is P 1-11 、P 1-12 、…、P 1-15 The corresponding probability vector of the second R-CNN output is P 2-11 、P 2-12 、…、P 2-15 Fusing probability vectors output by the two R-CNNs to obtain a recommended position B 2-11 、B 2-12 、…、B 2-15 Is a weighted probability vector P of (2) i ＝w 1 *P 1-i +w 2 *P 2-i Wherein w is 1 、w 2 The weights of the probability vectors output by the first R-CNN and the second R-CNN are respectively i=11, 12, … and 15. Determining recommended position B from the fused weighted probability vector 2-11 、B 2-12 、…、B 2-15 The category to which the recommendation position belongs, namely the category with the largest value in the weighted probability vector is taken as the category of the corresponding recommendation position. The decision module 640 recommends position B 2-11 、B 2-12 、…、B 2-15 As the position information, the weighted probability vector P 11 、P 12 、…、P 15 The class with the largest median is used as the corresponding class information to be output.
Recommended position B 1-1 、B 1-2 、…、B 1-10 The first R-CNN outputs and is corrected to B after the second R-CNN and the third R-CNN are corrected in turn 3-1 、B 3-2 、…、B 3-10 . The corresponding probability vector of the first R-CNN output is P 1-1 、P 1-2 、…、P 1-10 The corresponding probability vector of the second R-CNN output is P 2-1 、P 2-2 、…、P 2-10 The corresponding probability vector of the third R-CNN output is P 3-1 、P 3-2 、…、P 3-10 Fusing the probability vectors output by the three R-CNNs to obtain a recommended position B 3-1 、B 3-2 、…、B 3-10 Is a weighted probability vector P of (2) i ＝w 1 *P 1-i +w 2 *P 2-i +w 3 *P 3-i Wherein w is 1 、w 2 、w 3 The weights of probability vectors output by the first R-CNN, the second R-CNN and the third R-CNN are respectively i=1, 2, … and 10. Determining recommended position B from the fused weighted probability vector 3-1 、B 3-2 、…、B 3-10 The category to which the recommendation position belongs, namely the category with the largest value in the weighted probability vector is taken as the category of the corresponding recommendation position. The decision module 640 recommends position B 3-1 、B 3-2 、…、B 3-10 As the position information, the weighted probability vector P 1 、P 2 、…、P 10 The class with the largest median is used as the corresponding class information to be output.
In summary, the determination module 640 recommends position B 3-1 、B 3-2 、…、B 3-10 ，B 2-11 、B 2-12 、…、B 2-15 ，B 1-16 、B 1-17 、…、B 1-20 As position information, the corresponding probability vector P 1 、P 2 、…、P 10 ，P 11 、P 12 、…、P 15 ，P 1-16 、P 1-17 、…、P 1-20 The category with the largest median value is used as category information to be output.
In the model test stage, a plurality of test images are input into a trained indicator lamp recognition model, and the model outputs the position information and the category information of the indicator lamps of each test image. And calculating an evaluation index value of the indicator lamp identification model according to the position information and the category information output by the indicator lamp identification model and the real position and the real category of the indicator lamp in the test image. The evaluation index value may be, for example, precision (precision), recall (recall), or the like, but is not limited thereto. When the evaluation index value is greater than the preset threshold value, the indicator light recognition model is judged to be successfully generated, and the generated model is deployed to the application server 110, so that the application server 110 provides the indicator light recognition service for the user.
One embodiment of calculating an evaluation index value of the indicator light recognition model is given below, in which the evaluation index of the model includes accuracy and recall.
Inputting n test images into a trained indicator light recognition model, wherein the model outputs indicator light recognition results res of the test images 1 、res 2 、…、res n Each recognition result includes the detected position information r_box and the category information r_cls of the at least one indicator lamp. The labeling file corresponding to each test image is gt 1 、gt 2 、…、gt n The actual position g_box and the belonging category g_cls of the indicator lamp in the corresponding test image are recorded in each annotation file.
For each position information r_box detected by the model, the overlapping degree IoU of the position information r_box and each actual position g_box of the corresponding test image is calculated, and if IoU of the r_box and a certain actual position g_box in the test image is greater than a threshold (the threshold may be set to 0.7, for example), the position information r_box is recorded as effective position information r_validbox, and the corresponding actual position g_box is recorded as effective actual position g_validbox.
The accuracy (precision) and recall (recovery) of the model are calculated from the valid position information r_validbox and the valid actual position g_validbox.
The accuracy of the model is the ratio of the number of r_validboxes that are the same as the corresponding g_validbox class to the total number of r_boxes that the model detects.
The recall of the model is the ratio of the number of r_validboxes of the same class as the corresponding g_validbox to the total number of actual positions g_boxes of the n test images.
And setting thresholds of accuracy and recall rate respectively, and successfully generating the indicator light recognition model when the accuracy and recall rate of the model are both larger than the corresponding thresholds.
The generated model is deployed to the application server 110, and the application server 110 can provide the indicator light recognition service to the user. Specifically, referring to fig. 1, the user may take an image using the terminal device 120 in alignment with the position of the dashboard indicator, and the taken image may include background information such as a steering wheel, etc., or may be a separate indicator. Subsequently, the user uploads the photographed image to the application server 110. The application server 110 invokes the model interface to input the image into the indicator recognition model. The model will output the position information and category information of the indicator light in the image while giving a confidence (i.e. the probability that the position belongs to the category). When the confidence coefficient is greater than the preset threshold, the application server 110 returns the position and category information of the corresponding indicator lamp and the icon of the corresponding indicator lamp to the terminal device 120 for the user to distinguish and refer.
The method of any of claims 1-8, wherein the multi-scale feature extraction module is a Feature Pyramid Network (FPN), the region generation module is a region candidate network (RPN), and the target detection unit is a region-based convolutional neural network (R-CNN).
A10, the method according to any of claims 1-9, wherein in the process of training the indicator recognition model, the determining module uses the recommended position and recommended category output by the last target detecting unit as the position information and category information of the indicator in the instrument panel image.
A11. the method according to any of claims 1-10, further comprising, after said step of training said indicator light recognition model, the following model testing step:
taking an instrument panel image which marks the position and the category of the indicator lamp and does not participate in model training as a test image, and inputting a trained indicator lamp identification model so that the indicator lamp identification model outputs the position information and the category information of the indicator lamp in each test image;
calculating an evaluation index value of the indicator lamp recognition model according to the position information and the category information output by the indicator lamp recognition model and the real position and the real category of the indicator lamp in each test image;
And when the evaluation index value is larger than a preset threshold value, judging that the indicator lamp identification model is successfully generated.
The method according to claim 11, wherein the determining module is adapted to determine the position information and the category information of the indicator lamp in the dashboard image according to the recommended position and the recommended category outputted by each target detecting unit during the test of the trained indicator lamp recognition model.
The various techniques described herein may be implemented in connection with hardware or software or, alternatively, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions of the methods and apparatus of the present invention, may take the form of program code (i.e., instructions) embodied in tangible media, such as removable hard drives, U-drives, floppy diskettes, CD-ROMs, or any other machine-readable storage medium, wherein, when the program is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention.
In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. Wherein the memory is configured to store program code; the processor is configured to execute the method of the invention for identifying a dashboard indicator according to instructions in said program code stored in the memory.
By way of example, and not limitation, readable media comprise readable storage media and communication media. The readable storage medium stores information such as computer readable instructions, data structures, program modules, or other data. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Combinations of any of the above are also included within the scope of readable media.
In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system, or other apparatus. Various general-purpose systems may also be used with examples of the invention. The required structure for a construction of such a system is apparent from the description above. In addition, the present invention is not directed to any particular programming language. It will be appreciated that the teachings of the present invention described herein may be implemented in a variety of programming languages, and the above description of specific languages is provided for disclosure of enablement and best mode of the present invention.

Claims (14)

1. A method of dashboard light identification, executed in a computing device, adapted to generate a light identification model adapted to output location information and category information of a light in a dashboard image, the method comprising the steps of:
Acquiring a plurality of instrument panel images, wherein the instrument panel images comprise at least one indicator lamp;
marking the position and the category of each indicator light in the instrument panel image;
training the indicator lamp recognition model by taking the instrument panel image marked with the position and the category of the indicator lamp as a training image, wherein the indicator lamp recognition model comprises a multi-scale feature extraction module, a region generation module, a target detection module and a judgment module,
the multi-scale feature extraction module takes an instrument panel image as input and outputs a multi-scale feature map of the instrument panel image;
the region generation module comprises a plurality of region generation units, wherein each region generation unit takes a scale feature map output by the multi-scale feature extraction module as input and outputs candidate positions of the indicator lamps in the instrument panel image;
the target detection module comprises a plurality of cascaded target detection units, wherein the first target detection unit takes a plurality of candidate positions output by the region generation module as input, and the other target detection units take recommended positions output by the previous target detection units as input, so as to output recommended positions and recommended categories of the indicator lamps in the instrument panel image;
And the judging module takes the recommended positions and recommended categories output by the target detection units as inputs, and outputs the position information and the category information of the indicator lamps in the instrument panel image.
2. The method of claim 1, further comprising, prior to the step of annotating the location and category of each indicator light in the dashboard image, the step of:
and performing de-duplication processing on the plurality of instrument panel images.
3. The method of claim 2, wherein the deduplication process comprises the steps of:
extracting feature vectors of the instrument panel images respectively;
clustering the plurality of instrument panel images according to the feature vector;
and performing de-duplication treatment on each class respectively.
4. A method as claimed in claim 3, wherein the step of performing the deduplication process separately for each class comprises:
and if the similarity of the two instrument panel images in one class is larger than a first threshold value, deleting any one of the two instrument panel images, wherein the similarity is calculated according to the feature vector.
5. The method of any one of claims 1-4, wherein the step of annotating the location of each indicator light in the dashboard image comprises:
Marking the area range of each indicator lamp in the instrument panel image respectively;
for each area range in which the indicator light is located:
identifying connected domains in the region range;
calculating the area of each connected domain respectively, and taking the connected domain with the area larger than a second threshold value as an effective connected domain;
and taking the circumscribed rectangle of each effective communication domain as the position of the indicator lamp.
6. The method of any one of claims 1-5, wherein after the step of annotating the location and category of each indicator light in the dashboard image, further comprising the step of:
and mirroring the instrument panel image, changing the color space, blurring and rotating, and recording the position and the category of each processed indicator lamp.
7. The method of any one of claims 1-6, wherein prior to the step of annotating the location and category of each indicator light in the dashboard image, further comprising the step of:
dividing an instrument panel image into a plurality of recognition difficulty levels according to the definition of the image, wherein the recognition difficulty levels comprise simple, medium and difficult;
the step of taking the instrument panel image marked with the position and the category of the indicator light as the training image comprises the following steps:
And respectively taking out a plurality of instrument panel images from each recognition difficulty level as training images, wherein the number of the instrument panel images taken out from each recognition difficulty level accords with a preset proportion.
8. The method of claim 7, wherein in the preset ratio, a number of instrument panel images of medium recognition difficulty is greater than a number of instrument panel images of simple recognition difficulty is greater than a number of instrument panel images of difficult recognition difficulty.
9. The method of any of claims 1-8, wherein the multi-scale feature extraction module is a Feature Pyramid Network (FPN), the region generation module is a region candidate network (RPN), and the target detection unit is a region-based convolutional neural network (R-CNN).
10. The method of any one of claims 1-9, wherein the determination module uses the recommended position and recommended category output by the last target detection unit as the position information and category information of the indicator in the dashboard image in training the indicator recognition model.
11. The method of any of claims 1-10, further comprising, after the step of training the indicator light recognition model, the step of model testing of:
Taking an instrument panel image which marks the position and the category of the indicator lamp and does not participate in model training as a test image, and inputting a trained indicator lamp identification model so that the indicator lamp identification model outputs the position information and the category information of the indicator lamp in each test image;
calculating an evaluation index value of the indicator lamp recognition model according to the position information and the category information output by the indicator lamp recognition model and the real position and the real category of the indicator lamp in each test image;
and when the evaluation index value is larger than a preset threshold value, judging that the indicator lamp identification model is successfully generated.
12. The method of claim 11, wherein the determination module is adapted to determine the location information and the category information of the indicator in the dashboard image according to the recommended location and the recommended category outputted by each of the object detection units in the course of testing the trained indicator recognition model.
13. A computing device, comprising:
at least one processor and a memory storing program instructions;
the program instructions, when read and executed by the processor, cause the computing device to perform the method of identifying a dashboard indicator as defined in any one of claims 1-12.
14. A readable storage medium storing program instructions that, when read and executed by a computing device, cause the computing device to perform the method of identifying a dashboard indicator as recited in any of claims 1-12.
CN201911322879.0A 2019-12-20 2019-12-20 Method for identifying instrument panel indicator lamp and computing equipment Active CN111178200B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911322879.0A CN111178200B (en) 2019-12-20 2019-12-20 Method for identifying instrument panel indicator lamp and computing equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911322879.0A CN111178200B (en) 2019-12-20 2019-12-20 Method for identifying instrument panel indicator lamp and computing equipment

Publications (2)

Publication Number Publication Date
CN111178200A CN111178200A (en) 2020-05-19
CN111178200B true CN111178200B (en) 2023-06-02

Family

ID=70653966

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911322879.0A Active CN111178200B (en) 2019-12-20 2019-12-20 Method for identifying instrument panel indicator lamp and computing equipment

Country Status (1)

Country Link
CN (1) CN111178200B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111639647B (en) * 2020-05-22 2023-07-25 深圳市赛为智能股份有限公司 Indicator light state identification method and device, computer equipment and storage medium
CN111652102A (en) * 2020-05-27 2020-09-11 国网山东省电力公司东营供电公司 Power transmission channel target object identification method and system

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9779314B1 (en) * 2014-08-21 2017-10-03 Waymo Llc Vision-based detection and classification of traffic lights
CN109508580A (en) * 2017-09-15 2019-03-22 百度在线网络技术（北京）有限公司 Traffic lights recognition methods and device
WO2019232831A1 (en) * 2018-06-06 2019-12-12 平安科技（深圳）有限公司 Method and device for recognizing foreign object debris at airport, computer apparatus, and storage medium

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9779314B1 (en) * 2014-08-21 2017-10-03 Waymo Llc Vision-based detection and classification of traffic lights
CN109508580A (en) * 2017-09-15 2019-03-22 百度在线网络技术（北京）有限公司 Traffic lights recognition methods and device
WO2019232831A1 (en) * 2018-06-06 2019-12-12 平安科技（深圳）有限公司 Method and device for recognizing foreign object debris at airport, computer apparatus, and storage medium

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
刘全周 ; 贾鹏飞 ; 李占旗 ; 王述勇 ; 王启配 ; .基于深度学习的汽车仪表标识辨别系统设计.新型工业化.2018,(06),全文. *

Also Published As

Publication number Publication date
CN111178200A (en) 2020-05-19

Similar Documents

Publication Publication Date Title
CN109961009B (en) Pedestrian detection method, system, device and storage medium based on deep learning
US11321847B2 (en) Foreground-aware image inpainting
US9978003B2 (en) Utilizing deep learning for automatic digital image segmentation and stylization
US10453204B2 (en) Image alignment for burst mode images
CN109815843B (en) Image processing method and related product
US20180232887A1 (en) Utilizing deep learning for boundary-aware image segmentation
CN106447721B (en) Image shadow detection method and device
CN112529026B (en) Method for providing AI model, AI platform, computing device and storage medium
CN109376631B (en) Loop detection method and device based on neural network
WO2014174932A1 (en) Image processing device, program, and image processing method
CN109191424B (en) Breast mass detection and classification system and computer-readable storage medium
US9025889B2 (en) Method, apparatus and computer program product for providing pattern detection with unknown noise levels
US11392800B2 (en) Computer vision systems and methods for blind localization of image forgery
CN111626295B (en) Training method and device for license plate detection model
CN111178200B (en) Method for identifying instrument panel indicator lamp and computing equipment
CN109165654B (en) Training method of target positioning model and target positioning method and device
CN113095333A (en) Unsupervised feature point detection method and unsupervised feature point detection device
US11367206B2 (en) Edge-guided ranking loss for monocular depth prediction
CN112132130A (en) Real-time license plate detection method and system for whole scene
CN112465050B (en) Image template selection method, device, equipment and storage medium
CN111353325A (en) Key point detection model training method and device
CN112132753A (en) Infrared image super-resolution method and system for multi-scale structure guide image
US11200708B1 (en) Real-time color vector preview generation
Li et al. Deblurring traffic sign images based on exemplars
CN116363019B (en) Image data enhancement method, system and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant