CN114998424B - Vehicle window position determining method and device and vehicle - Google Patents
Vehicle window position determining method and device and vehicle Download PDFInfo
- Publication number
- CN114998424B CN114998424B CN202210929798.2A CN202210929798A CN114998424B CN 114998424 B CN114998424 B CN 114998424B CN 202210929798 A CN202210929798 A CN 202210929798A CN 114998424 B CN114998424 B CN 114998424B
- Authority
- CN
- China
- Prior art keywords
- image
- detected
- points
- determining
- window
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20112—Image segmentation details
- G06T2207/20132—Image cropping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a vehicle window position determining method and device and a vehicle. Wherein, the method comprises the following steps: acquiring an image to be detected, wherein the image to be detected is used for representing a vehicle window of a vehicle; performing Gaussian mapping on an image to be detected to obtain a thermodynamic diagram; determining target key points on the vehicle window in the image to be detected based on the thermodynamic diagram; and determining the position of the vehicle window based on the target key point. The invention solves the technical problem of low accuracy of positioning the vehicle window of the vehicle.
Description
Technical Field
The invention relates to the field of vehicles, in particular to a method and a device for determining positions of vehicle windows and a vehicle.
Background
At present, the position of a driver and information in a seat can be acquired by positioning a window area, so that the driving behavior of the driver is analyzed, and therefore, the accurate positioning of the window position has important significance.
In the related art, a positioning algorithm is usually used for determining the position of the vehicle window, but the positioning algorithm is tedious and time-consuming, has low precision and is difficult to adapt to complex scenes, and the technical problem of low accuracy in positioning the vehicle window still exists.
For the technical problem of low accuracy of positioning the vehicle window, no effective solution is provided at present.
Disclosure of Invention
The embodiment of the invention provides a vehicle window position determining method and device and a vehicle, and at least solves the technical problem that the accuracy of positioning a vehicle window of the vehicle is low.
According to an aspect of an embodiment of the present invention, there is provided a position determining method of a window, including: acquiring an image to be detected, wherein the image to be detected is used for representing a vehicle window of a vehicle; performing Gaussian mapping on an image to be detected to obtain a thermodynamic diagram; determining target key points on the vehicle window in the image to be detected based on the thermodynamic diagram; and determining the position of the vehicle window based on the target key point.
Optionally, performing gaussian mapping on an image to be detected to obtain a thermodynamic diagram, including: identifying the image to be detected to obtain key points in the image to be detected, wherein the key points comprise: four angular points of the image to be detected and a central point between the four angular points; and performing Gaussian mapping on the image to be detected based on the plurality of key points to obtain a plurality of thermodynamic diagrams.
Optionally, based on the key point, performing gaussian mapping on the image to be detected to obtain a thermodynamic diagram, including: obtaining a marked image obtained by marking key points in an image to be detected to obtain a plurality of marked images, wherein only one key point is marked in the marked image, and the number of the marked images is the same as that of the key points; the plurality of marker images are subjected to Gaussian mapping to obtain a plurality of thermodynamic diagrams.
Optionally, the method comprises: respectively determining a peak value in each thermodynamic diagram; and determining the peak value as the confidence of the key point in each thermodynamic diagram, wherein the confidence is used for characterizing the matching degree of the key point and the window.
Optionally, the method comprises: and determining the key points with the confidence degrees larger than the confidence degree threshold value as target key points.
Optionally, determining the position of the window based on the target key point includes: obtaining sub-pixel coordinates of a target key point; and determining the position of the vehicle window based on the sub-pixel point coordinates.
Optionally, the method comprises: and correcting the coordinates of the sub-pixel points based on the four angular points and a preset topological relation between the central points of the four angular points.
According to another aspect of the embodiments of the present invention, there is also provided a position determining apparatus of a window, including: the device comprises an acquisition unit, a detection unit and a processing unit, wherein the acquisition unit is used for acquiring an image to be detected, and the image to be detected is used for representing the window of the vehicle; the processing unit is used for carrying out Gaussian mapping on the image to be detected to obtain a thermodynamic diagram; the first determining unit is used for determining target key points on the vehicle window in the image to be detected based on the thermodynamic diagram; and the second determining unit is used for determining the position of the vehicle window based on the target key point.
According to another aspect of the embodiment of the invention, a vehicle is also provided. The vehicle is used for executing the position determining method of the vehicle window of the embodiment of the invention.
According to another aspect of the embodiments of the present invention, there is also provided a processor. The processor is configured to execute a program, where the program executes the method for determining a position of a window according to the embodiment of the present invention.
According to another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium. The computer-readable storage medium includes a stored program, where when the program is executed, the apparatus in which the computer-readable storage medium is located is controlled to execute the method for determining a position of a window according to the embodiment of the present invention.
In the embodiment of the invention, an image to be detected is obtained, wherein the image to be detected is used for representing the vehicle window of a vehicle; performing Gaussian mapping on an image to be detected to obtain a thermodynamic diagram; determining target key points on the vehicle window in the image to be detected based on the thermodynamic diagram; and determining the position of the vehicle window based on the target key point. That is to say, the Gaussian thermodynamic diagram is obtained by performing Gaussian mapping on the acquired vehicle window area image, the target key point of the vehicle window area image is determined based on the Gaussian thermodynamic diagram, and the position of the vehicle window is determined according to the target key point, so that the technical effect of improving the accuracy of positioning the vehicle window is achieved, and the technical problem of low accuracy of positioning the vehicle window is solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
fig. 1 is a flowchart of a window position determining method according to an embodiment of the present invention;
fig. 2 is a flowchart of another window position determining method according to an embodiment of the present invention;
fig. 3 is a schematic view of a position determining apparatus of a window according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
In accordance with an embodiment of the present invention, there is provided an embodiment of a window position determining method, it should be noted that the steps illustrated in the flowchart of the drawings may be performed in a computer system, such as a set of computer executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
Fig. 1 is a flowchart of a window position determining method according to an embodiment of the present invention, and as shown in fig. 1, the method includes the following steps.
And S102, acquiring an image to be detected, wherein the image to be detected is used for representing the vehicle window of the vehicle.
In the technical scheme provided by the step S102 of the present invention, an image to be detected may be obtained by detecting a high definition video image sequence of the electronic gate, where the image to be detected may be used to represent a window of a vehicle, and may be a window area image, for example, a window area image obtained by shooting in a vehicle driving process; the image of the window area of the vehicle in a difficult sample scene such as overexposure or overnight can be used.
Optionally, an image at least at one moment can be obtained through the electronic gate, an image sequence is obtained, license plate detection and vehicle head detection can be performed on the image sequence, the approximate position of the vehicle window area is determined according to the position prior relation between the license plate and the vehicle head, the image sequence is cut according to the estimated approximate position, the whole area where the license plate is located can be removed from the vehicle head area, and therefore an image to be detected (vehicle window area image) is obtained, wherein a license plate detection algorithm and a vehicle head detection algorithm can be target detection algorithms.
For example, an image sequence can be obtained by acquiring an image at a certain moment photographed by a camera of a traffic road, and a semi-automatic labeling tool can be used for labeling the key point position of an image to be detected in the image sequence.
And step S104, carrying out Gaussian mapping on the image to be detected to obtain a thermodynamic diagram.
In the technical solution provided by step S104 of the present invention, after the image to be detected is obtained, gaussian mapping may be performed on the image to be detected to obtain a thermodynamic diagram, where the thermodynamic diagram may be a gaussian thermodynamic diagram.
Optionally, the size of the image to be detected may be adjusted to a fixed size, for example, 256 × 256, and based on the coordinates of the midpoint of the image to be detected after the size adjustment, the image to be detected after the size adjustment is subjected to gaussian mapping according to the following formula:
the coordinate value of the generated thermodynamic diagram (64 × 64) can be represented by (x, y), and the value ranges of (x, y) can be x respectively
（0，64），y
(0, 64) the scaling factor that generates the Gaussian kernel may be passed
To carry out the presentation of the contents,
it may be 1.3, and it should be noted that this is only an example, and the size of the scaling factor and the value range of (x, y) are not specifically limited.
And step S106, determining target key points on the vehicle window in the image to be detected based on the thermodynamic diagram.
In the technical scheme provided by the step S106 of the present invention, after the thermodynamic diagram is obtained, the target key points on the vehicle window in the image to be detected may be determined based on the thermodynamic diagram, where the target key points may be four corner points and four center points of the image to be detected, and may be used to position the vehicle window.
For example, the point with the largest thermal value in the thermodynamic diagram may be determined as a target key point on the vehicle window in the image to be detected, wherein the target key point may be determined by the sub-pixel coordinate of the point with the largest thermal value.
And step S108, determining the position of the vehicle window based on the target key point.
In the technical solution provided in step S108 of the present invention, after the target key point is determined, the position of the window may be determined based on the target key point on the window.
Alternatively, the position of the window may be determined based on the position of the target keypoint.
In the above steps S102 to S108 of the present application, a gaussian thermodynamic diagram is obtained by performing gaussian mapping on the obtained image to be detected, where the image to be detected is used to represent a window of a vehicle, a target key point on the window in the image to be detected is determined based on the thermodynamic diagram, and a position of the window is determined based on the key point. In other words, the Gaussian thermodynamic diagram is obtained by performing Gaussian mapping on the acquired vehicle window area image, the target key point of the vehicle window area image is determined based on the Gaussian thermodynamic diagram, and the position of the vehicle window is determined according to the target key point, so that the technical effect of improving the accuracy of positioning the vehicle window is achieved, and the technical problem of low accuracy of positioning the vehicle window is solved.
The above-described method of this embodiment is further described below.
As an alternative embodiment, in step S104, the gaussian mapping is performed on the image to be detected to obtain a thermodynamic diagram, including: identifying the image to be detected to obtain key points in the image to be detected, wherein the key points comprise: four corner points of an image to be detected and a central point between the four corner points; and performing Gaussian mapping on the image to be detected based on the plurality of key points to obtain a plurality of thermodynamic diagrams.
In the embodiment of the invention, the image to be detected can be identified, the key points in the image to be detected can be obtained, and the image to be detected is subjected to Gaussian mapping based on a plurality of key points, so that a plurality of thermodynamic diagrams can be obtained, wherein the key points can comprise four corner points of the image to be detected and a central point between the four corner points.
Optionally, before the image to be detected is identified to obtain the key points, the bayonet data of various scenes are collected to obtain the true values of the image to be detected of a plurality of scenes and the corresponding key points in the image to be detected, so as to construct a training data set of the key points of the image to be detected, after the training data set is trained in a network construction based on the key point training data set, the key points in the image to be detected are obtained by inputting the image to be detected, wherein the plurality of scenes can include a shooting scene under normal illumination, a shooting scene under night illumination, a shooting scene under rainy day illumination, a scene under windshield reflection and the like, the scene description here is only an example, and any scene type meeting the requirements of an actual project scene can construct the training data set of the key points of the image to be detected.
Optionally, eight key points of the car window can be labeled by using a semi-automatic labeling tool to obtain a true value of the key points of the car window, four key points in the eight key points can be four corner points of the car window, other four key points can be central points of four sides of the car window, the central points of the four sides of the car window can be obtained through position constraint relations between the four corner points and the central points of the four sides of the car window, and the positions of the central points of the four sides can be manually finely adjusted to enable the central points to be attached to the car window and the edge of the car body.
Optionally, because the size of the window area of different vehicle types is inconsistent with the proportional relation between the eight corresponding angular points, in the process of acquiring the image to be detected, in order to enhance the robustness of the model, the image to be detected of different vehicle types can be acquired for training, so as to achieve the purpose of improving the accuracy of identifying the image to be detected to obtain the key point, wherein the different vehicle types can include passenger vehicles, common passenger vehicles, large trucks, trucks and the like, the vehicle types are only exemplified here, and the data of the image to be detected of any vehicle conforming to the actual project can be acquired.
Optionally, after the training data set is constructed, the labeled training data set may be preprocessed, so as to achieve the purpose of improving the training effect of the data set.
For example, a three primary color (Red Green Blue, abbreviated as RGB) image can be converted into a three-channel grayscale image; the position of the center point of the car window can be obtained according to the positions of the four corner points of the car window, the maximum value of the width and the height of the car window is expanded by 1.5 times, and the image of the car window area is zoomed and cut; the image of the car window area can be cut in a rotating mode by taking four corner points and the car window central point as rotating reference points and an angle [ -10,10] as a rotating angle; the noise processing can be carried out on the image through processing such as salt and pepper noise adding, gaussian fuzzification processing, high exposure scenes, dark low exposure scenes and the like; the window area image after noise processing may be normalized, for example, by subtracting 128 from the pixel value and dividing by 256, thereby completing the pre-processing of the labeled training data set.
Optionally, after preprocessing the labeled training data set, a lightweight model and a loss-related function can be constructed, and the key point positioning model is trained.
For another example, a lightweight model can be built based on a lightweight network (mobile) structure, the coding network can include a convolutional network (depth-wise) structure and a convolutional network (point-wise) structure, the decoding network can include a deconvolution structure of nearest neighbor interpolation, and the number of channels of the model is pruned under the condition that the positioning accuracy of the key point is not affected, so that the computational power requirement of the platform is adapted, and the time consumption of model operation is reduced.
Optionally, in the training process of the model, a loss function may be trained based on a key point thermodynamic diagram regression model, so that the model achieves the purpose of improving the attention of the position of the key point in the thermodynamic diagram in the training process, and the loss function may be represented by the following formula:
wherein, the predicted value can be represented by y, and the real value can be represented by y
Is expressed by an exponential factor which can be expressed by
Is expressed by the threshold value
Is expressed by a scaling factor
Expressed, the error gain factor can be passed
Expressed, the mean square error gain factor can be represented by A, the constant by C, and the exponential factor
Can be that1.8, error gain factor
May be 15.3, threshold
May be 0.6, scaling factor
May be 1.2 and the mean square error gain factor may be
The constant can be
It should be noted that, the size of the exponential factor, the error gain factor, the threshold, the scaling factor, etc. is not specifically limited, which is only an example.
Optionally, in the training of the loss function, for the background pixels, as the training loss value decreases, the background value gradually decreases to 0; for the foreground pixel, as the training loss value is reduced, the error is gradually reduced, so that the aims of reducing the influence of other pixels and accelerating the convergence of the model are fulfilled, and therefore the position of the key point can be the highest bright point in the thermodynamic diagram and can also be the foreground pixel.
In the embodiment of the invention, training can be carried out based on the collected true value data sets of the key points in the images to be detected of a plurality of scenes and different vehicle types and the corresponding images to be detected, the key points in the images to be detected are identified by inputting the images to be detected, and therefore, the images to be detected are subjected to Gaussian mapping based on a plurality of key points to obtain a plurality of thermodynamic diagrams.
As an alternative embodiment, based on the key point, performing gaussian mapping on the image to be detected to obtain a thermodynamic diagram, including: obtaining a marked image obtained by marking key points in an image to be detected to obtain a plurality of marked images, wherein only one key point is marked in the marked image, and the number of the marked images is the same as that of the key points; the plurality of marker images are subjected to Gaussian mapping to obtain a plurality of thermodynamic diagrams.
In the embodiment of the invention, based on the key points, the key points of the image to be detected can be marked by using a semi-automatic marking tool, the marked image after marking is obtained, a plurality of marked images are obtained, and the plurality of marked images can be subjected to Gaussian mapping respectively to obtain a plurality of thermodynamic diagrams.
Optionally, the keypoints may be labeled through a true value, for example, 1 may be used to indicate that the mark position is a keypoint, only one keypoint may be marked in the mark image, the number of the mark images may be the same as the number of the keypoints, for example, eight mark images may correspond to eight key points, and the eight mark images correspond to eight thermodynamic diagrams.
As an alternative embodiment, a peak value in each thermodynamic diagram is determined separately; and determining the peak value as the confidence of the key point in each thermodynamic diagram, wherein the confidence is used for characterizing the matching degree of the key point and the window.
In the embodiment of the present invention, after obtaining a plurality of thermodynamic diagrams, a peak value in each thermodynamic diagram may be determined, and the peak value may be determined as the confidence of a key point in each thermodynamic diagram, where the peak value may be a maximum value in the thermodynamic diagram and may be represented by argmax; the confidence level can be used for characterizing the matching degree of the key point and the car window.
Alternatively, the peak in the thermodynamic diagram may be determined by the following equation:
wherein, the value corresponding to each point in the thermodynamic diagram can be represented by feature.
Optionally, a peak value in the thermodynamic diagram may be determined, a point corresponding to the peak value may be determined as a key point, and the peak value may be determined as a confidence coefficient of the key point, where the higher the confidence coefficient is, the higher the matching degree of the key point and the vehicle window is; the lower the confidence, the lower the degree of matching of the key point to the window.
As an alternative embodiment, the keypoints with the confidence degrees larger than the confidence degree threshold value are determined as the target keypoints.
In the embodiment of the present invention, the key points with the confidence greater than the confidence threshold may be determined as target key points, where the confidence threshold may be used to determine target key points in the image to be detected, and may be a manual preset value, which is not limited specifically here.
Optionally, it may be determined whether the confidence of the keypoint is greater than a confidence threshold, and if the confidence of the keypoint is greater than the confidence threshold, the keypoint is determined to be the target keypoint; if the confidence of the keypoint is not greater than the confidence threshold, the keypoint is not considered as the target keypoint.
As an alternative embodiment, in step S108, determining the position of the window based on the target key point includes: obtaining sub-pixel coordinates of a target key point; and determining the position of the vehicle window based on the sub-pixel point coordinates.
In the embodiment of the invention, after the target key point is determined, the sub-pixel coordinates of the target key point can be obtained, and the position of the vehicle window is determined based on the sub-pixel coordinates.
Optionally, the sub-pixel coordinates of the target key point may be determined based on the target key point, where the sub-pixel coordinates may be represented by the following formula:
optionally, the position of the window may be determined according to the coordinate correspondence of the target key points, and when the confidence degrees of the eight target key points of the window all satisfy the confidence degree threshold, the position of the window may be directly determined, where the confidence degree of a certain corner point of the four corner points of the window may be determined by the corner point of the center of the window.
In the embodiment of the invention, the sub-pixel coordinates of the target key point are obtained by introducing the thermodynamic diagram, and the position of the vehicle window is determined based on the sub-pixel coordinates, so that the accuracy of positioning the vehicle window of the vehicle is improved.
As an optional embodiment, the coordinates of the sub-pixel points are corrected based on a preset topological relation between the four corner points and the central point between the four corner points.
In the embodiment of the present invention, the coordinates of the sub-pixel points may be modified based on a preset topological relationship between four corner points in the key points and a central point between the four corner points, where the topological relationship may include whether a vertical coordinate between key points on upper and lower horizontal sides is within a set horizontal change threshold range, whether a horizontal coordinate between key points on left and right vertical sides is within a set vertical change threshold range, whether lengths of the upper and lower horizontal sides and the left and right vertical sides are within a set ratio threshold range, and whether an included angle between the upper and lower horizontal sides is within a set included angle threshold range.
For example, it can be determined whether the vertical coordinate between the key points on the upper and lower horizontal sides is within the set horizontal variation threshold range, and if not, the sub-pixel coordinates can be corrected to make the vertical coordinate between the key points on the upper and lower horizontal sides within the set horizontal variation threshold range; whether the abscissa between the key points of the left vertical side and the right vertical side is within a set vertical change threshold range can be judged, and if not, the coordinates of the sub-pixel points can be corrected to enable the abscissa between the key points of the left vertical side and the right vertical side to be within the set vertical change threshold range; the length of the upper transverse edge, the length of the lower transverse edge, the length of the left vertical edge, the length of the right vertical edge and the length of the upper transverse edge, the length of the lower transverse edge, the length of the left vertical edge and the length of the right vertical edge can be judged whether to be within a set ratio threshold range, and if not, the coordinates of the sub-pixel points can be corrected to enable the lengths of the upper transverse edge, the lower transverse edge, the left vertical edge and the right vertical edge to be within the set ratio threshold range; whether the included angle of the upper transverse edge and the lower transverse edge is within the set included angle threshold range or not can be judged, and if not, the coordinates of the sub-pixel points can be corrected to enable the included angle of the upper transverse edge and the lower transverse edge to be within the set included angle threshold range.
In the embodiment of the invention, the sub-pixel points at the four corner positions can be corrected according to the topological relation corresponding to the key points, so that the accuracy of detecting the four corner positions is improved.
According to the embodiment, the Gaussian map is performed on the acquired vehicle window area image to obtain the Gaussian thermodynamic diagram, the target key point of the vehicle window area image is determined based on the Gaussian thermodynamic diagram, and the position of the vehicle window is determined according to the target key point, so that the technical effect of improving the accuracy of positioning the vehicle window is achieved, and the technical problem of low accuracy of positioning the vehicle window is solved.
Example 2
The technical solutions of the embodiments of the present invention will be illustrated below with reference to preferred embodiments.
In the intelligent transportation system, the position of a driver and information in a seat can be acquired by positioning a window area, so that the driving behavior of the driver is analyzed, and meanwhile, an important basis is provided for searching vehicles.
In a related technology, a traditional method for positioning a car window area based on image texture and gradient analysis is provided, the method can be used for quickly positioning the car window based on a color difference mean value or gradient, but the method needs more manually set parameters and has strong regularity and still has the technical problem of low adaptability to a scene.
In another related technology, a machine learning method based on a point regression model is provided, and positions of four corner points of a vehicle window are positioned by training a positioning model, but the method still has the technical problem of low positioning accuracy.
In another related technology, a machine learning method based on a multi-stage model from coarse positioning to fine positioning is provided, the method adopts a two-stage positioning model, firstly, four points are coarsely positioned through a large model, and then, four points are finely positioned through four small models, but the method still has the technical problem of complexity.
In order to solve the above problems, an embodiment of the present invention provides a method for determining a position of a window, which is based on a low computational power lightweight model, and includes locating positions of key points of the window by gaussian thermodynamic diagram regression, accurately regressing information in the gaussian thermodynamic diagram by using characteristics of a convolution network, and calculating positions of sub-pixel points by using a position of a maximum value and a neighborhood position, thereby completing post-processing of the gaussian thermodynamic diagram, and finally locating the window by using eight key points and redundancy thereof.
The following further describes embodiments of the present invention.
Fig. 2 is a flowchart of another window position determining method according to an embodiment of the present invention, as shown in fig. 2, which includes the following steps.
In step S201, a window approximate region position is acquired.
In the embodiment of the invention, an image at least at one moment can be obtained through the electronic gate to obtain an image sequence, the image sequence can be subjected to license plate detection and vehicle head detection, the approximate position of the vehicle window area is determined according to the position prior relation between the license plate and the vehicle head, the image sequence is cut according to the estimated approximate position, the whole area where the license plate is located can be removed from the vehicle head area, and an image to be detected (vehicle window area image) is obtained, wherein the license plate detection algorithm and the vehicle head detection algorithm can be target detection algorithms.
For example, an image sequence can be obtained by acquiring an image at a certain moment shot by a camera of a traffic road, and the positions of key points of the image to be detected in the image sequence can be labeled by a semi-automatic labeling tool.
Step S202, a key point data set is constructed.
In the embodiment of the invention, the bayonet data of various scenes can be collected to obtain the images to be detected of a plurality of scenes and the truth values of the corresponding key points in the images to be detected, so as to construct the training data set of the key points of the images to be detected, and after the training data set is trained in a network constructed based on the key point training data set, the key points in the images to be detected are obtained by inputting the images to be detected, wherein the scenes can comprise a shooting scene under normal illumination, a shooting scene under night illumination, a shooting scene under rainy illumination, a scene under windshield reflection and the like, the scene description is only an example, and any scene type meeting the requirements of actual project scenes can construct the training data set of the key points of the images to be detected.
Optionally, eight key points of the car window can be marked by using a semi-automatic marking tool to obtain a true value of the key points of the car window, four key points in the eight key points can be four corner points of the car window, other four key points can be central points of four edges of the car window, the central points of the four edges of the car window can be obtained through position constraint relations between the four corner points and the central points of the four edges, and the positions of the central points of the four edges can be manually finely adjusted to enable the central points to be attached to the car window and the edge of the car body.
Optionally, because the size of the window area of different vehicle types is inconsistent with the proportional relation between the corresponding eight corner points, in the process of acquiring the image to be detected, in order to enhance the robustness of the model, the image to be detected of different vehicle types can be acquired for training, so as to achieve the purpose of improving the accuracy of identifying the image to be detected to obtain the key points, wherein different vehicle types can include passenger vehicles, common passenger vehicles, large trucks, trucks and the like, the vehicle type is only an example, and any image data to be detected which accords with the actual project vehicle can be acquired.
Optionally, after the training data set is constructed, the labeled training data set may be preprocessed, so as to achieve the purpose of improving the training effect of the data set.
For example, a three primary color (Red Green Blue, abbreviated as RGB) image can be converted into a three-channel grayscale image; the position of the center point of the car window can be obtained according to the positions of the four corner points of the car window, the maximum value of the width and the height of the car window is expanded by 1.5 times, and the image of the car window area is zoomed and cut; the image of the car window area can be cut in a rotating mode by taking four corner points and the car window central point as rotating reference points and an angle [ -10,10] as a rotating angle; the noise processing can be carried out on the image through processing such as salt and pepper noise adding, gaussian fuzzification processing, high exposure scenes, dark low exposure scenes and the like; the window area image after noise processing may be normalized, for example, by subtracting 128 from the pixel value and dividing by 256, thereby completing the pre-processing of the labeled training data set.
And step S203, building a model and training.
In the embodiment of the invention, after the labeled training data set is preprocessed, a lightweight model and a related loss function can be built, and the key point positioning model is trained.
Optionally, a lightweight model may be built based on a lightweight network (mobile) structure, the coding network may include a convolutional network (depth-wise) structure and a convolutional network (point-wise) structure, the decoding network may include a deconvolution structure of nearest neighbor interpolation, and the number of channels of the model is pruned without affecting the positioning accuracy of the key point, so that the computation force requirement of the platform is adapted, and the time consumed by running the model is reduced.
Optionally, in the training process of the model, a loss function may be trained based on a key point thermodynamic diagram regression model, so that the model achieves the purpose of improving the attention of the position of the key point in the thermodynamic diagram in the training process, and the loss function may be represented by the following formula:
wherein, the predicted value can be represented by y, and the real value can be represented by y
Is expressed by an exponential factor which can be expressed by
Indicating that the threshold value can be passed
Is expressed by a scaling factor
Expressed, the error gain factor can be passed
Expressed as a mean square error gain factor, which can be represented by A, a constant by C, and an exponential factor
May be 1.8, error gain factor
May be 15.3, threshold
May be 0.6, scaling factor
May be 1.2 and the mean square error gain factor may be
The constant can be
It should be noted that, the size of the exponential factor, the error gain factor, the threshold, the scaling factor, etc. is not specifically limited, which is only an example.
Optionally, in the training of the loss function, for the background pixels, as the training loss value decreases, the background value gradually decreases to 0; for the foreground pixel, as the training loss value is reduced, the error is gradually reduced, so that the aims of reducing the influence of other pixels and accelerating the convergence of the model are fulfilled, and therefore the position of the key point can be the highest bright point in the thermodynamic diagram and can also be the foreground pixel.
In step S204, a thermodynamic diagram is acquired.
In the embodiment of the invention, the size of the image to be detected can be adjusted and input into the vehicle window key point positioning model, so that the thermodynamic diagrams of eight key points of the image to be detected are obtained.
Optionally, the size of the image to be detected may be adjusted to a fixed size, for example, 256 × 256, and based on the coordinates of the midpoint of the image to be detected after the size adjustment, the image to be detected after the size adjustment is subjected to gaussian mapping according to the following formula:
the coordinate value of the generated thermodynamic diagram (64 × 64) can be represented by (x, y), and the value ranges of (x, y) can be x respectively
（0，64），y
(0, 64) the scaling factor that generates the Gaussian kernel may be passed
To carry out the presentation of the contents,
it may be 1.3, and it should be noted that this is only an example, and the size of the scaling factor and the value range of (x, y) are not specifically limited.
In step S205, the window area position is determined.
In the embodiment of the present invention, after obtaining a plurality of thermodynamic diagrams, a peak value in each thermodynamic diagram may be determined, and the peak value may be determined as the confidence of a key point in each thermodynamic diagram, where the peak value may be a maximum value in the thermodynamic diagram and may be represented by argmax; the confidence level can be used for characterizing the matching degree of the key point and the car window.
Alternatively, the peak in the thermodynamic diagram may be determined by the following equation:
wherein, the value corresponding to each point in the thermodynamic diagram can be represented by feature.
Optionally, a peak value in the thermodynamic diagram may be determined, a point corresponding to the peak value may be determined as a key point, and the peak value may be determined as a confidence coefficient of the key point, where the higher the confidence coefficient is, the higher the matching degree between the key point and the vehicle window is; the lower the confidence, the lower the degree of matching of the key point to the window.
Optionally, the keypoints with the confidence degrees larger than the confidence degree threshold value may be determined as target keypoints, where the confidence degree threshold value may be an important basis for determining the target keypoints in the image to be detected.
For example, if the confidence of the keypoint is greater than the confidence threshold, determining the keypoint as the target keypoint; if the keypoint confidence is not greater than the confidence threshold, then the keypoint is not considered as the target keypoint.
Alternatively, the sub-pixel coordinates of the target keypoint may be determined based on the target keypoint, wherein the sub-pixel coordinates may be represented by the following formula:
optionally, the position of the window may be determined according to the coordinate correspondence of the target key points, and when the confidence degrees of the eight target key points of the window all satisfy the confidence degree threshold, the position of the window may be directly determined, where the confidence degree of a certain corner point of the four corner points of the window may be determined by the corner point of the center of the window.
Optionally, the sub-pixel points at the four corner positions can be corrected according to the topological relation corresponding to the key points, so that the accuracy of detecting the four corner points is improved.
For example, the position relationship of the key points needs to satisfy the corresponding topological relationship, and the topological relationship may include whether the ordinate between the key points on the upper and lower horizontal sides is within the set horizontal variation threshold range, whether the abscissa between the key points on the left and right vertical sides is within the set vertical variation threshold range, whether the lengths of the upper and lower horizontal sides and the left and right vertical sides are within the set ratio threshold range, and whether the included angle between the upper and lower horizontal sides is within the set included angle threshold range.
For example, it can be determined whether the vertical coordinate between the key points on the upper and lower horizontal sides is within the set horizontal variation threshold range, and if not, the sub-pixel coordinates can be corrected to make the vertical coordinate between the key points on the upper and lower horizontal sides within the set horizontal variation threshold range; whether the abscissa between the key points on the left vertical side and the right vertical side is within a set vertical change threshold range can be judged, and if the abscissa between the key points on the left vertical side and the right vertical side is not within the set vertical change threshold range, the coordinates of the sub-pixel points can be corrected to enable the abscissa between the key points on the left vertical side and the right vertical side to be within the set vertical change threshold range; the length of the upper transverse edge, the length of the lower transverse edge, the length of the left vertical edge, the length of the right vertical edge and the length of the upper transverse edge, the length of the lower transverse edge, the length of the left vertical edge and the length of the right vertical edge can be judged whether to be within a set ratio threshold range, and if not, the coordinates of the sub-pixel points can be corrected to enable the lengths of the upper transverse edge, the lower transverse edge, the left vertical edge and the right vertical edge to be within the set ratio threshold range; whether the included angle of the upper transverse edge and the lower transverse edge is within the set included angle threshold range or not can be judged, and if not, the coordinates of the sub-pixel points can be corrected to enable the included angle of the upper transverse edge and the lower transverse edge to be within the set included angle threshold range.
According to the embodiment, the Gaussian map is performed on the acquired vehicle window area image to obtain the Gaussian thermodynamic diagram, the target key point of the vehicle window area image is determined based on the Gaussian thermodynamic diagram, and the position of the vehicle window is determined according to the target key point, so that the technical effect of improving the accuracy of positioning the vehicle window is achieved, and the technical problem of low accuracy of positioning the vehicle window is solved.
Example 3
According to the embodiment of the invention, the invention further provides a device for determining the position of the vehicle window. It should be noted that the window position determining apparatus may be used to execute the window position determining method in embodiment 1.
Fig. 3 is a schematic view of a position determining apparatus of a window according to an embodiment of the present invention. As shown in fig. 3, the window position determining apparatus 300 may include: an acquisition unit 301, a first processing unit 302, a second processing unit 303 and a third processing unit 304.
The acquiring unit 301 is configured to acquire an image to be detected, where the image to be detected is used to represent a window of a vehicle;
the processing unit 302 is configured to perform gaussian mapping on an image to be detected to obtain a thermodynamic diagram;
a first determining unit 303, configured to determine a target key point on a vehicle window in an image to be detected based on the thermodynamic diagram;
a second determining unit 304, configured to determine the position of the window based on the target key point.
Optionally, the processing unit 302 includes: the first processing module is used for identifying the image to be detected to obtain key points in the image to be detected, wherein the key points comprise: four angular points of the image to be detected and a central point between the four angular points; and performing Gaussian mapping on the image to be detected based on the plurality of key points to obtain a plurality of thermodynamic diagrams.
Optionally, the first processing module comprises: the first processing submodule is used for acquiring a marked image obtained by marking key points in an image to be detected to obtain a plurality of marked images, wherein only one key point is marked in the marked image, and the number of the marked images is the same as that of the key points; and performing Gaussian mapping on the plurality of marked images to obtain a plurality of thermodynamic diagrams.
Optionally, the first processing module comprises: a first determining submodule for determining a peak value in each thermodynamic diagram respectively; and determining the peak value as the confidence of the key point in each thermodynamic diagram, wherein the confidence is used for characterizing the matching degree of the key point and the window.
Optionally, the first processing module comprises: and the second determining submodule is used for determining the key points with the confidence degrees larger than the confidence degree threshold value as target key points.
Optionally, the first processing module comprises: the third determining submodule is used for acquiring the sub-pixel coordinates of the target key point; and determining the position of the vehicle window based on the sub-pixel point coordinates.
Optionally, the second processing module comprises: and the correction submodule is used for correcting the coordinates of the sub-pixel points based on the four angular points and a preset topological relation between central points of the four angular points.
In the embodiment of the invention, an image to be detected is obtained through an obtaining unit, wherein the image to be detected is used for representing the vehicle window of a vehicle; performing Gaussian mapping on an image to be detected through a processing unit to obtain a thermodynamic diagram; determining target key points on a vehicle window in an image to be detected based on the thermodynamic diagram through a first determining unit; and determining the position of the vehicle window based on the target key point through a second determination unit. In other words, the Gaussian thermodynamic diagram is obtained by performing Gaussian mapping on the acquired vehicle window area image, the target key point of the vehicle window area image is determined based on the Gaussian thermodynamic diagram, and the position of the vehicle window is determined according to the target key point, so that the technical effect of improving the accuracy of positioning the vehicle window is achieved, and the technical problem of low accuracy of positioning the vehicle window is solved.
Example 4
According to an embodiment of the present invention, there is also provided a vehicle for executing the position determination method of a window in embodiment 1.
Example 5
According to an embodiment of the present invention, there is also provided a processor configured to execute a program, where the program executes the position determining method of the window in embodiment 1 when running.
Example 6
According to an embodiment of the present invention, there is also provided a computer-readable storage medium including a stored program, wherein the program executes the position determining method of the window in embodiment 1.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present invention, the description of each embodiment has its own emphasis, and reference may be made to the related description of other embodiments for parts that are not described in detail in a certain embodiment.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described apparatus embodiments are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or may not be executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (6)
1. A method for determining a position of a window, comprising:
acquiring an image to be detected, wherein the image to be detected is used for representing a vehicle window of a vehicle;
carrying out Gaussian mapping on the image to be detected to obtain a thermodynamic diagram;
determining target key points on the vehicle window in the image to be detected based on the thermodynamic diagram;
determining the position of the vehicle window based on the target key point;
wherein, to waiting to examine the image carry out the gaussian mapping, obtain thermodynamic diagram, include: identifying the image to be detected to obtain key points in the image to be detected, wherein the key points comprise: four corner points of the image to be detected and a central point between the four corner points; based on the plurality of key points, carrying out Gaussian mapping on the image to be detected to obtain a plurality of thermodynamic diagrams;
determining target key points on the vehicle window in the image to be detected based on the thermodynamic diagram, comprising: respectively determining a peak value in each thermodynamic diagram; determining the peak value as a confidence coefficient of a key point in each thermodynamic diagram, wherein the confidence coefficient is used for representing the matching degree of the key point and the vehicle window; and determining the key points with the confidence degrees larger than a confidence degree threshold value as the target key points.
2. The method according to claim 1, wherein performing gaussian mapping on the image to be detected based on the key points to obtain the thermodynamic diagram comprises:
obtaining a marked image obtained by marking key points in the image to be detected to obtain a plurality of marked images, wherein only one key point is marked in the marked image, and the number of the marked images is the same as that of the key points;
and performing Gaussian mapping on the plurality of marked images respectively to obtain a plurality of thermodynamic diagrams.
3. The method of claim 1, wherein determining the location of the window based on the target keypoints comprises:
obtaining sub-pixel coordinates of the target key points;
and determining the position of the vehicle window based on the sub-pixel point coordinates.
4. The method of any of claim 3, further comprising:
and correcting the coordinates of the sub-pixel points based on the four angular points and a preset topological relation between central points of the four angular points.
5. A position determining device for a window, comprising:
the device comprises an acquisition unit, a detection unit and a processing unit, wherein the acquisition unit is used for acquiring an image to be detected, and the image to be detected is used for representing the window of a vehicle;
the processing unit is used for carrying out Gaussian mapping on the image to be detected to obtain a thermodynamic diagram;
a first determining unit, configured to determine a target key point on the vehicle window in the image to be detected based on the thermodynamic diagram;
the second determining unit is used for determining the position of the vehicle window based on the target key point;
the processing unit performs Gaussian mapping on the image to be detected to obtain a thermodynamic diagram through the following steps: the image to be detected is identified to obtain key points in the image to be detected, wherein the key points comprise: four corner points of the image to be detected and a central point between the four corner points; based on the plurality of key points, carrying out Gaussian mapping on the image to be detected to obtain a plurality of thermodynamic diagrams;
the first determination unit determines a target key point on the vehicle window in the image to be detected based on the thermodynamic diagram by: respectively determining a peak value in each thermodynamic diagram; determining the peak value as a confidence coefficient of a key point in each thermodynamic diagram, wherein the confidence coefficient is used for representing the matching degree of the key point and the vehicle window; and determining the key points with the confidence degrees larger than a confidence degree threshold value as the target key points.
6. A vehicle, characterized by being adapted to carrying out the method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210929798.2A CN114998424B (en) | 2022-08-04 | 2022-08-04 | Vehicle window position determining method and device and vehicle |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210929798.2A CN114998424B (en) | 2022-08-04 | 2022-08-04 | Vehicle window position determining method and device and vehicle |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114998424A CN114998424A (en) | 2022-09-02 |
| CN114998424B true CN114998424B (en) | 2022-10-21 |
Family
ID=83022858
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210929798.2A Active CN114998424B (en) | 2022-08-04 | 2022-08-04 | Vehicle window position determining method and device and vehicle |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114998424B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112990197A (en) * | 2021-03-17 | 2021-06-18 | 浙江商汤科技开发有限公司 | License plate recognition method and device, electronic equipment and storage medium |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108182383B (en) * | 2017-12-07 | 2021-07-20 | 浙江大华技术股份有限公司 | Vehicle window detection method and device |
| CN108428248B (en) * | 2018-03-14 | 2021-04-13 | 苏州科达科技股份有限公司 | Vehicle window positioning method, system, equipment and storage medium |
| CN110276344B (en) * | 2019-06-04 | 2023-11-24 | 腾讯科技（深圳）有限公司 | Image segmentation method, image recognition method and related device |
| CN110532984B (en) * | 2019-09-02 | 2022-10-11 | 北京旷视科技有限公司 | Key point detection method, gesture recognition method, device and system |
| CN112052807B (en) * | 2020-09-10 | 2022-06-10 | 讯飞智元信息科技有限公司 | Vehicle position detection method, device, electronic equipment and storage medium |
| CN114067406A (en) * | 2021-11-18 | 2022-02-18 | 厦门熵基科技有限公司 | Key point detection method, device, equipment and readable storage medium |
| CN114550144A (en) * | 2022-02-21 | 2022-05-27 | 中国第一汽车股份有限公司 | Safety belt detection method, device, equipment and storage medium |
-
2022
- 2022-08-04 CN CN202210929798.2A patent/CN114998424B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112990197A (en) * | 2021-03-17 | 2021-06-18 | 浙江商汤科技开发有限公司 | License plate recognition method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114998424A (en) | 2022-09-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109785291B (en) | Lane line self-adaptive detection method | |
| US11042966B2 (en) | Method, electronic device, and storage medium for obtaining depth image | |
| CN112016614A (en) | Construction method of optical image target detection model, target detection method and device | |
| CN107705254B (en) | City environment assessment method based on street view | |
| CN110428425B (en) | Sea-land separation method of SAR image based on coastline vector data | |
| CN112598922B (en) | Parking space detection method, device, equipment and storage medium | |
| WO2006081018A1 (en) | Object-of-interest image capture | |
| CN111768332A (en) | Splicing method of vehicle-mounted all-around real-time 3D panoramic image and image acquisition device | |
| CN111382625A (en) | Road sign identification method and device and electronic equipment | |
| CN114693760A (en) | Image correction method, device and system and electronic equipment | |
| CN108364320A (en) | camera calibration method, terminal device and computer readable storage medium | |
| Wang et al. | Combining semantic scene priors and haze removal for single image depth estimation | |
| CN111723805B (en) | Method and related device for identifying foreground region of signal lamp | |
| JP2020095621A (en) | Image processing device and image processing method | |
| CN114170565A (en) | Image comparison method and device based on unmanned aerial vehicle aerial photography and terminal equipment | |
| CN114998424B (en) | Vehicle window position determining method and device and vehicle | |
| CN112528994A (en) | Free-angle license plate detection method, license plate identification method and identification system | |
| CN111062384A (en) | Vehicle window accurate positioning method based on deep learning | |
| JP2017174380A (en) | Recognition device, method for recognizing object, program, and storage medium | |
| CN116310889A (en) | Unmanned aerial vehicle environment perception data processing method, control terminal and storage medium | |
| CN114898321A (en) | Method, device, equipment, medium and system for detecting road travelable area | |
| CN113537154B (en) | Unmanned aerial vehicle building aerial photograph neural network automatic color correction method | |
| CN114820707A (en) | Calculation method for camera target automatic tracking | |
| CN114332814A (en) | Parking frame identification method and device, electronic equipment and storage medium | |
| CN110008918B (en) | Method for identifying posture of motorcycle simulator driver |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |