CN110276371A - A kind of container angle recognition methods based on deep learning - Google Patents
A kind of container angle recognition methods based on deep learning Download PDFInfo
- Publication number
- CN110276371A CN110276371A CN201910367932.2A CN201910367932A CN110276371A CN 110276371 A CN110276371 A CN 110276371A CN 201910367932 A CN201910367932 A CN 201910367932A CN 110276371 A CN110276371 A CN 110276371A
- Authority
- CN
- China
- Prior art keywords
- plane
- container
- corner fittings
- coordinate
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
Abstract
The container angle recognition methods based on deep learning that the present invention relates to a kind of.It needs that container angle is fixed using fork truck in actual industrial environment, needs to obtain the spatial positional information of corner fittings first.The system deployment that color camera is combined with depth camera is convenient and efficient, and precision is high, so it is most extensive to carry out corner fittings detection using visual information.It carries out Container inspection system using traditional vision algorithm to be difficult to extract effective feature, precision is lower, and speed is slower.The present invention carries out feature extraction by YOLO neural network, a kind of method for carrying out container angle detection using neural network and Feature Correspondence Algorithm is proposed, the deviation angle of container angle coordinate and container angle plane and camera plane can be obtained under different workplaces.
Description
Technical field
The invention belongs to computer vision fields, and in particular to a kind of container angle identification side based on deep learning
Method.
Background technique
The detection of target object is carried out by visual information, obtaining position of the target object in picture is computer vision
An important topic.In the identification of container angle, primary is exactly the position for identifying container in picture, so
Coordinate further according to container in space afterwards judges the deviation angle of container front side and camera plane, and on this basis
The coordinate of container angle in space is judged, so the accuracy of target detection has significant effect subsequent step.Currently,
The method of target detection mainly has: 1) based on the target detection of conventional method；2) based on the object detection method of study.
Traditional object detection method is all rapidly to carry out feature under the premise of guaranteeing abundant extraction, accurate feature
It calculates and predicts.But the feature of traditional method for extracting is essentially all the feature of low level, artificial selection.These features are opposite
It is more intuitive, it is easy by human intelligible, but a large amount of, multi-class targets cannot be expressed well, and accuracy rate is lower.
Object detection method based on study refers to using deep learning, from the high level of data focusing study target object
Feature, these features have better ability to express.These methods are usually the information extracted in whole picture, are not merely used
A part of feature of object, so the object detection method based on study not only makes the essence of target detection relative to conventional method
Degree and speed are all greatly improved, and have better noiseproof feature, that is, robustness is stronger.
Summary of the invention
The container angle recognition methods based on deep learning that in view of the deficiencies of the prior art, the present invention proposes a kind of.
The present invention combine with color camera under scene high-resolution color picture and the collected depth map of depth camera,
Intensity map, using YOLO-v3 neural network and Principal Component Analysis detect container angle in coordinate in the real world, with
And the angle of plane where plane where corner fittings and camera.It comprises the concrete steps that:
Step (1) is denoted as C using the high-resolution colour picture that color camera obtains a container region, uses
The depth map that depth camera obtains in container region is denoted as D, intensity map is denoted as I；
Step (2) uses the container and corner fittings in YOLO-v3 neural network sense colors picture；It is specific as follows:
1. color image is inputted in trained YOLO-v3 neural network；Neural network can automatically to input picture into
Then row feature extraction is predicted coordinate of the target object in color image using the feature extracted, is finally passed through
Multiple possible coordinates are screened, the maximum coordinate of possibility is exported, is accurate to pixel；The wherein detection mesh of YOLO-v3
It is designated as container and corner fittings, can finally export three as a result, being container and two corner fittings respectively；
2. judging whether network output is effective；Judge by the way that whether the coordinate to two corner fittings is located in container coordinate
The correctness of web results is only correct just to enter in next step；Concrete mode is as follows:
Wherein, flag indicates whether result is effective, xbox_min,xbox_max,ybox_min,ybox_maxIt is container respectively in coloured silk
Transverse and longitudinal coordinate range in chromatic graph, i, j are the transverse and longitudinal coordinate of corner fittings respectively；
Container region in color image is converted to grayscale image by step (3), is denoted as G；It is detected using SURF operator
Key point in grayscale image G and intensity map I, and both the description operator of two picture of FLANN algorithmic match is used, and is calculated pass
The distance between key point；Threshold value is selected according to the codomain of distance, matched key point is then used as less than threshold value；To matched key
Point carries out perspective transform, obtains transformation matrix H, and outline all match points in intensity map, depth is regarded as in this region
Spend camera in container coordinate, and using be located at G in corner fittings position key point in I corresponding key point as the seat of corner fittings
Mark；
Position of the step (4) according to container in depth map, extracts the three-dimensional coordinate data of container plane；Because
Plane where the depth areas of the container of previous step not just corner fittings, it is therefore desirable to container angle plane location
It is screened in domain；Specific screening step is as follows:
If the depth map coordinate of two corner fittings is respectively (x1,y1) and (x2,y2), then point (m, the n) Ying Fuhe in region with
Lower condition:
The point for meeting conditions above is modeled, to obtain the angle of container angle plane and camera plane；Specifically
Steps are as follows:
Areal model is established to plane:
Ax+By+Cz+D=0
Wherein A, B, C, D are the parameter of plane, and x, y, z is the coordinate value of Plane-point；
According to the confidence level figure of depth camera, the corresponding confidence level of each three-dimensional coordinate is compared with threshold value, when setting
Reliability reaches requirement, is just included in calculating point and concentrated, on the contrary then abandon；
The parameter of plane normal vector is solved using Principal Component Analysis Algorithm, calculation method is as follows:
(a) covariance matrix of all the points in plane is solved；Formula are as follows:
Σ=E (hhT)-E(h)E(hhT)
(b) feature vector and characteristic value of covariance matrix are solved, wherein the corresponding feature vector of minimal eigenvalue is exactly
The corresponding normal vector of plane, that is, parameter A, B, C in plane equation；
(c) all valid data in the three-dimensional coordinate data of container plane are substituted into equation, multiple D values is acquired, to D
Value is averaging, and obtains an accurate D；
(d) corner fittings plane is being obtained after the equation of depth camera coordinate system, using the method in solid geometry, calculating
The angle of vertical plane and corner fittings plane where depth camera out；It is as follows using formula:
Wherein, angle of the θ between two planes,WithRespectively normal vector and corner fittings of the camera perpendicular to floor
The normal vector of plane；
Utilize step (1)-(4), the three-dimensional coordinate of the angle and corner fittings of plane and camera plane where obtaining corner fittings.
Beneficial effects of the present invention: the powerful capability of fitting of neural network is utilized in the method for the present invention, detects object
The specific location of body in the picture, then using the color image of target object as sample, to find the correspondence of object in depth map
Position avoids depth camera and the cumbersome conversion process of color camera, and obtained container angle position is accurate, plane included angle
Estimation is accurate.
Specific implementation step
Step (1) obtains a high-resolution colour picture using color camera and is denoted as C, obtains packaging using depth camera
Depth map in case region is denoted as D, intensity map is denoted as I.
Step (2) uses the container and corner fittings in YOLO-v3 neural network sense colors picture.It is specific as follows:
1. color image is inputted in trained YOLO neural network.Neural network can automatically carry out input picture special
Sign is extracted, and is then predicted using the feature extracted coordinate of the target object in color image, finally by more
A possible coordinate is screened, and is exported the maximum coordinate of possibility, is accurate to pixel.In the present invention, the detection target of YOLO
For container and corner fittings, three results (being container and two corner fittings respectively) can be finally exported.
2. judging whether network output is effective.Judge by the way that whether the coordinate to two corner fittings is located in container coordinate
The correctness of web results is only correct just to enter in next step.Concrete mode is as follows:
Wherein, flag indicates whether result is effective, xbox_min,xbox_max,ybox_min,ybox_maxIt is container respectively in coloured silk
Transverse and longitudinal coordinate range in chromatic graph, i, j are the transverse and longitudinal coordinate of corner fittings respectively.
Container region in color image is converted to grayscale image by step (3), is denoted as G.It is detected using SURF operator
Key point in grayscale image G and intensity map I, and both the description operator of two picture of FLANN algorithmic match is used, and is calculated pass
The distance between key point.Suitable threshold value is selected according to the codomain of distance, threshold value is used to screen matched key point.To key point
Perspective transform is carried out, transformation matrix H, and all match points in intensity map center is obtained, depth is regarded as into this region
The coordinate of container in camera, and using be located at G in corner fittings position key point in I corresponding key point as the seat of corner fittings
Mark.
Position of the step (4) according to container in depth map extracts the three-dimensional coordinate data (point of container plane
Cloud).Because of plane where the depth areas of the container of previous step not just corner fittings, it is therefore desirable to flat to container angle
It is screened face region.Specific screening step is as follows:
If the depth map coordinate of two corner fittings is respectively (x1,y1) and (x2,y2), then point (m, the n) Ying Fuhe in region with
Lower condition:
The point for meeting conditions above is modeled, to obtain the angle of container angle plane and camera plane.Specifically
Steps are as follows:
Areal model is established to plane:
Ax+By+Cz+D=0
Wherein A, B, C, D are the parameter of plane, and x, y, z is the coordinate value of Plane-point.
The parameter of plane normal vector is solved using Principal Component Analysis Algorithm, calculation method is as follows:
(a) covariance matrix of all the points in plane is solved.Formula are as follows:
Σ=E (xxT)-E(x)E(xT)
(b) feature vector and characteristic value of covariance matrix are solved, wherein the corresponding feature vector of minimal eigenvalue is exactly
The corresponding normal vector of plane, that is, parameter A, B, C in plane equation.
(c) by cloud all valid data substitute into equation, acquire multiple D values, to D value be averaging, obtain one compared with
For accurate D.
By Principal Component Analysis Algorithm, an available accurate plane equation, still, because depth camera is adopted
There are much noises for the data collected, to plane equation accuracy there are larger impact, need to pre-process initial data.
Method particularly includes:
According to the confidence level figure of depth camera, the corresponding confidence level of each three-dimensional coordinate is compared with threshold value, when setting
Reliability reaches requirement, is just included in calculating point and concentrated, on the contrary then abandon.
(d) plane is being obtained after the equation of depth camera coordinate system, using the method in solid geometry, calculate depth
The angle of vertical plane and corner fittings plane where spending camera.It is as follows using formula:
Wherein, angle of the θ between two planes,WithRespectively normal vector and corner fittings of the camera perpendicular to floor
The normal vector of plane.
Utilize above 4 steps, the three-dimensional coordinate of the angle and corner fittings of plane and camera plane where corner fittings can be obtained.
Claims (1)
1. a kind of container angle recognition methods based on deep learning, which is characterized in that this method comprises the concrete steps that:
Step (1) is denoted as C using the high-resolution colour picture that color camera obtains a container region, uses depth
The depth map that camera obtains in container region is denoted as D, intensity map is denoted as I；
Step (2) uses the container and corner fittings in YOLO-v3 neural network sense colors picture；It is specific as follows:
1. color image is inputted in trained YOLO-v3 neural network；Neural network can automatically carry out input picture special
Sign is extracted, and is then predicted using the feature extracted coordinate of the target object in color image, finally by more
A possible coordinate is screened, and is exported the maximum coordinate of possibility, is accurate to pixel；Wherein the detection target of YOLO-v3 is
Container and corner fittings can finally export three as a result, being container and two corner fittings respectively；
2. judging whether network output is effective；Network is judged by the way that whether the coordinate to two corner fittings is located in container coordinate
As a result correctness is only correct just to enter in next step；Concrete mode is as follows:
Wherein, flag indicates whether result is effective, xbox_min,xbox_max,ybox_min,ybox_maxIt is container respectively in cromogram
Transverse and longitudinal coordinate range, i, j are the transverse and longitudinal coordinate of corner fittings respectively；
Container region in color image is converted to grayscale image by step (3), is denoted as G；Gray scale is detected using SURF operator
Scheme the key point in G and intensity map I, and use the description operator of two picture of FLANN algorithmic match, and calculates the two key point
The distance between；Threshold value is selected according to the codomain of distance, matched key point is then used as less than threshold value；To matched crucial click-through
Row perspective transform obtains transformation matrix H, and all match points are outlined in intensity map, and depth phase is regarded as in this region
The coordinate of container in machine, and using be located at G in corner fittings position key point in I corresponding key point as the coordinate of corner fittings；
Position of the step (4) according to container in depth map, extracts the three-dimensional coordinate data of container plane；Because upper one
The depth areas of the container of step plane not just where corner fittings, it is therefore desirable to container angle plane region into
Row screening；Specific screening step is as follows:
If the depth map coordinate of two corner fittings is respectively (x1,y1) and (x2,y2), then the point (m, n) in region should meet following item
Part:
The point for meeting conditions above is modeled, to obtain the angle of container angle plane and camera plane；Specific steps
It is as follows:
Areal model is established to plane:
Ax+By+Cz+D=0
Wherein A, B, C, D are the parameter of plane, and x, y, z is the coordinate value of Plane-point；
According to the confidence level figure of depth camera, the corresponding confidence level of each three-dimensional coordinate is compared with threshold value, works as confidence level
Reach requirement, is just included in calculating point and concentrated, it is on the contrary then abandon；
The parameter of plane normal vector is solved using Principal Component Analysis Algorithm, calculation method is as follows:
(a) covariance matrix of all the points in plane is solved；Formula are as follows:
Σ=E (hhT)-E(h)E(hhT)
(b) feature vector and characteristic value of covariance matrix are solved, wherein the corresponding feature vector of minimal eigenvalue is exactly plane
Corresponding normal vector, that is, parameter A, B, C in plane equation；
(c) all valid data in the three-dimensional coordinate data of container plane are substituted into equation, acquires multiple D values, D value is asked
It is average, obtain an accurate D；
(d) corner fittings plane is being obtained after the equation of depth camera coordinate system, using the method in solid geometry, calculate depth
The angle of vertical plane and corner fittings plane where spending camera；It is as follows using formula:
Wherein, angle of the θ between two planes,WithRespectively normal vector and corner fittings plane of the camera perpendicular to floor
Normal vector；
Utilize step (1)-(4), the three-dimensional coordinate of the angle and corner fittings of plane and camera plane where obtaining corner fittings.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910367932.2A CN110276371B (en) | 2019-05-05 | 2019-05-05 | Container corner fitting identification method based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910367932.2A CN110276371B (en) | 2019-05-05 | 2019-05-05 | Container corner fitting identification method based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110276371A true CN110276371A (en) | 2019-09-24 |
| CN110276371B CN110276371B (en) | 2021-05-07 |
Family
ID=67959670
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910367932.2A Active CN110276371B (en) | 2019-05-05 | 2019-05-05 | Container corner fitting identification method based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110276371B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111292261A (en) * | 2020-01-17 | 2020-06-16 | 杭州电子科技大学 | Container detection and locking method based on multi-sensor fusion |
| CN113408321A (en) * | 2020-03-16 | 2021-09-17 | 中国人民解放军战略支援部队信息工程大学 | Real-time target detection method and device for lightweight image and video data |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106949848A (en) * | 2017-03-15 | 2017-07-14 | 深圳市深视智能科技有限公司 | A kind of high-precision laser 3D profiles phone structural detection method |
| US20170214901A1 (en) * | 2016-01-22 | 2017-07-27 | Samsung Electronics Co., Ltd. | Method and apparatus for obtaining depth image by using time-of-flight sensor |
| CN107680092A (en) * | 2017-10-12 | 2018-02-09 | 中科视拓（北京）科技有限公司 | A kind of detection of container lock and method for early warning based on deep learning |
| CN107687953A (en) * | 2017-09-12 | 2018-02-13 | 哈尔滨市科佳通用机电股份有限公司 | A kind of lorry failure automatic checkout equipment |
| CN107832767A (en) * | 2017-11-14 | 2018-03-23 | 深圳码隆科技有限公司 | Container number identification method, device and electronic equipment |
| CN108564620A (en) * | 2018-03-27 | 2018-09-21 | 中国人民解放军国防科技大学 | Scene depth estimation method for light field array camera |
| CN109165541A (en) * | 2018-05-30 | 2019-01-08 | 北京飞鸿云际科技有限公司 | Coding method for vehicle component in intelligent recognition rail traffic vehicles image |
-
2019
- 2019-05-05 CN CN201910367932.2A patent/CN110276371B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170214901A1 (en) * | 2016-01-22 | 2017-07-27 | Samsung Electronics Co., Ltd. | Method and apparatus for obtaining depth image by using time-of-flight sensor |
| CN106949848A (en) * | 2017-03-15 | 2017-07-14 | 深圳市深视智能科技有限公司 | A kind of high-precision laser 3D profiles phone structural detection method |
| CN107687953A (en) * | 2017-09-12 | 2018-02-13 | 哈尔滨市科佳通用机电股份有限公司 | A kind of lorry failure automatic checkout equipment |
| CN107680092A (en) * | 2017-10-12 | 2018-02-09 | 中科视拓（北京）科技有限公司 | A kind of detection of container lock and method for early warning based on deep learning |
| CN107832767A (en) * | 2017-11-14 | 2018-03-23 | 深圳码隆科技有限公司 | Container number identification method, device and electronic equipment |
| CN108564620A (en) * | 2018-03-27 | 2018-09-21 | 中国人民解放军国防科技大学 | Scene depth estimation method for light field array camera |
| CN109165541A (en) * | 2018-05-30 | 2019-01-08 | 北京飞鸿云际科技有限公司 | Coding method for vehicle component in intelligent recognition rail traffic vehicles image |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111292261A (en) * | 2020-01-17 | 2020-06-16 | 杭州电子科技大学 | Container detection and locking method based on multi-sensor fusion |
| CN113408321A (en) * | 2020-03-16 | 2021-09-17 | 中国人民解放军战略支援部队信息工程大学 | Real-time target detection method and device for lightweight image and video data |
| CN113408321B (en) * | 2020-03-16 | 2023-08-22 | 中国人民解放军战略支援部队信息工程大学 | Real-time target detection method and device for lightweight image and video data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110276371B (en) | 2021-05-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102657532B (en) | Height measuring method and device based on body posture identification | |
| CN110175576A (en) | A kind of driving vehicle visible detection method of combination laser point cloud data | |
| CN107767400B (en) | Remote sensing image sequence moving target detection method based on hierarchical significance analysis | |
| CN106023257B (en) | A kind of method for tracking target based on rotor wing unmanned aerial vehicle platform | |
| CN105005999B (en) | It is a kind of based on obstacle detection method of the computer stereo vision towards apparatus for guiding blind | |
| CN109949361A (en) | A kind of rotor wing unmanned aerial vehicle Attitude estimation method based on monocular vision positioning | |
| CN109460709A (en) | The method of RTG dysopia analyte detection based on the fusion of RGB and D information | |
| CN105225230A (en) | A kind of method and device identifying foreground target object | |
| CN104200200B (en) | Fusion depth information and half-tone information realize the system and method for Gait Recognition | |
| CN102567994B (en) | Infrared small target detection method based on angular point gaussian characteristic analysis | |
| CN110334678A (en) | A kind of pedestrian detection method of view-based access control model fusion | |
| CN108346160A (en) | The multiple mobile object tracking combined based on disparity map Background difference and Meanshift | |
| CN106682641A (en) | Pedestrian identification method based on image with FHOG- LBPH feature | |
| CN104933738A (en) | Visual saliency map generation method based on local structure detection and contrast | |
| CN104102909B (en) | Vehicle characteristics positioning and matching process based on lenticular information | |
| CN104700088B (en) | A kind of gesture track recognition method under the follow shot based on monocular vision | |
| CN104143077B (en) | Pedestrian target search method and system based on image | |
| CN110006444B (en) | Anti-interference visual odometer construction method based on optimized Gaussian mixture model | |
| CN106204594A (en) | A kind of direction detection method of dispersivity moving object based on video image | |
| KR101460313B1 (en) | Apparatus and method for robot localization using visual feature and geometric constraints | |
| CN104200199A (en) | TOF (Time of Flight) camera based bad driving behavior detection method | |
| CN110020627A (en) | A kind of pedestrian detection method based on depth map and Fusion Features | |
| CN110276371A (en) | A kind of container angle recognition methods based on deep learning | |
| Tian et al. | Human Detection using HOG Features of Head and Shoulder Based on Depth Map. | |
| CN110147748A (en) | A kind of mobile robot obstacle recognition method based on road-edge detection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |