CN114937190A - Method and system for judging seed cotton opening effectiveness - Google Patents
Method and system for judging seed cotton opening effectiveness Download PDFInfo
- Publication number
- CN114937190A CN114937190A CN202210607648.XA CN202210607648A CN114937190A CN 114937190 A CN114937190 A CN 114937190A CN 202210607648 A CN202210607648 A CN 202210607648A CN 114937190 A CN114937190 A CN 114937190A
- Authority
- CN
- China
- Prior art keywords
- data
- sampling
- seed cotton
- opening
- depth
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01H—MEASUREMENT OF MECHANICAL VIBRATIONS OR ULTRASONIC, SONIC OR INFRASONIC WAVES
- G01H17/00—Measuring mechanical vibrations or ultrasonic, sonic or infrasonic waves, not provided for in the preceding groups
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Quality & Reliability (AREA)
- Treatment Of Fiber Materials (AREA)
- Preliminary Treatment Of Fibers (AREA)
Abstract
The invention discloses a method and a system for judging seed cotton opening effectiveness, which belong to the technical field of impurity sorting and deep learning. The invention aligns and fuses the line scanning image data and the line scanning 3D depth data, simultaneously collects the vibration data of the opener, improves the judgment accuracy, effectively reduces the performance requirement on computer hardware under the condition of the same sampling rate and sampling length, and is beneficial to reducing the operation and maintenance cost.
Description
Technical Field
The invention belongs to the technical field of impurity sorting and deep learning, and particularly relates to a method and a system for judging seed cotton opening effectiveness.
Background
The cotton is called seed cotton after being picked, the seed cotton is mixed with a large amount of impurities, and the foreign fiber impurity removal is needed before high-quality ginned cotton is formed. The opening of the seed cotton is an important link of an impurity removal production line, and if the opening effect is poor, the impurity removal rate and the carrying capacity of a sorting link are affected. At present, the opening link depends on non-feedback mechanical garnetting, and the opening quality changes along with different feeding speeds, seed cotton impurity contents and the like, so that the opening effect is insufficient, and the condition of subsequent sorting cannot be met. The conventional method does not need to judge the step of feeding seed cotton into opening and opening for many times, and the method increases time and energy and sets maintenance cost and has low efficiency, so that the effective judgment of seed cotton opening is a technical problem to be solved urgently for the seed cotton foreign fiber impurity removal industry.
Disclosure of Invention
The technical problem is as follows: aiming at the problems in the prior art, the invention aims to provide a method and a system for judging seed cotton opening validity to judge the seed cotton opening validity.
The technical scheme is as follows: in order to solve the technical problems, the technical scheme adopted by the invention is as follows:
a method for judging seed cotton opening effectiveness is characterized in that a linear array camera and a 3D depth camera are used for obtaining an image and three-dimensional information of opened seed cotton in a seed cotton foreign fiber impurity removal production line, line scanning image data and line scanning 3D depth data are aligned and fused, cotton conglomeration surface depth information is additionally obtained through the 3D depth data on the basis of image recognition, and opening effectiveness is judged by combining vibration data of an opener; the method specifically comprises the following steps:
step 1, determining the number h of pixels scanned by a linear array camera once, determining the number j of data scanned by a 3D depth camera once, and determining the sampling times t corresponding to the opening average duration k Determining the sampling times t corresponding to the time length from the discharge of the opener to the shooting position c ；
Step 2, determining sampling frequency and total sampling time, and determining a marking standard according to an impurity removal effect, namely marking the sampling of the corresponding time period as effective by using the one-hot code if the impurity removal effect meets the requirement, and otherwise marking the sampling as ineffective;
step 3, continuously sampling seed cotton RGB images subjected to one-time opening on a conveyor belt at a fixed sampling rate by using a linear array camera, and selecting latest w times of sampling data X (X) from continuously sampled data -w+1 ,X -w+2 ,…,X 0 ]，X 0 Representing the current sample, X being the sample image t times ago -t ；
Step 4, continuously sampling the 3D depth data of the seed cotton at the same position in the step 3 by using a 3D depth camera at the same sampling rate of the linear array camera, and selecting the 3D depth data Y (Y) of the latest w times of sampling from the continuously sampled data -w+1 ,Y -w+2 ,…,Y 0 ]，Y 0 Representing the current sample, and the sample data t times before is Y -t ；
Step 5, continuously sampling vibration data of the opener by using n vibration sensors arranged in the opener at the same sampling rate of the linear array camera, and selecting latest w times of sampling data from the continuously sampled data, wherein the expression is as follows:
in the formula (I), the compound is shown in the specification,
is sampled data t times earlier (in the formula, the
)，t 0 Counting the offset for a sample, Z -w+1-t0 Is (t) 0 + w-1) the next previous sample data,
is (t) 0 + w-2) the next previous sample data, w being the number of sample data;
Step 7, fusing X scaled And Y scaled Obtaining U, U being X scaled And Y scaled The fused matrix has the following formula:
in the formula, avgpool (1) For average pooling to a specified size in a second dimension, stack (0) For stacking in the first dimension, U belongs to R 4×min(h,j)×w ；
Step 8, extracting vibration characteristics by using an LSTM network, and converting Z scaled Input into LSTM network, t 0 The output corresponding to the next previous sample is recorded as
The status is recorded as
n is the dimensionality of a vector, and specifically refers to the number of vibration sensors installed in the opener;
step 9, extracting image data and 3D depth data by using a transform encoder structure, constructing a neural network jointly formed by encoder structures of 4 independent transform networks, inputting U as input, splitting according to a first dimension of U, respectively inputting the encoder structures of the 4 independent transform networks, adding position codes along a third dimension, and enabling a second dimension to correspond to word embedding length in the transform networks;
step 10, stacking the 4 vectors in the step 9 into a two-dimensional matrix U; ', U; ' stacking the resulting two-dimensional matrix;
step 11, fusing the output of a neural network formed by the encoder structures of the LSTM network in the step 8 and the 4 independent transform networks in the step 9 in an additive mode, and fusing U; ' already,
And
o is obtained, and O is U; ' already,
And
fusing the obtained matrices to respectively
And
repeating the stacking for 4 times to obtain H epsilon R 4×n And C ∈ R 4×n And fusing by applying the following formula:
W 1 ∈R 4×n
W 2 ∈R 4×n
in the formula, W 1 ，W 2 For the learnable weight parameter matrix, the sign "+" refers to the matrix addition, the sign
Are multiplied by the elements, H is
A matrix obtained by repeating the stacking 4 times, C is
Performing repeated stacking for 4 times to obtain a matrix, wherein R represents a real number tensor space, and n is the number of vibration sensors installed in the opener;
step 12, inputting the O in the step 11 into a multilayer feedforward neural network, and obtaining an opening validity judgment vector through softmax:
in the formula, e 1 For rendering ineffective the opening, e 2 For opening to be effective, T is the data sampling time period, T is the sampling time point, P is the probability, if P (e) 1 If T belongs to T) is more than 0.5, the opening is invalid, otherwise, the opening is valid.
Preferably, in step 7, the image data and the 3D depth data are fused in a matrix stacking manner to achieve the purpose of spatial and temporal alignment.
Preferably, according to step 9, the data obtained by fusing the multi-image data and the 3D depth data is divided by channels and used as the input of the independent and parallel transform encoder structure, and the dimension corresponding to the pixel or data size of a single sample is used as the word embedding length of the transform encoder structure.
Preferably, in step 6, the data of the line camera, the 3D depth camera and the vibration sensor are corrected, and the formula is as follows:
in the formula, μ is the mean value of each type of data, and σ is the standard deviation of each type of data.
Preferably, the encoder structure of the transform network in step 9 is formed by connecting a plurality of encoder units, each encoder unit includes a multi-head self-attention layer, a regularization layer, a splicing structure, a jump connection structure, and a plurality of parallel feedforward networks, a last encoder unit in the plurality of encoder units is connected to a full connection layer with n neurons, and n is the number of vibration sensors.
The invention also provides a judging system for seed cotton opening effectiveness, which comprises a vibration sensor arranged on the opener, a linear array camera arranged above the conveying belt, a 3D depth camera arranged above the conveying belt and a computer electrically connected with the vibration sensor, the linear array camera and the 3D depth camera, wherein the shooting direction of the linear array camera is perpendicular to the conveying belt and is used for shooting images of seed cotton on the conveying belt; the 3D depth camera is used for shooting a 3D depth image of seed cotton on the conveyor belt; the vibration sensor is used for collecting vibration data of the opener, and the computer processes the data according to the method for judging seed cotton opening effectiveness as claimed in claim 1 and according to the vibration data received from the vibration sensor, the seed cotton image data of the linear array camera and the seed cotton 3D depth data of the 3D depth camera, and judges the seed cotton opening effectiveness.
Preferably, the included angle between the shooting direction of the 3D depth camera and the conveyor belt is beta, and beta is more than or equal to 40 degrees and less than or equal to 60 degrees.
Has the advantages that: compared with the prior art, the invention has the following advantages: (1) the invention aligns and fuses the line scanning image data and the line scanning 3D depth data, and additionally obtains the cotton cluster surface depth information through the 3D depth data on the basis of image identification, so that the network can deduce by utilizing the color image, the surface fluctuation of the cotton cluster, the shape of the cotton fiber, the cotton cluster density and other information at the same position, and the judgment accuracy is improved.
(2) The invention aligns the time of the vibration sensor data and the cotton characteristic data after opening, reduces the time span between the data, thereby reducing the network complexity, effectively reducing the performance requirement on the computer hardware under the condition of the same sampling rate and sampling length, and being beneficial to reducing the operation and maintenance cost.
(3) The invention samples the opened cotton and collects the vibration data of the opener, so that the data input into the network has good reflecting capacity on the ineffectiveness of opening caused by uneven feeding speed, uneven indexes of the fed cotton and mechanical reasons of the opener, and the judgment accuracy is improved.
(4) The invention adopts different network structures to process cotton images and depth data after vibration and opening, utilizes the characteristic that LSTM is more sensitive to the time sequence data sequence to extract vibration characteristics, utilizes the characteristic that the structure of a transducer encoder has high parallelism and the characteristic that the extraction capability of the characteristics of long sequence data is higher to process linear array camera and linear array depth camera data which have large data volume, time and space attributes and need larger sequence length to reflect space information.
(5) The method adopts an additive mode to fuse the image and the depth characteristics of the cotton subjected to vibration and opening, utilizes the LSTM structure to make up the deficiency of the transform encoder structure in the causal sequence, and improves the discrimination accuracy.
Drawings
FIG. 1 is a schematic diagram of the system architecture of the present invention;
FIG. 2 is a diagram of an image and depth feature extraction network architecture;
FIG. 3 is a schematic flow diagram of the process of the present invention;
FIG. 4 is a block diagram of a multi-layer feedforward neural network.
Detailed Description
The present invention will be further illustrated by the following specific examples, which are carried out on the premise of the technical scheme of the present invention, and it should be understood that these examples are only for illustrating the present invention and are not intended to limit the scope of the present invention.
As shown in fig. 1, the embodiment provides a discrimination system for seed cotton opening effectiveness, which includes a vibration sensor 2, a line camera 4, a 3D depth camera 5 and a computer 6, wherein the vibration sensor 2 is an acceleration vibration sensor, 4 sensors are installed on the inner side of a housing of the opener 1, vibration data of the opener 1 are collected, the model of the acceleration vibration sensor is 4835A of TE Connectivity corporation, the sampling rate is 500Hz, and the interpolation is 1 kHz; the moving speed of the conveyor belt 1 is 0.5m/s, a shooting position 31 is arranged on the conveyor belt 12 meters away from the opener 1, the time from the opening position to the shooting position 31 is 4s, the line camera 4 is arranged right above the shooting position 31 on the conveyor belt 3, the shooting direction of the line camera 4 is perpendicular to the conveyor belt 3, the model of the line camera 4 is LA-GC-02K05B of Teledyne DALSA, and the line camera 4 is used for shooting images of seed cotton 9 on the conveyor belt 3; the 3D depth camera 5 is also arranged above the conveyor belt 3, is obliquely above the shooting position 31, forms an angle of 45 degrees with the conveyor belt 3, and is used for shooting a depth image of the seed cotton 9 from the direction of 45 degrees, the 3D depth camera 5 comprises a depth camera 51 and a laser light source 52, and the depth camera 51 and the laser light source 52 adopt Ranger 3V 3DR3-30NE31111 of the West Ke (SICK) company and a matched light source thereof; the line frequency of the two cameras, namely the linear array camera 4 and the depth camera 51, is set to be 1kHz, and the output is down-sampled to 384 pixels; the computer 6 is electrically connected with the vibration sensor 2, the linear array camera 4 and the 3D depth camera 5; and the computer 6 processes the data according to the vibration data of the vibration sensor 2, the seed cotton image data of the linear array camera 4 and the seed cotton 3D depth data of the 3D depth camera 5, and judges the seed cotton opening effectiveness.
As shown in fig. 2 and fig. 3, the present embodiment further provides a method for determining the seed cotton opening validity, which includes the following steps:
step 1, determining the number of pixels of single scanning to be h, h being 384, determining the number j of data of single scanning according to the performance of a 3D depth camera, j being 384, and determining the sampling times t corresponding to the opening average time length according to the performance of an opener k ，t k 3000, the sampling times corresponding to the time length from the discharge of the opener to the shooting position is determined to be t according to the performance of the conveying belt and the scanning position of the camera c ，t c ＝1000。
And 2, determining sampling frequency and total sampling time according to the performance of the computer, and determining a marking standard according to the impurity removal effect, namely marking the sampling of the corresponding time period as effective by using the one-hot code if the impurity removal effect meets the requirement, and otherwise marking the sampling as ineffective.
Step 3, continuously sampling the once-opened seed cotton RGB images on the conveyor belt at a fixed sampling rate by using a linear array camera, and selecting the nearest data in the continuously sampled data3000 times of sampling data, X ═ X -2999 ,X -2998 ,…,X 0 ]∈R 3×384×3000 。
Step 4, continuously sampling the depth information of the seed cotton at the same position in the step 3 by using a high-speed 3D depth camera, and selecting 3D depth data Y ═ Y [ Y ] of the latest 3000 times of sampling from the continuously sampled data -2999 ,Y -2998 ,…,Y 0 ]∈R 1×384×3000 。
Step 5, continuously sampling vibration data of the opener by using 4 vibration sensors arranged in the opener at the same sampling rate of the linear array camera, selecting latest w times of sampling data from the continuously sampled data,
due to t 0 2500, w 3000, so Z ═ Z -5499 ,Z -5498 ,…,Z -2500 ]∈R 1×4×3000 。
in the formula, mu is the mean value of various data, sigma is the standard deviation of various data, and X is obtained scaled 、Y scaled 、Z scaled 。
Step 7, fusing X scaled And Y scaled Obtaining U, U epsilon to R 4×min(h,j)×w (ii) a Namely, the data corrected by the linear array camera and the 3D depth camera are fused, and the formula is as follows:
in the formula, avgpool (1) For average pooling to a specified size in a second dimension, stack (0) In order to stack in the first dimension direction, h is j in this embodiment, so U is stack (0) (X stack ，Y stack )。
Step 8, extracting vibration characteristics by using LSTM network, and converting Z scaled Input LSTM network (Long-short term memory Artificial neural network), t of which 0 (t 0 2500) of the data before the sampling point is recorded as h -2500 The state is marked as c -2500 ；
Step 9, extracting image data and 3D depth data by using a transform encoder structure, as shown in fig. 2, U is used as an input of a 2-layer transform encoder, and is split according to a first dimension, and is respectively input into 4 independent transform network encoder structures, each transform network encoder structure is formed by connecting a plurality of encoder units, and one encoder unit is formed by a multi-head self-attention layer, a regularization layer, a splicing structure, a jump connection structure, and a plurality of parallel feedforward networks, and the difference between the transform encoder structure and the transform encoder structure is that a full connection layer with n neurons is added behind the encoder unit, n is the number of vibration sensors, and n is 4; and adding position codes along the third dimension, wherein the second dimension corresponds to the word embedding length in the transform network, and adding a 3-layer fully-connected network at the network output end.
Step 10, as shown in fig. 2, stacking the 4 vectors with the same output size and the same number of the vibration sensors in the step 9 into a two-dimensional matrix U; '.
Step 11, fusing U; ', h -2500 And c -2500 Obtaining O, O is epsilon to R 4×n ；
Is a fusion U; ', h -2500 And c -2500 Respectively mixing h with -2500 And c -2500 Repeating the stacking for 4 times to obtain H epsilon R 4×n And C ∈ R 4 ×n And applications ofThe following formula is fused:
W 1 ∈R 4×n
W 2 ∈R 4×n
in the formula, W 1 ，W 2 For the learnable weight parameter matrix, the sign "+" refers to the matrix addition, the sign
It means multiplication by elements. The calculation is performed according to the above formula, and the obtained matrix O has 4 rows and n columns, and thus belongs to the real number tensor space R 4×n Is recorded as O ∈ R 4×n 。
Step 12, inputting the value O in the step 11 into a multilayer feedforward neural network shown in the figure 4, and obtaining an opening effectiveness judgment vector through softmax:
in the formula, e 1 For rendering ineffective the opening, e 2 For opening to be effective, T is the data sampling period, if P (e) 1 If T belongs to T) is more than 0.5, the opening is invalid, otherwise, the opening is valid; where 0.5 is a fixed value, this value is used for the reason: 1. according to the existing practice, softmax used in the neural network output layer of the classification task usually takes 0.5 as a discrimination threshold; 2. the neural network is expected to output a value of 0-1 to characterize the probability, so that the value of 0.5 is beneficial to learning the expectation of the neural network in the network training process.
The computer 6 of this embodiment is connected with the control system of conveyer belt 3, and when computer 6 judged opening nature effectively through the internal processing procedure, next edulcoration link was carried with the seed cotton to computer 6 control conveyer belt 3, and when judging opening nature inefficacy, computer 6 control conveyer belt 3 continued opening with seed cotton conveying back opener 1, improved the operating efficiency through the effective judgement to opening nature, reduced the time of opening, reduced the equipment maintenance cost, the energy saving.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be construed as the protection scope of the present invention.
Claims (7)
1. A distinguishing method for seed cotton opening effectiveness is characterized in that a linear array camera and a 3D depth camera are used for obtaining an image and three-dimensional information of opened seed cotton in a seed cotton foreign fiber impurity removal production line, line scanning image data and line scanning 3D depth data are aligned and fused, cotton conglomerate surface depth information is additionally obtained through the 3D depth data on the basis of image recognition, and opening effectiveness is judged in combination with opener vibration data; the method specifically comprises the following steps:
step 1, determining the number h of pixels scanned by a linear array camera once, determining the number j of data scanned by a 3D depth camera once, and determining the sampling times t corresponding to the opening average time length k Determining the sampling times t corresponding to the time length from the discharge of the opener to the shooting position c ；
Step 2, determining sampling frequency and total sampling time, and determining a marking standard according to an impurity removal effect, namely marking the sampling of the corresponding time period as effective by using a single hot code if the impurity removal effect meets the requirement, otherwise marking the sampling as ineffective;
step 3, continuously sampling seed cotton RGB images subjected to one-time opening on a conveyor belt at a fixed sampling rate by using a linear array camera, and selecting latest w times of sampling data X (X) from continuously sampled data -w+1 ,X -w+2 ,…,X 0 ]，X 0 Representing the current sample, X being the sample image t times ago -t ；
Step 4, continuously sampling the 3D depth data of the seed cotton at the same position in the step 3 by using a 3D depth camera at the same sampling rate of the linear array camera, and selecting the 3D depth data Y (Y) of the latest w times of sampling from the continuously sampled data -w+1 ,Y -w+2 ,…,Y 0 ]，Y 0 Representing the current sampleAnd the sampled data t times before is Y -t ；
Step 5, continuously sampling vibration data of the opener by using n vibration sensors arranged in the opener at the same sampling rate of the linear array camera, and selecting latest w times of sampling data from the continuously sampled data, wherein the expression is as follows:
in the formula (I), the compound is shown in the specification,
is sampled data t times earlier (in the formula, the
)，t 0 Counting the offset for a sample, Z -w+1-t0 Is (t) 0 + w-1) the next previous sample data,
is (t) 0 + w-2) the next previous sample data, w being the number of sample data;
step 6, X, Y, Z is corrected to obtain X scaled 、Y scaled 、Z scaled ；
Step 7, fusing X scaled And Y scaled Obtaining U, U being X scaled And Y scaled The fused matrix has the following formula:
in the formula, avgpool (1) For average pooling to a specified size in a second dimension, stack (0) To be in first dimensionStacking in the direction of degree, U is formed by R 4×min(h,j)×w ；
Step 8, extracting vibration characteristics by using an LSTM network, and converting Z scaled Input into LSTM network, t 0 The output corresponding to the next previous sample is recorded as
The status is recorded as
n is the dimensionality of a vector, and specifically refers to the number of vibration sensors installed in the opener;
step 9, extracting image data and 3D depth data by using a transform encoder structure, constructing a neural network jointly formed by encoder structures of 4 independent transform networks, inputting U as input, splitting according to a first dimension of U, respectively inputting the encoder structures of the 4 independent transform networks, adding position codes along a third dimension, and enabling a second dimension to correspond to word embedding length in the transform networks;
step 10, stacking the 4 vectors in the step 9 into a two-dimensional matrix U'; u'; namely stacking the two-dimensional matrix;
step 11, fusing the output of a neural network formed by encoder structures of the LSTM network in the step 8 and the 4 independent transform networks in the step 9 in an additive mode, and fusing U'; a
And
obtaining O, wherein O is U'; a (c)
And
fusing the resulting matrices, respectively
And
repeating the stacking for 4 times to obtain H epsilon R 4×n And C ∈ R 4×n And fusing by applying the following formula:
W 1 ∈R 4×n
W 2 ∈R 4×n
in the formula, W 1 ，W 2 For the learnable weight parameter matrix, the sign "+" refers to the matrix addition, the sign
Are multiplied by the elements, H is
A matrix obtained by repeating the stacking 4 times, C is
Performing repeated stacking for 4 times to obtain a matrix, wherein R represents a real number tensor space, and n is the number of vibration sensors installed in the opener;
step 12, inputting the O in the step 11 into a multilayer feedforward neural network, and obtaining an opening validity judgment vector through softmax:
in the formula, e 1 For opening to be ineffective e 2 For opening to be effective, T is the data sampling time period, T is the time point of sampling, P is the probability, if P (e) 1 If T belongs to T) is more than 0.5, the opening is invalid, otherwise, the opening is valid.
2. The method for judging seed cotton opening validity according to claim 1, wherein the image data and the 3D depth data in the step 7 are fused in a matrix stacking mode for achieving the purpose of spatial and temporal alignment.
3. The method for judging seed cotton opening validity according to claim 2, wherein in step 9, the fused data of the multi-image data and the 3D depth data is divided by channels to be used as the input of an independent parallel transform encoder structure, and the dimension corresponding to the pixel or data volume of single sampling is used as the word embedding length of the transform encoder structure.
4. The method for judging the effectiveness of opening seed cotton according to claim 1, wherein in the step 6, the data of the linear array camera, the 3D depth camera and the vibration sensor are corrected according to the following formula:
in the formula, μ is the mean value of each type of data, and σ is the standard deviation of each type of data.
5. The method for judging seed cotton opening validity according to claim 1, wherein the encoder structure of the transform network in the step 9 is formed by connecting a plurality of encoder units, each encoder unit comprises a multi-head self-attention layer, a regularization layer, a splicing structure, a jump connection structure and a plurality of parallel feedforward networks, the last encoder unit in the plurality of encoder units is connected with a full connection layer with n neurons, and n is the number of vibration sensors.
6. The seed cotton opening effectiveness judging system is characterized by comprising a vibration sensor (2) arranged on an opener (1), a linear array camera (4) arranged above a conveying belt (3), a 3D depth camera (5) arranged above the conveying belt (3) and a computer (6) electrically connected with the vibration sensor (2), the linear array camera (4) and the 3D depth camera (5), wherein the shooting direction of the linear array camera (4) is perpendicular to the conveying belt (3) and is used for shooting an image of seed cotton on the conveying belt (3); the 3D depth camera (5) is used for shooting a 3D depth image of seed cotton on the conveyor belt (3); the vibration sensor (2) is used for collecting vibration data of the opener (1), and the computer (6) processes the data according to the method for judging seed cotton opening effectiveness of claim 1 and according to the vibration data received from the vibration sensor (2), the seed cotton image data of the line camera (4) and the seed cotton 3D depth data received from the 3D depth camera (5), so as to judge the seed cotton opening effectiveness.
7. The system for discriminating the opening validity of seed cotton according to claim 6, wherein an included angle between the photographing direction of the 3D depth camera (5) and the conveyor belt (3) is β, and β is more than or equal to 40 ° and less than or equal to 60 °.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210607648.XA CN114937190B (en) | 2022-05-31 | 2022-05-31 | Method and system for judging seed cotton opening effectiveness |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210607648.XA CN114937190B (en) | 2022-05-31 | 2022-05-31 | Method and system for judging seed cotton opening effectiveness |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114937190A true CN114937190A (en) | 2022-08-23 |
| CN114937190B CN114937190B (en) | 2023-05-16 |
Family
ID=82866782
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210607648.XA Active CN114937190B (en) | 2022-05-31 | 2022-05-31 | Method and system for judging seed cotton opening effectiveness |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114937190B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150197062A1 (en) * | 2014-01-12 | 2015-07-16 | Zohar SHINAR | Method, device, and system of three-dimensional printing |
| CN108284079A (en) * | 2018-02-13 | 2018-07-17 | 南京林业大学 | A kind of unginned cotton residual film sorting unit and algorithm based on high light spectrum image-forming and deep learning |
| CN108458989A (en) * | 2018-04-28 | 2018-08-28 | 江苏建筑职业技术学院 | A kind of Coal-rock identification method based on Terahertz multi-parameter spectrum |
| US20190122378A1 (en) * | 2017-04-17 | 2019-04-25 | The United States Of America, As Represented By The Secretary Of The Navy | Apparatuses and methods for machine vision systems including creation of a point cloud model and/or three dimensional model based on multiple images from different perspectives and combination of depth cues from camera motion and defocus with various applications including navigation systems, and pattern matching systems as well as estimating relative blur between images for use in depth from defocus or autofocusing applications |
| CN110222784A (en) * | 2019-06-13 | 2019-09-10 | 郑州轻工业学院 | Fusion in short-term with it is long when depth characteristic solar battery sheet defect inspection method |
| CN111476767A (en) * | 2020-04-02 | 2020-07-31 | 南昌工程学院 | High-speed rail fastener defect identification method based on heterogeneous image fusion |
| CN112461848A (en) * | 2020-12-16 | 2021-03-09 | 石河子大学 | Laser-enhanced seed cotton residual film online detection system |
| CN113297972A (en) * | 2021-05-25 | 2021-08-24 | 国网湖北省电力有限公司检修公司 | Transformer substation equipment defect intelligent analysis method based on data fusion deep learning |
-
2022
- 2022-05-31 CN CN202210607648.XA patent/CN114937190B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150197062A1 (en) * | 2014-01-12 | 2015-07-16 | Zohar SHINAR | Method, device, and system of three-dimensional printing |
| US20190122378A1 (en) * | 2017-04-17 | 2019-04-25 | The United States Of America, As Represented By The Secretary Of The Navy | Apparatuses and methods for machine vision systems including creation of a point cloud model and/or three dimensional model based on multiple images from different perspectives and combination of depth cues from camera motion and defocus with various applications including navigation systems, and pattern matching systems as well as estimating relative blur between images for use in depth from defocus or autofocusing applications |
| CN108284079A (en) * | 2018-02-13 | 2018-07-17 | 南京林业大学 | A kind of unginned cotton residual film sorting unit and algorithm based on high light spectrum image-forming and deep learning |
| CN108458989A (en) * | 2018-04-28 | 2018-08-28 | 江苏建筑职业技术学院 | A kind of Coal-rock identification method based on Terahertz multi-parameter spectrum |
| CN110222784A (en) * | 2019-06-13 | 2019-09-10 | 郑州轻工业学院 | Fusion in short-term with it is long when depth characteristic solar battery sheet defect inspection method |
| CN111476767A (en) * | 2020-04-02 | 2020-07-31 | 南昌工程学院 | High-speed rail fastener defect identification method based on heterogeneous image fusion |
| CN112461848A (en) * | 2020-12-16 | 2021-03-09 | 石河子大学 | Laser-enhanced seed cotton residual film online detection system |
| CN113297972A (en) * | 2021-05-25 | 2021-08-24 | 国网湖北省电力有限公司检修公司 | Transformer substation equipment defect intelligent analysis method based on data fusion deep learning |
Non-Patent Citations (3)
| Title |
|---|
| CHENGLIANG ZHANG等: "Detection of Impurity Rate of Machine-Picked Cotton Based on Improved Canny Operator" * |
| 倪超等: "基于短波近红外高光谱和深度学习的籽棉地膜分选算法" * |
| 高雅文;王忆如;段峥;宋金丰;张冬;张云;倪超;: "基于多信息融合的籽棉异纤识别分拣系统设计与实现" * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114937190B (en) | 2023-05-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108171184B (en) | Method for re-identifying pedestrians based on Simese network | |
| Xu et al. | Towards end-to-end license plate detection and recognition: A large dataset and baseline | |
| CN110569814B (en) | Video category identification method, device, computer equipment and computer storage medium | |
| CN112037269B (en) | Visual moving target tracking method based on multi-domain collaborative feature expression | |
| CN109583506B (en) | Unsupervised image identification method based on parameter transfer learning | |
| CN108491856B (en) | Image scene classification method based on multi-scale feature convolutional neural network | |
| CN109858467B (en) | Face recognition method and device based on key point region feature fusion | |
| CN111507275B (en) | Video data time sequence information extraction method and device based on deep learning | |
| CN111488805A (en) | Video behavior identification method based on saliency feature extraction | |
| CN111639580A (en) | Gait recognition method combining feature separation model and visual angle conversion model | |
| Zubić et al. | From chaos comes order: Ordering event representations for object recognition and detection | |
| CN116958786A (en) | Dynamic visual identification method for chemical waste residues based on YOLOv5 and ResNet50 neural network | |
| WO2024010120A1 (en) | Waste classification system based on vision-hyperspectral fusion data | |
| CN115861210A (en) | Transformer substation equipment abnormity detection method and system based on twin network | |
| CN114596273B (en) | Intelligent detection method for multiple defects of ceramic substrate by using YOLOV4 network | |
| CN116094792A (en) | Encryption malicious stream identification method and device based on space-time characteristics and attention mechanism | |
| Rasyid et al. | Digital Image Processing for Detecting Industrial Machine Work Failure with Quantization Vector Learning Method | |
| WO2023065472A1 (en) | Efficient cross-camera target re-recognition method based on similarity | |
| CN114937190A (en) | Method and system for judging seed cotton opening effectiveness | |
| CN114926900A (en) | Human body action online detection method with separated foreground and background | |
| CN214608445U (en) | Cigarette packet appearance quality detection device and cigarette packet packaging machine | |
| CN114581769A (en) | Method for identifying houses under construction based on unsupervised clustering | |
| CN114723953A (en) | Deep neural network for image source detection | |
| CN114283319A (en) | Locomotive wheel set tread stripping identification method | |
| Sugandi et al. | Face recognition based on pca and neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |