CN101169829A - Automated detection method for improving template comparison of electronic product assembly line - Google Patents

Automated detection method for improving template comparison of electronic product assembly line Download PDF

Info

Publication number
CN101169829A
CN101169829A CNA2006101501065A CN200610150106A CN101169829A CN 101169829 A CN101169829 A CN 101169829A CN A2006101501065 A CNA2006101501065 A CN A2006101501065A CN 200610150106 A CN200610150106 A CN 200610150106A CN 101169829 A CN101169829 A CN 101169829A
Authority
CN
China
Prior art keywords
image
target model
measured
model
comparison
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CNA2006101501065A
Other languages
Chinese (zh)
Inventor
林宸生
黄国纮
吴俊旻
吴国彰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
QIYI TECHNOLOGY INTERNATIONAL Co Ltd
Original Assignee
QIYI TECHNOLOGY INTERNATIONAL Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by QIYI TECHNOLOGY INTERNATIONAL Co Ltd filed Critical QIYI TECHNOLOGY INTERNATIONAL Co Ltd
Priority to CNA2006101501065A priority Critical patent/CN101169829A/en
Publication of CN101169829A publication Critical patent/CN101169829A/en
Pending legal-status Critical Current

Links

Images

Abstract

An automatic monitoring method for improved templet comparison of electronic element production line includes the following steps: a setting step, a loading step, a regulating step, a comparing step and a finishing step. The method is applied on various grains or cross-label positioning in liquid crystal display detection, has process result with high precision and high speed, can perform comparison process using design templets with any shape, can perform ignore operation process aimed at a set special gray level to improve difference of characteristic part of a target templet, and shield with a feature weight matrix with any shape to strengthen feature of the target templet to improve identification rate of comparison operation. Thus, the invention has the advantages that the target templet has feature weight shielding part with any shape, the target templet can be used as ignore operation setting, the target templet has shielding matrix part with any shape, and brightness and contact of the target templet can be automatically regulated.

Description

The automated detection method of the improvement model comparison of electronic product assembly line
Technical field
The present invention relates to a kind of automated detection method of improvement model comparison of electronic product assembly line.
Background technology
On the industrial detection related application, exist many programs that need classification or identification comparison, as electric circuit inspection of identification kind of object or its otherness, circuit board etc., and characteristics that these actions have are repeatability height, also because such repeatability, we can simplify such manner of execution, define a standard jig, repeat its data group is done the work of searching and comparing.
In the field of computer vision, image processing and Figure recognition,, simultaneously, has the huge characteristic of operational data because that image is the folding of a light intensity function and two amounts of reflective function is long-pending.We carry out so-called pre-treatment, aftertreatment and relevant figure to a figure and strengthen action, the processing of these actions, be nothing but to take away some inadequacies, incoherent noise data, the data of share of only will being correlated with remain, and such flow process, last purpose is nothing but in order to do object identification, comparison and a classification for us.
And the work of comparison identification, how many overlapping relations that just needs clearly to define earlier between data and the data has, and also must allow recognition time not increase because of the huge of data volume as far as possible in the action of numerous and diverse repetition.When carrying out the comparison of data overlapping relation, we select model comparison method, simply say, exactly all standard jig figures are stored in the computing machine in advance.When computing machine ran into an image to be measured, just the plate pattern database data that this image to be measured and all are stored in advance accessed one by one and compares, and the come out plate pattern of approaching this image to be measured of comparison can reach the purpose of SHAPE DETECTION identification.
Simple comparison method, exactly two figures are placed on the same position, make two images subtract each other then, or calculate the area that overlaps between two images, usually do so also and can exhaust a lot of times, for example the classification of defects test pattern of circuit board printing firm just has up to a hundred, and comparison will be used up a large amount of computing times one by one.Bi Dui method usually needs accurate localization like this, otherwise perhaps two images are to have staggered in the position, but content is identical, and subtracting each other the many some elements in back still can residual gray-scale value, so can be two kinds of same image multi-formly be judged to be two different images.
Therefore before the SHAPE DETECTION identification, should do the analysis of some structures to test pattern, be engaged in SHAPE DETECTION identification and comparison according to the feature of structure then, and model (template) promptly can be considered the sub-image with image structure feature, utilize model to be engaged in the work of shape comparison, help to simplify the flow process of whole SHAPE DETECTION identification.
Suppose that image is f (x), model is g (x), and then the mathematical expression of the model of one dimension comparison method is expressed as follows:
d(Y)=∑[f(x)-g(x-Y)] 2
Wherein Σ = Σ x = - m m If arrived two-dimentional system, then Σ = Σ x = - m m Σ y = - n n ;
M, n are the matrix size of model;
Model is made matrix move, compare identically more, d (Y) value is more little, when comparison fits like a glove:
d(Y)=0;
The model of a common image is a two dimension, so the mathematical expression of model comparison method is represented and can be amended as follows:
Σ [ i , j ] ∈ R ( f - g ) 2 ;
Taking advantage of comes can get: Σ [ i , j ] ∈ R ( f - g ) 2 = Σ [ i , j ] ∈ R f 2 + Σ [ i , j ] ∈ R g 2 - 2 Σ [ i , j ] ∈ R fg ;
Suppose f, g then has only in the following formula for fixing
Figure A20061015010600085
Value can be because matrix move and changes, and confidential relation is arranged with the result of model comparison because Σ [ i , j ] ∈ R f 2 + Σ [ i , j ] ∈ R g 2 > 2 Σ [ i , j ] ∈ R fg , Therefore
Figure A20061015010600087
Value big more, then
Figure A20061015010600088
Value more near 0, also just the expression comparison goodness of fit is bigger;
Order Σ [ i , j ] ∈ R fg = M [ i , j ] , That is, M [ i , j ] = Σ k = 1 m Σ l = 1 n g [ k , l ] f [ i + k , j + l ] ;
We originally supposed f, and g is for fixing, but in fact, and after the model decision, g is for fixing, but image f change along with moving of model, that is the object of same shape, may since polishing light field inhomogeneous will make in brighter place, its
Figure A20061015010600093
Value bigger, for fear of this situation, we can take normalized related operation (normalized cross-correlation) to handle, even also:
C fg [ i , j ] = Σ k = 1 m Σ l = 1 n g [ k , l ] f [ i + k , j + l ] ;
And the mathematical expression of model comparison method is represented and can be amended as follows:
M [ i , j ] = C fg [ i , j ] { Σ k = 1 m Σ l = 1 n f 2 [ i + k , j + l ] } 1 / 2 ;
The mathematical expression of model comparison method also can following form be represented:
max [ i , j ] ∈ R | f - g | ;
Or Σ [ i , j ] ∈ R | f - g | ;
Mean that value is little to a threshold values when following, its goodness of fit is bigger.
We can design model according to the shape of predetermined image to be measured, go to compare with designed model then with remaining image to be measured, model can move in each zone of image to be measured when implementing comparison, check the result of the long-pending computing of its folding one by one, when there is the figure of determinand in a certain zone in the image to be measured, very big value will appear in the result of the long-pending computing of folding, and we are the coordinate at decidable determinand place, and the process of whole computing is very similar with the shielding computing.
Existing system produces following shortcoming:
1, plate pattern does not have the feature weight shielding part.Existing system is compared with computing machine temporary plate pattern and image to be measured merely, and plate pattern does not have the design of feature weight shielding part, can't compare at the keypoint part of image to be measured, so need to compare with the integral body of each image to be measured, data are quite huge, and comparison speed is slow.
2, plate pattern can't do to ignore the setting of GTG value.If image to be measured only is arranged in several corners in a zone, then existing plate pattern can't be made to ignore the GTG value with the zone beyond several corners of the image to be measured of correspondence and set, and when comparison, still need be compared in whole zones, and is quite inconvenient and consuming time.
3, plate pattern can't change shape.Suppose that image to be measured is circle or geometric figure, then plate pattern has only squarely, can't make alteration of form in advance by corresponding image to be measured.
4, plate pattern can't be done the adjustment of brightness or contrast.For the image to be measured of constantly weeding out the old and bring forth the new, have may be just on brightness or contrast some elementary errors different, and temporary plate pattern possibility shape is identical with it, but brightness or contrast slightly variant (in fact not influencing the quality of image to be measured), so in comparison process, promptly may be because of these two factor errors, the problem that makes comparison process occur not being inconsistent.
Therefore, be necessary to develop new technology to solve above-mentioned shortcoming.
Summary of the invention
Technical problem underlying to be solved by this invention is, overcome the above-mentioned defective that prior art exists, and the automated detection method that the improvement model that a kind of electronic product assembly line is provided is compared, it has the feature weight shielding part of arbitrary shape, making the target model can make to ignore the GTG value sets, make the target model have the shielding matrix portion of arbitrary shape, and can adjust the brightness and the contrast of target model automatically.
The technical solution adopted for the present invention to solve the technical problems is:
A kind of automated detection method of improvement model comparison of electronic product assembly line is characterized in that, comprises the following steps:
(1). set step: get a target model, this target model is defined as T I, j, k(p, q), its size is P * Q, 1≤i≤3,1≤p≤P, 1≤q≤Q, elder generation sets the predetermined GTG value and the distribution range thereof ignored of this target model, sets the predetermined characteristic weight matrix of this target model again, to finish a predetermined target model;
(2). be written into step: load an image to be measured, and should image to be measured be defined as Ii (x, y), size is M * N, 1≤i≤3;
(3). set-up procedure:, adjust the brightness of this target model and contrast automatically to predetermined value near this image to be measured according to the regional GTG value institute value that calculates this image to be measured;
(4). the comparison step: whether comparison target model conforms to the image of being scheduled to be measured;
(5). completing steps: the target model that comparison obtains coincideing on this image to be measured.
The automated detection method of the improvement model comparison of aforesaid electronic product assembly line, wherein in described setting step:
(1) GTG value and distribution range are ignored in setting: described image I to be measured i(x, y) pixel insignificant plain its (B) the GTG value is (N for R, G 1, N 2, N 3), the plain total quantity of insignificant point is n; Cooperate this condition, on this target model, establish a shielding matrix portion, ignore GTG value distribution range with what this shielding matrix portion defined this target model;
(2) target setting model feature weight shielding: this target model (T I, j, kThe feature weight matrix of (p, q)) be W (p, q), matrix size is P * Q;
W ( p , q ) = W ( 1,1 ) Λ W ( 1 , Q ) M O M W ( P , 1 ) Λ W ( P , Q ) ;
Cooperate above-mentioned condition, on this target model, establish at least one feature weight shielding part,, on this target model, finish the shielding computing that design of GTG value and feature weight matrix are ignored in adding to set the shared zone of feature weight matrix of this target model.
The automated detection method of the improvement model comparison of aforesaid electronic product assembly line, wherein in described set-up procedure:
(1) the regional GTG value of calculating image to be measured: this image I to be measured i(x, y) coordinate points (x, y) (B) the GTG value is respectively (I for R, G 1(x, y), I 2(x, y), I 3(x, y)); This target model is positioned at this image I to be measured i(x, y) on, (j k) is target model T with coordinate I, j, k(p, q) the model coordinate method of relative initial point; This target model T I, j, k(p, q) coordinate points (B) the GTG value is respectively for R, G
(T 1,j,k(p,q),T 2,j,k(p,q),T 3,j,k(p,q));
(2) adjust the brightness of target model automatically: be in dynamic adjustment mode, make this target model move to the brightness optimization at the predetermined block place of image to be measured, with reach overall image be fit to be correlated with before processing, feature extraction, than reciprocity image processing;
(3) adjust target model contrast automatically: be to adjust image contrast, be fit to do pre-treatment, feature extraction to reach overall image,, and can compare the reinforcement computing, and promote image contrast and dynamic range characteristics than reciprocity image processing to optimization.
The automated detection method of the improvement model comparison of aforesaid electronic product assembly line, wherein dynamically adjusting brightness is the average gray value A of the image to be measured at the predetermined block place of definition I, j, k:
If T I, j, k(p, q)=N i
B then I, j, k(p, q)=0;
If T I, j, k(p, q) ≠ N i
B then I, j, k(p, q)=I i(j+p, k+q);
A i , j , k = 1 P × Q - n { 1 3 Σ i = 1 3 Σ p = 1 P Σ q = 1 Q B i , j , k ( p , q ) } ;
A I, j, kRepresentative, on this image to be measured so that (j k) be initial point, and size is (R, G, B) the average GTG value of target image at the block place of P * Q;
And the average gray value AT of objective definition model iFor:
A T i = 1 P × Q - n { 1 3 Σ i = 1 3 Σ p = 1 P Σ q = 1 Q [ T i ( j + p , k + q ) ] } ;
AT wherein iRepresent (R, G, B) the average GTG value, and T of size for the target model of P * Q I, j, k(p, q) ≠ N i
The automated detection method of the improvement model comparison of aforesaid electronic product assembly line, wherein average GTG value be can determine overall image bright partially/partially secretly, and can cooperate a size along with average GTG value again, and change the method for the pixel gray level value of comparison target model:
Set a luminance reference value LV i
If | AT i-A I, j, k|>LV i
T then i' (x, y)=(k iA I, j, k-AT i)+T i(x, y);
Wherein, LV iAdjustment luminance reference value for the average GTG value of the image to be measured at predetermined block place;
T i' (x y) is the matrix gray-scale value of the target model of process adjustment;
k iFor the matrix gray scale of target model is adjusted coefficient.
The automated detection method of the improvement model comparison of aforesaid electronic product assembly line is wherein in described comparison step: relatively move with target model and several images to be measured, calculate the diversity factor D of each image to be measured that corresponds to and target model; If operation result gets diversity factor D I (1≤i≤3), then judge D iWhether less than a reservation threshold V I (1≤i≤3), for not, represent image to be measured and target model to misfit as answer, as answer for being to represent image to be measured and target model identical.
The automated detection method of the improvement model comparison of aforesaid electronic product assembly line, wherein comparison mode is with the antipode sum total value D between the image to be measured of target model and opposite position 1(j, k) the assessment goodness of fit is as less than a threshold values V 1, promptly judge and coincide:
If T i(p, q)=N i
D then i(p, q)=O;
If T i(p, q) ≠ N i
D then I, j, k(p, q)=[| I i(j+p, k+q)-T i(p, q) |] * W (p, q) };
D 1 ( j , k ) = { Σ i = 1 3 Σ p = 1 P Σ q = 1 Q [ d i , j , k ( p , q ) ] } ;
D 1(j, k) representative is on this image to be measured, with coordinate (j, k) be initial point, to x coordinate direction extended distance P, to y coordinate direction extended distance Q, target image with this block, with size be the target model of P * Q, do each pixel RGB GTG value and subtract each other, get its absolute value and the value that adds up mutually is D 1(j, k);
As D 1(j, k)≤V 1
Then so that (j k) is initial point, and to x coordinate direction extended distance P, to y coordinate direction extended distance Q, the target image of this block promptly coincide with the target model.
The automated detection method of the improvement model of aforesaid electronic product assembly line comparison, wherein comparison mode is to be matrix directions with the target model to move, and subtracts each other with the RGB GTG value of the image to be measured of opposite position, and gives square, again the addition sum total; Obtain difference value D 1Less than threshold values V 1, then decidable this locate to be target model place;
If T i(p, q)=N i
D then i(p, q)=O;
If T i(p, q) ≠ N i
D then I, j, k(p, q)=[| I i(j+p, k+q)-T i(p, q) |] * W (p, q) };
D 1 ( j , k ) = Σ i = 1 3 Σ p = 1 P Σ q = 1 Q [ d i j , k ( p , q ) ] 2 ;
D 1(j, k) representative on this image to be measured, with coordinate (j k) be initial point, to x coordinate direction extended distance P, to y coordinate direction extended distance Q, with this block image, with size be the target model of P * Q, do that the RGB diversity factor sums up square;
As D 1(j, k)≤V 1
Then so that (j k) is initial point, and to x coordinate direction extended distance P, to y coordinate direction extended distance Q, this block image promptly coincide with the target model.
The automated detection method of the improvement model of aforesaid electronic product assembly line comparison, wherein comparison mode are the maximum RGB difference value D that asks for target model and the image to be measured of corresponding coordinate 2, be used for assessing the goodness of fit, as less than a threshold values V 2, promptly judge and coincide:
If T i(p, q)=N i
D then i(p, q)=O;
If T i(p, q) ≠ N i
D then I, j, k(p, q)=[| I i(j+p, k+q)-T i(p, q) |] * W (p, q) };
D 2 ( j , k ) = max { Σ i = 1 3 [ d i , j , k ( p , q ) ] } ;
D 2(j k) represents on this image to be measured, and (j k) is initial point with coordinate, to x coordinate direction extended distance P, to y coordinate direction extended distance Q, with this block image, with size be the target model of P * Q, does each pixel RGB GTG value and subtract each other, and its maximal value is D 2(j, k);
As D 2(j, k)≤V 2
Then so that (j k) is initial point, and to x coordinate direction extended distance P, to y coordinate direction extended distance Q, this block image promptly coincide with the target model.
The automated detection method of the improvement model comparison of aforesaid electronic product assembly line, wherein the presumptive area of target model is to cooperate shielding matrix portion computing, and forms a given shape target model.
The invention has the beneficial effects as follows that the feature weight shielding part that it has arbitrary shape makes the target model can make to ignore the GTG value and sets, and makes the target model have the shielding matrix portion of arbitrary shape, and can adjust the brightness and the contrast of target model automatically.
Description of drawings
The present invention is further described below in conjunction with drawings and Examples.
Fig. 1 is a schematic flow sheet of the present invention
Fig. 2 is an actual mechanical process calcspar of the present invention
Fig. 3 is the synoptic diagram of model comparison of the present invention
Fig. 4 A, Fig. 4 B, Fig. 4 C and Fig. 4 D are the assignment procedure synoptic diagram one, two, three and four of target model of the present invention
Fig. 5 is a wherein embodiment synoptic diagram of model matrix of the present invention
Fig. 6 is that model matrix of the present invention is searched for the comparison synoptic diagram
Fig. 7 is GTG brightness statistics figure of the present invention
Fig. 8 is the area schematic of image definition to be measured of the present invention
Fig. 9 is the design diagram of the target model of given shape of the present invention
The number in the figure explanation:
11 set step 12 load step
13 set-up procedures, 14 comparison steps
15 completing steps, 20 images to be measured
30 target model 30A target block
30B given shape target model 30C tool local feature is emphasized the target model
31 shielding matrix portions 311 are shielding portion not
312 shielding portions, 32 feature weight shielding parts
No feature weight portion of 321 feature weight portions 322
Embodiment
The present invention is a kind of automated detection method of improvement model comparison of electronic product assembly line, and as shown in Figures 1 and 2, it comprises the following steps:
1. set step 11: get a target model 30 (consulting Fig. 3, Fig. 4 A and Fig. 4 B), this target model 30 is defined as T I, j, k(p, q), its size is P * Q, 1≤i≤3,1≤p≤P, 1≤q≤Q; Set earlier this target model 30 ignore GTG value and distribution range (shown in annex one figure B), for another example shown in Fig. 4 C, set the feature weight shielding part 32 (shown in annex one figure C) of this target model 30, next shown in Fig. 4 D, set the matrix (shown in annex one figure D) of this target model 30, to finish a predetermined target model (shown in annex one figure E);
2. be written into step 12: load an image 20 to be measured, and should be defined as I by image 20 to be measured i(x, y), size is M * N, 1≤i≤3 (showing as Fig. 3); And begin comparison by the initial point of this image 20 to be measured;
3. set-up procedure 13: calculate the regional GTG value of this image 20 to be measured, and according to calculating institute's value, adjust the brightness of this target model 30 and the predetermined value of extremely approaching this image 20 to be measured of contrast (shown in annex one figure A) automatically;
4. comparison step 14: the target model 30 finished of comparison and the image of being scheduled to be measured 20 whether conform to (shown in annex one figure E);
5. completing steps 15: the target model 30 that comparison obtains coincideing on this image 20 to be measured.
It so is the automated detection method of the improvement model comparison of electronic product assembly line of the present invention.
Consult Fig. 1 and Fig. 2, set in the step 11 at this:
1, GTG value and distribution range are ignored in setting: this image I to be measured i(x, y) pixel insignificant plain its (B) the GTG value is (N for R, G 1, N 2, N 3), the plain total quantity of insignificant point is n; Cooperate this condition, on this target model 30 (shown in Fig. 4 A), establish a shielding matrix portion 31, ignore GTG value distribution range (shown in Fig. 4 B) with what this shielding matrix portion 31 defined this target model 30;
2, target setting model feature weight shielding: this target model 30 (T I, j, kThe feature weight matrix of (p, q)) be W (p, q), matrix size is P * Q;
W ( p , q ) = W ( 1,1 ) Λ W ( 1 , Q ) M O M W ( P , 1 ) Λ W ( P , Q ) ;
Cooperate above-mentioned condition, be provided with at least one feature weight shielding part 32 (the present invention establishes four feature weight shielding parts 32) in this target model 30, so set the shared zone of feature weight matrix (shown in Fig. 4 C) of this target model 30, next shown in Fig. 4 D, on this target model 30, finish the shielding computing that design of GTG value and feature weight matrix are ignored in adding.
In this set-up procedure 13:
1, calculates the regional GTG value of image to be measured: this image I to be measured i(x, y) coordinate points (x, y) (B) the GTG value is respectively (I for R, G 1(x, y), I 2(x, y), I 3(x, y)); This target model T I, j, k(p q) is positioned at this image I to be measured i(x, y) on, (j k) is target model T with coordinate I, j, k(p, q) the model coordinate system of relative initial point; This target model T I, j, k(p, q) coordinate points (B) the GTG value is respectively (T for R, G 1, j, k(p, q), T 2, j, k(p, q), T 3, j, k(p, q));
2, adjust the brightness of target model automatically: the mode that can dynamically adjust, make this target model 30 move to the brightness optimization at the predetermined block place of image 20 to be measured, with reach overall image be fit to be correlated with before processing, feature extraction, than reciprocity image processing;
The technological means that above-mentioned dynamic adjustment brightness is used is the average gray value A of the image to be measured 20 at the predetermined block place of definition I, j, k:
If T I, j, k(p, q)=N i
B then I, j, k(p, q)=0;
If T I, j, k(p, q) ≠ N i
B then I, j, k(p, q)=I i(j+p, k+q);
A i , j , k = 1 P × Q - n { 1 3 Σ i = 1 3 Σ p = 1 P Σ q = 1 Q B i , j , k ( p , q ) } ;
A I, j, kRepresentative, on this image 20 to be measured so that (j k) be initial point, and size is (R, G, B) the average GTG value of target model 30 at the block place of P * Q.
The average gray value AT of objective definition model 30 in like manner iFor:
AT i = 1 P × Q - n { 1 3 Σ i = 1 3 Σ p = 1 P Σ q = 1 Q [ T i ( j + p , k + q ) ] } ;
AT wherein iRepresent (R, G, B) the average GTG value, and T of size for the target model 30 of P * Q I, j, k(p, q) ≠ N i
Because average GTG value, can determine overall image bright partially or dark partially, the present invention reaches the whole comparison mechanism that do not influence because of light source is strong and weak for improving the comparison accuracy rate, can cooperate a size again, and change the method for the pixel gray level value of comparison target model 30 along with average GTG value:
Set a luminance reference value LV i
If | AT i-A I, j, k|>LV i
T then i' (x, y)=(k iA I, j, k-AT i)+T i(x, y);
Wherein, LV iAdjustment luminance reference value for the average GTG value of the image to be measured 20 at predetermined block place;
T i' (x y) is the matrix gray-scale value of the target model 30 of process adjustment;
k iFor the matrix gray scale of target model 30 is adjusted coefficient;
Whereby, cooperate, and image 20 to be measured is carried out the work that luminance dynamic is adjusted as Fig. 5 and the predetermined image to be measured 20 of target model 30 (matrix) scanning shown in Figure 6; In 8 * 4 matrix of this target model 30, have 32 pixel values (pixel), calculate average GTG value with formula, if the measures of dispersion of image to be measured 20 average gray values of the average GTG value of target model 30 and predetermined block is greater than LV i, promptly represent this target model 30 bright too partially or dark partially, need add that one adjusts parameter (no special parameter is as long as be adjusted to predetermined luminance according to actual needs).
3, adjust target model contrast automatically: be to adjust image contrast to optimization, with reach make overall image be fit to do pre-treatment, feature extraction, than reciprocity image processing, as shown in Figure 7, its longitudinal axis is a number of pixels, transverse axis is image greyscale value (0~255), gray-scale value is proportional to the lightness of image, can being learnt by following formula specific strength (Contrast) an of image:
C i = S im 1 - S im 2 S im 1 + S im 2 ;
S Im1, S Im2For accounting for the gray-scale value of the maximum number of pixels of whole image among the intensity profile figure, both gray difference values heal when big, and are also just big to specific strength.
Compare reinforcement (Contrast Enhacement) computing whereby, with promote image to dynamic range characteristics when; Find out assembly place of an image greyscale value, make outer maximum gradation value of this assembly place and minimum gradation value, the gray scale extension of this image is come, use 0 to 255 whole half-tone information, so make things convenient for eye-observation for the bound of contrast intensification's computing.Compare with the expansion computing, this method can make the saturated some prime number order of gray-scale value reduce, and can avoid whole the situation that the image greyscale value is higher again.
General contrast intensification (contrast enhancement) formula is:
I i ' ( x , y ) = I i ( x , y ) - S im 1 S im 2 - S im 1 × 255 ;
And the present invention sets a contrast reference value CV iAnd the GTG brightness statistics figure that obtains through scanning by target model 30 (matrix), define zone (the Region Of Interest shown in Fig. 7 (the GTG brightness statistics figure of target example edition 30), be called for short ROI), it adds up number preceding 1/4th and the average GTG value S of back 1/4th with pixel It1And S It2, calculate its contrast C Ti
The x coordinate is 0 to 255 GTG value, and y coordinate f (x) is the quantity of this pixel gray level value on the target model, the valid pixel of target model be (P * Q-n), order:
S iT 1 = [ Σ x = 0 x 1 x × f ( x ) ] 1 4 ( P × Q - n ) ;
S iT 2 = [ Σ x = x 2 255 x × f ( x ) ] 1 4 ( P × Q - n ) ;
F (x) ≠ N wherein i, x 1For pixel adds up number in preceding 1/4th image greyscale value, Σ x = 0 x 1 f ( x ) = 1 4 ( P × Q - n ) , x 2For pixel adds up number in preceding 3/4ths image greyscale value, Σ x = 0 x 2 f ( x ) = 3 4 ( P × Q - n ) ;
The then contrast of target model C Ti = S iT 1 - S iT 2 S iT 1 + S iT 2 ;
In like manner, the contrast of the image to be measured 20 in the predetermined block C Ii = S iI 1 - S iI 2 S iI 1 + S iI 2 ;
Target model 30 (matrix) with Fig. 5 is an example, if the contrast strength difference (C of this target model 30 and image 20 to be measured Ti-C Ii) less than reference value CV iThe time, image 20 then to be measured is to belong to image clearly, does not just need to adjust the contrast of target model 30; If (C Ti-C Ii) greater than reference value CV iThe time, the contrast of then representing image 20 to be measured then must be adjusted much smaller than this target model 30
The contrast of target model 30 is C Ti = S iT 1 - S iT 2 S iT 1 + S iT 2 ;
The contrast of former block image C Ii = S iI 1 - S iI 2 S iI 1 + S iI 2 ;
If (C Ti-C Ii)>CV i, and CV iAdjustment luminance reference value for the average block image greyscale value of image 20 to be measured;
Then target model 30 is done and is downgraded the contrast processing:
T then i' ( x , y ) = T i ( x , y ) - S iT 1 S iT 2 - S iT 1 × ( S iI 2 - S iI 1 ) + S iI 1 ;
T i' (x y) is the gray-scale value of target model 30 (matrix) behind the contrast intensification;
In this comparison step 14: relatively move (as shown in Figure 6) with target model shown in Figure 5 30 and a plurality of images 20 to be measured, in moving process, calculate the diversity factor D of each image to be measured that corresponds to 20 and target model 30; If operation result gets diversity factor D I (1≤i≤3), then judge D iWhether less than a reservation threshold V I (1≤i≤3), for not, represent image 20 to be measured to misfit as answer with target model 30, as answer for being to represent image 20 to be measured identical with target model 30.
In fact, comparison mode have at least following several:
1, utilizes the antipode sum total value D of 20 of the images to be measured of target model 30 and opposite position 1(j k), also is to be used for assessing the goodness of fit, as less than a threshold values V 1, promptly judge and coincide:
If T i(p, q)=N i
D then i(p, q)=O;
If T i(p, q) ≠ N i
D then I, j, k(p, q)=[| I i(j+p, k+q)-T i(p, q) |] * W (p, q) }
D 1 ( j , k ) = { Σ i = 1 3 Σ p = 1 P Σ q = 1 Q [ d i , j , k ( p , q ) ] } ;
D 1(j, k) representative is on this image 30 to be measured, with coordinate (j, k) be initial point, to x coordinate direction extended distance P, to y coordinate direction extended distance Q, target image 20 with this block, with size be the target model 30 of P * Q, do each pixel RGB GTG value and subtract each other, get its absolute value and the value that adds up mutually is D 1(j, k);
As D 1(j, k)≤V 1
Then so that (j k) is initial point, and to x coordinate direction extended distance P, to y coordinate direction extended distance Q, the target image 20 of this block promptly coincide with target model 30.
2, target model 30 moves with matrix directions, subtracts each other with the RGB GTG value of the image to be measured 30 of opposite position, and gives square, again the addition sum; Obtain difference value D 1Less than threshold values V 1, then decidable this locate to be the model place;
If T i(p, q)=N i
D then i(p, q)=O;
If T i(p, q) ≠ N i
D then Ij, k(p, q)=[| I i(j+p, k+q)-T i(p, q) |] * W (p, q) }
D 1 ( j , k ) = Σ i = 1 3 Σ p = 1 P Σ q = 1 Q [ d ij , k ( p , q ) ] 2 ;
D 1(j, k) representative on this image 20 to be measured, with coordinate (j k) be initial point, to x coordinate direction extended distance P, to y coordinate direction extended distance Q, with this block image, with size be the target model 30 of P * Q, do that the RGB diversity factor sums up square;
As D 1(j, k)≤V 1
Then so that (j k) is initial point, and to x coordinate direction extended distance P, to y coordinate direction extended distance Q, this block image promptly coincide with target model 30.
3, ask for the maximum RGB difference value D of target model 30 and the image to be measured 20 of corresponding coordinate 2, be used for assessing the goodness of fit, as less than a threshold values V 2, promptly judge and coincide:
If T i(p, q)=N i
D then i(p, q)=O;
If T i(p, q) ≠ N i
D then I, j, k(p, q)=[| I i(j+p, k+q)-T i(p, q) |] * W (p, q) };
D 2 ( j , k ) = max { Σ i = 1 3 [ d i , j , k ( p , q ) ] } ;
D 2(j, k) representative is on this image 20 to be measured, with coordinate (j, k) be initial point, to x coordinate direction extended distance P, to y coordinate direction extended distance Q, with this block image, with size be the target model 30 of P * Q, does each pixel RGB GTG value and subtract each other, and its maximal value is D 2(j, k);
As D 2(j, k)≤V 2
Then so that (j k) is initial point, and to x coordinate direction extended distance P, to y coordinate direction extended distance Q, this block image promptly coincide with target model 30.
Shown in the figure A of annex one, it is on actual LCD (Liquid Crystal Display the is called for short LCD) production line, is used to locate the detection image of calibration, target model for taking out in the black line frame, Sibai look square is the location square frame on the target model; When comparison, the background that this kind detects image is often comparatively complicated, designs (shown in the figure B of annex one) so add the GTG value of ignoring of grey.
Yet the model search of this type, need comparison result comparatively accurately, so in the design of target model feature weight shielding, can emphasize the proportion (shown in the figure C of annex one) of its diversity factor at discontinuous local feature place, the corresponding white box that detects image, the feature weight value is 4, posting and background intersection, its diversity factor are for the also suitable importance of the comparison of whole image to be measured, and weighted value is 8; Be depicted as the figure D of annex one and add the arithmograph that the model of ignoring the design of GTG value and feature weight shield; Be depicted as target model after the design in the synoptic diagram that detects on the image as the figure E of annex one.
Suppose one 100 * 100 model, have 10000 some elements, its zone that need compare (Region Of Interest, be called for short ROI) may have only 20 some elements, adopt sequence to represent a model, represent that than general use matrix model will save a lot of figure places certainly, if the model image is big more, or the zone of comparison is more little in the shared ratio of model image, and it is also just favourable more to use sequential value to represent so.
Dan Se model for example, its sequence can be write as A=a 11a 12a 13... ..a N3
A wherein 11Be the plain GTG of first point in the model comparison zone, a 12a 13Be the plain x of first point in the model comparison zone, y coordinate figure.
In fact, target model 30 of the present invention is except square, also can be any specific shape, as shown in Figure 9, capture a square target model 30, have a target block 30A (may be circle) on it, other establishes a shielding matrix portion 31 and (is roughly 12 * 12 matrix, this target model 30 is identical with it), this shielding matrix portion 31 has not a shielding portion 311 (being defined as 0) and a shielding portion 312 (being defined as 1) at least; This not shielding portion 311 be corresponding to this target block 30A; Cooperate this target model 30 to carry out computing with this shielding matrix portion 31, obtain a given shape target model 30B; To specify in this part, be this given shape target model 30B profile should with this not shielding portion 31 be all zigzag (also can say it is the matrix profile), the present invention beautifies pattern.
Establish a feature weight shielding part 32 again, it has feature weight portion 321 at least and does not have feature weight portion 322, and this feature weight portion 32 for example further has the feature weight district of different proportions such as 4,6, cooperate this given shape target model 30B to carry out computing with this feature weight shielding part 32, produce the tool local feature and emphasize target model 30C; It can be in order to carry out image to be measured 20 comparisons as shown in Figure 6.
Advantage of the present invention and effect are as described below:
1, the target model has the feature weight shielding part of arbitrary shape.No matter the present invention is to use the target model of what shape, can add a feature weight shielding part, with at the keypoint part of image to be measured (no matter keypoint part is in angle or middle position, no matter also keypoint part is square, arc or geometric configuration), carry out feature weight shielding computing, directly compare, speed is fast and accurate.
2, the target model can do to ignore the setting of GTG value.Suppose that image to be measured is to be positioned at four square corners, and the target model is square, then the present invention can ignore the setting of GTG value with the full work in the zone beyond four corners of target model, so in comparison process, its corresponding image to be measured position in addition of target model is left in the basket entirely, significantly improves the precision and the speed of comparison.
3, the target model has the shielding matrix portion of arbitrary shape.For comparing various different image to be measured (for example electronic package of Yuan Xing circuit board or geometric configuration) fast; the present invention on square target model, establish one with the corresponding shielding matrix of image to be measured; make square-mesh standard specimen plate directly be configured to the shape of corresponding image to be measured; be applied to LCD and detect each based fine particles or cross mark location, the more accurate result faster that reaches is arranged.
4, can adjust the brightness and the contrast of target model automatically.For adapting to the image to be measured of different brightness and contrast, after having calculated the regional GTG value of image to be measured, the controlled target model is adjusted to brightness and the contrast near this image to be measured automatically, improves the precision of comparison process.
The above, it only is preferred embodiment of the present invention, be not that the present invention is done any pro forma restriction, every foundation technical spirit of the present invention all still belongs in the scope of technical solution of the present invention any simple modification, equivalent variations and modification that above embodiment did.
In sum, the present invention is on structural design, use practicality and cost benefit, it is required to meet industry development fully, and the structure that is disclosed also is to have unprecedented innovation structure, have novelty, creativeness, practicality, the regulation that meets relevant patent of invention important document is so mention application in accordance with the law.
[annex one]
Figure A is an actual detection image synoptic diagram of the present invention
Figure B is the synoptic diagram of ignoring the GTG value that model of the present invention adds grey
Figure C is the synoptic diagram of model feature weight matrix of the present invention
Figure D is the shielding arithmograph that design of GTG value and feature weight matrix are ignored in adding of the present invention
Figure E is the synoptic diagram of model on the detection image after the design of the present invention.

Claims (10)

1. the automated detection method of the improvement model of electronic product assembly line comparison is characterized in that, comprises the following steps:
(1). set step: get a target model, this target model is defined as T I, j, k(p, q), its size is P * Q, 1≤i≤3,1≤p≤P, 1≤q≤Q, elder generation sets the predetermined GTG value and the distribution range thereof ignored of this target model, sets the predetermined characteristic weight matrix of this target model again, to finish a predetermined target model;
(2). be written into step: load an image to be measured, and should image to be measured be defined as Ii (x, y), size is M * N, 1≤i≤3;
(3). set-up procedure:, adjust the brightness of this target model and contrast automatically to predetermined value near this image to be measured according to the regional GTG value institute value that calculates this image to be measured;
(4). the comparison step: whether comparison target model conforms to the image of being scheduled to be measured;
(5). completing steps: the target model that comparison obtains coincideing on this image to be measured.
2. according to the automated detection method of the improvement model of claim 1 described electronic product assembly line comparison, it is characterized in that in described setting step:
(1) GTG value and distribution range are ignored in setting: described image I to be measured i(x, y) pixel insignificant plain its (B) the GTG value is (N for R, G 1, N 2, N 3), the plain total quantity of insignificant point is n; Cooperate this condition, on this target model, establish a shielding matrix portion, with this shielding square
What battle array portion defined this target model ignores GTG value distribution range;
(2) target setting model feature weight shielding: this target model (T I, j, kThe feature weight matrix of (p, q)) be W (p, q), matrix size is P * Q;
W ( p , q ) = W ( 1,1 ) Λ W ( 1 , Q ) M O M W ( P , 1 ) Λ W ( P , Q ) ;
Cooperate above-mentioned condition, on this target model, establish at least one feature weight shielding part,, on this target model, finish the shielding computing that design of GTG value and feature weight matrix are ignored in adding to set the shared zone of feature weight matrix of this target model.
3. the automated detection method of the improvement model of electronic product assembly line according to claim 1 comparison is characterized in that in described set-up procedure:
(1) the regional GTG value of calculating image to be measured: this image I to be measured i(x, y) coordinate points (x, y) (B) the GTG value is respectively (I for R, G 1(x, y) I 2(x, y) I 3(x, y); This target model is positioned at this image I to be measured i(x, y) on, (j k) is target model T with coordinate I, j, k(p, q) the model coordinate method of relative initial point; This target model T I, j, k(p, q) coordinate points (B) the GTG value is respectively for R, G
(T 1,j,k(p,q),T 2,j,k(p,q),T 3,j,k(p,q));
(2) adjust the brightness of target model automatically: be in dynamic adjustment mode, make this target model move to the brightness optimization at the predetermined block place of image to be measured, with reach overall image be fit to be correlated with before processing, feature extraction, than reciprocity image processing;
(3) adjust target model contrast automatically: be to adjust image contrast, be fit to do pre-treatment, feature extraction to reach overall image,, and can compare the reinforcement computing, and promote image contrast and dynamic range characteristics than reciprocity image processing to optimization.
4. the automated detection method of the improvement model of electronic product assembly line according to claim 3 comparison is characterized in that described dynamic adjustment brightness is the average gray value A of the image to be measured at the predetermined block place of definition I, j,, k:
If T I, j, k(p, q)=N i
B then I, j, k(p, q)=0;
If T I, j, k(p, q) ≠ N i
B then I, j, k(p, q)=I i(j+p, k+q);
A i , j , k = 1 P × Q - n { 1 3 Σ i = 1 3 Σ p = 1 P Σ q = 1 Q B i , j , k ( p , q ) } ;
A I, j, kRepresentative, on this image to be measured so that (j k) be initial point, and size is (R, G, B) the average GTG value of target image at the block place of P * Q;
And the average gray value AT of objective definition model iFor:
T i = 1 P × Q - n { 1 3 Σ i = 1 3 Σ p = 1 P Σ q = 1 Q [ T i ( j + p , k + q ) ] } ;
AT wherein iRepresent (R, G, B) the average GTG value, and T of size for the target model of P * Q I, j, k(p, q) ≠ N 1
5. the automated detection method of the improvement model of electronic product assembly line according to claim 4 comparison, it is characterized in that described average GTG value be can determine overall image bright partially/dark partially, and can cooperate a size along with average GTG value again, and change the method for the pixel gray level value of comparison target model:
Set a luminance reference value LV i
If | AT i-A I, j, k|>LV i
T then i' (x, y)=(k iA I, j, k-AT i)+T i(x, y);
Wherein, LV iAdjustment luminance reference value for the average GTG value of the image to be measured at predetermined block place;
T i' (x y) is the matrix gray-scale value of the target model of process adjustment;
k iFor the matrix gray scale of target model is adjusted coefficient.
6. the automated detection method of the improvement model of electronic product assembly line according to claim 1 comparison, it is characterized in that in described comparison step: relatively move with target model and several images to be measured, calculate the diversity factor D of each image to be measured that corresponds to and target model; If operation result gets diversity factor D I (1≤i≤3), then judge D iWhether less than a reservation threshold V I (1≤i≤3), for not, represent image to be measured and target model to misfit as answer, as answer for being to represent image to be measured and target model identical.
7. the automated detection method of the improvement model of electronic product assembly line according to claim 6 comparison is characterized in that described comparison mode is with the antipode sum total value D between the image to be measured of target model and opposite position 1(j, k) the assessment goodness of fit is as less than a threshold values V 1, promptly judge and coincide:
If T i(p, q)=N i
D then i(p, q)=0;
If T i(p, q) ≠ N i
D then I, j, k(p, q)=[| I i(j+p, k+q)-T i(p, q) |] * W (p, q) };
D 1 = ( j , k ) = { Σ i = 1 3 Σ p = 1 P Σ q = 1 Q [ d i , j , k ( p , q ) ] } ;
D 1(j, k) representative is on this image to be measured, with coordinate (j, k) be initial point, to x coordinate direction extended distance P, to y coordinate direction extended distance Q, target image with this block, with size be the target model of P * Q, do each pixel RGB GTG value and subtract each other, get its absolute value and the value that adds up mutually is D 1(j, k);
As D 1(j, k)≤V 1
Then so that (j k) is initial point, and to x coordinate direction extended distance P, to y coordinate direction extended distance Q, the target image of this block promptly coincide with the target model.
8. the automated detection method of the improvement model of electronic product assembly line according to claim 6 comparison, it is characterized in that described comparison mode is to be matrix directions with the target model to move, subtract each other with the RGB GTG value of the image to be measured of opposite position, and give square, again the addition sum total; Obtain difference value D 1Less than threshold values V 1, then decidable this locate to be target model place;
If T i(p, q)=N i
D then i(p, q)=0;
If T i(p, q) ≠ N i
D then I, j, k(p, q)=[| I i(j+p, k+q)-T i(p, q) |] * W (p, q) };
D 1 ( j , k ) = Σ i = 1 3 Σ p = 1 P Σ q = 1 Q [ d i , j , k ( p , q ) ] 2 ;
D 1(j, k) representative on this image to be measured, with coordinate (j k) be initial point, to x coordinate direction extended distance P, to y coordinate direction extended distance Q, with this block image, with size be the target model of P * Q, do that the RGB diversity factor sums up square;
As D 1(j, k)≤V 1
Then so that (j k) is initial point, and to x coordinate direction extended distance P, to y coordinate direction extended distance Q, this block image promptly coincide with the target model.
9. the automated detection method of the improvement model of electronic product assembly line according to claim 6 comparison is characterized in that described comparison mode is to ask for the maximum RGB difference value D of target model and the image to be measured of corresponding coordinate 2, be used for assessing the goodness of fit, as less than a threshold values V 2, promptly judge and coincide:
If T i(p, q)=N i
D then i(p, q)-O;
If T i(p, q) ≠ N i
D then I, j, k(p, q)=[| I i(j+p, k+q)-T i(p, q) |] * W (p, q) };
D 2 ( j , k ) = max { Σ i = 1 3 [ d i . j . k ( p , q ) ] } ;
D 2(j k) represents on this image to be measured, and (j k) is initial point with coordinate, to x coordinate direction extended distance P, to y coordinate direction extended distance Q, with this block image, with size be the target model of P * Q, does each pixel RGB GTG value and subtract each other, and its maximal value is D 2(j, k);
As D 2(j, k)≤V 2
Then so that (j k) is initial point, and to x coordinate direction extended distance P, to y coordinate direction extended distance Q, this block image promptly coincide with the target model.
10. the automated detection method of the improvement model of electronic product assembly line according to claim 1 comparison, the presumptive area that it is characterized in that described target model is to cooperate shielding matrix portion computing, and forms a given shape target model.
CNA2006101501065A 2006-10-27 2006-10-27 Automated detection method for improving template comparison of electronic product assembly line Pending CN101169829A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CNA2006101501065A CN101169829A (en) 2006-10-27 2006-10-27 Automated detection method for improving template comparison of electronic product assembly line

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CNA2006101501065A CN101169829A (en) 2006-10-27 2006-10-27 Automated detection method for improving template comparison of electronic product assembly line

Publications (1)

Publication Number Publication Date
CN101169829A true CN101169829A (en) 2008-04-30

Family

ID=39390448

Family Applications (1)

Application Number Title Priority Date Filing Date
CNA2006101501065A Pending CN101169829A (en) 2006-10-27 2006-10-27 Automated detection method for improving template comparison of electronic product assembly line

Country Status (1)

Country Link
CN (1) CN101169829A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104792263A (en) * 2015-04-20 2015-07-22 合肥京东方光电科技有限公司 Method and device for determining to-be-detected area of display mother board
CN108037442A (en) * 2017-12-08 2018-05-15 郑世珍 A kind of detection device for integrated circuit plate electronic component
CN108198141A (en) * 2017-12-28 2018-06-22 北京奇虎科技有限公司 Realize image processing method, device and the computing device of thin face special efficacy

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104792263A (en) * 2015-04-20 2015-07-22 合肥京东方光电科技有限公司 Method and device for determining to-be-detected area of display mother board
US10210605B2 (en) 2015-04-20 2019-02-19 Boe Technology Group Co., Ltd. Method and device for detecting boundary of region on display motherboard
CN108037442A (en) * 2017-12-08 2018-05-15 郑世珍 A kind of detection device for integrated circuit plate electronic component
CN108037442B (en) * 2017-12-08 2019-11-19 江苏富澜克信息技术有限公司 A kind of detection device for integrated circuit board electronic component
CN108198141A (en) * 2017-12-28 2018-06-22 北京奇虎科技有限公司 Realize image processing method, device and the computing device of thin face special efficacy
CN108198141B (en) * 2017-12-28 2021-04-16 北京奇虎科技有限公司 Image processing method and device for realizing face thinning special effect and computing equipment

Similar Documents

Publication Publication Date Title
Molina et al. On the detection of defects on specular car body surfaces
CN109934814B (en) Surface defect detection system and method thereof
Liu et al. Machine vision based online detection of PCB defect
CN108355981B (en) Battery connector quality detection method based on machine vision
US6868175B1 (en) Pattern inspection apparatus, pattern inspection method, and recording medium
CN102496161B (en) Method for extracting contour of image of printed circuit board (PCB)
CN111693549A (en) Method for detecting and classifying defects of mobile phone cover plate glass
CN106803244A (en) Defect identification method and system
CN108416766A (en) Bilateral incidence type light guide plate defective vision detection method
CN101251496A (en) Defect detection device and defect detection method
Zhongxiang et al. Evaluation of three-dimensional surface roughness parameters based on digital image processing
CN108492327B (en) AOI image matching method, system, readable storage medium and intelligent device
US20010012390A1 (en) Pattern inspection equipment, pattern inspection method, and storage medium storing pattern inspection program
CN101169829A (en) Automated detection method for improving template comparison of electronic product assembly line
CN112767359A (en) Steel plate corner detection method and system under complex background
CN114677340A (en) Concrete surface roughness detection method based on image edge
CN107328371A (en) Sub-pix contours extract based on Gaussian and the optimization using Softmax recurrence in the case where metal plate detects scene
Zhou et al. Circle detection with model fitting in polar coordinates for glass bottle mouth localization
US7538750B2 (en) Method of inspecting a flat panel display
Wang et al. Assembly defect detection of atomizers based on machine vision
TW201512649A (en) Method of chip detects inspecting, system therefor, and computer program product thereof
CN115753791B (en) Defect detection method, device and system based on machine vision
CN113077429B (en) Speckle quality evaluation method based on adjacent sub-area correlation coefficient
CN114693681A (en) Zipper production defect intelligent detection method and system based on optical means
CN114897797A (en) Method, device and equipment for detecting defects of printed circuit board and storage medium

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C02 Deemed withdrawal of patent application after publication (patent law 2001)
WD01 Invention patent application deemed withdrawn after publication

Open date: 20080430