CN114741804A - Hole inner surface bidirectional reflection distribution function measurement and modeling method - Google Patents

Hole inner surface bidirectional reflection distribution function measurement and modeling method Download PDF

Info

Publication number
CN114741804A
CN114741804A CN202210339025.9A CN202210339025A CN114741804A CN 114741804 A CN114741804 A CN 114741804A CN 202210339025 A CN202210339025 A CN 202210339025A CN 114741804 A CN114741804 A CN 114741804A
Authority
CN
China
Prior art keywords
hole
point
endoscope
brdf
lens
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210339025.9A
Other languages
Chinese (zh)
Inventor
盛强
申亮
李程
蔡乐平
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shaanxi University of Science and Technology
Original Assignee
Shaanxi University of Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shaanxi University of Science and Technology filed Critical Shaanxi University of Science and Technology
Priority to CN202210339025.9A priority Critical patent/CN114741804A/en
Publication of CN114741804A publication Critical patent/CN114741804A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F30/00Computer-aided design [CAD]
    • G06F30/10Geometric CAD
    • G06F30/17Mechanical parametric or variational design
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F30/00Computer-aided design [CAD]
    • G06F30/20Design optimisation, verification or simulation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2119/00Details relating to the type or aim of the analysis or the optimisation
    • G06F2119/02Reliability analysis or reliability optimisation; Failure analysis, e.g. worst case scenario performance, failure mode and effects analysis [FMEA]
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02PCLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Geometry (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Pure & Applied Mathematics (AREA)
  • Mathematical Optimization (AREA)
  • Mathematical Analysis (AREA)
  • Computational Mathematics (AREA)
  • Endoscopes (AREA)

Abstract

A hole inner surface bidirectional reflection distribution function measuring and modeling method comprises the following specific steps: 1. acquiring image brightness information of a hole with a known inner surface three-dimensional shape through an endoscope, 2, measuring a BRDF value of the inner surface of the hole according to the radiant irradiance of each point of the inner surface of the hole and the reflected radiance reflected into the lens direction, 3, establishing an inBRDF model of the inner surface of the hole according to the relation between the BRDF of the inner surface of the hole and the three-dimensional shape of the hole, 4, measuring the BRDF value of the inner surface of the hole made of a specific material through experiments, and determining the optimal parameter of the inBRDF model of the specific material by adopting a genetic algorithm; based on the brightness of the endoscopic image and the hole inner surface normal distribution measurement hole inner surface BRDF measurement method, a Phong model is improved to establish a hole inner surface bidirectional reflection distribution function inBRDF statistical model, and the relation between the hole inner surface BRDF and the three-dimensional morphology is established; the method provided by the invention can effectively measure the bidirectional reflection distribution function of the inner surface of the hole, cavity and cavity structure, has high efficiency and strong equipment universality, and the established inBRDF model can quickly and accurately obtain the BRDF value of the continuous angle of the inner surface of the hole and can be applied to the quality detection of the inner surface of the hole based on machine vision.

Description

Hole inner surface bidirectional reflection distribution function measurement and modeling method
Technical Field
The invention relates to the technical field of machining, in particular to a method for measuring and modeling a bidirectional reflection distribution function of an inner surface of a hole.
Background
The hole parts are the most common part forms in machining and production and manufacturing, and detection of the quality of the hole parts, particularly the quality of the inner surface of the hole, has been a difficult problem for enterprises for a long time. The reflection characteristic refers to a distribution rule of the object surface to a light energy mirror and diffuse reflection after the object surface receives light irradiation, and is generally described by using a ratio of incident irradiance of a point light source to reflection radiance in an observation direction, namely a Bidirectional Reflection Distribution Function (BRDF). The method for obtaining the reflection image of the surface of the object under the specific illumination condition and realizing the reconstruction of the object shape by means of the reflection characteristic is widely applied to the fields of machine vision, quality detection, remote sensing detection and the like, and the key for realizing the inner surface quality detection based on the machine vision is to obtain the reflection characteristic of the inner surface of the hole.
The measurement of the BRDF requires that irradiance and radiance are respectively obtained from different incidence and reflection angles by means of special equipment to be solved, the measurement result is discrete data of a specific incidence angle and a specific reflection angle, mathematical modeling is required to be carried out according to the limited measurement result in order to realize quality detection and three-dimensional reconstruction based on machine vision, and the BRDF value of any angle is obtained.
The BRDF modeling method comprises a theoretical analytical method and an engineering statistical method, wherein the BRDF model is established by the theoretical analytical method by means of an optical principle and calculus, parameters of the model have actual physical significance, such as a T-S model, an M-B model and the like, and the theoretical analytical method is high in accuracy and large in calculated amount. The engineering statistical method is a more common method, and the method expresses BRDF by using a relatively simple function on the basis of experience, and then fits variables in the function by using experimental results to improve the generalization capability of the model, wherein a Phong model, a five-parameter model and the like are commonly used.
Patent application CN200810226219.8 discloses a real object material reflection attribute modeling method based on a linear light source, wherein a specific angle BRDF value is determined by a direct measurement method under the condition of the linear light source, and three modeling parameters are obtained based on a Word model. Patent application CN201610157317.5 discloses an illumination modeling method based on real material measurement data, which comprises the steps of obtaining BTF material BRDF by a direct measurement method, decomposing the BTF material BRDF into two parts of geometric information and illumination information by analysis, and utilizing the illumination information to perform BRDF fitting and modeling. The BRDF direct measurement result adopted by the patent application is subjected to model derivation, and only BRDF measurement and modeling on the outer surface of the material can be realized. Patent application CN103699810A discloses a modeling method for rough surface microwave band bidirectional reflection distribution function, which is a typical BRDF indirect measurement method, wherein the rough surface bidirectional reflection distribution function is obtained according to the relation between rough surface scattering coefficient and bidirectional reflection distribution function, but the BRDF measurement only in microwave band can not realize the BRDF measurement under visible light frequency, and the BRDF measurement and modeling for the inner surfaces of hole, hole and cavity structures can not be realized.
Disclosure of Invention
In order to overcome the defects in the prior art, the invention aims to provide a hole inner surface bidirectional reflection distribution function measuring and modeling method, which is used for establishing a hole inner surface bidirectional reflection distribution function inBRDF statistical model based on a Phong model, has high accuracy and strong universality, can accurately describe the reflection characteristics of the hole inner surface, and can be applied to the detection of the quality of the hole inner surface based on machine vision.
In order to achieve the purpose, the invention adopts the technical scheme that:
a hole inner surface bidirectional reflection distribution function measuring and modeling method specifically comprises the following steps:
step 1, acquiring image brightness information of a known inner surface three-dimensional shape hole through an endoscope;
step 2, measuring BRDF (bidirectional reflectance distribution function) values of the inner surface of the hole according to the radiant irradiance of each point of the inner surface of the hole and the reflected radiance of the direction reflected into a lens;
step 3, establishing an inBRDF model of the bidirectional reflection distribution function of the inner surface of the hole according to the relation between the BRDF value of the inner surface of the hole and the three-dimensional shape of the hole;
and 4, measuring the BRDF value of the inner surface of the specific material hole through experiments, and determining the optimal parameter of the inBRDF model of the specific material by adopting a genetic algorithm.
The step 1 specifically comprises the following steps:
1) fixing a movable lifting platform 1 on an optical workbench 2, fixing an endoscope probe 3 downwards at the front end of a workbench cantilever 4, vertically placing a hole part 5 on the movable lifting platform 1 and fixing the hole part by a clamp 6, adjusting the horizontal direction of the movable lifting platform 1 and observing an endoscope display 7, and when the center of an image is basically coincident with the axis of the hole, adjusting the vertical direction of the movable lifting platform 1 to ensure that the endoscope probe 3 is deep into the hole part 5 and obtain an endoscopic image 8;
2) after obtaining the endoscopic image 8, firstly carrying out pretreatment, including gray level conversion and median filtering noise reduction; secondly, extracting an edge information image of the endoscopic image by adopting an edge detection algorithm; then, determining the center coordinates of the edge information image of the endoscopic image, namely the center 9 of the endoscopic image, by adopting a Hough transformation method;
3) taking the center 9 of the endoscopic image as the center of a circle, intercepting a circular ring image 10 of the inner surface of the hole with the width of h pixels, adopting a bilinear interpolation algorithm to convert the circular ring image 10 into an expanded image 11 of the inner surface of the hole, and finally obtaining the digital brightness information of the image.
The step 2 specifically comprises the following steps:
the center of the front end face of the endoscope probe 3 is an endoscope lens 12, a plurality of LEDs are uniformly distributed around the endoscope lens as an endoscope light source 13, and the distance between the endoscope light source 13 and the endoscope lens 12 is 1-3 mm; when the main optical axis of the endoscope lens 12 coincides with the hole axis, the central position of the endoscope lens is set as the space coordinate origin O, the distance from the endoscope light source 13 to the center O of the endoscope lens is set to be delta, and when m point light sources have the same luminous intensity I0When the light is irradiated to the Q (x, y, z) point on the inner surface of the hole, the incident direction vectors can be respectively expressed as
Figure RE-GDA0003686893270000041
The inner surface is used as a secondary light source to perform mirror reflection and diffuse reflection on the received light energy, wherein the energy in the mirror reflection direction cannot enter the lens of the endoscope and only in the receiving direction
Figure RE-GDA0003686893270000042
The light energy of the diffuse reflection part of (a) is imaged as a point q (x ', y ', z ') on the endoscope light sensing plane 14 and finally converted into the digital image brightness on the endoscope display 8; according to the cosine law of luminous intensity, the incident irradiance E of the point Q can be expressed as formula (1):
Figure RE-GDA0003686893270000043
wherein alpha is1、α2…αmRespectively in the normal direction of the point Q
Figure RE-GDA0003686893270000044
And the incident direction of the light source
Figure RE-GDA0003686893270000045
Figure RE-GDA0003686893270000046
Angle of (l)1、l2…lmThe optical paths from the point light sources to the Q point on the inner surface of the hole part 5 can be expressed as
Figure RE-GDA0003686893270000047
Figure RE-GDA0003686893270000048
The spatial coordinates (x, y, z) and the normal direction (n) of each point on the inner surface of the hole part 5x,ny, nz) The method can be obtained by establishing a curved surface equation through the known three-dimensional morphology in the hole, establishing the curved surface equation for any hole with the known inner surface morphology, drawing a point cloud distribution diagram on the inner surface of the hole by using Matlab software, and then acquiring a normal vector (n) of each point by using a pcnormals functionx,ny,nz) (ii) a Obtaining coordinates (x, y, z) of each point on the inner surface of the hole part 5 and the normal direction through surface modeling
Figure RE-GDA0003686893270000059
(nx, ny,nz) Then, according to the vector relation, the following results are obtained:
Figure RE-GDA0003686893270000051
Figure RE-GDA0003686893270000052
Figure RE-GDA0003686893270000053
Figure RE-GDA0003686893270000054
substituting equation (2-1) … … above for equation (2-m) into equation (1), the incident irradiance received at the surface Q-point on the bore part 5 can be expressed as:
Figure RE-GDA0003686893270000055
in the optical imaging system, a quantitative relation shown in formula (4) exists between the reflection radiance L of any point of the surface of an object in the receiving direction of a lens and the incident irradiance E' of the point in a photosensitive plane:
Figure RE-GDA0003686893270000056
where E' is the incident irradiance on the endoscope photosurface 14, i.e., the image brightness, τ is the transmittance of the optical system, d is the lens diameter, f is the endoscope focal length,
Figure RE-GDA0003686893270000057
for the lens receiving direction and the endoscopeThe included angle of the main optical axis direction of the lens 12 can be expressed as
Figure RE-GDA0003686893270000058
The reflection radiance L of each point on the inner surface of the hole part 5 can be obtained according to the brightness information of the endoscopic image obtained in the step 1 and is shown in a formula (5):
Figure RE-GDA0003686893270000061
according to the definition of the bidirectional reflection distribution function, the BRDF value is equal to the ratio of the radiance L of the receiving direction of the lens on the surface of the object to the irradiance E of the incident direction of the light source, and the luminous intensity I of the specific endoscopic imaging system0The focal length f, the diameter d of the lens, the transmittance tau and the distance delta between the light source and the lens are all determined by the equipment, and eta is 4f2/I0·π·τ·d2Then, the bidirectional reflectance distribution function in the endoscopic imaging environment can be expressed as formula (6):
Figure RE-GDA0003686893270000062
in the formula (6), η is determined by the endoscope itself used when the inner surface of the hole is photographed, E' is the image brightness of each point in the endoscopic image, and the incident radiation E received by the point Q on the inner surface of the hole member 5 can be obtained by the formula (3).
The step 3 specifically comprises the following steps:
from the formula (6), under the condition of endoscope imaging, the BRDF value of the inner surface of the hole is determined by the space position and three-dimensional shape of each point of the inner surface, namely
Figure RE-GDA0003686893270000063
Because the positions of the endoscope lens 12 and the endoscope point light source 13 are fixed in the endoscopic imaging environment, the space coordinates of any point on the inner surface of the hole part 5 not only determine the incidence and reflection directions of each point, but also further determine the optical path l of the incident light1、l2、…lmReflected light in the direction of the lensWhen the distance delta between the endoscope light source and the lens is ignored, the optical path of the incident light is approximately equal to the optical path of the reflected light in the direction of the lens, namely l1=l2=...=lmI, the angle of incidence of each point source is approximately equal to the angle of reflection, i.e. θ1=θ2=...=θmθ; at this time, the factors affecting the BRDF on the inner surface of the hole can be summarized as two independent variables, namely, the BRDF is equal to f (θ, l), where θ reflects the three-dimensional morphology of the inner surface of the hole, and l reflects the spatial distance from the lens to the surface of the object; because the energy attenuation of light is very slow along with the increase of the distance in the propagation process, the factor influencing the BRDF on the inner surface of the hole under the peeping environment is considered to be the light deflection angle, namely BRDF is f (theta);
the light deflection angle of each point in the hole can be solved according to the space coordinates and the three-dimensional appearance of each point on the inner surface of the hole as shown in a formula (7):
Figure RE-GDA0003686893270000071
respectively obtaining BRDF values of all points on the inner surface of the hole and corresponding light deflection angles according to a formula (6) and a formula (7), counting the BRDF values and establishing a mathematical model as the measurement result is still discrete data, and obtaining the BRDF value of any angle; the BRDF of the inner surface of the hole can be regarded as being composed of a specular reflection component and a diffuse reflection component of a tangent plane, so that an inner surface inBRDF model is established based on an improved Phong model, and the relation between the BRDF value and a ray deflection angle theta, namely the three-dimensional morphology is established; the inBRDF model is shown in equation (8):
Figure RE-GDA0003686893270000072
wherein the first term on the right side of the equal sign represents the diffuse reflection component, following Lambert's law, and the second term represents the specular reflection component, exp (-a θ)b) Is a function of the internal surface topography profile. k is a radical ofd、ksA, b are undetermined parameters, where kdAnd ksCharacterised by diffuse and specular componentsThe size is related to the material and the reflectivity of the inner surface, and the a and the b represent the light deflection angle distribution of each point of the inner surface of the hole and are related to the three-dimensional appearance.
The step 4 specifically comprises the following steps:
in order to obtain abundant three-dimensional appearance and normal distribution of the inner surface of the hole, holes with abundant inner appearance or a plurality of holes with different apertures made of the same material are adopted for measurement to obtain a group of BRDF measurement data fmAnd its corresponding ray declination angle theta.
Determining the optimal parameters of the inBRDF model by adopting a genetic algorithm, and substituting the measured ray deflection angle theta into the inBRDF model to obtain a group of BRDF calculation data fcMeasuring data fmAnd calculating data fcThe Root Mean Square Error (RMSE) of (a) is minimized as a target to establish a genetic algorithm objective function, which can be expressed as formula (9):
E(kd,ks,a,b)=min∑g(θ)×[fc(kd,ks,a,b)-fm(θ)]2 (9)
where g (θ) is a weighting function of the adjustment error.
The specific method for determining the center 9 of the endoscopic image by using the Hough transform method in the step 1 and the step 2) comprises the following steps:
the Hough transformation detects an object with a specific shape through a summation algorithm, and the process is to obtain a result meeting the requirement of the specific shape by calculating the maximum value of a summation result in a parameter space; the Hough transform is specified as follows:
a) detecting an image edge according to the discontinuity of the image gray value, and obtaining and extracting boundary points;
b) converting the general equation (a-x)2+ (b-y)2 ═ R2 of the circle from the x-y coordinate system to the parameter space a-b coordinate system, so that a point on the circular arc in the x-y coordinate system corresponds to a circle in the parameter space;
c) a plurality of points are arranged on an arc in an x-y coordinate system, and a plurality of circles are arranged in the a-b coordinate system correspondingly; because points on the same arc in the x-y coordinate system have the same circle center, the corresponding circles of the points in the parameter space are intersected at one point, and the point is the circle center of the corresponding arc;
d) and counting the number of circles at the local intersection points, wherein the maximum value of the number of circles can be determined as the center of the original image, namely the center 9 of the endoscopic image.
The basic process of determining the optimal parameters of the inBRDF model by adopting the genetic algorithm in the step 4 is as follows: randomly generating a set of initial parameters (k) between given value ranges, e.g. (0-10)d,ksA, b), calculating the sum of the root mean square errors of all data according to the initial parameters, then changing the parameters according to the data iteration step size and continuously solving the sum of the root mean square errors, stopping iteration when the sum of the root mean square errors is smaller than a specific threshold or the iteration step number reaches a specified maximum value, and outputting a group of parameters (k) when the sum of the root mean square errors is minimumd,ksAnd a, b), namely the optimal parameters of the inBRDF model of the material.
The invention has the beneficial effects that:
1. the method for measuring the bidirectional reflection distribution function of the inner surface of the hole overcomes the contradiction that the BRDF direct measurement method needs to continuously change the shooting angle and the space in the hole is limited, has high efficiency and strong equipment universality, and can be applied to BRDF measurement of the inner surfaces of different holes, cavities and other structures.
2. The inBRDF modeling method provided by the invention is simple and convenient to operate, has small modeling difficulty, and can quickly and accurately obtain the BRDF value of the continuous angle of the inner surface of the hole.
3. The inBRDF model established by the invention has high fitting precision, can better describe the reflection characteristic of the inner surface of the hole and can be applied to the quality detection of the inner surface of the hole based on machine vision.
Drawings
Fig. 1 is a flow chart of BRDF measurement and modeling of the inner surface of the hole.
Fig. 2 is a schematic structural view of an endoscopic image capturing device for the inner surface of a hole.
Fig. 3 is a schematic diagram of the basic principle of endoscopic image processing.
Fig. 4 is a schematic view of the imaging principle of the endoscope.
In the figure: 1. a movable lifting platform; 2. an optical bench; 3. an endoscopic probe; 4. A table boom; 5. a bore component; 6. a part clamp; 7. an endoscopic display; 8. endoscopic images; 9. the center of the endoscopic image; 10. a circular ring image of the inner surface of the hole; 11. developing an image on the inner surface of the hole; 12. an endoscope lens; 13. an endoscope light source; 14. the endoscope is provided with a light sensing plane.
Detailed Description
The structural and operational principles of the present invention will be described in further detail below with reference to the accompanying drawings.
A hole inner surface bidirectional reflection distribution function measurement and modeling method comprises the following specific measurement schemes:
step 1, acquiring image brightness information of a known inner surface three-dimensional shape hole through an endoscope:
1) fixing a movable lifting platform 1 on an optical workbench 2, fixing an endoscope probe 3 downwards at the front end of a workbench cantilever 4, vertically placing a hole part 5 on the movable lifting platform 1 and fixing the hole part by a clamp 6, adjusting the horizontal direction of the movable lifting platform 1 and observing an endoscope display 7, and when the center of an image is basically coincident with the axis of the hole, adjusting the vertical direction of the movable lifting platform 1 to enable the endoscope probe 3 to be deep into the hole part 5 and obtain an endoscopic image 8, see fig. 1; the endoscopic image capture scheme is shown in figure 2.
2) After the endoscopic image 8 is obtained, preprocessing is firstly carried out, including gray level conversion and median filtering noise reduction; secondly, extracting an edge information image of the endoscopic image by adopting an edge detection algorithm; then, determining the center coordinates of the edge information image of the endoscopic image, namely the center 9 of the endoscopic image, by using a Hough transformation method, which comprises the following specific steps:
the Hough transformation detects an object with a specific shape through a summation algorithm, and the process is to obtain a result meeting the requirement of the specific shape by calculating the maximum value of a summation result in a parameter space; the Hough transform is specified as follows:
a) detecting an image edge according to the discontinuity of the image gray value, and obtaining and extracting boundary points;
b) general equation (a-x) of the circle2+(b-y)2=R2From the x-y coordinate system to the a-b coordinate system of the parameter space, one of the arcs in the x-y coordinate systemPoint corresponds to a circle in the parameter space;
c) a plurality of points are arranged on an arc in an x-y coordinate system, and a plurality of circles are arranged in a corresponding a-b coordinate system; because points on the same arc in the x-y coordinate system have the same circle center, the corresponding circles of the points in the parameter space are intersected at one point, and the point is the circle center of the corresponding arc;
d) and counting the number of circles at the local intersection points, wherein the maximum value of the number of circles can be determined as the center of the original image, namely the center 9 of the endoscopic image.
3) Taking the center 9 of the endoscopic image as the center of a circle, intercepting a circular ring image 10 of the inner surface of the hole with the width of h pixels, adopting a bilinear interpolation algorithm to convert the circular ring image 10 into an expanded image 11 of the inner surface of the hole, and finally obtaining the digital brightness information of the image. The endoscopic image processing basic principle is shown in fig. 3.
Step 2, according to the radiant irradiance of each point on the inner surface of the hole and the reflection radiance reflected into the lens direction, measuring the BRDF value of the inner surface of the hole:
the center of the front end face of the endoscope probe 3 is an endoscope lens 12, a plurality of LEDs are uniformly distributed around the endoscope lens as an endoscope light source 13, and the distance between the endoscope light source 13 and the endoscope lens 12 is 1-3 mm; when the main optical axis of the endoscope lens 12 coincides with the hole axis, the central position of the endoscope lens is set as the space coordinate origin O, the distance from the endoscope light source 13 to the center O of the endoscope lens is set to be delta, and when m point light sources have the same luminous intensity I0When the light is irradiated to the Q (x, y, z) point on the inner surface of the hole, the incident direction vectors can be respectively expressed as
Figure RE-GDA0003686893270000121
The inner surface is used as a secondary light source to perform mirror reflection and diffuse reflection on the received light energy, wherein the energy in the mirror reflection direction cannot enter the lens of the endoscope and only in the receiving direction
Figure RE-GDA0003686893270000122
Can be imaged as a point q (x ', y ', z ') on the endoscope light sensing plane 14 and ultimately converted to digital image brightness on the endoscope display 8. The imaging principle of the endoscope is shown in figure 4Shown in the figure. The incident irradiance E of the point Q can be expressed as formula (1) according to the cosine law of the luminous intensity;
Figure RE-GDA0003686893270000123
wherein alpha is1、α2…αmRespectively in the normal direction of the point Q
Figure RE-GDA0003686893270000124
And the incident direction of the light source
Figure RE-GDA0003686893270000125
Figure RE-GDA0003686893270000126
Angle of (l)1、l2…lmThe optical paths from the point light sources to the Q point on the inner surface of the hole part 5 can be expressed as
Figure RE-GDA0003686893270000127
Figure RE-GDA0003686893270000128
The spatial coordinates (x, y, z) and the normal direction (n) of each point on the inner surface of the hole part 5x,ny, nz) The method can be obtained by establishing a curved surface equation through the known three-dimensional shape in the hole, establishing the curved surface equation for any hole with the known inner surface shape, drawing a point cloud distribution diagram on the inner surface in the hole by using Matlab software, and then acquiring a normal vector (n) of each point by using a pcnormals functionx,ny,nz) (ii) a Obtaining coordinates (x, y, z) of each point on the inner surface of the hole part 5 and the normal direction through surface modeling
Figure RE-GDA00036868932700001210
(nx, ny,nz) Then, according to the vector relation, the following results are obtained:
Figure RE-GDA0003686893270000129
Figure RE-GDA0003686893270000131
Figure RE-GDA0003686893270000132
Figure RE-GDA0003686893270000133
substituting equation (2-1) … … above for equation (2-m) into equation (1), the incident irradiance received at the surface Q-point on the bore part 5 can be expressed as:
Figure RE-GDA0003686893270000134
in the optical imaging system, a quantitative relation shown in formula (4) exists between the reflection radiance L of any point of the surface of an object in the receiving direction of a lens and the incident irradiance E' of the point in a photosensitive plane.
Figure RE-GDA0003686893270000135
Where E' is the incident irradiance on the endoscope photosurface 14, i.e., the image brightness, τ is the transmittance of the optical system, d is the lens diameter, f is the endoscope focal length,
Figure RE-GDA0003686893270000136
the angle between the lens receiving direction and the main optical axis direction of the endoscope lens 12 can be expressed as
Figure RE-GDA0003686893270000137
Obtaining the brightness of the endoscopic image according to the step 1The information is obtained by obtaining the reflection radiance L of each point on the inner surface of the hole part 5 as shown in the formula (5).
Figure RE-GDA0003686893270000138
The BRDF value is equal to the ratio of the radiance L of the lens receiving direction of the object surface to the irradiance E of the incident direction of the light source according to the definition of the bidirectional reflection distribution function. For a particular endoscopic imaging system, the luminous intensity I0The focal length f, the lens diameter d, the transmittance tau, and the distance delta between the light source and the lens are determined by the device itself, where eta is 4f2/I0·π·τ·d2Then, the bidirectional reflectance distribution function in the endoscopic imaging environment can be expressed as formula (6):
Figure RE-GDA0003686893270000141
in the formula (6), η is determined by the endoscope itself used when the inner surface of the hole is photographed, E' is the image brightness of each point in the endoscopic image, and the incident radiation E received by the point Q on the inner surface of the hole member 5 can be obtained by the formula (3).
Step 3, establishing a hole inner surface bidirectional reflection distribution function inBRDF model according to the relation between the BRDF value of the hole inner surface and the three-dimensional shape of the hole inner surface:
from the formula (6), under the condition of endoscope imaging, the BRDF value of the inner surface of the hole is determined by the space position and three-dimensional shape of each point of the inner surface, namely
Figure RE-GDA0003686893270000142
Because the positions of the endoscope lens 12 and the endoscope point light source 13 are fixed in the endoscopic imaging environment, the space coordinates of any point on the inner surface of the hole part 5 not only determine the incidence and reflection directions of each point, but also further determine the optical path l of the incident light1、l2、…lmAnd the optical path l of the reflected light in the direction of the lens, wherein the optical path of the incident light is approximately equal to the optical path in the direction of the reflected light into the lens after neglecting the distance delta between the light source of the endoscope and the lens, namely1=l2=...=lmI, the angle of incidence of each point source is approximately equal to the angle of reflection, i.e. θ1=θ2=...=θmθ; at this time, the factors affecting the BRDF on the inner surface of the hole can be summarized as two independent variables, namely, the BRDF is equal to f (θ, l), where θ reflects the three-dimensional morphology of the inner surface of the hole, and l reflects the spatial distance from the lens to the surface of the object; because the energy attenuation of light is very slow along with the increase of the distance in the propagation process, the factor influencing the BRDF on the inner surface of the hole under the peeping environment is considered to be the light deflection angle, namely BRDF is f (theta);
the light deflection angle of each point in the hole can be solved according to the space coordinates and the three-dimensional appearance of each point on the inner surface of the hole as shown in a formula (7):
Figure RE-GDA0003686893270000151
respectively obtaining BRDF values of all points on the inner surface of the hole and corresponding light deflection angles according to a formula (6) and a formula (7), counting the BRDF values and establishing a mathematical model as the measurement result is still discrete data, and obtaining the BRDF value of any angle; the BRDF of the inner surface of the hole can be considered to be composed of a specular reflection component and a diffuse reflection component of a tangent plane, so that an inBRDF model of the inner surface is established based on an improved Phong model, and the relation between the BRDF value and a light deflection angle theta, namely the three-dimensional morphology is established; the inBRDF model is shown in formula (8):
Figure RE-GDA0003686893270000152
wherein the first term on the right side of the equal sign represents the diffuse reflection component, following Lambert's law, and the second term represents the specular reflection component, exp (-a θ)b) Is a function of the internal surface topography profile. k is a radical ofd、ksA, b are undetermined parameters, where kdAnd ksThe sizes of diffuse reflection components and specular reflection components are respectively represented and are related to the inner surface material and the reflectivity, and the a and b represent the light deflection angle distribution of each point on the inner surface of the hole and are related to the three-dimensional shape.
Step 4, measuring the BRDF value of the inner surface of the specific material hole by an experiment, and determining the optimal parameter of the inBRDF model of the specific material by adopting a genetic algorithm:
in order to obtain abundant three-dimensional morphology and normal distribution of the inner surface of the hole, the hole with abundant inner morphology or a plurality of holes with different apertures made of the same material are adopted for measurement to obtain a group of BRDF measurement data fmAnd its corresponding ray declination angle theta.
Determining the optimal parameters of the inBRDF model by adopting a genetic algorithm, and substituting the measured ray deflection angle theta into the inBRDF model to obtain a group of BRDF calculation data fcMeasuring data fmAnd calculating data fcThe Root Mean Square Error (RMSE) of (a) is minimized as a target to establish a genetic algorithm objective function, which can be expressed as formula (9):
E(kd,ks,a,b)=min∑g(θ)×[fc(kd,ks,a,b)-fm(θ)]2 (9)
where g (θ) is a weighting function of the adjustment error.
The basic process for determining the optimal parameters of the inBRDF model by adopting the genetic algorithm is as follows: randomly generating a set of initial parameters (k) between given value ranges, e.g. (0-10)d,ksA, b), calculating the sum of the root mean square errors of all data according to the initial parameters, then carrying out parameter change according to the data iteration step size such as 0.01 and continuously solving the sum of the root mean square errors, stopping iteration when the sum of the root mean square errors is smaller than a specific threshold value or the iteration step number reaches a specified maximum value, and outputting a group of parameters (k) when the sum of the root mean square errors is minimumd,ksAnd a, b), namely the optimal parameters of the inBRDF model of the material.
The gray level conversion, the median filtering and the edge detection in the step 2) are all conventional basic operations in the field of image processing, and are not repeated.
The above embodiments are only intended to illustrate the technical solutions of the present invention, but not to be exhaustive or limited, and it should be understood by those skilled in the art that modifications may be made to the technical solutions described in the foregoing embodiments, or equivalent replacements may be made to some technical features, and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the scope of the claims of the present invention.

Claims (7)

1. A hole inner surface bidirectional reflection distribution function measurement and modeling method is characterized in that: the specific measurement scheme comprises:
step 1, acquiring image brightness information of a known inner surface three-dimensional shape hole through an endoscope;
step 2, measuring BRDF (bidirectional reflectance distribution function) values of the inner surface of the hole according to the radiant irradiance of each point of the inner surface of the hole and the reflected radiance of the direction reflected into a lens;
step 3, establishing an inBRDF model of the bidirectional reflection distribution function of the inner surface of the hole according to the relation between the BRDF value of the inner surface of the hole and the three-dimensional shape of the hole;
and 4, measuring the BRDF value of the inner surface of the hole of the specific material through experiments, and determining the optimal parameter of the inBRDF model of the specific material by adopting a genetic algorithm.
2. The method of claim 1, wherein the method comprises: the step 1 specifically comprises the following steps:
1) fixing a movable lifting platform (1) on an optical workbench (2), fixing an endoscope probe (3) at the front end of a workbench cantilever (4) downwards, vertically placing a hole part (5) on the movable lifting platform (1) and fixing the hole part by a clamp (6), adjusting the horizontal direction of the movable lifting platform (1) and observing an endoscope display (7), and adjusting the vertical direction of the movable lifting platform (1) when the center of an image is basically coincident with the axis of the hole, so that the endoscope probe (3) is deep into the hole part (5) and obtains an endoscopic image (8);
2) after the endoscopic image (8) is obtained, preprocessing is firstly carried out, including gray level conversion and median filtering noise reduction; secondly, extracting an edge information image of the endoscopic image by adopting an edge detection algorithm; then, determining the center coordinate of the edge information image of the endoscopic image, namely the center (9) of the endoscopic image, by adopting a Hough transformation method;
3) the method comprises the steps of taking the center (9) of an endoscopic image as the center of a circle, intercepting a circular ring image (10) of the inner surface of a hole with the width of h pixels, converting the circular ring image (10) into an expanded image (11) of the inner surface of the hole by adopting a bilinear interpolation algorithm, and finally obtaining digital brightness information of the image.
3. The method of claim 1, wherein the method comprises: the step 2 specifically comprises the following steps:
the center of the front end face of the endoscope probe (3) is an endoscope lens (12), a plurality of LEDs are uniformly distributed around the endoscope probe as an endoscope light source (13), and the distance between the endoscope light source (13) and the endoscope lens (12) is 1-3 mm; when the main optical axis of the endoscope lens (12) is coincident with the axis of the hole, the central position of the endoscope lens is set as a space coordinate origin O, the distance from the endoscope light source (13) to the center O of the endoscope lens is delta, and when m point light sources have the same luminous intensity I0When the light is irradiated to the Q (x, y, z) point on the inner surface of the hole, the incident direction vectors can be respectively expressed as
Figure RE-FDA0003686893260000021
The inner surface is used as a secondary light source to perform mirror reflection and diffuse reflection on the received light energy, wherein the energy in the mirror reflection direction cannot enter the lens of the endoscope and only has the receiving direction
Figure RE-FDA0003686893260000022
The light energy of the diffuse reflection part of (a) is imaged as a point q (x ', y ', z ') on the endoscope photosensitive plane (14) and finally converted into the digital image brightness on the endoscope display (8); according to the cosine law of luminous intensity, the incident irradiance E of the point Q can be expressed as formula (1):
Figure RE-FDA0003686893260000023
wherein alpha is1、α2…αmRespectively in the normal direction of the point Q
Figure RE-FDA0003686893260000024
And the incident direction of the light source
Figure RE-FDA0003686893260000025
Figure RE-FDA0003686893260000026
Angle of (l)1、l2…lmThe optical paths from the point light sources to the Q point on the inner surface of the hole part (5) can be expressed as
Figure RE-FDA0003686893260000027
Figure RE-FDA0003686893260000028
The space coordinates (x, y, z) and the normal direction (n) of each point on the inner surface of the hole part (5)x,ny,nz) The method can be obtained by establishing a curved surface equation through the known three-dimensional morphology in the hole, establishing the curved surface equation for any hole with the known inner surface morphology, drawing a point cloud distribution diagram on the inner surface of the hole by using Matlab software, and then acquiring a normal vector (n) of each point by using a pcnormals functionx,ny,nz) (ii) a Obtaining coordinates (x, y, z) of each point on the inner surface of the hole part (5) and normal direction by surface modeling
Figure RE-FDA0003686893260000031
Then, according to the vector relation, the following steps are found:
Figure RE-FDA0003686893260000032
Figure RE-FDA0003686893260000033
Figure RE-FDA0003686893260000034
Figure RE-FDA0003686893260000035
substituting equation (2-1) … … above for equation (2-m) into equation (1), the incident irradiance received at the surface Q-point on the bore part (5) can be expressed as:
Figure RE-FDA0003686893260000036
in the optical imaging system, a quantitative relation shown in formula (4) exists between the reflected radiance L of any point on the surface of an object in the receiving direction of a lens and the incident irradiance E' of the point on a photosensitive plane:
Figure RE-FDA0003686893260000041
wherein E' is incident irradiance on an endoscope photosensitive plane (14), namely image brightness, tau is the transmittance of an optical system, d is the diameter of a lens, f is the focal length of an endoscope,
Figure RE-FDA0003686893260000042
the included angle between the lens receiving direction and the main optical axis direction of the endoscope lens (12) can be expressed as
Figure RE-FDA0003686893260000043
The reflection radiance L of each point on the inner surface of the hole part (5) can be obtained according to the brightness information of the endoscopic image obtained in the step 1 and is shown in a formula (5):
Figure RE-FDA0003686893260000044
defining BRDF values from bi-directional reflectance distribution functionsEqual to the ratio of the radiance L of the receiving direction of the lens on the surface of the object to the irradiance E of the incident direction of the light source. For a particular endoscopic imaging system, its luminous intensity I0The focal length f, the diameter d of the lens, the transmittance tau and the distance delta between the light source and the lens are all determined by the equipment, and eta is 4f2/I0·π·τ·d2Then, the bidirectional reflectance distribution function in the endoscopic imaging environment can be expressed as formula (6):
Figure RE-FDA0003686893260000045
in the formula (6), η is determined by the endoscope itself used when the inner surface of the hole is photographed, E' is the image brightness of each point in the endoscopic image, and the incident radiation E received by the point Q on the inner surface of the hole member 5 can be obtained by the formula (3).
4. The method of claim 1, wherein the method comprises: the step 3 specifically comprises the following steps:
from the formula (6), under the condition of endoscope imaging, the BRDF value of the inner surface of the hole is determined by the space position and three-dimensional shape of each point of the inner surface, namely
Figure RE-FDA0003686893260000046
Because the positions of the endoscope lens (12) and the endoscope point light source (13) are fixed in the endoscopic imaging environment, the space coordinates of any point on the inner surface of the hole part (5) not only determine the incidence and reflection directions of each point, but also further determine the optical path l of the incident light1、l2、…lmAnd the optical path l of the reflected light in the direction of the lens, wherein the optical path of the incident light is approximately equal to the optical path in the direction of the reflected light into the lens after neglecting the distance delta between the light source of the endoscope and the lens, namely1=l2=...=lmL, the angle of incidence of each point source is approximately equal to the angle of reflection, i.e. θ1=θ2=...=θmθ; the factors influencing the BRDF on the inner surface of the hole can be summarized into two independent light beam deflection angles theta and light path lengths lA vertical variable, namely BRDF ═ f (θ, l), where θ reflects the three-dimensional topography of the inner surface of the hole and l reflects the spatial distance from the lens to the surface of the object; because the energy attenuation of light is very slow along with the increase of the distance in the propagation process, the factor influencing the BRDF on the inner surface of the hole under the peeping environment is considered to be the light deflection angle, namely BRDF is f (theta);
the light deflection angle of each point in the hole can be solved according to the space coordinates and the three-dimensional appearance of each point on the inner surface of the hole as shown in a formula (7):
Figure RE-FDA0003686893260000051
obtaining BRDF values of each point on the inner surface of the hole and corresponding light deflection angles according to a formula (6) and a formula (7), counting the BRDF values and establishing a mathematical model as the measurement result is still discrete data, and obtaining the BRDF value at any angle; the BRDF of the inner surface of the hole can be regarded as being composed of a specular reflection component and a diffuse reflection component of a tangent plane, so that an inner surface inBRDF model is established based on an improved Phong model, and the relation between the BRDF value and a ray deflection angle theta, namely the three-dimensional morphology is established; the inBRDF model is shown in equation (8):
Figure RE-FDA0003686893260000052
wherein the first term on the right side of the equal sign represents the diffuse reflection component, following Lambert's law, and the second term represents the specular reflection component, exp (-a θ)b) Is an internal surface topography distribution function; k is a radical ofd、ksA, b are undetermined parameters, where kdAnd ksRespectively representing the size of the diffuse reflection component and the specular reflection component, relating to the inner surface material and the reflectivity, and a and b representing the light deflection angle distribution of each point on the inner surface of the hole, relating to the three-dimensional appearance.
5. The method of claim 1, wherein the method comprises: the step 4 specifically comprises the following steps:
in order to obtain abundant three-dimensional appearance and normal distribution of the inner surface of the hole, holes with abundant inner appearance or a plurality of holes with different apertures made of the same material are adopted for measurement to obtain a group of BRDF measurement data fmAnd its corresponding ray declination angle θ;
determining the optimal parameters of the inBRDF model by adopting a genetic algorithm, and substituting the measured ray deflection angle theta into the inBRDF model to obtain a group of BRDF calculation data fcMeasuring data fmAnd calculating data fcThe Root Mean Square Error (RMSE) of (a) is minimized as a target to establish a genetic algorithm objective function, which can be expressed as formula (9):
E(kd,ks,a,b)=min∑g(θ)×[fc(kd,ks,a,b)-fm(θ)]2 (9)
where g (θ) is a weighting function of the adjustment error.
6. The method of claim 2, wherein the method comprises: the specific method for determining the center (9) of the endoscopic image by using the Hough transform method in the step 1 and the step 2) comprises the following steps: the Hough transformation detects an object with a specific shape through a summation algorithm, and the process is to obtain a result meeting the requirement of the specific shape by calculating the maximum value of a summation result in a parameter space; the Hough transform is specified as follows:
a) detecting an image edge according to the discontinuity of the image gray value, and obtaining and extracting boundary points;
b) general equation (a-x) of the circle2+(b-y)2=R2Converting the x-y coordinate system into a parameter space a-b coordinate system, and enabling a point on an arc in the x-y coordinate system to correspond to a circle in the parameter space;
c) a plurality of points are arranged on an arc in an x-y coordinate system, and a plurality of circles are arranged in the a-b coordinate system correspondingly; because points on the same arc in the x-y coordinate system have the same circle center, the corresponding circles of the points in the parameter space are intersected at one point, and the point is the circle center of the corresponding arc;
d) and counting the number of circles at the local intersection points, wherein the maximum value of the number of circles can be identified as the center of the original image, namely the center (9) of the endoscopic image.
7. The method of claim 5, wherein the method comprises: the basic process for determining the optimal parameters of the inBRDF model by adopting the genetic algorithm is as follows: randomly generating a set of initial parameters (k) between given value ranges, e.g. (0-10)d,ksA, b), calculating the sum of the root mean square errors of all data according to the initial parameters, then changing the parameters according to the data iteration step size and continuously solving the sum of the root mean square errors, stopping iteration when the sum of the root mean square errors is smaller than a specific threshold or the iteration step number reaches a specified maximum value, and outputting a group of parameters (k) when the sum of the root mean square errors is minimumd,ksAnd a, b), namely the optimal parameters of the inBRDF model of the material.
CN202210339025.9A 2022-04-01 2022-04-01 Hole inner surface bidirectional reflection distribution function measurement and modeling method Pending CN114741804A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210339025.9A CN114741804A (en) 2022-04-01 2022-04-01 Hole inner surface bidirectional reflection distribution function measurement and modeling method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210339025.9A CN114741804A (en) 2022-04-01 2022-04-01 Hole inner surface bidirectional reflection distribution function measurement and modeling method

Publications (1)

Publication Number Publication Date
CN114741804A true CN114741804A (en) 2022-07-12

Family

ID=82278858

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210339025.9A Pending CN114741804A (en) 2022-04-01 2022-04-01 Hole inner surface bidirectional reflection distribution function measurement and modeling method

Country Status (1)

Country Link
CN (1) CN114741804A (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160371880A1 (en) * 2013-08-30 2016-12-22 Honda Motor Co., Ltd. Design layer data creation device and method, and design simulation device
CN109684712A (en) * 2018-12-20 2019-04-26 国家卫星气象中心 A kind of creation method, device, electronic equipment and the storage medium of BRDF model
CN112149310A (en) * 2020-10-09 2020-12-29 哈尔滨工业大学 BRDF (bidirectional reflectance distribution function) modeling method for material surface in space proton irradiation environment

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160371880A1 (en) * 2013-08-30 2016-12-22 Honda Motor Co., Ltd. Design layer data creation device and method, and design simulation device
CN109684712A (en) * 2018-12-20 2019-04-26 国家卫星气象中心 A kind of creation method, device, electronic equipment and the storage medium of BRDF model
CN112149310A (en) * 2020-10-09 2020-12-29 哈尔滨工业大学 BRDF (bidirectional reflectance distribution function) modeling method for material surface in space proton irradiation environment

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
QIANG SHENG: "Measurement and modeling of reflection characteristics of hole inner surface based on endoscopic image", MEASUREMENT 190(2022)110742, 18 January 2022 (2022-01-18) *
史卫朝;郑建明;李言;李旭波;安倩楠;: "加工表面双向反射分布函数的测量与建模", 光学学报, no. 10, 11 May 2018 (2018-05-11) *
梁琼;赵慧民;葛任贤;熊建斌;杨盛雄;苏冠元;: "基于椭圆拟合的改进Hough变换算法在眼控系统中的应用研究", 广东技术师范大学学报, no. 03, 25 June 2020 (2020-06-25) *
袁艳;孙成明;张修宝;: "空间目标表面材料光谱双向反射分布函数测量与建模", 物理学报, no. 03, 15 March 2010 (2010-03-15) *

Similar Documents

Publication Publication Date Title
Xu et al. Line structured light calibration method and centerline extraction: A review
Isa et al. Design and analysis of a 3D laser scanner
Zhang et al. A robust surface coding method for optically challenging objects using structured light
CN111707221B (en) Multi-exposure scattering signal fusion surface roughness measurement method
CN109141273A (en) A kind of high-speed moving object distortion measurement system and method based on DMD
CN118130477B (en) Die-casting alloy workpiece defect detection method and system based on visual recognition
Wang et al. Highly reflective surface measurement based on dual stereo monocular structured light system fusion
CN111344553B (en) Method and system for detecting defects of curved object
Fu et al. High dynamic range structured light 3-D measurement based on region adaptive fringe brightness
Sheng et al. Measurement and modeling of reflection characteristics of hole inner surface based on endoscopic image
Dong et al. Illumination variation robust circular target based on digital image correlation method
CN114741804A (en) Hole inner surface bidirectional reflection distribution function measurement and modeling method
CN115164776B (en) Three-dimensional measurement method and device for fusion of structured light decoding and deep learning
KR20020000443A (en) A 3-D shape measuring method and system using a adaptive area clustering
CN114241059B (en) Synchronous calibration method for camera and light source in photometric stereo vision system
CN112414316B (en) Strain gauge sensitive grid size parameter measuring method
CN112782201B (en) Lobster eye optical device calibration method based on X-ray focusing image
CN110006924B (en) Method for detecting two-dimensional profile of micro defect on surface of optical element
Li et al. Photometric stereo for general isotropic reflectances by spherical linear interpolation
CN114609094A (en) Inner surface BRDF (bidirectional reflectance distribution function) measuring method based on brightness of endoscopic image and normal distribution of inner surface of hole
CN108759720B (en) Smooth surface profile measuring method
CN112213704A (en) Target scattering cross section calculation method and device
Wang et al. Precision circular target location in vision coordinate measurement system
Svetkoff et al. Influence of object structure on the accuracy of 3-D systems for metrology
US20020110278A1 (en) Structure-guided image measurement method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination