WO2023028955A1 - Procédé et appareil de mesure - Google Patents

Procédé et appareil de mesure Download PDF

Info

Publication number
WO2023028955A1
WO2023028955A1 PCT/CN2021/116262 CN2021116262W WO2023028955A1 WO 2023028955 A1 WO2023028955 A1 WO 2023028955A1 CN 2021116262 W CN2021116262 W CN 2021116262W WO 2023028955 A1 WO2023028955 A1 WO 2023028955A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
level size
pixel
size
world
Prior art date
Application number
PCT/CN2021/116262
Other languages
English (en)
Inventor
Xiaohui Wang
Yinjie ZHENG
Guansheng LIANG
Qian Sun
Zhijin Zhang
Ke Xiao
Fusheng He
Hui WENG
Xing Liu
Caijun HANG
Shanshan Wang
Original Assignee
Telefonaktiebolaget Lm Ericsson (Publ)
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Telefonaktiebolaget Lm Ericsson (Publ) filed Critical Telefonaktiebolaget Lm Ericsson (Publ)
Priority to PCT/CN2021/116262 priority Critical patent/WO2023028955A1/fr
Publication of WO2023028955A1 publication Critical patent/WO2023028955A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/14—Measuring arrangements characterised by the use of optical techniques for measuring distance or clearance between spaced objects or spaced apertures
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/02—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness
    • G01B11/022—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness by means of tv-camera scanning
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B21/00—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant
    • G01B21/02—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant for measuring length, width, or thickness
    • G01B21/04—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant for measuring length, width, or thickness by measuring coordinates of points
    • G01B21/042—Calibration or calibration artifacts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30204—Marker

Definitions

  • the present disclosure generally relates to image processing, and more specifically, to a method and apparatus for measurement.
  • a device may be enhanced with the computer vision technology to extract the contour of a gap of an object in an image, acquire the gap width in pixels, and determine the gap width in the real world according to how many millimeters (mm) correspond to 1 pixel.
  • mm millimeters
  • the cost of manual detection may be very high because professional devices and workers may be needed.
  • the manual detection may introduce an artificial measurement error, as it is easy to misread the measurement result.
  • the computer vision technology may be applied for the gap width measurement, most solutions based on computer vision technology may have high failure rate of image calibration because an object such as a chessboard in an image may not be found, especially for the chessboard located in the complicated background.
  • obtaining image calibration parameters may be a prerequisite for gap width measurement. Therefore, it may be desirable to implement the gap width measurement in a more efficient way.
  • Various embodiments of the present disclosure propose a solution for measurement, which can enable the size of an object in an image, e.g., gap width, etc. to be measured more accurately by using the computer vision technology.
  • pixel-level size may refer to the size of an object (e.g., a gap, a hole, a screw, etc. ) measured at the pixel level or the image level.
  • the pixel-level width of a gap or the pixel-level distance between two points may be measured according to the pixel coordinates and represented in pixels.
  • the term “world-level size” described in this document may refer to the actual physical size of an object (e.g., a gap, a hole, a screw, etc. ) measured at the real-world level.
  • the world-level width of a gap or the world-level distance between two points may be measured according to the real-world physical coordinates and represented in metric units such as millimeter (mm) .
  • image segmentation may refer to an image processing procedure that can be used to segment or extract a part of an original image.
  • the image segmentation may be applied to identify the contour of an object in an image, so as to segment or extract the object part from the image.
  • a method for measurement which may be performed by an apparatus such as a server, a client device, etc.
  • the method comprises: measuring a pixel-level size of one or more markers in a first image by applying image segmentation to the one or more markers in the first image.
  • each of the one or more markers may be designed in concentric circles way and have a predefined world-level size.
  • the method further comprises: determining a mapping relationship between a pixel-level size and a world-level size, according to the measured pixel-level size and the predefined world-level size of the one or more markers.
  • each of the one or more markers may include multiple concentric circles and each circle may have a predefined world-level size (e.g., radius, diameter, perimeter, etc. ) .
  • the mapping relationship between the pixel-level size and the world-level size may be determined by: building up the mapping relationship between the pixel-level size and the world-level size, according to size measurements on one or more of the multiple concentric circles.
  • the mapping relationship between the pixel-level size and the world-level size may be determined further by: verifying the mapping relationship between the pixel-level size and the world-level size and/or accuracy of image calibration, according to size measurements on remaining of the multiple concentric circles.
  • the first image may be an image calibrated based at least in part on one or more distortion correction parameters.
  • the one or more distortion correction parameters may be determined by performing a training process based on chessboard segmentation.
  • the training process based on the chessboard segmentation may have chessboard images as input data and corresponding mask images as output data.
  • the chessboard segmentation can extract chessboard parts from the chessboard images by using the corresponding mask images.
  • the training process based on the chessboard segmentation may be performed by using a convolution neural network (e.g., a UNet model, etc. ) or any other suitable artificial intelligence algorithms (e.g., deep learning algorithms) .
  • a convolution neural network e.g., a UNet model, etc.
  • any other suitable artificial intelligence algorithms e.g., deep learning algorithms
  • the one or more distortion correction parameters may be used to calibrate a second image (e.g., by using an image calibration model such as OpenCV, etc. ) prior to applying image segmentation to an object in the second image.
  • an image calibration model such as OpenCV, etc.
  • the method according to the first aspect of the present disclosure may further comprise: measuring a pixel-level size of an object in a second image by applying image segmentation to the object in the second image.
  • the second image may be generated from a third image which includes the object and has a smaller size than the second image.
  • the method according to the first aspect of the present disclosure may further comprise: determining a world-level size of the object, according to the measured pixel-level size of the object and the mapping relationship between the pixel-level size and the world-level size.
  • the third image may be cropped from a fourth image which has a same size with the second image.
  • the position of the third image in the second image may be same as the position of the third image in the fourth image.
  • the third image may be a region of interest (ROI) portion from the fourth image
  • the fourth image may be the original image captured by a camera.
  • ROI region of interest
  • only the ROI portion from the original image may be transmitted to a server to generate the image to be measured, so as to save the amount of data transmission.
  • the third image may have the image quality meeting a predetermined criterion.
  • the image quality of the third image may be equal to or higher than a predefined quality level.
  • the image quality score of the third image may be equal to or less than a predefined score.
  • the image segmentation applied to the object in the second image may be implemented in a segmentation model (e.g., a UNet model, etc. ) .
  • the segmentation model may be trained by using data augmentation with distortion parameter setting.
  • the pixel-level size of the object in the second image may be measured by: determining one or more boundaries of the object in the second image, according to a result of the image segmentation.
  • the one or more boundaries may be indicated by subpixel coordinates of a set of segment points.
  • the one or more boundaries may include a first boundary and a second boundary.
  • the set of segment points may include one or more segment points on the first boundary and one or more corresponding segment points on the second boundary.
  • the pixel-level size of the object in the second image may be represented by one or more distance values.
  • Each of the one or more distance values may indicate a distance between a segment point on the first boundary and a corresponding segment point on the second boundary.
  • the method according to the first aspect of the present disclosure may further comprise: triggering an alarm in response to one or more of the following events:
  • ⁇ an average value of the one or more distance values is equal to or larger than a first threshold
  • ⁇ a difference between the largest distance value and the smallest distance value among the one or more distance values is equal to or larger than a second threshold
  • ⁇ an average value of the one or more distance values is equal to or less than a third threshold
  • ⁇ a difference between the largest distance value and the smallest distance value among the one or more distance values is equal to or less than a fourth threshold.
  • first, second, third and fourth thresholds mentioned above may be predefined as the same value or different values, depending on various application requirements.
  • an apparatus which may be implemented as a server or a client device, etc.
  • the apparatus may comprise one or more processors and one or more memories comprising computer program codes.
  • the one or more memories and the computer program codes may be configured to, with the one or more processors, cause the apparatus at least to measure a pixel-level size of one or more markers in a first image by applying image segmentation to the one or more markers in the first image.
  • each of the one or more markers may be designed in concentric circles way and have a predefined world-level size.
  • the one or more memories and the computer program codes may be configured to, with the one or more processors, cause the apparatus at least further to determine a mapping relationship between a pixel-level size and a world-level size, according to the measured pixel-level size and the predefined world-level size of the one or more markers.
  • the one or more memories and the computer program codes may be configured to, with the one or more processors, cause the apparatus according to the second aspect of the present disclosure at least to perform any step of the method according to the first aspect of the present disclosure.
  • a computer-readable medium storing computer program codes which, when executed on a computer, cause the computer to perform any step of the method according to the first aspect of the present disclosure.
  • Various exemplary embodiments according to the present disclosure can enable size measurement (e.g., gap width measurement, etc. ) to be implemented in an easier and more accurate way.
  • the special design of a marker can make the mapping relationship between the pixel-level size and the world-level size determined by using the marker more precise and generalized.
  • application of various exemplary embodiments can enhance measurement performance and resource efficiency, e.g., by processing only the ROI part of the image, and/or training the image segmentation/calibration model to improve robustness, etc.
  • Fig. 1A is a diagram illustrating exemplary chessboard segmentation according to an embodiment of the present disclosure
  • Fig. 1B is a diagram illustrating an exemplary contour annotation according to an embodiment of the present disclosure
  • Fig. 1C is a diagram illustrating an exemplary mask image according to an embodiment of the present disclosure
  • Fig. 1D is a diagram illustrating exemplary training pairs according to an embodiment of the present disclosure
  • Fig. 2A is a diagram illustrating an exemplary marker according to an embodiment of the present disclosure
  • Fig. 2B is a diagram illustrating a set of exemplary markers according to an embodiment of the present disclosure
  • Fig. 2C is a diagram illustrating an exemplary image with the tested object and markers according to an embodiment of the present disclosure
  • Fig. 2D is a diagram illustrating exemplary segmentation results for inner circles of different markers according to an embodiment of the present disclosure
  • Fig. 2E is a diagram illustrating exemplary verification of measurement results by other circles according to an embodiment of the present disclosure
  • Fig. 2F is a diagram illustrating exemplary generation of anomalies according to an embodiment of the present disclosure.
  • Fig. 3A is a diagram illustrating an exemplary ROI according to an embodiment of the present disclosure
  • Fig. 3B is a diagram illustrating an exemplary check of image quality according to an embodiment of the present disclosure
  • Fig. 3C is a diagram illustrating an exemplary image generated from a ROI part according to an embodiment of the present disclosure
  • Fig. 3D is a diagram illustrating exemplary gap segmentation results according to an embodiment of the present disclosure.
  • Fig. 3E is a diagram illustrating exemplary upper and lower boundaries of a gap according to an embodiment of the present disclosure
  • Fig. 3F is a diagram illustrating an exemplary subpixel representation according to an embodiment of the present disclosure.
  • Figs. 4A-4B are diagrams illustrating exemplary use cases of measurement according to some embodiments of the present disclosure.
  • Fig. 5 is a flowchart illustrating a procedure of gap width measurement according to an embodiment of the present disclosure
  • Fig. 6 is a flowchart illustrating a method according to an embodiment of the present disclosure.
  • Fig. 7 is a block diagram illustrating an apparatus according to an embodiment of the present disclosure.
  • the terms “first” , “second” and so forth refer to different elements.
  • the singular forms “a” and “an” are intended to include the plural forms as well, unless the context clearly indicates otherwise.
  • the term “based on” is to be read as “based at least in part on” .
  • the term “one embodiment” and “an embodiment” are to be read as “at least one embodiment” .
  • the term “another embodiment” is to be read as “at least one other embodiment” .
  • Other definitions, explicit and implicit, may be included below.
  • Size measurement plays an important role in the manufacturing process.
  • Various on-the-spot investigations show that the size measurement such as gap width measurement may just rely on workers manually using vernier caliper or other tools for measurement.
  • the cost of manual detection may be expensive, and it is easy to misread the measurement result, so the gap width may not be detected with high precision because of human eyes fatigue.
  • step iv) Using the corresponding relationship calculated in step ii) to acquire the size of the gap (e.g., the gap width) in the real world.
  • the existing solutions using the computer vision technology may have the following defects:
  • various exemplary embodiments of the present disclosure propose a solution for gap width measurement by using the computer vision technology.
  • the proposed solution can implement calibration in a more robust way, which can successfully conquer the issue that OpenCV fails to detect cross-corner points if the background of an image is complex.
  • the proposed solution may be easily deployed and provide a simple way to process a ROI part of an image so as to save bandwidth resource.
  • a marker with the pre-defined size may be utilized to calculate a mapping relationship between the pixel-level size and the world-level size. It may be easy to segment the marker from the image.
  • the design of markers may be beneficial to mapping relationship calculation and testing.
  • more abnormal samples for gap width measurement may be obtained by distorting them with different coefficients.
  • an image quality evaluator such as Blind/Referenceless Image Spatial QUality Evaluator (BRISQUE) may be used to check the image quality and the samples with bad quality may be abandoned.
  • BRISQUE Blind/Referenceless Image Spatial QUality Evaluator
  • a subpixel scheme may be used to make the measurement result more precise.
  • a solution for gap width measurement may be divided into two parts or phases, including Part I “calibration and training phase” and Part II “detection phase” .
  • Part I aims at getting a mapping relationship between a pixel-level size and a world-level size and acquiring a gap segmentation model based on artificial intelligence (e.g., a deep learning algorithm, a convolution neural network, etc. ) .
  • Part II aims at measuring the gap width of an object.
  • the two parts may include the following steps:
  • Step 1 Rectify an image by calibration.
  • an image segmentation model e.g., a convolution network for biomedical image segmentation such as UNet, etc.
  • an image calibration model e.g., OpenCV application programming interface (API) , etc.
  • API application programming interface
  • Step 2 Obtain standard markers (which may be generated by using AutoCAD) and paste the markers onto an object to be tested.
  • Step 3 Use an image segmentation model (e.g., a convolution network for biomedical image segmentation such as UNet, etc. ) to segment a marker and calculate the mapping relationship between a pixel-level size and a world-level size (e.g., between a pixel-level distance/width and a world-level distance/width, etc. ) .
  • an image segmentation model e.g., a convolution network for biomedical image segmentation such as UNet, etc.
  • a mapping relationship between a pixel-level size and a world-level size e.g., between a pixel-level distance/width and a world-level distance/width, etc.
  • the calculation operation in Step 3 of Part I may need to be implemented after the calibration operation in Step 1 of Part I, although this is not necessary in some embodiments.
  • Step 4 Collect gap samples. Most gap images are usually normal. Besides the conventional image augmentation, different distortion coefficients may be used to collect more anomalies from the gap images to make the result of image segmentation/calibration more robust.
  • Step 5 Train an image segmentation model (e.g., a convolution network for biomedical image segmentation such as UNet, etc. ) for gap segmentation.
  • an image segmentation model e.g., a convolution network for biomedical image segmentation such as UNet, etc.
  • Step 1 Transmit only a ROI part of an original image to a server or the cloud side so as to save bandwidth resource.
  • Step 2 Check the image quality. Raise an alarm if the image quality is equal to or lower than a threshold.
  • Step 3 Calibrate the image.
  • the image calibration may be performed on an image generated with the ROI part and a specific background, e.g., a white background, etc.
  • Step 4 Segment a gap in the ROI part of the image to measure the gap width by using the image segmentation model (e.g., UNet, etc. ) trained in Step 5 of Part I.
  • the image segmentation model e.g., UNet, etc.
  • Step 5 Get the corresponding points of the gap. For example, a predetermined number (e.g., 5, etc. ) of groups of points may be obtained, and each group has two points which may be respectively represented by the pixel/subpixel coordinates on two boundaries (e.g., upper boundary and lower boundary) of the gap.
  • a predetermined number e.g., 5, etc.
  • each group has two points which may be respectively represented by the pixel/subpixel coordinates on two boundaries (e.g., upper boundary and lower boundary) of the gap.
  • Fig. 1A is a diagram illustrating exemplary chessboard segmentation according to an embodiment of the present disclosure.
  • an image segmentation model such as UNet model may be used to segment a chessboard image. After segmentation of the chessboard image, only a chessboard part is kept and the background area of the chessboard image is turned to white.
  • the first row of images corresponds to the original chessboard images; the second row of images corresponds to the mask images where the white area is used to segment a target object (i.e. the chessboard part) from the original image and the black area is used to mask the background of the original image; and the third row of images corresponds to the results of image segmentation, i.e., the chessboard parts of the original chessboard images.
  • Fig. 1B is a diagram illustrating an exemplary contour annotation according to an embodiment of the present disclosure.
  • the contour of a target object to be segmented may need to be identified.
  • an application such as Labelme may be used to annotate the contour of the chessboard part in the original chessboard image.
  • the right part of Fig. 1B shows the chessboard part annotated by Labelme.
  • Fig. 1C is a diagram illustrating an exemplary mask image according to an embodiment of the present disclosure.
  • the mask image shown in Fig. 1C may be used to segment or extract the chessboard part from the original chessboard image shown in Fig. 1B.
  • OpenCV API may be used to get the mask image in Fig. 1C, e.g., by polygon filling. It can be appreciated that the mask image also may be obtained by using other suitable applications or technologies in addition to OpenCV API.
  • the UNet model for image segmentation may be trained until it converges.
  • the training process may be implemented, e.g., by repeatedly collecting samples for chessboard segmentation, including the original chessboard images as shown in Fig. 1B and the mask images as shown in Fig. 1C.
  • Fig. 1D is a diagram illustrating exemplary training pairs according to an embodiment of the present disclosure.
  • the training pairs include training input data and training output data.
  • the training pair for input and output may consist of the original image to be segmented and its corresponding mask image.
  • the original chessboard image is the training input data
  • the mask image is the training label or training output data.
  • the chessboard segmentation may be added before image calibration which may be implemented by applying OpenCV API, so that the failure rate of image calibration by OpenCV can be dramatically reduced.
  • the image segmentation may be applied to trace the object part, determine its contour, and get the mask image by only keeping the object part and turning the background to white.
  • the masked image may be provided to an image calibration model such as OpenCV API to implement different post-processing such as calibration.
  • the image calibration model may be used to calculate distortion correction parameters, e.g., a distortion related matrix, etc.
  • the distortion correction parameters may be used to rectify an image by calibration.
  • one or more standard markers may be designed to determine a mapping relationship between a pixel-level size and a world-level size.
  • the design of a marker may need to meet one or more requirements as below:
  • the shape of the marker can cause the marker to be easily segmented.
  • the parameter (s) related to the pixel-level size (e.g., width, distance, etc. ) can be easily acquired.
  • the world-level size of the marker can be pre-defined very precisely.
  • the marker may have a shape generated or determined by using a graphics application such as AutoCAD.
  • the marker can be easily fastened on the tested object.
  • the marker can be used to verify the mapping relationship between the pixel-level size and the world-level size, so as to generalize the mapping relationship as a measurement rule.
  • the marker may be designed in concentric circles way.
  • a circle may be a good choice for the marker because it may be very easy to segment a circle and the radius/diameter/perimeter of the circle can be determined precisely, e.g., by using OpenCV API with cv2. findContours and cv2. minClosingCircle through the mask image.
  • Fig. 2A is a diagram illustrating an exemplary marker according to an embodiment of the present disclosure.
  • the exemplary marker is designed to have three circles, and the radiuses from inner to outer are 0.75mm, 1.25mm, 2.75mm, respectively. It can be appreciated that the marker may have more or less circles and the radius values may include other suitable pre-defined values.
  • the circle radius may be designed such that the gap width of the tested object is between the radius values of the middle circle and the outer circle. For example, if the gap width is about 2mm, the radius value (s) of the circle (s) may be pre-defined as shown in Fig. 2A according to the gap width.
  • the inner circle of the marker may be utilized to build up the mapping relationship between a pixel-level size and a world-level size.
  • other circles may be utilized to verify whether the measurement rule (i.e., the mapping relationship between the pixel-level and world-level sizes) can be generalized or not. This may be facilitated by designing the marker so that the gap width is between the radiuses of the middle circle and the outer circle.
  • Fig. 2B is a diagram illustrating a set of exemplary markers according to an embodiment of the present disclosure.
  • the markers may be generated by a graphics application such as AutoCAD and printed on a piece of paper.
  • the size of a marker may be pre-defined according to the size of the tested object and/or other specific requirements. It can be appreciated that the number, shape, size and arrangement of the markers shown in Fig. 2B are just examples, and other suitable numbers, shapes, sizes and arrangements of the markers may be applied according to different application scenarios and service requirements.
  • Fig. 2C is a diagram illustrating an exemplary image with the tested object and markers according to an embodiment of the present disclosure. As shown in Fig. 2C, three parts each containing four markers are cropped from the paper printed with markers and pasted onto the tested object. It can be appreciated that although three cropped parts each containing four markers are used in the embodiment of Fig. 2C, more or less cropped parts with each part containing more or less markers may be applicable for size measurement on the tested object. In an embodiment, one or more images containing the tested object and the pasted markers may be captured, and the pixel-level size of one or more circles in the marker (s) can be measured by applying image calibration and segmentation on the image.
  • Fig. 2D is a diagram illustrating exemplary segmentation results for inner circles of different markers according to an embodiment of the present disclosure.
  • An image segmentation model such as UNet may be used to segment or extract a marker portion, e.g., from the image shown in Fig. 2C.
  • the centroid and radius of the marker can be acquired by OpenCV API, after getting the mask image.
  • the mask images used to extract the inner circle images of three markers are illustrated in the left column, and the extracted inner circle images containing the contour and centroid are illustrated in the right column.
  • 12 markers are pasted onto the tested object (e.g., as shown in Fig. 2C) .
  • the pixel-level measurement may be performed to get the respective radius results of inner circles of these 12 markers.
  • the pixel-level width of the object is 108 pixels
  • this mapping relationship between the pixel-level size and the world-level size may be verified by the measurement results of other circles (e.g., the middle circles and/or the outer circles) of the markers.
  • Fig. 2E is a diagram illustrating exemplary verification of measurement results by other circles according to an embodiment of the present disclosure.
  • the image segmentation model such as UNet may be used to extract other concentric circle images of the markers.
  • the pixel-level measurement may be performed to get the respective radius results of the other concentric circles, e.g., including the middle circles and/or the outer circles.
  • the mapping relationship between the pixel-level size and the world-level size can be calculated in a manner similar to that described in connection with Fig. 2D.
  • mapping relationship meets the expectation, e.g., a difference between different calculation results of mapping relationship equal to or less than a certain threshold, it may be considered that the generalization of mapping relationship is successfully verified. Otherwise, the mapping relationship may need to be recalculated, and/or the image calibration may need to be adjusted.
  • some data augmentation including anomalies may be introduced to improve performance of image segmentation and image calibration for the object such as a gap to be tested.
  • Fig. 2F is a diagram illustrating exemplary generation of anomalies according to an embodiment of the present disclosure.
  • a conventional image processing procedure it may be required to get rid of image distortion.
  • tactfully utilizing different distortion settings to generate anomalies may be able to make the image processing performance more robust to different scenarios.
  • different distortion coefficients may be used to collect more anomalies to make the image segmentation/calibration results more robust.
  • four gap images may be generated according to different distortion coefficients. The edges of these gaps are not horizontal and some are slightly bent. These gap images may be used as input for training the image segmentation model such as UNet.
  • the trained image segmentation model can handle diversified gap images and provide better segmentation performance.
  • the size of a target object such as gap width may be detected and measured.
  • the size of an image captured by an industrial camera may be very large. For example, the size of the captured image with 2448 *2048 resolution may reach to 14.3M.
  • a portion of the originally captured image i.e. a ROI of the image, may be cropped and only this portion of the image may be transmitted to a server at which the size of the detected object may be measured by applying the computer vision technology on the ROI of the image.
  • Fig. 3A is a diagram illustrating an exemplary ROI according to an embodiment of the present disclosure.
  • the image on the left is the original image where the portion framed by a box is a ROI part, and the image on the right only includes the ROI part.
  • the ROI may be pre-defined by configuring the camera setting.
  • An object to be measured such as a gap shown in Fig. 3A is included in the ROI part of the image.
  • only the ROI part of the image may be transmitted, e.g., via a communication network such as fourth generation (4G) /long term evolution (LTE) and fifth generation (5G) /new radio (NR) networks, to a server to measure the size of the object. Since the size of the ROI is less than the size of the original image, the amount of transmitted image data may be decreased significantly.
  • 4G fourth generation
  • LTE long term evolution
  • 5G fifth generation
  • NR new radio
  • the image quality may not be always good since the focal length of the camera may be changed accidentally or as other issues happen. So the image quality may need to be checked before further handling. For example, one or more image samples (e.g., the original images or only ROI parts of the original images) may be checked so as to identify the image sample (s) with satisfied image quality. In an embodiment, only the identified image sample (s) can be provided to the server for size measurement.
  • one or more image samples e.g., the original images or only ROI parts of the original images
  • Fig. 3B is a diagram illustrating an exemplary check of image quality according to an embodiment of the present disclosure.
  • the quality of the image on the left is better than that of the image on the right.
  • Various image quality assessment approaches e.g., BRISQUE, etc. may be applied for checking the image quality.
  • BRISQUE it may calculate a mean subtracted contrast normalized (MSCN) coefficient and finally uses a support vector machine (SVM) regressor.
  • MSCN mean subtracted contrast normalized
  • SVM support vector machine
  • the image on the left in Fig. 3B may have a clear image score such as 60.3, and the image on the right in Fig. 3B may have a blur image score such as 81.6.
  • the higher score means the lower quality.
  • an alarm may be raised if the image score of a sample is higher than a threshold, and then this sample may be dropped.
  • Fig. 3C is a diagram illustrating an exemplary image generated from a ROI part according to an embodiment of the present disclosure.
  • the image shown in Fig. 3C may be generated based on a background image such as a pure white image in a memory of the server.
  • the size of the background image may be equal to the original image from which the ROI part is cropped.
  • the image in Fig. 3C may be generated by putting the ROI part to a position on the background image which is the same as its place on the original image.
  • image calibration may be performed for the generated image, e.g., according to one or more distortion correction parameters acquired by Step 1 of Part I.
  • Fig. 3D is a diagram illustrating exemplary gap segmentation results according to an embodiment of the present disclosure.
  • An image segmentation model e.g., the UNet model trained as described with respect to Fig. 2F, etc.
  • the contour tracking function of the image segmentation model may be used to extract the upper and lower boundaries of the gap.
  • the gap segmentation results may indicate the left and right boundaries of the gap, the inner and outer boundaries of the gap, and/or any other possible parameters reflecting the gap width, etc.
  • one or more segment points on one boundary of the gap and the corresponding segment points on the other boundary of the gap may be identified as required.
  • Fig. 3E is a diagram illustrating exemplary upper and lower boundaries of a gap according to an embodiment of the present disclosure. As shown in Fig. 3E, five groups of segment points on the upper and lower boundaries of the gap are identified and used to divide the upper/lower boundary into six parts. For example, each segment point on the lower/upper boundary may be identified by the closest pixel coordinate in x-axis. In an embodiment, a subpixel representation may be used to refine the pixel coordinate.
  • Fig. 3F is a diagram illustrating an exemplary subpixel representation according to an embodiment of the present disclosure.
  • a zoom-in image of one boundary (e.g., the upper boundary shown in Fig. 3E) of a gap is shown in Fig. 3F.
  • the coordinate of the boundary may be represented in more precise manner. For instance, for the pixel coordinate (120, 220) , if the subpixel representation is applied, the corresponding subpixel coordinate may be (120.25, 220.50) .
  • the float point value of the subpixel coordinate may be acquired so that the measurement result is more accurate. It can be appreciated that in addition to the subpixel representation scheme, other suitable schemes also may be used to get the precise contour and make the measurement result more accurate.
  • the distance between one segment point and its corresponding segment point in each group of segment points may be used to indicate a gap width.
  • five Euclidean distance values can be calculated for the five groups of segment points shown in Fig. 3E.
  • an alarm may be raised if at least one of the following conditions is met:
  • Figs. 4A-4B are diagrams illustrating exemplary use cases of measurement according to some embodiments of the present disclosure.
  • Fig. 4A shows screw hole size measurement according to an exemplary embodiment
  • Fig. 4B shows putty shape measurement according to another exemplary embodiment.
  • the measurement results may be used to determine whether the size and/or shape of an object meets the requirement, e.g., whether the size of the object is too large or too small.
  • Fig. 5 is a flowchart illustrating an exemplary procedure of gap width measurement according to an embodiment of the present disclosure.
  • the procedure may be performed by using the computer vision technology.
  • the procedure may include the calibration and training phase and the detection phase.
  • a chessboard image may be provided 511 to an image segmentation model such as UNet to get a chessboard image segmentation result.
  • the chessboard image segmentation result may be sent 512 to an OpenCV calibration API to get 513 one or more distortion correction parameters.
  • an image containing one or more markers may be sent 521 to an image calibration model such as OpenCV and calibrated according to the one or more distortion correction parameters 514.
  • a portion containing one or more markers may be cropped from the calibrated image and provided 522 to an image segmentation model such as UNet to implement marker segmentation.
  • the marker extracted from the cropped image portion by the image segmentation model may be used to calculate 523 the pixel-level radius value of a circle in the marker, e.g. by using UNet and OpenCV API.
  • the calculated pixel-level radius value of the circle may be used, together with the pre-defined world-level radius value of the circle, to get 524 the mapping relationship between the pixel-level size and the world-level size.
  • the mapping relationship between the pixel-level size and the world-level size may be verified 525 so as to implement generalization of the mapping relationship.
  • one or more samples each reflecting a gap in the tested object may be collected to generate 531 one or more distortion coefficients for data augmentation.
  • the one or more distortion coefficients may be used 532 for training data labeling, so as to implement 533 UNet model training.
  • the trained UNet model may be used 534 as a gap segmentation model.
  • an image containing an object with a gap to be detected may be captured or retrieved by a camera.
  • the ROI part of the image may be transmitted 541 to a server for gap measurement.
  • the image quality of the ROI part may be checked prior to implementing image calibration, which is realized by applying distortion correction parameters, 515, on an image generated based on the ROI part.
  • the image generated based on the ROI part may be provided 542 to a gap segmentation model such as the trained UNet model to implement gap segmentation on the object.
  • a gap segmentation model such as the trained UNet model to implement gap segmentation on the object.
  • the pixel-level gap width may be calculated 543 according to the pixel/subpixel coordinates of several segment points on the boundaries. According to an embodiment, the subpixel representation may be used to improve accuracy of the measured gap width. According to the mapping relationship between the pixel-level size and the world-level size 526, the pixel-level gap width can be translated 544 into the world-level gap width which may be output 545 as the measurement result of gap width.
  • the proposed solution according to various exemplary embodiments can greatly increase success rate of image calibration even if the image background is complicated.
  • a special image data augmentation scheme may be used to generate more anomalies for image segmentation, so as to make the size measurement become more robust.
  • some image samples with low quality may be dropped to make the measurement result more reliable.
  • the subpixel scheme may be used to perfect the measurement result.
  • a deep learning algorithm may be applied to make the proposed solution more robust to a light environment change. It can be appreciated that the proposed solution may be scalable to other use cases for size automatic measurement.
  • UNet model and OpenCV API as described above are just examples, and according to different application requirements, various models and algorithms such as deep learning algorithms, convolution neural networks, and image processing technologies for segmentation and calibration may be applicable for various exemplary embodiments according to the present disclosure.
  • Fig. 6 is a flowchart illustrating a method 600 according to an embodiment of the present disclosure.
  • the method 600 illustrated in Fig. 6 may be performed by an apparatus which may be able to utilize the computer vision technology or be communicatively coupled to an image processor.
  • the apparatus may comprise a server, a client device or any other suitable device which is capable of implementing size measurement by using the computer vision technology.
  • the apparatus can measure a pixel-level size of one or more markers in a first image (e.g., the image shown in Fig. 2C) by applying image segmentation to the one or more markers in the first image, as shown in block 602.
  • each of the one or more markers may be designed in concentric circles way and have a predefined world-level size.
  • the apparatus can determine a mapping relationship between a pixel-level size and a world-level size, as shown in block 604.
  • each of the one or more markers may include multiple concentric circles (as shown in Fig. 2A, Fig. 2B and Fig. 2C) and each circle may have a predefined world-level size (e.g., radius, diameter, perimeter, etc. ) .
  • the apparatus can determine the mapping relationship between the pixel-level size and the world-level size by building up the mapping relationship between the pixel-level size and the world-level size, according to size measurements on one or more of the multiple concentric circles.
  • the apparatus can determine the mapping relationship between the pixel-level size and the world-level size further by verifying the mapping relationship between the pixel-level size and the world-level size and/or accuracy of image calibration, according to size measurements on remaining of the multiple concentric circles. In some cases, if the mapping relationships determined according to the size measurements on different concentric circles are inconsistent, the mapping relationship built up previously may need to be modified, and/or the current image calibration may be inaccurate and require adjustment.
  • the first image may be an image calibrated based at least in part on one or more distortion correction parameters.
  • the one or more distortion correction parameters may be determined by performing a training process based on chessboard segmentation (e.g., as described with respect to Figs. 1A-1D and Fig. 5) .
  • the training process based on the chessboard segmentation may have chessboard images as input data and corresponding mask images as output data.
  • the chessboard segmentation can extract chessboard parts from the chessboard images by using the corresponding mask images.
  • the training process based on the chessboard segmentation may be performed by using a convolution neural network (e.g., a UNet model, etc. ) or any other suitable artificial intelligence algorithms (e.g., deep learning algorithms, etc. ) .
  • a convolution neural network e.g., a UNet model, etc.
  • any other suitable artificial intelligence algorithms e.g., deep learning algorithms, etc.
  • the one or more distortion correction parameters may be used to calibrate a second image (e.g., by using an image calibration model such as OpenCV, etc. ) prior to applying image segmentation to an object (e.g., a gap, etc. ) in the second image.
  • an image calibration model such as OpenCV, etc.
  • the apparatus can measure a pixel-level size of an object (e.g., a gap, etc. ) in a second image (e.g., the image shown in Fig. 3C) by applying image segmentation to the object in the second image (e.g., as described with respect to Figs. 3C-3E) .
  • the second image may be generated from a third image (e.g., the right image shown in Fig. 3A) which includes the object and has a smaller size than the second image.
  • the apparatus can determine a world-level size of the object, according to the measured pixel-level size of the object and the mapping relationship between the pixel-level size and the world-level size.
  • the third image may be cropped from a fourth image (e.g., the left image shown in Fig. 3A) which has a same size with the second image.
  • the position of the third image in the second image may be same as the position of the third image in the fourth image.
  • the third image may be a ROI portion from the fourth image, and the fourth image may be the original image captured by a camera.
  • the third image may have an image quality which meets a predetermined criterion.
  • the image quality of the third image may be equal to or higher than a predefined quality level.
  • the image quality score of the third image may be equal to or less than a predefined score.
  • the image segmentation applied to the object in the second image may be implemented in a segmentation model (e.g., a UNet model, etc. ) .
  • the segmentation model may be trained by using data augmentation with distortion parameter setting (e.g., as described with respect to Fig. 2F) .
  • the apparatus can measure the pixel-level size of the object in the second image by determining one or more boundaries of the object in the second image, according to a result of the image segmentation.
  • the one or more boundaries may be indicated by subpixel coordinates of a set of segment points.
  • the one or more boundaries may include a first boundary and a second boundary.
  • the set of segment points may include one or more segment points on the first boundary and one or more corresponding segment points on the second boundary (e.g., as described with respect to Fig. 3E) .
  • the pixel-level size of the object in the second image may be represented by one or more distance values.
  • Each of the one or more distance values may indicate a distance between a segment point on the first boundary and a corresponding segment point on the second boundary.
  • the apparatus may trigger an alarm in response to one or more of the following events:
  • ⁇ an average value of the one or more distance values is equal to or larger than a first threshold
  • ⁇ a difference between the largest distance value and the smallest distance value among the one or more distance values is equal to or larger than a second threshold
  • ⁇ an average value of the one or more distance values is equal to or less than a third threshold
  • ⁇ a difference between the largest distance value and the smallest distance value among the one or more distance values is equal to or less than a fourth threshold.
  • first, second, third and fourth thresholds mentioned above may be predefined as the same value or different values, depending on various application requirements.
  • Figs. 5-6 may be viewed as method steps, and/or as operations that result from operation of computer program code, and/or as a plurality of coupled logic circuit elements constructed to carry out the associated function (s) .
  • the schematic flow chart diagrams described above are generally set forth as logical flow chart diagrams. As such, the depicted order and labeled steps are indicative of specific embodiments of the presented methods. Other steps and methods may be conceived that are equivalent in function, logic, or effect to one or more steps, or portions thereof, of the illustrated methods. Additionally, the order in which a particular method occurs may or may not strictly adhere to the order of the corresponding steps shown.
  • Fig. 7 is a block diagram illustrating an apparatus 700 according to various embodiments of the present disclosure.
  • the apparatus 700 may comprise one or more processors such as processor 701 and one or more memories such as memory 702 storing computer program codes 703.
  • the memory 702 may be non-transitory machine/processor/computer readable storage medium.
  • the apparatus 700 may be implemented as an integrated circuit chip or module that can be plugged or installed into a server or a client device such that the server or the client device may be able to implement the method as described with respect to Fig. 6.
  • the one or more memories 702 and the computer program codes 703 may be configured to, with the one or more processors 701, cause the apparatus 700 at least to perform any operation of the method as described in connection with Fig. 6.
  • the one or more memories 702 and the computer program codes 703 may be configured to, with the one or more processors 701, cause the apparatus 700 at least to perform more or less operations to implement the proposed methods according to the exemplary embodiments of the present disclosure.
  • the various exemplary embodiments may be implemented in hardware or special purpose chips, circuits, software, logic or any combination thereof.
  • some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the disclosure is not limited thereto.
  • firmware or software which may be executed by a controller, microprocessor or other computing device, although the disclosure is not limited thereto.
  • While various aspects of the exemplary embodiments of this disclosure may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.
  • the exemplary embodiments of the disclosure may be practiced in various components such as integrated circuit chips and modules. It should thus be appreciated that the exemplary embodiments of this disclosure may be realized in an apparatus that is embodied as an integrated circuit, where the integrated circuit may comprise circuitry (as well as possibly firmware) for embodying at least one or more of a data processor, a digital signal processor, baseband circuitry and radio frequency circuitry that are configurable so as to operate in accordance with the exemplary embodiments of this disclosure.
  • exemplary embodiments of the disclosure may be embodied in computer-executable instructions, such as in one or more program modules, executed by one or more computers or other devices.
  • program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types when executed by a processor in a computer or other device.
  • the computer executable instructions may be stored on a computer readable medium such as a hard disk, optical disk, removable storage media, solid state memory, random access memory (RAM) , etc.
  • RAM random access memory
  • the function of the program modules may be combined or distributed as desired in various embodiments.
  • the function may be embodied in whole or partly in firmware or hardware equivalents such as integrated circuits, field programmable gate arrays (FPGA) , and the like.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Geometry (AREA)
  • Image Analysis (AREA)

Abstract

Un procédé de mesure (600) consiste à : mesurer une taille de niveau de pixel d'un ou de plusieurs marqueurs dans une première image par application d'une segmentation d'image au ou aux marqueurs dans la première image (602); chacun du ou des marqueurs peut être conçu en cercles concentriques et avoir une taille de niveau de monde prédéfinie; déterminer une relation de mappage entre une taille de niveau de pixel et une taille de niveau de monde, en fonction de la taille de niveau de pixel mesurée et de la taille de niveau de monde prédéfinie du ou des marqueurs (604).
PCT/CN2021/116262 2021-09-02 2021-09-02 Procédé et appareil de mesure WO2023028955A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/116262 WO2023028955A1 (fr) 2021-09-02 2021-09-02 Procédé et appareil de mesure

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/116262 WO2023028955A1 (fr) 2021-09-02 2021-09-02 Procédé et appareil de mesure

Publications (1)

Publication Number Publication Date
WO2023028955A1 true WO2023028955A1 (fr) 2023-03-09

Family

ID=85410788

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/116262 WO2023028955A1 (fr) 2021-09-02 2021-09-02 Procédé et appareil de mesure

Country Status (1)

Country Link
WO (1) WO2023028955A1 (fr)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101520514A (zh) * 2008-02-28 2009-09-02 株式会社拓普康 目标及使用其的三维形状测定装置
JP2011047725A (ja) * 2009-08-26 2011-03-10 Alpine Electronics Inc 撮像装置の取り付け位置の推定方法およびキャリブレーションシステム
CN102121821A (zh) * 2010-12-23 2011-07-13 江西洪都航空工业集团有限责任公司 落震试验轮胎压缩量和轮轴位移的图像测量方法
CN103323229A (zh) * 2013-07-08 2013-09-25 重庆工业职业技术学院 基于机器视觉的五轴数控机床旋转轴误差检测方法
CN109556514A (zh) * 2018-12-03 2019-04-02 广东正业科技股份有限公司 光轴标定方法、图像坐标系间转换关系的测法及成像系统
CN109934874A (zh) * 2017-12-18 2019-06-25 比亚迪股份有限公司 摄像头标定中基于同心圆环模板进行求点的方法和装置
CN109934830A (zh) * 2019-03-18 2019-06-25 杭州电子科技大学 一种基于图像计算的实时物体转动角度测量装置和方法
CN110207605A (zh) * 2019-06-13 2019-09-06 广东省特种设备检测研究院东莞检测院 一种基于机器视觉的金属结构变形的测量装置及方法

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101520514A (zh) * 2008-02-28 2009-09-02 株式会社拓普康 目标及使用其的三维形状测定装置
JP2011047725A (ja) * 2009-08-26 2011-03-10 Alpine Electronics Inc 撮像装置の取り付け位置の推定方法およびキャリブレーションシステム
CN102121821A (zh) * 2010-12-23 2011-07-13 江西洪都航空工业集团有限责任公司 落震试验轮胎压缩量和轮轴位移的图像测量方法
CN103323229A (zh) * 2013-07-08 2013-09-25 重庆工业职业技术学院 基于机器视觉的五轴数控机床旋转轴误差检测方法
CN109934874A (zh) * 2017-12-18 2019-06-25 比亚迪股份有限公司 摄像头标定中基于同心圆环模板进行求点的方法和装置
CN109556514A (zh) * 2018-12-03 2019-04-02 广东正业科技股份有限公司 光轴标定方法、图像坐标系间转换关系的测法及成像系统
CN109934830A (zh) * 2019-03-18 2019-06-25 杭州电子科技大学 一种基于图像计算的实时物体转动角度测量装置和方法
CN110207605A (zh) * 2019-06-13 2019-09-06 广东省特种设备检测研究院东莞检测院 一种基于机器视觉的金属结构变形的测量装置及方法

Similar Documents

Publication Publication Date Title
US11443133B2 (en) Computer vision system for industrial equipment gauge digitization and alarms
CN110930390B (zh) 基于半监督深度学习的芯片管脚缺失检测方法
WO2018195797A1 (fr) Procédé de détection visuelle, dispositif de détection et robot
WO2018209941A1 (fr) Procédé de détection de sous-pixel de trou circulaire de carte de circuit imprimé haute densité fondé sur un ajustement gaussien
KR102073468B1 (ko) 비전 시스템에서 컬러 이미지에 대해 컬러 후보 포즈들의 점수화를 위한 시스템 및 방법
WO2020119467A1 (fr) Procédé et dispositif de génération d'image de profondeur dense à haute précision
CN112489140A (zh) 姿态测量方法
CN110866949A (zh) 中心点定位方法、装置、电子设备和存储介质
CN116228780A (zh) 基于计算机视觉的硅片缺陷检测方法及系统
WO2023028955A1 (fr) Procédé et appareil de mesure
CN115423804B (zh) 影像标定方法及装置、影像处理方法
CN116430069A (zh) 机器视觉流体流速测量方法、装置、计算机设备及存储介质
CN113470103B (zh) 车路协同中相机作用距离确定方法、装置和路侧设备
CN110619629A (zh) 一种cpu插座检测方法和设备
CN111325802B (zh) 一种直升机风洞试验中圆形标记点识别匹配方法
CN114283081A (zh) 基于金字塔加速的深度恢复方法、电子设备及存储介质
KR101383827B1 (ko) 인쇄회로기판의 솔더링 영역 자동검출 시스템 및 방법
US10958899B2 (en) Evaluation of dynamic ranges of imaging devices
CN117173156B (zh) 基于机器视觉的极片毛刺检测方法、装置、设备及介质
KR20160148735A (ko) 카메라 광중심 측정을 위한 영상 처리 장치 및 그 방법
Zhu et al. Image quality evaluation method for surface crack detection based on standard test chart
CN117392127B (zh) 一种显示面板边框的检测方法、装置和电子设备
US20230326029A1 (en) Method for processing images, electronic device, and storage medium
CN118050376A (zh) 检测方法、检测装置、相机系统和计算机设备
CN114119944A (zh) 一种图像检测装置、方法及计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21955494

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE