US7978914B2 - Image processing system - Google Patents

Image processing system Download PDF

Info

Publication number
US7978914B2
US7978914B2 US10/597,554 US59755404A US7978914B2 US 7978914 B2 US7978914 B2 US 7978914B2 US 59755404 A US59755404 A US 59755404A US 7978914 B2 US7978914 B2 US 7978914B2
Authority
US
United States
Prior art keywords
processing system
image processing
contour point
contour
image data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related, expires
Application number
US10/597,554
Other languages
English (en)
Other versions
US20090129699A1 (en
Inventor
Lars Beikirch
Sergej Fedorenko
Dittmar Schwarz
Joachim Ihlefeld
Lutz Wenert
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Baumer Optronic GmbH
Original Assignee
Baumer Optronic GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Baumer Optronic GmbH filed Critical Baumer Optronic GmbH
Assigned to BAUMER OPTRONIC GMBH reassignment BAUMER OPTRONIC GMBH ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: BEIKIRCH, LARS, FEDORENKO, SERGEJ, IHLEFELD, JOACHIM, DR., SCHWARZ, DITTMAR, WENERT, LUTZ
Publication of US20090129699A1 publication Critical patent/US20090129699A1/en
Application granted granted Critical
Publication of US7978914B2 publication Critical patent/US7978914B2/en
Expired - Fee Related legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/12—Edge-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/28—Indexing scheme for image data processing or generation, in general involving image processing hardware
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10081—Computed x-ray tomography [CT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30204—Marker
    • G06T2207/30208—Marker matrix
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior

Definitions

  • the invention relates to an image processing system in general, and to an image processing system for omnidirectional reading of optical patterns, for example of one-dimensional and two-dimensional codes, in particular.
  • Image processing for automatic pattern recognition is currently one of the most demanding and excellent research fields. The potential applications are virtually unlimited.
  • Simple precursors of pattern recognition are the widely used laser scanner for detection of one-dimensional barcodes.
  • digital image processing goes well beyond this since it can be used not only to identify one-dimensional barcodes, but also two-dimensional codes or complex images, in particular and in addition the structure which is unknown from the start as far as identification of how this is also made possible by the eye in conjunction with the brain.
  • One possible application is the identification of a code which is applied at an undefined point to a motor vehicle in the production line, in order to allow the production process to be comprehensively controlled and monitored. Since the codes are relatively small in comparison to the cars, and can be applied at different points, this application is extremely demanding with respect to the required quality and resolution. However, the different illumination in a production line also results in considerable difficulties for image identification. This is associated with ever more stringent demands for the processing speed.
  • Image processing systems are typically based on a digital two-dimensional image which is produced, for example, by a CCD camera.
  • the two-dimensional images which are produced by the camera and can be processed further digitally are then passed to the pattern recognition.
  • other imaging systems for example X-ray CT scans, MRI scans or scanners can also be used, to name but one possible choice.
  • the identification process can be split into at least two phases. Firstly, pixels which can in principle be associated with a basic structure are found, and secondly pixels which have been found and belong to the same basic structure are combined in order to actually determine the shape, position, size etc of this basic structure.
  • the image information is typically in the form of a brightness distribution and, possibly, a color distribution in the image, as well.
  • the following text refers only to monochrome processing, but it is obvious that the processing can also be carried out in a corresponding manner for different color components of the image.
  • the two-dimensional digital image is assumed, that is to say a discrete matrix of pixels, which is produced by the camera. In the matrix, each pixel contains at least one brightness value.
  • the image information can be broken down into simple objects which are bounded by lines or edges which are defined by a particularly high brightness gradient, frequently also referred to as a discontinuity.
  • the discontinuity points which have been found in this way are then associated with one specific edge.
  • the basic structures determined in this way finally represent the sought discontinuity edges, which represent a schematic form of the image content. This procedure is referred to as edge detection.
  • One possible way to approach this problem is to scan the brightness or to determine the brightness function along one direction. The function is then differentiated and subjected to a threshold value analysis. If the threshold value is exceeded at a point, then it is assumed that an edge is present at this point.
  • the gradient magnitude and direction of the image are determined in a lattice composed of node points by means of Gaussian network operators.
  • the decision as to whether a node point is associated with an edge path is made on the basis of the gradient magnitude.
  • the resolution is restricted by the image matrix, so that the resolution is inadequate for many fields of application.
  • major refinements of the image matrix in order to increase the resolution cannot be carried out indefinitely, either, since the processing time would otherwise explode.
  • the described method is therefore disadvantageous and, in particular, is barely suitable for use in the demanding exemplary embodiment described above in the field of motor vehicle production.
  • many other applications, such as precise edge measurements, can also be carried out only to a restricted extent in this way.
  • One object of the present invention is therefore to provide an image processing system which ensures high resolution with fast processing at the same time, in order in particular to allow processing in video real time.
  • a further object of the invention is to provide an image processing system which ensures reliable identification of the image information, in particular reliable and low-error edge detection and segmentation, as well as efficient object association and efficient compression of the image information.
  • Another object of the invention is to provide an image processing system which is cost-effective, and which in particular can be manufactured in series production.
  • Yet another object of the invention is to provide an image processing system which avoids or at least reduces the disadvantages of known solutions.
  • an image processing system which is aimed in particular at omnidirectional reading of optical patterns, for example one-dimensional and two-dimensional codes.
  • the image processing system according to the invention is, however, even suitable for carrying out image identification on complex objects.
  • the image processing system comprises an imaging system, for example a CCD camera, a scanner or a CT scanner, etc. for provision of an image datastream, which is initially in analog form.
  • the image datastream can also be recorded by a plurality or large number of cameras (for example stereo systems, 3D processing, robot control, facet eye).
  • the analog image datastream produced by the imaging system is then converted by means of an analog/digital converter to digital image data, to be more precise to a digital image datastream.
  • An illumination device is preferably also included, in order to achieve matched illumination of the image.
  • the digital image datastream is also further processed by a digital processing device or unit, with the processing device comprising a pipeline processor and a computation device, or a microcomputer, inter alia with means for edge detection.
  • the core of the image processing system is a plurality of convolvers or convolution components which operate in parallel and/or using the time-division multiplex method, and which carry out convolution operations on the incoming digital image datastream.
  • gray-scale value profiles are first of all produced from the image datastream, and are convolved.
  • the convolvers carry out a cross-correlation process on the image data, with a convolution core with a zero mean value being used for this purpose.
  • the parallel hardware-based processing has made it possible to achieve a breakthrough in the processing speed. It is thus possible to carry out processing in video real time by use of the method according to the invention.
  • the digital image data is preferably convolved by means of the hardware convolvers in parallel in time and/or using the time-division multiplexing method with the image datastream representing an at least two-dimensional image, and the convolvers operating in parallel in time, in different directions.
  • the image processing system preferably has an edge detection unit for each direction or scanning direction, with each edge detection unit having at least one hardware convolver.
  • scanning in each of four directions rotated through 45° with respect to one another has been found to be an advantageous compromise between hardware complexity and identification quality, so that four edge detection units are used in this case, and four separate convolvers are provided, operating in parallel in time.
  • gray-scale profiles are produced and cross-correlated in parallel for each scanning direction in order to define their point of inversion as a zero point for the cross-correlation.
  • the incoming image datastream is preferably convolved in pairs in each edge detection unit, with the two convolvers in one pair operating in the same direction.
  • one convolver pair is provided for each direction.
  • each gray-scale profile comprises n pixels which are adjacent along the scanning direction. This results in a 1 ⁇ n environment. Furthermore, the convolution of the first n ⁇ 1 pixels of each gray-scale profile and of the last pixels or n ⁇ 1 pixels shifted by one is carried out in pairs, so that one first and one second convolution result is in each case calculated for each n ⁇ n environment and for each direction. It has been found to be particularly advantageous to operate within a 6 ⁇ 6 environment for this purpose.
  • the incoming digital image datastream is thus convolved within an n ⁇ n, in particular 6 ⁇ 6 environment, in the processing device which, in particular is a preprocessing unit, in a first main process by means of a plurality of pairs, in particular four pairs, of convolvers which operate in parallel in time and operate in four directions which are in each case rotated through 45°.
  • This shifted convolution in pairs allows a wide range of options for further processing, as will be explained in the following text.
  • Contour points are then produced and a decision-making component is used to decide whether the respective contour point is valid or invalid.
  • the subset of valid contour points produced in this way is then entered in a contour point list in a contour point memory.
  • One contour point list is in consequence produced in each case for each direction.
  • the decision as to whether a contour point is valid or invalid is in this example made as follows. First of all, operands are determined by means of the convolution results of the convolvers, for a logic decision.
  • the image processing system also has at least one decision unit, to be more precise, each edge detection unit has in each case one decision unit, which produces a logical variable as a function of the convolution results or operands.
  • the contour point is classified as valid or invalid as a function of the logical variable, and is entered in the contour point list if it is valid.
  • the logic decision as to whether a contour point is classified as valid is preferably made as follows.
  • the logical variable is represented by a Boolean vector with a plurality of logical variables.
  • a first of the logical variables B[1] depends on the difference between the convolution results associated with one convolver pair exceeding an absolute magnitude threshold value.
  • a second of the logical variables B[2] depends on a mathematical sign difference between the convolution results associated with one convolver pair, and a third of the logical variables B[4] depends on whether one of the convolution results associated with one convolver pair is equal to zero.
  • the valid/invalid decision is then made as a function of a logic operation on the first, second and/or third logical variables, and the contour point is entered in the contour point list.
  • the logical operation is in the form of a Boolean operation B[1] AND (B[2] OR B[4]) and the contour point is entered in the contour point list when the result of the logical operation is TRUE.
  • a contour point is entered in the contour point list only when a) the convolution result of the operands which are associated with one convolver pair has different mathematical signs and the difference between the operands in the predetermined direction exceeds a threshold value, or when b) one and only one operand is equal to zero and the difference between the operands in the predetermined direction exceeds a threshold value.
  • the contrast value is determined, for example, as the difference between the sum of the respective two outer pixels of the 1 ⁇ 6 environment used for the convolution process. Another option for definition of the contrast value is, for example, the difference between the convolution results.
  • the contrast value is not only used as a further decision-making criterion for the validity of a contour point, but, if it is classified as being valid, the contour point is entered in a memory, at least with the contrast value associated with the values X position, Y position.
  • the contrast value is advantageously also used later for determination of a neighborhood criterion.
  • a further aspect of the invention is the use of a neighborhood processor, which reads and links the contour points.
  • This linking process is part of a second main process, which is carried out at a time after the first main process.
  • the contour points (P(X,Y, contrast value)) of adjacent lines and/or columns are read by the neighborhood processor, are then linked in order to satisfy a neighborhood criteria, and are once again entered in a contour point list.
  • the neighborhood criterion is determined as a function of the contour point separation and/or the contrast values of the adjacent contour points. This classification on the basis of distance and/or contrast or gray-scale value has been found to be particularly suitable for identification of cohesive and separate objects as such.
  • a further embodiment of the invention is based on the creation of a segment list as a contour point list to which at least one segment number is added.
  • Object association is preferably carried out on the contour points as well, in the processing device.
  • the segment list is read by a statistical processor in a third main process, which is carried out at a time after the second main process.
  • the statistical processor calculates statistical moments for the objects, in particular statistical moments for the objects in each direction.
  • the statistical moments are preferably calculated as far as the second, third and/or fourth order, in particular for the objects in each direction.
  • All of the processing can be carried out in video real time, since the image data is supplied as an image datastream to the processing device, and the contour point list is actually produced while the image datastream is being supplied.
  • contour point lists are stored in one or more memories for the running time of the image and they are sorted on the basis of directions and are stored in a partition form when the contour point memory is read again, before processing by the neighborhood processor. (As described below).
  • One major advantage of the image processing system according to the invention is, however, also based on the fact that a pixel interpolation process can be carried out in order to reduce the digitization noise, thus achieving a resolution which is better than the resolution of the image produced by the camera.
  • sub-pixel resolution is achieved, and the contour points are stored with sub-pixel resolution in the contour point list. This has not been possible until now, in particular with the known threshold value methods.
  • At least one gradient is also calculated for each scanning direction, and is preferably stored in the contour point list.
  • the gradient or gradients corresponds or correspond in particular to a contrast value in the scanning direction, and may be used as a measure of the quality of the contour point, and as a neighborhood criterion.
  • the difference between the two convolution operations in the same direction, and shifted by one pixel in the scanning direction, is calculated as a decision criterion for entry of a contour point in the associated contour point list.
  • a gradient or a plurality of gradients and the difference between two convolution results are calculated in the respective filter direction with respect to the delay time of the image, and at least one of the gradients and/or said difference is stored as a contrast within the contour point list.
  • the contents of the contour point list are then copied to a first memory for each direction.
  • the first memory is preferably an external memory with respect to the first processing device in which case the contents of the contour point list are copied from the processing device for each direction and after each line of the imaging system to the first memory.
  • contour points are preferably also segmented in the processing device.
  • an interface of a data storage unit for this purpose copies the current line from the contour point list, after processing progress of the neighborhood processor, separately on the basis of directions to a second memory, to be more precise to a general purpose memory of the neighborhood processor.
  • the neighborhood processor preferably then uses a neighborhood criterion to define neighborhoods of the contour points, segments the contour points and/or enters the segment numbers and coincidences in an extended contour point list.
  • the neighborhood criterion of contour points is calculated by the distance and the contrast comparison.
  • the Euclidean distance or else some other distance measure is used as the distance measure.
  • a statistical processor preferably calculates statistical moments on the basis of the contour point list.
  • the data obtained in this way specifically the scanning direction, the statistical moments, the size of the environment and the gradient or contrast are now output via an output interface in the following output format [Direction (DIR), Moment (S), Segment number (SegNo), Contrast (CONT)].
  • DIR Direction
  • S Moment
  • S Segment number
  • CONT Contrast
  • FIG. 1 shows a gray-scale image of a two-dimensional data matrix code
  • FIG. 2 shows an enlargement of a detail from FIG. 6 .
  • FIG. 3 shows a selection of an edge profile for the 90° direction in order to produce two gray-scale profiles for parallel convolution
  • FIG. 4 shows an illustration, in the form of a graph, of the profile of the first derivative of a typical gray-scale profile based on location
  • FIG. 5 shows an illustration of the length extension of the X and Y coordinates for sub-pixel interpolation
  • FIG. 6 shows an illustration, in the form of a graph, of the contour point list in the 90° direction
  • FIG. 7 shows the calculation of the coordinates with sub-pixel accuracy
  • FIG. 8 shows the data structure of the contour point list
  • FIG. 9 shows an example of a detail of a hexadecimal contour point list (“Hexdump”)
  • FIG. 10 shows an illustration in the form of a graph on the approximated and filtered digital paths
  • FIG. 11 shows a block diagram of the image processing system according to the invention
  • FIG. 12 shows a block of an edge detection unit according to the invention
  • FIG. 13 shows a block diagram of a decision unit according to the invention
  • FIG. 14 shows a block diagram of a hardware divider
  • FIGS. 15 to 17 show the method of operation of the neighborhood processor in the network
  • FIG. 18 shows a program flowchart for the distance vector and segment number calculation
  • FIG. 19 shows a block diagram of the neighborhood processor
  • FIG. 20 shows a block diagram of the segment number generator
  • FIG. 21 shows a hardware calculation of the validity of a contour point in the edge detection unit and the decision unit shown in FIGS. 12 and 13 , broken down on the basis of the processor clock cycle
  • FIG. 22 shows an illustration of a contour with four scanning directions
  • FIG. 23 shows a schematic illustration of the image processing system with a code to be identified on a motor vehicle.
  • the omnidirectional image processing system 1 ( FIG. 11 ) according to the invention, or the code reader, comprises an imaging system 2 , an analog/digital converter 4 , a digital processing device 6 , a computation device 8 , a microcomputer which is not shown, and an optional illumination device 9 ( FIG. 23 ).
  • This camera 10 represents an exemplary embodiment of the imaging system 2 and of the analog/digital converter 4 and produces digital image data, in the form of a digital image datastream 12 continuously or on a triggered basis.
  • the imaging system 2 may also be a scanner, a CT scanner or the like, for example even an arrangement of a plurality of imaging systems.
  • the image data is shown in the form of gray-scale image in a two-dimensional illustration in FIG. 1 .
  • FIG. 1 shows a slightly rotated data matrix code 18 which is intended to be decoded.
  • a contour point list 20 which is shown in the form of a two-dimensional illustration in FIG. 6 , is produced in video real time in a first main process or first processing step in the digital processing device or preprocessing unit 6 .
  • the digital processing device 6 is in the form of a pipeline processor.
  • FIG. 2 shows an illustration of a detail from the contour point list shown in FIG. 6 .
  • the measurement accuracy of the contour point coordinates directly influences the approximation quality of the network on which the code is based, and thus the reading rate.
  • the location of a contour point 22 is defined by the position of the point of inflection of the gray-scale profile, which is scanned essentially at right angles to the contour direction.
  • the points of inversion of four gray-scale profiles are calculated for each pixel, at the angles 0°, 45°, 90° and 135°, in video real time with sub-pixel accuracy.
  • the point of inflection is calculated via the zero point of the cross-correlation function from the gray-scale profile in the predetermined direction and with the normalized first derivative of the gray-scale profile of a typical edge, based on location, FIG. 4 .
  • FIG. 3 shows the choice of an edge profile for the 90° direction corresponding to FIG. 2 .
  • the local 6 ⁇ 6 environment NBD (neighborhood) of a pixel is stored as an image element in a high-speed RAM, within which image element, as will be explained in the following text, a convolution process is carried out.
  • FIG. 4 shows an illustration in the form of a graph of the profile of the first derivative 24 of the typical gray-scale profile for the 90° direction, based on location.
  • the convolution core is in each case the same for the 0° and 90° as well as for the 45° and 135° scanning directions, and it may also be the same for all the scanning directions.
  • the two convolution products F 90 _ 0 and F 90 _ 1 which are calculated in parallel in time are now calculated for the 90° direction as:
  • This calculation is known as a cross-correlation, with five pixel operators being used in this example.
  • hardware convolvers which operate in parallel in time or operate using the time-division multiplex method are used for this purpose in order to calculate two parallel convolution products F 90 _ 0 and F 90 _ 1 , with the calculation in each case being terminated within the duration of one pixel.
  • the contrast CONT 90: NBD[ 3,1 ]+NBD[ 3,2 ] ⁇ NBD[ 3,5 ] ⁇ NBD[ 3,6] and other functions which will be explained in the following text are calculated by hardware.
  • a list entry is produced in a hardware-based contour point list for the contour point CP[i], which is provided with the sequential index i.
  • CP[i] — 90: [ X, Y, CONT 90]
  • the X and Y coordinates are obtained from the coordinates x i and y i of that pixel which is imaged onto the element NBD[3,3] in the 6 ⁇ 6 environment NBD.
  • the resolution of the X and Y coordinates has been extended by two bits in comparison to the original image, that is to say the resolution has been increased by a factor of four in each direction.
  • This sub-pixel interpolation ensures an improved contour quality even in environments subject to interference, since on the one hand only local threshold values are used, and on the other hand the contour point is largely position-invariant in the depth of focus range of the objective.
  • FIG. 5 shows an illustration of the resolution extension of the X and Y coordinates for sub-pixel interpolation.
  • FIG. 6 shows an illustration in the form of a graph of the contour point list 20 in the 90° direction.
  • contour point lists for the 0°, 45° and 135° directions are calculated in the same way as for the 90° direction, with x being used for the direction in the following text.
  • the lists calculated in this way are stored in a RAM organized in such a way that the contour points are stored in lines in a sequence organized on the basis of their indices i.
  • the algorithm described above is carried out in real time, with respect to the incoming image datastream, by hardware in the processing device 6 as follows.
  • FIG. 11 shows the design of the processing device 6 in the form of a block diagram.
  • the digital processing device 6 or preprocessing unit has a data interface 32 , a memory interface 34 , 12 registers RG 1 to RG 12 as well as a computation device 8 .
  • the figure does not show a digital microcomputer, for example a DSP or other standardized computer which can access the data storage unit 36 .
  • the computation device 8 in turn comprises a register matrix 48 , a plurality of n edge detection units EDU 1 to EDUn and a neighborhood processor 90 .
  • a data storage device 36 is connected to the processing device 6 by means of the memory interface 34 .
  • the image data in the image acquisition unit 10 is transmitted in real time to the data interface 32 (IAU interface) of the preprocessing unit 6 .
  • the data interface 32 is connected to the memory interface 34 (DSU interface) to the external data storage unit 36 (DSU) and is able to make a plurality of, typically eight, write/read accesses to the RAM during the transmission time of one pixel.
  • the data storage unit 36 has or comprises as modules the frame/field memory 38 , the line memory 40 , the contour point memory 42 , the segment memory 44 , and the general purpose memory 46 .
  • the modules 38 to 46 can be provided in a standard memory with a high data rate, or else in a plurality of memories of different speed.
  • One good memory architecture has or comprises three independent high-speed 16-bit memory modules, comprising one dynamic memory M 1 (DRAM) and two solid-state memories M 2 and M 3 (SRAM).
  • DRAM dynamic memory
  • SRAM solid-state memories
  • Table 1 The association between the logic memories and the physical memories is described in Table 1. This architecture results in a data rate of 48-times the pixel rate.
  • the image data and the contour point lists in each case represent larger matrices and are formatted in blocks in the memory interface 34 , and are stored in real time in the dynamic memory M 1 .
  • the current image datastream is likewise stored in the line memory M 2 40 , but in this case only with a reduced depth of six lines.
  • the memory interface 34 reads a matrix of 2 ⁇ 5 pixels from the line memory 40 , and writes the current pixel and the previous pixel as a 16-bit word to the line memory 40 . This results in a datastream which contains two columns of the image matrix with a length of six pixels, and which is stored in the registers RG 1 to RG 12 . The contents of this register are then shifted in lines in two clock cycles to the register matrix NBD 6 ⁇ 6 48 .
  • the register matrix 48 also contains a column which is required as buffer store, but is not shown.
  • the register matrix 48 is represented by 42 registers. Implementations with high-speed internal SRAMs are likewise possible.
  • the sub-pixel coordinates of contour points can be determined directly from the register matrix 48 .
  • the associated gray-scale profiles GP 0 , GP 45 , GP 90 , GP 135 are first of all taken for the edge detection units EDU 1 to EDU 4 with the direction 0°, 45°, 90°, 135°.
  • NBD is an abbreviation for neighborhood, as those skilled in the art will be aware.
  • FIG. 12 illustrates in detail one of the four identical edge detection units, which operate in parallel.
  • the pixels P 1 to P 6 are passed from the register matrix 48 to the NBD multiplexer 52 , with the six pixels being firmly linked to the inputs of an NBD multiplexer 52 of the edge detection unit for the respective direction, in this case EDU 1 .
  • the calculation is carried out by the edge detection unit EDU 1 with the aid of a pipeline processor.
  • the datastream is processed in the edge detection unit, that is to say without any branches, with the processing starting with the delay of six pixels, of course, for the 6 ⁇ 6 environment that is used, but with the procedure from here on being carried out in video real time.
  • the edge detection unit EDU 1 has at least one hardware convolver pair 54 , which are connected to outputs Out 1 to Out 3 of the NBD multiplexer 52 , and which carry out the convolution operations in the datastream, in each case in parallel, for P 1 to P 5 and P 2 to P 6 .
  • the two convolution products in pairs for one direction Fx_ 0 and Fx_ 1 are in this case calculated as follows within one pixel clock cycle and “on the fly” from the data P 1 to P 5 and P 2 to P 6 in the data pipelines DP 1 and DP 2 , to be more precise in the convolver pair 54 .
  • the clocking of the pipeline processor that is used as the basis for the edge detection unit EDU 1 is illustrated in FIG. 21 .
  • the data items P 1 to P 6 are written from the NBD multiplexer 52 to a first additional element Add 1 of the convolver pair 54 , with the first additional element Add 1 being firmly linked to the outputs Out 1 and Out 2 .
  • the gray-scale value of the pixels which are symmetrical in pairs about the point of symmetry of the convolution core 24 that is to say P 2 and P 4 , P 1 and P 5 for Fx_ 0 as well as P 3 and P 5 , P 2 and P 6 for Fx_ 1 , are added in the first additional element Add 1 to give the first intermediate results P 2 +P 4 , P 1 +P 5 , P 3 +P 5 as well as P 2 +P 6 .
  • the first intermediate results are multiplied in a first multiplier Mult 1 by the associated values of the convolution core 24 , specifically CONV[ 1 ] and CONV[ 2 ], and in the clock cycles Clock 4 and Clock 6 they are further added in a second additional element Acc 1 .
  • the data items P 3 and P 4 are read in a corresponding manner in the clock cycles Clock 0 to Clock 3 in the parallel data pipeline DP 2 , which is firmly linked to the output Out 3 of the NBD multiplexer 52 , and are multiplied in the clock cycles Clock 4 to Clock 7 by the point of symmetry CONV[ 3 ] of the convolution core 24 by means of a second multiplier Mult 2 .
  • the convolution results Fx_ 0 and Fx_ 1 are finally calculated in a further adder 55 by addition of the results obtained from the data pipelines DP 1 and DP 2 (the clocking is not illustrated in FIG. 21 ).
  • the convolution products Fx_ 0 and Fx_ 1 (convolution results) from the convolver pair are thus respectively produced at the output 56 of the convolver pair 54 and at the output of the data pipelines DP 1 and DP 2 .
  • This method of calculation of the convolution results is made possible by the symmetry of the convolution core 24 and makes the calculation simpler than by carrying out five multiplication operations in each case.
  • the gradient Grad_ 1 corresponds to the contrast, and the gradient Grad_ 2 is used for artifact suppression. Both gradients are then subjected in a decision unit 60 to a threshold value test with a programmable threshold Grad_ 1 Threshold and Grad_ 2 Threshold respectively. The way in which this calculation is clocked is likewise illustrated in FIG. 21 .
  • the decision unit 60 or the decision-making process is illustrated in detail in FIG. 13 .
  • the decision unit 60 synchronizes the input data and produces a logical variable for each pixel, in the form of a Boolean vector B[k],
  • the result of the Boolean vector thus depends on the convolution results, to be more precise on a total of six operands, which are calculated from the gray-scale value profile and the convolution results.
  • the logical variable B[1] depends on whether the absolute magnitude of the difference between the convolution products Fx_ 0 and Fx_ 1 of one convolver pair 54 exceeds the threshold Fx_Threshold. Furthermore, the logical variable B[2] depends on whether there is a change in the mathematical sign between the two convolution results Fx_ 0 and Fx_ 1 which are associated with one convolver pair 54 . Furthermore, the logical variables B[3] and B[4] depend on whether the convolution result Fx_ 1 is not equal to zero and the convolution result Fx_ 0 is equal to zero.
  • the decision unit 60 has a demultiplexer 62 on the input side, which passes the convolution results Fx_ 0 and Fx_ 1 to two separate registers RG Fx_ 0 and RG Fx_ 1 , respectively. From there, the results are passed to a subtractor 64 and to a logic unit 66 , as well as a comparator 67 with a magnitude-forming input, in order to determine the values B[1] to B[4].
  • the variables B[5] and B[6] test for overshooting of the threshold value Grad_ 1 Threshold and Grad_ 2 Threshold of the gradients Grad_ 1 and Grad_ 2 , respectively by means of a comparator 68 .
  • the corresponding contour point is entered in the contour point list when and only when the complete validity criterion is Contour_point_valid 70 TRUE.
  • a sub-pixel interpolation process is also carried out as shown in FIG. 7 and FIG. 14 .
  • the divider 80 which is illustrated in detail in FIG. 14 , operates sequentially in a simple manner by accumulating the difference Fx_ 0 ⁇ Fx_ 1 in the denominator in an addition element 82 until four-times the value of the counter is reached in a multiplication element 84 .
  • the value ⁇ in the tables LUT 1 and LUT 2 is then transformed to the high-resolution Cartesian coordinate system. During this process, offsets are taken into account by the digital network.
  • the values x i and y i correspond to the coordinates of the pixel which is mapped onto the element NBD[ 3 , 3 ] in the 6 ⁇ 6 environment NBD, taking account of the transformation to high-resolution Cartesian coordinate system.
  • the contour point list 20 is stored organized in lines, so that a format which has typically been shortened to 32 bits is sufficient in this case, in which only the low part of the y coordinate is stored.
  • a separate edge detection unit EDU 1 to EDUn is provided for each direction.
  • Data records which are transmitted to the memory interface 34 are produced at the output of each edge detection unit.
  • the memory interface 34 buffers the data records DS 1 to DSn associated with one direction, and stores the data records DS 1 to DSn in a progressive sequence based on the FIFO principle in the memory M 1 . If a change occurs in the line address, the memory interface 34 adds a data record which marks the line change, in the same way as an end identification after the image end.
  • the storage of the contour point list 20 in the memory M 1 completes the first step or main process of contour point generation.
  • the contour point lists are now loaded line by line in the memory M 3 .
  • the memory interface 34 expands the data in the contour point list CP[i] to the complete structure length that is now required for reading the information.
  • the memory M 3 is organized as a ring buffer with a typical memory depth of four lines, and has four independent buffers for the 0°, 45°, 90° and 135° directions. After identification of a line end marking, the writing process is in each case stopped, and the neighborhood processor 90 is started.
  • the current, most recently written line is written to the neighborhood processor 90 .
  • list data is processed using the sub-pixel format according to the invention.
  • the neighborhood can be defined flexibly depending on the object to be achieved.
  • the insertion of the contrast threshold value prevents or makes more difficult the combination of contours of different strength.
  • a different distance measure for example the Euclidean distance, may, of course, also be used instead of the simplified distance.
  • the neighborhood relationships between contour points 22 in this example are determined as follows.
  • a Pointer CurPtr in the first direction to be investigated is first of all set to the first contour point in the current line j, and a Pointer AboveStartPtr and a Pointer AbovePtr are set to the first contour point in the line j ⁇ 1, and the addressed contour points are written to the neighborhood processor 90 .
  • the neighborhood processor 90 segments contour points as a function of the neighborhood criterion NC, and enters identical segment numbers when neighborhood is confirmed for adjacent elements. If a contour point has no neighbors with segment numbers which have already been allocated, a new segment number is allocated. If it is found that a single contour point links segments with a different segment number, the entry is made in a coincidence list.
  • FIGS. 15 , 16 and 17 show the method of operation of the neighborhood processor 90 in the network.
  • the points P 11 , P 12 etc. from the previous above line j ⁇ 1 as well as the points P 21 , P 22 etc. from the current line j include the respective contour point addresses with sub-pixel accuracy, the contrast value CONT as well as the segment number SegNo.
  • FIG. 15 shows the initial state.
  • the Pointer AboveStartPtr in this case points to P 11 , the Pointer CurPtr to P 21 , and the Pointer OldCurPtr is invalid.
  • FIG. 16 shows the status for the transition to the next point P 22 to be processed.
  • the Pointer CurPtr now points to P 22 , the Pointer OldCurPtr to P 21 , and the Pointer AboveStartPtr is (iteratively) shifted to P 12 .
  • FIG. 17 shows which points are investigated during testing for neighborhood with the line located above: the pointer AbovePtr is initiated with the value of AboveStartPtr, and then runs via P 12 , P 13 and P 14 .
  • the current contour point to be processed is read (Read current contour point record, RCCP), corresponding to CurPtr, as described above.
  • AbovePtr is first of all initialized from the current AboveStartPtr (this is not shown in the flowchart for clarity reasons).
  • CurPtr has still not received a segment number after the end of the iteration of AbovePtr, a new segment number is produced and is allocated to it (Create new segment number for current contour point, NSC). Finally, the old CurPtr is stored in OldCurPtr and CurPtr is incremented (Increment current pointer and update OldCurent pointer, ICP). This completes the transition to the next contour point to be processed, and the entire procedure is repeated until all of the contour points have been processed.
  • FIG. 8 shows a single contour point, having or comprising a filter direction, coordinates, a contrast measure, a segment number and coincidence information, stored in a 64-bit long data structure in the contour point list CP_long.
  • the splitting of the bits between the individual data items as illustrated in FIG. 8 allows high memory efficiency.
  • All of the resultant contour points from all of the filter directions are stored in a common continuous list in the memory.
  • an organization structure is maintained, which is defined by the coordinates of the contour points and the filter direction by means of which the contour point has been produced.
  • the points are stored in the form of pixel lines, in each case from left to right, starting with the top left-hand corner of the image, with contour points having the same pixel coordinates being stored separately on the basis of the filter directions.
  • the end of the list is marked by an entry in which all of the bits are set to 0.
  • FIG. 9 shows an example of a hexadecimal contour point list CP_long, in the form of a so-called “Hexdump”, with two contour points per line.
  • segment number determined in this way and the coincidence information are likewise written to the data structure CP_long. Once the segment numbers for a complete line have been determined, the relevant line is written via the memory interface 34 to the memory M 1 . This completes the second main process of segmentation.
  • the neighborhood processor 90 which is illustrated in detail in FIG. 19 has two registers RG CurPtr and RG OldCurPtr, to which the corresponding contour points are written from the line memory 40 . These registers are connected directly to a multiplexer 92 , in the same way as the line memory.
  • the contour point separation is determined by means of a subtraction element 94 , and is subjected by means of a comparator 96 to a threshold value analysis with a maximum distance value.
  • the segment number generator 98 is illustrated in detail in FIG. 20 .
  • the third main process in which the statistical moments S are calculated, is carried out with a time delay after the second main process.
  • a contour point sequence passes through a computation unit, defined as a statistical processor, suitable for determination of the statistical moments.
  • the X and Y coordinates as well as the contrast CONT are copied from the structure of the contour point and are accumulated in accordance with the equations for moment calculation.
  • a structure with moments S in this example up to the second order, but for specific applications also a higher order, is stored in the memory M 3 as follows:
  • a calculation up to the third order is particularly advantageous for identification of non-linear, for example curved, edges or structures.
  • This list is stored via an interface that is not shown and can be internally compressed further, or can be read directly by a microcomputer.
  • the list Obj_Moments[i] has a considerably smaller data volume than the original gray-scale value image.
  • the compression rate reaches values of about 1:200 to 1:5000 with respect to the original image of the imaging system or of the CCD camera, depending on the image content, without this resulting in any data losses for the network approximation.
  • FIG. 22 shows one implementation, with four directions DIR 1 to DIR 4 .
  • the directions DIR 1 to DIR 4 are rotated through 90° with respect to the scanning directions.
  • the path lies within the direction sector DIR 2 , and records paths in the angle range ⁇ 22.5° . . . 67.5°> in the Cartesian coordinate system.
  • the difference gradients are annotated m_ 0 , m_ 45 , m_ 90 and m_ 135 and are calculated as follows, approximated by means of Taylor series:
  • centroids (x 0 , y 0 ) of the digital paths are calculated using:
  • An object filter which, in particular, suppresses noise, paints identical segments which have been found in different directions and, when required, combines segments on the basis of the coincidence information.
  • the object filter criterion ( n> 2) AND ( abs ( m — DIR ) ⁇ arctan (22.5°)) can be used for this purpose.
  • FIG. 10 shows an illustration in the form of a graph of the approximated and filtered digital paths 104 .
  • the resultant segments are combined with sub-pixel quality to form objects, depending on the application to be implemented.
  • these are initially network structures, which are then used for scanning the gray-scale values of the code.
  • the method which is carried out on the image processing system according to the invention can be implemented omnidirectionally in video real time.
  • the recording of all the available contours allows the network to be locally reconstructed, so that it is possible to read even damaged codes or codes distorted by parallex.
  • the computation performance of the microprocessor used is adequate for the network reconstruction and decoding proposed here in real time.
  • the high-speed image processing described above, in particular segmentation of gray-scale value images with sub-pixel resolution is largely generally applicable and can also be used for a large number of other applications, for example in industrial metrology, robot control, for inspection tasks and for high-speed segmentation tasks in the logistics fields (reading scripts in gray-scale value images, reading codes) and for high-speed automatic object identification, for example in the point-of-sale (POS) area.
  • POS point-of-sale
  • biometric forms and structures for example for fingerprint, iris and facial identification.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
US10/597,554 2004-01-30 2004-01-30 Image processing system Expired - Fee Related US7978914B2 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2004/000831 WO2005073911A1 (fr) 2004-01-30 2004-01-30 Systeme de traitement image

Publications (2)

Publication Number Publication Date
US20090129699A1 US20090129699A1 (en) 2009-05-21
US7978914B2 true US7978914B2 (en) 2011-07-12

Family

ID=34814527

Family Applications (1)

Application Number Title Priority Date Filing Date
US10/597,554 Expired - Fee Related US7978914B2 (en) 2004-01-30 2004-01-30 Image processing system

Country Status (5)

Country Link
US (1) US7978914B2 (fr)
EP (1) EP1709587B1 (fr)
CA (1) CA2554453A1 (fr)
DE (1) DE502004005457D1 (fr)
WO (1) WO2005073911A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11694074B2 (en) 2018-09-07 2023-07-04 Samsung Electronics Co., Ltd. Integrated circuit that extracts data, neural network processor including the integrated circuit, and neural network device

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
SE528063C2 (sv) 2004-11-12 2006-08-22 Zi Decuma Ab Metod och anordning för segmenteringsbaserad igenkänning
WO2008054325A1 (fr) * 2006-11-02 2008-05-08 Soo Seng Ng Dispositif de support pour la manutention de matériaux
FR2938680B1 (fr) 2008-11-14 2012-11-16 Saint Louis Inst Procede de construction en temps reel de vecteurs prototypes a partir de donnees d'entree d'un processus neuronal
DE102009006660A1 (de) 2009-01-29 2010-08-05 Baumer Innotec Ag Pipeline-Recheneinrichtung zur Verbindung von Konturelementen aus Bilddaten
US9565966B2 (en) * 2010-10-28 2017-02-14 Voice Systems Technology, Inc. Coffee filter qualification apparatus and methodology
CN107644398B (zh) * 2017-09-25 2021-01-26 上海兆芯集成电路有限公司 图像插补方法及其相关图像插补装置
US10990840B2 (en) * 2019-03-15 2021-04-27 Scenera, Inc. Configuring data pipelines with image understanding
CN110929779B (zh) * 2019-11-19 2023-05-02 腾讯科技（深圳）有限公司 重建神经元质量检测方法、有序点云分类方法及装置
CN111583292B (zh) * 2020-05-11 2023-07-07 浙江大学 一种面向双光子钙成像视频数据的自适应图像分割方法

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4910786A (en) * 1985-09-30 1990-03-20 Eichel Paul H Method of detecting intensity edge paths
US5151953A (en) 1990-12-10 1992-09-29 Harris Corporation Single chip 2-D convolver
EP0587301A2 (fr) 1992-08-12 1994-03-16 International Business Machines Corporation Méthode pour extraire des points d'inflexion pour la reconnaissance optique de caractères
WO1996018158A1 (fr) 1994-12-09 1996-06-13 United Parcel Service Of America, Inc. Procede et appareil pour la convolution simultanee de multiples images binaires numeriques a l'aide d'un seul convolutionneur dote d'un masque binaire pour determiner les densites de pixels
US5796868A (en) * 1995-12-28 1998-08-18 Cognex Corporation Object edge point filtering system for machine vision
US6064768A (en) 1996-07-29 2000-05-16 Wisconsin Alumni Research Foundation Multiscale feature detector using filter banks
CA2376683A1 (fr) 1999-06-07 2000-12-14 Metrologic Instruments, Inc. Identification d'emballage individuel et systeme de dimensionnement utilisant un procede de balayage radar laser
US20040047498A1 (en) * 2000-11-22 2004-03-11 Miguel Mulet-Parada Detection of features in images
US7894675B2 (en) * 2003-07-18 2011-02-22 Lockheed Martin Corporation Method and apparatus for automatic linear object identification using identified terrain types in images

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5611426A (en) * 1995-10-12 1997-03-18 Point Group Corporation Packaging assembly for compact discs
US5638952A (en) * 1996-04-29 1997-06-17 Kim; Jung Won Portable storage case for a plurality of compact discs and/or 3.5 inch floppy disketters
US5791468A (en) * 1997-09-25 1998-08-11 Arthur Meyer Paperboard compact disk package
NL1017409C2 (nl) * 2001-02-20 2002-08-22 Fountain Tech Bv Verpakkingsinrichting voor in hoofdzaak plaatvormige informatiedragers en werkwijze voor vervaardiging daarvan.

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4910786A (en) * 1985-09-30 1990-03-20 Eichel Paul H Method of detecting intensity edge paths
US5151953A (en) 1990-12-10 1992-09-29 Harris Corporation Single chip 2-D convolver
EP0587301A2 (fr) 1992-08-12 1994-03-16 International Business Machines Corporation Méthode pour extraire des points d'inflexion pour la reconnaissance optique de caractères
WO1996018158A1 (fr) 1994-12-09 1996-06-13 United Parcel Service Of America, Inc. Procede et appareil pour la convolution simultanee de multiples images binaires numeriques a l'aide d'un seul convolutionneur dote d'un masque binaire pour determiner les densites de pixels
US5796868A (en) * 1995-12-28 1998-08-18 Cognex Corporation Object edge point filtering system for machine vision
US6064768A (en) 1996-07-29 2000-05-16 Wisconsin Alumni Research Foundation Multiscale feature detector using filter banks
CA2376683A1 (fr) 1999-06-07 2000-12-14 Metrologic Instruments, Inc. Identification d'emballage individuel et systeme de dimensionnement utilisant un procede de balayage radar laser
US20040047498A1 (en) * 2000-11-22 2004-03-11 Miguel Mulet-Parada Detection of features in images
US7894675B2 (en) * 2003-07-18 2011-02-22 Lockheed Martin Corporation Method and apparatus for automatic linear object identification using identified terrain types in images

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
"Canadian Office Action for Canadian International Application No. CA 2,554,453", Jan. 17, 2011, Publisher: Canadian Patent Office, Published in: Canada.
International Preliminary Examination Report.
Paul R. Beaudet, Time and Space Multiplexing Focal Plane Convolvers, Proceedings of the Spie, Jan. 16, 1989-Jan. 18, 1989, pp. 90-98, XP009037462, vol. 1071, Optical Sensors and Electronic Photography.
R. Massen et al., Real-time Symbol Extraction from Grey-level Images, Proceedings of the Spie, Nov. 17, 1987-Nov. 18, 1987, pp. 19-25, XP009037482, vol. 860, Real Time Image Processing Concepts and Technologies.

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11694074B2 (en) 2018-09-07 2023-07-04 Samsung Electronics Co., Ltd. Integrated circuit that extracts data, neural network processor including the integrated circuit, and neural network device

Also Published As

Publication number Publication date
EP1709587B1 (fr) 2007-11-07
US20090129699A1 (en) 2009-05-21
CA2554453A1 (fr) 2005-08-11
DE502004005457D1 (de) 2007-12-20
WO2005073911A1 (fr) 2005-08-11
EP1709587A1 (fr) 2006-10-11

Similar Documents

Publication Publication Date Title
Prewitt Object enhancement and extraction
US6516087B1 (en) Method for real time correlation of stereo images
Broggi Parallel and local feature extraction: A real-time approach to road boundary detection
US10621446B2 (en) Handling perspective magnification in optical flow processing
US20100128789A1 (en) Method and apparatus for processing video sequences
JP2007000205A (ja) 画像処理装置及び画像処理方法並びに画像処理プログラム
US7978914B2 (en) Image processing system
CN111753669A (zh) 基于图卷积网络的手部数据识别方法、系统和存储介质
CN112419372B (zh) 图像处理方法、装置、电子设备及存储介质
Verly et al. Adaptive mathematical morphology for range imagery
CN112435223A (zh) 目标检测方法、装置及存储介质
US6249603B1 (en) Efficient search for a gray-level pattern in an image
EP0460960A2 (fr) Traitement de données
US7388584B2 (en) Method and program for determining insides and outsides of boundaries
Guichard et al. Contrast invariant image analysis and PDE’s
Ben-Tzvi et al. A dynamic combinatorial Hough transform
CN115731179A (zh) 轨道部件检测方法、终端及存储介质
JP3300092B2 (ja) 画像特徴抽出装置
CN110309738B (zh) 一种对oct指纹图像进行标注的方法
JP2016085694A (ja) 画像処理装置及びその制御方法、プログラム
US20040013299A1 (en) System and method for contrast enhanced registration with complex polynomial interpolation
Choi et al. An interpolation method for strong barrel lens distortion
CN116403269B (zh) 一种遮挡人脸解析方法、系统、设备及计算机存储介质
Wei et al. Foveate wavelet transform for camera motion recovery from videos
US6310983B1 (en) Efficient search for a gray-level pattern in an image using ranges of sums

Legal Events

Date Code Title Description
AS Assignment

Owner name: BAUMER OPTRONIC GMBH, GERMANY

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BEIKIRCH, LARS;FEDORENKO, SERGEJ;SCHWARZ, DITTMAR;AND OTHERS;REEL/FRAME:018651/0438

Effective date: 20060824

STCF Information on status: patent grant

Free format text: PATENTED CASE

CC Certificate of correction
FPAY Fee payment

Year of fee payment: 4

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 8

FEPP Fee payment procedure

Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

LAPS Lapse for failure to pay maintenance fees

Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STCH Information on status: patent discontinuation

Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362

FP Lapsed due to failure to pay maintenance fee

Effective date: 20230712