SE522437C2 - Method and apparatus for extracting information from a target area within a two-dimensional graphic object in an image - Google Patents

Abstract

A method is presented for extracting information from a target area (101) within a two-dimensional graphical object (100) having a plurality of predetermined features (23) with known characteristics in a first plane. An image (102) is read where the object (100) is located in a second plane, which is a priori unknown. A plurality of candidates (108) to the features in the second plane are identified in the image. A transformation matrix (H) for projective mapping between the second and first planes is calculated from the identified feature candidates. The target area (101) of the object is transformed from the second plane into the first plane. Finally, the target area is processed so as to extract the information.

Description

l5 20 25 30 522 437 2 The high resolution of these sensors makes it possible to take pictures of objects with sufficiently high accuracy to be able to process the pictures with satisfactory results.

However, an image taken from a handheld device gives rise to rotational and perspective effects.

In order to thus extract and interpret the desired information within the image, a projecting transformation is needed. Such a projecting transformation requires at least four different point correspondences where no three points lie on the same line.

SUMMARY OF THE INVENTION In light of the above, an object of the invention is to facilitate detection of a known two-dimensional object in an image to enable extraction of desired information stored in a target area within the object, even if the image is recorded in a unpredictable environment, and thus at unknown angles, rotation and light conditions.

Another object is to achieve a universal detection method, which, with a minimum of adjustments, is adaptable to a number of known objects. Yet another object is to provide a detection method which is efficient in terms of computing power and memory usage, and which is thus particularly suitable for handheld image recording devices.

The above objects are generally achieved by a method and an apparatus according to the appended independent claims.

According to the invention, there is thus provided a method for extracting information from a target area within a two-dimensional graphic object having a plurality of predetermined features with, in a first plane, known character - ristik. The method comprises the steps of: reading an image in which said object is placed in a second plane, said second plane being unknown in advance, identifying in said image a plurality of candidates for said predetermined features in said second plane, that from said identified plurality feature candidates, calculating a transformation matrix for projecting imaging between said second and first planes, transforming said target area on said object from said second plane to said first plane, and processing said target area to extract said information.

The device according to the invention is advantageously a hand-held device used for detection and interpretation of a known two-dimensional object in the form of a sign in a single image, which is registered at an unknown angle, rotation and light conditions. In order to locate the known sign in such an image, specific features of the sign are identified. The feature identification is preferably based on the edges of the sign. This enables a solution that is adaptable to most existing signs, as the features are as general as possible and common to most signs. To find lines based on the edges of the sign, an edge detector based on the Gaussian core is preferably used. Once all edge points have been identified, they are grouped into lines. The Gaussian core is also used to advantage to locate the gradient of the edge points. The corner points on the inside of the edges are then used as special feature candidates. These corner points are obtained when the lines running along the edges intersect. 10 15 20 25 30 LW BJ NJ $> OJ * J 4 In an alternative embodiment, if there are other very prominent features on the sign (eg points with a specific grayscale, color, intensity or luminescence), these can be used instead of or in addition to edges, as such prominent features are easier to detect.

Once a specific set of feature candidates has been identified, an algorithm based on the algorithm commonly known as RANSAC is executed to verify that the features are set correctly and to calculate a transformation matrix. After ensuring that the features are in the correct geometric arrangement, any target area of the object can be transformed, extracted and interpreted with, for example, an interpreter for OCR or barcodes.

Other objects, features and advantages of the present invention will become apparent from the following detailed description, from the appended dependent claims, as well as from the drawings.

Brief Description of the Drawings A preferred embodiment of the present invention will now be described in more detail, with reference to the accompanying drawings.

Fig. 1 is a schematic view of an image recording device according to the invention, in the form of a hand-held device.

Fig. 1a shows both the image recording device in Fig. 1 and a computer environment in which the device can be used.

Fig. 2 is a block diagram illustrating important parts of the image recording apparatus shown in Fig. 1. <2 ii. ": R '~. 10 15 20 25 30 522 437 5 Fig. 3 is a flow chart illustrating the overall steps performed by the method according to the invention.

Fig. 4 is a flow chart illustrating in more detail one of the steps in Fig. 3.

Fig. 5 is a diagram for illustrating a mask for smoothing and derivation, which is applied to a recorded image during a step in the method illustrated in Figs. 3 and 4.

Figs. 6-17 are photographs illustrating processing of a recorded image during different steps of the method illustrated in Figs. 3 and 4.

Detailed Description of a Preferred Embodiment The remainder of this specification has the following layout: Section A provides a general overview of the method and apparatus of the preferred embodiment.

To better understand the material covered by this specification, an introduction to projecting geometry in terms of homogeneous notation is given and a camera projection matrix is described in section B.

Section C provides an explanation of how the transformation matrix or homography matrix can be obtained when similarities in feature points have been identified.

An explanation of what kind of feature should be chosen, and why, can be found in section D.

Section E describes a line detection algorithm.

Section F provides a description of the type of information that can be obtained from lines.

Once the feature points have been identified, the homography matrix can be calculated, which is done using a RANSAC algorithm, as explained in section G. Section H describes how the desired information can be extracted from the target area.

Finally, section I touches on some alternative embodiments.

A. General Overview A preferred embodiment will now be described, in which the object to be recognized and read from is a sign 100, which is shown at the bottom of Fig. 1. However, it should be emphasized that the invention is not limited to signs only. The sign 100 is intended to look like any other sign. The target area 101, from which the information is to be extracted and interpreted, is the area with the numbers "12345678", and is indicated in Fig. 1 by a dashed frame. As will be appreciated, the plate 100 does not contain much information which can be used as a feature.

As with many other signs, the sign 100 is surrounded by a frame. The edges of this frame give rise to lines. The preferred embodiment is based on the use of these lines as features. However, any type of feature can be used as long as at least a total of four feature points can be distinguished. If the sign contains some specific features (for example, dots with a specific color), these can be used instead of, or in addition to, the frame, as they are usually easier to detect.

Fig. 1 illustrates an image generating handheld device 300, which implements the device according to the preferred embodiment, and by means of which the method according to the preferred embodiment can be performed. The handheld device 300 has a housing 1 having approximately the same shape as a conventional highlighter. A short side of the housing has a window 2, through which images are registered for various image-based functions of the handheld device. 10 15 20 25 30 522 457 7 Essentially, the housing 1 contains an optics part, an electronics part and a power supply.

The optical part comprises a number of light sources 6, such as LEDs, a lens system 7 and an optical image sensor 8, which interfaces with the electronics part. The LEDs 6 are intended to illuminate a surface of the object (sign) 100, which at any moment is within sight of the window 2. The lens system 7 is intended to project, as accurately as possible, an image of the surface of the photosensitive sensor. sensor 8. The optical sensor 8 may consist of an area sensor, such as a CMOS sensor or a CCD sensor with a built-in A / D converter. Such sensors are commercially available. The optical sensor 8 can produce VGA ("Video Graphics Array") images with a resolution of 640x480 and 24 bit color depth. Thus, the optics part forms a digital camera.

In this example, the power supply of the handheld device 300 is a battery 12, but alternatively it may be a mains connection (not shown).

As shown in more detail in Fig. 2, the electronics part comprises a processing device 20 with storage means 21. The processing device 20 can be implemented by a commercially available microprocessor, such as a CPU ("Central Processing Unit") or a DSP ("Digital Signal Processor"). "). Alternatively, the processing device 20 may be implemented as an ASIC ("Application-Specific Integrated Circuit"), as discrete analog and digital components, or any combination thereof.

The storage means 21 comprises various types of memory (RAM) programmed programs 22 for performing the method according to it such as a working memory and a read only memory (ROM). Asso-preferred embodiment is stored in the storage means 21.

In addition, the storage means 21 comprises a set of object feature definitions 23 and a set of internal camera parameters 24, the purpose of which will be described in more detail later. Registered images are stored in an area 25 in the storage means 21.

As shown in Fig. 1a, the handheld device 300 may be connected to a computer 200 via a transmission link 301. The computer 200 may be a standard personal computer with circuits and programs which enable communication with the handheld device 300 through a communication interface 210. For this purpose, the electronics part may also comprise a transceiver 26 for transmitting information to / from the computer 200. The transceiver 26 is preferably adapted for short-range radio communication in accordance with, for example, the Bluetooth standard on the 2.4 GHz ISM band ("Industrial, Scientific and Medical"). "). The transceiver may, however, alternatively be adapted for infrared communication (such as IrDA - "Infrared Data Association", which is indicated by dashed lines at 26 ') or wired serial communication (such as RS232, which is indicated by dashed lines at 26'). ), or virtually any available standard for short-range communication between a handheld device and a computer.

The electronics part further comprises buttons 27, by means of which the user can control the hand-held device 300 and specifically switch between its various modes of functionality.

Optionally, the handheld device 300 may include a monitor 28, such as a liquid crystal display (LCD).

In the context of the present invention, as shown in Fig. 3, the generally important function of the handheld device 300 is to identify a known two-dimensional object 100 in an image which is registered by the handheld device 300 at an unknown angle, rotation and illumination (steps 31-33 in Fig. 3). Once the two-dimensional object has been identified in the registered image, a transformation matrix (step 34 in Fig. 3) is determined for the purpose of projecting (step 35 in Fig. 3) the target area 101 within the registered image of the two-dimensional object 100 by projection. to a plan that is suitable for further processing of the information within the target area.

The target area 101 can simply be transformed into a predetermined first plane, which may be the normal plane of the optical axis of the handheld device 300, so that it appears that the image was recorded directly in front of the window 2 of the handheld device 300, rather than at an unknown angle and rotation.

The first plane comprises a number of features which can be used for the transformation. These features can be obtained directly from the physical object 100, to be imaged by direct measurements at the object itself.

Another way to obtain such information is to take a picture of the object and to measure the image itself.

Finally, the transformed target area is processed by, for example, optical character recognition (OCR) or bar code interpretation, to extract the desired information (steps 36 and 37 in Fig. 3). To this end, the preferred embodiment includes at least one of an OCR module 29 and a bar code module 29 '. Advantageously, such modules 29 or 29 'can be implemented as program code 22, which is stored in the storage means 21 and executed by the processing device 20.

The extracted information can be used in many different ways, either internally in the handheld device 10 22 20 25 30 522 437 10 300, or externally in the computer 200, after being transmitted over the transmission link 301.

Non-limiting examples of use include a night watchman who checks where and when during his night shift he was at different locations, by capturing images of generally identical signs 100 containing different information as he walks around the protected area, a sales assistant using the handheld device 300 for inventory purposes, tracking goods in industrial areas, or for registering registration numbers for cars and other vehicles.

The handheld device 300 can advantageously provide other image-based services, such as scanner functionality and mouse functionality.

The scanner functionality is used to register text. The user moves the input device 300 over the text he wants to register. The optical sensor 8 detects images with partially overlapping content. The images are composed of the processing device 20. Each character in the composite image is located and by means of, for example, the software for neural networks in the processing device 20, its corresponding ASCII character is determined. The text thus converted to character-coded format can be stored, in the form of a text string, in the handheld device 300 or transmitted over the link 301 to the computer 200. The scanner functionality is described in more detail in the applicant's publication No. WO98 / 20446, which is incorporated herein by this reference.

The mouse functionality can be used to control a cursor on the computer screen 200 of the computer 200. When the handheld device 300 is moved over an outer base surface, the optical sensor 8 detects a plurality of partially overlapping images. The processing device 20 determines position signals for the marker of the computer 200 based on the relative positions of the recorded images, which are determined by the content of the images. The mouse functionality is described in more detail in the applicant's publication No. WO99 / 60469, which is hereby incorporated by reference.

Additional image-based services can be provided by the handheld device 300, such as traditional image or camcorder functionality, drawing tools, translation of scanned text, address book, calendar or Short Messages Services via a mobile phone , such as a GSM telephone ("Global System for Mobile Communications", not shown in Fig. 1).

B. Projecting Geometry This chapter introduces the main geometric ideas and designations necessary for understanding the material covered in the remainder of this specification.

Introduction In Euclidean geometry, the coordinate pair (LJO in Euclidean space R2 can represent a point in the real plane. Thus, it is common to identify a plane with R2. If one considers R2 as a vector space, coordinates can be identified as vectors. This section will to introduce homogeneous representation of points and lines in a plane.The homogeneous representation provides a consistent designation for projecting images of points and lines.This designation will be used to explain images between different representations of planes.

Homogeneous coordinates A line in a plane is represented by the equation ax + by + c == 0, where different choices on a, b and c give rise to different lines. The vector representation of this line is "'íf-šš lâcïi (j: åífšš lšíllíišf fi ë -Éší-i. ÉWI íïx-'íß 10 15 20 25 12 l = (mb4 fl T. On the other hand, also represents the equation (hÛx + (MÛy + kc = O the same line for a constant k, different from zero.Thus, the correspondence between lines and vectors is not identical, since two vectors related by an overall scaling are considered to be equal.A class of equivalents with vectors under this equivalence ratio is known as homogeneous vectors. The set of equivalence classes for vectors in R3 - {0ßÅDT forms the projecting space P2. The designation - (OQJDT means that the vector (OQÅDT is excluded.

A point represented by the vector x = (x¿0T lies on the line l = (mbøjt if and only if ax + by + c == 0. This equation can be written as a scalar product of two vectors, (x¿ @ UQLhcf ~ = 0. Here the point is represented as a three-vector (LyJ), by adding a last coordinate of l to the 2-vector Using the same terminology as above, we note that (knk% kjQLhcf '= 0 , which means that the vector kQLyJ) represents the same point as Qnyl) for all constants k that are non-zero, thus the vector set kU¿% DT can be considered as the homogeneous representation of the point (x¿ÛT in R2. An arbitrary homogeneous vector representing a point has the form T x = (x1, x2, x3) xl xz T '2 This vector represents the point (x, x) 1 R, 3 3 if x3 # 0.

A point represented as a homogeneous vector is thus also part of the projecting space P2. A special case of a point x = (xUxpxQT in P2 is when x3 = 0.

This does not represent a finite point R2. In P2, these (F. C931 t flm: 10 15 15 25 25 522 457 13 points are known as ideal points, or infinity points.

The set of all ideal points is represented by x = (xvxP0fÄ This set lies on a single line known as the infinity line and is denoted by the vector La = (QOJY. Through calculations one can verify that ljx = (o, o, 1) (x1, x2 , o) f = o.

Homographs or projected images When points are mapped from one plane to another, the ultimate goal is to find a single function that images each point from the first plane uniquely to a point in the second plane.

A projectivity is an invertible mapping h from P2 - + P2 such that x1, xz and x3 are on the same line if and only if hßq), hßq) and hßg) do so (see Hartley, R. and Zissermann, A., " Multiple View Geometry in computer vision ", Cambridge University Press, 2000). A projectivity is also called a colonization, a projective transformation or a homography.

This image can also be written as Mk) = L fi g where x, MX) e P2 and H 'are a non-singular 3x3 matrix. H is called a homography matrix. From now on we will denote x '= h (x), which gives: or only x' = Hx. Since both x 'and x are homogeneous representations of points, H can be changed by multiplying any arbitrary constant from zero without changing the homography transformation. This means that H is only determined up to a scale. A matrix like this is called a homogeneous matrix. Consequently, H has only eight degrees of freedom, and the scale can be chosen so that one of its constituents (for example, hg) can be assumed to be 1. However, if the coordinate system's origin of H is mapped to a point at infinity, it can be proved that h9 = 0 and scaling H so that h9 = 0 can therefore lead to unstable results. Another way to select a representation for a homography matrix is to require that the amount ß fl = 1.

Camera projection matrix A camera is an image from the three-dimensional world to the two-dimensional image. This image can be described as: x P11 P12 P13 P14 y = P21 P22 P23 P24 Z P31 P32 Paz P34 ~ N '~ <:> <or shorter, x == PX. X is the homogeneous representation of the point in the three-dimensional coordinate frame. x is the corresponding homogeneous representation of the point in the two-dimensional coordinate frame. P is the homogeneous 3x4 camera projection matrix. For a complete derivation of P (see Hartley, R., and Zissermann, A., "Multiple View Geometry in Computer Vision", Cambridge University Press, 2000, pp. 139-144), where the camera projection matrix for the basic holding camera is derived. P can be factorized as: P = 1 In this case, K is the 3x3 calibration matrix, which contains the camera's internal parameters. R is a 3x3 rotation matrix and t is the 3xl translation vector. This factorization will be used below.

On plane Assume that we are only interested in mapping points from the world coordinate frame that is in the same plane n. Since we are free to choose our world coordinate frame as we wish, we can, for example, define n: Z == O.

This reduces the equation above. If we denote the columns in the camera projection matrix with p ,, we get: xy: bl P2 P3 P4 1 .X: bn P2 Pa Y '1 The image between the points x ”= (ÅQYJ) T on n, and their corresponding points on the image x ', is a common plane homography x' = Hx ,,, where H = [p1 pz p,].

Additional limitations If we have a calibrated camera, the calibration matrix K will be known and we can obtain additional information. Since P = 1 \ 1131 :; 20:12 üjä ífäië-§.f; f '; «<: 10 15 20 25 30 522 437 16 and the calibration matrix K is not invertible, we can obtain: K_1P = R [I i" t] = K_liP1 P2 p: P4i = K_l [h1 hz P3 hai- The first two columns of the rotation matrix R are equivalent to the first two columns of K "Uï. If we denote these two columns by r, and rz, we get: [fl rzi = K4 | ïhl hzi 'Since the rotation matrix is orthogonal, Q and 13 should be orthogonal and of the same length. However, as we have mentioned before, H is only determined up to scale, which means that Q and Q will not be normalized, but will still be of the same length.

Conclusion: With a calibrated camera we obtain two additional constraints on H: where [fl rzizK-lihx hz] ~ C. Solution for the homography matrix H The first thing to consider when solving the equation for the homography matrix H is how many corresponding points x 'éè x needed. As mentioned in section B, H has eight degrees of freedom. Because we work in 2D, each point has limitations in two directions, and thus each point correspondence has two degrees of freedom. This means that a lower limit of four matching points in the two different coordinate frames is needed to ~ »5:22: father? -§ =: = »; ~" *; '; = f;': r 522 437 17 calculate the homography matrix H. This section will show different ways of solving the equation for H.

The algorithm for direct linear transformation (DLT) For each point correspondence we have the equation 5 x'i = Hx ,. . Note that since we are working with homogeneous vectors, x ',. and Hxi differ in scale.

The equation can also be expressed as a vector cross product x ',.> The scale factor will be removed. If we denote the 10th row in H with hfT, then Hxl. expressed as: h1Tx.

I Hx- = hzïx.

I I h3TX.

Using the same terminology as in section B, the cross product above can be expressed as: 3T 2T y: h Xi _W'i h Xi x ', .xHx ,. = w ',. h1Tx ,. -xßhaïxi = 0. 2T lr x'ih Xí-yßh X, - 15 Since hfTxi = xirhj for j = 1..3, we can rearrange the equation and obtain: OT - w ',. xf y ', xf h * x',.> - 'xT x'.xT OT h3 I l II We now face three linear equations with eight unknown elements (the nine elements in H minus one due to few? Lffnåé ïf} : ". âï _ = _ 1i ÉPÉ iä'-C-'f: ï '; Ii§Ä fi.? 2 íšïšíïnšwšrfi-'JZ 10 15 20 25 522 437 18 of the scale factor). Since, however, the third line is linearly dependent on the others two lines, provides only two of the equations useful information.

Therefore, each point correspondence gives us two equations. If we use four point correspondences, we will have eight equations with eight unknown elements. This system can now be solved using Gaussian elimination.

Another way to solve the system can be with the help of SVD, as described below.

Singular value factorization (SVD) In real life, we usually do not get the positions of the points to be exact, due to noise in the image. The solution to H will thus be inaccurate.

To get an H that is more accurate, we can use more than four point matches and then solve a definite system. If, on the other hand, the points are exact, the system will give rise to equations that are linearly dependent on each other, whereby we will again have eight equations that are linearly independent.

If we have n numbers for point agreement, we can denote the equation set with Ah == 0, where A is 2nx9 matrix, and One way to solve this system is by minimizing the Euclidean normal "Ah" instead, under the constraint "ML = k, where k is a constant from zero This last limitation is due to H being homogeneous Minimizing the normal "Ah" is the same as optimizing the problem: n fi nAh.

H flfl A solution to this problem can be obtained through SVD.

A detailed description of SVD is given in Golub, G. H., and Van Loan, C. F., "Matrix Computations", 3rd Edition, The John Hopkins University Press, Baltimore, MD, 1996.

With the help of SVD, the matrix A can be factorized to: A == USVT, where the last column with V gives the solution of h.

Constraints on the corresponding points If three points, of the four point similarities, are collinear, they will give rise to a sub-determined system (see Hartley, R., and Zissermann, A., "Multiple View Geometry in computer vision"). Cambridge University Press, 2000, page 74), and the solution from SVD will degenerate. We will thus be limited when we choose to be feature points, so that we do not choose collinear points.

D. Feature Restrictions An important question is how to find features in objects. Since the results are preferably assumed to be applicable to already existing signs, it is desirable to find features which are often used and which are easy to detect on an image. A good feature must meet as many of the following criteria as possible.

Be easy to detect; Be easy to distinguish; Be placed in a useful arrangement.

In this section you will find some different types of features that can be used to calculate the homography matrix H. 10 15 20 25 30 522 437 20 The features should in some way be associated with points, since the point matches are used to calculate H1 Program to find features , where the user can only change a few constants, is implemented according to the present invention and stored in the area 23 for defining object features in the storage means 21, to adapt the feature finder for specific objects.

A very common feature in most signs is different combinations of lines. Most signs are surrounded by an edge, which gives rise to a line. Many signs also have frames around them, giving rise to double lines that are parallel. Regardless of the type of feature found, it is important to gather as much information as possible from each individual feature. Since lines are frequently used features, a description of how different types of lines can be found will be given in section E.

Number of features Since the images are of 2D plane and have been captured by a handheld camera 300, the scene and image plane are related through a plane projective transformation. In section C, we concluded that at least four point matches are needed to calculate H. If four points in the stage plane and the four corresponding points in the image are found, then H can be calculated. The problem is that we do not know if we have the correct corresponding points. Therefore, a verification procedure to check whether H is correct must be performed. To do this, H can be verified with even more point matches. If the camera is calibrated, a verification of H with the camera's internal parameters as explained at the end of section B. 24 can be performed. Restrictions on lines I 2D have lines two degrees of freedom and in the same way as with dots can four lines - where no three lines 10 15 20 25 30 5222 437 21 coincide - used to calculate the homography matrix. However, the calculation needs to be slightly modified, since lines are transformed as P = I ¥ 4l, as opposed to points that are transformed as x% = Hk, for the same homography matrix H (see Hartley, R., and Zissermann, A., "Multiple View Geometry in computer vision ", Cambridge University Press, 2000, p. 15).

It is even possible to mix feature points and lines when calculating the homography matrix.

However, there are some additional limitations when doing this, as points and lines are interdependent. As proved in section C, four points, and in the same way four lines, have eight degrees of freedom. Three lines and a point are geometrically equivalent to four points, since three non-coincident lines define a triangle, and the corner points of the triangle uniquely define three points. In the same way, three non-collinear points and one line are equivalent to four lines, which have eight degrees of freedom. However, two points and two lines cannot be used to calculate the homography matrix. The reason is that a total of five lines and five points can be determined uniquely from two points and two lines.

The problem, however, is that four out of five lines coincide and that four of the five points are collinear. These two systems thus degenerate and cannot be used to calculate the homography matrix.

Select corner points In the preferred embodiment, the equation of the lines is not used in connection with the calculation of the homography matrix. Instead, the intersection points of the lines are calculated, and thus only these points are used in the calculations.

One of the reasons for doing this is due to the proportions between the coordinates (a, b and c) of the lines. lO 15 20 25 30 5212 437 22 In an image with VGA resolution, the values of the coordinates of a normalized line will be 0ShL b | s1, if os | c {s «/ 64o2 + 4802 = soo _ This means that c the coordinates are not in proportion to the a and b coordinates. The effect of this is that a small variation of the line gradient (ie the a- and b-coordinates) can result in a large variation of component c. This makes it difficult to verify line matches.

The problem with these proportional coordinates does not disappear when the intersection points of the lines are used instead of the parameters of the lines, it is just moved. This is just a way to normalize parameters, so that they can be easily compared with each other during the verification procedure.

E. Line Detection With reference to Figs. 4 and 5, details will now be given on how to determine candidates for feature points (ie step 33 in Fig. 3). Steps 41 and 42 in Fig. 4 will be described in this section, while steps 43 will be described in the next section.

Edges are defined as points where the gradients of the image are large in terms of grayscale, color, intensity or luminescence. Once all edge points in an image have been obtained, they can be analyzed to determine how many of them are in a straight line. These points can then be used as the basis for a line.

Extraction of border points There are several ways to extract points from the image. Most are based on threshing, area growth, region splitting and merging (see Gonzalez, R.).

C., and Woods, RE, "Digital Image Processing", Addison Wesley, Reading, MA, 1993, page 414). In practice, it is common to run a mask through the image. The definition of an edge is the point of intersection between two different homogeneous regions Thus, the masks are usually based on the calculation of a local derivative, and digital images usually absorb an indefinite amount of noise as a result of sampling.

Therefore, a smoothing mask is also preferable to the derivative mask, in order to reduce the noise. An equalization mask, which gives very nice results, is the Gaussian nucleus Ga: G @ (x) = ---: - where G is the standard deviation (or the width of the nucleus) and X is the distance from the point under investigation.

Instead of first running a smoothing mask over the image and then deriving it, it is advantageous to just take the convolution of the image with the derivative of the Gaussian core: íGa '(x) = __ x_ 1 Öx 02 \ / 2n'o'2 e-xz / Zaz Fig. 5 shows åGJx) for a = 1.2.

Since images are 2D, the filter is used in both the x and y directions. To distinguish the edge points n, the filtered points f (n) are selected, i.e. the result of the convolution of the image with the derivative of the Gaussian core, where g; ,. lO 15 20 25 522 437 24 f ("- 1) f (fl)> f (fl + 1), threshold where threshold is a selected threshold.

In Fig. 7, all the edge points detected from an original image 102 (Fig. 6) are marked with a "+" sign, as indicated by the reference numeral 103.

A Gaussian core with 0fl = L2 and threshold = 5 has been used here.

Extraction of line information Once all points have been obtained, it is possible to find the equation of the line of which they may form a part. The gradient of a point in the image is a vector that points in the direction in which the image intensity at the current point decreases the most. This vector is in the same direction as the normal to the possible line. Therefore, the gradient of all edge points must be found. To extract the X coefficient of the edge point, the derivative of the Gaussian core is applied in 2D, Û X 2 z 2: G x,: __ _e- (x + y) / 2o 'I öx ° (JO 2fl a4 in the image around the edge points. In this mask, (my) is the distance from the edge point. _ _ N _ -3a Usually an area of -3a is used where G is the standard deviation.

In the same way, the y coefficient can be extracted. As mentioned above, the normal of the line has the same direction as the gradient. Thus, the a and b coefficients of the line have been obtained. The last coordinate c can be easily calculated, since ax + by + c == 0. Preferably, the equation of the line will be normalized, so that the normal of the line will have the length 1: __ (a, b, c) T JM + b2) 'This means that the c-coordinate will have same l value as the distance from the line to the origin.

Clustering of edge points to lines To find out if edge points form part of a line, restrictions on the points must be applied. There are two main constraints: - The points must have the same gradient.

- The proposed line shall run through the points.

Since the image will be blurry, these restrictions only need to be met within the limit of a certain threshold. The threshold will of course depend on the circumstances under which the image was taken, the resolution of the image and the object in the image. Since all the data for the points are known, all that needs to be done is to group the points together and align lines to them (step 42 in Fig. 4). The following algorithm is used according to the preferred embodiment: For a certain number of loops, Step 1: Randomly select a point p = (x¿% DT, with line data l = (mb¿ÛT; Step 2: Find, using pnï-l others points pn = (rvyMDT, with line data ln = (aMbMcnY fl which lie on the same line; Step 3: See, using (aMbn) (mbf '> (L-% nm2), if these points have the same gradient as p; 3 - 22-1; * IÉ 1 Pa- * Jçi CÉÉ 2 š5 .. ~: = ÉÉ _ “-_ '¿fï I * l iii) f: i *' - '=» * "<=: zi;' = ïf. '' F Û ITT ~ 13 - il: - ï-íš-í _ ”fi” lO 15 20 25 30 522 437 26 Step 4: Adjust, using SVD, a new line l = (mb¿ fl T based on all the points that meet the conditions in step 2 and step 3, pn Repeat steps 2-3; Step 5: Repeat steps 2-4 twice; Step 6: If there are at least a certain number of points that meet these conditions, define these points to form a line; End.Repeat with the remaining points.

This algorithm randomly selects a point. The equation for the line of which this point may form is already known. Now the algorithm finds all other points that have the same gradient and are on the same line as the first point. Both of these checks must be performed within a certain threshold value. In step 2, the algorithm checks if the point is closer to the line than the distance thresl. In step 3, the algorithm checks whether the gradients of the two points are the same. If they are, the product of the gradients must be 1. Again, due to inaccuracy, it is sufficient if the product is greater than Üf4hnß2).

Because the edge points are not exactly positioned, and because the gradients will not have the exact value, a new line is calculated in step 4. This line is calculated, using SVD, in the following way, based on all the points that meet the conditions in step 2 and step 3. The points must also meet the condition (x¿¿D @ Lb¿ÛT = 4l Thus, an nx3 matrix consisting of these points is assembled, and like section C is optimized using SVD n fi nAl, IMS to obtain better accuracy, step 2 and step 3 are repeated. To further increase the accuracy, 10 15 20 25 30 522 437 occurs As will be appreciated by those skilled in the art, the values of the threshold numbers will need to be determined depending on an actual application.

Fig. 8 shows the lines 104 found, as well as the edge points 103 used in the example above.

If the edge points used are omitted, it is easier to see how good the approximation of the estimated lines is, see Fig. 9.

F. Information extracted from the lines To calculate the homography matrix H, four corresponding points are needed from the two coordinate frames. Since many lines are available, additional information can be provided.

Crossing points Corners are common features on signs. However, there are usually many corners on a sign that are not interesting, for example if there is text on the sign, the letters will give rise to a lot of corners that are uninteresting. Now that the lines consisting of edges have been obtained, the corner points of the edges can be easily calculated (step 43 in Fig. 4) by taking the cross product for two lines: xc = li> The vector XC will be the homogeneous representative of the point at which lines L and lj intersect. If the third coordinate of xc = 0, then xc is the point of infinity, and the lines L and lj are parallel.

These intersections, combined with the information from the lines, will provide even more information. A verification of whether the lines really have edge points at the intersection points, or whether the intersection point is in the extension of the lines, can be performed. 10 15 20 25 30 522 437 28 This information can then be compared with the feature points sought, since information regarding whether they should have edge points at the intersections is known. In this way, crossroads that are uninteresting can be eliminated. Items that are uninteresting can have different origins.

One possibility is that they constitute crossroads that should be there, but which in this specific case are not used. Another possibility is that they are generated by lines, which should not exist, but which have nevertheless arisen due to disturbing elements in the image.

In Fig. 10, all intersections are marked with a "+" sign, as seen at 105. The actual corners of the frame are marked with a "*" sign, as seen at 106.

Parallel lines Another common feature of signs is frames, which give rise to parallel lines. If only lines originating in frames are interesting, all lines that do not have a parallel equivalent, ie a line with a nearby normal in the opposite direction, can be discarded.

Because the line is transformed, parallel lines in 3D values may appear to be non-parallel on the 2D image scene.

However, lines that are close to each other will still be parallel, within a certain margin of error.

The result of an algorithm which finds parallel lines 107, 107 'is shown in Fig. 11.

Once all the sets of parallel lines have been found, it is possible to figure out which lines are candidates for being a line corresponding to the inner edge of a frame. If the cross products for all these lines are calculated, a set of points is obtained which are possible candidates for inner corner points in a frame, as marked with "*" characters at 108 in Fig. 12.

ZílíIÃi »<3? -víïfš fZê .Z r ïšïšïš? 10 15 20 25 30 522 437 29 Continuous edge points By chance, it is possible that the line detection algorithm generates a line that actually consists of a lot of small edges that lie on a straight line. For example, edges of letters written on a straight line can give rise to such a line. If only lines consisting of continuous edge points are interesting, it is desirable to eliminate these other lines. One way to do this is to take the midpoint of all the edge points on the line. From this point, a few more points along the line are extrapolated. Now the difference in intensity is checked on both sides of the line at the selected points. If the intensity differences at the points do not exceed a certain threshold value, the line is not made up of continuous edge points.

With this algorithm, not only lines derived from non-continuous edge points will be eliminated, but the algorithm will also eliminate thin lines on the image. This is a positive effect if only borders derived from thick frames are to be used as features. In Fig. 13, the same algorithms used previously have been applied to Figure 102 shown in Fig. 6. The only difference in the algorithms is that no check has been made as to whether the lines consist of continuous edge points along edges.

Fig. 14 shows an enlargement of the result from the algorithm, with checks of continuous edge points placed on line 109 at the lower part of the numbers "12345678". The algorithm gave a negative result, in terms of whether there was a continuous edge pattern or not. Fig. 15 is an enlargement of the same algorithm, applied to line 110 at the lower part of the frame. Här iii-lv íïïå íëïà iii-z 23: “= 2> znt" ÅI “¿> šJäêšáš» låílíšlääíšlíïëfi-â .Sšiï -Iïš «.;“ ~ .- = ': -: =: ¿: J> k < : 212352 (Lä '_35 íl <šE' * «ï.f.í; ': <: lO 15 20 25 30 522 437 30 gave the algorithm a positive result, in that the edge points are continuous.

G. Calculation of the homography matrix H Once the feature candidates in the image have been obtained, they must be matched against features from the original plate, which have known coordinates. If four feature candidates have been found, their coordinates can be matched with the corresponding coordinates of feature points of the object, which are stored in the area 23 of the storage means 21, and the homography matrix H can be calculated. Since more candidates for the interesting features than the intended ones are likely to be found, a verification procedure must be performed.

This procedure must verify that the selected features for feature points have been performed with correct matching. Thus, if there are many candidates for possible feature points, the homography matrix should be calculated several times and verified each time, to check whether it is the correct point correspondence or not.

Advantageously, this matching procedure is optimized using Fischler and Bolles' RANSAC algorithm (see Fischler, M. A., and Bolles, R. C., "Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography", Comm.

Assoc. Comp. Mach., 24 (6): 38l-395, 1981).

RANSAC RANSAC (The RANdom Sample and Consensus algorithm) is an estimation algorithm, which has the ability to work with very large sets of possible matches.

The best way to determine the homography matrix H is to calculate H for all possible combinations, verify each solution and then use the conformity that has the best verification result. Verification procedures, as described below, can be performed in various ways.

Since calculating H for each possible combination is very time consuming, this is not a very good approach when the algorithm is to be performed in real time. The RANSAC algorithm is also an algorithm for hypothesis and verification, but it works in a different way. Instead of systematically working its way through the possible features, it randomly selects its points of conformity and then calculates the homography matrix and performs the verifications.

RANSAC shall repeat this procedure a certain number of times and then decide to use the set of conformities which has the best verification result.

The advantages of the RANSAC procedure are that it is more robust when there are many possible features, and that it tests conformities in random order. If the point matches are tested systematically, and the algorithm happens to start with a point that is incorrect, all matches that this point can give rise to must be verified by the algorithm. This does not happen with RANSAC, as a point will only be matched with a single possible point match, and new feature points will then be selected for matching with each other. The RANSAC matching procedure is performed only a specified number of times, and the best solution is selected.

Because the points are randomly selected, sometimes the correct match, or at least one that is close to the correct one, has been selected, and these point matches can be used to calculate H.

Verification procedures Once the homography matrix has been calculated, it must be verified that correct point conformities have been used.

This can be done in a few different ways. 10 15 20 25 30 522 457 32 A fifth feature The most common way to verify H 'is to use additional feature points. In this case, even more than the four features of the original objects must be known. The remaining points from the original object can be transformed into the coordinate system of the image. Then a verification procedure can be performed, to check whether the points have been found in the image. The more additional features found, the greater the likelihood that the correct set of point matches has been selected.

Camera internal parameters If the camera is calibrated, it is possible to verify the putative homography matrix with the camera's internal parameters 24, which are stored in the storage means 21 (see the discussion in the previous sections). This introduces further restrictions on the selected feature points. If the points represent the corner of a rectangle, the first and second rows, q and rz will give rise to the same value, if the points are correctly matched up to a rotation error of the rectangle of 180 degrees. This is obvious because if a rectangle is rotated 180 degrees, it will give rise to the exact same rectangle. In the same way, a square can be rotated 90, 180 or 270 degrees, and still give rise to the exact same square. In all these cases, Q and rz will still be orthogonal. Although this verification procedure may have a rotational error, if corners of a rectangle are used as feature points, it is still very useful, since rectangles are common features. The rotation error can easily be checked later. lf rv-. * VV 10 15 20 25 30 Verification errors Depending on how the feature points are selected, errors can still occur when the feature points are verified. As mentioned above, the homography matrix is a homogeneous matrix, and only determined up to a scale. If the object has points that are at exactly the same layout as the feature and verification points, except that they are rotated and / or up to scale, the verification procedure will give rise to exactly the same values as if the correct point conformity had been found . Therefore, it is important to choose features that are as distinctive as possible.

Restrictions on RANSAC RANSAC is based on randomness only. If even more information is available, this should obviously be used to optimize the RANSAC algorithm. Some restrictions that can be added are as follows.

Stop if the solution is found Instead of repeating the calculations in the procedure a specific number of times, it is possible to stop if the verification indicates that a solution that is good has been found. In order to determine whether a solution is good or not, a condition can be made, as if at least a certain number of feature points have been found in the verification procedure, this must be the correct homography matrix. If the camera's internal parameters are used as a verification procedure, a stop can be made if rl and r are very close to have the same length and are orthogonal.

Feature points on the same line The restriction that only such a set of feature points should be used, where no three points are allowed to lie on the same line, can be included in the RANSAC algorithm.

After four points are randomly selected, it is possible to check whether three of them are on the same line, before proceeding with the calculation of the homography matrix. Combined with the following two restrictions, this control is very time efficient.

Convex envelope The convex envelope of any set of S points is the smallest convex polygon IQ for which each point in S lies either on the boundary of P; or in its interior. Two of the most common algorithms used to calculate the convex envelope are Graham's scan and Jarvis' march. Both of these algorithms use a technique called "rotary sweeping" (see Cormen, T. H., Leiserson, C. E., and Rivest, R. L., "Introduction to Algorithms", The Massachusetts Institute of Technology, 1990, p. 898). When calculating the convex casing, these algorithms will also bring about the order of the corners, as they appear on the casing, in the counterclockwise order. Graham's scan works in Ob fl gn) time, unlike Jarvis's march which works in 00%) time, where n is the number of points and h is the number of corners.

Because projecting images are line-preserving, they must also preserve the convex envelope. In a set of four points, where no three points lie on the same line, the convex housing will consist of either three or four of the points. This means that in two sets of corresponding points, both of their convex housings will consist of either three or four points. A check of this can, after the two sets of four points have been selected, be included in the RANSAC algorithm.

Qi '/ ~. 10 15 20 25 30 522 457 35 Systematic search The principle behind RANSAC is to randomly select four points, match them with four supposedly matching points, which are also chosen at random, and then reject these points and select new ones. It is possible to modify this algorithm and include some systematic operations. Once the two sets of four points have been selected, all possible combinations of matching between these points can be tested. This means that there are 4h = 24 different combinations to test. If the above restrictions have been included, this number can be greatly reduced. First, one can ensure that no three points of the four points in each set are on the same line. Then you can check if both sets have the same number of points in the convex housing. If this is the case, the order of the points on the casing will also be obtained, and now the points can only be matched with each other in either three or four different ways, depending on how many points the casing consists of.

Thus, out of 24 possible combinations, 0, 3 or 4 putative point matches have been reached. Of course, calculating the convex envelope and ensuring that no three points are on the same line is time consuming, but this is insignificant compared to calculating the homography matrix 24 times.

H. Target Area Extraction Once the homography matrix is known, any area from the image can be extracted so that it will appear as if the image was taken from a location directly in front of it. To do this extraction, all the points within the area of interest will be transformed into the image plane according to the selected solution. fa: - - ~ f: lå: Z-š <3: Éšlïgåíilffiïfhš: sï-Q-Lííäké: Zíï “(29 - -å. = '.“ i <§ <: 10 15 20 25 30 522 437 36 Because the picture is a discrete coordinate frame, it consists of pixels with integer values, but the transformed points will probably not be integers.

Therefore, a bilinear interpolation (see, for example, Heckbert, P. S., "Graphics Gems IV", Academic Press, Inc., 1994) must be made to obtain the intensity from the image. The transformed image can be recreated from either the grayscale intensity, or all three intensity levels can be obtained from the original color image.

Fig. 16 shows the target area 101 of the image 102 in Fig. 6, found by the algorithms above.

In Fig. 17, the target area 101 'has been transformed so that, for example, OCR or bar code interpretation can follow (steps 36 and 37 in Fig. 3). In this example, a resolution of 128 pixels in the x-direction was selected.

I. Alternative Embodiments The invention has been described above with reference to a preferred embodiment. However, embodiments other than those described above are equally possible within the scope of the invention, as defined in the appended claims. Specifically, it is observed that the invention may be practiced in portable devices other than that described above, for example, mobile telephones, portable digital assistants (PDAs), PDAs, calendars, communication devices, etc.

Furthermore, it is possible, within the scope of the invention, to perform some of the steps of the inventive method in the external computer 200, rather than in the handheld device 300. For example, it is possible to transmit the transformed target area 101 as a digital image (JPEG, GIF , TIFF, BMP, EPS, etc.) Over the link 301 to the computer 200, which will then perform the actual processing of the transformed target area 101, to extract the desired information (OCR text, bar code, etc).

Of course, the computer 200 may, in a conventional manner, be connected to a local network, or a global network, such as the Internet, which enables the extracted information to be passed on to further applications outside the handheld device 300 and the computer 200. Alternatively, the extracted information is communicated via a mobile telephone, which is functionally connected to the handheld device 300 via IrDA, Bluetooth or cable (not shown in the drawings). "" ('"~ Í? $ Ï *

Claims (26)

10 15 20 25 30 522 457 38 PATENT REQUIREMENTS A method of extracting information from a target area (101) within a two-dimensional graphic object (100) having a plurality of predetermined features (23) having, in a first plane, known characteristics, characterized by the steps of: reading an image (102) in which said object (100) is located in a second plane, said second plane being known in advance, in said image identifying a plurality of candidates (108) for said predetermined features (23) in said second plane, that from said identified plurality of feature candidates, calculating a transformation matrix (H) for projecting imaging between said second and first planes, transforming said target area (101) from said second plane to said first plane, and processing said target area to extract said information. The method of claim 1, wherein said plurality of predetermined features (23) are read from a memory (21) before said plurality of feature candidates (108) are identified. The method of claim 1 or 2, wherein said plurality of predetermined features (23) include at least four features. The method of claim 3, wherein said at least four predetermined features are four points, four lines, three points and one line, or one point and three lines. 10 15 20 25 30 522 437 39 The method of claim 3, wherein said at least four predetermined features are four points, said plurality of feature candidates (108) being identified by: locating edge points (103) as points in said image (102) having large gradients, that clustering said edge points to lines (104), and A to determine said plurality of feature candidates as intersection points (105, 106, 108) between any two of said lines. The method of claim 5, wherein said intersection points (105, 106, 108) are at four corner points on a frame in said two-dimensional graphic object. A method according to any one of the preceding claims, wherein said transformation matrix (H) is calculated by: selecting from said identified plurality of feature candidates randomly as many feature candidates as in said plurality of feature candidates (23), calculating a hypothetical transformation matrix for said randomly selected candidates. and said plurality of predetermined features, verifying the hypothetical transformation matrix, repeating the steps above several times, and as said information matrix (H) selecting the specific hypothetical transformation matrix with best results from the verification step. A method according to claim 6 or 7, wherein the hypothetical transformation matrix is verified by means of at least one further predetermined feature. A method according to any one of claims 6-8, wherein said plurality of predetermined features (23) comprises íï (I: fi. "'_L'flï' lll 1,2 S355 ÉÉÉÉ iè '-." -' ~. ':: = ï í {rl¿>. Zíï '321 (f- “š» fi fšíšêß. iï 10 15 20 25 30 522 437 40 at least four points and wherein said step of random selection is limited to a set of four feature candidates, which does not include three aligned points. The method of claim 9, wherein said step of randomly selecting is further limited by calculating the complex envelope of said feature candidates. A method according to any one of the preceding claims, wherein said plurality of predetermined features (23) comprise at least one point having a value for grayscale, color, intensity or luminescence, which is clearly distinct from surrounding points in said two-dimensional graphic objects (100). A method according to any one of the preceding claims, wherein said two-dimensional graphic object (100) is a sign. A method according to any preceding claim, wherein said processing step comprises optical character recognition (OCR) of said target area (101). A method according to any preceding claim, wherein said processing step comprises bar code interpreting said target area (101). A method according to any preceding claim, wherein said processing step comprises transferring said target area (101) to an external computer (200). A method according to any one of the preceding claims, wherein said first plane is the image plane of said read image (102). A method according to any one of claims 1-15, wherein said first plane is the image plane of a previously read image. A method according to claims 1-17, wherein said plurality of predetermined features (23) are obtained by direct measurement at said previously read image. 10 15 20 25 30 522 4s7ö 41 A computer program product which is directly chargeable to an internal memory (21) in a processing device (20), the computer program product comprising program code (22) for performing the steps according to any of claims 1-18 when executed by said processing device. A computer program product according to claim 19, comprising on a computer readable medium (21). The handheld image generating device (300) having storage means (21) and a processing device (20), the storage means containing program code (22) for performing the steps according to any one of claims 1-18 when executed by said processing device. An apparatus for extracting information from a target area (101) within a two-dimensional graphic object (100), having a plurality of predetermined features (23) having, in a first plane, known characteristics, the apparatus comprising an image sensor (8). ), a processing device (20) and storage means (21), characterized by a first area (25) in said storage means (21), said first area being adapted for storing an image (102) recorded by said image sensor (8), in which said object (100) is located in a second plane, said second plane being known in advance, and a second area (23) in said storage means (21), said second area being adapted for storage of said plurality of predetermined features, wherein: said processing device (20) is arranged to read said image (102) from said first area (25), to read said plurality of predetermined features from said second area (23), to identify in said image a plural candidates to the said feature in said second plan, that from said identified feature candidates .--. f '«- v, v, -., -, ~» vy y, _... _ f ~ »--'-_ -; -¿ '-. f ~ «-,» ~. . <'~ .- ^ -: f. - «-; = q-x, ..., _,., ~, ._> _, _», 1 ~~. f- a; .f-.fï,, _ ._ Law-_: ~ ut! .get Izæf. 1.; - gvïu: ger: -Åchß 12min; ïcuitffi fl i SJ; of; : šz - vir-nflfz 2 = .m-¿'f-J v ..- aqf fl iwàízi. Calculating a transformation matrix (H) for projecting imaging between said second and first planes, transforming said target area (101) in said object from said second plane to said first plane, and after the transformation extracting said information from said màlomràde. The apparatus of claim 22, further comprising an optical character recognition (OCR) module (29) adapted to extract said information from said target area (101). The apparatus of claim 22, further comprising a module (29 ') for bar code interpretation, which is adapted to extract said information from said target area (101). A device according to any one of claims 22-24 in the form of a handheld device (300). The device of any of claims 22-24, wherein said device comprises a handheld device (300) and a computer (200). ; -; 1 = ^ <;;;: = .- k =; : Elf íï