WO2014092546A1

WO2014092546A1 - Image processing system and method for extracting a spatial relationship between objects in an image

Info

Publication number: WO2014092546A1
Application number: PCT/MY2013/000251
Authority: WO
Inventors: Yeek TAN SIEOW; Seng TAN YEW; Lukose Dickson
Original assignee: Mimos Bhd
Current assignee: Mimos Bhd
Priority date: 2012-12-13
Filing date: 2013-12-11
Publication date: 2014-06-19
Anticipated expiration: 2015-06-13
Also published as: MY181673A

Abstract

The present invention relates to an image processing system (100). The image processing system (100) is able to compute and analyse spatial relationship between objects detected in an image. The image processing system (100) comprises of an image segmentation and labelling component (110), a blob detection component (120), a spatial relationship extractor component (130), and a domain knowledge base (140). The image processing system (100) extracts spatial relationship between objects in an image by performing a surface subdivision computation, two-dimensional spatial relation computation, three-dimensional spatial relation computation and spatial relation extender.

Description

IMAGE PROCESSING SYSTEM AND METHOD FOR EXTRACTING A SPATIAL RELATIONSHIP BETWEEN OBJECTS IN AN IMAGE

FIELD OF INVENTION

The present invention relates to an image processing system. More particularly, the present invention relates to an image processing system and method for extracting spatial relationship between object in an image.

BACKGROUND OF THE INVENTION

Spatial relation specifies how an object is located in relation to a reference object. Spatial relation plays an important role in image processing system as it provides a better understanding of an image for further processing.

Muda et al. (2009) relates to the development of an enhanced automatic annotation system which extracts spatial information about objects in an image. The approach uses region boundaries and region labels to generate annotations describing absolute object positions and also relative position between pairs of objects. A domain ontology and spatial information ontology are also used to extract more complex information about the relative closeness of objects to the viewer.

However, such system still lacks of a precise way to describe the spatial relations between objects that exist in an image. The depth information in 2D images is still limited for identifying far and near information between objects. In particular, the system only uses ontology to store height magnitude of an object and thereon, using the height magnitude and height of bounding rectangles to calculate "far" and "near" spatial relationship. Thus, most of the existing systems use only a single point to determine spatial relations between objects. The existing spatial relationship enrichment methods are still yet to be fully explored, especially with the incorporation of additional knowledge from ontology.

Therefore, there is a need to provide an image processing system that is able to extract spatial relationship between objects and also addresses the aforementioned problems. SUMMARY OF INVENTION

The present invention provides an image processing system (100). The image processing system (100) comprises an image segmentation and labelling component (110), and a blob detection component (120). The image processing system (100) is characterised in that it further includes a spatial relationship extractor component (130) connected to a domain knowledge base (140), and wherein said spatial relationship extractor component (130) is configured to perform a surface subdivision computation, two-dimensional spatial relation computation, three- dimensional spatial relation computation and spatial relation extender.

The present invention also provides a method for extracting spatial relationship between objects in an image. The method is characterised by the steps of identifying and classifying objects in the image; computing a bounding box and mass for each object in the image; performing a surface subdivision computation for each object in the image; performing a two-dimensional relation computation for each object pair in the image; performing a three-dimensional relation computation for each object pair in the image; and extending the spatial relation for each object pair in the image. Preferably, the step of performing a surface subdivision computation for each object in the image includes dividing a bounding box for an object into N x M cells, wherein N and M are predetermined parameters; computing a fill-density for each cell; determining whether the fill-density for each cell exceeds a predetermined threshold; if the cell exceeds the predetermined threshold, status of the cell is defined as active; and if the cell is below the predetermined threshold, status of the cell is defined as inactive.

Preferably, the step of performing a two-dimensional relation computation for each object pair in the image includes selecting two objects in the image as an object pair; connecting each active cell in a first object to a corresponding active cell in a second object via a vector; computing and storing angle direction of each vector; determining whether 2D spatial relationship for the selected object pair can be defined as "within" based on the stored angel directions of the active cells; wherein if the 2D spatial relationship of object pair cannot be defined as "within", computing a mean of all angle directions of the vectors and mapping the computed mean using mapping rules to determine 2D spatial relationship of the object pair; and extending the assigned 2D spatial relationship by applying a transitive rule, an inverse rule and a symmetric rule. Preferably, the step of performing a three-dimensional relation computation for each object pair in the image includes the steps of selecting two objects in the image as an object pair; identifying the selected objects of the object pair; retrieving mass order of magnitude information for each object from an ontology in a domain knowledge base (140); computing mass of each object of the object pair; retrieving height order of magnitude for each object from an ontology in a domain knowledge base (140); computing height of each object of the object pair; computing a ratio of the computed height of the first object over the computed height of the second object; normalizing a ratio of the actual height of the first object over the actual height of the second object; determining whether the computed ratio of computed heights is greater than the normalized ratio of actual heights, wherein determining that the first object is farther than the second object in the image if the computed ratio of computed heights is lesser than the normalized ratio of actual heights; computing a ratio of the computed mass of the first object over the computed mass of the second object; normalizing a ratio of the actual mass of the first object over the actual mass of the second object; and comparing whether the computed ratio of computed masses is greater than the normalized ratio of actual masses, wherein determining that the first object is nearer than the second object in the image if the computed ratio of computed masses is greater than the normalized ratio of actual masses, and determining that the first object is farther than the second object in the image if the computed ratio is lesser than the normalized ratio.

Preferably, the step of extending the spatial relation for each object pair in the image includes the steps of selecting an object pair; retrieving the spatial relationships for the object pair; and permuting all spatial relationships for the object pair based on the retrieved the spatial relationships.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. FIG. 1 shows a block diagram of an image processing system (100) according to an embodiment of the present invention. FIGS. 2(a-e) show flowcharts of a method for extracting spatial relationship between objects in an image according to an embodiment of the present invention.

FIGS. 3(a-c) show an image being processed based on the method as shown in FIG. 2a

FIGS. 4(a-d) show an object pair in an image being processed based on the method as shown in FIGS. 2b and 2c.

FIG. 5 shows an example of an ontology stored in a domain knowledge base (140) of the image processing system of FIG. 1.

DESCRIPTION OF THE PREFFERED EMBODIMENT

A preferred embodiment of the present invention will be described herein below with reference to the accompanying drawings. In the following description, well known functions or constructions are not described in detail since they would obscure the description with unnecessary detail.

Referring to FIG. 1 , there is shown a block diagram of an image processing system (100) according to an embodiment of the present invention. The image processing system (100) is able to compute and analyse spatial relationship between objects detected in an image. The image processing system (100) comprises of image segmentation and labelling component (110), a blob detection component (120) and a spatial relationship extractor component (130). Each component can either be a hardware component, a software component or a combination thereof.

The image segmentation and labelling component (110) is configured to perform a series of image pre-processing steps such as image segmentation, region classification and region labelling. By performing those pre-processing steps, the image segmentation and labelling component (110) is able to obtain a plurality of different segmented regions in an image labelled with an object class. The image segmentation and labelling component (110) is connected to the blob detection component (120).

The blob detection component (120) is configured to perform a bounding box computation and mass computation on the labelled regions. The blob detection component (120) is connected to the image segmentation and labelling component (110) and the spatial relationship extractor component (130).

The spatial relationship extractor component (130) is connected to a domain knowledge base (140) and it is configured to perform a surface subdivision computation, two-dimensional or 2D spatial relation computation, three-dimensional or 3D spatial relation computation and spatial relation extender.

Referring to FIG. 2a, there is shown a flowchart of a method for extracting spatial relationship between objects in an image. Initially, as in step 201, the image segmentation and labelling component (110) receives an image or an image uniform resource locator (URL). If the image segmentation and labelling component (110) receives an image URL, the image segmentation and labelling component (110) downloads the image based on the URL provided. FIG. 3a shows an example of an image inputted to the image segmentation and labelling component (110). Thereon, the image segmentation and labelling component (110) performs a sanity validation on the image to ensure that the image received is not corrupted and is of the correct format and type. Next, the image segmentation and labelling component (110) identifies and classifies objects in the image by using segmentation and object labelling techniques. The segmentation technique used by the image segmentation and labelling component (110) may either be a threshold-based technique, an edge- based technique, a region-based technique, or any other segmentation technique. The object labelling technique used by the image segmentation and labelling component (110) may either be a supervised and unsupervised multi-class model, a statistical model or any other object labelling technique. Each object is labelled by overlaying a pre-designated colour before the result is output as a segmented and labelled image. FIG. 3b shows an example of a segmented and labelled image based on the image of FIG. 3a. The segmented and labelled image is then sent to the blob detection component (120). In step 202, the blob detection component (120) detects all the colour-labelled objects and thereon, computes its bounding box and mass. The bounding box of an object is the smallest possible rectangle containing said object and is represented by coordinates of the top-leftmost coordinate and the bottom-rightmost coordinate of the bounding box. The mass is the size of the area used by the object and it is measured by number of pixels. The blob detection component (120) uses sequential region labelling techniques to compute the bounding box and mass for each object. FIG. 3c shows an example of an image computed with the bounding box and mass for each object based on the image of FIG. 3b. The output of the image with the computed bounding boxes and masses from the blob detection component (120) is sent to the spatial relationship extractor component (130).

In steps 203 to 206, the spatial relationship extractor component (130) receives the output from the blob detection component (120) and performs a surface subdivision computation process, a two-dimensional or 2D relation computation process, a three-dimensional or 3D relation computation process, and a spatial relation extender process.

Referring now to FIG. 2b, there is shown the surface subdivision computation process performed by the spatial relationship extractor component (130). Initially, as in step 301 the bounding box for an object is divided into N x M cells, wherein N and M are predetermined parameters. FIG. 4a shows an example of two objects in an image divided into x M cells, wherein the predetermined parameter for N and M is 4.

Thereon, a fill-density for a cell in the grids is computed as in step 302. The fill-density is an estimated percentage of used space or non-white space over the total space of the cell. The status of fill-density for the cell is determined. If the fill-density is determined to exceed a predetermined threshold, the status of the cell is deemed as active (decision 303 and step 304). Otherwise, the status of a cell is deemed as inactive as in decision 303 and step 305. FIG. 4b shows the status of cell density for each cell of the two objects as shown in FIG. 4a, wherein the inactive cells are marked with 'x'. Thereon, the fill-density is computed for another cell of the bounding box until the status of fill-density for each cell has been determined (decision 306 and step 302).

Once the status of fill-density for each cell in the bounding box has been determined, the process is repeated for each object detected in the image.

Referring now to FIG. 2c, there is shown the 2D relation computation process performed by the spatial relationship extractor component (130). The 2D relation computation process is to compute directional angle connecting two objects in an image. Initially, two objects in the image are selected as an object pair as in step 401. Thereon, a first active cell of a first object is selected and a corresponding active cell in a second object is determined as in decisions 402 and 403.

If there is no corresponding active cell in the second object, the spatial relationship extractor component (130) returns to decision 402 to select another active cell in the first object. If there is a corresponding active cell in the second object, the spatial relationship extractor component (130) connects both active cells via a vector. FIG. 4c shows an example of the active cells of two objects connected via vectors. The angle direction of the vector is computed and stored (steps 404 and 405). Thereon, the method returns decision 402 to select another active cell in the first object.

If there all of the active cells in the first object have been selected, the spatial relationship extractor component (130) determines whether the 2D spatial relationship for the selected object pair is defined as "within" as in decisions 402 and 406. The spatial relationship extractor component (130) determines whether the 2D spatial relationship can be defined as "within" based on the stored angel directions of the active cells. Preferably, the 2D spatial relationship is defined as "within" if the stored angle directions have an approximately even distribution of angles throughout four quadrants, wherein a first quadrant is within a range of 0 degrees to 90 degrees, a second quadrant is within a range of 90 degrees to 180 degrees, a third quadrant is within a range of 180 degrees to 270 degrees, and a fourth quadrant is within a range of 270 degrees to 360 degrees.

If the 2D spatial relationship of object pair is determined not to be defined as "within", the spatial relationship extractor component (130) computes a mean of the stored angle directions of the active cells as in decision 406 and step 407. The mean angle is then used to map the 2D spatial relationship between the two objects based on a mapping rules as in decision 408. Table 1 shows an example of the mapping rules and FIG. 4d shows an example of a mapping result between two objects based on the mean of the angle directions for each active cell. Based on the mapping rules, the spatial relationship extractor component (130) assigns the 2D spatial relationship to the selected object pair as in step 409. Thereon, the spatial relationship extractor component (130) extends the assigned 2D spatial relationship by applying a transitive rule, an inverse rule and a symmetric rule as in step 410. The transitive rule is used to derive an extended spatial relationship between a first object and a third object based on a 2D spatial relationship between a first object and a second object and a 2D spatial relationship between a second object and a third object. The extended spatial relationship using the transitive rule includes IS-LEFT-OF, IS- RIGHT-OF, IS-ON-TOP-OF, IS-BOTTOM-OF, IS-TOP-LEFT-OF, IS-TOP-RIGHT- OF, IS-BOTTOM-LEFT-OF, IS-BOTTOM-RIGHT-OF, IS-NEARER-TO, and IS- FARTHER-TO. For example, if a 2D spatial relationship between object A and object B is defined as "object A IS-ON-TOP-OF object B" and a 2D spatial relationship between object B and object C is defined as "object B IS-ON-TOP-OF object C", then the extended spatial relationship between object A and object C using the transitive rule is derived as "object A IS-ON-TOP-OF object C". The inverse rule is used to derive an extended spatial relationship which is an inversed 2D spatial relationship of object pair. For example, if the 2D spatial relationship between object A and object C is as "object A IS-ON-TOP-OF object C", the extended spatial relationship using the inverse rule is derived as "object C IS-ON-BOTTOM-OF object A". The symmetric rule is used to derive an extended spatial relationship between three objects based on a 2D spatial relationship between a first object and a second object and a 2D spatial relationship between a second object and a third object. The extended spatial relationship using the symmetric rule includes IS-AT-THE-SAME-SIDE-OF. As an example, if a 2D spatial relationship between object P and object R is defined as "object P IS-ON-LEFT-OF object R" and a 2D spatial relationship between object T and object R is defined as "object T IS-ON-LEFT-OF object R", then the extended spatial relationship using the symmetric rule is derived as "object P IS-AT-THE- SAME-SIDE-OF object T". If the 2D spatial relationship of the object pair is determined to be defined as

"within", the spatial relationship extractor component (130) assigns the 2D spatial relationship of the object pair as "within" as in decision 406 and step 409. Thereon, the spatial relationship extractor component (130) extends the assigned 2D spatial relationship by applying the transitive rule, the inverse rule and the symmetric rule as in step 410.

The process is repeated for each object pair in the image as in decision 411. The computed 2D spatial relationships and the extended spatial relationships for each object pair are stored in the domain knowledge base (140).

Referring now to FIG. 2d, there is shown the 3D relation computation process performed by the spatial relationship extractor component (130). The computation of three-dimensional spatial relationship namely, "far" and "near" relations uses height order of magnitude information and mass order of magnitude information provided by the domain knowledge base (140). The magnitude information is an actual width, height, depth or mass of an object in real world. The height order and mass order are predetermined constant coefficients for normalizing an actual height and mass of an object. The height order and mass order are stored in the domain knowledge base (140). The comparison of height and mass information in the image enables the determination of the distance of the objects from an image capture point.

In step 501, an object pair which includes a first object and a second object is selected. Thereon, the spatial relationship extractor component (130) identifies the selected objects and retrieves the mass order of magnitude information for each object from the domain knowledge base (140). In the domain knowledge base (140), there is provided a pre-built ontology wherein a specific object is pre-mapped to a mass order, an actual mass of the specific object and an actual height of the specific object. FIG. 5 shows an example of the pre-built ontology in the domain knowledge base (140). In step 502, mass of each object of the object pair is computed, wherein the mass is measured by number of pixel.

In step 503, the height order of magnitude information for each object of the object pair is retrieved from the domain knowledge base (140).

In step 504, height of each object of the object pair is computed, wherein the height is measured by number of pixel. Thereon, a ratio of the computed height of the first object (hi) over the computed height of the second object (hj) is compared to a normalized ratio of the actual height of the first object over the actual height of the second object. The normalized ratio is obtained by H*H/H_j, wherein H is the retrieved height order, /-/, is an actual height of the first object, and H_j is an actual height of the second object. If h h_j is lesser than H*H/H_j, it is determined that the first object is farther than the second object in the image as in decision 505 and step 508.

Otherwise, a ratio of the computed mass of the first object (mi) over the computed mass of the second object (m,) is compared to a normalized ratio of the actual mass of the first object over the actual mass of the second object as in decisions 505 and 506. The normalized ratio is obtained by M*M/M_j, wherein M is the retrieved mass order, M, is an actual mass of the first object, and M_j is an actual mass of the second object. If rn/m_j is greater than M*M/M_j, it is determined that the first object is nearer than the second object in the image as in decision 506 and step 507. Otherwise, it is determined that the first object is farther than the second object in the image as in decision 506 and step 508.

The process is repeated for each object pair in the image. The computed 3D spatial relationship for each object pair is stored in the domain knowledge base (140).

Referring now to FIG. 2e, there is shown the spatial relation extender process performed by the spatial relationship extractor component (130). In steps 601 and 602, an object pair is selected and the spatial relationships for the object pair is retrieved from the domain knowledge base (140). In step 603, the extended spatial relationships for the object pair are retrieved from the domain knowledge base (140). In step 604, the spatial relationship extractor component (130) outputs and permutes all spatial relationships of the selected object pair which include the 2D spatial relationship, the 3D spatial relationship, and the extended spatial relationship. Thus, the spatial relationship extractor component (130) generates new or undiscovered spatial relationships for the selected object pair based on the computed spatial relationships as stored in the domain knowledge base (140).

The process is then repeated for each object pair in the image.

While embodiments of the invention have been illustrated and described, it is not intended that these embodiments illustrate and describe all possible forms of the invention. Rather, the words used in the specifications are words of description rather than limitation and various changes may be made without departing from the scope of the invention.

Angle Properties Spatial Relationship

Min < 22.5° & Max > 337.5°; or

Min < 22.5° & Max < 22.5°; or Right Of

Min > 337.5° & Max > 337.5°.

Mean = 22.5° to 67.5° Bottom Right Of

Mean = 67.5° to 112.5° Bottom Of

Mean = 112.5° to 157.5° Bottom Left Of

Mean = 157.5° to 202.5° Left Of

Mean = 202.5° to 247.5° Top Left Of

Mean = 247.5° to 292.5° Top Of

Mean = 292.5° to 337.5° Top Right Of

Angle presents all round Within Of

Table 1

REFERENCES

1. Muda, Zurina, Lewis, Paul H., Payne, Terry R. and Weal, Mark (2009) Enhanced Image Annotations Based on Spatial Information Extraction and

Ontologies. At IEEE International Conference On Signal & Image Processing 2009 (ICSIPA2009), Kuala Lumpur, Malaysia, 18 - 19 Nov 2009. lEEExplore.

Claims

1. An image processing system (100) comprising an image segmentation and labelling component (110), and a blob detection component (120); wherein said image processing system (100) is characterised in that it further includes a spatial relationship extractor component (130) connected to a domain knowledge base (140), and wherein said spatial relationship extractor component (130) is configured to perform a surface subdivision computation, two-dimensional spatial relation computation, three-dimensional spatial relation computation and spatial relation extender.

A method for extracting spatial relationship between objects in an image is characterised by the steps of:

a) identifying and classifying objects in the image;

b) computing a bounding box and mass for each object in the image; c) performing a surface subdivision computation for each object in the image;

d) performing a two-dimensional relation computation for each object pair in the image;

e) performing a three-dimensional relation computation for each object pair in the image; and

f) extending the spatial relation for each object pair in the image.

The method as claimed in claim 2, wherein step (c) is characterised by the steps of:

a) dividing a bounding box for an object into N x M cells, wherein N and M are predetermined parameters;

b) computing a fill-density for each cell;

c) determining whether the fill-density for each cell exceeds a predetermined threshold;

d) if the cell exceeds the predetermined threshold, status of the cell is defined as active; and

e) if the cell is below the predetermined threshold, status of the cell is defined as inactive.

4. The method as claimed in claim 2, wherein step (d) is characterised by the steps of:

a) selecting two objects in the image as an object pair;

b) connecting each active cell in a first object to a corresponding active cell in a second object via a vector;

c) computing and storing angle direction of each vector;

d) determining whether 2D spatial relationship for the selected object pair can be defined as "within" based on the stored angel directions of the active cells; wherein if the 2D spatial relationship of object pair cannot be defined as "within", computing a mean of all angle directions of the vectors and mapping the computed mean using mapping rules to determine 2D spatial relationship of the object pair; and e) extending the assigned 2D spatial relationship by applying a transitive rule, an inverse rule and a symmetric rule.

5. The method as claimed in claim 2, wherein step (e) is characterised by the steps of:

a) selecting two objects in the image as an object pair;

b) identifying the selected objects of the object pair;

c) retrieving mass order of magnitude information for each object from an ontology in a domain knowledge base (140);

d) computing mass of each object of the object pair;

e) retrieving height order of magnitude for each object from an ontology in a domain knowledge base (140);

f) computing height of each object of the object pair;

g) computing a ratio of the computed height of the first object over the computed height of the second object;

h) normalizing a ratio of the actual height of the first object over the actual height of the second object;

i) determining whether the computed ratio of computed heights is greater than the normalized ratio of actual heights, wherein determining that the first object is farther than the second object in the image if the computed ratio of computed heights is lesser than the normalized ratio of actual heights; j) computing a ratio of the computed mass of the first object over the computed mass of the second object;

k) normalizing a ratio of the actual mass of the first object over the actual mass of the second object; and

I) comparing whether the computed ratio of computed masses is greater than the normalized ratio of actual masses, wherein determining that the first object is nearer than the second object in the image if the computed ratio of computed masses is greater than the normalized ratio of actual masses, and determining that the first object is farther than the second object in the image if the computed ratio is lesser than the normalized ratio.

6. The method as claimed in claim 2, wherein step (f) is characterised by the steps of:

a) selecting an object pair;

b) retrieving the spatial relationships for the object pair; and

c) permuting all spatial relationships for the object pair based on the retrieved the spatial relationships.