WO2011115570A1

WO2011115570A1 - Apparatus and methods for analysing goods packages

Info

Publication number: WO2011115570A1
Application number: PCT/SG2010/000093
Authority: WO
Inventors: Dmitry Nechiporenko
Original assignee: Azimuth Intellectual Products Pte. Ltd.
Priority date: 2010-03-16
Filing date: 2010-03-16
Publication date: 2011-09-22

Abstract

An apparatus for constructing a data model of a plurality of goods forms an association between a first candidate side of a candidate goods package extracted from a first image of a first side of the plurality of goods packages and a second candidate side of the candidate goods package extracted from a second image of a second side of the plurality of goods packages. An apparatus for performing a determination of whether a candidate goods package is of a known type of goods package operates on a plurality of indicator values to perform the determination, each of the plurality of indicator values representing a provisional indication of whether the candidate goods package is of the known type of goods package.

Description

APPARATUS AND METHODS FOR ANALYSING GOODS PACKAGES

The invention relates to an apparatus and a method for constructing a data model of a plurality of goods packages. The invention also relates to an apparatus and a method for performing a determination of whether a candidate goods package is of a known type of goods package. The invention also extends to machine- (computer-) readable media having stored thereon machine-readable instructions for executing, in a machine, the aforementioned methods. The invention has particular, but not exclusive, application for analysing a plurality of goods packages on a pallet to facilitate automated warehouse management.

In logistics, inbound and outbound cargo control is typically an error-prone, expensive and time-consuming process requiring a substantial amount of work maintaining WMS (Warehouse Management Systems) and ERPs (Enterprise

Resource Planning Systems). The results of this cargo control are often hard to evaluate and contain far too little data to be of any great assistance to the warehouse management process. In a typical current scenario, inbound goods are checked with three main steps:

1) Determine what has arrived and from which supplier

2) Count how many cases have arrived, which articles and what quantities

3) Determination of damaged or missing goods For outbound goods the steps are as follows:

4) Count how many cases have arrived, which articles and what quantities

5) Determination of damaged or missing goods

Steps 1) 'Determining what has arrived and from which supplier' and 3)

'Determination of damaged or missing goods' are principally manual activities and therefore are error prone processes. Typically a warehouse worker visually inspects boxes looking for logos and part numbers and then enters this data onto a paper form. At some later time, this form will be manually keyed into some type of spreadsheet or management system. There is a high degree of data loss as well as inaccuracy.

Counting 'how many cases have arrived, which articles and what quantities' in step 2) is typically done by warehouse workers using manual barcode scanners. Barcodes generally include information on articles, quantities, serial numbers, order numbers, and carton/pallet IDs. In some cases it may include country of origin and some supplementary information for vendor's IT system. The results of this data are often connected directly to a WMS or ERP system.

These existing methods have several prominent problems:

• The manually collected data is unreliable and thus has a low confidence rate.

• Barcode scanners must be operated in a rigorous sequential manner. All

barcodes must be collected in 'the proper' order. One missed scan could propagate an error throughout the entire sequence of barcodes.

• There is no way to accurately correlate the barcode data with manually collected paper data.

• Barcode data can easily be corrupted by scratches on the labels or presence of foreign material.

Some warehouses have implemented RFID (Radio Frequency Identification Device) tags as an alternative to manual tracking. This method is much more accurate, than barcode-reading combined with paper processing. It also much faster, as it only takes the truck driver with the pallet to pass before reading portal, to acquire the whole information on the chips from the pallet. However this method also has several disadvantages: • Cost: The cost of the RFID labels and reading equipment is very high; much higher than normal barcodes. This adds substantially to the cost of each and every tagged carton.

• Robustness: RF tags are sensitive to temperature, humidity, and magnetic fields.

This can be highly problematic in the typical 'uncontrolled' environment of a warehouse.

• Accessibility: RFID cannot be used in dense containers or within materials such as metals and liquids. These materials shield the radio waves resulting in a increased probability of errors. Such a condition forces the operator to revert back to the manual method which defeats the initial purpose.

Other optical recognition systems are available which allow a warehouse manager to recognise barcodes on, for example, goods cartons on a pallet (or text/colour information), and use this information as 'pallet content'. However, this is still not ideal because there is ambiguity as to which barcode (or serial number or other carton data value) belongs to which carton. Also, if a carton barcode is damaged there is no provision for error recovery.

And even when it is known that there should be, say, 20 cartons in a pallet, there is no guarantee that, having 20 carton data entries, some of them were not taken from the same carton (like in case each carton has labels on front and rear side and both sides are visible).

Techniques to mitigate these issues have been proposed by the Applicants in commonly-owned International Patent Application Nos. PCT/SG2009/000108 and PCT/SG2009/000472, the contents of which are hereby incorporated by reference in their entirety. Further techniques to mitigate these issues are set forth herein.

Accordingly, the invention is defined in the independent claims. Some optional features are defined in the dependent claims. Implementation of the techniques disclosed herein allows automated or semi- automated recognition of one or more goods packages in a plurality - for example a stack - of goods packages such as one might find disposed upon a pallet for shipment and/or storage in a goods warehouse. Implementation of the disclosed techniques allows for an initial modelling of a goods package (or stack thereof) by associating detected sides - candidate sides - of a package with one another. If there are spurious results, for example, if instead of recognising one goods package two are detected, implementation of the techniques disclosed may resolve such an inconsistency. For instance, if one side of a goods package is detected in a first view of the goods package (or of a stack of goods packages) and two possible sides are detected in a second view of the package from a second side, this might represent a conflict which must be resolved. Techniques are disclosed which allow for such conflicts to be resolved with reference to data relating to know types of goods packages, such as information relating to the expected contents of a shipment of goods packages, or a database of known packages. This known information represents "context information", reference to which might resolve any conflicts.

The invention will now be described, by way of example only, and with reference to the accompanying drawings in which:

Figure 1 is a block diagram representing an architecture for an apparatus for analysing goods packages;

Figure 2 is a series of images showing elevational views of a stack of goods cartons for analysis with the apparatus of Figure 1;

Figure 3 is a series of images showing goods packages represented in the images of Figure 2 after initial edge detection techniques have been applied to the images of Figure 2;

Figure 4 is a flow diagram illustrating steps in the side association process performed by the apparatus of Figure 1; Figure 5 is an image of a side of a goods package, illustrating elements of the goods package visible in an image of the goods package;

Figure 6 is a flow diagram illustrating steps in the process for indicator value derivation by the apparatus of Figure 1, and derivation of the indicator values, and also illustrating images and tables used in indicator value derivation;

Figure 7 is a flow diagram illustrating steps in the process for conflict resolution in the apparatus of Figure 1 using the indicator values derived in the process of Figure 6; and

Figure 8 is a flow diagram illustrating steps in a secondary process for conflict resolution in the apparatus of Figure 1.

As noted above, implementation of the techniques disclosed herein allows recognition of one or more goods packages. In the exemplary techniques disclosed, images of the goods package(s) are acquired and analysed for the recognition. One exemplary apparatus for acquisition of the images is that disclosed in commonly- owned International Patent Application No. PCT/SG2009/000157 the contents of which are hereby incorporated in their entirety, although the techniques disclosed therein may be augmented to include use of a colour camera and/or an overhead camera, as will be discussed further below. The images are processed and coarse recognition of the goods package(s) is performed. A coarse data model of the goods package(s) is constructed. Refinement of the coarse data model is then performed. In a first step, indicator values are derived for the goods package(s) indicating whether the goods package(s) could be of a known type of goods package. The indicator values are derived from the goods package(s) themselves and may relate to properties of the goods package, such as package dimensions, colour of the package, logos on the package, barcodes on the package, and text on the package, such as alphanumeric text, and including Roman characters, Cyrillic characters, Chinese characters and the like. The indicator values are used to resolve any conflicts in the coarse data modelling and/or to confirm the findings of the coarse data modelling- i.e. no correction may be required - and a refined data model of the goods package(s) is produced with, if appropriate, any conflicts removed.

The exemplary techniques presented herein are given in the context of the goods package being of cardboard or other card-like material, and being of generally cuboid shape. However, the techniques are readily extendable to all types of goods package, including goods cartons made of plastic, metal containers, wooden boxes/crates, paper/textile bags, packages of or wrapped in plastic film -whether clear (transparent) plastic film, or opaque/partially opaque film - or trays for placing goods in or on, with or without wrapping.

Turning first to Figure 1 apparatus 100 comprises a microprocessor 102 and a memory 104 for storing routines 106. The microprocessor 102 operates to execute the routines 106 to control operation of the apparatus 100 as will be described in greater detail below. Apparatus 100 processes a series 121 of images of a goods package 122 which, in the example of Figure 1, is in a stack 120 of goods packages. Apparatus 100 comprises storage memory 108 for receiving and storing the series of images 121. Storage 108 may also store the context database 109 - to be discussed in greater detail below - but the context database 109 may also be stored in a separate apparatus which apparatus 100 is able to communicate with.

Apparatus 100 is further comprises candidate package recognition module 110, side association module 112 (which performs coarse modelling of one or more goods packages in the stack 120 of good packages), indicator value derivation module 114, indicator value operation module 116 and data modelling module 118 which refines the coarse modelling performed in module 110. The operation of these modules will be discussed in more detail below.

In a first mode of operation to be described more fully below, apparatus 100 is configured to construct a data model of a plurality of goods packages. In doing so, apparatus 100 forms an association 124 between a first candidate side 122a of a candidate goods package 122 extracted from a first image 120a of a first side of the plurality of goods packages 120 and a second candidate side 122bl of the candidate goods package 122 extracted from a second image 120b of a second side of the plurality of goods packages 120.

In a second mode of operation to be described more fully below, apparatus 100 is configured to perform a determination of whether a candidate goods package 122 is of a known type of goods package (which may either be a known type from a database of know types of goods packages or one type of goods package expected in a particular shipment). In doing so, apparatus 100 operates on a plurality of indicator values (126) to perform the determination, each (126a, 126b,... 126n) of the plurality of indicator values representing a provisional indication of whether the candidate goods package 122 is of the known type of goods package.

That is, apparatus 100 may be configured to perform the functionality of both of the first and second modes of operation. However, it will be appreciated that for each of the first and second modes of operation, an apparatus for performing the stated functionality of the respective modes of operation may be provided as stand-alone apparatus.

Turning now to Figure 2, a series 221 of images comprising four orthogonal views 220a, 220b, 220c, 220d of sides of a stack 220 of goods packages disposed upon a pallet 230 is illustrated. In the present example, both greyscale and colour images are acquired for the sides of the goods package(s). As a further option, it is possible to obtain a plan view of the goods package(s) from an overhead camera (not shown) to obtain additional information concerning the layout of goods package(s).

As shown in view 220a, a first side of the stack 220 is shown. Visible in the stack is a plurality of goods packages disposed in a non-uniform manner on the pallet 230, where the individual types of packages are denoted with a #mark - that is, #1 denotes a first type of package and so on. Visible is a package #1, two packages #2, two packages #3 and a package #4. Also visible is an as-yet-unidentified object 236. In fact, this is a package only partially visible in the view 220a. Visible across packages #2 are packing strips 232 and across package #1 a broader line 234 which comprises packing tape.

As shown in view 220b, a second side of the stack 220 is shown. Visible in the stack in this view are three #1 packages, two #2 packages, one #3 package and two #4 packages. Also visible is an as-yet-unidentified object 240, also a package only partially visible in view 220b. A broad line 238 is visible across the packages #2, which comprises packing tape.

Similar observations may be made about objects which are visible in views 220c and 220d.

Because several items are partially obscured in the views 220, and because of items such as the packing strips 232 and packing tape 234, 238, then relatively

straightforward edge-detection techniques are likely to be insufficient to recognise the goods packages correctly.

Candidate package recognition module 110 of apparatus 100 operates in the following way to provide a more sophisticated package recognition technique. In doing so, module 110 performs edge detection techniques on both the greyscale and colour images which have been acquired. For instance, the edges of the package(s) will typically present a lower chromatic value in the greyscale images, and colour thresholding is applied in terms of the colour images in order to perform a detection of the goods carton edges. A benefit of using both colour and greyscale images is that it is possible simple edge detection with greyscale images only would detect the packing strips 232 and packing tape 234, 238 as box edges, thus leading to spurious results.

For instance, simple black and white/greyscale edge detection techniques applied to the image show in view 220a would lead to the result shown in Figure 3a, where packages #4 and 3 are correctly detected, but for packages #2, the edge detection technique incorrectly recognises these as being made up if discrete packages #2c, #2d, #2e and #2f because of packing strips 232. Additionally, package #1 is incorrectly recognised as two separate packages #lh and #li (122bl and 122b2) because of the presence of packing tape 234. A similar scenario is shown in Figure 1, where the side 122b of package 122 is mistakenly recognised as two separate package sides 122bl and 122b2.

After apparatus 100 applies colour thresholding, the results are shown in Figure 3b. Some correction is made in that each package #2 is recognised as being not two packages but one package. However, it is possible there are still errors; for instance, even after the colour thresholding application, package #1 might still be recognised as two separate packages because the packing tape 234 still leads the edge detection algorithm into thinking there are two packages.

Figure 3c shows the results after package recognition using both greyscale and colour imaging techniques for the second side, view 220b, of Figure 2.

Note that it is also possible for the colour thresholding step of the algorithm to refer to known data. For instance, this known data might comprise of data from context database 109 related to known types of packages, or to details of the expected shipment of packages on pallet 230. So, for instance, and taking the packages #2 as an example, if it is known that is not possible - or at least unlikely - that the pallet load will comprise four packages having the dimensions of four detected packages #2c, #2d, #2e and #2f, the apparatus 100 can make an intelligent guess that, in actual fact, these comprise two #2 packages. So, for example, if apparatus 100 has knowledge that the expected shipment is from a particular supplier which does not provide packages of the smaller dimensions of four detected packages 2c, #2d, #2e and #2f, then correction may be made.

Module 110 continues to apply the same techniques to all sides of the goods packages shown in views 220 of Figure 2. It is not shown in Figure 3, but recognition of side 4 in view 220b of Figure 2 yields the result that, on the lower level of the pallet 230 on side 4, there are three #1 packages then this presents a potential conflict, as module 100 has detected #1 package 122a in view 220b is adjacent to the two packages 122bl (#le) and 122b2 (#lf) as shown in Figure 3b. However, as both results are plausible, these are both kept e.g. by being written to a table or matrix - for further processing by the apparatus 100. In order for the apparatus 100 to build a data model of the goods package(s) from the candidate package sides - e.g. packages #le and #lf, #2 packages on side 1 and package side 240 on side 2 - derived by candidate package recognition module 110, apparatus 100 performs side association between candidate package sides from the module 110. So, in this mode of operation, apparatus 100 forms an association 124 between a first candidate side 122a of a candidate goods package 122 extracted from a first image 120a of a first side of the plurality of goods packages 120 and a second candidate side 122bl of the candidate goods package 122 extracted from a second image 120b of a second side of the plurality of goods packages 120. One step in forming the association comprises extracting a first dimensional parameter for the first candidate side from the first image and a second dimensional parameter for the second candidate side from the second image and identifying the second candidate side as a candidate side from a comparison of the first dimensional parameter with the second dimensional parameter. So, apparatus 100 determines positional/dimensional data of the candidate sides from the respective images and determines whether the candidate sides can be adjacent one another on the pallet 230. In the stack 120 of goods packages shown in figure 1, apparatus 100 recognises that candidate side 122a is a candidate side in the upper right hand side of the stack 120. On the adjacent side 2, candidate side 122b2 is the uppermost package (only one column of packages on side 2) and therefore these two candidate sides 122a and 122bl are likely to be sides of the same package 122. The positional data can be derived from pixel data in the series 121 of images, which can be correlated to actual dimension of the goods packages. Additionally, apparatus 100 forms a second association 126 between the first candidate side 122a and a third candidate side 122b2 of the candidate goods package extracted from the second image. That is, apparatus 100 knows from the positional/pixel/dimensional data that side 122b2 can also be another side of package 122 adjacent first candidate side 122a. However, identification of the third candidate side of the candidate goods package comprises a conflict. Apparatus 100 resolves the conflict with reference to data from a context database, as will be described in greater detail below, with respect to Figures 5 to 8. First, however, the process for side association will be described with reference to Figure 4. (For the sake of completeness, note that the third candidate side can also be a third side 122c of the candidate goods package 122 or, say, a top side (in the view of Figure 1) of package 122.)

Process 400 starts at 402, and one of the sides of the stack of goods package(s) is selected for processing. A candidate goods package on the side of the stack is selected at step 406. So, and referring to Figure 1, candidate goods package 122 is selected at this step. At step 408, a first side 122a of goods package 122 is selected. A candidate side 122a is also shown in view 220b of Figure 2. At step 410, apparatus 100 checks to determine whether there is a candidate match for the goods package on another side of the stack. In the example of Figure 1, apparatus 100 would determine that candidate side 122bl presented a possible (or candidate) match on side 2 of the stack for candidate side 122a on side 1 of the stack. When the check at step 410 returns a positive result then, at step 412, apparatus 100 selects the candidate side 122bl and determines whether there is a match of dimensional parameters at step 414 for the two candidate package sides 122a, 122bl. From a determination that candidate side 122a is at the upper right hand side of side 1 and that candidate side 122bl is the uppermost in the single column of side 2, apparatus 100 associates these sides with one another in step 416, forming association 124. It may be that because of imaging tolerance or improper calibration of the imaging apparatus, the positional data for the two candidate sides is not a complete match; for example, if comparing pixel data - which corresponds to dimensional data - apparatus 100 may consider from the comparison that the two candidate sides are not touching one another. In such instances, apparatus 100 is configured still to determine an association should still be formed for the two candidate sides of the dimensional data is within an acceptable tolerance.

When the check at step 410 yields a negative result, the process proceeds directly to step 420, to be discussed further below.

Going back to step 414, if no match of dimensional data can be determined - for example, apparatus 100 had, first, checked side two for candidate side 123b, where the dimensional data of the two candidate sides 122a and 123b will clearly not match - then process 400 loops back to step 410, to check if there are any more candidate matching package sides on another side of the stack.

After forming of the association 124 at step 416, apparatus 100 checks to determine if there are other candidate sides which might be sides of package 122 on other sides of the stack 120. In this case, apparatus 100 will find that candidate side 122b2 is a candidate match as a side of goods package 122 and the process loops back to step 412 for the association 126 to be formed between candidate sides 122a and 122b2.

If apparatus 100 has already checked all the candidate sides 122bl, 122b2 and 123b on side 2, it then checks on other sides of the stack 120. In the first instance, the apparatus would find at the next iterative check at step 418 that candidate side 122c on side 3 of the stack was a possible match, and the process loops back to step 412. If no other candidate matches are found at step 418, process 400 moves on to check at step 420 whether there are other candidate packages to be checked on the present side of the stack 121. For instance, this check yields that, for example, package 125 is a candidate package for modelling and the process loops back to step 406 to form associations for package 125.

If no more candidate packages are found on side 1 of the stack at step 420, the apparatus 100 checks at step 422 whether there are other sides of the stack for which the same operations are to be made and loops back at stop 404 to select the next side of the stack, where the process repeats as outlined above. If apparatus 100 determines at step 422 there are no more sides of the stack to check, the process ends at step 424.

Although the example of Figures 1 to 4 is provided in the context of there being multiple goods packages in a stack 121, it will be appreciated that the techniques disclosed can be applied to a single goods package to form an association between sides of the package, should it be necessary.

In effect, the side association module 112 provides a coarse modelling of the goods package(s). This may be sufficient for requirements, however, where there are conflicts - for example, the detection of two candidate package sides 122bl and 122b2 on side two of stack 121 when only one adjacent candidate side 122a is found on side one - then these may need to be resolved for full analysis of the stack of goods packages. The techniques to resolve any such conflicts are described in relation to Figures 5 to 8. However, it will be appreciated that the techniques of Figures 5 to 8 may be used where no conflict resolution is required, and these techniques simply provide higher confidence in the analysis of the goods package(s). The techniques of Figures 5 to 8 comprise another mode of operation of apparatus 100 and, as mentioned above, a stand-alone apparatus may be provided to utilise these techniques; that is, the techniques discussed with reference to Figures 5 to 8 may be provided separately or in conjunction with the techniques discussed with references to Figure 2 to 4.

When the techniques are used in conjunction with one another, as illustrated in Figure 1, apparatus 100 is able to resolve any conflict from, say, the identification of two candidate goods package sides 122bl and 122b2 instead of just one side 122b. The conflict is resolved with reference to data from a context database. Apparatus 100 resolves the conflict by performing a determination of whether a candidate goods package 122 is of a known type of goods package, the known type of goods package being defined by data in the context database 109, and to perform the determination by operating on a plurality of indicator values 126, each of the plurality of indicator values 126a, 126n,... 126n representing a provisional indication of whether the candidate goods package 122 is of the known type of goods package. In the following examples, the indicator values relate to an element of the candidate goods package visible in an image of the goods package.

An exemplary goods package and elements thereof is illustrated in Figure 5. The discussion is given in the context of the goods package being a goods carton made of cardboard, plastic or similar material, but the techniques are applicable to all types of goods packages. An acquired image of a side 500 of a goods package is illustrated. The side 500 has a length 514a and a height 514b, both of which may be defined in terms of pixels in the image. In the present example, Apparatus 100 is able to derive the length and height by finding the co-ordinates of the pixels of the corners of the side 500 and calculating the dimensions from a simple arithmetic operation on the co-ordinates, such as subtraction.

Visible in the image are labels 502 and 504, a vendor logo 506, handling/shipping marks 508 and barcode 510. Label 502 has co-ordinates 512a, 512b, 512c, 512d located at the four corners of the label. Information on labels 502, 504 includes human-readable alpha-numeric characters and barcodes. In the example of Figure 5, the image of side goods package side 500 is an 8-bit per pixel monochrome (e.g. greyscale) high resolution image. The techniques disclosed herein are readily extendable to use with colour images, but it has been found that in some

implementations better performance is achieved using a greyscale image.

Thus, elements of the candidate goods package may be derived from or relate to:

• package dimensions

· package logos

• package text, such as on the labels, or on the packages themselves

• package barcodes, either printed directly onto the package or on labels affixed to the package

• package colours and patterns (the arrangement/density and/or allocation of the elements on the package).

The package logos, barcodes, text strings and labels may be extracted and recognised in accordance with the techniques disclosed in, for example, commonly- owned International Patent Application No. PCT/SG2009/000472 mentioned above. This same application also discloses techniques for performing OCR on the packages and extracting goods package text. However, other label, logo, colour, pattern, barcode and text extraction and recognition techniques may also be applied.

Derivation of the indicator values is discussed with reference to process 600 of Figure 6. The process starts at step 602, and data is retrieved from the context database 109 at step 604. Apparatus 100 then performs dimension detection 606, logo detection 608, OCR/text search 610 and barcode detection 612. In the example of Figure 6, apparatus 100 performs steps 606, 608, 610 and 612 in parallel but, alternatively, they can be performed at different times, such as sequentially.

Dimension detection 606 is performed as discussed above. Apparatus 100 forms a package matrix as in Figure 6b where column "Package" indicates a number allocated to a package in the stack of packages, and the number in columns "side 1", "side 2", "side 3" and "side 4" indicate the order that package appears in the image of that side. Dimensions of the package visible in a particular view are written to the respective column "Size XY", where XY correspond to the dimensions 514a and 514b in the X and Y directions respectively of Figure 5.

Logo detection 608 includes retrieval of the expected shipment/article from an ERP system {not shown). The size for each expected article is then retrieved from the Context Database 109. A data file for logos for each expected article is retrieved from the context database 109 and a size filter is applied to select the particular logo which corresponds with the size of the candidate goods package. Logos are searched for and recognised in the goods package side 500 and a search is performed for all types of known goods packages in the context database 109 corresponding to the recognised logo. A table for known types of packages is shown in Figure 6c. The logos are compared and a corresponding confidence score is. calculated indicating a level of confidence in accuracy of whether the logo has been correctly detected or not. There are various techniques for calculating the logo confidence score. For instance, various readily-available algorithms can calculate a score based on combination of curves, lines, etc. and proportions thereof. Such a calculation makes a determination of which known logo the detected logo is similar to. For example, a certain combination of curves/lines may be present in full in one logo, and another may have something 80% of more important logo elements and 30% of secondary elements. From this a reasonable confidence score can be calculated, even if precise matching is not possible. Thresholding may then be applied to the scores.

OCR/Text search 610 and Barcode Detection 612 may operate to recognise text and barcode either printed on labels on the package or text and barcodes printed directly onto the package. If reading from a label apparatus 100 may also apply the label extraction techniques disclosed in commonly-owned International Patent

Application No. PCT/SG2009/000472 mentioned above. In any event, OCR is performed to extract text/characters from the package. An example of a printed label on a package is shown in Figure 6d, character strings extracted from the label are shown in Figure 6e and Figure 6f shows a word list generated from the extracted character strings, along with a corresponding confidence score that the words in the list are correctly derived. Again, the generation of the word list and the confidence score may be performed as disclosed in International Patent Application No.

PCT/SG 2009/000472. Apparatus 100 can be configured to write to (or keep in) the Word List words/character strings having a confidence score above a threshold.

For barcode detection, a list of expected barcodes is retrieved from the details of the expected shipment. Barcode recognition is performed on the package (see, for example, PCT/SG 2009/000472) and a detected barcode is illustrated Figure 6g. A table is illustrated in Figure 6h (in this example comprising only a single row) with the barcode details and a confidence score the barcode has been correctly recognised.

In one implementation, dimensional information for goods package elements is extracted for use in accordance with the techniques disclosed in

PCT/SG 2009/000472, and used to derive an indicator value - for example an element dimensional indicator value - as discussed below. Further, colour information can also be used to derive an indicator value - for example a specific colour information indicator value - as discussed below. Derivation of the various indicator values in module 114 is now discussed. The dimension indicator value is derived at step 614. In this step, apparatus 100/module 114 checks the dimensions of the candidate goods package against known dimensions of the expected articles. For instance, if the dimensions 514a, 514b of the side 500 of the goods package of Figure 5 are 100mm x 80mm, apparatus 100 checks through all the sizes of the expected articles for a possible match, applying, where appropriate, scaling factors to take into consideration imaging tolerances such as the distance the image acquisition apparatus from the carton varying. Thus, the combination of 100mm x 80 mm (and vice versa) is checked against the known dimensions of the expected articles and apparatus 100 returns a results table giving an indication of whether the candidate goods package is of a particular type, such as the Table 1 below. A confidence score may also be calculated, although this column is left blank in the table below. The apparatus may not be able to measure absolutely accurately the exact dimensions of a candidate good package from the image, but only relative dimensions. So, for example, a goods package with detected

dimensions of around, say, 50cm x 50cm towards the centre of an image (where distortions are at a minimum) will yield a higher score rating that it is a package of a known type having dimensions of 50cm x 50cm. If detected dimensions are 35cm x 70cm, (or 70cm x 35cm), the apparatus will provide a lower score rating that the detected goods package is of the type known to have 50cm x 50cm. This is especially true when we have a known goods carton type having dimensions of 70 cm x 35cm.

Table 1 In this example, apparatus 100 implements a neural network to determine these results. So, in the example of Table 1, apparatus 100 detects that the candidate goods package (Carton 1) could be of type E, F, G or H in the list of expected articles; that is, for each of the types of known goods package, an indicator value is derived indicating whether the candidate goods package is of the type of one of the expected articles. It will also be appreciated, that this comparison can be made against the list of known types of goods packages in the context database 109. Indeed, a comparison with the list of known types of goods packages in the context database 109 can be extrapolated thereafter to a determination of whether the candidate goods package is of a particular type in the list of expected articles if the expected article type is stored in the context database 109.

A logo indicator value is derived at step 616. A logo list is generated for each type of goods package in the expected shipment and/or context database 109. This list is compared with the results of the logo detection step 608 and from that a logo indicator value of whether the candidate goods package is of the known type of goods package is generated, such as in Table 2:

Table 2

Text indicator values are derived at step 618. In the present example, five text indicator values are derived: for keywords, for part numbers in the goods packages, for manufacturer of the goods in the goods package, for producer, and for country of shipment. (In general, the producer of a particular good may be the same as a manufacturer, but in some instances, goods may be manufactured by one party on behalf of a separate producer.) For each of the indicator values, a table similar to Tables 1 and 2 is generated.

Further, the indicators may not be just textual. For instance, an indicator may also relate to a colour scheme in the package derived by the apparatus from RGB information in a colour image of the package, or parts thereof. An indicator may also apply to patterns, images of trademarks, package handling marks (which, in principle, may be analysed in a manner similar to logos but specifying different categories, such as product type or package type). Each indicator type may have different versions; for example different logos 'Sharp" and text "Sharp" and "Sharp Corp". The apparatus may be set up to apply greater weighting for one indicator over another, as the result of one may be considered to be more reliable.

For instance, for keywords, a check word list is created by fetching words from the context database 109 for each type of goods package in the expected shipment list. Apparatus 100 checks to determine whether there is an exact match from the word list of Figure 6f with the check word list and/or checks for a distance between words in the keyword list and words in the list of Figure 6f. The distance between the words may be calculated in accordance with the techniques disclosed in International Patent Application No. PCT/SG 2009/000472. From this another table for keywords is generated, including keyword indicator values as to whether the candidate goods package is of a type in the expected shipment.

Similar operations are performed for part numbers where a check word list of part numbers is retrieved from the context database 109 and compared with the list of Figure 6f. A part number indicator value is derived accordingly. A manufacturer indicator value is derived by creating a check word list of manufacturers retrieved from the context database 109 and compared with the list of Figure 6f. A producer indicator value is derived by creating a check word list of producers retrieved from the context database 109 and compared with the list of Figure 6f. A country indicator value is derived by creating a check word list of countries retrieved from the context database 109 and compared with the list of Figure 6f.

A barcode indicator value is derived in a fairly similar way at step 620 where a list of barcode check words is retrieved from context database 109 comprising a list of alphanumeric strings corresponding with barcodes. Apparatus 100 checks to determine whether there is an exact match from the barcode word list of Figure 6h with the barcode check word list. From this another table for barcode keywords is generated, including barcode indicator values as to whether the candidate goods package is of a type in the expected shipment.

Thus, in the exemplary techniques disclosed above, a total of eight indicator values are derived for each type of known goods package (whether a known goods package from the expected shipment and/or from the known types of goods packages from the context database 109). Of course, it will be appreciated that other numbers of indicator values may also be used, whether higher or lower than eight. For instance, higher numbers of indicators for keywords/text, barcodes and logs may be employed. Additionally, one or more indicator values may be derived for colour information of the package (e.g. the colour of the package material). Apparatus 100 processes an image of the goods package and extracts RGB information from the image, and compares this with RGB information about a known type of goods package in the context database 109. This colour process can also be carried out for colour information relating to elements (labels, logos, etc.) on the goods package.

One or more indicators may be derived concerning logos. Some goods packages are typically provided with multiple logos thereon, such as a manufacturer's logo and a logo for the product itself.

One or more indicators may be derived concerning pattern or dimensional information of goods package elements. This includes data relating to the positioning of the elements on the package, either in absolute terms in relation to its position on the package itself, or relative one another. This information is derived and compared with goods element dimensional information about a known type of goods package in the context database 109. Examples of useful information which can be derived for this purpose is known spacing between, say, labels, barcodes, text, handling marks, and the like. Use of such an indicator and such data can assist apparatus 100 in detecting the orientation of a package because if goods package elements for a particular type of goods package have a known arrangement, and apparatus 100 recognises a known pattern in the arrangement of the elements, the orientation of the package can be derived from this.

But, following the present example, the eight indicator values comprise as follows:

• one indicator value for dimensions

• one indicator value for logo

• five indicator values for keywords/text

· one indicator value for barcodes.

Thus, apparatus 100 is able to derive indicator values from a determination of whether the element of the candidate goods package relates to a corresponding element of the known type of goods package.

The indicator values are operated upon in module 116 for increased confidence in the recognition of the candidate goods package. This also allows for resolution of any conflicts such as the conflict between the detection of two candidate goods package sides 122bl and 122b2 instead of one candidate side 122b, from Figure 1. Indicator value operation module 116 may comprise of, say, a gate logic module and/or a neural network. In the present example, the indicator values are in either "Y" (yes) or "N" (no) form, thus being essentially of logic form. Apparatus 100 is then able to perform, with indicator value operation module 116, logical operations on the indicator values.

In a first step, apparatus 100/module 116 performs the operation by performing a first logical AND operation comprising an AND of the indicator values. So, indicator values for, say, carton type A of the expected shipment are subjected to a logical AND operation. This may comprise an AND of all of the plurality of indicator values (for each of the known types of goods carton), thereby to produce a first logical result. So, for the first logical AND operation to produce a TRUE result, all eight of the indicator values for, say, carton type A would have to be in "Y" form. This would provide a result of reasonably high confidence that the candidate goods package was of type A.

In this example, module 116 is also configured to perform a second logical AND operation comprising an AND of a subset of the plurality of indicator values, thereby to produce a second logical result. So, for instance, an AND of seven (or six or other number) of the indicator values is performed. Additionally for this example, results can be calculated for all permutations of seven indicator values (of the eight listed above), and logical AND operations performed for each permutation result. The same operation can also be performed for permutations of six, five, four, three and/or two indicator values. Additionally, the result of each operation on the permutations of different numbers of indicators may be subjected to a logical AND operation. Regardless of the precise nature and combination and permutations of the indicator value AND values, the result is a second logical result. Module 116 performs a third logical AND operation comprising an AND of the first logical result and the second logical result. Very high confidence in the correct recognition of a candidate goods package may result from such an operation.

The technique is illustrated with reference to an example having three indicator values: X, Y and Z. So apparatus operates on two categories (input = 3 and 2). In category 3, apparatus 100 operates on a combination of (X & Y & Z) from each expected article. In category 2, apparatus 100 operates on a combination of (X & Y, X & Z, Y & Z) from each expected article. Apparatus 100/module 116 scans through the categories (from largest number of input to 2 input). It will record each category that has a TRUE state in its corresponding gate combination. Apparatus 100 extracts the combination/combinations that give the true state from the category. This is illustrated in Table 3:

Table 3

With the above result, apparatus 100 determines the candidate goods package is Article Type A as, as this is the only type of category where the first logical result, second logical result and third logical result are all "True".

If apparatus 100 produces a result of Table 4:

Table 4 the third logical result for all article categories is False.

As such, when operation on the plurality of indicator values returns a result which is not fully determinative of the known type of goods package - for example, the number of results is not one - apparatus 100 performs a weighted score calculation to determine a likelihood the candidate goods package is of a known type of goods package, as will now be described.

Generally speaking, apparatus 100 performs the weighted score calculation by identifying instances of a logical TRUE result from AND operations on the indicator values, and to apply a weighted score to each of the instances. That is, apparatus

100 extracts the instances of True from Table 4 or equivalent, and these are then put into another comparator/neural network. From the Gate Logic, each gate's component score and weight will be used as the input for the comparator. (The scores may be selected by a user when the neural network is being trained, and scores re-assigned based on results until matching, or approaching, real-world values.) Weighting values can be selected according to perceived confidence. An example is given in Table 5 when operating on the True results from Table 4:

Table 5 So, in Table 5 instances of True results from Table 4 have been retrieved, along with the corresponding score value. Weights are applied to this core and a resulting score calculated. In the example of Table 5, the highest score is for Article A and, thus, apparatus 100 identifies the candidate goods package as being of type A. That is, when operation on the plurality of indicator values returns a result that the candidate goods package may be of a first known type of goods package (e.g. type A) or of a second known type of goods package (e.g. type E or type G), the apparatus identifies the candidate goods package as being of the first known type of goods package or of the second known type of goods package from the weighted score calculation. In the present example, the candidate goods package is identified as being of Type A. The apparatus then discards any results it held for Type E or Type G. If, for example, Type A related to a first known type of goods carton having the dimensions of side 122a in Figure 1, type E related to a second known type of goods carton having dimensions of candidate side 122bl in Figure 1, the apparatus would correctly recognise the candidate goods package was of type A, thereby resolving the conflict which had arisen due to the inaccurate detection of side 122bl, and would ascribe candidate goods package sides 122bl and 122b2 as being a single side 122b.

Conversely, if the apparatus had incorrectly performed an initial detection of one candidate goods package, when, in fact, there were actually two candidate packages, each of the two packages can be identified by performing respected weighted score calculations as outlined above.

The operation of module 116 can also be described with reference to Figures 7 and 8. Starting first with Figure 7, a process 700 for operating on the indicator values starts at 702. Module 116 retrieves a candidate goods package, such as that illustrated in Figure 5, at step 704. The indicator values derived as described with reference to Figure 6 are retrieved at step 706 for a first type of known package, where logical operation - for example, one or more Boolean AND operations as described above - is conducted at step 708. Apparatus 100 checks for a positive result at step 710 and if found this is flagged at step 712 and written to a table, also as described above. Alternatively, a false result is flagged and written to a table at step 714. At step 716, apparatus 100/module 116 checks whether there are more known package types to check. If more are to be checked, the next type is retrieved at step 718 and the process loops around steps 706, 708, 710, 712/714 and 716, until apparatus 100 determines no more known types are to be checked. Apparatus 100 checks for the number of positive results from the logical AND operations (in

Table 3 or Table 4) at step 720. If the number is 1, then the candidate goods package is flagged according to type at step 722. Apparatus 100 checks to determine whether more goods packages are to be checked at step 724. If more are to be checked, apparatus 100 hops the process back to step 704, and the process repeats. If no more are to be checked, then the process ends at step 726. If at step 720 the results are not fully determinative of the type of known package which the candidate goods package is, apparatus 100 checks at 727 whether the results present a second conflict. That is, apparatus 100 operates to determine that the candidate goods package is one of multiple known types of goods package, and when operation on the plurality of indicator values returns a result which is not fully determinative of the known type of goods package, the apparatus is further configured to flag the result as a second conflict. Such a conflict may be a significant conflict, for example, an irresolvable conflict which can arise when, say, apparatus 100 detects the box might be from two different suppliers: "supplier 1" and "supplier 2". There will be instances where such second conflicts are not considered to be irresolvable, and the process 800 of Figure 8, to be discussed below, is followed to provide acceptable results. In other instances a user will consider such conflicts to be an error of sufficiently serious nature that it should be checked. In which case, the user avails itself of the optional functionality of apparatus 100 which allows it to be configured to flag such results as significant conflicts. Apparatus 100 can then be controlled to re-perform the initial edge detection algorithm on the images of the goods packages if only one package has been inadvertently recognised when two packages (one each for supplier 1 and supplier 2) should have been recognised. In such circumstances, apparatus 100 is configured to end the process of Figure 7 at step 726. Apparatus 100 can then be caused to re-perform the initial edge detection on the colour and greyscale images, described above.

Another example of such a significant conflict would be if apparatus 100 detected conflicting information - whether on a single goods package or in multiple packages within a pallet load - relating to, say, chemicals, and also to, say, foodstuffs. Another such example would be two conflicting part numbers extracted from an OCR process for a goods package, where it should not be possible for one package to have both part numbers. The same principle applies to any two types of goods or goods parameters which would not normally be shipped together, where detection of the two types of goods in the same shipment might represent an error requiring checking.

In such circumstances, apparatus 100 effectively makes use of a logical NOT operation where apparatus 100 can be configured to run a NOT operation such as NOT [(Propertyl) AND (Property2)], where detection of both "Propertyl" and "Property 2" represents a possible error which should not proceed unchecked.

The edge detection module is provided with threshold settings which can be varied to allow for a more accurate recognition of the goods packages. That is, apparatus 100 performs an edge detection process on images of the plurality of goods packages using user-specified thresholding parameters. So, if apparatus 100 is caused to re-perform the edge detection techniques, the re-performance can be tuned to allow for a more refined detection which to allow for detection of two packages - one each for supplier 1 (Propertyl) and supplier 2 (or Property2) - for subsequent processing. Thus, after edge detection has been re-performed, the process of Figure 4 is run again to associate sides of the candidate goods packages (coarse pallet reconstruction). The goods packages are thus re-modelled on the refined information derived from the detection of the significant (irresolvable) conflict. This is followed by the processes of Figures 6 and 7 and, if appropriate, Figure 8, thus allowing refinement of the initial results by varying the thresholding parameters of the edge detection process.

In circumstances where no significant conflict is detected at step 727, apparatus 100 refers to process 800 of Figure 8, at step 728. At step 802, apparatus 100 retrieves the True values calculated at step 712. Apparatus 100 applies weightings to the scores at step 804, before calculating aggregate scores at step 806. Apparatus 100 identifies the highest score at step 808 before referring back to step 722 of Figure 7 via 730 to flag the package according to type. Once the packages have been identified, refined data modelling is performed by apparatus 100 in module 118. For instance, indicator value operation module 116 resolves the conflict relating to candidate sides 122bl and 122b2, thereby allowing the apparatus to discard the association 126 between candidate side 122a and 122b2. Thus, a single association 124' is made between sides 122a and 122b.

The context database 109 can be created in a number of ways. Regardless of the manner followed, context database 109 contains data relating to:

• package dimensions

· package logos

• package text, including text found on package labels

• details of the packages themselves, such as typical colouring on the packages.

These details are selectively retrieved by apparatus 100 for use with the techniques disclosed above.

It will be appreciated that the invention has been described by way of example only. Various modifications may be made to the techniques described herein without departing from the spirit and scope of the appended claims. The disclosed techniques comprise techniques which may be provided in a stand-alone manner, or in combination with one another. Therefore, features described with respect to one technique may also be used and presented in combination with another technique.

Claims

1. An apparatus for constructing a data model of a plurality of goods packages, the apparatus comprising:

a processor; and

a memory for storing one or more routines which, when executed under control of the processor, control the apparatus to form an association between a first candidate side of a candidate goods package extracted from a first image of a first side of the plurality of goods packages and a second candidate side of the candidate goods package extracted from a second image of a second side of the plurality of goods packages.

2. The apparatus of claim 1 configured, under control of the processor, to extract a first dimensional parameter for the first candidate side from the first image and a second dimensional parameter for the second candidate side from the second image and to identify the second candidate side as a candidate side from a comparison of the first dimensional parameter with the second dimensional parameter.

3. The apparatus of claim 1 or claim 2 configured, under control of the processor, to form a second association between the first candidate side and a third candidate side of the candidate goods package extracted from the second image.

4. The apparatus of claim 3, wherein identification of the second candidate side and identification of the third candidate side of the candidate goods package comprises a conflict, and the apparatus is configured, under control of the processor, to resolve the conflict with reference to data from a context database.

5. The apparatus of claim 4 configured, under control of the processor, to resolve the conflict by performing a determination of whether the candidate goods package is of a known type of goods package, the known type of goods package being defined by data in the context database, and to perform the determination by operating on a plurality of indicator values, each of the plurality of indicator values representing a provisional indication of whether the candidate goods package is of the known type of goods package.

6. An apparatus for performing a determination of whether a candidate goods package is of a known type of goods package, the apparatus comprising:

a processor; and

a memory for storing one or more routines which, when executed under control of the processor, control the apparatus to operate on a plurality of indicator values to perform the determination, each of the plurality of indicator values representing a provisional indication of whether the candidate goods package is of the known type of goods package.

7. The apparatus of claim 5 or claim 6 configured, under control of the processor, to operate on indicator values where each of the indicator values relate to an element of the candidate goods package visible in an image of the goods package.

8. The apparatus of any of claims 5 to 7 configured, under control of the processor, to derive an indicator value from a determination of whether the element of the candidate goods package relates to a corresponding element of the known type of goods package.

9. The apparatus of any of claims 5 to 8, configured under control of the processor, to operate on indicator values where each of the indicator values are in logic form.

10. The apparatus of claim 9 configured, under control of the processor, to perform the operation by performing a first logical AND operation comprising an AND of the indicator values.

11. The apparatus of claim 10 configured, under control of the processor, to perform the operation by performing a logical AND operation comprising an AND of all of the plurality of indicator values, thereby to produce a first logical result.

12. The apparatus of claim 11 configured, under control of the processor, to perform the logical operation by performing a second logical AND operation comprising an AND of a subset of the plurality of indicator values, thereby to produce a second logical result, and to perform a third logical AND operation comprising an AND of the first logical result and the second logical result.

13. The apparatus of any of claims 5 to 12 configured, under control of the processor, to determine that the candidate goods package is one of multiple known types of goods package, and when operation on the plurality of indicator values returns a result which is not fully determinative of the known type of goods package, the apparatus is further configured to flag the result as a second conflict.

14. The apparatus of claim 13 configured, under control of the processor, to perform an edge detection process on images of the plurality of goods packages using user-specified thresholding parameters.

15. The apparatus of any of claims 5 to 14 configured, under control of the processor, to determine that the candidate goods package is one of multiple known types of goods package, and when operation on the plurality of indicator values returns a result which is not fully determinative of the known type of goods package, the apparatus is further configured to perform a weighted score calculation to determine a likelihood the candidate goods package is of a known type of goods package.

16. The apparatus of claim 15 configured, under control of the processor, to perform the weighted score calculation by identifying instances of a logical TRUE result from AND operations on the indicator values, and to apply a weighted score to each of the instances.

17. The apparatus of claim 16, wherein, when operation on the plurality of indicator values returns a result that the candidate goods package may be of a first known type of goods package or of a second known type of goods package, the apparatus is configured, under control of the processor, to identify the candidate goods package as being of the first known type of goods package or of the second known type of goods package from the weighted score calculation.

18. A method, implemented in an apparatus, for constructing a data model of a plurality of goods packages, the method comprising controlling a processor of the apparatus to form an association between a first candidate side of a candidate goods package extracted from a first image of a first side of the plurality of goods packages and a second candidate side of the candidate goods package extracted from a second image of a second side of the plurality of goods packages.

19. A method, implemented in an apparatus, for performing a determination of whether a candidate goods package is of a known type of goods package, the method comprising controlling a processor of the apparatus to operate on a plurality of indicator values to perform the determination, each of the plurality of indicator values representing a provisional indication of whether the candidate goods package is of the known type of goods package.

20. A machine-readable medium, having stored thereon machine-readable instructions for executing, in a machine, a method for constructing a data model of a plurality of goods packages, the method comprising controlling a processor of the apparatus to form an association between a first candidate side of a candidate goods package extracted from a first image of a first side of the plurality of goods packages and a second candidate side of the candidate goods package extracted from a second image of a second side of the plurality of goods packages.

21. A machine-readable medium, having stored thereon machine-readable instructions for executing, in a machine, a method, implemented in an apparatus, for performing a determination of whether a candidate goods package is of a known type of goods package, the method comprising controlling a processor of the apparatus to operate on a plurality of indicator values to perform the determination, each of the plurality of indicator values representing a provisional indication of whether the candidate goods package is of the known type of goods package.