AU2017279676A1 - Method, system and apparatus for comparing objects in images - Google Patents

Abstract

METHOD, SYSTEM AND APPARATUS FOR COMPARING OBJECTS IN IMAGES A method of comparing objects in images. A dictionary determined from a first set of images is received, the dictionary being determined using correspondences of codes of a first set of codes 5 based on pairwise similarities between images in the first set of images, wherein the dictionary is applied to feature vectors generated from the first set of images to generate the first set of codes. A second set of codes is determined for a second set of images using the received dictionary. An updated dictionary is generated using the first and second set of codes when a difference between codes of the first and second set of images exceeds a predetermined 10 threshold. Objects in a first and second image are compared using the updated dictionary. 14063804v1

Description

METHOD, SYSTEM AND APPARATUS FOR COMPARING OBJECTS IN IMAGES

TECHNICAL FIELD

The present invention relates generally to image processing and, in particular, to matching objects between two captured images to determine whether a candidate object is an object of interest. The present invention also relates to a method, apparatus and system for comparing objects in images, and to a computer program product including a computer readable medium having recorded thereon a computer program for comparing objects in images.

BACKGROUND

Public venues such as shopping centres, parking lots and train stations are increasingly subject to surveillance using large-scale networks of video cameras. Application domains of large-scale video surveillance include security, safety, traffic management and business analytics. In one example application from the security domain, a security officer may want to view a video feed containing a particular suspicious person in order to identify undesirable activities. In another example from the business analytics domain, a shopping centre may wish to track customers across multiple cameras in order to build a profile of shopping habits.

Many surveillance applications require methods, known as “video analytics”, to detect, track, match and analyse multiple objects of interest across multiple camera views. In one example, referred to as a “hand-off’ application, object matching is used to persistently track multiple objects across first and second cameras with overlapping fields of view. In another example application, referred to as “re-identification”, object matching is used to locate a specific object of interest across multiple cameras in the network with non-overlapping fields of view.

Cameras at different locations may have different viewing angles and work under different lighting conditions, such as indoor and outdoor. The different viewing angles and lighting conditions may cause the visual appearance of a person to change significantly between different camera views. In addition, a person may appear in a different orientation in different camera views, such as facing towards or away from the camera, depending on the placement of the camera relative to the flow of pedestrian traffic. Robust person matching in the presence of appearance change due to camera viewing angle, lighting and person orientation is difficult.

14063804vl

2017279676 20 Dec 2017

In most person matching methods, the appearance of a person is represented by a “descriptor”, also referred to as an “appearance descriptor” or “feature vector”. A descriptor is a derived value or set of derived values determined from the pixel values in an image of a person. One example of a descriptor is a histogram of colour values. Another example of a descriptor is a histogram of quantized image gradient responses.

In some known methods for person matching, known as “supervised learning”, a projection is learned from pairs of images of people captured from a pair of cameras. In each pair of images, the first image is captured from the first camera and the second image is captured from the second camera. Pairs of images of the same person are known as “positive” training images.

Pairs of images of different people are known as “negative” training images. Pairs of appearance descriptors extracted from positive training images are known as “positive” training samples. Pairs of appearance descriptors extracted from negative training images are known as “negative” training samples.

The projection is learned with information related to whether the image pairs are positive or negative training samples. In one known method, known as “distance metric learning”, a projection is learned to minimize a distance between the appearance descriptors in each positive training sample and maximize the distance between the appearance descriptors in each negative training sample. In another method, known as “linear discriminative analysis”, a set of projections are learned to separate appearance descriptors associated with different positive training samples in a common subspace. In another method, known as “canonical correlation analysis”, a set of projections are learned to maximize the correlation between the appearance descriptors in each positive training sample in a common subspace.

Supervised learning methods may be impractical due to the need for positive training images. In practice, generating a set of positive training images is time consuming and requires intense manual labour. Furthermore, people may appear infrequently in some camera views, such as remote perimeters, making the collection of a large set of positive training images impractical. Therefore, methods, known as “unsupervised learning”, resort to learning a discriminative representation of appearance descriptors without the need to capture large quantities of positive training images in every pair of cameras.

In some known unsupervised methods for person matching, known as “dictionary teaming”, a “dictionary” is learned to encode a compact, discriminative representation of an appearance descriptor. A dictionary consists of a set of dictionary “atoms” or basis vectors. An appearance

14063804vl

2017279676 20 Dec 2017 descriptor of a person can be reconstructed as a linear weighted sum of dictionary atoms, each atom being weighted by a coefficient. The coefficients for all dictionary atoms collectively form a “code”. Given an appearance descriptor, the corresponding code is determined by determining the weighted sum of dictionary atoms that minimizes a difference, known as a “reconstruction error”, between the appearance descriptor and a reconstruction of the appearance descriptor using the dictionary atoms. A dissimilarity score (e.g., Euclidean distance), between the codes of a pair of images determines if the pair of image is matched.

Dictionary learning methods for person re-identification usually use the entire training dataset for training. The computational cost of learning a dictionary is very high if the number of training samples in the training dataset is large. In addition, the need for memory storage in the learning process is high. In practice, a dictionary is often required to be updated when there is a change in the capturing environment such as time-of-day, seasonal changes, and dynamics of a scene. In one known method of dictionary learning, known as “online dictionary learning”, each dictionary atom is sequentially updated using block coordinate descent with warm restarts. The online dictionary learning method can handle a large scale training dataset. However, the online dictionary learning method fails to work for person re-identification as the online dictionary learning method ignores the similarities between images received from different cameras.

SUMMARY

It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

Disclosed are arrangements relating to comparing objects in images captured from a pair of cameras, based on unsupervised dictionary learning using images of objects.

According to one aspect of the present disclosure, there is provided a method of comparing objects in images, the method comprising:

receiving a dictionary determined from a first set of images, the dictionary being determined using correspondences of codes of a first set of codes based on pairwise similarities between images in the first set of images, wherein the dictionary is applied to feature vectors generated from the first set of images to generate the first set of codes;

14063804vl

2017279676 20 Dec 2017 determining a second set of codes for a second set of images using the received dictionary;

generating an updated dictionary using the first and second set of codes when a difference between codes of the first and second set of images exceeds a predetermined threshold; and comparing objects in a first and second image using the updated dictionary.

According to another aspect of the present disclosure there is provided, an apparatus for comparing objects in images, the apparatus comprising:

means for receiving a dictionary determined from a first set of images, the dictionary being determined using correspondences of codes of a first set of codes based on pairwise similarities between images in the first set of images, wherein the dictionary is applied to feature vectors generated from the first set of images to generate the first set of codes;

means for determining a second set of codes for a second set of images using the received dictionary;

means for generating an updated dictionary using the first and second set of codes based on a difference between codes of the first and second set of codes exceeding a predetermined threshold; and comparing objects in a first and second image using the updated dictionary.

According to still another aspect of the present disclosure there is provided, a system for comparing objects in images, the system comprising:

a memory for storing data and a computer program; and a processor coupled to the memory for executing the computer program, the computer program having instructions for:

receiving a dictionary determined from a first set of images, the dictionary being determined using correspondences of codes of a first set of codes based on pairwise

14063804vl

2017279676 20 Dec 2017 similarities between images in the first set of images, wherein the dictionary is applied to feature vectors generated from the first set of images to generate the first set of codes;

determining a second set of codes for a second set of images using the received dictionary;

generating an updated dictionary using the first and second set of codes based on a difference between codes of the first and second set of codes exceeding a predetermined threshold; and comparing objects in a first and second image using the updated dictionary.

According to still another aspect of the present disclosure there is provided, a non-transitory computer readable medium having a computer program stored on the medium for comparing objects in images, the program comprising:

code for receiving a dictionary determined from a first set of images, the dictionary being determined using correspondences of codes of a first set of codes based on pairwise similarities between images in the first set of images, wherein the dictionary is applied to feature vectors generated from the first set of images to generate the first set of codes;

code for determining a second set of codes for a second set of images using the received dictionary;

code for generating an updated dictionary using the first and second set of codes based on a difference between codes of the first and second set of codes exceeding a predetermined threshold; and comparing objects in a first and second image using the updated dictionary.

Other aspects are also disclosed.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more example embodiments of the invention will now be described with reference to the 25 following drawings, in which:

14063804vl

2017279676 20 Dec 2017

Fig. 1 shows a surveillance system comprising cameras connected to a computer system and a cloud server for analysing objects of interest, to which arrangements described may be applied;

Figs. 2A and 2B form a schematic block diagram of the computer system of Fig. 1 upon which the described arrangements can be practiced;

Fig. 3 is a schematic flow diagram showing a method of comparing objects between a query image and a gallery image using a learned dictionary;

Fig. 4 is a schematic flow diagram of a method of collecting a training dataset and determining a dictionary based on the collected training dataset as used in the method of Fig. 3;

Fig. 5 is a schematic flow diagram of a method of determining an appearance descriptor of an 10 object as used in the method of Fig. 3 and the method of Fig. 4;

Fig. 6 is a schematic flow diagram of a method of determining an initial dictionary using appearance descriptors as used in the method of Fig. 4;

Fig. 7 is a schematic flow diagram of a method of learning a dictionary as used in the methods of Figs. 6 and 8; and

Fig. 8 is a schematic flow diagram of a method of updating a dictionary using a previous dictionary and appearance descriptors as used in the method of Fig. 4.

DESCRIPTION OF THE EMBODIMENTS

Where reference is made in any one or more of the accompanying drawings to steps and/or features, which have the same reference numerals, those steps and/or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears.

It is to be noted that the discussions contained in the “Background” section and the section above relating to prior art arrangements relate to discussions of documents or devices which may form public knowledge through their respective publication and/or use. Such discussions should not be interpreted as a representation by the present inventors or the patent applicant that such documents or devices in any way form part of the common general knowledge in the art.

14063804vl

2017279676 20 Dec 2017

An image, such as an image 110 in Fig. 1, is made up of visual elements. The image 110 is captured using an image capture device such as a digital camera or video camera. The term “pixel”, also referred to as a “pixel location” or “image location”, refers in the present disclosure to one of the visual elements in a captured image. Each pixel of an image is described by one or more values characterising a property of a scene captured in the image. In one example, a single intensity value characterises a brightness of the scene at a pixel location. In another example, a triplet of values characterise a colour of the scene at the pixel location.

A “region”, also referred to as an “image region”, in an image refers to a collection of one or more spatially adjacent visual elements. The term “bounding box” refers to a rectilinear image region enclosing an object or part of an object in an image. In one example, bounding box 131 in Fig. 1 encloses a person 130. A “feature”, also referred to as an “appearance descriptor” or “descriptor”, represents a derived value or set of derived values determined from the pixel values in an image region. One example of an appearance descriptor is a histogram of colour values in an image region. Another example of an appearance descriptor is a histogram of quantized image gradient responses in a region.

The term “foreground mask” refers to a binary image with non-zero values at pixel locations corresponding to an object of interest. In one example, the terms “candidate object” and “object of interest” refer to a person in a crowd that has been identified as being of particular interest.

A non-zero pixel location in a foreground mask is known as a “foreground pixel”. In one arrangement, a foreground mask is determined using a statistical background pixel modelling method such as Mixture of Gaussian (MoG), wherein the background model is maintained over multiple frames with a static camera. In another arrangement, foreground detection is performed on Discrete Cosine Transform blocks. In yet another arrangement, a foreground mask is determined using unsupervised segmentation, for example, using superpixels. Any suitable method for determining a foreground mask may equally be practised.

Methods for comparing objects in images captured from a first pair of cameras, based on dictionary learning using images of objects, are described below.

Fig. 1 shows an example surveillance system 100 upon which arrangements described can be practiced. In one example, the described methods may be used to determine whether an object observed in an image 120 of a first scene captured by a first digital camera 125 has the same identity as a second object detected in an image 110 of a second scene captured by a second digital camera 115. In the arrangements described, the terms “object”, “person” and “target”

14063804vl

2017279676 20 Dec 2017 relate to “an object of interest” within at least partial view of one of the cameras 125 and 115. Disclosed arrangements may equally be applied when the images are captured with different environmental factors. For example, the images may be captured by different cameras simultaneously or at different times, or captured by the same camera at different times, including images that represent the same scene or different scenes.

The cameras 115 and 125 may be any type of image capture device suitable for capturing an image of a scene using a sensor such as an optical sensor, an infrared sensor, a radar sensor, and the like or be multi-sensor devices. The images used for matching objects are captured by the same type of sensor. The cameras 115 and 125 may be a digital camera, for example. The cameras 115 and 125 are connected to a computer system 150 as seen in Figs. 2A and 2B. The computer system 150 is connected to a cloud server 160, via a communications network 220, to which arrangements described may be applied. The cameras 115 and 125 may be in wired or wireless communication with the computer system 150, and the computer system 150 may be in wired or wireless communication with the cloud server 160.

The terms “re-identification”, “hand-off’ and “matching” relate to the task of relating an object of interest within at least partial view of a video camera (e.g., the camera 115) to another object within at least a partial view of the same or another video camera (e.g., the camera 125).

For the example in Fig. 1, a first object is known as a “query” object, and a second object is known as a “gallery” object. A query object may be a person who has been identified as being suspicious at an airport. A gallery object is a person in a crowded airport, where the person is merely a member of the crowd. The gallery object may also be referred to as a “candidate object”. In one example, a person 132 in an image 120 is a query object, and three people 130, 134 and 136 in an image 110 are gallery objects. The arrangements described may be applied to determine which of the three gallery objects 130, 134 and 136 is a best match for the query object 132.

The example of Fig. 1 may be applied to a range of applications. In one example, the computer system 150 may be configured to allow a security guard to select a query object through an interactive user interface, and return images of one or more gallery objects determined to match the query object. In another example, the computer system 150 may be configured to automatically select a query object and match the object across multiple distributed cameras in order to analyse the long-term behaviour of the object. In other arrangements, the computer system 150 is integral to one of the cameras 115 and 125.

14063804vl

2017279676 20 Dec 2017

While the described examples generally relate to surveillance tasks of monitoring persons, the described methods may equally be practised on other types of objects. For example, the described methods may be applied to track a suspicious vehicle. The described methods may also be implemented using different types of sensors including near infra-red (IR) cameras, radar sensors, and laser scanners.

For the example in Fig. 1, the query and gallery objects may be matched by extracting an appearance descriptor for the query object from a bounding box 133, and appearance descriptors for the gallery objects 130, 134 and 136 from the bounding boxes 131, 135 and 137 respectively. A similarity score may be determined between the query object and each gallery object based on the appearance descriptors. The gallery object with the greatest similarity to the query object is determined to have the same identity as the query object.

The similarity score is in part based on a learned representation of object appearance determined using dictionary learning. The learned representation, represented by atoms in the dictionary, is derived from a set of training images of objects of the same type as the query and gallery object without knowledge of the identity of the object in each image. The training images originate from a training data set, collected from the query and gallery cameras. For the example shown in Fig. 1, the training data 170 is collected from cameras 115 and 125 during a training phase of operation, and communicated from the computer system 150, via a communications network 220, to cloud server 160. A dictionary is learned on the cloud server

160 based on the training data 170, and transmitted back to the computer system 150.

Figs. 2A and 2B depict the computer system 150, upon which the arrangements described can be practiced.

As seen in Fig. 2A, the computer system 150 includes: a computer module 201; input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, one or more cameras such as the cameras 115 and 125, and a microphone 280; and output devices including a printer 215, a display device 214 and loudspeakers 217. An external Modulator-Demodulator (Modem) transceiver device 216 may be used by the computer module 201 for communicating to and from remote cameras such as the cameras 100 and 105 over the communications network 220 via a connection 221. The communications network 220 may be a wide-area network (WAN), such as the Internet, a cellular telecommunications network, or a private WAN. Where the connection 221 is a telephone line, the modem 216 may be a traditional “dial-up” modem. Alternatively, where the connection 221 is a high capacity (e.g., cable) connection, the

14063804vl

2017279676 20 Dec 2017 modem 216 may be a broadband modem. A wireless modem may also be used for wireless connection to the communications network 220.

The cameras 115 and 125 may be remote cameras forming part of a camera network. In the arrangements described, the cameras 115 and 125 relate to video cameras. The camera network may comprise any type of image capture devices capable of capturing image and/or video data from which vanishing points can be determined. The camera network may comprise a plurality of a single type of image capture device. In other arrangements, image capture devices integral or proximate to the computer module 201, can form part of the camera network.

The computer module 201 typically includes at least one processor unit 205, and a memory unit 206. For example, the memory unit 206 may have semiconductor random access memory (RAM) and semiconductor read only memory (ROM). The computer module 201 also includes an number of input/output (I/O) interfaces including: an audio-video interface 207 that couples to the video display 214, loudspeakers 217 and microphone 280; an FO interface 213 that couples to the keyboard 202, mouse 203, scanner 226, camera 116 and optionally a joystick or other human interface device (not illustrated); and an interface 208 for the external modem 216 and printer 215. In some implementations, the modem 216 may be incorporated within the computer module 201, for example within the interface 208. The computer module 201 also has a local network interface 211, which permits coupling of the computer system 150 via a connection 223 to a local-area communications network 222, known as a Focal Area Network (FAN). As illustrated in Fig. 2A, the local communications network 222 may also couple to the wide network 220 via a connection 224, which would typically include a so-called “firewall” device or device of similar functionality. The local network interface 211 may comprise an Ethernet circuit card, a Bluetooth® wireless arrangement or an IEEE 802.11 wireless arrangement; however, numerous other types of interfaces may be practiced for the interface 211.

The I/O interfaces 208 and 213 may afford either or both of serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standards and having corresponding USB connectors (not illustrated). Storage devices 209 are provided and typically include a hard disk drive (HDD) 210. Other storage devices such as a floppy disk drive and a magnetic tape drive (not illustrated) may also be used. An optical disk drive 212 is typically provided to act as a non-volatile source of data. Portable memory devices, such optical disks (e.g., CD-ROM, DVD, Blu-ray DiscTM), USB-RAM, portable, external hard

14063804vl

2017279676 20 Dec 2017 drives, and floppy disks, for example, may be used as appropriate sources of data to the system 150.

The components 205 to 213 of the computer module 201 typically communicate via an interconnected bus 204 and in a manner that results in a conventional mode of operation of the computer system 150 known to those in the relevant art. For example, the processor 205 is coupled to the system bus 204 using a connection 218. Likewise, the memory 206 and optical disk drive 212 are coupled to the system bus 204 by connections 219. Examples of computers on which the described arrangements can be practised include IBM-PC’s and compatibles, Sun Sparcstations, Apple MacTM or a like computer systems.

The methods described may be implemented using the computer system 150 wherein the processes of Figs. 3, 4, 5, 6, 7 and 8, to be described, may be implemented as one or more software application programs 233 executable within the computer system 150. In particular, the steps of the methods described are effected by instructions 231 (in Fig. 2B) in the software 233 that are carried out within the computer system 150. The software instructions 231 may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the corresponding code modules performs the methods described and a second part and the corresponding code modules manage a user interface between the first part and the user.

The software 233 may be stored in a computer readable medium, including the storage devices described below, for example. The software 233 is typically stored in the HDD 210 or the memory 206. The software is loaded into the computer system 150 from the computer readable medium, and then executed by the computer system 150. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., CD-ROM) 225 that is read by the optical disk drive 212. A computer readable medium having such software or computer program recorded on the computer readable medium is a computer program product. The use of the computer program product in the computer system 150 preferably effects an advantageous apparatus for implementing the methods described.

In some instances, the application programs 233 may be supplied to the user encoded on one or more CD-ROMs 225 and read via the corresponding drive 212, or alternatively may be read by the user from the networks 220 or 222. Still further, the software can also be loaded into the computer system 150 from other computer readable media. Computer readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and/or

14063804vl

2017279676 20 Dec 2017 data to the computer system 150 for execution and/or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, DVD, Blu-rayTM Disc, a hard disk drive, a ROM or integrated circuit, USB memory, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computer module 201. Examples of transitory or non-tangible computer readable transmission media that may also participate in the provision of software, application programs, instructions and/or data to the computer module 201 include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like.

The second part of the application programs 233 and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented upon the display 214. Through manipulation of typically the keyboard 202 and the mouse 203, a user of the computer system 150 and the application may manipulate the interface in a functionally adaptable manner to provide controlling commands and/or input to the applications associated with the GUI(s). Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface utilizing speech prompts output via the loudspeakers 217 and user voice commands input via the microphone 280.

Tig. 2B is a detailed schematic block diagram of the processor 205 and a “memory” 234. The memory 234 represents a logical aggregation of all the memory modules (including the

HDD 209 and semiconductor memory 206) that can be accessed by the computer module 201 in Tig. 2A.

When the computer module 201 is initially powered up, a power-on self-test (POST) program 250 executes. The POST program 250 is typically stored in a ROM 249 of the semiconductor memory 206 of Tig. 2A. A hardware device such as the ROM 249 storing software is sometimes referred to as firmware. The POST program 250 examines hardware within the computer module 201 to ensure proper functioning and typically checks the processor 205, the memory 234 (209, 206), and a basic input-output systems software (BIOS) module 251, also typically stored in the ROM 249, for correct operation. Once the POST program 250 has run successfully, the BIOS 251 activates the hard disk drive 210 of Tig. 2A. Activation of the hard disk drive 210 causes a bootstrap loader program 252 that is resident on the hard disk drive 210 to execute via the processor 205. This loads an operating system 253

14063804vl

2017279676 20 Dec 2017 into the RAM memory 206, upon which the operating system 253 commences operation. The operating system 253 is a system level application, executable by the processor 205, to fulfil various high level functions, including processor management, memory management, device management, storage management, software application interface, and generic user interface.

The operating system 253 manages the memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has sufficient memory in which to execute without colliding with memory allocated to another process. Furthermore, the different types of memory available in the system 150 of Fig. 2A must be used properly so that each process can run effectively. Accordingly, the aggregated memory 234 is not intended to illustrate how particular segments of memory are allocated (unless otherwise stated), but rather to provide a general view of the memory accessible by the computer system 150 and how such is used.

As shown in Fig. 2B, the processor 205 includes a number of functional modules including a control unit 239, an arithmetic logic unit (AFU) 240, and a local or internal memory 248, sometimes called a cache memory. The cache memory 248 typically includes a number of storage registers 244 - 246 in a register section. One or more internal busses 241 functionally interconnect these functional modules. The processor 205 typically also has one or more interfaces 242 for communicating with external devices via the system bus 204, using the connection 218. The memory 234 is coupled to the bus 204 using the connection 219.

The application program 233 includes the sequence of instructions 231 that may include conditional branch and loop instructions. The program 233 may also include data 232 which is used in execution of the program 233. The instructions 231 and the data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending upon the relative size of the instructions 231 and the memory locations 228-230, a particular instruction may be stored in a single memory location as depicted by the instruction shown in the memory location 230. Alternately, an instruction may be segmented into a number of parts each of which is stored in a separate memory location, as depicted by the instruction segments shown in the memory locations 228 and 229.

In general, the processor 205 is given a set of instructions which are executed therein. The processor 205 waits for a subsequent input, to which the processor 205 reacts to by executing another set of instructions. Each input may be provided from one or more of a number of sources, including data generated by one or more of the input devices 202, 203, data received from an external source across one of the networks 220, 202, data retrieved from one of the

14063804vl

2017279676 20 Dec 2017 storage devices 206, 209 or data retrieved from a storage medium 225 inserted into the corresponding reader 212, all depicted in Fig. 2A. The execution of a set of the instructions may in some cases result in output of data. Execution may also involve storing data or variables to the memory 234.

The arrangements described use input variables 254, which are stored in the memory 234 in corresponding memory locations 255, 256, 257. The arrangements described produce output variables 261, which are stored in the memory 234 in corresponding memory locations 262, 263, 264. Intermediate variables 258 may be stored in memory locations 259, 260, 266 and 267.

Referring to the processor 205 of Fig. 2B, the registers 244, 245, 246, the arithmetic logic unit (AFU) 240, and the control unit 239 work together to perform sequences of micro-operations needed to perform “fetch, decode, and execute” cycles for every instruction in the instruction set making up the program 233. Each fetch, decode, and execute cycle comprises:

• a fetch operation, which fetches or reads an instruction 231 from a memory location 228, 229, 230;

• a decode operation in which the control unit 239 determines which instruction has been fetched; and • an execute operation in which the control unit 239 and/or the AFU 240 execute the instruction.

Thereafter, a further fetch, decode, and execute cycle for the next instruction may be executed. Similarly, a store cycle may be performed by which the control unit 239 stores or writes a value to a memory location 232.

Each step or sub-process in the processes of Figs. 4, 5, 6, 7 and 8 is associated with one or more segments of the program 233 and is performed by the register section 244, 245, 247, the

AFU 240, and the control unit 239 in the processor 205 working together to perform the fetch, decode, and execute cycles for every instruction in the instruction set for the noted segments of the program 233.

14063804vl

2017279676 20 Dec 2017

The methods described may alternatively be implemented in dedicated hardware such as one or more integrated circuits performing the functions or sub functions. Such dedicated hardware may include graphic processors, digital signal processors, or one or more microprocessors and associated memories, and may reside on platforms such as video cameras.

Fig. 3 shows a method 300 of comparing objects between images by comparing codes determined using dictionary learning. The method 300 may be used to match the objects between images based on the comparison.

In one example, the method 300 is used to determine a gallery object in an image matched to the query object. The method 300 may be implemented by one or more software code modules of the application 233, stored in the hard disk drive 210 and being controlled under execution of the processor 205. In some arrangements, portions of the method 300 are executed on a processor of a cloud server computer, such as the cloud server 160.

The method 300 is described by way of example with reference to the query image 120 containing the object of interest 132 detected at the bounding box 133, and the gallery image

110 containing candidate objects 130, 134 and 136, detected at the bounding boxes 131, 135 and 137. In the example described, the method 300 is used to determine which of the detections 131, 135 and 137 is the object of interest 132, detected at 133. The following description provides details, examples and alternative implementations for the main steps of the method 300. Further details, examples and alternative implementations of step 305, 340 and

345 are described below.

The method 300 starts at a determining step 305. At step 305, a dictionary is determined (or “learned”), under execution of the processor 205, based on a training dataset collected from the query 125 and gallery 115 cameras. In some arrangements, the training images are processed on the computer system 150 to extract appearance descriptors, which are then transmitted to the cloud server 160, via the network 220, to learn a dictionary. In other arrangements, the training images are transmitted to the cloud server 160 for processing. In yet other arrangements, image processing and dictionary learning are processed locally on the computer system 150. A method 400 of collecting training images from query 125 and gallery 115 cameras and determining (or “learning”) a dictionary incrementally from the collected images, as executed at step 305 of method 300, is described below with reference to Fig. 4. The output of step 305 is a dictionary for determining codes. As described in detail below, the codes are determined by applying the

14063804vl

2017279676 20 Dec 2017 dictionary to appearance descriptors determined for the images captured by the query 125 and gallery 115 cameras. The dictionary may be stored in the memory 206.

The method 300 also starts at a receiving step 310. At execution of the step 310, at least one image containing a query object is received as input. For example, the image 120 is a query image received from a query camera 125 containing a query object 132. The image 120 may be stored in the memory 206. The method 300 progresses under execution of the processor 205 from the receiving step 310 to a detecting step 320. At the detecting step 320, a query object is detected from the received query images. One example of detecting the query object uses a pedestrian detection method to detect all persons in the query images. One pedestrian detection method learns a detector to search for persons within an image by scanning pixel locations. The detector produces a high score if the local image features inside the local search window meet certain criteria. The local image feature may be the histogram of oriented gradients or local binary pattern. Other pedestrian detection methods include a part-based detection method and a background subtraction method. The output of the pedestrian detection method is a set of bounding boxes. The image region defined by each bounding box contains a person.

The method 300 progresses under execution of the processor 205 from the detecting step 320 to a selecting step 330. In one arrangement, a user such as a security guard manually selects an automatically detected bounding box, such as the bounding box 133, as the query object via a graphical user interface executing on the module 201. In another arrangement, the user manually draws a bounding box containing an object to define the query object via a graphical user interface executing on the module 201. In yet another arrangement, an algorithm executing on the module 201 automatically selects an automatically detected bounding box, such as the bounding box 133, as the query object based on pre-defined rules. The output of step 330 is an image region within a bounding box for the query object.

The method 300 progresses under execution of the processor 205 from the step 330 to a determining step 340. An appearance descriptor for the query object is determined at step 340 based on pixels in the image region determined at step 330. A method 500 of determining an appearance descriptor of an object, as executed at step 340, will be described below with reference to Fig. 5. The output of step 340 is an appearance descriptor characterizing the query object.

The method 300 progresses under execution of the processor 205 from step 340 to a determining step 350, where a code for the query object is determined by applying the learned

14063804vl

2017279676 20 Dec 2017 dictionary determined at step 305 to the appearance descriptor determined at step 340. In one arrangement, the code a¹ for an appearance descriptor x determined at step 340 is determined by solving the Lasso problem shown in Equation (1) using a sparse coding algorithm (e.g., least angle regression algorithm).

a' — argminj|x — Da||₂ + zl|] or||i (1) where || · ||₂ denotes the 2-norm of a vector, ΙΗ^ denotes the 1-norm of a vector, D represents the learned dictionary, x represents the appearance descriptor, a represents the code, and Λ represents a regularisation factor. The regularisation factor may be pre-defined (e.g., λ = 0.01), or may be determined during the “training” phase using known methods such as cross10 validation. In another arrangement, the code a' for an appearance descriptor x is determined by solving a regularised least squares problem to determine the closed-form solution of Equation (2), as follows:

a' = (D^TD + pI')~¹D^Tx (2) where D^T represents the transpose of the dictionary matrix, I is an identity matrix, and p is a regularisation factor. In one arrangement, the regularization factor is pre-defined (e.g., p =

0.5). In another arrangement, the regularization factor is determined during the “training” phase using any suitable method such as cross-validation. The output of step 350 is a code corresponding to the query object selected at step 330. The code may be stored in the memory 206.

As seen in Fig. 3, the method 300 also starts at a receiving step 315. In one arrangement, steps 305, 310 and 315 start concurrently. In another arrangement, steps 305, 310 and 315 are executed at different times. In all arrangements, step 305 is executed before steps 350 and 355. In some arrangements, step 305 is executed during an initial installation or “training” phase, while the remaining steps in method 300 are executed during routine operation after the “training” phase has finished. Furthermore, steps 350 and 355 are executed before step 360 in all arrangements.

At execution of step 315, at least one image containing gallery objects is received as input. For example, the image 110 is a gallery image received from a gallery camera 115 containing gallery objects 132, 134 and 136. The method 300 progresses under execution of the processor

205 from step 315 to a detecting step 325. At step 325, a set of gallery objects is detected in the received gallery images. In one arrangement, step 325 is implemented for gallery objects in a

14063804vl

2017279676 20 Dec 2017 similar manner to step 320 for query objects. The output of step 325 is a set of bounding boxes, such as the bounding boxes 131, 135 and 137 corresponding to the gallery objects 130, 134 and 136.

The method 300 progresses under execution of the processor 205 from step 325 to a selecting step 335. At the selecting step 335, a gallery object is selected for comparing with the query object determined at step 330. In one arrangement, the gallery objects determined at detecting step 325 are stored in a list, for example in the memory 206, and a gallery object is selected by enumerating the objects in the list. In other arrangements, step 335 is implemented for gallery objects in a similar manner to step 330 for query objects. The output of the selecting step 335 is an image region within a bounding box for the gallery object. The image region output at step 335 may be stored in the memory 206.

The method 300 progresses under execution of the processor 205 from the step 335 to a determining step 345. An appearance descriptor for the gallery object is determined at step 345 based on pixels in the image region determined at step 335. Further details, examples and alternative implementations of the step 345 are described below with reference to Fig. 5. The output of step 345 is an appearance descriptor characterizing the gallery object. Again, the appearance descriptor output at step 345 may be stored in the memory 206.

The method 300 progresses under execution of the processor 205 from step 345 to a determining step 355, where a code for the gallery object is determined. In one arrangement, step 355 is implemented for the gallery object in a similar manner to step 350 for the query object. The output of step 355 is a code corresponding to the gallery object selected at step 335. The code output at step 355 may be stored in the memory 206.

After execution of steps 350 and 355, the method 300 progresses under execution of the processor 205 to a comparing step 360. At the comparing step 360, the code of the selected query object determined at step 350 and the code of the selected gallery object determined at step 355 are compared to determine whether the codes correspond indicating that the objects match (i.e., the objects) have the same identity. In one arrangement, a comparison is performed at step 360 by determining correspondences between the codes based a similarity or dissimilarity score between the codes. One example of a dissimilarity score is a Euclidean distance between the codes. One example of a similarity score is a Cosine distance between the codes. Any other suitable similarity or dissimilarity scores may be determined to compare two codes.

14063804vl

2017279676 20 Dec 2017

One example of determining whether the two objects match (i.e., have the same identity) is to compare a similarity score to a fixed threshold. If a similarity score exceeds the threshold, the query and gallery object are determined to match.

Another example of determining whether the query object matches the gallery object (i.e., the query object has the same identity as the gallery object) is to compare the similarity score to the similarity score between the query object and all other objects in the video stream. If the similarity score for the gallery object is greater than all other objects, then the query and gallery object are determined to have the same identity.

The computer system 150 generates a response if the two objects are determined match. In one example, the match is communicated to a user through a graphical user interface, for example, displayed on the display 214. In another example, the response is to tag the gallery object for further analysis, such as tracking the gallery object through the field of view of the gallery camera 115.

The method 300 concludes after completing the matching step 360.

The method 400 of collecting training images from query 125 and gallery 115 cameras and determining (or “learning”) a dictionary from the collected images, as executed at step 305 of method 300, is now described with reference to Fig. 4. The method 400 may be implemented at least in part as one or more software code modules of the software application program 233 resident in the hard disk drive 210 and being controlled in execution by the processor 205. The method 400 may in some arrangements be executed at least in part on a processor of the cloud server 160.

The method 400 starts at a collecting step 410. In execution of the step 410, query and gallery images containing a plurality of objects are collected from two or more (query and gallery) cameras installed at a customer site. For the example application in Fig. 1, cameras 115 and 125 are examples of cameras installed at a customer site, and images 110 and 120 are examples of images containing a plurality of objects. The images collected at step 410 may be stored in the memory 206.

The method 400 passes under execution of the processor 205 from step 410 to a detecting step 420. At step 420, a set of objects is detected in the images from the two or more cameras 115 and 125 collected at step 410. In one arrangement, at step 420, objects in the images are

14063804vl

2017279676 20 Dec 2017 detected in a similar manner to step 320 for detecting objects in query images. The output of step 420 is a set of bounding boxes, such as the bounding boxes 131, 133, 135 and 137 corresponding to the objects 130, 132, 134 and 136 in the images 110 and 120. The bounding boxes determined at step 420 may be stored in the memory 206.

The method 400 passes under execution of the processor 205 from step 420 to generating step 425, where pairs of training objects, each object in each pair being detected in a different camera, are generated based on the objects detected at step 420. In one arrangement, a first camera with the minimum number of detected objects over all cameras is determined, and a commensurate set of objects are randomly selected from a second camera. In another arrangement, a first camera with a maximum number of detected objects over all cameras is determined. Then, the bounding boxes corresponding to detected objects in a second camera are randomly configured to generate new pseudo-detections, such that the total number of detections in the second camera matches the number of detections in the first camera.

The method 400 progresses under execution of the processor 205 from the step 425 to a determining step 430. An appearance descriptor for each detected object is determined at step 430 based on pixels in the bounding box corresponding to the detected object, as determined at step 420. The appearance descriptor is determined in accordance with the method 500 which will be described below with reference to Fig. 5. The output of step 430 is a set of appearance descriptors describing the detected objects in the images. The set of appearance descriptors determined at step 430 is also known as the training data which may be stored in the memory 206.

The method 400 progresses under execution of the processor 205 from step 430 to a determining step 440. At step 440, the training data is used to determine (or learn) an initial dictionary. A method 600 of determining a dictionary using the training data, as executed at step 440, will be described below with reference to Fig. 6. The output of step 440 is an initial dictionary for determining codes of appearance descriptors from detected query and gallery objects.

After the initial dictionary is determined at step 440, the method 400 works iteratively using newly collected training data to update the initial dictionary determined at step 440. Each iteration starts from collecting step 450 and proceeds to a detecting step 460, a determining step 470, an updating step 480, and back to the collecting step 450. The appearance descriptors used

14063804vl

2017279676 20 Dec 2017 at the determining step 440 are labelled as previous appearance descriptors and the previous appearance descriptors will be used in the updating step 480.

As seen in Fig. 4, the method 400 proceeds, under execution of the processor 205, from step 440 to a collecting step 450 where one or more new query and gallery images containing a plurality of objects are collected from the query and gallery cameras 115 and 125, respectively. In one arrangement, the new images are collected in a similar manner to step 410. The new images collected at step 450 may be stored in the memory 206.

The method 400 passes under execution of the processor 205 from the step 410 to a detecting step 460, where objects are detected from the query and gallery images collected at step 450 in a similar manner to step 420. The output of step 460 is a set of bounding boxes containing the detected objects in the query and gallery images. The determined bounding boxes may be stored in the memory 206.

The method 400 passes under execution of the processor 205 from the step 460 to a determining step 470. An appearance descriptor for each detected object is determined at step 470 based on pixels in the bounding box corresponding to the detected object, as determined at step 460. In one arrangement, the appearance descriptor is determined in a similar manner to step 430. The output of step 470 is a set of appearance descriptors describing the detected objects in the images.

The method 400 proceeds under execution of the processor 205 from the step 470 to a updating step 480, where the initial dictionary determined at step 440 is updated using the previous appearance descriptors determined at step 430, and the new appearance descriptors determined at step 470. A method 800 of updating the dictionary as executed at step 480, will be described below with reference to Fig. 8. The output of step 480 is an updated dictionary for determining codes of appearance descriptors from detected query and gallery objects. In one arrangement, the new appearance descriptors determined at step 470 are appended to the previous appearance descriptors determined at step 430 at the end of each iteration. The combined set of appearance descriptors are then used at a next iteration. In an alternative arrangement, the new appearance descriptors determined at step 470 are appended to new data in the next iteration if there are few objects detected at step 460 in the current iteration.

In one arrangement, the dictionary is learned on a customer server such as the computer system 150 in Fig. 1. In another arrangement, the dictionary is learned at a cloud server such as the

14063804vl

2017279676 20 Dec 2017 cloud server 160. In the arrangement where the dictionary is learned at the cloud server 160, the training data determined at steps 430 and 470 are transmitted via the network 220 from the customer server to the cloud server prior to the execution of steps 440 and 480 respectively. After execution of steps 440 and 480, the learned dictionary may be transmitted via the network

220 from the cloud server 160 back to the customer server.

The method 400 concludes after completing the determining step 480.

The method 500 of determining an appearance descriptor of an object, as executed at the steps 340 and 345 of the method 300 and steps 430 of the method 400, is now described with reference to Fig. 5. The method 500 may be implemented within the module 201 as one or more software code modules of the software application program 233 resident in the hard disk drive 210 and being controlled by execution of the processor 205.

The method 500 starts at a receiving step 505, in which an image or image sequence containing an object and a corresponding bounding box, such as the bounding box selected at step 330 or 335, are received as input. The image or image sequence received at step 505 may be stored in the memory 206. In one arrangement, the bounding box contains the whole body of a person.

In one example, when the method 500 is applied to the query object 132 shown in Fig. 1, the image 120 and the bounding box 133 are received as input in step 505. In another example, the bounding boxes containing head regions of persons are received.

The method 500 passes under execution of the processor 205 from the step 505 to a determining step 510. At step 510, a foreground confidence mask is determined under execution of the processor 205 and may be stored in the memory 206. The foreground confidence mask assigns to each pixel in the bounding box received at step 505 a value indicating a confidence that the pixel belongs to an object. In one arrangement, a foreground confidence mask is determined at step 505 by performing foreground separation using a statistical background pixel modelling method such as Mixture of Gaussian (MoG), where the background model is maintained over multiple frames with a static camera.

The method 500 passes under execution of the processor 205 from step 510 to a refining step 520. At step 520, the bounding box received at step 505 is refined to tightly bound the body of the person, based on the foreground confidence mask determined at step 510. In one arrangement, the bounding box for the head region received at step 505 is converted to a full body bounding box by only including the pixels with a foreground confidence value determined

14063804vl

2017279676 20 Dec 2017 at step 510 higher than a predetermined threshold and within a predetermined region of the image based on the head region. One example of a predetermined region is a region of equal width and upper boundary as the head region, and extending down to four (4) times the height of the head region. In another arrangement, the bounding box for the whole body received at step 505 is refined (by shrinking or expanding) to include the pixels with a foreground confidence value determined at the step 510 greater than a predetermined threshold and within a predetermined distance from the body region. An example of a predetermined distance is a five (5) pixel margin around the bounding box.

The method 500 passes under execution of the processor 205 from step 520 to a pre-processing step 530. In execution of the pre-processing step 530, the image region inside the bounding box determined at step 520 is pre-processed for feature determination. In one arrangement, a weighting scheme is used to weight every pixel of the image region inside the bounding box determined at step 520. One example of the weighting scheme uses a 2-D Gaussian function to weight the pixels based on the spatial locations. A pixel located close to the centre of the bounding box is assigned a higher weight than a pixel located farther from the centre of the bounding box. Another example of the weighting scheme uses the foreground confidence mask determining step 510 to weight the pixels based on the foreground confidence at the corresponding location. In another arrangement, the observed object in the bounding box determined at step 520 is rectified to a vertical orientation, which reduces a variation in the visual appearance of an object due to the viewpoint of the camera. In yet another arrangement, colour normalization is applied to the image inside the bounding box determined at step 520 to compensate for lighting variations across cameras.

The method 500 passes under execution of the processor 205 from step 530 to a determining step 540. At step 540, a plurality of feature channels for the pre-processed image generated in the step 530 are determined under execution of the processor 205. At each feature channel, each pixel in the image is assigned a feature value. In one arrangement, a feature channel is the red colour value at each pixel. In another arrangement, a feature channel is the green colour value at each pixel. In another arrangement, a feature channel is the blue colour value at each pixel. In another arrangement, a feature channel is a local binary pattern (LBP) at each pixel. In another arrangement, a feature channel is an image gradient magnitude at each pixel.

The method 500 passes under execution of the processor 205 from step 540 to a determining step 550. At step 550, the appearance descriptor, also referred to as a feature vector, is

14063804vl

2017279676 20 Dec 2017 determined from the feature channels determined at the step 540. The determined appearance descriptor may be stored in the memory 206 under execution of the processor 205.

In one arrangement, the appearance descriptor is determined at step 550 by dividing an image into regions and concatenating a spatial distribution of colour and texture features determined in each region. The colour feature component consists of colour histograms determined independently over a predetermined number of horizontal stripes (e.g., fifteen (15) horizontal stripes uniformly spaced from top to bottom of the image), based on the colour feature channels determined at step 540. The shape feature component is a “histogram of oriented gradients” (HOG) descriptors determined based on the image gradient feature channel determined at step

540. The texture feature component consists of histograms determined independently over predetermined regions (e.g. dividing the image according to a uniform grid), based on the LBP feature channel determined at step 540. The appearance descriptor is formed by concatenating the above components to form a single vector. In one arrangement, the descriptor is replaced with the square root of the values in the descriptor to reduce the effect of noise. In another arrangement, the appearance descriptor is determined at step 550 by encoding appearance as the difference between histograms across pairs of local regions.

In one arrangement, at step 550, the appearance descriptor is transformed by a subspace projection, in which the appearance descriptor is projected to a low-dimensional subspace. One example of a subspace projection is a Principal Component Analysis (PCA) projection. Another example of a subspace projection is a Locally-Linear Embedding (LLE).

In another arrangement, at step 550, the appearance descriptor is transformed by a nonlinear projection. In one example, a projection is obtained by performing a PC A in a reproducing kernel Hilbert space. In another example, a projection is obtained using a low rank approximation method (e.g., Nystrom approximation method). A set of representative appearance descriptors may be selected from the training dataset and then a PCA may be performed on the representative appearance descriptors to determine eigenvalues and eigenvectors. The projection of an appearance descriptor is determined using the eigenvalues and eigenvectors and the pairwise similarities between the appearance descriptor to be projected and the representative appearance descriptors.

The method 500 concludes after completing the determining step 550. An appearance descriptor is typically in the form of a vector and may also be referred to as a feature vector.

14063804vl

2017279676 20 Dec 2017

The steps 510 to 550 effectively operate to determine feature vectors based on properties of pixels in the received image or sequence of images.

The method 600 of determining a dictionary, as executed at step 440 of the method 400, will now be described with reference to Fig. 6.

The method 600 leams a dictionary by minimising an objective function comprised of a reconstruction error with code regulariser and manifold regulariser. Denote as Xq — {Xq and X_G = {XcCOlY the N appearance descriptors of the objects from the query camera and N appearance descriptors of the objects from the gallery camera, respectively, as determined at step 430 of method 400. The objective function is described in accordance with Equation (3), as follows:

min^HX - ZM||£ + ZR_C(A) + βΡ_Μ(Α) (3) where || · ||_F denotes the Frobenius norm of a matrix, D — [d(l), ..., d(K)] represents the dictionary comprised of K dictionary atoms, X — [Xq,X_g] represents input data comprised of 2N appearance descriptors extracted from training images, A — [ot(l), ... ,a(2N)] represents

2N codes for corresponding appearance descriptors based on the dictionary D, and Rc(A) and R_m (A) represent the code regulariser and manifold regulariser, respectively. The regularisation factors λ and β control the balance between the reconstruction error and the regularisers. In one arrangement, the code regulariser R_C(A) — Σ^ι 11^(0 Hi measures the sparsity of a code a(i) using Q-norm. In another arrangement, the code regulariser R_C(A) — Σμ=ιΙΙ«(0II2 measures the magnitude of a code a(t) using Q-norm. If the trace of a matrix B is defined as trace (B), the manifold regulariser R_M(A) in one arrangement is R_M(A) — trace(ALA^T), which makes the atoms of the dictionary respect the neighbouring relationship underlying the input data using the Laplacian matrix L. If two appearance descriptors x(i) and x(f) are close to each other in the feature space, then the corresponding codes α(ί) and a(f) are also close to each other in the encoding space.

The method 600 may be implemented within the module 201 as one or more software code modules of the software application program 233 resident in the hard disk drive 210 and being controlled by execution of the processor 205.

The method 600 starts at a receiving step 605, where appearance descriptors determined from training images are received as input under execution of the processor 205.

14063804vl

2017279676 20 Dec 2017

The method 600 passes under execution of the processor 205 from step 605 to a determining step 610. At step 610, an assignment between objects in training images received from query and gallery cameras is determined using appearance descriptors under execution of the processor 205. In one arrangement, the assignment is determined by the “Hungarian assignment algorithm” that solves the assignment problem based on a cost matrix. Each element of the cost matrix is the dissimilarity between two appearance descriptors extracted respectively from two objects. One object is detected from a training image received from the query camera 125 and the other object is detected from a training image received from the gallery camera 115. One example of a dissimilarity measure is a Euclidean distance between appearance descriptors.

Another example of a dissimilarity measure is a Cosine distance between appearance descriptors. Any other suitable dissimilarity measure may be used to determine the dissimilarity between appearance descriptors. In another arrangement, the assignment is determined by using a probabilistic graph matching algorithm that solves the assignment problem using “Maximum a posteriori inference”.

Based on the assignment determined at step 610, each object in an image received from the gallery camera 115 is assigned as a matched object to an object in an image received from the query camera 125. The indexes of appearance descriptors in Xq and X_G are re-ordered based on the assignment. The i-th appearance descriptor Xq(i) in Xq and i-th appearance descriptor x_G (i) in X_G are considered to be matched based on the assignment. In other words, matching descriptors in Xq and X_G have the same ordinality. The output of step 610 is appearance descriptors Xq and X_G re-ordered based on the assignment. The appearance descriptors X_Q and X_G may be stored in the memory 206.

The method 600 progresses under execution of the processor 205 from step 610 to a generation and refinement step 615. At the step 615, an affinity matrix is generated using appearance descriptors Xq and X_G re-ordered at step 610. Each element of the affinity matrix, Wy, is the similarity score between the i-th appearance descriptor Xq(T) in Xq and they-th appearance descriptor x_G(j) in X_G. Each diagonal element of the affinity matrix W is the similarity score between the i-th appearance descriptor Xq (i) in Xq and the i-th appearance descriptor x_G (i) in X_G, which is considered as a match to x_Q (i) based on the assignment determined at step 610.

One example of a similarity score used for generating the affinity matrix is a reciprocal

Euclidean distance between appearance descriptors. Another example of a similarity score is a Gaussian kernel value between appearance descriptors. Any other suitable similarity score may be used to determine the similarity between appearance descriptors. The affinity matrix is then

14063804vl

2017279676 20 Dec 2017 refined using a diffusion process that iteratively improves the affinity matrix by exploring the underlying characteristic of the data based on the pairwise similarities in the affinity matrix. In one arrangement, a smoothing operator is determined as P — C~^rW where C is a degree matrix with diagonal elements Q j determined in accordance with Equation (4) as follows:

<Υ=Σ,Μγ,· (4) where H^z. j is the element of the affinity matrix W at the z'-th row and /-th column. The affinity matrix is iteratively updated by using the smoothing operator P as follows:

W(t + 1) = W(t)P + I where W(t) represents the affinity matrix obtained at the t-th iteration. After being iteratively updated by the smoothing operator for a selected number of iterations (e.g., 10), the affinity matrix is normalized as W — WCf In another arrangement, the smoothing operator P is constructed by taking n (e.g., n — 5) largest entries at each row of the affinity matrix. The remaining entries at each row of the smoothing operator are set to zero. The affinity matrix is then iteratively updated by using the smoothing operator P as follows:

W(t + 1) = PW(f)P^T + I

The refined affinity matrix output at step 615 may be stored in the memory 206.

The method 600 passes under execution of the processor 205 from step 615 to a determining step 620. At step 620, a Laplacian matrix L is determined using the affinity matrix W generated at step 615 in accordance with Equation (5), as follows:

L = C - W (5) where C is the degree matrix with diagonal elements Q j determined from the affinity matrix W in accordance with Equation (4). Some entries of the Laplacian matrix L are then assigned to be zero by comparing each entry to a predetermined threshold (e.g., 0.4). The Laplacian matrix L output at step 620 may be stored in the memory 206.

The method 600 passes under execution of the processor 205 from step 620 to an initialization step 625, wherein the dictionary and the model parameters for dictionary learning are initialized.

In one arrangement, the dictionary is initialised by a random matrix where each element is a random number generated according to a predetermined distribution (e.g. a Gaussian

14063804vl

2017279676 20 Dec 2017 distribution). In another arrangement, the dictionary is initialised by using a K-SVD algorithm. In one arrangement, the initial values of the model parameters K, λ, and β in Equation (3) may be predetermined (e.g., K — 128, λ — 0.01 and β — 1.0). In another arrangement, the initial values of the model parameters K, λ, and β are determined by using cross-validation. The initial dictionary and model parameters output at step 625 may be stored in the memory 206.

The method 600 passes under execution of the processor 205 from step 625 to a learning step 630, where the dictionary is learned using the appearance descriptors re-ordered at step 610, the Laplacian matrix determined at step 620, and the dictionary and model parameters initialized at step 625. A method 700 of learning the dictionary, as executed at step 630, will be described hereinafter with reference to Fig. 7. The dictionary output at step 630 may be stored in the memory 206.

The method 600 progresses under execution of the processor 205 from step 630 to a determining step 635, wherein the codes for the appearance descriptors are determined based on the dictionary determined at step 630. At step 635, the code for each appearance descriptor is determined in a similar manner to step 350 and step 355 of method 300 in Fig. 3 that determine codes for query and gallery objects respectively. The determined codes may be stored in the memory 206.

The method 600 passes under execution of the processor 205 from step 635 to a decision step 640, where the method 600 determines whether the maximum number of iterations has been reached. The maximum number of iterations may be pre-defined to be an integer larger than zero. At step 640, if the maximum number of iterations has been reached, Yes, then the method 600 concludes after completing the decision step 640 and the output of the method 600 is the learned dictionary which may be stored in the memory 206 under execution of the processor 205. If the maximum number of iterations has not been reached, No, then the method 600 proceeds from the decision step 640 to a determination step 645.

At step 645, an assignment between the codes received from step 635 is determined in a similar manner to step 610. The indexes of the appearance descriptors in Xq and X_G and the indexes of codes for the appearance descriptors are re-ordered based on the assignment determined at step 645. The appearance descriptors and codes output at step 645 may be stored in the memory 206.

The method 600 passes under execution of the processor 205 from step 645 to a generation and refinement step 650, where an affinity matrix is generated using the codes re-ordered at step

14063804vl

2017279676 20 Dec 2017

645. At step 650, the affinity matrix is generated and is refined using a diffusion process in a similar manner to step 615.

The method 600 passes under execution of the processor 205 from step 650 to a determination step 660, wherein a Laplacian matrix L is determined using the affinity matrix determined at step 650. The Laplacian matrix L is determined in a similar manner to step 620. The Laplacian matrix L output at step 660 may be stored in the memory 206.

The method 600 passes under execution of the processor 205 from step 660 to a teaming step 665, where the dictionary is learned using the appearance descriptors re-ordered at step 645, the Laplacian matrix determined at step 660, the model parameters initialized at step 625, and the dictionary learned at step 630. A method 700 of learning the dictionary, as executed at step 665, will be described hereinafter with reference to Fig. 7.

The method 600 returns under execution of the processor 205 from the learning step 665 to the determination step 635.

The method 700 of learning a dictionary, as executed at the steps 630 and 665 of the method 15 600, is now described with reference to Tig. 7. The method 700 may be implemented within the module 201 as one or more software code modules of the software application program 233 resident in the hard disk drive 210 and being controlled by execution of the processor 205.

The method 700 starts at a receiving step 710, where appearance descriptors, a Laplacian matrix, the model parameters initialized at step 625, and an estimated dictionary are received as input under execution of the processor 205. The estimated dictionary is the dictionary initialized at step 625, or the dictionary output from the previous learning process at step 665 of method 600.

The method 700 passes under execution of the processor 205 from step 710 to a selection step 720, where a maximum number of iterations is selected. In one arrangement, the maximum number of iteration is selected to be twenty (20). The maximum number of iterations output at step 720 may be stored in the memory 206.

The method 700 passes under execution of the processor 205 from step 720 to a determination step 730. At step 730, the codes for the received descriptors are determined using the estimated dictionary, Laplacian matrix L, and model parameters received from the step 710. The codes are

14063804vl

2017279676 20 Dec 2017 determined by minimizing the objective function described in accordance with Equation (6), as follows:

min^HX — DA |j^· + AR_C(A) + /Jtrace^iX) (6) where D represents the estimated dictionary. The determination of the codes is based on the selection of the code regulariser R_C(A).

In one arrangement, the code regulariser /?_C(A) = Σ^=ιΙΙ«(0ΙΙι is selected to measure the sparsity of a code a(i) using X-norm. In the arrangement where the code regulariser R_C(A) — Σί=ί ll«(0 Hi is selected to measure the sparsity of a code a(i) using X-norm, the codes are determined by using a sparse coding algorithm. One example of a sparse coding algorithm is a feature-sign search algorithm. Another example of a sparse coding algorithm is an alternating direction method of multipliers.

In another arrangement, the code regulariser R_C(A) — Σί=ι 11 «(0II2 is selected to measure the magnitude of a code a(T) using Z₂-norm. In the arrangement where the code regulariser Rc(A) = ΣίίιΙΙαίΟΙΙζ is selected to measure the magnitude of a code a(Z) using Z₂-norm, the codes A are determined by solving a “Sylvester equation” described in accordance with Equation (7), as follows:

(D^TD + λΙ)Α + βΑί =D^TX (7)

The codes output at step 730 may be stored in the memory 206.

The method 700 passes under execution of the processor 205 from step 730 to an updating step

740. At step 740, the dictionary is updated using the codes determined at step 730. In one arrangement, the dictionary D is updated in accordance with Equation (8), as follows:

D = XA^T (AA^T + μΐ)-¹ (8) where μ is a pre-defined regularisation factor (e.g., μ — 0.01). In another arrangement, the dictionary is updated using a “gradient descent method”.

The method 700 passes under execution of the processor 205 from step 740 to a decision step 750, where the maximum number of iterations selected at step 720 is checked. At step 750, if the maximum number of iterations has been reached, Yes, then the method 700 concludes after completing the decision step 750 and the output of the method 700 is the learned dictionary. If

14063804vl

2017279676 20 Dec 2017 the maximum number of iterations has not been reached, No, then the method 700 proceeds from the decision step 750 to a determination step 730.

The method 700 concludes after the maximum number of iterations has been reached in the decision step 750. The output of the method 700 is a dictionary for determining codes of appearance descriptors.

The method 800 of updating a dictionary, as executed at step 480 of the method 400, will now be described with reference to Fig. 8. The method 800 receives a dictionary, model parameters, a set of previously collected appearance descriptors determined at step 430, and a set of new appearance descriptors determined at step 470 as input. The method 800 then updates the received dictionary using the received appearance descriptors.

The method 800 may be implemented within the module 201 as one or more software code modules of the software application program 233 resident in the hard disk drive 210 and being controlled by execution of the processor 205.

The method 800 starts at a receiving step 805, where a dictionary, model parameters, a set of previously collected appearance descriptors determined at step 430, and a set of new appearance descriptors determined at step 470, are received under execution of the processor 205.

The method 800 passes under execution of the processor 205 from step 805 to a determining step 810, where the codes for all received appearance descriptors are determined based on the received dictionary. The code for each appearance descriptor is determined in a similar manner to step 350 and step 355 of the method 300 in Fig. 3. The determined codes may be stored in the memory 206.

The method 800 proceeds under execution of the processor 205 from step 810 to a determining step 815, where the difference between two probability distributions of the codes for previously collected appearance descriptors and the codes for new appearance descriptors is determined. If there is a change in capturing environment and associated environmental conditions such as time-of-day, seasonal changes, lighting and other dynamics of a scene, the difference between two probability distributions of the codes will be large. In one arrangement, the difference between two probability distributions is determined at step 815 using the maximum mean discrepancy that is the Euclidean distance between the mean of the codes for previously

14063804vl

2017279676 20 Dec 2017 collected appearance descriptors and the mean of the codes for new appearance descriptors. In another arrangement, the probability distribution of codes is determined using a Gaussian mixture model. The Kullback-Leibler divergence may then be used for determining the difference between two probability distributions. The output of step 815 is the difference between two probability distributions of codes.

The method 800 proceeds under execution of the processor 205 from step 815 to a decision step 820, where the difference determined at step 815 is checked. At step 820, if the difference is less than a predetermined threshold (e.g., 0.5), No, then the method 800 concludes after completing the decision step 820 and returns the received dictionary as the output. If the difference is determined to be larger than the predetermined threshold, Yes, then the method 800 proceeds from the decision step 820 to a determining step 830.

At the determining step 830, an assignment between the codes received from step 810 is determined in a similar manner to step 645 in Fig. 6 and is determined by the Hungarian assignment algorithm. The indexes of the appearance descriptors and the indexes of codes for the appearance descriptors are re-ordered based on the assignment. The re-ordered appearance descriptors and codes output at step 830 may be stored in the memory 206.

The method 800 passes under execution of the processor 205 from step 830 to a selection step 840. At step 840, pairwise similarities among re-ordered codes determined at step 830 are determined and the pairs of codes with the highest similarities are selected. Each pair of codes is composed of two codes with the index determined at step 830. The similarity between a pair of codes is determined using a Cosine distance or a Euclidean distance. A number of code pairs with the highest pairwise similarities may be selected. In one arrangement, the number of pairs is predetermined (e.g., 150). The selected pairs of codes and the corresponding pairs of appearance descriptors are the output of step 840 and may be stored in the memory 206.

The method 800 passes under execution of the processor 205 from step 840 to a generation and refinement step 850, where an affinity matrix is generated using the selected pairs of codes determined at step 840. At step 850, the affinity matrix is generated and is refined using a diffusion process in a similar manner to step 615 of Fig. 6.

The method 800 continues under execution of the processor 205 from step 850 to a determining step 860, wherein a Faplacian matrix is determined using the affinity matrix determined at step

14063804vl

2017279676 20 Dec 2017

850. The Laplacian matrix is determined in a similar manner to step 620 in Fig. 6. The Laplacian matrix output at step 860 may be stored in the memory 206.

The method 800 passes under execution of the processor 205 from step 860 to a generation step 870, where an updated dictionary is generated using the pairs of re-ordered appearance descriptors selected at step 840, the Laplacian matrix determined at step 860, the received model parameters at step 805, and a previously determined dictionary. At step 870, a dictionary is generated in a similar manner to step 665 of Fig. 6. In the first iteration, the previously determined dictionary is the dictionary received at step 805. At step 870, a dictionary is updated using the selected pairs of re-ordered appearance descriptors and the previously determined dictionary. In the subsequent iterations, the dictionary generated at step 870 in the previous iteration is used as the previously determined dictionary.

The method 800 passes under execution of the processor 205 from step 870 to a decision step 880, where the maximum number of iterations has been reached. The maximum number of iterations is pre-defined to be an integer larger than zero. If the maximum number of iterations has not been reached, then the method 800 returns to the determining step 830. If the maximum number of iterations has been reached, then the method 800 concludes after completing the decision step 880 and the output of the method 800 is the updated dictionary generated at step 870.

The output of the method 800 is an updated dictionary for determining codes of appearance descriptors, which may be stored in the memory 206 under execution of the processor 205.

The arrangements described are applicable to the computer and data processing industries and particularly for image processing.

The foregoing describes only some embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

14063804vl

Claims (16)

1. CLAIMS:

   2017279676 20 Dec 2017

   1. A method of comparing objects in images, the method comprising:

   receiving a dictionary determined from a first set of images, the dictionary being

   5 determined using correspondences of codes of a first set of codes based on pairwise similarities between images in the first set of images, wherein the dictionary is applied to feature vectors generated from the first set of images to generate the first set of codes;

   determining a second set of codes for a second set of images using the received dictionary;

   10 generating an updated dictionary using the first and second set of codes when a difference between codes of the first and second set of images exceeds a predetermined threshold; and comparing objects in a first and second image using the updated dictionary.
2. 2. The method according to claim 1, wherein generating the updated dictionary using the

   15 first and second set of codes uses matching pairs of images selected from the first and second set of images.
3. 3. The method according to claim 1, wherein the difference between the first and second set of codes is caused by a change in environmental conditions of scenes of the first and second set of images.

   20
4. 4. The method according to claim 1, wherein the objects are a person, the person being located in images captured from two different cameras.
5. 5. The method according to claim 1, wherein at least one matching pair of images selected from the first and second set of images uses an image from the first set and an image from the second set.

   14063804vl

   2017279676 20 Dec 2017
6. 6. The method according to claim 1, wherein each image of each pair of images used for the pairwise similarities is sourced from a different camera.
7. 7. The method according to claim 1, wherein updating the received dictionary uses the 5 received dictionary to initialise the updated dictionary.
8. 8. The method according to claim 1, further comprising detecting the objects in the first and second image using bounding boxes.
9. 9. The method according to claim 8, wherein each bounding box contains the whole body of a person.
10. 10 10. The method according to claim 8, wherein each bounding box contain head regions of persons.
11. 11. The method according to claim 1, further comprising generating pairs of training objects, each training object being detected in a different camera.
12. 12. The method according to claim 1, further comprising determining an appearance 15 descriptor for each of the compared objects.
13. 13. The method according to claim 12, wherein the dictionary is updated using the appearance descriptors.
14. 14. The method according to claim 1, further comprising determining a foreground confidence mask for each of the images.

    20 15. An apparatus for comparing objects in images, the apparatus comprising:

    means for receiving a dictionary determined from a first set of images, the dictionary being determined using correspondences of codes of a first set of codes based on pairwise similarities between images in the first set of images, wherein the dictionary is applied to feature vectors generated from the first set of images to generate the first set of codes;

    14063804vl

    2017279676 20 Dec 2017 means for determining a second set of codes for a second set of images using the received dictionary;

    means for generating an updated dictionary using the first and second set of codes based on a difference between codes of the first and second set of codes exceeding a predetermined

    5 threshold; and comparing objects in a first and second image using the updated dictionary.

    16. A system for comparing objects in images, the system comprising:

    a memory for storing data and a computer program; and

    10 a processor coupled to the memory for executing the computer program, the computer program having instructions for:

    receiving a dictionary determined from a first set of images, the dictionary being determined using correspondences of codes of a first set of codes based on pairwise similarities between images in the first set of images, wherein the dictionary is applied
15. 15 to feature vectors generated from the first set of images to generate the first set of codes;

    determining a second set of codes for a second set of images using the received dictionary;

    generating an updated dictionary using the first and second set of codes based on a difference between codes of the first and second set of codes exceeding a

    20 predetermined threshold; and comparing objects in a first and second image using the updated dictionary.
16. 17. A non-transitory computer readable medium having a computer program stored on the medium for comparing objects in images, the program comprising:

    code for receiving a dictionary determined from a first set of images, the dictionary 25 being determined using correspondences of codes of a first set of codes based on pairwise

    14063804vl

    2017279676 20 Dec 2017 similarities between images in the first set of images, wherein the dictionary is applied to feature vectors generated from the first set of images to generate the first set of codes;

    code for determining a second set of codes for a second set of images using the received dictionary;

    5 code for generating an updated dictionary using the first and second set of codes based on a difference between codes of the first and second set of codes exceeding a predetermined threshold; and comparing objects in a first and second image using the updated dictionary.

    CANON KABUSHIKI KAISHA

    10 Patent Attorneys for the Applicant/Nominated Person

    SPRUSON & FERGUSON

    14063804vl

    1/9

    29Υ12Ί96Ί6 20 Dec 2017