EP4690143A1 - Context aware identification of unknown objects in image frames - Google Patents

Abstract

An image processing device includes at least one processor which is operative to determine a scene category for an image generated by an image sensor. The operations detect known objects in the image and determine region proposals in the image where unknown objects are possibly located. The operations determine provisional object classes for unknown objects in the region proposals based on relationships between the scene category, the known objects, and the region proposals in the image. The operations provide to an application the known objects and the provisional object classes for the objects in the region proposals.

Description

CONTEXT AWARE IDENTIFICATION OF UNKNOWN OBJECTS IN IMAGE

FRAMES

TECHNICAL FIELD

[001] The present disclosure relates to image processing and, more particularly, to identifying new or unknown objects within image frames from cameras and other imaging sensors.

BACKGROUND

[002] The world is diverse and dynamic in the number, type, and configuration of objects that can be captured in image frames, such as still images or video, from cameras and other imaging sensors. Object detection models should accordingly be adaptable to the particular user needs, use cases and application requirements.

[003] Object detection algorithms assign a class to objects and localize them in images. A conventional close-set object detector relies on a pre-trained deep learning model with a set of object classes to identify these objects with acceptable accuracy. The close-set object detector is limited to detecting only objects that are members of a predefined set of objects, where the predefined set of objects may have been used to train the object detector or may be referenced by the object detector when attempting to detect objects. If a new object appears, that is, an object not included in the training set, a close-set detector will treat unknown objects as background or overfit them to labeled objects. This will cause missing information or generate misinformation or erroneous classifications. One of the challenges is to leam and identify new classes of objects, i.e., assign a category to objects that appear in a scene that is not included in the training set of the model.

[004] This problem is addressed in different ways according to the learning process followed. Some approaches try to add new object classes incrementally to the model. Incremental learning or continual learning can include retraining the model with the old and new object classes. One approach disclosed in reference [1] generates training data on-the- fly for the new object classes (including the labels and bounding boxes) and propose an end- to-end learning system. Operations address “catastrophic forgetting,” a problem which can commonly occur with these approaches, in which the old classes are forgotten due to the retraining process with the new classes.

[005] Other approaches, such as disclosed in references [2,3], addressed the problem as an open-set object detection, which tries to identify unknown object classes. It can include two stages: a conventional deep learning model with an offline trained model to detect the known objects and a second stage for the unknown detection. The disclosed operations build a dataset of unknown objects to train a network to categorize the new objects as “unknown”. Another approach disclosed in reference [4] extends the above by adding a third stage that incrementally learns the unknown categories. This approach is called open-world detection. These approaches are very computationally expensive and time-consuming; sometimes, taking minutes before having a class for the new object, and making them unsuitable for realtime applications.

[006] In addition, some approaches attempt to recognize new categories without training examples relying on contextual information, such as disclosed in references [5, 6, 7], and previously detected objects. These approaches are called zero-shot object detectors. Some of these and other references propose end-to-end deep networks based on object classes available during the training stage and assume that the unknown objects are not present in the images contained in the training set. For instance, reference [6] proposes to infer new objects using known objects and their visual relationships to infer classes. Also, reference [7] incorporates visual features combined with semantic information to build a network based on graph modeling to infer the new object classes.

[007] The main drawbacks of current approaches when identifying unknown objects are three-fold. First, the learning-based operational strategies need a large amount of data to train the models and are computationally expensive since several networks are needed to incorporate the new objects on the fly. Second, the approaches rely on the initially known objects to build the relations and infer the unknown object classes without considering the errors and misclassifications of the close-set detectors. Third, the approaches without training need to prune training sets to remove unknown objects and use more networks or subnetworks to assign the classes to the unknown objects. SUMMARY

[009] Some embodiments disclosed herein are directed to an image processing device that includes at least one processor which is operative to determine a scene category for an image generated by an image sensor and to detect known objects in the image. The processor determines region proposals in the image where unknown objects are possibly located. The processor determines provisional object classes for unknown objects in the region proposals based on relationships between the scene category, the known objects, and the region proposals in the image. The processor provides to an application the known objects and the provisional object classes for the objects in the region proposals.

[0010] Some other embodiments are directed to a corresponding method by an image processing device. The method includes determining a scene category for an image generated by an image sensor, and detecting known objects in the image. The method includes determining region proposals in the image where unknown objects are possibly located. The method includes determining provisional object classes for unknown objects in the region proposals based on relationships between the scene category, the known objects, and the region proposals in the image. The method includes providing to an application the known objects and the provisional object classes for the objects in the region proposals [0011] As will be explained in further detail below, these and further embodiments disclosed herein enable determination of object classes for unknown objects in an image, and may be performed while using a close-set object detector which, e.g., has not been trained to classify the unknown objects. These and other advantages may be obtained through the use of a scene category that is determined for an image and the use of objects that have been detected in the image, to select and apply relationships that enable determination of provisional object classes to unknown objects in the image.

[0012] Other image processing devices and methods according to embodiments will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such image processing devices and methods be included within this description, be within the scope of the present disclosure, and be protected by the accompanying claims. Moreover, it is intended that all embodiments disclosed herein can be implemented separately or combined in any way and/or combination. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Aspects of the present disclosure are illustrated by way of example and are not limited by the accompanying drawings. In the drawings:

[0014] Figure 1 illustrates an image processing device which provides common object detections, uncommon object detections, and provisional object detections to an application in accordance with some embodiments of the present disclosure;

[0015] Figure 2 illustrates a flowchart of operations that can be performed by the image processing device in accordance with some embodiments of the present disclosure;

[0016] Figure 3 illustrates a graphical representation in accordance with one embodiment of the present disclosure of a training object set used by the close-set object detection module, the set of detected objects, the set of detected common objects, the set of detected uncommon objects, the scene-based-likely object set, a set of objects more likely found in the scene category, and a set of candidate objects that are more likely to be found near the set of detected common objects;

[0017] Figures 4a through 4e illustrate an example image and objects that have been determined through operations in accordance with some embodiments of the present disclosure;

[0018] Figure 5 illustrates a flowchart of more general operations that can be performed by an image processing device in accordance with some embodiments of the present disclosure; and

[0019] Figure 6 illustrates components of an image processing device which can be configured to operate according any of the embodiments disclosed herein.

DETAILED DESCRIPTION

[0020] Inventive concepts will now be described more fully hereinafter with reference to the accompanying drawings, in which examples of embodiments of inventive concepts are shown. Inventive concepts may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of various present inventive concepts to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present/used in another embodiment.

[0021] Conventional close-set object detection models assume that all the object classes to be detected appear during training. Therefore, unknown objects are treated as background or are overfitted to labelled objects. Incremental learning approaches take a long time before having a class for the novel objects (e.g., references [1, 4]); they need first to build a dataset for the unknown object on the fly, then retrain the model, having the new object class available for the application after a couple of minutes.

[0022] Zero-shot based methods build solutions to detect unknown objects without training by applying an object detector that identifies the known objects and based on that, infers some relations that lead to a new object class for the unknown objects in the image. The main problem of these approaches is that they rely on the object detector outputs. The performance of zero-shot detection models depends on how well the known objects are learned to be detected correctly. These deep learning-based object detectors (e.g., reference [5]) mainly focus on initially known objects to classify and localize the new classes by exploiting several relationships. However, the detectors are imperfect and sometimes they output erroneous classifications of objects even though they are trained to detect specific objects. If the known objects are wrongly classified, the relationships built will be wrong, and consequently, the new object classes will probably be erroneous.

[0023] Most of these approaches need several networks or subnetworks to address the problem, for example, having a network to first classify into unknown, known and background, to then apply the inference of new object classes. In reference [5], they define meta-class categories for the objects (known, unknown, and background) and then refine the new classes using a second network that leams the relationships between the meta-categories. This makes the process more complex, heavy, and time-consuming.

[0024] Various prior approaches for identifying unknown objects based on contextual information, assume that the known objects are available during training while the unknown objects are not (e.g., references [5, 6, 7]). These approaches pre-train a model on a base dataset for only known object classes. As the training data cannot contain unknown objects, several mechanisms are applied to prune the training data before pre-train the model not to ruin the inference during testing. In reference [7], they divide the dataset between seen and unseen classes, crop the objects within the bounding box of the image and then save them for further training.

[0025] That constitutes a problem because to apply these approaches, there is a need to know the unknown objects in advance to remove them from the dataset and retrain the object detectors to apply their solutions. So, available object detectors need a pruning phase to be used for known classification, making them unsuitable for real-world deployment and applications. [0026] Embodiments of the present disclosure are directed to identifying new or unknown objects in images of an environment.

[0027] In accordance with some embodiments, operations assign a class to unknown objects considering contextual information extracted from the environment. The operations determine a provisional object class considering the scene type, other objects detected, and the relationships between them. In this way, the operations can provide as much information as possible, not only the outputs of a close-set object detector but the provisional classes for unknown objects.

[0028] The operations may use an image processing device that runs a conventional close-set object detector along with a scene recognition model which is configured according to some present embodiments. The input to the device can be single image (also "image frame) or a sequence of images. The scene recognition model outputs a scene category for the image frame (e.g., living room, bedroom, kitchen). The object detector extracts the region proposals from the image and assigns an object class to some of them according to its pre-trained model. This information is sent to the context-aware detection module (CAD), which first verifies the known objects given by the object detector by explicitly exploiting the scene information.

[0029] Then, it extracts the region proposals containing the unknown objects. An object class is assigned to each proposal using the relations of the contextual information of the image. Finally, redundant or ambiguous classifications are removed. This way, the operations can assign new object classes to unknown objects without training by leveraging surrounding information.

[0030] The context-aware detector (CAD) module can operate to take advantage of different contextual information to assign a provisional object class (POC) for unknown objects in an image. The operations can exploit the scene information to verify the initially known objects, assuming that the object detector module can output erroneous classifications and that the scene classifier offers good performance in terms of accuracy, providing beneficial information in disambiguating the recognized objects such as described in reference [12], This way, the scene information is used to verify that the detected objects correspond to common or uncommon objects in the scene, enriching the information about the initial detections, leading to a better inference of the new object classes. The operations can then exploit the relationships between the output of the scene recognition model, the detected objects by the close-set object detector, and the region proposals to determine provisional object classes that represent the most meaningful classes for the unknown objects without training.

[0031] The CAD module operations to categorize unknown objects by leveraging relationships between contextual information. New objects can be more accurately determined by improving the operational ability to distinguish new objects from background and allow more robust classification of the new objects. The operations use the scene information to verify the initial object detections and improve the generation of relationships, without necessarily assuming that the initially detected objects are correct. These operations may avoid use of inappropriate relations that lead to incorrect new object classes. Provisional object classes (POCs) can be used to facilitate construction of more enriched three dimensional (3D) scene graphs and 3D environments considering the detected objects along with new objects and identified relationships. The operations may enable the interaction in 3D models for extended reality (XR) and robotic applications, which will positively affect the user experience and application operational performance and feature richness.

[0032] Operations according to some embodiments can enable a device to perform object detection using contextual information without requiring training. Additionally, the operations do not need explicit human intervention to annotate new object classes, such as is required by various known object detection approaches, and can avoid the need to prune a training set to remove unknown objects.

[0033] Some embodiments are now explained in the context of Figure 1. Figure 1 illustrates an image processing device 50 which provides common object detections, uncommon object detections, and provisional object detections to an application 108 in accordance with some embodiments of the present disclosure.

[0034] The image processing device 50 may include an image sensor 100 that provides images, which are also referred to, without limitation, as image frames. The image sensor 100 may be any type of image-based sensor, including without limitation a monocular camera, stereo cameras, etc. Alternatively, the image processing device 50 may obtain images from a remote image sensor 100. The image processing device 50 includes an object detection (OD) module 104, a scene recognition (SR) module 102, a context-aware detection (CAD) module 106, and may further include or output data for use by an application 108. [0035] As will be explained in further detail below, the SR module 102 and the OD module 104 both obtain an image captured by the image sensor 100. The SR module 102 processes the image to output a scene category (S t) to the CAD module 106. The OD module 104 processes the image to output region proposals (R_a) and object detections (O k) to the CAD module 106. The region proposals (R_a) can correspond to where unknown objects may be located in the image. The CAD module 106 responsively processes and outputs common object detections (O_c), uncommon object detections (O_uc), and provisional object classes (POCs) for use as input to the application 108. The POCs can be assigned for each region proposal that was not assigned an object class by the OD module 104. The application 108 can be operable to use the received information.

[0036] Various embodiments are now explained that may include some or all of the following operational steps:

1) Determine a scene category (S t) for an image.

2) Detect known objects (O_k) which are present in the image.

3) Verify the detected known objects based on the scene information, such can be designated as common objects (O_c) and uncommon objects (O_uc).

4) Determine region proposals containing unknown objects (using as input R_a).

5) Determine the provisional object class (POC) for each region proposal.

6) Remove redundant or ambiguous classifications for the region proposals.

7) Send the new object information and the object detections to the application allowing more knowledge of the environment.

[0037] Reference is now made to the flowchart of Figure 2 to explain in more detail operations that can be performed by the image processing device 50 and, more particularly, by the OD module 104, the SR module 102, and the CAD module 106 in accordance with some embodiments of the present disclosure.

[0038] Operational step 200 obtains an image from the image sensor 100.

[0039] Operational step 202 determines a scene category for the image.

[0040] The scene recognition (SR) module 102 processes the image to determine a scene category S t that is captured in the image. For example, the SR module 102 may categorize an image as containing the scene of a kitchen, living room, bedroom, defined type of business room, defined type of building, defined type of outdoor setting, or another defined scene category. The scene category S t may be determined based on recognizing features and/or objects in the image and relating the recognition to one of a plurality of defined categories of scenes. The SR module 102 may process the image using a neural network model, such as disclosed in reference [8] or reference [9], which has been trained to output a scene category from an input image.

[0041] Alternatively or additionally, the scene category S t may be determined based on metadata provided with the image, such as geographical location of the image sensor when the image was captured, direction of view of the image sensor when the image was captured, identifier of a Bluetooth device which was detected proximately in time to when the image was captured and which has a location known to the SR module 102, identifier of a WiFi device which was detected proximately in time to when the image was captured and which has a location known to the SR module 102, etc. Alternatively or additionally, the scene category S t may be determined based on relatedness in elapsed time between a present image and an earlier image for which a scene category has been previously determined, and/or based on relatedness in geographic location between between the present image and the earlier image for which a scene category has been previously determined.

[0042] The SR module 102 outputs the scene category S t, and may further output a confidence score indicating a level of confidence that the scene is that scene category. [0043] Operational step 204 detects objects present in the image.

[0044] The close-set object detector (OD) module 104 processes the image to detect objects present in the image and their corresponding regions in the images. The OD module 104 may process the image concurrently with the SR module 102 processing the image. For example, the image may be provided simultaneously to both the SR module 102 and the OD module 104 to initiate their respective processing. The OD module 104 outputs a set of known objects O_k = {o_i,.. . ,o_n} (which are limited by the object classes in the training set used by the OD module) and outputs bounding boxes B_k = {b_i,... ,b_n} (also called regions) which may enclose or otherwise define location of the detected objects, which includes known objects and may further including bounding boxes (region proposals) where unknown objects are possibly located. The OD module 104 may further output confidence scores, where each confidence score indicates a level of confidence for an associated one of the detected objects C_k = {c i,... ,c_n}.

[0045] The OD module 104 may be a one-stage object detector (e.g., which may be based on reference [10]), a two-stage object detector (e.g., which may be based on reference [11]), or may be based on another multi-stage object detector. The OD module 104 may include a pretrained deep learning model with a defined set of object classes which are used to identify objects in the image.

[0046] Operational step 206 verifies the detected objects from the OD module 104.

[0047] The operations use the scene information and known objects as input to predict new object classes. In this step, the scene information is used to verify that individual objects in the set of the detected objects O_k correspond to the scene category S_t. In some embodiments, any of the individual objects which are not able to be verified can be removed or otherwise not further used from the set of the detected objects O k. The verification step can avoid further operations extracting inappropriate relationships that could negatively affect or prevent subsequent operations to determine new objects among the detected objects O_k. [0048] The operations obtain a scene-based-likely object set O_e that represents the set of objects that are more likely to be found in the particular scene category S_t, e.g., were determined to satisfy a defined rule such as more than a threshold value of likelihood of being found in the particular scene category S t. The scene-based-likely object set O_e may be extracted from a commonsense knowledge base which associates objects that are likely to be found in defined scene categories (e.g. which may be based on ConceptNet). The operations use the scene-based-likely object set O_e to obtain a set of common objects O_c by comparing the detected objects O k to the scene-based-likely object set O_e. The comparison may identify matches therebetween and/or identify where similarity between objects in the sets satisfies a defined similarity rule. The set of common objects O_c may be the intersection of objects belonging to the detected objects O_k and to the scene-based-likely object set O_e, such as:

O_c = O k A O_e

[0049] Objects in the set of detected objects O k which are not found in the scene-based- likely object set O_e may be designated as uncommon objects O_uc, which means that these objects are unlikely to be found in the scene category S_t:

O_uc = O k \ O_e

[0050] Figure 3 illustrates a graphical representation in accordance with one embodiment of a training object set O_od used by the close-set object detection module 104, the set of detected objects O_k, the set of detected common objects O_c, the set of detected uncommon objects O_uc, the scene-based-likely object set O_e, a set of objects more likely found in the scene category S_t, and a set of candidate objects that are more likely to be found near the set of detected common objects O_c.

[0051] For the set of common objects O_c, the associated bounding boxes and confidence scores can be registered in the vectors (data structures defined in memory) B_c and C_c, respectively. Similarly, for the set of uncommon objects O_uc, the associated bounding boxes and confidence scores can be registered in the vectors (data structures defined in memory) B_uc and C ue.

[0052] The set of common objects O_c is used for extracting relationships between the objects and the scene category S_t. In some embodiments, the set of uncommon objects O_uc is only used for the extraction of spatial relationships as described in the operations below.

[0053] Operational step 208 determines the region proposals containing the unknown objects. [0054] Step 208 defines regions (called "region proposals") in the image where unknown objects are possibly located. Later operational steps attempt to identify the unknown objects in the defined regions. These operations can rely on operation 202 by the OD module 104 to determine from the image a set of region proposals R u = {r_i, ... ,r_m} containing the potential unknown objects. Each region proposal may be represented by r = [x, y, h, w, f] having the top-left x,y coordinates, the width and height of the region and the feature vector f that can be determined from the neural network. Although rectangular regions are described and shown in the use case of Figures 4a-4e (described below) for ease of illustration, the regions may be defined to have any shape.

[0055] The region proposals can be determined by a region proposal module (RPM) of a two- stage object detector. This type of detector has two stages: localization and classification. The localization stage generates a set of regions for all possible objects (R d that are then sent to the classification stage, which determines the object class according to a set of categories in the training set.

[0056] For a one-stage object detector, one option to determine the region proposals for all possible objects R a is to change the configuration of the anchors (as they influence the size of the convolutional layers and the number of boxes produced to predict the objects). The generated bounding boxes may be filtered to consider only the predictions that satisfy a defined rule, e.g., by being above a threshold value. The threshold value can be relaxed (e.g., reduced) to keep all the possible objects (bounding boxes priors or anchor boxes) in the image and not only those corresponding to the training set. This operation changes the configuration of the predetermined anchors.

[0057] To identify the potential regions containing unknown objects R_u, the operations select only the regions in R a that were not assigned an object class, which may include subtracting the bounding boxes (B_k) of the detected objects. Then, the region proposals containing the unknown objects are obtained by R u = R_a\ B k.

[0058] Another alternative operation uses category-independent object proposals (classagnostic proposal), e.g., based on the disclosure of reference [2], to determine the set of region proposals R a from the image.

[0059] Operational step 210 extracts relationships between objects and the scene category S t, where the relationships may be retrieved from a database using the scene category S_t. The relationships may define spatial relationships of paterns (raster paterns) in an image that form individual objects and/or define relationships between various objects which are defined as being associated with the scene category S t. An example relationship that may be extracted 210 can define raster paterns in the image of objects that are types of home furniture, office furniture, buildings, vehicles, etc. Alternatively or additionally, the relationship that may be extracted 210 can define how types of objects can be spatially related (e.g., oriented) to each other, such as how chairs are to be oriented in the image relative to a table. The database may define known relationships which have been learned and/or defined between objects present in various categories of scenes. The relations may be extracted from a commonsense knowledge base (e.g., the multilingual knowledge graph known as ConceptNet and/or Visual Genome).

[0060] Operational step 212 determines the provisional object classes (POCs).

[0061] The inputs to this step can include the scene category (S ), a set of region proposals containing the unknown objects R_u, the set (vector) of common objects (O c, B e, C_c) and the set (vector) of uncommon objects (O tic, B uc, C ue).

[0062] The provisional object class (POC) can be determined 212 using the relations (from step 210) of the contextual information contained in the image and each region proposal. The operations may assign a provisional object class based on an iterative (or repetitive processing one unknown object at a time) process flow that includes at least some of the following steps: i. An operational stage determines scene-object relationships. The operational stage obtains the objects that are more likely found in scene category S t that are not included in the training set (O_od in Figure 3) of the OD module 104. The training set O_od (Figure 3) is the set of object on which the close-set OD module 104 has been trained. The symbol O st represents the set of objects that are more likely (e.g., with a high probability) of being found in in the scene category S t.

O st = O e \ O od In one example implementation, data of the scene category is used to query a knowledge base defining the most common objects that can be found in that scene category. The knowledge base, such as ConceptNet, stores semantic information about human defined knowledge and/or machine learned observations. The most common objects that can be found in this scene category may be determined from, e.g., a graph neural network or algorithm which may be located on a cloud-based or other network server. Alternatively, a user may tell the system the most common objects in this scene category. ii. Another operational stage determines object co-occurrences. The operational stage obtains the set of objects from O st that are more likely to be next to (proximately located to or adjacent to) the common objects detected O c in the image frame. The operational stage outputs a set of candidate objects O _p that are more likely to be found near the common objects detected. Co-occurrence statistics can be used in this step and obtained from a commonsense knowledge base (e.g., as disclosed in reference [3]). iii. Another operational stage determines the spatial relationships of each region proposal. The operational stage analyzes the location of each region proposal with respect to the detected objects in the image. The operations find the spatial relationships (e.g., in front of, next to, behind of, left, right, near, far from, size, shape) between O c, O tic, O _p and R u. as follow:

• Relations O p O c: for each object in O _p, the operations extract the spatial relations with respect to the common objects O c.

• Relations p O uc for each object in O _p, the operations extract the spatial relations with respect to the uncommon objects O uc.

• Relations R u - O c: the operations extract the spatial relations of each region proposal in R u with respect to the common objects O c.

• Relations R u - O_uc the operations extract the spatial relations of each region proposal in R u with respect to the uncommon objects O uc.

• Relations R u - O_p: the operations find the size and shape relations between R u and O _p. By exploiting bounding boxes or shapes of the proposals and

O _p the operations can further refine the object classes. The operations can use attributes such as contours (convex, concave) and bounding boxes size (comparisons like larger or smaller than other objects). iv. Operations match the spatial relations. The operations compare the relations O p O c, O p O tic, R M - O_c, R M - O uc, and R u O p to find the most dominant object class for each region proposal. That is the object class in O _p that matches the best of the extracted relationships, in accordance with one embodiment. The confidence score for each POC can be obtained through a similarity strategy (e.g., as disclosed in reference [4]). The operations can assign a provisional object class (POC) to each region proposal in R u that is not included in the training set of the close-set object detector without the need for training by leveraging prior contextual information. v. Another operational stage associates 214 each provisional object class with the corresponding feature vector f obtained from each region proposal. This information can then be exposed to the user or sent to one or more applications. An illustration of the output of the system is shown in Figure 4e, which is described in detail below. vi. Another operational stage compares feature vectors to transfer POC knowledge to speed up the process. The operations can use the feature vectors to speed up assigning new object classes. For instance, if the operations determine a POC_1 for a region proposal r_l, then the operations can compare (step 218) its feature vectorTyf with the vectors of the remaining proposals to find feature correspondences to see if there are any similar regions. If the similarity is determined to be higher than a certain threshold (i.e.,/\Z ~f_2 the operations can assume that regions r_l and r_2 correspond to the same object class (POC 1); therefore, it would not be necessary to infer the POC for r_2. vii. In another embodiment, the POC information (object class and feature vector) can be locally stored in order to be used for the next image frames. In this case, to identify the potential regions containing unknown objects R_u, we can also consider the previous POC to filter the region proposals further. This way, it can increase the efficiency of the method by avoiding the re-assignation of a previously object class determined.

[0063] Operational step 214 associates the POC with a feature vector of the object in one of the region proposals ("compared region proposal"). Operational step 216 queries (determines) features vectors of objects in the other region proposals. Operational step 218 determines whether the feature vector of the object in the one region proposal satisfies a similarity rule (e.g., for being sufficiently similar) relative to the feature vector of the object in any of the other region proposals. When the feature vector of the object in the compared region proposal satisfies the similarity rule relative to the feature vector of the object in another region proposal, the POC assigned to the object of the compared region proposal can also be transferred 220 (also assigned) to the object in the other region proposal. In this manner, the same POC can be assigned to the two of more objects that have feature vectors which satisfy the similarity rule. For example, once an unknown object has been determined to have a chair POC, another unknown object in the image which has a sufficiently similar feature vector can also be determined to have the chair POC, which thereby substantially reduces the processor and memory resource utilization by the operations to process objects in the image.

[0064] In contrast, when the feature vector of the object in the compared region proposal does not satisfy the similarity rule relative to the feature vector of the object in another one of the region proposals, the objects in the other region proposals can be separately processed to determine their POCs through repetition of the operations 212-220.

[0065] Operational step 222 removes redundant and/or ambiguous classifications.

[0066] If two region proposals are too close, it could indicate that they correspond to the same object. However, these region proposals can be assigned (determined to have) different object classes where, for example, one of the object classes assigned to one of the region proposals may be incorrect or less correct relative to the other object class assigned to the other region proposal. In order to avoid such ambiguity, the operations may calculate overlap between the two regions (r _ui, r_uj) by applying Intersection over Union (loU), such as below:

IoU(r_ui, r_uj) > /beta

[0067] The term /beta can be threshold used to decide whether two bounding boxes correspond to the same object. When loU is greater than /beta, the operations assume two bounding boxes correspond to the same object, and choose the provisional object class (POC) with the highest confidence score. In contrast, when loU is less than /beta, the operations assume the regions contain different objects and so both regions are kept with their provisional object classes.

[0068] In another embodiment, the redundant regions can be removed before inferring the POC by applying, for instance, a non-maximum suppression (NMS) to remove such duplicates. For example, when two region proposals extracted in operational step 208 have more than a threshold amount of overlap, the operations can select to use only one of the region proposals or otherwise combine the two region proposals into a single region proposal to be processed in the subsequent operational steps.

[0069] Application of various of the above described operations to example objects of a nonlimiting example image scenario is now explained with reference to Figures 4a through 4e. [0070] Figure 4a illustrates an image that is obtained from the image sensor 100.

[0071] Figure 4b illustrates that operations have detected objects in the image that are operationally classified as a car (o_l), a chair (o_2), another chair (o_3), and a sofa (o_4). Operations have also determined the objects corresponds to a living room scene category (S_l).

[0072] Figure 4c illustrates that detected objects have been verified through further operations using the information for the living room scene category to classify the set of uncommon objects to include the car (o_uci), and to classify the set of common objects to include the chair (o_ci), the other chair (o_C2), and the sofa fo es). The car (o_uci) is classified as an uncommon object because it is not determined to be sufficiently related to the living room scene category. In contrast, the chair (o_ci), the other chair (o_C2), and the sofa (o_cs) are classified as common objects because they are determined to be sufficiently related to the living room scene category.

[0073] Figure 4d illustrates that region proposals 400 (window), 401 (lamp partially obscured by sofa o cs). 402 (plant in pot), and 403 (table between chairs o_ci and o_C2) from the object detection module 104 and the verification operation are used to identify locations where unknown objects may be located. Operations use defined relationships between the scene category, the common and uncommon objects, and the region proposals to determine provisional object classes for the unknown objects at least partially within the region proposals 400. The provisional object class (POC) may be stored in a vector or other data structure with an association to the respective unknown object. The POC can include a descriptive label (e.g., "chair") or other identifier of the object, and may include a confidence value which indicates a level of confidence that the descriptive label or other identifier is correct for the respective unknown object.

[0074] Figure 4e illustrates that the operations output to the application 108 and/or to another device, the known objects (both common and uncommon objects) and the POCs of the unknown objects.

[0075] Embodiments of the present disclosure are not limited to the particular operations and order of operations shown in Figure 2 and described above. These and other embodiments can include only some of the operations described above and without being limited to the examples above. Figure 5 illustrates a flowchart of more general operations that can be performed by an image processing device in accordance with some embodiments of the present disclosure. The operations of Figure 5 are described below with cross-reference to non-limiting examples that have been described above as operational steps in Figure 2. The cross-references to operational steps in Figure 2 are non-limiting examples and do not limit the scope of the operational steps in Figure 5.

[0076] Referring to Figure 5, an image processing device includes at least one processor that performs operations to determine 500 (e.g., 202 in Fig. 2) a scene category for an image generated by an image sensor. The operations detect 502 (e.g., 204 in Fig. 2) known objects in the image. The operations determine 504 (e.g., 208 in Fig. 2) region proposals in the image where unknown objects are possibly located. The operations determine 506 (e.g., 212 in Fig. 2) provisional object classes for unknown objects in the region proposals based on relationships between the scene category, the known objects, and the region proposals in the image. The operations provide 508 (e.g., 224 in Fig. 2) to an application the known objects and the provisional object classes for the objects in the region proposals.

[0077] In a further embodiment, the operations to determine 500 (e.g., 202 in Fig. 2) the scene category for the image, can include to recognize features and/or objects in the image, and then select the scene category from among a set of scene categories based on relating the recognized features and/or objects to defined features and/or objects which are associated with the selected scene category.

[0078] In a further embodiment, the operations to determine 500 (e.g., 202 in Fig. 2) the scene category for the image, can include to process the image through a neural network model to output the scene category, the neural network model having been trained to relate defined features and/or objects in images to defined scene categories.

[0079] In a further embodiment, the operations to determine 500 (e.g., 202 in Fig. 2) the scene category for the image, can include to select the scene category from among a set of scene categories based on metadata provided with the image that identifies at least one of: geographical location of the image sensor when the image was captured; direction of view of the image sensor when the image was captured; identifier of a Bluetooth device which was detected proximately in time to when the image was captured and which has a location known to the image processing device, and identifier of a WiFi device which was detected proximately in time to when the image was captured and which has a location known to the image processing device. [0080] In a further embodiment, the operations further include to obtain (e.g., 206 in Fig. 2) a scene-based-likely object set containing objects that are more likely to be found in images having the scene category, and identify (e.g., 206 in Fig. 2) a set of common objects containing objects among the known objects that satisfy a rule for matching objects of the scene-based-likely object set, wherein the identification of the set of common objects assigns object classes to the common objects. The operations can identify (e.g., 206 in Fig. 2) a set of uncommon objects containing objects among the known objects that do not satisfy the rule for matching objects of the scene-based-likely object set, wherein the identification of the set of uncommon objects assigns object classes to the uncommon objects. The operations can then determine 504 (e.g., 208 in Fig. 2) the region proposals where unknown objects in the image are possibly located, based on selecting regions of the image containing an object that is not in either of the set of common objects and the set of uncommon objects.

[0081] In a further embodiment, the operations further include to extract (e.g., 210 in Fig. 2) relationships between the scene category and objects having defined object classes. The operations then determine 506 (e.g., 212 in Fig. 2) the provisional object classes for the unknown objects in the region proposals based on applying the extracted relationships to identify relatedness between the sets of known and unknown objects having defined object classes and the unknown objects in the region proposals of the image.

[0082] The extracted relationships may define raster patterns in images of objects having defined object classes.

[0083] The extracted relationships may define spatial relationships between objects having defined object classes.

[0084] In a further embodiment, the operations to determine 506 (e.g., 212 in Fig. 2) the provisional object classes for the unknown objects in the region proposals, may include to obtain a first subset of the objects of the scene-based-likely object set that are not in a training set of objects used for the detection 502 (e.g., 204 in Fig. 2) of the known objects in the image. The operations determine a second subset of objects that are determined that are more likely to be next to the common objects in the first subset of the objects, e.g., satisfy a defined proximity rule. The operations extract spatial relationships between the set of common objects, the set of uncommon objects, the second set of objects, and the region proposals. The operations then determine the provisional object classes for the unknown objects in the region proposals based on comparing the spatial relationships to find a most dominant object class corresponding to each of the region proposals. For each of the regions proposals, the operations associate the associated most dominant object class with the corresponding region proposal.

[0085] In a further embodiment, the operations may select the first subset of the objects of the scene-based-likely object set based on satisfying a first rule for occurring with at least a first threshold probability for the scene category and based on not being in the training set of objects used for the detection 502 (e.g., 204 in Fig. 2) of the known objects in the image. [0086] In a further embodiment, based on determining 506 (e.g., 212 in Fig. 2) a provisional object class for one of the unknown objects in one of the region proposals, the operations further include to determine (e.g., 218 in Fig. 2) whether a feature vector of the one of the unknown object matches any of the feature vectors of any of the other unknown objects of the other region proposals. Responsive to determining (e.g., 218 in Fig. 2) at least one match, the operations transfer (e.g., 220 in Fig. 2) the provisional object class for the one of the unknown objects to each of the matched at least one other unknown object.

[0087] In a further embodiment, the operations further include to identify (e.g., 222 in Fig. 2) a pair of region proposals in the image that satisfy a rule defining allowable overlap between regions. The operations eliminate (e.g., 222 in Fig. 2) one of the region proposals of the pair and the associated provisional object class of the image in the one of the region proposals, from the provisional object classes for the objects in the region proposals that are provided 508 (e.g., 224 in Fig. 2) to the application.

[0088] Figure 6 illustrates components of an image processing device 50 which can be configured to operate according any one or more of the embodiments disclosed herein. The image processing device 50 can include at least one processor 600 (processor), at least one memory 610 (memory), at least one communication interface 620 (communication interface), and may further include an image sensor 100. The processor 600 is operationally connected to these various components. The memory 610 stores instructions that are executed by the processor 600 to perform operations and methods according to any one or more of the embodiments disclosed herein. The memory 610 may include the scene recognition module 102, the object detection module 104, the context-aware detection module 106, and may further include the application 108. The processor 600 may include one or more data processing circuits, such as a general purpose and/or special purpose processor (e.g., microprocessor and/or digital signal processor), which may be collocated or distributed across one or more data networks.

[0089] In accordance with the above discussion, operations process the image to assign a class to unknown objects using contextual information of the scene captured in the image. The operations are not limited to outputting the results of a close-set object detector, instead being able to further output the provisional classes for unknown objects without requiring training. This information can enhance the performance of several applications.

[0090] One example use case is for a semantic Simultaneous Localization and Mapping (SLAM) scenario, such as where a mobile device computes a map of the environment using 3D reconstruction, and compute a semantic understanding of the environment properties using scene/object detection, classification, and segmentation. In this task, the object information is used along with geometric and visual information to localize the device in the environment. Having more informative information about the environment can improve the SLAM process and knowing the common and uncommon objects can lead to trust more or less in the objects that will contribute to better decisions.

[0091] Another example use case is for semantic maps (e.g., 3D scene graphs). These graphs are hierarchical representations of the environment that collect various information, including semantics. Traditionally, the objects included in a 3D scene graph (3DSG) are limited to those detected by a conventional close-set object detector. Therefore, the POC information can be integrated into a 3DSG, enriching them with more object information and their relations. This can enable a more detailed understanding of the scene and can be helpful for other tasks such as navigation and path planning, among others, reducing inappropriate actions due to the unawareness of the existence of some objects.

[0092] Another example use case is for object detection in real-world scenarios. The present approach can be used to power conventional close-set object detectors in real -world environments. These detectors are limited by the number of objects they are trained on. Present embodiments can expand the object categories and allowing the application to have more information about the environment without the need to retrain.

[0093] Another example use case is for human-machine interaction. The operations can expose not only the information about the objects that the object detector can identify but also the provisional objects in the scene and the classification of common and uncommon objects, which can lead to more and better use/human interaction.

[0094] Another example use case is for object search task. The operations can be used with search strategies which typically rely on semantic information to predict the location of particular objects. The information on the provisional objects can help to guide the search for objects in the environment, making these strategies more efficient by reducing the search time and the explored area. [0095] Another example use case is for manipulation tasks. A mobile manipulator can benefit from various of the present embodiments, especially in cluttered environments with many and sometimes occluded objects. For instance, if the task requires grasping a fragile object which is close to other unknown objects, having provisional information about the unknown objects and their relationships can contribute to the strategy for a better approach to the object.

[0096] Further definitions and embodiments are now explained below.

[0097] In the above description of various embodiments of present inventive concepts, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of present inventive concepts. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which present inventive concepts belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense expressly so defined herein.

[0098] When an element is referred to as being "connected", "coupled", "responsive", or variants thereof to another element, it can be directly connected, coupled, or responsive to the other element or intervening elements may be present. In contrast, when an element is referred to as being "directly connected", "directly coupled", "directly responsive", or variants thereof to another element, there are no intervening elements present. Like numbers refer to like elements throughout. Furthermore, "coupled", "connected", "responsive", or variants thereof as used herein may include wirelessly coupled, connected, or responsive. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Well-known functions or constructions may not be described in detail for brevity and/or clarity. The term "and/or" includes any and all combinations of one or more of the associated listed items.

[0099] It will be understood that although the terms first, second, third, etc. may be used herein to describe various elements/operations, these elements/operations should not be limited by these terms. These terms are only used to distinguish one element/operation from another element/operation. Thus, a first element/operation in some embodiments could be termed a second element/operation in other embodiments without departing from the teachings of present inventive concepts. The same reference numerals or the same reference designators denote the same or similar elements throughout the specification. [00100] As used herein, the terms "comprise", "comprising", "comprises", "include", "including", "includes", "have", "has", "having", or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components or functions but does not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof. Furthermore, as used herein, the common abbreviation "e.g.", which derives from the Latin phrase "exempli gratia," may be used to introduce or specify a general example or examples of a previously mentioned item and is not intended to be limiting of such item. The common abbreviation "i.e.", which derives from the Latin phrase "id Est," may be used to specify a particular item from a more general recitation.

[00101] Example embodiments are described herein with reference to block diagrams and/or flowchart illustrations of computer-implemented methods, apparatus (systems and/or devices) and/or computer program products. It is understood that a block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by computer program instructions that are performed by one or more computer circuits. These computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and/or other programmable data processing circuit to produce a machine, such that the instructions, which execute via the processor of the computer and/or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions/acts specified in the block diagrams and/or flowchart block or blocks, and thereby create means (functionality) and/or structure for implementing the functions/acts specified in the block diagrams and/or flowchart block(s).

[00102] These computer program instructions may also be stored in a tangible computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer- readable medium produce an article of manufacture including instructions which implement the functions/acts specified in the block diagrams and/or flowchart block or blocks. Accordingly, embodiments of present inventive concepts may be embodied in hardware and/or in software (including firmware, resident software, micro-code, etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as "circuitry," "a module" or variants thereof. [00103] It should also be noted that in some alternate implementations, the functions/acts noted in the blocks may occur out of the order noted in the flowcharts. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Moreover, the functionality of a given block of the flowcharts and/or block diagrams may be separated into multiple blocks and/or the functionality of two or more blocks of the flowcharts and/or block diagrams may be at least partially integrated. Finally, other blocks may be added/inserted between the blocks that are illustrated, and/or blocks/operations may be omitted without departing from the scope of inventive concepts. Moreover, although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[00104] Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present inventive concepts. All such variations and modifications are intended to be included herein within the scope of present inventive concepts. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments, which fall within the spirit and scope of present inventive concepts. Thus, to the maximum extent allowed by law, the scope of present inventive concepts is to be determined by the broadest permissible interpretation of the present disclosure including the following examples of embodiments and their equivalents and shall not be restricted or limited by the foregoing detailed description. [00105] The enumerated references made above are defined below:

[00106] [1] Li, D„ Tasci, S„ Ghosh, S„ Zhu, J., Zhang, J., & Heck, L. (2019,

November). RILOD: Near real-time incremental learning for object detection at the edge. In Proceedings of the 4th ACM/IEEE Symposium on Edge Computing (pp. 113-126).

[00107] [2] Sonogashira, M., liyama, M., & Kawanishi, Y. (2022). Towards Open-

Set Scene Graph Generation With Unknown Objects. IEEE Access, 10, 11574-11583.

[00108] [3] Zheng, J., Li, W„ Hong, J., Petersson, L„ & Barnes, N. (2022).

Towards Open-Set Object Detection and Discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3961-3970).

[00109] [4] Joseph, K. J., Khan, S., Khan, F. S., & Balasubramanian, V. N. (2021).

Towards open world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5830-5840) [00110] [5] Rahman, S., Khan, S., & Porikli, F. (2018). Zero-shot object detection:

Learning to simultaneously recognize and localize novel concepts. In Asian Conference on Computer Vision (pp. 547-563). Springer, Cham.

[00111] [6] Luo, R., Zhang, N., Han, B., & Yang, L. (2020, April). Context-aware zero-shot recognition. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 07, pp. 11709-11716).

[00112] [7] Nie, H., Wang, R., & Chen, X. (2022). From Node to Graph: Joint

Reasoning on Visual-Semantic Relational Graph for Zero-Shot Detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1109-1118).

[00113] [8] Simonyan, K., & Zisserman, A. (2015, April 10). Very Deep

Convolutional Networks for Large-Scale Image Recognition. arXiv.

[00114] [9] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D.,

Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. CV- Foundation.

[00115] [10] Bochkovskiy, A., Wang, C.-Y., & Liao, H.-Y. M. (2020, April 23).

Yolov4: Optimal Speed and accuracy of object detection. arXiv.

[00116] [11] Ren, S„ He, K„ Girshick, R„ & Sun, J. (2015). Faster R-CNN:

Towards Real-Time Object Detection with Region Proposal Networks. NeurlPS.

[00117] [12] Zeng, D„ Liao, M„ Tavakolian, M„ Guo, Y„ Zhou, B„ Hu, D„ ... &

Liu, L. (2021). Deep learning for scene classification: A survey. arXiv preprint arXiv:2101.10531.

Claims

CLAIMS:

1. An image processing device (50) comprising at least one processor (600) operative to: determine (202,500) a scene category for an image generated by an image sensor (100); detect (204,502) known objects in the image; determine (208,504) region proposals in the image where unknown objects are possibly located; determine (212,506) provisional object classes for unknown objects in the region proposals based on relationships between the scene category, the known objects, and the region proposals in the image; and provide (224,508) to an application the known objects and the provisional object classes for the objects in the region proposals.

2. The image processing device (50) of Claim 1, wherein to determine (202,500) the scene category for the image, the at least one processor (600) is further operative to: recognize features and/or objects in the image; and select the scene category from among a set of scene categories based on relating the recognized features and/or objects to defined features and/or objects which are associated with the selected scene category.

3. The image processing device (50) of any of Claims 1 to 2, wherein to determine (202,500) the scene category for the image, the at least one processor (600) is further operative to: process the image through a neural network model to output the scene category, the neural network model having been trained to relate defined features and/or objects in images to defined scene categories.

4. The image processing device (50) of any of Claims 1 to 3, wherein to determine (202,500) the scene category for the image, the at least one processor (600) is further operative to: select the scene category from among a set of scene categories based on metadata provided with the image that identifies at least one of: geographical location of the image sensor when the image was captured; direction of view of the image sensor when the image was captured; identifier of a Bluetooth device which was detected proximately in time to when the image was captured and which has a location known to the image processing device, and identifier of a WiFi device which was detected proximately in time to when the image was captured and which has a location known to the image processing device.

5. The image processing device (50) of any of Claims 1 to 4, wherein the at least one processor (600) is further operative to: detect known objects in the image using a close-set object detector.

6. The image processing device (50) of any of Claims 1 to 5, wherein the at least one processor (600) is further operative to: obtain (206) a scene-based-likely object set containing objects that are more likely to be found in images having the scene category; identify (206) a set of common objects containing objects among the known objects that satisfy a rule for matching objects of the scene-based-likely object set, wherein the identification of the set of common objects assigns object classes to the common objects; and identify (206) a set of uncommon objects containing objects among the known objects that do not satisfy the rule for matching objects of the scene-based-likely object set, wherein the identification of the set of uncommon objects assigns object classes to the uncommon objects; and determine (208,504) the region proposals where unknown objects in the image are possibly located, based on selecting regions of the image containing an object that is not in either of the set of common objects and the set of uncommon objects.

7. The image processing device (50) of Claim 6, wherein the at least one processor (600) is further operative to: extract (210) relationships between the scene category and objects having defined object classes; determine (212,506) the provisional object classes for the unknown objects in the region proposals based on applying the extracted relationships to identify relatedness between the sets of known and unknown objects having defined object classes and the unknown objects in the region proposals of the image.

8. The image processing device (50) of Claim 7, wherein the extracted relationships define raster patterns in images of objects having defined object classes.

9. The image processing device (50) of any of Claims 7 to 8, wherein the extracted relationships define spatial relationships between objects having defined object classes.

10. The image processing device (50) of any of Claims 7 to 9, wherein to determine (212,506) the provisional object classes for the unknown objects in the region proposals based on applying the extracted relationships to identify relatedness between the sets of known and unknown objects having defined object classes and the unknown objects in the region proposals of the image, the at least one processor (600) is further operative to: obtain a first subset of the objects of the scene-based-likely object set that are not in a training set of objects used for the detection (204,502) of the known objects in the image; determine a second subset of objects that are determined that are more likely to be next to the common objects in the first subset of the objects; extract spatial relationships between the set of common objects, the set of uncommon objects, the second set of objects, and the region proposals; determine the provisional object classes for the unknown objects in the region proposals based on comparing the spatial relationships to find a most dominant object class corresponding to each of the region proposals; and for each of the regions proposals, associate the most dominant object class with the corresponding region proposal.

11. The image processing device (50) of Claim 10, wherein: the first subset of the objects of the scene-based-likely object set are selected based on satisfying a first rule for occurring with at least a first threshold probability for the scene category and based on not being in the training set of objects used for the detection (204,502) of the known objects in the image..

12. The image processing device (50) of any of Claims 1 to 11, the at least one processor (600) is further operative to: based on determining (212,506) a provisional object class for one of the unknown objects in one of the region proposals, determining (218) whether a feature vector of the one of the unknown object matches any of the feature vectors of any of the other unknown objects of the other region proposals, and responsive to determining (218) at least one match, transferring (220) the provisional object class for the one of the unknown objects to each of the matched at least one other unknown object.

13. The image processing device (50) of any of Claims 1 to 12, wherein the at least one processor (600) is further operative to: identify (222) a pair of region proposals in the image that satisfy a rule defining allowable overlap between regions; and eliminate (222) one of the region proposals of the pair and the associated provisional object class of the image in the one of the region proposals, from the provisional object classes for the objects in the region proposals that are provided (224,508) to the application.

14. A method by an image processing device comprising: determining (202,500) a scene category for an image generated by an image sensor; detecting (204,502) known objects in the image; determining (208,504) region proposals in the image where unknown objects are possibly located; determining (212,506) provisional object classes for unknown objects in the region proposals based on relationships between the scene category, the known objects, and the region proposals in the image; and providing (224,508) to an application the known objects and the provisional object classes for the objects in the region proposals.

15. The method of Claim 14, wherein the determining (202,500) of the scene category for the image, comprises: recognizing features and/or objects in the image; and selecting the scene category from among a set of scene categories based on relating the recognized features and/or objects to defined features and/or objects which are associated with the selected scene category.

16. The method of any of Claims 14 to 15, wherein the determining (202,500) of the scene category for the image, comprises: processing the image through a neural network model to output the scene category, the neural network model having been trained to relate defined features and/or objects in images to defined scene categories.

17. The method of any of Claims 14 to 16, wherein the determining (202,500) of the scene category for the image, comprises: selecting the scene category from among a set of scene categories based on metadata provided with the image that identifies at least one of: geographical location of the image sensor when the image was captured; direction of view of the image sensor when the image was captured; identifier of a Bluetooth device which was detected proximately in time to when the image was captured and which has a location known to the image processing device, and identifier of a WiFi device which was detected proximately in time to when the image was captured and which has a location known to the image processing device.

18. The method of any of Claims 14 to 17, wherein the detecting (204,502) of known objects in the image uses a close-set object detector.

19. The method of any of Claims 14 to 18, further comprising: obtaining (206) a scene-based-likely object set containing objects that are more likely to be found in images having the scene category; identifying (206) a set of common objects containing objects among the known objects that satisfy a rule for matching objects of the scene-based-likely object set, wherein the identification of the set of common objects assigns object classes to the common objects; and identifying (206) a set of uncommon objects containing objects among the known objects that do not satisfy the rule for matching objects of the scene-based-likely object set, wherein the identification of the set of uncommon objects assigns object classes to the uncommon objects; and determining (208,504) the region proposals where unknown objects in the image are possibly located, based on selecting regions of the image containing an object that is not in either of the set of common objects and the set of uncommon objects.

20. The method of Claim 19, further comprising: extracting (210) relationships between the scene category and objects having defined object classes; determining (212,506) the provisional object classes for the unknown objects in the region proposals based on applying the extracted relationships to identify relatedness between the sets of known and unknown objects having defined object classes and the unknown objects in the region proposals of the image.

21. The method of Claim 20, wherein the extracted relationships define raster patterns in images of objects having defined object classes.

22. The method of any of Claims 20 to 21, wherein the extracted relationships define spatial relationships between objects having defined object classes.

23. The method of any of Claims 20 to 22, wherein the determining (212,506) of the provisional object classes for the unknown objects in the region proposals based on applying the extracted relationships to identify relatedness between the sets of known and unknown objects having defined object classes and the unknown objects in the region proposals of the image, comprises: obtaining a first subset of the objects of the scene-based-likely object set that are not in a training set of objects used for the detection (204,502) of the known objects in the image; determining a second subset of objects that are determined that are more likely to be next to the common objects in the first subset of the objects; extracting spatial relationships between the set of common objects, the set of uncommon objects, the second set of objects, and the region proposals; determining the provisional object classes for the unknown objects in the region proposals based on comparing the spatial relationships to find a most dominant object class corresponding to each of the region proposals; and for each of the regions proposals, associating the most dominant object class with the corresponding region proposal.

24. The method of Claim 23, wherein: the first subset of the objects of the scene-based-likely object set are selected based on satisfying a first rule for occurring with at least a first threshold probability for the scene category and based on not being in the training set of objects used for the detection (204,502) of the known objects in the image.

25. The method of any of Claims 14 to 24, further comprising: based on determining (212,506) a provisional object class for one of the unknown objects in one of the region proposals, determining (218) whether a feature vector of the one of the unknown object matches any of the feature vectors of any of the other unknown objects of the other region proposals, and responsive to determining (218) at least one match, transferring (220) the provisional object class for the one of the unknown objects to each of the matched at least one other unknown object.

26. The method of any of Claims 14 to 25, further comprising: identifying (222) a pair of region proposals in the image that satisfy a rule defining allowable overlap between regions; and eliminating (222) one of the region proposals of the pair and the associated provisional object class of the image in the one of the region proposals, from the provisional object classes for the objects in the region proposals that are provided (224,508) to the application.