EP4445331A1

EP4445331A1 - Object detection device, object detection method, and object detection system

Info

Publication number: EP4445331A1
Application number: EP21967291.2A
Authority: EP
Inventors: ManYiu CHOW; Mitsuhiro Kitani; Jun Koike; Atsushi Oda
Original assignee: Hitachi Ltd
Current assignee: Hitachi Ltd
Priority date: 2021-12-10
Filing date: 2021-12-10
Publication date: 2024-10-16
Also published as: WO2023105800A1; EP4445331A4; JP7660264B2; JP2024545451A

Abstract

Provided is an object detection device that is capable of detecting occluded objects with a high degree of accuracy. The object detection device includes an image acquisition unit for receiving a target image that includes a first target object, an image segmentation unit for identifying an occluded area of the target image in which the first target object is occluded by a first occluding object, an image conversion unit for generating a first recovery image by converting the occluded area of the target image to a first recovery mask, a generator unit for generating a first recovered image in which the first recovery mask has been replaced with a first predicted image, and an object detection unit for detecting the first target object and generating an object detection result.

Description

OBJECT DETECTION DEVICE, OBJECT DETECTION METHOD, AND OBJECT DETECTION SYSTEM

The present disclosure generally relates to an object detection device, object detection method, and object detection system.

In recent years, with the continued growth of information technology and Internet of Things (IoT), sensor devices are increasingly used to collect data in a variety of fields. This collected data can then be analyzed to detect and measure various physical phenomena and gather valuable insights.

One aspect of information technology and IoT systems relates to the detection and tracking of objects in images collected by cameras. The detection and tracking of objects has applications in the surveillance, healthcare, retail, and transportation domains. Objects of interest could include, for instance, humans, animals, vehicles, packages, or the like. However, in some cases, the object of interest in an image may only be partially visible due to occlusion by other objects. In such cases, it is difficult for existing object detection methods to provide accurate object detection results.

Accordingly, in view of such challenges, methods have been proposed for detecting occluded objects in images.
As an example, US10002309B2 (Patent Document 1) discloses “A method includes the following steps. A video sequence including detection results from one or more detectors is received, the detection results identifying one or more objects. A clustering framework is applied to the detection results to identify one or more clusters associated with the one or more objects. The clustering framework is applied to the video sequence on a frame-by-frame basis. Spatial and temporal information for each of the one or more clusters are determined. The one or more clusters are associated to the detection results based on the spatial and temporal information in consecutive frames of the video sequence to generate tracking information. One or more target tracks are generated based on the tracking information for the one or more clusters. The one or more target tracks are consolidated to generate refined tracks for the one or more objects.”

Additionally, KR102090487B1 (Patent Document 2) discloses “Disclosed is an object detection apparatus and method using a fusion sensor. A method of detecting an object using a fusion sensor includes collecting rider data and radar data using a rider sensor and a radar sensor for a search area to detect an object; Extracting objects present in the search area based on the collected lidar data and radar data, respectively; Setting an area of interest of the lidar sensor using the collected lidar data, and generating an occluded depth using an object extracted based on the lidar data; Setting a region of interest of the radar sensor using the collected radar data; And determining whether an object existing in the search area is a moving object by using an area of interest or occlusion depth of the lidar sensor and an area of interest of the radar sensor.”

US10,002,309B2 KR102090487B1

Patent Document 1 discloses a technique in which, once a target object has been correctly identified in a particular frame, a clustering method is used to track the target object through subsequent frames in which it may be occluded. The clustering method of Patent Document 1 requires the collection of a video sequence, and a cluster is associated with the target object based on the spatial and temporal information of the target object in consecutive frames.

The technique disclosed in Patent Document 1, however, depends on first correctly identifying the target object in a frame in which it is not occluded before it can be tracked through subsequent frames in which it may be occluded. Accordingly, the technique disclosed in Patent Document 1 is not suitable in cases in which a video sequence in which the target object is not occluded in one or more frames is not available. Further, as the technique disclosed in Patent Document 1 performs clustering based on the temporal and spatial relationship of the target object in subsequent frames of the video sequence, clustering cannot be accurately performed in situations where the image capture rate is slow, as the distance of the target object between subsequent frames may be too large for cluster association.

Patent Document 2 discloses a sensor fusion method to determine the depth of partially occluded objects based on both radar and LiDAR measurements.

The technique disclosed in Patent Document 2, however, focuses on determining the depth of partially occluded objects, and is limited in its ability to detect the class of the target object. Accordingly, the technique of Patent Document 2 is not suitable for situations in which accurate class identification of target objects is important, such as automobile or railroad applications in which the risk of a particular object to automobile or train safety may depend on the class of the object (e.g., dynamic objects may pose a greater risk than static objects).

Accordingly, it is an object of the present disclosure to provide a device, method, and system for object detection that is capable of detecting occluded objects with a high degree of accuracy.

One representative example of the present disclosure relates to an object detection device including an image acquisition unit for receiving a target image that at least includes a first target object; an image segmentation unit for processing the target image to identify an occluded area of the target image in which at least the first target object is at least partially occluded by a first occluding object, and assign a first object label to the first occluding object that indicates an object class of the first occluding object; an image conversion unit for generating a first recovery image by converting the occluded area of the target image to a first recovery mask based on the first object label; a generator unit for processing the first recovery image to generate a first recovered image in which the first recovery mask has been replaced with a first predicted image that includes at least a first predicted object associated with a second object label that indicates an object class corresponding to both the first predicted object and the first target object; and an object detection unit for processing the first recovered image to detect the first target object and generating an object detection result that includes at least a location of the first target object and a target object label indicating an object class of the first target object.

According to the present disclosure it is possible to provide a device, method, and system for object detection that is capable of detecting occluded objects with a high degree of accuracy.

Problems, configurations, and effects other than those described above will be made clear by the following description of the embodiments for carrying out the invention.

FIG. 1 illustrates an example computing architecture for executing the embodiments of the present disclosure. FIG. 2 illustrates an example configuration of an object detection system, according to embodiments. FIG. 3 illustrates a block diagram of an inference phase process of the object detection system, according to embodiments. FIG. 4 illustrates a block diagram of a training phase process of the object detection system, according to embodiments. FIG. 5 illustrates a block diagram of an image segmentation training process, according to embodiments. FIG. 6 illustrates a block diagram of a set of transformation units, according to embodiments. FIG. 7 illustrates an example of a target image, a recovery image, and a recovered image, according to embodiments.

Description of Embodiment(s)

Hereinafter, embodiments of the present invention will be described with reference to the Figures. It should be noted that the embodiments described herein are not intended to limit the invention according to the claims, and it is to be understood that each of the elements and combinations thereof described with respect to the embodiments are not strictly necessary to implement the aspects of the present invention.
Various aspects are disclosed in the following description and related drawings. Alternate aspects may be devised without departing from the scope of the disclosure. Additionally, well-known elements of the disclosure will not be described in detail or will be omitted so as not to obscure the relevant details of the disclosure.
The words “exemplary” and/or “example” are used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” and/or “example” is not necessarily to be construed as preferred or advantageous over other aspects. Likewise, the term “aspects of the disclosure” does not require that all aspects of the disclosure include the discussed feature, advantage, or mode of operation.
Further, many aspects are described in terms of sequences of actions to be performed by, for example, elements of a computing device. It will be recognized that various actions described herein can be performed by specific circuits (e.g., an application specific integrated circuit (ASIC)), by program instructions being executed by one or more processors, or by a combination of both. Additionally, the sequence of actions described herein can be considered to be embodied entirely within any form of computer readable storage medium having stored therein a corresponding set of computer instructions that upon execution would cause an associated processor to perform the functionality described herein. Thus, the various aspects of the disclosure may be embodied in a number of different forms, all of which have been contemplated to be within the scope of the claimed subject matter.
Aspects of the disclosure relate to utilizing semantic segmentation to identify an occluding object that occludes a target object, replacing the area of the occluding object with a recovery mask, and subsequently utilizing a generative adversarial neural network (GAN) to generate a recovered image in which the recovery mask has been replaced with a predicted image that indicates at least a predicted object that shares an object class with the target object. By performing object detection with respect to this recovered image, it is possible to achieve a highly accurate object detection result.
More particularly, in order to recognize the occluded area in an image, high-resolution semantic segmentation may be utilized to identify the occluding object that occludes the target object. Here, in the case of a railroad or automotive application, examples of the occluding object may include electric poles, trees, and other barriers, and the target object may include people, animals, automobiles or the like. Accordingly, as used in the present disclosure, “occluded area” refers to the area in the image that corresponds to the occluding object identified by semantic segmentation.
In a training phase, the occluded area identified using the high-resolution semantic segmentation may be used to estimate model weights for the occluded object recovery ability of a patch label classifier, a discriminator and a generator in a GAN. Additionally, in an inference phase, the occluded area can be used to guide a GAN to generate a predicted image to replace the occluded area. Here, the occluded area may be represented using a binary recover mask.
Further, during a training phase, the generator in the GAN may include global/ patch discriminators and an additional patch label classifier to guide the training of the generator to recover the object class of the target object with greater accuracy. More particularly, these global/ patch discriminators may receive the occluded area identified using the semantic segmentation to generalize the model for the occluded area from a set of labels, and guide the generator to generate a realistic predicted image to replace the occluded area.
Further, in embodiments, aspects of the disclosure relate to utilizing a plurality of transformation units (e.g., additional GANs) to enhance the quality and modify features (e.g., weather conditions, lighting conditions) of input images. The outputs from these transformation units may be weighted and combined into a single image using a set of convolution layers. By fusing the results of multiple transformation units, it is possible to balance the weights of each transformation unit and reduce the likelihood of false detection.
In this way, according to the present disclosure, it is possible to provide a device, method, and system for object detection that is capable of detecting occluded objects with a high degree of accuracy.
Turning now to the Figures, FIG. 1 depicts a high-level block diagram of a computer system 100 for implementing various embodiments of the present disclosure, according to embodiments. The mechanisms and apparatus of the various embodiments disclosed herein apply equally to any appropriate computing system. The major components of the computer system 100 include one or more processors 102, a memory 104, a terminal interface 112, a storage interface 113, an I/O (Input/Output) device interface 114, and a network interface 115, all of which are communicatively coupled, directly or indirectly, for inter-component communication via a memory bus 106, an I/O bus 108, bus interface unit 109, and an I/O bus interface unit 110.
The computer system 100 may contain one or more general-purpose programmable central processing units (CPUs) 102A and 102B, herein generically referred to as the processor 102. In embodiments, the computer system 100 may contain multiple processors; however, in certain embodiments, the computer system 100 may alternatively be a single CPU system. Each processor 102 executes instructions stored in the memory 104 and may include one or more levels of on-board cache.
In embodiments, the memory 104 may include a random-access semiconductor memory, storage device, or storage medium (either volatile or non-volatile) for storing or encoding data and programs. In certain embodiments, the memory 104 represents the entire virtual memory of the computer system 100, and may also include the virtual memory of other computer systems coupled to the computer system 100 or connected via a network. The memory 104 can be conceptually viewed as a single monolithic entity, but in other embodiments the memory 104 is a more complex arrangement, such as a hierarchy of caches and other memory devices. For example, memory may exist in multiple levels of caches, and these caches may be further divided by function, so that one cache holds instructions while another holds non-instruction data, which is used by the processor or processors. Memory may be further distributed and associated with different CPUs or sets of CPUs, as is known in any of various so-called non-uniform memory access (NUMA) computer architectures.
The memory 104 may store all or a portion of the various programs, modules and data structures for processing data transfers as discussed herein. For instance, the memory 104 can store an object detection application 150. In embodiments, the object detection application 150 may include instructions or statements that execute on the processor 102 or instructions or statements that are interpreted by instructions or statements that execute on the processor 102 to carry out the functions as further described below.
In certain embodiments, the object detection application 150 is implemented in hardware via semiconductor devices, chips, logical gates, circuits, circuit cards, and/or other physical hardware devices in lieu of, or in addition to, a processor-based system. In embodiments, the object detection application 150 may include data in addition to instructions or statements. In certain embodiments, a camera, sensor, or other data input device (not shown) may be provided in direct communication with the bus interface unit 109, the processor 102, or other hardware of the computer system 100. In such a configuration, the need for the processor 102 to access the memory 104 and the object detection application 150 may be reduced.
The computer system 100 may include a bus interface unit 109 to handle communications among the processor 102, the memory 104, a display system 124, and the I/O bus interface unit 110. The I/O bus interface unit 110 may be coupled with the I/O bus 108 for transferring data to and from the various I/O units. The I/O bus interface unit 110 communicates with multiple I/O interface units 112, 113, 114, and 115, which are also known as I/O processors (IOPs) or I/O adapters (IOAs), through the I/O bus 108. The display system 124 may include a display controller, a display memory, or both. The display controller may provide video, audio, or both types of data to a display device 126. Further, the computer system 100 may include one or more sensors or other devices configured to collect and provide data to the processor 102.
As examples, the computer system 100 may include biometric sensors (e.g., to collect heart rate data, stress level data), environmental sensors (e.g., to collect humidity data, temperature data, pressure data), motion sensors (e.g., to collect acceleration data, movement data), or the like. Other types of sensors are also possible. The display memory may be a dedicated memory for buffering video data. The display system 124 may be coupled with a display device 126, such as a standalone display screen, computer monitor, television, or a tablet or handheld device display.
In one embodiment, the display device 126 may include one or more speakers for rendering audio. Alternatively, one or more speakers for rendering audio may be coupled with an I/O interface unit. In alternate embodiments, one or more of the functions provided by the display system 124 may be on board an integrated circuit that also includes the processor 102. In addition, one or more of the functions provided by the bus interface unit 109 may be on board an integrated circuit that also includes the processor 102.
The I/O interface units support communication with a variety of storage and I/O devices. For example, the terminal interface unit 112 supports the attachment of one or more user I/O devices 116, which may include user output devices (such as a video display device, speaker, and/or television set) and user input devices (such as a keyboard, mouse, keypad, touchpad, trackball, buttons, light pen, or other pointing device). A user may manipulate the user input devices using a user interface in order to provide input data and commands to the user I/O device 116 and the computer system 100, and may receive output data via the user output devices. For example, a user interface may be presented via the user I/O device 116, such as displayed on a display device, played via a speaker, or printed via a printer.
The storage interface 113 supports the attachment of one or more disk drives or direct access storage devices 117 (which are typically rotating magnetic disk drive storage devices, although they could alternatively be other storage devices, including arrays of disk drives configured to appear as a single large storage device to a host computer, or solid-state drives, such as flash memory). In some embodiments, the storage device 117 may be implemented via any type of secondary storage device. The contents of the memory 104, or any portion thereof, may be stored to and retrieved from the storage device 117 as needed. The I/O device interface 114 provides an interface to any of various other I/O devices or devices of other types, such as printers or fax machines. The network interface 115 provides one or more communication paths from the computer system 100 to other digital devices and computer systems; these communication paths may include, for example, one or more networks 130.
Although the computer system 100 shown in FIG. 1 illustrates a particular bus structure providing a direct communication path among the processors 102, the memory 104, the bus interface 109, the display system 124, and the I/O bus interface unit 110, in alternative embodiments the computer system 100 may include different buses or communication paths, which may be arranged in any of various forms, such as point-to-point links in hierarchical, star or web configurations, multiple hierarchical buses, parallel and redundant paths, or any other appropriate type of configuration. Furthermore, while the I/O bus interface unit 110 and the I/O bus 108 are shown as single respective units, the computer system 100 may, in fact, contain multiple I/O bus interface units 110 and/or multiple I/O buses 108. While multiple I/O interface units are shown which separate the I/O bus 108 from various communications paths running to the various I/O devices, in other embodiments, some or all of the I/O devices are connected directly to one or more system I/O buses.
In various embodiments, the computer system 100 is a multi-user mainframe computer system, a single-user system, or a server computer or similar device that has little or no direct user interface, but receives requests from other computer systems (clients). In other embodiments, the computer system 100 may be implemented as a desktop computer, portable computer, laptop or notebook computer, tablet computer, pocket computer, telephone, smart phone, or any other suitable type of electronic device.
Next, an example configuration of an object detection system according to the embodiments of the present disclosure will be described with reference to FIG. 2.
FIG. 2 illustrates an example configuration of an object detection system 200, according to the embodiments of the present disclosure. As illustrated in FIG. 2, the object detection system 200 primarily includes an image capture device 210, a client device 220, and an object detection device 230. The image capture device 210, the client device 220, and the object detection device 230 may be communicably connected via a communication network 225 such as a local area network (LAN) or the Internet.
The image capture device 210 is device configured to capture image data. The image data captured by the image capture device may include still images or videos (that is, a sequence of chronological image frames). As examples, the image capture device 210 may include a stationary camera (e.g., a surveillance camera) mounted in a predetermined location, a camera mounted on an automobile or a train, a camera included in a mobile computing device (e.g., the camera in a smartphone or tablet) or the like. In embodiments, the image capture device may be configured to capture a target image 212 that includes a target object and transmit it to the object detection device 230 via the communication network 225.
The client device 220 may include a computing device for managing the object detection process performed by the object detection device 230 and viewing and confirming an object detection result 222 transmitted from the object detection device 230 as a result of the object detection process. In embodiments, the client device 220 may be used to designate target objects for detection, set parameters of the object detection device 230, or the like. As examples, the client device 220 may include a smartphone, tablet device, laptop computer, desktop computer, or other suitable computing device.
The object detection device 230 is a computing device configured to perform the object detection process according to the embodiments of the present disclosure in order to detect a target object that is at least partially occluded by an occluded object. As illustrated in FIG. 2, the object detection device 230 may include an image acquisition unit 232, an image segmentation unit 234, an image conversion unit 236, a generator unit 238, and an object detection unit 240. In embodiments, the object detection device 230 may be implemented using the computer system 100 illustrated in FIG. 1, such that the functions of the image acquisition unit 232, the image segmentation unit 234, the image conversion unit 236, the generator unit 238, the object detection unit 240, and other functions of the object detection device 230 are carried out by means of the object detection application 150.
It should be noted that, while an example of the object detection device 230 including the image acquisition unit 232, the image segmentation unit 234, the image conversion unit 236, the generator unit 238, and the object detection unit 240 is illustrated in FIG. 2, the present disclosure is not limited hereto, and other functional units (for example, functional units used in the training process) of the object detection device 230 will be described herein.
The image acquisition unit 232 is a functional unit configured to receive the target image 212 captured by the image capture device 210. Here, the target image 212 may include an image in which a target object is at least partially occluded by an occluding object. Herein, an example will be described in which the target image 212 is a 3-channel RGB image, but the present disclosure is not limited thereto. As an example, in the case that the image capture device 210 is a camera mounted on a driverless train, the image capture device 210 may capture a target image 212 of the surroundings of the driverless train that includes a target object for which class identification is desirable. Here, the occluding object refers to any object that partially blocks, hides, or obscures the target object in the target image 212. In embodiments, it is preferable that the occluding object block less than or equal to 30% of the target object in the target image 212. With reference to the driverless train example, the occluding object may include a crossing device, an electric pole, a tree, a building, or any other object that partially obscures the target object.
The image segmentation unit 234 is a functional unit configured to process the target image acquired by the image acquisition unit 232 to identify an occluded area of the target image 212 in which the target object is at least partially occluded by an occluding object, and assign an object label to the occluding object that indicates the object class of the occluding object. Here, an object class refers to the type classification of an object (e.g., person, dog, tree), and an object label refers to a metadata tag that indicates an object class. In embodiments, the function of the image segmentation unit 234 may be realized using a suitable semantic segmentation technique. As examples, the image segmentation unit 234 may include a fully convolutional network, DeepLab, Atrous convolution, Spatial Pyramidal Pooling, a global convolutional network, a spatio-temporal fully convolutional network, or the like.
The image conversion unit 236 is a functional unit configured to generate a recovery image by converting the occluded area of the target image 212 to a recovery mask based on the object label of the occluding object.
The generator unit 238 is a functional unit for processing the recovery image generated by the image conversion unit 236 to generate a recovered image in which the recovery mask has been replaced with a predicted image. Here, the predicted image is placed at substantially the same location and occupies substantially the same area as the recovery mask (e.g., the same area as the occluded area). In embodiments, as will be described later, the generator unit 238 may include the trained generator of a generative adversarial network (GAN).
The object detection unit 240 is a functional unit configured to process the recovered image generated by the generator unit 238 to detect the target object and generate an object detection result 222. The function of the object detection unit 240 may be realized using any suitable object detection technique. As examples, the object detection unit 240 may include Histogram of Oriented Gradients (HOG), Region-based Convolutional Neural Networks (R-CNN), Fast R-CNN, Faster R-CNN, Region-based Fully Convolutional Network (R-FCN), Single Shot Detector (SSD), Spatial Pyramid Pooling (SPP-net), YOLO (You Only Look Once) or the like.
Next, a block diagram of an inference phase process of the object detection system will be described with reference to FIG. 3.
In embodiments, aspects of the disclosure relate to a training phase process 400 in which the object detection device 230 is trained to achieve a high degree of object detection accuracy and an inference phase process 300 in which the trained object detection device 230 is used to generate object detection results with respect to a target image. Accordingly, FIG. 3 illustrates a block diagram of the inference phase process 300 of the object detection system 200, according to embodiments.
First, at Step S310, the image segmentation unit 234 receives the target image 212 captured by the image acquisition unit 232 (not shown in FIG. 3) and processes the target image 212 to identify an occluded area of the target image 212 in which the target object is at least partially occluded by an occluding object, and assigns an object label to the occluding object. Here, the occluded area of the target image 212 refers to the entire area of the occluding object. The object label refers to a data label that identifies the predicted class of the occluding object. In embodiments, the object label may also designate the location of the occluding object in the image. As an example, in the case that the target image includes a target object of a person that is partially occluded by an electric pole, at Step S310 the image segmentation unit 234 may identify the entire area of the electric pole as the occluded area, and assign a first object label of “Electric Pole” to the electric pole.
Next, at Step S320, the image conversion unit 236 generates a recovery image 325 by converting the occluded area of the target image 212 to a recovery mask based on the object label assigned at Step S310. Here, the recovery mask refers to an image used to indicate and isolate the region of the target image 212 to be processed by the generator unit 238 in Step S330. In embodiments, the recovery mask may include a binary image in which those pixels that correspond to the occluded area are indicated by a first pixel value (e.g., “1”) and those pixels that do not correspond to the occluded area are indicated by a second pixel value (e.g., “0”).
Next, at Step S330, the generator unit 238 processes the recovery image 325 generated by the image conversion unit 236 in Step S320 to generate a recovered image 335 in which the recovery mask has been replaced with a predicted image. Here, the predicted image refers to an artificially generated image to replace the recovery mask (that, is, the occluding area). The predicted image includes a predicted object that shares an object class with the target object. In the case that the region occluded by the occluding object in the target image 212 includes a plurality of occluded objects having different object classes, the predicted image may include a plurality of predicted objects having object classes that respectively correspond to the occluded objects.
As an example, in the case that the target image 212 includes an image in which an occluding object occludes a person, an automobile, the ground, and the sky, the generator unit 238 may generate a predicted image in which predicted objects of a person, an automobile, the ground, and the sky replace the recovery mask (that, is, the occluding area).
Next, at Step S340, the object detection unit 240 processes the recovered image 335 generated by the generator unit 238 at Step S330 to detect the target object and generate an object detection result 222. Here, the object detection result 222 may include a collection of data that indicates at least a target object label that indicates the object class of the target object. In embodiments, the object detection result 222 may include additional information regarding the target object, such as the location of the target object in the image, the trajectory (e.g., predicted path of motion) of the target object, or the like. Subsequently, the object detection result 222 may be transmitted to the client device 220 via the communication network 225.
According to the inference phase process 300 of the object detection system 200 illustrated in FIG. 3, it is possible to provide a method capable of detecting occluded objects with a high degree of accuracy.
Next, a block diagram of a training phase process of the object detection system will be described with reference to FIG. 4.
In embodiments, aspects of the disclosure relate to a training phase process 400 in which the object detection device 230 is trained to achieve a high degree of object detection accuracy and an inference phase process 300 in which the trained object detection device 230 is used to generate object detection results with respect to a target image. Accordingly, FIG. 4 illustrates a block diagram of a training phase process 400 of the object detection system, according to embodiments.
First, at Step S405, the image segmentation unit 234 is used to process a training image 401 to identify occluding objects (e.g., those objects that partially occlude a target object) in the training image 401. The image segmentation unit 234 may output a set of object labels 407 that indicate the object class and location information for the occluding objects in the training image 401 (e.g., coordinates indicating the occluding area corresponding to the occluding object). Here, the training image 510 may include an image of a scene in which a target object is occluded by an occluding object. In addition, at Step S405, it is assumed that the image segmentation unit 234 has already been trained to accurately identify occluding objects.
In addition, the image segmentation unit 235 may generate a set of pixel classification data 414 by performing a function of labeling each pixel of the training image as “real” or “fake,” such that each pixel corresponding to the occluding area is labeled as “fake” and each pixel not corresponding to the occluding area is labeled as “real.” This set of pixel classification data 414 can be used by the discriminator unit described later to evaluate the recovered image 413 generated by the generator unit 238.
Further, the image segmentation unit 235 may use a set of occluded object labels 418 to replace the object labels corresponding to the occluding objects with the object labels corresponding to the occluded objects, and generate an occluded object labeled image in which the occluded objects have been labeled with their correct object classes. More particularly, the image segmentation unit 235 may generate the occluded object labeled image by replacing the object label of each pixel of the occluded area with the object label of the object that it occluded. In embodiments, the set of occluded object labels 418 may be generated based on a set of annotated target object labels created by a user or administrator (e.g., and stored in a database) that indicate the correct object classes for the occluded objects in the training image 401.
Next, at Step S410, the image conversion unit 236 generates a recovery image 411 by converting the occluded area of the training image 401 to a recovery mask based on the object labels 407. As described herein, the recovery mask may include a binary image in which those pixels that correspond to the occluded area are indicated by a first pixel value (e.g., “1”) and those pixels that do not correspond to the occluded area are indicated by a second pixel value (e.g., “0”).
Next, at Step S412, the generator unit 238 processes the recovery image 411 to generate a recovered image 413 (e.g., a second recovered image) in which the recovery mask has been replaced with a predicted image that indicates one or more predicted objects that share object classes with the respective occluded objects of the training image 401. It should be noted that, at this stage, as the generator unit 238 has not been fully trained, the accuracy of the recovered image 413 generated by the generator unit 238 at Step S412 may be low. However, by repeating the training phase process 400 described herein, the generator unit 238 can be trained to increase its accuracy and generate accurate recovered images 413 in which the occluded area occluded by an occluding object can be replaced with a predicted image including one or more predicted objects having the same object class as the occluded object in the training image 401.
Next, at Step S414, a discriminator unit receives the recovered image 413 from the generator unit 238, the pixel classification data 414 from the image segmentation unit 234, and a reference image 415 that indicates the target object in a state in which it is not occluded by an occluding object, and is trained using the received data to identify the area corresponding to the predicted image of the received recovered image 413. Put differently, the discriminator unit is configured to distinguish whether the recovered image 413 is real (e.g., an original image) or fake (e.g., generated by the generator unit). Here, the discriminator unit may include a classifier unit of a GAN. In embodiments, the discriminator unit may include a global discriminator configured to evaluate the recovered image 413 in its entirety and a patch discriminator configured to evaluate the predicted image within the recovered image 413.
Here, the reference image 415 may be an image that illustrates substantially the same scene as the training image 401 except that the target object is not occluded by the occluding object. By providing the reference image 415 to the discriminator unit, the discriminator unit can be trained to avoid mis-labeling pixels corresponding to the target object. At Step S416, using the pixel classification data 414 and the reference image 415, the discriminator unit may be trained to generate a first set of feedback weights 417 based on the image quality level of the recovered image 413. Here, the image quality level refers to a quantitative measure of the degree to which the recovered image 413 resembles a real image (e.g., recovered images 413 that are more difficult for the discriminator to distinguish as fake may be considered to have a higher image quality level). This first set of feedback weights 417 may be back-propagated to the generator unit 238. Subsequently, the parameters of the generator unit 238 may be adjusted based on this first set of feedback weights 417 in order to facilitate the generation of more accurate recovered images 413.
Next, at Step S419, a patch label classifier unit receives the recovered image 413 from the generator unit 238, the set of occluded object labels 418 from the image segmentation unit 234 (e.g., the occluded object labeled image), and the reference image 415, and is trained using the received data to classify the object labels of the predicted objects in the received image 413. In embodiments, the patch label classifier unit may compare the object labels identified for the predicted objects in the recovered image 413 with the occluded object labels 418 to evaluate the accuracy of the object labels of the predicted objects in the recovered image 413, and generate a second set of feedback weights 420 based on the correlation between the object labels of the predicted objects in the recovered image 413 and the occluded object labels 418. This second set of feedback weights 420 may be back-propagated to the generator unit 238. Subsequently, the parameters of the generator unit 238 may be adjusted based on this second set of feedback weights 420 in order to facilitate the generation of recovered images 413 with more accurate predicted objects. Put differently, the generator unit 238 can be trained to generate recovered images 413 having predicted objects that share the same object class as the target objects that were occluded in the training image 401.
Next, at Step S422, a consistency management unit calculates, based on the recovered image 413 and the training image 401, a consistency loss value that indicates a degree of information loss between the recovered image 413 and the training image 401. The consistency loss value is a value that increases with the amount of pixels of the non-occluded area of the training image 401 that are not present in the recovered image 413. This consistency loss value may be back-propagated to the generator unit 238. Subsequently, the parameters of the generator unit 238 may be adjusted in order to reduce the consistency loss value for future recovered images 413. In this way, the generator unit 238 can be trained to maximize the amount of visual information maintained between the training image 401 and the recovered image 413 for the non-occluding area (e.g., to avoid generating recovered images 413 in which non-occluded portions of the training image 401 are lost).
According to the training phase process 400 described above, the discriminator unit can be trained to distinguish whether the recovered images 413 generated by the generator unit 238 are real or fake, and the generator unit 238 can be trained to generate recovered images 413 that fool the discriminator unit (e.g., images that the discriminator unit classifies as real images, despite being generated by the generator unit 238). In this way, by repeating the training phase process 400 until the generator achieves a desired accuracy level, the generator unit 238 can be trained to generate realistic recovered images in which the area occluded by occluding objects has been replaced with predicted images that illustrate objects corresponding to the occluded objects in the original target image.
Next, a block diagram of a training phase process of the image segmentation unit will be described with reference to FIG. 5.
As described herein, aspects of the disclosure relate to utilizing the image segmentation unit 234 to process the target image acquired by the image acquisition unit 232 to identify an occluding object that at least partially occludes a target object. As examples, the image segmentation unit 234 may include a fully convolutional network, DeepLab, Atrous convolution, Spatial Pyramidal Pooling, a global convolutional network, a spatio-temporal fully convolutional network, or the like. Accordingly, in order to accurately identify the occluding objects in the target image, it is desirable to perform an image segmentation training process 500 to train the image segmentation unit 234. FIG. 5 illustrates a block diagram of an image segmentation training process 500, according to embodiments.
At Step S520, an untrained image segmentation unit 234 is used to process a training image 510. Here, the training image 510 may include an image of a scene in which a target object is occluded by an occluding object. The image segmentation unit 234 processes the training image 510 to attempt to identify the object class of the occluding object together with the occluding area (e.g., that is, the area enclosed by the occluding object). The results output by the image segmentation unit 234 are compared with a set of ground truth data 530 that indicates the correct labels and occluding areas for the occluding objects (e.g., trees, traffic signals, electric poles, crossing devices) in the training image 510, and a loss value is calculated based on the difference between the results of the image segmentation unit 234 and the ground truth data.
Subsequently, the parameters of the image segmentation unit 234 are adjusted (e.g., via back propagation or the like) to reduce the loss value calculated based on the difference between the results of the image segmentation unit 234 and the ground truth data. By repeating this process over multiple iterations with different training images 510 and corresponding ground truth data 530, the image segmentation unit 234 can be trained to accurately identify the object labels and occluding areas of occluding objects. This image segmentation training process 500 may be repeated until the image segmentation accuracy of the image segmentation unit 234 achieves a predetermined accuracy threshold (e.g., 90%, 95%).
In this way, the image segmentation unit 234 can be trained to accurately identify the object labels and occluding areas of occluding objects.
Next, a block diagram of a set of transformation units will be described with reference to FIG. 6.
In embodiments, aspects of the disclosure relate to the recognition that in some cases, target images may have poor image quality due to weather conditions, lighting conditions, image resolution, or the like. In such situations, the poor image quality may negatively affect the accuracy of object detection performed on these target images. Accordingly, aspects of the disclosure relate to utilizing a set of transformation units configured to perform image transformation operations on a target image to facilitate accurate object detection. Accordingly, FIG. 6 illustrates a set of transformation units 600, according to the embodiments of the present disclosure.
In embodiments, the set of transformation units 600 may be configured in parallel with the image segmentation unit 234, the image conversion unit 236, and the generator unit 238 described above. As an example, the set of transformation units 600, the image segmentation unit 234, the image conversion unit 236, and the generator unit 238 may be configured as different layers within a neural network. As will be described below, a set of output images 640 generated by the set of transformation units 600 and the generator unit 238 may be aggregated into a single combined output.
It should be noted that, as the image segmentation unit 234, the image conversion unit 236, the generator unit 238, and the recovered image 335 have been described above, the descriptions thereof will be omitted here.
As illustrated in FIG. 6, the set of transformation units 600 includes a first transformation unit 610, a second transformation unit 620, and a third transformation unit 630. Each of the set of transformation units 600 may be configured using a generator (e.g., the generator of a GAN) trained to perform a different image transformation on the target image 212. More particularly, each of the set of transformation units 600 may be configured to perform an image transformation operation to increase the similarity of the target image 212 to the training images (e.g., an object detection training image) used to train the object detection unit 240. By increasing the similarity of the target image 212 to the training images used to train the object detection unit 240, it is possible to facilitate accurate object detection.
The first transformation unit 610 may be configured to perform a resolution transformation operation on the target image 212 to generate a first transformed target image 612 having a resolution different than the target image 212. For instance, the first transformation unit 610 may perform a resolution transformation operation on the target image 212 to generate a first transformed target image 612 that has an increased resolution with respect to the target image 212 in order to facilitate object detection.
The second transformation unit 620 may be configured to perform a weather transformation operation to generate a second transformed target image 622 having a weather condition different than the target image 212. For instance, in the case that the target image 212 illustrates a scene having foggy weather, the second transformation unit 620 may perform a weather transformation operation on the target image 212 to generate a second transformed target image 622 in which the target image 212 has been transformed to a scene having sunny weather in order to facilitate object detection.
The third transformation unit 630 may be configured to perform a lighting transformation operation to generate a third transformed target image 632 having a lighting condition different than the target image 212. For instance, in the case that the target image 212 illustrates a nighttime scene, the third transformation unit 630 may perform a lighting transformation operation on the target image 212 to generate a third transformed target image 632 in which the target image 212 has been transformed to a daytime scene in order to facilitate object detection.
Next, a transformation management unit 645 may receive the target image 212, the recovered image 335, the first transformed target image 612, the second transformed target image 622, and the third transformed target image 632, and assign a first set of weights to the target image, a second set of weights to the recovered image 335, a third set of weights to the first transformed target image 612, a fourth set of weights to the second transformed target image 622, and a fifth set of weights to the third transformed target image 632. Here, the first, second, third, fourth, and fifth set of weights are weights that indicate the degree to which the features corresponding to each respective image should be reflected in the final combined transformed target image.
In embodiments, the transformation management unit 645 may be trained to assign the first, second, third, fourth, and fifth set of weights based on the degree to which the features of each of the target image 212, the recovered image 335, the first transformed target image 612, the second transformed target image 622, and the third transformed target image 632 are represented in the training images used to train the object detection unit 240, such that images that include features (lighting conditions, weather conditions, resolution) that are more similar to the training images used to train the object detection unit 240 are given a greater weight. In embodiments, the transformation management unit 645 may be implemented using the different model layers of a GAN.
Next, the transformation management unit 615 may combine the target image 212, the recovered image 335, the first transformed target image 612, the second transformed target image 622, and the third transformed target image 632 into a combined transformed target image 655 based on the first set of weights, the second set of weights, the third set of weights, the fourth set of weights, and the fifth set of weights (S650). In this way, a combined transformed target image 655 can be generated that reflects the characteristics of each of the target image 212, the recovered image 335, the first transformed target image 612, the second transformed target image 622, and the third transformed target image 632 in accordance with the first, second, third, fourth, and fifth set of weights. Put differently, a combined transformed target image 655 that has been transformed to have greater similarity to the training images used to train the object detection unit 240 can be generated.
Next, the combined transformed target image 655 can be input to the object detection unit 240. The object detection unit 240 may process the combined transformed target image 655 to detect the target object and generate an object detection result 222, as described herein.
According to the set of transformation units 600 described herein, it is possible to perform image transformations on a target image 212 to increase the similarity of the target image 212 with the training images used to train the object detection unit 240, and thereby facilitate accurate object detection.
Next, an example of a target image, a recovery image, and a recovered image will be described with reference to FIG. 7.
As described herein, aspects of the disclosure relate to converting a target image 212 in which one or more target objects are at least partially occluded by one or more occluding objects to a recovery image 720 in which the area occluded by the occluding objects is represented as a binary recovery mask, and subsequently generating a recovered image 335 in which the recovery mask has been replaced with a predicted image illustrating predicted objects that correspond to (e.g., share object classes with) the occluded target objects of the target image 212. FIG. 7 illustrates an example of a target image 212, a recovery image 720, and a recovered image 335, according to embodiments.
As an example, in the target image 212 illustrated in FIG. 7, a first target object of an automobile is occluded by a railroad crossing device 705, and a second target object of a person 712 is occluded by a railroad crossing device 715. Accordingly, by processing the target image 212 using the image segmentation unit 234, the area occluded by both the railroad crossing devices 715, 715 can be identified, and an object label of “railroad crossing device” can be assigned thereto.
Next, the image segmentation unit 234 can generating recovery image 720 by converting the occluded area of the target image 212 to a recovery mask 725 based on the object label.
Subsequently, the generator unit 238 can process the recovery image 720 to generate a recovered image 335 in which the recovery mask 725 has been replaced with a first predicted image 745 that includes one or more predicted objects that have the same object class as the target objects, respectively. More particularly, as shown in the recovered image 335, the area of the target image 212 occluded by the railroad crossing devices 705 has been replaced with a predicted image illustrating an automobile 752, and the area of the target image 212 occluded by the railroad crossing device 715 has been replaced with a predicted image illustrating a person 762, in addition to the appropriate background images of the sky and ground that were occluded in the target image 212.
The recovery image 720 generated in this way can subsequently be input to an object detection unit (e.g., the object detection unit 240) to facilitate generation of the target objects of the automobile and the person with a high degree of accuracy.
According to the object detection device, object detection method, and object detection system according to the present disclosure, as a generator unit (e.g., the generator in a GAN) can be trained to generated a recovered image in which the occluded area of a target image has been replaced with a predicted image that indicates one or more predicted objects having the same object class as the respective occluded objects in the target image, and subsequently performing object detection on this recovered image, it is possible to generate an accurate object detection result even in cases where target objects are occluded.
Further, aspects of the disclosure relate to utilizing a plurality of transformation units (e.g., additional GANs) to enhance the quality and modify features (e.g., weather conditions, lighting conditions) of input images. The outputs from these transformation units may be weighted and combined into a single image using a set of convolution layers. By fusing the results of multiple transformation units, it is possible to balance the weights of each transformation unit to generate an image that has higher similarity with the images used to train the object detection unit. As such, the likelihood of false detection can be reduced, further improving the accuracy of object detection.
In this way, according to the present disclosure, it is possible to provide a device, method, and system for object detection that is capable of detecting occluded objects with a high degree of accuracy.
The present invention may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing.
A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
Embodiments according to this disclosure may be provided to end-users through a cloud-computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing may be defined as a computing capability that provides an abstraction between the computing resource and its underlying technical architecture (e.g., servers, storage, networks), enabling convenient, on-demand network access to a shared pool of configurable computing resources that can be rapidly provisioned and released with minimal management effort or service provider interaction. Thus, cloud computing allows a user to access virtual computing resources (e.g., storage, data, applications, and even complete virtualized computing systems) in “the cloud,” without regard for the underlying physical systems (or locations of those systems) used to provide the computing resources.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
While the foregoing is directed to exemplary embodiments, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow. The descriptions of the various embodiments of the present disclosure have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the various embodiments. As used herein, singular forms are intended to include the plural forms as well, unless the context clearly indicates otherwise. “Set of,” “group of,” etc. are intended to include one or more. It will be further understood that the terms “includes” and/or “including,” when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. In the previous detailed description of exemplary embodiments of the various embodiments, reference was made to the accompanying drawings (where like numbers represent like elements), which form a part hereof, and in which is shown by way of illustration specific exemplary embodiments in which the various embodiments may be practiced. These embodiments were described in sufficient detail to enable those skilled in the art to practice the embodiments, but other embodiments may be used and logical, mechanical, electrical, and other changes may be made without departing from the scope of the various embodiments. In the previous description, numerous specific details were set forth to provide a thorough understanding the various embodiments. But, the various embodiments may be practiced without these specific details. In other instances, well-known circuits, structures, and techniques have not been shown in detail in order not to obscure embodiments.

200... Object detection system
210… Image capture device
212… Target image
220… Client device
222… Object detection result
225… Communication network
230… Object detection device
232… Image acquisition unit
234… Image segmentation unit
236… Image conversion unit
238… Generator unit
240… Object detection unit
300… Inference phase process
335... Recovered image
400…Training phase process
500…Image segmentation training process
600…Set of transformation units

Claims

An object detection device comprising:
an image acquisition unit for receiving a target image that at least includes a first target object;
an image segmentation unit for processing the target image to identify an occluded area of the target image in which at least the first target object is at least partially occluded by a first occluding object, and assign a first object label to the first occluding object that indicates an object class of the first occluding object;
an image conversion unit for generating a first recovery image by converting the occluded area of the target image to a first recovery mask based on the first object label;
a generator unit for processing the first recovery image to generate a first recovered image in which the first recovery mask has been replaced with a first predicted image that includes at least a first predicted object associated with a second object label that indicates an object class corresponding to both the first predicted object and the first target object; and
an object detection unit for processing the first recovered image to detect the first target object and generating an object detection result that includes at least a location of the first target object and a target object label indicating an object class of the first target object.
The object detection device according to claim 1, wherein:
the generator unit is further configured to generate, with respect to a training image in which a second target object is at least partially occluded by a second occluding object, a second recovered image in which a second recovery mask corresponding to the second occluding object has been replaced with a second predicted image that indicates at least a second predicted object associated with a third object label that indicates an object class of the second predicted object; and
the object detection device further comprises a discriminator unit configured to generate a first set of feedback weights based on an image quality level of the second recovered image,
wherein the discriminator unit is trained based on:
a first labeled training image generated by processing the training image with the image segmentation unit to assign a fourth object label to the second occluding object that indicates an object class of the second occluding object, and
a reference image that includes the second target object not occluded by an occluding object.
The object detection device according to claim 2, further comprising:
a patch label classifier unit configured to generate a second set of feedback weights based on a correlation of the third object label of the second predicted object and a fifth object label indicating an object class of the second target object,
wherein the patch label classifier unit is trained based on:
a second labeled training image generated by processing the training image with the image segmentation unit to assign, based on a set of annotated target object labels, the fifth object label to the second target object; and
the reference image.
The object detection device according to claim 3, further comprising:
a consistency management unit configured to calculate, based on the second recovered image and the training image, a consistency loss value that indicates a degree of information loss between the second recovered image and the training image.
The object detection device according to claim 4, wherein:
the generator unit is trained to generate recovery images based on the first set of feedback weights, the second set of feedback weights, and the consistency loss value.
The object detection device according to claim 1, further comprising:
a set of transformation units configured to generate a set of transformed target images, wherein the set of transformation units include:
a first transformation unit configured to perform a resolution transformation operation on the target image to generate a first transformed target image having a resolution different than the target image;
a second transformation unit configured to perform a weather transformation operation to generate a second transformed target image having a weather condition different than the target image; and
a third transformation unit configured to perform a lighting transformation operation to generate a third transformed target image having a lighting condition different than the target image.
The object detection device according to claim 6, further comprising a transformation management unit configured to:
assign a first set of weights to the target image based at least on a similarity of the target image to an object detection training image used to train the object detection unit;
assign a second set of weights to the first recovered image based at least on a similarity of the first recovered image to the object detection training image;
assign a third set of weights to the first transformed target image based at least on a similarity of the first transformed target image to the object detection training image;
assign a fourth set of weights to the second transformed target image based at least on a similarity of the second transformed target image to the object detection training image;
assign a fifth set of weights to the third transformed target image based at least on a similarity of the third transformed target image to the object detection training image; and
combine the target image, the first recovered image, the first transformed target image, the second transformed target image, and the third transformed target image into a combined transformed target image based on the first set of weights, the second set of weights, the third set of weights, the fourth set of weights, and the fifth set of weights.
An object detection system comprising:
an image capture device for capturing a target image that at least includes a first target object;
an object detection device for detecting the first target object in the target image and generating an object detection result; and
a client device for viewing the object detection result,
wherein the image capture device, the object detection device, and the client device are communicably connected via a communication network, and
the object detection device includes:
an image acquisition unit for receiving the target image from the image acquisition unit;
an image segmentation unit for processing the target image to identify an occluded area of the target image in which at least the first target object is at least partially occluded by a first occluding object, and assign a first object label to the first occluding object that indicates an object class of the first occluding object;
an image conversion unit for generating a first recovery image by converting the occluded area of the target image to a first recovery mask based on the first object label;
a generator unit for processing the first recovery image to generate a first recovered image in which the first recovery mask has been replaced with a first predicted image that includes at least a first predicted object associated with a second object label that indicates an object class corresponding to both the first predicted object and the first target object; and
an object detection unit for processing the first recovered image to detect the first target object, generating the object detection result that includes at least a location of the first target object and a target object label indicating an object class of the first target object, and transmitting the object detection result to the client device.
An object detection method comprising:
generating, using an untrained generator unit, with respect to a training image in which a first target object is at least partially occluded by a first occluding object, a first recovered image in which a first recovery mask corresponding to the first occluding object has been replaced with a first predicted image that indicates at least a first predicted object associated with a first object label that indicates an object class of the first predicted object;
training a discriminator unit based on a first labeled training image generated by processing the training image with an image segmentation unit to assign a second object label to the first occluding object that indicates an object class of the first occluding object and a reference image that includes the first target object not occluded by an occluding object;
generating, by using the discriminator unit to evaluate the first recovered image, a first set of feedback weights based on an image quality level of the first recovered image;
training a patch label classifier unit based on the reference image and a second labeled training image generated by processing the training image with the image segmentation unit to assign, based on a set of annotated target object labels, a third object label to the first target object that indicates an object class of the first target object;
generating, using the patch label classifier unit to process the first recovered image, a second set of feedback weights based on a correlation of the first object label of the first predicted object and the third object label of the first target object;
calculating, using a consistency management unit, based on the first recovered image and the training image, a consistency loss value that indicates a degree of information loss between the first recovered image and the training image;
generating a trained generator unit by training, based on the first set of feedback weights, the second set of feedback weights, and the consistency loss value, the untrained generator unit to generate recovery images;
receiving, using an image acquisition unit, a target image that includes a second target object;
identifying, by processing the target image with the image segmentation unit, an occluded area of the target image in which at least the second target object is at least partially occluded by a second occluding object, and assigning a fourth object label to the second occluding object that indicates an object class of the second occluding object;
generating, using an image conversion unit, a second recovery image by converting the occluded area of the target image to a second recovery mask based on the fourth object label;
generating, using the trained generator unit to process the second recovery image to generate a second recovered image in which the second recovery mask has been replaced with a second predicted image that includes at least a second predicted object associated with a fifth object label that indicates an object class corresponding to both the second predicted object and the second target object; and
generating, using an object detection unit to process the second recovered image, an object detection result that includes at least a location of the second target object and a target object label indicating an object class of the second target object.