JP2019204505A - Object detection deice, object detection method, and storage medium - Google Patents

Abstract

To provide an object detection device, an object detection method, and storage medium, which enable better detection of associated objects.SOLUTION: An object detection device is provided, comprising an extraction unit for extracting features from an image, a determination unit configured to determine spatial relationships among individual feature points in the image on the basis of the extracted features, and a detection unit configured to detect an object area in the image on the basis of the determined spatial relationships, where objects herein imply associated objects in the image.SELECTED DRAWING: Figure 2

Description

  The present invention relates to image processing, in particular, for example, object detection processing.

  In the monitoring system, a person is generally a main monitoring target. In general, people wear, hold or use some objects on a daily basis (for example, glasses, bags, suitcases, wheelchairs, etc.), so these objects monitor people in an auxiliary manner. To be used. In the present specification, these objects are called, for example, attachments of related persons. In the monitoring process, the person recognition process is generally the main processing operation, and the most necessary basic information used for the person recognition process is information such as the position of the person in the moving image and related objects. Therefore, the ability to detect people and related objects from video / images with high recall directly affects the accuracy of the person recognition process. In the present specification, the person recognition processing includes, for example, person attribute recognition, person verification (verification of target person ID), person image search, person behavior / behavior recognition or analysis (for example, target person Analysis of movement between the subject and other objects, etc.).

  Non-Patent Document 1 discloses an exemplary object detection technique for detecting a person and related objects from a video / image with high recall. In general, it is as follows. First, various levels of features are extracted from the input image using a neural network. For example, extract low-level features for small scale objects, medium-level features for medium-scale objects, and high-level features for large-scale objects. Next, using the corresponding pre-generated candidate area generation network, the relevant information of the candidate area of the object (for example, the position of the candidate area, the score of the candidate area, and the feature of the candidate area) is extracted from the features of each level. To do.

  In the object detection technique (for example, the above-described exemplary technique), generally only candidate areas having scores that do not fall below a preset threshold value, or only candidate areas having scores ranked in the top N are output as final outputs. Become. In other words, the final output candidate region is regarded as a region of objects (people and objects) that can be detected from the image. However, for example, an object obstructed by other objects in the image, such as a suitcase obstructed by a person concerned, a wheelchair obstructed by a seated person, a bag affected by a shadow placed on the ground, For objects that are affected by lighting, the characteristics of those objects cannot be completely extracted from the image. Therefore, even if the candidate of those objects can be detected from the image using the technique exemplified above, the score of the candidate area of the object is low, so the score of the candidate area of the object is below a preset threshold value. Or, the scores of the candidate areas of these objects are not ranked in the top N. As a result, the candidate area of such an object does not become the final output, the object from the image cannot be detected, and finally the high reproduction rate of the object is affected.

"Feature Pyramid Networks for Object Detection" (Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie, CVPR 2017).

  In view of the related art, the present invention solves at least one of the above problems.

  An object detection apparatus provided in one aspect of the present invention includes an extraction unit that extracts a feature from an image, a determination unit that determines a spatial relationship between individual feature points in the image based on the extracted feature, And detecting means for detecting a region of the object in the image based on the determined spatial relationship, wherein the object is an associated object in the image.

  An object detection method provided in another aspect of the present invention includes an extraction step of extracting features from an image, a determination step of determining a spatial relationship between individual feature points in the image based on the extracted features, And detecting a region of the object in the image based on the determined spatial relationship, wherein the object is an associated object in the image.

  An object detection apparatus provided in a further aspect of the present invention is a feature extraction means for extracting a feature from a current video frame in a video, and detects a candidate region of an object from the current video frame based on the extracted feature. Based on the candidate area detection means, the spatial relationship determination means for determining the spatial relationship between the candidate areas based on the detection result of the previous video frame of the current video frame, and the spatial relationship between the candidate areas, And determining the rank of the candidate area, and ranking determination means for determining the candidate area after determining the rank as the area of the object, and the detection result of the previous video frame of the current video frame is the object detection apparatus described above. It is obtained using.

  In yet another aspect of the invention, a storage medium for storing instructions executable by a processor is provided to enable the object detection method described above.

  In the present invention, the spatial relationship between the individual feature points is used when detecting the object region so that the detection of the object region is regulated. This makes it possible to better detect related objects. Since related objects are generally more useful for monitoring people during the monitoring process, not only the reproducibility of object detection, but also the effect of monitoring people can be improved according to the present invention.

  Further other features and advantages of the present invention will become apparent from the embodiments described below with reference to the accompanying drawings.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present invention in order to explain the principles of the present invention.
It is a block diagram which shows the hardware constitutions which can implement the technique which concerns on embodiment of this invention. It is a block diagram which shows the structure of the object detection apparatus which concerns on the 1st Embodiment of this invention. FIG. 3 is a block diagram illustrating a configuration of a detection unit 230 illustrated in FIG. 2. It is a figure which shows the flowchart of the object detection concerning the 1st Embodiment of this invention. It is a schematic block diagram of the pre-generation model utilized in the flowchart of FIG. It is a figure which shows the flowchart of the detection step shown by FIG. 4 concerning the 1st Embodiment of this invention. It is a figure which shows the flowchart of the order | rank determination step shown by FIG. 6 concerning the 1st Embodiment of this invention. It is a block diagram which shows the structure of the object detection apparatus concerning the 2nd Embodiment of this invention. It is a figure which shows the flowchart of the production | generation method which produces | generates the model applicable to this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that the following description is merely exemplary and exemplary in nature and is not intended to limit the invention and its application or uses. The relative arrangement of elements and steps, numerical expressions, numerical values, and the like described in the embodiments do not limit the scope of the present invention unless otherwise specified. Moreover, techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail, but should be part of the specification where appropriate.

  In the drawings, like reference numerals and letters indicate like items. Therefore, it should be noted that once an item is defined in one figure, it need not be discussed in the following figure.

  The inventor has found that in the actual scene of object detection, there are generally several objects that have a specific relationship (especially spatial relationship) to each other, and these objects are generally referred to as related objects. . Furthermore, on the other hand, interacting cases (eg, being blocked from each other) are likely to occur between related objects. On the other hand, in the monitoring process, the related object is useful for monitoring a person. In this way, the inventor found that this particular relationship (especially the spatial relationship) that exists between related objects in the object detection process is, for example, even when some related objects are occluded. We have found that the detection of the area of the object can be constrained. As a result, the object detection reproducibility can be improved, and the effect of monitoring a person can be enhanced. In the present invention, a person (for example, a subject) and a person are wearing / holding / using, such as a suitcase that a woman and the woman are pulling, a wheelchair that a man and the man are sitting on, etc. Objects can be considered as related objects. A woman and her child's child, and two overlapping people located in front and back can also be considered as related objects. Neighboring objects (for example, the target object and other objects next to the target object), such as a suitcase and a bag placed on it, a person's shadow and a bag partially or entirely covered with the shadow, It can be considered as a related object. However, it is clear that this need not be limited.

In the present invention, a spatial relationship existing between related objects represents a spatial constraint between the related objects. For example, in two related objects, a spatial relationship between them (for example, a spatial constraint between regions corresponding to the two related objects) includes at least some of the following restrictions.
-Relative positional relationship between two objects (eg direction relationship, distance relationship, etc.)
For example, in the case of a computer placed on a desk, the directional relationship between the computer and the desk is “on the desk”. For example, in the case of a person / animal on the lawn, the directional relationship between the person / animal and the lawn is “on the grass”. For example, in the case of a woman guiding a child to hold and walk, the distance relationship between the woman and the child is “adjacent to, close to”. However, it is clear that this need not be limited.
-Topological relationship between two objects (for example, overlapping relationship, inclusion relationship, adjacency relationship, etc.)
For example, in the case of a man sitting in a wheelchair, the phase relationship between the man and the wheelchair is “overlapping relationship”. For example, in the case of a woman holding a child, the phase relationship between the woman and the child is an “inclusion relationship”. In the case of a woman having a parasol, the phase relationship between the woman and the parasol is an “adjacent relationship”. However, it is clear that this need not be limited.
・ Relative shape relationship between two objects
For example, like a man sitting in a wheelchair, spatial regulation between a person and a wheelchair also becomes a “relative shape relationship”. However, it is clear that this need not be limited.

  The present invention has been proposed in view of the above consideration results, and will be described in detail below with reference to the accompanying drawings.

[Hardware configuration]
First, a hardware configuration capable of realizing the technique described below will be described with reference to FIG.

  The hardware configuration 100 includes, for example, a central processing unit (CPU) 110, a random access memory (RAM) 120, a read only memory (ROM) 130, a hard disk 140, an input device 150, an output device 160, a network interface 170, and a system. A bus 180 is included. Further, the hardware configuration 100 may be implemented in, for example, a camera, video camera, personal digital assistant (PDA), smartphone, tablet, laptop, desktop computer, or other suitable electronic device.

  The object detection processing of the present invention in one implementation method is configured by hardware or firmware, and functions as a module or component of the hardware configuration 100. For example, the object detection apparatus 200 described in detail below with reference to FIG. 2 and the object detection apparatus 800 described in detail below with reference to FIG. 8 are used as modules or components of the hardware configuration 100. The The object detection process of the present invention in another implementation method is configured by software stored in the ROM 130 or the hard disk 140 and executed by the CPU 110. For example, a process 400 described in detail with reference to FIG. 4 and a process 900 described in detail with reference to FIG. 90 are used as programs stored in the ROM 130 or the hard disk 140.

  The CPU 110 is any suitable programmable control device such as a processor, and can execute various functions described below by executing various application programs stored in the ROM 130 or the hard disk 140 (memory or the like). it can. The RAM 120 temporarily stores programs and data loaded from the ROM 130 or the hard disk 140, and various processes executed by the CPU 110 (for example, the techniques shown in FIGS. 3 to 7 and 9 described later) and other uses. It is also used as a space for performing possible functions. The hard disk 140 stores various information such as an operating system (OS), various applications, control programs, videos, images, pre-generated models, and pre-defined data (for example, thresholds (THs)). Store.

  In one embodiment, the input device 150 allows a user to exchange information with the hardware configuration 100. In one embodiment, the user can input images / video / data through the input device 150. In other embodiments, the user can initiate corresponding processing of the present invention through the input device 150. Further, the input device 150 may employ various forms such as buttons, a keyboard, or a touch screen. In another embodiment, input device 150 is utilized to receive images / video output from specialized electronic devices such as digital cameras, video cameras, and / or network cameras.

  In one embodiment, the output device 160 is utilized to display the detection results (position, score, features, etc. of the detected area of the object) to the user. Furthermore, the output device 160 can adopt various forms such as a CRT (Cathode Ray Tube) and a liquid crystal display. In other embodiments, the output device 160 may pass the detection results to subsequent processing, such as person recognition processing (eg, person attribute recognition, person matching, person image search, and person behavior / behavior recognition or analysis). Used for output.

  The network interface 170 provides an interface for connecting the hardware configuration 100 to the network. For example, the hardware configuration 100 can perform data communication with other electronic devices connected via a network via the network interface 170. Alternatively, the hardware configuration 100 may include a wireless interface for performing wireless data communication. The system bus 180 provides a data transfer path for mutual data transfer among the CPU 110, RAM 120, ROM 130, hard disk 140, input device 150, output device 160, network interface 170, and the like. System bus 180 is referred to as a bus, but is not limited to a particular data transmission technique.

  The hardware configuration 100 described above is merely exemplary and is not intended to limit the present invention and its application or use. Furthermore, for simplicity, only one hardware configuration is shown in FIG. However, multiple hardware configurations can be used as needed.

Object detection
Next, object detection in the present invention will be described with reference to FIGS.

  FIG. 2 is a block diagram showing the configuration of the object detection apparatus 200 according to the first embodiment of the present invention. Here, some or all of the modules shown in FIG. 2 may be realized by dedicated hardware. As illustrated in FIG. 2, the object detection device 200 includes an extraction unit 210, a determination unit 220, and a detection unit 230.

  First, the input device 150 shown in FIG. 1 receives an image output from a specific electronic device (for example, a video camera). Next, the input device 150 transfers the received image to the object detection apparatus 200 via the system bus 180.

  Then, as shown in FIG. 2, the extraction unit 210 extracts features from the received image. Here, the extraction unit 210 uses an existing feature extraction algorithm such as a local binary pattern (LBP) algorithm, a Gabor algorithm, a scale invariant feature transformation (SIFT) algorithm, a neural network (NN) algorithm, and the like from an image. Features can be extracted. Here, the extracted features may be, for example, gradient features, edge features, apparent features, semantic features, etc. in the image.

  The determination unit 220 determines a spatial relationship between individual feature points in the image based on the extracted features. Here, the feature point is a point on the extracted feature. Further, as a spatial relationship between any two feature points, according to whether the two feature points belong to the same region or different regions, the spatial correlation between the feature points is “a positional correlation within the same region ( That is, it is categorized into “positional relationship between intra categories)” and “positional relationship between different areas (ie, positional relationship between inter categories)”. Furthermore, the spatial relationship between the determined feature points has a spatial relationship value corresponding thereto. Here, in the case of a spatial relationship between two feature points, the corresponding spatial relationship value represents the probability that the two feature points belong to the spatial relationship.

  In one embodiment, the determination unit 220 may determine a spatial relationship between feature points based on the extracted features according to a predefined rule. The predefined rules here may be stored in a corresponding recording device, for example the storage device 240 shown in FIG. Here, the storage device 240 may be the ROM 130 or the hard disk 140 shown in FIG. 1, or may be a server or an external storage device connected to the object detection apparatus 200 via a network (not shown). As described above, in the embodiment, the determination unit 220 first acquires a predefined rule from the recording device 240, and performs a determination process corresponding to the spatial relationship.

  In another embodiment, a model used to determine the spatial relationship between feature points (ie, a pre-generated model) is used to conveniently determine the spatial relationship in various scenes. A relationship is trained / pre-generated according to a labeled training sample and stored in a corresponding storage device (eg, storage device 240). Here, the details of the method for generating the pre-generated model will be described later with reference to FIG. On the other hand, the determination unit 220 determines a spatial relationship between feature points based on the extracted features by using a pre-generated model.

  Furthermore, in order to improve the processing speed of object detection, the above pre-generated model is generated as shown in FIG. Here, the pre-generated model includes a part for extracting features and a part for determining a spatial relationship. Alternatively, the extraction unit 210 may extract features from the image using a pre-generated model. In this case, on the one hand, the extraction unit 210 acquires a pre-generated model from the storage device 20, and on the other hand, the extraction unit 210 extracts features from the image using the pre-generated model. Furthermore, in this case, the determination unit 220 may use the pre-generated model acquired by the extraction unit 210 from the storage device 240 without acquiring the corresponding pre-generated model from the storage unit 240.

  Returning to the description of FIG. After the spatial relationship between the individual feature points is determined, the detection unit 230 detects a region of the object in the image based on the determined spatial relationship. Here, the object is preferably a related object in the image, and the area of the detected object includes, for example, the position of the area, the score of the area, and the features encompassed by the area. Here, the score of one area indicates the probability that the area belongs to an object of a certain category, and the feature included in one area is a feature among the features extracted by the extraction unit 210 belonging to the area. It is.

  In one embodiment, the detection unit 230 may detect the region of the object from the image by directly using the determined spatial relationship. Specifically, first, the detection unit 230 clusters individual feature points based on the determined spatial relationship. Here, the clustering result can be regarded as one region, and the spatial relationship between the feature points in each clustering result belongs to the “intra category spatial relationship” described above. Then, the detection unit 230 sets the corresponding region as the final detection region of the object based on the spatial relationship between the feature points belonging to different clustering results (that is, the above “intercategory spatial relationship”). judge. Here, the clustering result whose mutual distance is less than a predetermined threshold (for example, TH1) can be regarded as the final detection region of the object, and the clustering results overlapping each other can be regarded as the final detection, for example. .

  In other embodiments, a more relevant object region can be preferentially output, and in order to make the position of the detected object region more accurate, the detection unit 230 may be configured as shown in FIG. The candidate area detecting unit 231 and the order determining unit 232 may be included.

  As illustrated in FIG. 3, the candidate area detection unit 231 detects an object candidate area from the image based on the feature detected by the detection unit 210. Here, the candidate area detection unit 231 uses an existing area detection algorithm, for example, a selective search algorithm, an edge box algorithm, an object algorithm, etc. A region can be detected. Further, as described above, the pre-generated model shown above may include a part for feature extraction and a part for determining a spatial relationship. In order to further increase the processing speed of object detection, the pre-generated model 9 may include a part to be detected in a candidate area of an object when generated as shown in FIG. Therefore, the candidate area detection unit 231 may detect the candidate area of the object from the image based on the features extracted using the pre-generated model. In this case, the candidate area detection unit 231 may detect a candidate area of the object from the image using the pre-generated model acquired by the detection unit 210 from the storage device 240. Here, the candidate area in which the object is detected includes, for example, the position of the candidate area, the score of the candidate area, and a feature including the candidate area. Here, the score of one candidate area indicates the probability that the candidate area belongs to an object of a certain category. For example, the score of a candidate area may be obtained by categorizing the candidate areas.

  Next, as shown in FIG. 3, after the candidate area is detected from the image, the rank determining unit 232 determines the rank of the detected candidate area based on the spatial relationship determined by the determining unit 220, The candidate area after the rank determination is set as an object detection area.

  Further, the above pre-generated model may include a part for directly detecting the object area when the object is generated as shown in FIG. 9 in addition to the part for detecting the object candidate area. . Therefore, in other embodiments, the detection unit 230 may detect the region of the object in the image based on the spatial relationship determined using the pre-generated model directly. In this case, the detection unit 230 may detect an object region from the image using the pre-generated model acquired by the extraction unit 210 from the storage device 240.

  Returning to FIG. 2, after the object region is detected from the image, the detection unit 230 sets the object region having a score that is defined in advance or having a score equal to or higher than the threshold as the final detection result or the top N regions. An area having a ranking score is set as a final detection result, and the final detection result is transferred to the output device 160 via the system bus 180 shown in FIG. The position, score, and feature) of the image to the user, or subsequent processing such as person recognition processing (for example, recognition or analysis of person attribute recognition, person matching, person image search, person behavior / behavior, etc.) The object detection area is output to.

  A flowchart 400 shown in FIG. 4 is a process corresponding to the object detection apparatus 200 shown in FIG. The following description is performed by the extraction unit 210, the determination unit 220, and the detection unit 230 that perform corresponding processing using a pre-generated model. Here, the schematic configuration of the pre-generated model used in the processing is as shown in FIG. 5, for example. However, it is clear that there is no need to be limited to this.

  As illustrated in FIG. 4, in the extraction step S410, the extraction unit 210 acquires a pre-generated model from the storage device 240, and receives an image using the acquired pre-generated model (particularly, a portion in which features are extracted). Extract features from.

  In determination step S420, determination unit 220 determines a spatial relationship between feature points based on the extracted features using a pre-generated model (particularly, a portion for determining a spatial relationship therein).

  In detection step S430, detection unit 230 detects an object region from the image using the acquired pre-generated model. Here, the object is preferably an associated object in the image. As described above, in order to preferentially output a region of a more relevant object and to make the position of the detected region of the object more accurate, in one implementation phase, the detection unit 230 may 6 to detect the area of the object in the image.

  As shown in FIG. 6, in candidate region detection step S431, the candidate region detection unit 231 is extracted by the extraction unit 210 using a pre-generated model (particularly, a portion for detecting candidate regions therein). Based on the feature, a candidate region of the object is detected from the image. In the rank determination step S432, the rank determination unit 232 determines the rank of the candidate area based on the spatial relationship determined by the determination unit 220, and sets the candidate area after the rank determination as the final detection area of the object. To do. In one embodiment, the rank determination unit 232 determines the rank of candidate regions according to FIG.

  As illustrated in FIG. 7, in step S4321, the rank determining unit 232 determines a spatial relationship between candidate areas detected by the candidate area detecting unit 231. Specifically, for any two candidate regions, the rank determination unit 232 determines the spatial relationship between the two candidate regions based on the mutual spatial relationship between the feature points included in the two candidate regions. judge. Here, as long as there are two corresponding feature points having a specific spatial relationship in the two candidate regions, the two candidate regions can be regarded as having a specific spatial relationship.

  In one embodiment, for any two candidate regions, the rank determination unit 232 determines a spatial relationship between any two feature points between the two candidate regions, and a spatial relationship between the two candidate regions. Judge as. Preferably, for example, a spatial relationship between two feature points at the center position of these two candidate regions may be determined as a spatial relationship between the two candidate regions. Here, the spatial relationship value of the spatial relationship between the two candidate regions is regarded as the spatial relationship value of the spatial relationship between the two candidate regions. For example, a spatial relationship between two feature points having the maximum spatial relationship value is determined as a spatial relationship between the two candidate regions. Here, the maximum spatial relationship value is regarded as the spatial relationship value of the spatial relationship between the two candidates.

  In another embodiment, the rank determination unit 232 uses, for any two candidate regions, the space between the two candidate regions using all the spatial relationships between the feature points that exist between the two candidate regions. The relationship. Preferably, for example, on the one hand, the spatial relationship between feature points is voted and the spatial relationship with the largest number of votes is determined as the spatial relationship between the two candidate regions. On the other hand, all the spatial relation values belonging to the spatial relation with the largest number of votes are averaged, weighted and summed, or maximized, and the obtained value is the spatial relation value of the spatial relation between the two candidate areas. Is considered.

  Returning to FIG. In step S4322, the rank determination unit 232 updates the score of the candidate region based on the spatial relationship value of the spatial relationship after the determination between the candidate regions. In one embodiment, the rank determination unit 232 may update the score of the candidate area by calculation between the matrices. Specifically, for example, a matrix composed of the spatial relationship values of the determined spatial relationships between the candidate regions and a matrix composed of the scores of the candidate regions are subjected to a mathematical operation (for example, matrix multiplication). Then, the result obtained after the calculation is used as the updated score of the candidate area. In another embodiment, when a target object (for example, a target person) to be detected is specified, the rank determination unit 232 may select a candidate area of an object (for example, an accessory of the target person) related to the target object. Just update the score. In particular. For example, first, a related object having a maximum spatial relationship value is determined from a related object having a spatial relationship to the target object, and a maximum spatial relationship value is determined to update the score of the candidate region. It is superimposed on the score of the candidate object candidate area.

  Further, in order to narrow the determination range of the spatial relationship between the candidate regions in order to improve the processing speed, as shown in FIG. 7, step S4320 determines the spatial relationship between the candidate regions before S4321 (that is, the spatial relationship between the candidate regions). It may be included in the previous). As shown in FIG. 7, in step S4320, the rank determination unit 232 acquires corresponding auxiliary information. This auxiliary information is, for example, information about a specific detection task, information about a specific detection scene, and the like.

  As the specific detection task, generally, a target object (for example, a target person) to be detected is specified, that is, position information and category information of the target object are generally given. Furthermore, it is generally desirable that the preferentially detected object be another object related to the target object (for example, an accessory around the target person).

  Therefore, regarding the specific detection task, the auxiliary information acquired by the rank determination unit 232 is, for example, position information and category information of at least one target object. On the other hand, since the category information of the target object is known, the rank determination unit 232 may clearly determine the type of spatial relationship existing between the target object and another object. For example, when the target object is a target person, the spatial relationship between the target object and other objects is “spatial relationship between a certain person and another person” and “between a certain person and another object. "Spatial relationship between" and "spatial relationship between a certain object". On the other hand, since the position information of the target object is known, the order determination unit 232 roughly determines which candidate region should be determined without determining the spatial relationship between all candidate regions. Can be defined. Therefore, when the rank determination unit 232 determines the spatial relationship between candidate regions in step S4321, only the spatial relationship between specific candidate regions needs to be determined, thereby improving the processing speed. Can do.

  Further, regarding the specific detection task, when the target object is a target person, the auxiliary information obtained by the rank determination unit 232 is, for example, joint point information of at least one target object (that is, the target person). is there. Here, the joint point information of the target person can be acquired using manual labeling or a joint point detection method. Further, the rank determination unit 232 may acquire an operation corresponding to the spatial relationship between the target person and the person / object associated with the target person by categorizing or recognizing the joint points of the target person. For example, when the target person is pulling a suitcase, the action corresponding to the spatial relationship between the target person and the suitcase is “pull”. Therefore, when the rank determination unit 232 determines a specific spatial relationship between the candidate regions in S4321, only the specific spatial relationship between the specific candidate regions is determined, and further, the specific corresponding to the specific movement is determined. Only the spatial relationship is determined, and the processing speed can be further improved.

  For a particular detected scene, generally a particular spatial relationship exists between that scene and the objects in the scene (eg, people, animals, etc.). For example, in grass / ploughs, flying animals (eg birds) are generally flying in the air and are unlikely to be walking on the ground. In addition, people or walking animals (such as sheep) generally walk on the ground, and rarely fly in the air. Therefore, for the specific detection scene, the auxiliary information acquired by the rank determining unit 232 is, for example, scene information (that is, background information of the input image). Furthermore, the rank determination unit 232 may specifically determine a specific spatial relationship between a specific object and a scene according to the scene information. Therefore, when the rank determination unit 232 determines the spatial relationship between the candidate regions in S4321, only a specific spatial relationship may be determined without determining all the spatial relationships. As a result, the processing speed can be improved.

  Returning to FIG. When the object region is detected from the image, the detection unit 230 sets the detection region of the object having a score equal to or higher than the threshold value or attached to the top N as the final detection result, as illustrated in FIG. The detection result is sent to the output device 160 via the system bus 180, and the final object detection result (for example, the position, score, and feature of the region) is displayed for the user, or person recognition processing (for example, person attribute recognition) The object detection area is output to subsequent processing such as person matching, person image search, and recognition or analysis of person behavior / behavior. For example, for recognition or analysis of a person's behavior / behavior or the like, the object area detected by the object detection apparatus 200 shown in FIG. 2 is preferably the target person and the target person / The object that is gripped / used and the area of the other person in the vicinity of the target person, so that the behavior / behavior between the target person and the accessory or the neighboring person can be determined from the spatial relationship between the areas. Can be directly recognized or analyzed. Further, for example, in the case of the target person and other persons adjacent to the target person, when the spatial relationship between the regions is “inclusion relationship”, the behavior of the target person is, for example, “holding” It can be inferred as “holding”. In addition, regarding the target person and the attachment of the target person, if the spatial relationship between the areas is “adjacency relationship”, it is inferred that the action of the target person is, for example, “grasping” it can. Also, in a human image search within a video segment, the spatial relationship between the target person and the target person's appendages generally does not vary significantly. Therefore, it may only be determined whether the target person in the segment of the video having a spatial relationship between the detected regions is similar. For example, it is only determined whether the target person pulling the suitcase in the video segment is similar.

  According to the first embodiment of the present invention, since the spatial relationship between individual feature points in the image is used when an object region is detected, these spatial relationships regulate object region detection. This makes it possible to better detect related objects. Since related objects are generally useful for monitoring a person during the monitoring process, not only the reproducibility of object detection but also the effect of monitoring a person can be improved according to the present invention.

  In the first embodiment of the present invention, the object detection operation is executed within one image. Since the spatial relationship between objects generally does not change significantly within a short duration, the present invention can also be used to perform object detection within a segment of video. FIG. 8 is a block diagram showing a configuration of an object detection apparatus 800 according to the second embodiment of the present invention. Here, some or all of the modules shown in FIG. 8 may be realized by dedicated hardware. As illustrated in FIG. 8, the object detection apparatus 800 includes a feature extraction unit 810, a candidate area detection unit 820, a spatial relationship determination unit 830, and a rank determination unit 840.

  First, the input device 150 shown in FIG. 1 receives a segment of video output from a particular electronic device (eg, a video camera, etc.) or input by a user. Next, the input device 150 transfers the received video to the object detection apparatus 800 via the system bus 180.

  Next, as shown in FIG. 8, the feature extraction unit 810 extracts features from the current video frame in the received video. Since the operation of feature extraction unit 810 is the same as that of extraction unit 210 shown in FIG. 2, the description thereof will not be repeated here.

  The candidate area detection unit 820 detects an object candidate area from the current video frame based on the features extracted by the feature extraction unit 810. Since the operation of the candidate area detection unit 820 is the same as that of the candidate area detection unit 231 shown in FIG. 3, the description thereof is not described here.

  The spatial relationship determination unit 830 determines the spatial relationship between candidate regions detected by the candidate region detection unit 820 based on the detection result of previous frames with respect to the current video frame. Here, the detection result of the previous frame with respect to the current video frame may be obtained according to the first embodiment of the present invention. In one embodiment, for example, a spatial relationship between regions of an object detected from any of the previous video frames is a spatial relationship between candidate regions in the current video frame. In other embodiments, for example, a wide range of spatial relationships between regions of objects detected from N video frames of the previous video frame (eg, obtained by performing mathematical operations such as weighting or averaging). ) Is a spatial relationship between candidate regions in the current video frame.

The rank determination unit 840 determines the ranks of the candidate areas detected by the candidate area detection unit 820 based on the spatial relationship between the candidate areas determined by the spatial relationship determination unit 830,
The candidate area after the rank determination is set as an object area.

  When the object area is detected from the current video frame, the rank determination unit 840 sets the object area having a score equal to or higher than a predetermined threshold as the final detection result or ranks the top N areas. 1 is used as the final detection result, and the final detection result is transferred to the output device 160 via the system bus 180 of FIG. The object's position, score, and features) for the user, or for subsequent processing such as person recognition processing (for example, person attribute recognition, person matching, person image search, person behavior / behavior recognition or analysis, etc.) The detection area is output.

  As an application example of the second embodiment of the present invention, the object detection apparatus 800 shown in FIG. 8 may be used to track a person in a video. Specifically, in the current video frame in the video, if the person in the current video frame can be successfully tracked using commonly used person tracking devices, the person in the current video frame is generally used. Detected using a person tracking device. If the person in the current video frame cannot be tracked successfully using a commonly used person tracking apparatus, the person in the current video frame may be detected using the object detection apparatus 800 shown in FIG. . This achieves person tracking throughout the video.

[Model generation]
As described in the first embodiment of the present invention, a model applicable to the present invention (ie, a pre-generated model) is pre-trained / pre-generated according to the learning of samples labeled with spatial relationships. The Here, as described above, in order to improve the processing speed of the present invention, for example, as shown in FIG. 5, the pre-generated model applied to the present invention determines, for example, the part to extract features and the spatial relationship. And a portion for detecting a region / candidate region. In the present invention, the pre-learning model may be generated based on training of samples labeled with spatial relationships using a deep learning method (for example, a neural network method). Here, each part of the pre-generated model in the present invention is configured by a network of a plurality of layers, for example, a part for extracting features is configured by an N layer network, and a part for determining a spatial relationship is configured by an M layer network In addition, the part for detecting the region / candidate region may be configured by a T layer network. Here, N, M, and T are natural numbers, and the values indicated by them may be the same or different.

  In one embodiment, in order to shorten the time required to generate the pre-generated model, the part for extracting the feature, the part for determining the spatial relationship, and the part for detecting the region / candidate region in the model are Updated simultaneously by means of propagation. FIG. 9 is a flowchart 900 schematically illustrating a generation method for generating a model applicable to the present invention. A flowchart 900 shown in FIG. 9 will be described using an example in which a neural network that generates a model applicable to the present invention is used. However, it is clear that this need not be limited. Here, the generation method according to FIG. 9 can also be executed by the hardware configuration 100 shown in FIG.

  As shown in FIG. 9, first, the CPU 110 shown in FIG. 1 acquires an initial neural network and a plurality of training samples set in advance by the input device 150. Here, each training sample is labeled with a spatial relationship, region location, and object category. The spatial relationship labeled in the training sample is, for example, “with / without spatial relationship”, “to which category the spatial relationship belongs”, and the like.

  Next, in step S910, on the other hand, the CPU 110, on the other hand, the training sample, a part of the current neural network for extracting features (for example, an initial neural network) and a part of the current neural network for determining the spatial relationship (Initial neural network) to get the spatial relationship that exists in the training sample. On the other hand, the CPU 110 determines a loss (eg, first loss Loss1) between the obtained spatial relationship and the sample spatial relationship. Here, the sample spatial relationship may be obtained according to the spatial relationship labeled in the training sample. The first loss Loss1 represents an error between the spatial relationship value of the predicted spatial relationship obtained using the current neural network and the spatial relationship value of the sample spatial relationship (ie, the real spatial relationship value), where The error is measured by distance, for example. For example, the first loss Loss1 can be obtained by the following equation (1).

Here, j represents the number of the spatial relation category to which the object in the training sample belongs, C represents the maximum number of spatial relation categories, and y _j represents the real spatial relation value of the object in the spatial relation category j. , P _j indicates the predicted spatial category value of the object of the spatial relationship category j.

  On the other hand, in step S920, CPU 110 passes the training samples through all current neural networks (eg, initial neural network), and obtains the object region / scenario region position and the object category of the object. That is, the CPU 110 passes the training sample through the current neural network of the part for feature extraction, the current neural network of the part for determining the spatial relationship, and the partial neural network for detecting the object area / candidate area. The object area / candidate area position and the object category of the object are obtained. On the other hand, for the obtained object region / candidate region position, the CPU 110 determines a loss (for example, a second loss Loss2) between the obtained object region / candidate region position and the sample region position. Here, the sample region location can be obtained according to the location of the region labeled on the training sample. Here, the second loss Loss2 represents an error between the prediction region / candidate region position obtained using the current neural network and the sample region position, and the error is measured by the distance. For example, the second loss Loss2 is obtained by the following equations (2) and (3).

Here, smooth _L1 (x) represents the difference between the region / candidate region position and the actual region position of the object, x represents the abscissa of the upper left corner of the object region / candidate region position, and y represents the object Represents the ordinate of the upper left corner of the position of the area / candidate area, w represents the width of the object area / candidate area, h represents the height of the object area / candidate area, and t ⁿ _i represents the object category An area / candidate area position of ⁿ objects is represented, and v ⁿ _i represents an actual area position of an object having an object category n.

  For the object category of the obtained object, the CPU 119 determines a loss (for example, the second loss Loss3) between the object category of the obtained object and the sample object category. Here, the sample object category can be obtained according to the object category labeled in the training sample. The third loss Loss3 represents an error between the predicted object category obtained by using the current neural network and the sample object category (that is, the real object category), and this error can be measured by, for example, distance. For example, the third loss Loss3 is obtained by the following equation (4).

Here, m represents the number of the object category of objects that training sample belongs, M represents the maximum number of objects category of objects that training sample belongs, y _m represents the real object category of objects in the object category m, p _m represents the predicted object category of the object of object category m.

  Returning to FIG. 9, in step S930, the CPU 110 determines a predetermined value based on the total loss (that is, the first loss Loss1, the second loss Loss2, and the third loss Loss3) obtained by all the current neural networks. It is determined whether or not the condition is satisfied. For example, if the sum of three losses is compared with a threshold sum (eg, TH2) and the sum of three losses / weighted sum is less than or equal to TH2, all current neural networks satisfy the predetermined condition It is determined and output as a final neural network (ie, a pre-generated model). Here, the final neural network is output to the storage device 240 shown in FIG. 2, for example, for the object detection described with reference to FIGS. If the sum of three losses / weighted sum is greater than TH2, it is determined that all the current neural networks do not satisfy the predetermined condition, and the generation process proceeds to step S940.

  In step S940, CPU 110 updates the parameters of each layer of the current neural network for determining the spatial radius based on first loss Loss1. Here, the parameter of each layer is, for example, the weight of each convolution layer of the current neural network. In one example, the parameters for each layer are updated based on the first loss Loss1, for example by using a stochastic gradient descent method.

  In step S950, CPU 110 updates the parameters of each layer in the current neural network of the part for detecting the object area / candidate area based on second loss Loss2 and third loss Loss3. The parameter of each layer here is also the weight of the convolution layer in the current neural network, for example. In one embodiment, the parameters for each layer are updated based on the second loss Loss2 and the third loss Loss3, for example using a stochastic gradient descent method.

  In step S960, CPU 110 updates parameters of each layer of the current neural network of the feature extraction portion based on first loss Loss1, second loss Loss2, and third loss Loss3. Here, the parameter of each layer is also a weight in each convolution layer in the current neural network, for example. In one example, the parameters for each layer are also obtained using a stochastic gradient descent method with a first loss Loss1. Updated based on the second loss Loss2 and the third loss Loss3. Thereafter, the generation process proceeds to step S910 again.

  In the flowchart shown in FIG. 9, whether or not the sum / weight sum of three losses of the first loss Loss1, the second loss Loss2, and the third loss Loss3 satisfies a predetermined condition is the current neural network. The condition to stop the update of. However, it is clear that this need not be limited. For example, although step S930 is omitted, after the number of updates to the current neural network reaches a predetermined number, the corresponding update operation is stopped.

  All of the above units are exemplary and / or preferred modules for performing the processes described in this disclosure. These units may be hardware units such as field programmable gate arrays (FPGAs), digital signal processors, application specific integrated circuits, and / or software modules such as computer readable programs. The units for performing each step are not described in detail above. However, when there are steps to perform a particular process, it may be a corresponding functional module or unit (implemented by hardware and / or software) for performing the same process. The technical solutions of all combinations of steps according to the description and the units corresponding to these steps are included in the disclosure content of this application as long as the technical solutions constituted by them are complete and applicable.

  The method and apparatus of the present invention can be implemented in a number of ways. For example, the method and apparatus of the present invention may be implemented by software, hardware, firmware, or any combination thereof. The above order of steps of the method is intended to be exemplary only, and the steps of the method of the invention are not limited to the order specifically described above unless otherwise specified. Further, in some embodiments, the present invention can also be implemented as a program recorded on a recording medium that includes machine-readable instructions for performing the method according to the present invention. Therefore, the present invention also includes a recording medium on which a program for executing the method according to the present invention is recorded.

  While several specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above embodiments are merely exemplary and do not limit the scope of the invention. Those skilled in the art will recognize that the above embodiments can be modified without departing from the scope and spirit of the invention. The scope of the present invention is defined by the appended claims.

Claims (21)

An object detection device,
Extraction means for extracting features from the image;
Determining means for determining a spatial relationship between individual feature points in the image based on the extracted features;
An object detection apparatus comprising: detecting means for detecting a region of an object in the image based on the determined spatial relationship; and wherein the object is an associated object in the image.   The object according to claim 1, wherein the determination unit determines a spatial relationship between the feature points and a spatial relationship value thereof based on the extracted features using a pre-generated model. Detection device. The detection means includes
Candidate area detecting means for detecting a candidate area of an object from an image based on the extracted features;
The object detection according to claim 1, further comprising: a rank determination unit that determines the rank of the candidate area based on the determined spatial relationship and sets the candidate area after the rank determination as the area of the object. apparatus.   The object detection apparatus according to claim 3, wherein the determination unit determines a spatial relationship between the feature points based on the extracted features and a spatial relationship value thereof using a pre-generated model.   The object detection apparatus according to claim 4, wherein the rank determination unit updates a score of the candidate area based on a spatial relation value of a spatial relation between the candidate areas.   The spatial relationship between the two candidate game odds in any two candidate regions is determined based on a mutual spatial relationship between feature points included in the two candidate regions. 6. The object detection device according to 5. In any two candidate regions, the spatial relationship between the two candidate regions is
Position information and category information of at least one target object in the two candidate areas,
Coupling point information of at least one target object in the two candidate regions;
Background information of the image,
The object detection apparatus according to claim 6, wherein the determination is based on at least one of the following. A spatial relationship between two candidate regions represents a spatial constraint between the two candidate regions,
The spatial constraint is at least a relative positional relationship between the two candidate regions,
A phase relationship between the two candidate regions;
The object detection apparatus according to claim 5, comprising at least one of a relative shape relationship between the two candidate regions. The extraction means extracts features from the image using a pre-generated model;
The object detection apparatus according to claim 2, wherein the detection unit detects an object region from the image using the pre-generated model.   The object detection apparatus according to claim 9, wherein the pre-generated model is generated based on a training sample labeled with a spatial relationship using a deep learning method. The pre-generated model has at least three parts: a part for extracting features, a part for determining a spatial relationship, and a part for detecting a region of an object;
In the generation process of the pre-generated model, a current part for extracting features, a current part for determining a spatial relationship, and a current part for detecting a region of an object are obtained by means of back propagation. The object detection apparatus according to claim 10, wherein the object detection apparatus is updated at the same time. An object detection method,
An extraction step for extracting features from the image;
A determination step of determining a spatial relationship between individual feature points in the image based on the extracted features;
An object detection method comprising: detecting a region of an object in the image based on the determined spatial relationship; and wherein the object is an associated object in the image.   13. The determination step according to claim 12, wherein a spatial relationship between the feature points and a spatial relationship value thereof are determined based on the extracted features using a pre-generated model. Object detection method. The detecting step includes
A candidate area detection step of detecting a candidate area of the object from the image based on the extracted features;
The object detection method according to claim 12, further comprising: determining a rank of the candidate area based on the determined spatial relationship, and determining a rank of the candidate area after the rank determination as the object area. Method.   15. The object detection method according to claim 14, wherein in the determination step, a spatial relationship between the feature points based on the extracted features and a spatial relationship value thereof are determined using a pre-generated model. .   The object detection method according to claim 15, wherein, in the rank determination step, a score of the candidate area is updated based on a spatial relation value of a spatial relation between the candidate areas.   The spatial relationship between the two candidate game odds in any two candidate regions is determined based on a mutual spatial relationship between feature points included in the two candidate regions. 17. The object detection method according to 16. The spatial relationship between two candidate regions represents the spatial constraint between the two candidate regions,
The spatial constraint is at least a relative positional relationship between the two candidate regions,
A phase relationship between the two candidate regions;
The object detection method according to claim 16, comprising at least one of a relative shape relationship between the two candidate regions. In the extraction step, the features are extracted from the image using a pre-generated model,
The object detection method according to claim 13 or 15, wherein in the detection step, the region of the object is detected from the image using the pre-generated model. Feature extraction means for extracting features from the current video frame in the video;
Candidate area detection means for detecting a candidate area of an object from the current video frame based on the extracted features;
A spatial relationship determination means for determining a spatial relationship between the candidate regions based on a detection result of a previous video frame of the current video frame;
Based on the spatial relationship between the candidate areas, the rank of the candidate areas is determined, and the candidate areas after the determination of the ranks have rank determination means for area of the object,
12. The object detection device according to claim 1, wherein the detection result of the previous video frame of the current video frame is obtained using the object detection device according to any one of claims 1 to 11.   A storage medium for storing a command for executing the object detection method according to any one of claims 12 to 19 when the computer executes the program.