JPWO2019235050A1 - Object detection method and object detection device - Google Patents

Abstract

The object detection method is preset in a candidate detection step (S1) for detecting a candidate for an area occupied by a plurality of objects on an image, and for each detected candidate, with the above candidate as one candidate and around one candidate. Detection based on the relationship score detection step (S2) for detecting the relationship score representing the likelihood of a predetermined relationship with another candidate of one candidate by referring to other candidates in the range, and a predetermined algorithm. An object determination step (S3) for determining the regions of a plurality of objects from the determined candidates, and a relationship determination step (S3) for determining the relationship between a plurality of objects based on the relationship scores detected for the pair of candidates corresponding to the determined regions. S4) and is included.

Description

The present invention relates to an object detection method and an object detection device for detecting a plurality of objects and relationships between the plurality of objects from an image.

In recent years, a technique has been proposed in which a plurality of objects are detected from an image and a relationship existing between the plurality of objects is estimated. For example, in Non-Patent Document 1, a plurality of CNNs (Convolutional Neural Networks) are used to gradually narrow down the candidates for objects and the candidates for relationships, thereby estimating the relationships between the plurality of objects and them. More specifically, a CNN for detecting a person from an image, a CNN for detecting an object other than the person (for example, an umbrella), and a CNN for detecting the relationship between them are prepared, and each CNN prepares a person, After extracting each candidate of object and relationship, each extracted candidate is integrated, and then convolution is performed to narrow down each candidate.

Ji Zhang, et al., “Relationship Proposal Networks”, Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference, July 21-26, 2017

However, in the technique of Non-Patent Document 1, intermediate processing (integration and convolution) for narrowing down the candidates is required, and after convolution is performed by a plurality of CNNs, convolution is performed again after the integration of each candidate. , The convolution process is also redundant. Therefore, in the technique of Non-Patent Document 1, it is difficult to estimate (detect) a plurality of objects and the relationship between the objects at high speed from the image.

The present invention has been made to solve the above problems, and an object of the present invention is to provide an object detection method and an object detection device capable of detecting a plurality of objects and their relationships from an image at high speed. There is.

The object detection method according to one aspect of the present invention is an object detection method for detecting a plurality of objects and the relationship between the plurality of objects from an image, and detects candidates for a region occupied by the plurality of objects on the image. With reference to the candidate detection step to be performed and other candidates within a preset range around the one candidate, with the candidate as one candidate for each detected candidate, the other candidate of the one candidate is referred to. A relationship score detection step that detects a relationship score that represents the likelihood of a predetermined relationship with respect to a candidate, and an object determination step that determines the region of the plurality of objects from the detected candidates based on a predetermined algorithm. It includes a relationship determination step of determining the relationship of the plurality of objects based on the relationship score detected for the pair of candidates corresponding to the region.

The object detection device according to another aspect of the present invention is an object detection device that detects a plurality of objects and the relationship between the plurality of objects from an image, and can be a candidate for a region occupied by the plurality of objects on the image. In addition to detecting, for each of the detected candidates, the candidate is set as one candidate, and other candidates within a preset range around the one candidate are referred to, and the other candidate of the one candidate is referred to. A candidate / relationship score detection unit that detects a relationship score representing the likelihood of a predetermined relationship with respect to the object, and an object determination unit that determines the region of the plurality of objects from the detected candidates based on a predetermined algorithm. Includes a relationship determination unit that determines the relationship of the plurality of objects based on the relationship score detected for the pair of candidates corresponding to the region.

In order to detect candidates in the area occupied by a plurality of objects on the image and detect the relationship score indicating the likelihood of the relationship between the detected candidates, the detection of the candidate and the detection of the relationship score, for example, a single CNN. It can be performed simultaneously (in parallel, collectively) by the neuro operation used. This makes it possible to detect a plurality of objects and the relationships between the objects at high speed based on the relationship score.

It is a block diagram which shows the schematic structure of the object detection apparatus of embodiment of this invention. It is explanatory drawing which shows typically the structure of the candidate / relationship score detection part which the object detection apparatus has. It is a flowchart which shows the process flow of the object detection method by the said object detection apparatus. It is explanatory drawing which shows typically the procedure of the detection of a candidate by the said candidate / relationship score detection part. It is explanatory drawing which shows typically the example of detecting a plurality of candidates with respect to a specific grid. It is explanatory drawing which shows typically the example of detecting a plurality of candidates with respect to other grids. It is explanatory drawing which shows typically the class which has the maximum class score for each grid. It is explanatory drawing which shows typically the procedure of the detection of the relational score by the said candidate / relational score detection part. It is explanatory drawing which shows the feature map output from the said candidate / relationship score detection part. It is explanatory drawing which shows the state which the object determination part of the object detection apparatus determined the region of a plurality of objects from a plurality of candidates. It is explanatory drawing which graphically represented the relationship about two candidates selected from the candidates corresponding to each area shown in FIG. It is explanatory drawing which shows typically the relationship between two candidates before and after the threshold processing. It is explanatory drawing which shows typically the relationship between two candidates before and after narrowing down a relational score. It is a block diagram which shows the other structure of the said object detection apparatus. It is a flowchart which shows the process flow of the object detection method by the object detection apparatus of FIG.

An embodiment of the present invention will be described below with reference to the drawings. The present invention is not limited to the following contents.

[Configuration of object detection device]
FIG. 1 is a block diagram showing a schematic configuration of the object detection device 1 of the present embodiment. The object detection device 1 is composed of, for example, a terminal device such as a personal computer, and is connected to at least one surveillance camera 10 installed in the store via a communication line (whether wired or wireless). There is. When the data of the image (moving image or still image) acquired by the shooting by the surveillance camera 10 is input to the object detection device 1, the object detection device 1 executes a process by the object detection method described later, thereby performing a process. Multiple objects on the image and the relationships between each object are detected.

In addition, in this specification, "relationship" includes at least one of the relationship or state between a plurality of objects, and the action or action of one object on the other object. Therefore, for example, when a person holds an umbrella on an image, the state of "holding" or the action of "holding" corresponds to the above relationship. In addition, actions or states such as "a person holding an umbrella" or "belonging" to B are also included in the above relationship.

The object detection device 1 does not necessarily have to be connected to the surveillance camera 10, and may have a configuration in which image data can be acquired from the outside. For example, the object detection device 1 acquires the image data by receiving an e-mail with an image data file attached from another terminal device, or reads the image data recorded on the recording medium by a reading device. Therefore, the image data necessary for object detection may be acquired. Hereinafter, the details of the object detection device 1 will be described.

The object detection device 1 includes a control unit 2, a storage unit 3, an input unit 4, a display unit 5, a communication unit 6, and a detection processing unit 7.

The control unit 2 is composed of, for example, a central processing unit (CPU), operates according to an operation program stored in the storage unit 3, and controls the operation of each unit of the object detection device 1.

The storage unit 3 is a memory that stores the above-mentioned operation program, image data acquired by the surveillance camera 10, and the like. The storage unit 3 is composed of, for example, a hard disk, but may be appropriately selected from recording media such as a RAM (Random Access Memory), a ROM (Read Only Memory), an optical disk, a magneto-optical disk, and a non-volatile memory. ..

The input unit 4 is composed of, for example, a keyboard, a mouse, a touch pad, a touch panel, and the like, and receives various instruction inputs by the user. The display unit 5 is a device that displays various information such as an image acquired by the surveillance camera 10 and a result detected by the detection processing unit 7 described later (for example, the relationship between objects), and is, for example, a liquid crystal display. It consists of devices. The communication unit 6 is an interface for communicating with the outside (including the surveillance camera 10), and includes input / output terminals and the like. For example, when the surveillance camera 10 and the object detection device 1 wirelessly communicate with each other (for example, transmission / reception of image data), the communication unit 6 may be configured to include an antenna, a transmission / reception circuit, a modulation circuit, a demodulation circuit, and the like. Good.

The detection processing unit 7 is a block that performs object detection processing for detecting a plurality of objects and the relationship between the plurality of objects from an image, and is a candidate / relationship score detection unit 7a, an object determination unit 7b, and a relationship determination unit 7c. It is composed including and. The detection processing unit 7 is composed of, for example, a GPU (Graphics Processing Unit) which is an arithmetic unit specialized in real-time image processing, but may be composed of the same CPU as the control unit 2 or a separate CPU, and other CPUs. It may be composed of an arithmetic unit.

The candidate / relationship score detection unit 7a is composed of, for example, CNN, and when an image is input, it detects candidates for an area occupied by a plurality of objects on the image (a rectangular frame also called a bounding box). , Detects a relationship score that represents the likelihood (probability, reliability) of a predetermined relationship between a plurality of detected candidates. In detecting the above candidates, for example, Joseph Redmon, et al., "You Only Look Once: Unified, Real-Time Object Detection", Computer Vision and Pattern Recognition (cs.CV), Submitted on 8 Jun 2015 ( The technology described in v1), last revised 9 May 2016 (hereinafter abbreviated as “YOLO”) can be adopted, but for the detection of the above relationship score and the determination of the relationship based on the relationship score, “YOLO” Is not mentioned at all.

Further, the candidate / relationship score detection unit 7a has a relative position of the candidate with respect to each of the plurality of grids set on the image (for example, the position of the center of the candidate with respect to the center of the grid) and a relative size of the candidate with respect to the image (for example, the position of the center of the candidate with respect to the center of the grid). CNN detects at least one of the following parameters: vertical and horizontal dimensions), a detection score that represents the likelihood of a candidate for an object, and a class score that represents the likelihood of a class (type) of an object in the grid. The likelihood of the candidate with respect to the object represents the certainty that the candidate indicates the object and / or the certainty that the candidate fits the object. The likelihood of a class of an object represents the certainty of the class to which the object belongs in the grid. In detecting each of the above parameters (relative positions of candidates, etc.), the "YOLO" technology can be adopted.

Here, the description of the CNN constituting the candidate / relationship score detection unit 7a of the present embodiment will be supplemented.

FIG. 2 is an explanatory diagram schematically showing the configuration of the candidate / relationship score detection unit 7a (CNN). The CNN includes an input layer 11, a convolutional layer 12, a pooling layer 13, and an output layer 14. The convolution layer 12 and the pooling layer 13 may have at least one set, but may have a plurality of sets.

Each of the above-mentioned layers constituting the CNN has a plurality of nodes (or units), and at least a part of the plurality of nodes is connected by an edge between the layers. A neural network is an information processing system that imitates a human neural network, and the above-mentioned nodes represent an engineering neuron model corresponding to a human nerve cell. Each layer has a function called an activation function (response function), and edges have weights. Therefore, the value output from the node of each layer is calculated from the value of the node of the previous layer, the weight of the edge, and the activation function of the layer. The edge weight can be changed by learning.

Data (pixel value) of each pixel constituting one image is input to each node of the input layer 11. The convolution layer 12 filters the values output from a predetermined node in the previous layer to obtain a feature map. The pooling layer 13 further reduces the feature map output from the convolution layer 12 to obtain a new feature map. The output layer 14 is the final layer of the CNN, and from the node values of the previous layer, the edge weights, and the activation function of the output layer 14, each of the above parameters (region candidate, relationship score, candidate) is used. Relative position of, relative size of candidates, detection score, class score) is output. The output layer 14 may be composed of a convolution layer to be filtered, or a fully connected layer that combines the outputs from all the nodes of the previous layer, performs a predetermined operation, and outputs each parameter. It may be composed of.

As the learning algorithm of the candidate / relationship score detection unit 7a (CNN), for example, an image data with a correct answer is used, and an output value from the output layer 14 obtained when the above image data is input and a value indicating the correct answer are used. Backpropagation is an error backpropagation method in which the weight of each layer (edge) is sequentially changed from the output layer 14 side to the input layer 11 side by using the steepest descent method so that the squared error of is minimized. ) Can be used. By learning the CNN in advance in this way, when an image containing a plurality of objects to be detected is input to the CNN, each of the above-mentioned parameters is output from the CNN to detect (estimate) them. be able to.

The object determination unit 7b determines a region of a plurality of objects from a plurality of object candidates detected by the candidate / relationship score detection unit 7a based on a predetermined algorithm. The relationship determination unit 7c determines the relationship between a plurality of objects based on the relationship scores detected for the pair of candidates corresponding to the region (of the plurality of objects) determined by the object determination unit 7b. The details of determining the region of the object and determining the relationship between the objects will be described in the following description of the object detection method.

[Object detection method]
Next, the object detection method of the present embodiment will be described. FIG. 3 is a flowchart showing a processing flow of the object detection method by the object detection device 1 of FIG. The object detection method of the present embodiment includes a candidate detection step (S1), a relationship score detection step (S2), an object determination step (S3), and a relationship determination step (S4). Hereinafter, a more detailed description will be given.

(S1; candidate detection step)
In S1, the candidate / relationship score detection unit 7a of the detection processing unit 7 detects a candidate for a region occupied by a plurality of objects on the image by a neuro operation (CNN) based on a predetermined algorithm. More details are as follows.

FIG. 4 schematically shows a procedure for detecting a candidate by the candidate / relationship score detecting unit 7a. As shown in the figure, the candidate / relationship score detection unit 7a has a plurality of grids of H × W (4 × 4 = 16 as an example in FIG. 4) with H and W as integers of 2 or more. For each grid when divided into, two candidate Bs are detected with a class score, which is common to all classes. As the input image, here, features are extracted by the folding layer 12 and the pooling layer 13 from the image input to the output layer 14 (see FIG. 2) of the CNN, that is, the image input to the input layer 11. Consider the image after. In the following description, the input image is also simply referred to as an image.

When the number of candidate Bs to be detected for each grid is K, K = 2 in FIG. 4, but K can be set to a value other than 2. Further, the value of K may be set (to a different value) for each class.

Further, in S1, the candidate / relationship score detection unit 7a uses the relative position of the candidate B with respect to each of the plurality of grids set on the image, the relative size of the candidate B with respect to the image, the detection score of the candidate B, and each grid. Further detect the class score of. Here, the relative positions of the candidate B with respect to each grid are the relative positions of the center of the candidate B in the vertical direction (H direction) and the horizontal direction (W direction) with respect to the center of each grid, with respect to the vertical direction and the horizontal direction. It is two-dimensional information. Further, the relative size of the candidate B is the vertical and horizontal sizes of the candidate B with respect to the vertical and horizontal sizes of the entire image, and is the same as the relative position of the candidate B in the vertical and horizontal directions. It is two-dimensional information about. Further, the detection score of the candidate B is information on the certainty that the candidate B indicates an object on the image and / or information on the certainty that the candidate B fits the object, and P = 1. Or 2, it can be said that it is P-dimensional information. Further, the class score can be said to be C-dimensional information because C types are detected for each grid, where C is the number of classes to be set.

Therefore, if the detection of the relational score described later is not considered, the feature map output from the candidate / relational score detection unit 7a (output layer 14 of CNN) is represented by H × W × {K (2 + 2 + P) + C)}. It will be the tensor on the 3rd floor.

FIG. 5 schematically shows an example of detecting a plurality of candidate Bs for a specific grid. In FIG. 4, the vertical and horizontal sizes of the candidate B to be detected are shown as constant, but since the CNN constituting the candidate / relationship score detection unit 7a has been learned in advance using a predetermined image pattern, the candidate to be detected can be detected. The vertical and horizontal sizes of B can be changed according to the image or the object on the image. For example, the position of each grid is represented by (h, w) with the coordinates in the H direction as h and the coordinates in the W direction as w, and the grid (see the diagonal grid) of (h, w) = (1, 2). _{), The candidate / relationship score detection unit 7a detects two candidates B 1} and B ₂ whose centers are located in the grid as candidates B of a plurality of object regions (here, K = 2). Since it is set, the number of candidates detected is two). In the example of FIG. 5, the candidate / relationship score detection unit 7a shows an example in which a frame that fits an object (for example, an umbrella) is detected as _{candidate B 2.}

FIG. 6 schematically shows an example of detecting a plurality of candidate Bs with respect to other grids. Here, an example of detecting candidate B for each grid of (h, w) = (3,2), (3,4), (4,3) is shown. In this way, the same detection of candidate B as in FIG. 5 is performed for other grids, and finally, a plurality of candidate Bs are detected for all grids.

FIG. 7 schematically shows the class having the maximum class score for each grid based on the class score detected for each grid (the difference between the classes is shown by the difference in hatching). In the figure, as an example, the area of the grid with the highest class score for "umbrella", the area of the grid with the highest class score for "person with an umbrella", and the class for "other person". It is shown separately from the area of the grid that has the highest score. When the candidate / relationship score detection unit 7a detects the class score for each grid, it is possible to detect which class is most likely in each grid based on the class score.

(S2; Relationship score detection step)
In S2, the candidate / relationship score detection unit 7a detects the relationship score representing the likelihood of the relationship between each candidate by a neuro operation (CNN) based on a predetermined algorithm. The relationship score detection step of S2 is executed in parallel with the candidate detection step of S1 described above. Here, the candidate detection step of S1 and the relationship score detection step of S2 are collectively referred to as a candidate / relationship score detection step S10. Hereinafter, the details of the relationship score detection step of S2 will be described.

FIG. 8 schematically shows a procedure for detecting a relationship score by the candidate / relationship score detection unit 7a. Here, one candidate detected in any grid i within the range of H'× W'(where H'<H, W'<W) is defined as B (k, i). However, k is any integer from 1 to K, i is any integer from 1 to H'× W', and the grid i is the i-th within the range of H'× W'. Point to the grid of. Further, let B (k, j) be another candidate detected in any grid j within the range of H'x W'. However, j is an integer from 1 to H'xW', and grid j points to the jth grid within the range of H'xW'.

As shown in the figure, the candidate / relationship score detection unit 7a extracts one candidate B (k, i) from each candidate B detected in S1 and of the one candidate B (k, i). One candidate B (k, i) with reference to other candidates B (k, j), B (k, j + 1), ... Within the surrounding preset range of H'x W'. Relationship scores for other candidates B (k, j), B (k, j + 1), ... Are detected, and this process is performed for each candidate B detected in S1 (one extracted from each candidate B). (Change the candidates in order). Since the CNN constituting the candidate / relationship score detection unit 7a is learned in advance using a predetermined image pattern, the relationship score of one candidate B (k, i) with respect to another candidate B (k, j) or the like is obtained. Can also be detected by neuro-calculation on CNN. In FIG. 8, for convenience, one candidate B (k, i) and other candidates B (k, j), B (k, j + 1), ... Are shown in a fixed size. The point that the vertical and horizontal sizes change according to the image or the object is the same as the above.

Here, when the case where the relation R between the two candidates exists is set to R = 1, and the case where the relation R does not exist is set to R = 0, the case of one candidate B (k, i) in the grid i is set to The presence / absence of a relation R with respect to another candidate B (k, j) in the grid j and its probability (relationship score P) are expressed as follows, for example. That is, the existence of the relationship R of one candidate B (k, i) in the grid i with respect to the other candidate B (k, j) in the grid j is such that R = 1 | B (k, i), B ( It is represented by k, j), and the probability that the relationship R exists (relationship score P) is represented by P {R = 1 | B (k, i), B (k, j)}. Similarly, the absence of the relation R of one candidate B (k, i) in the grid i to the other candidate B (k, j) in the grid j is R = 0 | B (k, i) ,. It is represented by B (k, j), and the probability that the relation R does not exist (relationship score P) is represented by P {R = 0 | B (k, i), B (k, j)}. The candidate / relationship score detection unit 7a detects at least the relationship score P (in the case of R = 1) in which the relationship R exists between the candidates by CNN. In addition, P {R = 1 | B (k, i), B (k, j)} and P {R = 0 | B (k, i), B (k, j)} described above are both. , 0 to 1 and P {R = 1 | B (k, i), B (k, j)} + P {R = 0 | B (k, i), B (k, j)} = 1.

Further, when one candidate B (k, i) corresponds to the area of "person" on the image and the other candidate B (k, j) corresponds to the area of "umbrella", one candidate B (k, i) corresponds to the area of "umbrella". As an example of the relationship R of the candidate B (k, i) with respect to the other candidate B (k, j), "having" in the case of "a person holding an umbrella" can be considered. In this case, the probability that one candidate B (k, i) in the grid i has a relationship R (relationship of "having") with respect to another candidate B (k, j) in the grid j (relationship score P). Can be expressed as P {having = 1 | B (k, i), B (k, j)}.

FIG. 9 shows a feature map output from the candidate / relationship score detection unit 7a (CNN output layer 14). In the above-mentioned relationship score P (here, the score for R = 1 is considered), there are K candidates B (k, i) for each grid within the range of H'× W', and candidate B (k). , J) also exists, so the total number of relational scores P obtained within the range of H'xW'is H'xW'xKxK (= H'W'KK). Become. In the present embodiment, the candidate / relationship score detection unit 7a detects the candidate in S1 (neuro calculation by CNN) and the detection of the relationship score P in S2 (neuro calculation by CNN) by a single CNN. Do it all together (at the same time, in parallel). As a result, the feature map output from the candidate / relationship score detection unit 7a is a tensor on the third floor represented by H × W × {K (2 + 2 + P) + C) + H'W'KK}, as shown in the figure. It becomes.

(S3; object determination step)
In S3, the object determination unit 7b determines the regions of a plurality of objects from the candidates B of the plurality of objects detected in S1 based on a predetermined algorithm. Here, the above-mentioned predetermined algorithm is Non-Maximum Suppression (NMS). The NMS is recognized as the same class, and among the overlapping regions (candidate B), the region (or candidate) having the lower score (detection score or class score obtained in S1) is removed, and the region having the higher score is removed. It is an algorithm that suppresses duplication by leaving an area (or candidate). The "overlap" of the two regions (or candidates) can be determined based on the IOU (Intersection over Union) value and the threshold value. The IoU value is, for example, when focusing on two candidates p and q, {(the area of the region included in both the candidate p and the candidate q) / (the area of the region included in at least one of the candidates p and q)}. It refers to a value (ratio), and when this IoT value is equal to or greater than a threshold value, it can be determined that the two candidates p and q "overlap".

In FIG. 10, the object determination unit 7b applied NMS to the plurality of candidates (see the frame of the broken line) shown in FIG. 6 to determine the regions B _A , B _B , _BC , and _BD of the plurality of objects. Indicates the state. Areas B _A , B _B , _BC , and _BD are the areas of "the person with an umbrella", the area of "umbrella", and the "face of the person facing the person with an umbrella", respectively, on the image. It corresponds to the area, "the whole person facing the person holding the umbrella". In this way, by applying NMS, an appropriate candidate can be narrowed down from the candidates of the overlapping region, and the narrowed down candidate can be appropriately determined as the region of the object on the image.

(S4; relationship determination step)
In S4, the relationship determination unit 7c determines the relationship between a plurality of objects based on the relationship score P detected in S2 for the candidate pair (pair) corresponding to the region determined in S3. More specifically, it is as follows.

First, the relationship determination unit 7c graphically represents the relationship for the pair of candidates corresponding to the region determined in S3. FIG. 11 is a graphical representation of the relationship between two candidates selected from the candidates B _a , B _b , B _c , and B _{d corresponding to the respective regions B A} , B _B , _BC , and _{BD shown in FIG.} It is a figure that was made. In FIG. 11, the relationship between the two candidates is indicated by an arrow, and the thickness (line width) of the arrow is expressed corresponding to the value of the relationship score P. The candidate at the start point of the arrow corresponds to one candidate B (k, i) detected in any grid i within the range of H'× W'shown in FIG. The candidate at the end point corresponds to another candidate B (k, j) detected in any grid j within the range of H'x W'.

In FIG. 11, for example, the arrow _{from candidate B a} to candidate B _{b has a relationship between candidate B a} (“person”) and candidate B _b (“umbrella”) (a relationship in which a person “holds” an umbrella). It shows that there is, and that the relationship score P, which indicates the likelihood of the relationship of "having", is the maximum (because the line width of the arrow is the largest). For any of the two candidates selected from the candidates B _a , B _b , B _c , and B _d , the relationship score P between them has already been detected in the relationship score detection step of S2, so that the relationship determination unit In 7c, as shown in FIG. 11, the relationship between the two candidates can be graphed using the relationship score P.

As shown in FIG. 11, when there are a plurality of relationship scores P (when the number of arrows is a plurality in FIG. 11), the relationship determination unit 7c performs threshold processing on the plurality of relationship scores P and is less than the threshold value. Exclude the relationship score P. FIG. 12 schematically shows the relationship between the two candidates before and after the threshold processing. By excluding the relationship score P below the threshold value in this way, only the arrow indicating the relationship score P above the threshold value remains. As a result, it is possible to narrow down the candidate pairs for which the relationship needs to be determined from the plurality of candidate pairs. In the figure, as a result of the threshold processing, only the relationship score P1 obtained _{from the candidate B a} to the candidate B _{b and the relationship score P2 obtained from the candidate B b} to the candidate B _a remain, and a pair of a plurality of candidates remains. It is shown that the pairs for which the relationship needs to be determined _{are narrowed down to the pair of candidate B a} and the pair of candidate B _b.

As described above, as the relationship score P obtained for the pair consisting of the _{two candidates B a} and B _{b, the relationship score P1 obtained from one candidate B a} to the other candidate B _b and the other candidate B _b When there is a relationship score P2 obtained for one candidate B _a , the relationship determination unit 7c excludes the smaller relationship score. FIG. 13 schematically shows the relationship between the two candidates before and after narrowing down the relationship score P. By excluding the smaller relationship score (relationship score P2 in the figure) from the relationship scores P1 and P2 obtained for the same two candidates B _a and B _{b, only the highly reliable relationship score P1 remains.} .. As a result, the relationship between the two candidates B _a and B _b , that is, the relationship between the two objects corresponding to the two _{candidates B a} and B _{b can be appropriately determined based on the remaining relationship score P1.} it can. In the example of the figure, the relationship determination unit 7c determines the relationship between _{the object corresponding to the candidate B a} (“person”) and the object corresponding to the _{candidate B b (“umbrella”) based on the remaining relationship score P1.} , The person will decide the relationship of "holding" an umbrella.

〔effect〕
As described above, in the object detection method of the present embodiment, the candidate / relationship score detection unit 7a detects candidate B in the area occupied by a plurality of objects on the image (S1), and for each detected candidate B, With the above candidate as one candidate B (k, i), the relationship score P with respect to the other candidate B (k, j) within the preset range of H'× W'is detected (S2). Then, the object determination unit 7b determines the regions of a plurality of objects from the plurality of candidates B based on a predetermined algorithm (for example, NMS) (S3), and the relationship determination unit 7c determines a plurality of regions based on the relationship score P. The relationship between the objects of (S4) is determined. The detection of the candidate B and the detection of the relationship score P are performed collectively by the CNNs constituting the candidate / relationship score detection unit 7a, that is, a single CNN (at the same time, in parallel, in a non-stepwise manner, in a single-shot). Therefore, there is no need for intermediate processing (integration, convolution) for candidate selection as in the conventional technology that extracts candidates such as people using multiple CNNs, integrates the extracted candidates, and convolves again. Yes, the convolution process is not redundant. This makes it possible to detect a plurality of objects and their relationships at high speed from the image based on the relationship score. As a result, the object detection method and the object detection device 1 of the present embodiment can be easily applied to applications that require such high-speed detection.

In particular, since the candidate / relationship score detection unit 7a performs the detection of the candidate B in S1 and the detection of the relationship score P in S2 in parallel by the neuro operation, a CNN for detecting a plurality of objects and relationships is generated. It can be one type, and can detect a plurality of objects and relationships with a simpler configuration and at a higher speed than the above-mentioned conventional technique.

Further, the above neuro operation is an operation by CNN. In this case, since the detection of the candidate B and the detection of the relationship score P can be performed simultaneously in parallel with a single CNN, a plurality of objects and their relationships can be detected at high speed based on the detected relationship score P. It is definitely possible to do so.

The neural network constituting the candidate / relationship score detection unit 7a is not limited to CNN, and may be an MLP (multilayer perceptron) in which the nodes of each layer are fully connected, or an RNN (Recurrent). It may be another neural network such as Neural Networks). However, in relationship recognition (detection), it is necessary to model (learn) a much more complicated appearance (image) than in general object type recognition, but it is easy to process. It is desirable to use CNN because it enables recognition with a small amount of training data.

Further, the candidate / relationship score detection unit 7a includes the relative position of the candidate B with respect to each of the plurality of grids set on the image, the relative size of the candidate B with respect to the image, the detection score of the candidate B with respect to the object, and the object on the grid. Further detect at least one of the parameters of the class score of. In this case, the candidate B can be detected by effectively utilizing the conventional "YOLO" technology, and the region of the object can be determined from the plurality of candidate Bs by the NMS processing based on the detection score or the like. Become.

[Other configurations of object detection device]
FIG. 14 is a block diagram showing another configuration of the object detection device 1. The object detection device 1 of FIG. 14 is exactly the same as the configuration of FIG. 1 except that the detection processing unit 7 further has an explanatory text creation unit 7d. Further, FIG. 15 is a flowchart showing a processing flow of the object detection method by the object detection device 1 of FIG. 14, except that the explanatory text creation step (S5) is added after the relationship determination step (S4). It is exactly the same as the flowchart of 3. Hereinafter, parts different from those in FIGS. 1 and 3 will be described.

The description creation unit 7d creates a description of the image based on the region of the plurality of objects determined in S3 and the relationship between the plurality of objects determined in S4. Such an explanatory text creating unit 7d is composed of a neural network such as CNN or RNN that performs a neuro operation. The neural network of the explanatory text creation unit 7d is configured separately from the neural network of the candidate / relationship score detection unit 7a (distinguished from each other).

For example, to determine the region B _B region B _A and the object of the object the object determination unit 7b is on the image by the NMS ( "person") ( "Umbrella") (see FIG. 10), the relationship determining unit 7c are two When the relationship between the candidates B _a and B _b corresponding to the objects B _A and B _B (for example, "having") is determined (see FIG. 13), the explanation creating unit 7d has an explanation saying "a person has an umbrella". (S5; description creation step). The data of the created explanatory text is stored in, for example, the storage unit 3, and is read out and used as needed. As an example, the above explanatory text (data) is displayed on the display unit 5 together with the image, or is transmitted to an external terminal together with the image data via the communication unit 6.

As described above, the description creation unit 7d creates a description of the image based on the areas and relationships of a plurality of objects, thereby realizing an application that presents the description to the user together with the image. It becomes possible. Thereby, the convenience of the object detection device 1 can be improved. In addition, the explanatory text creation unit 7d can easily create an explanatory text by a neuro operation using CNN or the like.

[Programs and recording media]
The object detection device 1 described in the present embodiment can be configured by, for example, a computer (PC) in which a predetermined program (application software) is installed. By reading and executing the above program by a computer (for example, a control unit 2 as a CPU), each unit of the object detection device 1 can be operated to execute each process (each process) described above. Such a program is acquired by downloading from the outside via a network, for example, and is stored in the storage unit 3. Further, the program is recorded on a computer-readable recording medium such as a CD-ROM (Compact Disk-Read Only Memory), and the computer reads the program from the recording medium and stores it in the storage unit 3. There may be.

[Other]
The object detection method and the object detection device of the present embodiment described above may be expressed as follows. In addition, the contents described in the present embodiment also include a program and a recording medium expressed as follows.

1. 1. An object detection method for detecting the relationship between a plurality of objects and the plurality of objects from an image, which is a candidate detection step for detecting a candidate for a region occupied by the plurality of objects on the image, and for each of the detected candidates. , With the candidate as one candidate, with reference to other candidates within a preset range around the one candidate, a relationship representing the likelihood of a predetermined relationship of the one candidate with respect to the other candidate. A relationship score detection step for detecting a score, an object determination step for determining the region of the plurality of objects from the detected candidates based on a predetermined algorithm, and a pair of candidates corresponding to the determined region are detected. A method for detecting an object, which includes a relationship determination step for determining the relationship between the plurality of objects based on the relationship score.

2. The object detection method according to 1, wherein the candidate detection step and the relationship score detection step are executed in parallel by a neuro operation.

3. 3. The object detection method according to 2, wherein the neuro operation is an operation by CNN.

4. In the candidate detection step, the relative position of the candidate with respect to each of the plurality of grids set on the image, the relative size of the candidate with respect to the image, the detection score representing the likelihood of the candidate with respect to the object, and the above. The object detection method according to any one of 1 to 3, wherein at least one parameter of a class score representing the likelihood of the class of the object in the grid is further detected.

5. The object detection method according to 4, wherein the predetermined algorithm in the object determination step is Non-Maximum Suppression.

6. The object detection method according to any one of 1 to 5, wherein in the relationship determination step, when a plurality of the relationship scores are present, the relationship scores below the threshold value are excluded.

7. In the relationship determination step, as the relationship score obtained for the pair consisting of the two candidates, the relationship score obtained from one candidate of the pair to the other candidate and the one from the other candidate of the pair The object detection method according to any one of 1 to 6 above, wherein the smaller relationship score is excluded when there is a relationship score obtained for the candidate.

8. Any of the above 1 to 7, further comprising a descriptive text creation step of creating a descriptive text of the image based on the region of the plurality of objects determined and the relationship of the plurality of objects. The object detection method according to.

9. The object detection method according to 8, wherein the explanatory text is created by a neuro operation in the explanatory text creation step.

10. The relationship between the plurality of objects is any one of the above 1 to 9, characterized in that the relationship or state between the plurality of objects, the action or action of one object on the other object, or at least one of them is included. The described object detection method.

11. An object detection device that detects a plurality of objects and the relationship between the plurality of objects from an image, detects candidates for an area occupied by the plurality of objects on the image, and detects the candidates for each of the detected candidates. With reference to other candidates within a preset range around the one candidate, a relationship score representing the likelihood of a predetermined relationship of the one candidate with respect to the other candidate is detected. A pair of a candidate / relationship score detection unit, an object determination unit that determines the region of the plurality of objects from the detected candidates, and a candidate pair corresponding to the determined region are detected based on a predetermined algorithm. An object detection device including a relationship determination unit that determines the relationship between the plurality of objects based on the relationship score.

12. 11. The object detection device according to 11, wherein the candidate / relationship score detection unit performs detection of the candidate and detection of the relationship score in parallel by a neuro operation.

13. The object detection device according to the above 12, wherein the neuro operation is an operation by CNN.

14. The candidate / relationship score detection unit is a detection representing the relative position of the candidate with respect to each of the plurality of grids set on the image, the relative size of the candidate with respect to the image, and the likelihood of the candidate with respect to the object. The object detection apparatus according to any one of 11 to 13, further detecting at least one parameter of a score, a class score representing the likelihood of a class of the object in the grid.

15. The object detection device according to 14, wherein the predetermined algorithm used by the object determination unit to determine the region is Non-Maximum Suppression.

16. The object detection device according to any one of 11 to 15, wherein the relationship determination unit excludes a relationship score less than a threshold value when a plurality of relationship scores are present.

17. As the relationship score obtained for the pair consisting of the two candidates, the relationship determination unit includes a relationship score obtained from one candidate of the pair to the other candidate and the one from the other candidate of the pair. The object detection device according to any one of 11 to 16, wherein the smaller relationship score is excluded when there is a relationship score obtained for the candidate.

18. One of the above 11 to 17, further comprising a description creating unit that creates a description of the image based on the region of the plurality of objects determined and the relationship of the plurality of objects. The object detection device according to.

19. The object detection device according to 18, wherein the explanatory text creating unit creates the explanatory text by a neuro operation.

20. The relationship between the plurality of objects is one of 11 to 19, wherein the relationship includes at least one of a relationship or a state between the plurality of objects, and an action or action of one object on the other object. The object detection device described.

21. A program that causes a computer to execute the object detection method according to any one of 1 to 10.

22. A computer-readable recording medium on which the program according to 21 is recorded.

Although the embodiments of the present invention have been described above, the scope of the present invention is not limited to this, and can be extended or modified within a range that does not deviate from the gist of the invention.

The present invention can be used in applications such as surveillance camera systems that analyze human behavior from images and applications that estimate relationships existing between a plurality of objects from images at high speed.

1 Object detection device 7a Candidate / relationship score detection unit 7b Object determination unit 7c Relationship determination unit 7d Explanation creation unit

Claims (20)

An object detection method for detecting the relationship between a plurality of objects and the plurality of objects from an image.
A candidate detection step for detecting a candidate for an area occupied by the plurality of objects on the image,
For each of the detected candidates, the candidate is set as one candidate, and the other candidates within the preset range around the one candidate are referred to, and the predetermined relationship of the one candidate with respect to the other candidate. The relationship score detection step that detects the relationship score that represents the likelihood of
An object determination step of determining the region of the plurality of objects from the detected candidates based on a predetermined algorithm.
An object detection method including a relationship determination step of determining the relationship of a plurality of objects based on the relationship score detected for a pair of candidates corresponding to the determined area. The object detection method according to claim 1, wherein the candidate detection step and the relationship score detection step are executed in parallel by a neuro operation. The object detection method according to claim 2, wherein the neuro operation is an operation by CNN. In the candidate detection step, the relative position of the candidate with respect to each of the plurality of grids set on the image, the relative size of the candidate with respect to the image, the detection score representing the likelihood of the candidate with respect to the object, and the above. The object detection method according to any one of claims 1 to 3, further detecting at least one parameter of a class score representing the likelihood of the class of the object in the grid. The object detection method according to claim 4, wherein the predetermined algorithm in the object determination step is Non-Maximum Suppression. The object detection method according to any one of claims 1 to 5, wherein in the relationship determination step, when a plurality of the relationship scores are present, the relationship scores below the threshold value are excluded. In the relationship determination step, as the relationship score obtained for the pair consisting of the two candidates, the relationship score obtained from one candidate of the pair to the other candidate and the one from the other candidate of the pair The object detection method according to any one of claims 1 to 6, wherein the smaller relationship score is excluded when there is a relationship score obtained for the candidate. The description according to any one of claims 1 to 7, further comprising a description creation step of creating a description of the image based on the region of the plurality of objects determined and the relationship of the plurality of objects. Object detection method. The object detection method according to claim 8, wherein in the description creation step, the description is created by a neuro operation. The object according to any one of claims 1 to 9, wherein the relationship between the plurality of objects includes at least one of a relationship or a state between the plurality of objects, and an action or operation of one object on the other object. Detection method. An object detection device that detects the relationship between a plurality of objects and the plurality of objects from an image.
A candidate for an area occupied by the plurality of objects on the image is detected, and for each of the detected candidates, the candidate is set as one candidate, and other candidates within a preset range around the one candidate. With reference to, a candidate / relationship score detection unit that detects a relationship score representing the likelihood of a predetermined relationship of the one candidate with respect to the other candidate.
An object determination unit that determines the region of the plurality of objects from the detected candidates based on a predetermined algorithm.
An object detection device including a relationship determination unit that determines the relationship of a plurality of objects based on the relationship score detected for a pair of candidates corresponding to the determined area. The object detection device according to claim 11, wherein the candidate / relationship score detection unit performs detection of the candidate and detection of the relationship score in parallel by a neuro operation. The object detection device according to claim 12, wherein the neuro operation is an operation by CNN. The candidate / relationship score detection unit is a detection representing the relative position of the candidate with respect to each of the plurality of grids set on the image, the relative size of the candidate with respect to the image, and the likelihood of the candidate with respect to the object. The object detection apparatus according to any one of claims 11 to 13, further detecting at least one parameter of a score, a class score representing the likelihood of the class of the object in the grid. The object detection device according to claim 14, wherein the predetermined algorithm used by the object determination unit to determine the region is Non-Maximum Suppression. The object detection device according to any one of claims 11 to 15, wherein the relationship determination unit excludes relationship scores below a threshold value when a plurality of relationship scores are present. As the relationship score obtained for the pair consisting of the two candidates, the relationship determination unit includes a relationship score obtained from one candidate of the pair to the other candidate and the one from the other candidate of the pair. The object detection device according to any one of claims 11 to 16, which excludes the smaller relationship score when there is a relationship score obtained for the candidate. The invention according to any one of claims 11 to 17, further comprising a description creating unit that creates a description of the image based on the region of the plurality of objects determined and the relationship of the plurality of objects. Object detector. The object detection device according to claim 18, wherein the explanatory text creating unit creates the explanatory text by a neuro operation. The object according to any one of claims 11 to 19, wherein the relationship between the plurality of objects includes at least one of a relationship or a state between the plurality of objects, and an action or operation of one object on the other object. Detection device.

Images