JP2020042608A - Detection apparatus and program - Google Patents

Abstract

To provide a detection apparatus which can detect an object in a high-resolution image at a high speed.SOLUTION: A detection apparatus includes: a first acquisition unit 1 which acquires a first candidate area as an area with motion from a first image; a first detection unit 2 which detects an area and type of an object for a window where the first candidate area has been acquired out of windows defined on the first image, to detect the area and type of the object in the whole of the first image. The first detection unit 2 detects at least an area of a target. The apparatus includes: a second acquisition unit 3 which acquires a second candidate area as an area corresponding to the area detected in the first image from a second image obtained by imaging the same scene as that captured in the first image in a different viewpoint; and a second detection unit 4 which detects the object only for a window where the second candidate area has been acquired out of windows defined on the second image, to detect the object in the whole of the second image.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a detection device and a program suitable for application to, for example, an image in a high-resolution multi-viewpoint video and capable of quickly detecting an object in an image even with a high-resolution image.

  In recent years, an object detection technique has been developed that recognizes a class of an object from an image using deep learning (deep learning) and also estimates a region (bounding box) where the object exists in the image. For example, in Non-Patent Document 1, as a two-stage structure, in addition to a convolutional layer for extracting a feature map and a network for extracting an object candidate region (Region Proposal Network), classification and regression results are obtained. An object detection technology configured by an output network is disclosed. Although the method of Non-Patent Document 1 achieves highly accurate detection, there is a problem that the use of a network for extracting an object candidate region makes the calculation heavy.

  Therefore, in order to speed up the calculation, a method that requires only one network instead of a two-stage structure has been proposed. For example, in YOLOv3 (You Only Look Once version 3) of Non-Patent Document 2, the image is divided into grids, and object detection is performed by one simple network for each area, thereby realizing high speed. For example, in the case of the SSD (Single Shot Detector) of Non-Patent Document 3, object detection is similarly performed using one simple network, but the recognition accuracy of small objects is improved by further introducing multiscale. I let it.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.2015.Faster r-cnn: Towards real-time object detection with region proposal networks.In Advances in neural information processing systems.91-99. Joseph Redmon and Ali Farhadi. 2018.YOLOv3: An Incremental Improvement.arXiv (2018). Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. 2016. SSD: Single shot multibox detector. In European conference on computer vision.Springer, 21-37. C. Stauffer; W. Grimson (August 1999) .Adaptive background mixture models for real-time tracking (PDF). IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2. pp. 246-252.

  However, even with the methods described in Non-Patent Documents 2 and 3 that realize high-speed and high-precision detection as described above, the detection target is small and sparse in a high-resolution image such as a 4K image. When the method is applied to the case where the data exists, there is a problem that the calculation time becomes longer.

  FIG. 1 is a diagram showing an image in a soccer video as a typical example of an image in which the problem occurs, but the same situation may occur in other sports videos and videos of other genres. In FIG. 1, a vast soccer field F is photographed in a bird's-eye view over the entire image P, and as an example of an object to be detected, a certain soccer player PL and a ball B (each having a humanoid shape and (Shown as a dot shape) is very small on the field F in the image P, and is photographed as being sparsely present in the image P. Note that, in a normal soccer video, other objects (whether or not to be detected) such as other players and spectator seats are also captured in the image P, but in FIG. Other objects are omitted.

  In FIG. 1, as a numerical example of the size, the image P is a 4K image and has a size of 4096 pixels in width × 2160 pixels in height, whereas a rectangular area (bounding box) surrounding the player PL is 100 pixels in width. It is about 40 pixels in height, and the rectangular area surrounding the ball B is about 10 pixels in width × 10 pixels in height, which is extremely small with respect to the entire image P and exists sparsely.

A numerical example of the calculation time when the detection methods of Non-Patent Documents 2 and 3 are applied to the image P as shown in FIG. 1 (as confirmed by the present inventors) is as follows. That is, consider a case in which a high-resolution image P is input without being reduced without being adjusted to an input size assumed by YOLOv3 or SSD (for example, in the case of YOLOv3, 416 × 416 size as described below). . When using NVIDIA's GTX 1080Ti as the computer environment, for example, the time required to execute YOLO in one sliding window (Sliding Window), which is 416 pixels wide by 416 pixels long, is about 25 ms (milliseconds). . In this case, the time required to apply the sliding window over the entirety of one 4K image P and execute YOLO is 1278 ms as shown in the following formula, and it is assumed that video is processed in real time. , Calculation time is large.
25ms x (4096 x 2160) / (416 x 416) = 1278ms

  In FIG. 1, a typical example of a sliding window having the size 416 × 416 in the image P is shown as the window W.

  Further, when each image P is an image of a certain viewpoint in a multi-view image, considering that YOLO is executed on the entire multi-view image (at a certain time), the number of times is 1278 ms as described above. The calculation time required for multiplying the number of viewpoints of the viewpoint video is required. For example, in the case of a multi-viewpoint video including four viewpoints, the calculation time of 4112 times per time frame is required, which makes it more difficult to perform real-time processing as a video.

  As described above, in the related art, even when a target to be detected sparsely exists to some extent in a high-resolution image, the detection is performed in proportion to the high-resolution image. There is a problem that the amount of calculation for this increases. Further, in the related art, when the image is a multi-viewpoint image, similarly, there is a problem that the calculation amount for detection increases in proportion to the number of viewpoints.

  SUMMARY OF THE INVENTION In view of the above-described problems of the related art, a first object of the present invention is to provide a detection device and a program capable of detecting an object in an image at a high speed even if the image has a high resolution. A second object of the present invention is to provide a detection device and a program capable of detecting an object in an image at high speed even if the image is each image in a multi-view image.

  In order to achieve the above object, the present invention is a detection device, a first acquisition unit for acquiring a first candidate region as a region having motion from the first image, and each window defined on the first image A first detection that detects a target region and / or type in the entire first image by detecting a target region and / or type only for a window in which the first candidate region is acquired. And a first part. Further, the first detection unit detects at least a target area, and a second image obtained by photographing a scene common to that captured in the first image from another viewpoint is used by the first detection unit. A second acquisition unit that acquires a second candidate region as a region corresponding to the target region detected from the first image, and the second candidate region is acquired from each window defined on the second image. A second detection unit that detects the target area and type for only the window that has been detected, thereby detecting the target area and type in the entire second image. . In addition, the present invention is a detection device, for the first image, the target region has already been detected, by capturing a scene common to that captured in the first image from another viewpoint From the obtained second image, a second obtaining unit that obtains a second candidate region as a region corresponding to the target region detected from the first image, and among the windows defined on the second image, A second detection unit that detects a target region and / or type in the entirety of the second image by detecting a target region and / or type for only the window in which the second candidate region is obtained; The third feature is to provide the following. A fourth feature is that the program is a program that causes a computer to function as the detection device according to the first to third features.

  According to the first feature, a high-resolution image can be detected at high speed by applying a detection process using a window only to a moving area. According to the second or third feature, from the second image obtained by photographing the same scene as that captured in the first image from another viewpoint, the region where the target has already been detected in the first image By applying the detection process using the window only to the area of the second image corresponding to the image, it is possible to perform high-speed detection even for a high-resolution image.

FIG. 4 is a diagram illustrating a schematic example of an image in which a problem occurs. It is a functional block diagram of a detecting device concerning one embodiment. FIG. 9 shows a schematic example of a camera arrangement for capturing a multi-viewpoint image in a case where the number of viewpoints is four. 5 is a flowchart of an operation of the detection device according to one embodiment. It is an example of the result of applying the background subtraction method using the mixture normal distribution model in the first detection unit. FIG. 7 is a diagram illustrating a schematic example of a process performed by a first detection unit. FIG. 7 is a diagram for explaining that a relative position detected in a window is converted into an absolute position in an image by a first detection unit. It is a figure for explaining one embodiment which acquires a 2nd candidate field with a 2nd acquisition part. FIG. 9 is a diagram for explaining an enlargement process by a second detection unit.

  FIG. 2 is a functional block diagram of the detection device according to one embodiment. As shown in FIG. 2, the detection device 10 includes a first acquisition unit 1, a first detection unit 2, a second acquisition unit 3, and a second detection unit 4. The detection device 10 can perform the following first operation and second operation as its overall operation. In the first operation, the first image as an input is received by the first acquisition unit 1 and the first detection unit 2, and the detection result in the first image (the range occupied by the object in the first image) is received from the first detection unit 2. And the detection result of the type of the target). In the second operation, the second image as an input is received by the second acquisition unit 3 and the second detection unit 4, and the detection result in the second image (the range occupied by the object in the second image) is received from the second detection unit 4. And the detection result of the type of the target).

  In one embodiment (hereinafter, referred to as an embodiment EA), the detection device 1 performs the above-described first operation, and then obtains information obtained in the first operation (obtained by the first detection unit 2). The above-mentioned second operation can be further performed by using the information of the first region as the range occupied by the object from the obtained detection result. In another embodiment (hereinafter, referred to as an embodiment EB), the detection device 1 may perform only the above-described first operation, and may not perform the second operation. In this case, the detection device 10 may be configured such that the second acquisition unit 3 and the second detection unit 4 are omitted and only the first acquisition unit 1 and the first detection unit 2 are provided.

  Note that the first image and the second image as inputs to the detection device 10 can be prepared as having the following relationship. That is, as shown in FIG. 2, the image at the first viewpoint in a certain multi-viewpoint image (a plurality of images obtained by photographing a common scene at a common time with cameras having different arrangements (viewpoints)) is the first image. , An image at a second viewpoint different from the first viewpoint is a second image.

  FIG. 3 shows a case where an image such as the image P (image of a video of a soccer match) shown in FIG. 1 as a schematic example described above is taken as an image at each viewpoint of the multi-view image. Example of a camera arrangement for the case where the number of viewpoints is four is shown. Each of the four cameras C10, C20, C30, and C40 captures a field F where a soccer match is being performed as a common scene with different arrangements (viewpoints). FIG. 3 schematically shows the field F and each of the cameras C10, C20, C30, and C40 as viewed from above. The objects such as the player PL and the ball B shown in FIG. Is omitted. Assuming that the images obtained by the cameras C10, C20, C30, and C40 capturing the field F as a common scene at a common time are images P10, P20, P30, and P40, respectively, these image groups (P10, P20, and P30) , P40) constitute a multi-view image as each time frame image in the multi-view video.

  As shown in FIG. 2, the schematic processing contents of each unit 1 to 4 of the detection device 1 are as follows.

  The first obtaining unit 1 obtains, as a first candidate area R1, an area in the input first image P1 that is determined to have a movement as a foreground. The obtained first candidate region R1 is output to the first detection unit 2.

  Here, in the first acquisition unit 1, each time t on the video (t is a time index and t = 1, 2, 3,..., And the current time as the time of interest for the description is t and The same applies to the first image P1 (t) as a frame image of the current time t and the first image P1 (t) of one or more past times tk (k ≧ 1). By referring to the image P1 (tk) and the motion information as the foreground in the first image P1 (t) at the current time t, the first candidate region R1 (t) at the current time t Can be requested.

  Note that the first candidate region R1 (t) can be an arbitrary number of regions (connected regions) in the first image P1 (t). The same applies to a first region in a first image described later with respect to the first detection unit 2 and a second candidate region in a second image described later with respect to the second acquisition unit 3.

  The first detection unit 2 is obtained by the first acquisition unit 1 at each time t for each window W that is defined in advance as moving in the first image P1 (t) for the target detection processing. Only the window W in which the first candidate region R1 (t) is included inside the window W is to be processed, and the detection process of the region and the type of the imaged target is performed, whereby the first image The detection result of the region and the type of the object to be photographed with respect to the entire P1 (t) is obtained.

  As described above, the first detection unit 2 performs the process of detecting the region of the target imaged and determining the type of all of the multiple windows W that are predefined as moving in the first image P1 (t). Rather, by performing the process only on the window W that includes the first candidate region R1 (t), the distribution of the target captured in the first image P1 (t) is sparse. In some cases, the processing can be completed at high speed. This schematic example will be described later with reference to FIG.

  In the embodiment EA (in which both the first operation and the second operation are performed), the detection result obtained by the first detection unit 2 (that is, the detection result of the object captured in the first image P1 (t)) The information of the target area obtained from the (area and type information) is also output to the second acquisition unit 3 as the information of the first area D1 (t).

  The second acquisition unit 3 at each time t, in the input second image P2 (t), the region corresponding to the first region D1 (t) obtained from the first detection unit 2, the second candidate region R2 (t). The obtained second candidate region R2 (t) is output to the second detection unit 4.

  Here, in the second acquisition unit 3, the first camera parameters previously determined for the first camera capturing the first image P1 (t) and the second camera P2 (t) capturing the second image P2 (t) A second camera parameter determined in advance for the camera, determined as a relationship between the pixel position (x1, y1) in the first image P1 (t) from the pixel position in the second image P2 (t) (x2, y2) by applying the conversion H (homography conversion H described later) to the first region D1 (t), the first region D1 (t) in the first image P1 (t) is converted to the second image P2 The second candidate area R2 (t) can be obtained based on the area H (D1 (t)) mapped to the area H (D1 (t)) in (t).

  The second detection unit 4 performs the same processing as that performed by the first detection unit 2 using the first image P1 (t) and the first candidate region R1 (t). By using the candidate region R2 (t), it is possible to obtain the detection result of the region and the type of the object to be photographed with respect to the entire second image P2 (t).

  That is, the second detection unit 4 performs, at each time t, the second acquisition unit 3 for each window W defined in advance as moving in the second image P2 (t) for the target detection processing. By performing processing for detecting the region and type of the target to be photographed, only the window W in which the obtained second candidate region R2 (t) is included inside the window W is processed, It is possible to obtain the detection result of the region and the type of the object to be photographed for the entire two images P2 (t). This schematic example will be described later with reference to FIG.

  Accordingly, for the same reason as described for the first detection unit 2 (only a part of all the windows W is subjected to the detection processing), the second image P2 (t For example, when the distribution of the target imaged in the parentheses is sparse, the processing can be completed at high speed.

  FIG. 4 is a flowchart of the operation of the detection device 10 according to one embodiment (one embodiment corresponding to the above-described embodiment EA in which both the first operation and the second operation are performed). FIG. 4 shows each step in the case where the detection processing is performed on the first image and the second image as the images at each time t of the multi-view video.

  As a prerequisite for the explanation of each step in FIG. 4, a multi-view video to be subjected to detection processing by the detection device 10 is captured by each of cameras C1, C2,..., CN of N (N ≧ 2) viewpoints. It is assumed that multi-view images P1 (t), P2 (t),..., PN (t) at each time t are configured, and among these, P1 (t) is a first image, Each of the remaining N-1 P2 (t),..., PN (t) is used as a second image. In the detection device 10, any one of the multi-viewpoint images P1 (t), P2 (t),..., PN (t) based on such general N (N ≧ 2) viewpoints is used. Can be set in advance so that the image of the viewpoint is used as the first image and the remaining N-1 viewpoint images are used as the second images. Here, it is necessary to assign variable notations (P1 (t), P2 (t),..., PN (t)) for explanation, and thus, it is possible to set arbitrarily without loss of generality. P1 (t) is set as the first image as the images of the viewpoints, and only the images P2 (t),..., PN (t) of the other N-1 viewpoints are set as the second images.

  The assignment of the variable notation is also referred to as the first image being the image P1 (t) by the camera C1 and the second image being the image P2 (t) by the camera C2 in the schematic description of each of the above-described units 1 to 4. Is matched. That is, what is processed as the second image described in the schematic description is not necessarily the image of only one camera C2, and as described below with reference to FIG. 4, the other cameras C3,. The image of the first image (excluding the camera C1) may additionally be present.

  When the flow of FIG. 4 is started, the process first proceeds to step S10. In step S10, the detection device 10 acquires the multi-view images P1 (t), P2 (t),..., PN (t) at the current time t as input data, and then proceeds to step S12. In step S12, for the first image P1 (t) of the multi-view image at the current time t acquired as an input in step S10, the first acquisition unit 1 and the first detection unit 2 have been described in the schematic description. By performing the above processing, the region and the type of the object being photographed in the first image P1 (t) are detected, and then the process proceeds to step S14.

  In step S14, one of the multi-view images at the current time t acquired as an input in step S10, for which the detection processing is not completed in the second image Pn (t) (any of n = 2, 3,..., N) (This is referred to as a second image Pn (t).) The second acquisition unit 3 and the second detection unit 4 perform the processing as described in the schematic description, thereby obtaining the second image Pn (t). After detecting the region and the type of the object photographed in the parentheses, the process proceeds to step S16.

  As described in the schematic description, when performing the detection processing on the second image Pn (t) in step S14, the first area D1 of the detection results obtained by the first detection unit 2 in step S12. After the second acquisition unit 3 refers to the information of (t), the processing by the second acquisition unit 3 and the second detection unit 4 is performed.

  In step S16, with respect to the current time t, the detection processing in step S14 is performed on all of the N-1 second images Pn (t) specified by the index n (n = 2, 3,..., N). It is determined whether or not has been completed. If all of the N-1 items have been completed (that is, if the determination is affirmative), the process proceeds to step S20, and if there are uncompleted ones (ie, The process proceeds to step S18 (if a negative determination is made).

  In step S18, for the current time t, of the N-1 second images Pn (t) specified by the index n (n = 2, 3,..., N), the processing in step S14 has not been performed. After selecting one of the completions, the process returns to step S14. In the returned step S14, the detection processing is performed on the second image Pn (t) selected as the processing not completed in step S18.

  In addition, instead of returning from step S18 to step S14, at the time when the process proceeds from step S12 to step S14, all of the N-1 second images Pn (t) are in a state where the detection processing is not completed. , Any one of them may be a detection target. For each time t, the process of step S14 is executed only N-1 times for each of the N-1 second images Pn (t), and the order of execution (the order of n) is It may be set in advance, and in step S18, setting according to the set order may be performed. For example, the order may be set as n = 2, 3,..., N in ascending order.

  In step S20, the detection result in the first image P1 (t) obtained in step S12 with respect to the current time t, and the second image Pn (t) obtained in step S14 executed N-1 times (n = 2, 3,..., N), and the detection device 10 outputs the detection result as a detection result for the entire input multi-viewpoint image P1 (t), P2 (t),. Then, the process proceeds to step S22. In step S22, the current time t is updated to the next latest time t + 1, and the process returns to step S10.

  As described above, the multi-viewpoint images P1 (t), P2 (t),..., PN (t) as frames of the multi-viewpoint video at each time t = 1, 2, 3,. With respect to, the detection result can be output in step S20.

  Hereinafter, details of the processing contents in each of the units 1 to 4 of the detection device 10 will be described.

<First acquisition unit 1>
The first acquisition unit 1 acquires a foreground area from the first image P1 (t) having a role as a “base viewpoint” in the multi-viewpoint images P1 (t), P2 (t),..., PN (t). I do. For example, in the case of a soccer video as in the schematic example of FIG. 1, areas such as a plurality of players PL and balls B are acquired as foreground areas.

  The base viewpoint is a term related to the following technical significance in the contents already described. That is, in the embodiment EA, the information of the first region D1 (t) in the first image P1 (t), which is the base viewpoint, is referred to by the second acquisition unit 3, whereby the second image Pn other than the base viewpoint is displayed. The first image is a prerequisite for the detection of the second image in that the detection target can be narrowed down as the second candidate region R2 (t) in (t) (n = 2, 3,..., N). Therefore, it is called as the base viewpoint.

  Specifically, the first acquisition unit 1 can acquire the first candidate region R1 (t) as the foreground region by the background subtraction method which is an existing method. For example, a moving target can be separated as a foreground by using a background difference method using a mixed normal distribution disclosed in Non-Patent Document 4 mentioned above. In other words, in the method of Non-Patent Document 4 and the like, for the problem that the background fluctuates due to wind or the like, or the lighting environment changes due to the position of the sun or the movement of clouds, a mixture normal distribution (Mixture of Gaussian Distribution, It is possible to cope with modeling of the used background. Since MoG updates the background model sequentially using newly observed images, it can handle slow changes in the lighting environment, such as changes in the position of the sun.

  FIG. 5 shows an example in which a player and a ball are extracted as a foreground by applying the background difference method by the MoG to the first image as a soccer video shown in the schematic example of FIG. , Background pixels are shown in black.

<First detector 2>
In the first detection unit 2, among the windows W (sliding windows) defined in advance as moving in the first image P1 (t), the first candidate area acquired as the foreground by the first acquisition unit 1 R1 (t) is limited to only the window W that is included in the window W, and a detector such as YOLO disclosed in the above-mentioned Non-Patent Document 2 which is an existing method (the image is captured in the window W) By applying the target region and type detector, a target region and type detection result is obtained as a detection result in the entire first image P1 (t).

  FIG. 6 is a diagram illustrating a schematic example of the processing by the first detection unit 2 separately from [1] to [4]. [1] is an example of a window W defined in an image in advance to apply a detector such as YOLO. Here, as a schematic example, the entire image area is divided into eight horizontally and three vertically. A total of 24 window examples are shown. [2] is an example of the same foreground extraction result as in FIG. 5, but from the viewpoint of ensuring visibility in the description with reference to FIG. Indicated by. [3] is obtained by assigning the predetermined window defined in [1] to the foreground extraction result of [2], and [4] is the foreground extracted among the windows assigned in [3]. The six windows to which YOLO and the like are applied due to the presence of are shown in gray.

  As can be seen in [4] of FIG. 6, the first detection unit 2 does not apply a detector such as YOLO to all of the 24 windows in total, and detects only the six windows where the foreground exists. By applying the sparse object in the image (that is, the remaining 18 windows are excluded from the application), the sparse object in the image can be quickly detected.

  Note that the example of FIG. 6 is an example of a window W defined in advance in which the entire image area is divided vertically and horizontally so that different windows do not overlap with each other. As the windows, windows having overlaps between windows at different positions may be used. For example, the size of the window W is defined as that of eight horizontal divisions and three vertical divisions, which is the same as that of FIG. 6, and the slide width does not use the entire horizontal and vertical window widths as shown in FIG. It is also possible to set it to half the window width both vertically and vertically. In this case, assuming that adjacent windows have the same window W size as in FIG. 6 and have an overlap of half the window width, 8 × 2-1 = 15 steps horizontally and 3 × 2-1 = The window can be slid to a total of 15 × 5 = 75 positions of 5 steps. Alternatively, the definition of the window W according to an existing window sliding method for searching in an image (for example, one used in template matching or the like) may be used. May be allowed to go outside the image.

  The first detection unit 2 can acquire information of a target region and a type in a window W (a window W in which a foreground exists) by applying a detector such as YOLO. (Here, the type information can be acquired as information of a pair of a type and its reliability value, such as “type: person, reliability: 0.8”.) Further, the acquired window W In the first image P1 (t), the information on the position occupied by the window W in the first image P1 (t) is added to the information on the relative position area in the first image P1 (t). It is possible to obtain the information of the target area.

  FIG. 7 shows that, from the region detected by the first detection unit 2 as a relative coordinate value in the window W as described above, the region is converted into a coordinate value in the entire first image P1 (t). It is a figure for explaining.

In FIG. 7, the first detection unit 2 detects the j-th window W (j) in the first image P1 (t), and the i-th target area is defined as a rectangular area Obj-1 (i). It shows that it was detected. In this case, the information of the area of the rectangular area Obj-1 (i) is represented by the upper left vertex (x _r ¹ (i), y _r ¹ (i)) indicated by a black dot (●) in FIG. It can be represented by two coordinate values of the lower vertex (x _r ² (i), y _r ² (i)). These two coordinate values are obtained as relative coordinate values within the window W (j), with the upper left vertex of the window W (j) as the origin (reference position). On the other hand, in the window W (j), the position coordinates of the upper left vertex (the upper left vertex indicated by a black circle (●) in FIG. 7) in the first image P1 (t) are (x _w ¹ (i), y _w ¹ (i)).

Therefore, in the first detection unit 2, the following equations (1A) to (1D), the upper left and right in the window W (j) expressing the i-th rectangular area Obj-1 (i) to be detected. The relative coordinate values (x _r ¹ (i), y _r ¹ (i)) and (x _r ² (i), y _r ² (i)) of the lower two points are calculated in the entire image P1 (t). absolute coordinates of the two points ^{_{(x 1 1 (i),}} y 1 1 (i)) and ^{_{(x 1 2 (i),}} y 1 2 (i)) to convert into an image P1 (t) An area detection result within the whole can be obtained.
x ₁ ¹ (i) = x _r ¹ (i) + x _w (j)… (1A)
y ₁ ¹ (i) = y _r ¹ (i) + y _w (j)… (1B)
x ₁ ² (i) = x _r ² (i) + x _w (j)… (1C)
y ₁ ² (i) = y _r ² (i) + y _w (j)… (1D)

  Here, as shown in FIG. 7, the direction of the coordinate axes in the image (including in the window) is + x (the x coordinate increases in the right direction with respect to the horizontal direction in both the relative coordinates and the absolute coordinates). Direction), and the downward direction is + y (the direction in which the y coordinate increases). This is the same for the other coordinates described below.

Note that the information of the first region D1 (t) in the image P1 (t) obtained by the first detection unit 2 includes all detected target regions Obj-1 (i) (i = 1, 2,...). , M; M is the respect the total number) of the detected region, the absolute coordinate values of the two points ^{_{(x 1 1 (i),}} y 1 1 (i)) and ^{_{(x 1 2 (i),}} y 1 2 ( It can be expressed by giving i)). Note that the information of the first area D1 (t) is obtained by using a predetermined ratio instead of using the target area Obj-1 (i) detected as a rectangle (a rectangle including the detection target therein) as it is. Only an enlarged version may be used. By enlarging, the second candidate area Rn (t) (n = 2, 3,..., N) described below with respect to the second acquisition unit 3 is also acquired as having a margin, and the conversion described later is performed. Even if there is an error in H _{n, 1} , detection in the second image can be made more reliable.

In FIG. 7, relative coordinates ( _xr ¹ (i), _yr ² (i)) of the lower left vertex of the rectangular area Obj-1 (i) in the window W (j) are further defined as white points (点). It is shown. This is shown for reference in relation to FIG. The relative coordinates (x _r ¹ (i), y _r ² (i)) of the lower left vertex are also calculated by the above formulas (1A) and (1D) using the absolute coordinates (x ₁ ¹ (i), y ₁ ) in the image. ² (i)).

<Second acquisition unit 3>
In the second acquisition unit 3, the information of the first region D1 (t) detected by the first detection unit 2 with respect to the first image P1 (t) is calculated based on the pixel coordinates of the first image P1 (t). By applying a predetermined homography transformation H _{n, 1} to the image coordinates of the two images Pn (t) (n = 2, 3,..., N), the second candidate region Rn (t) (n = 2, 3,…, N).

FIG. 8 is a diagram for explaining an embodiment in which the second acquisition unit 3 acquires the second candidate region Rn (t) (n = 2, 3,..., N). As shown in [1], the target rectangular area detected in the first image P1 (t) is defined as an area Obj-1 (i). (That is, the information of the first area D1 (t) is information of the area Obj-1 (i) for all detected targets.) The area Obj-1 (i) of [1] in FIG. 7 is the same as the area Obj-1 (i) shown in FIG. 7, and as the absolute coordinates in the image P1 (t), the horizontal direction below (+ y direction) the rectangular area Obj-1 (i). The lower left vertex (○) and the lower right vertex (●), which are both ends of one side, are coordinates (x ₁ ¹ (i), y ₁ ² (i)) and (x ₁ ² (i), y ₁ ² ( i)).

As schematically shown in [2] of FIG. 8, the second acquisition unit 3 converts the homography conversion H _{n, 1} into a horizontal line segment (the length thereof) below the rectangular area Obj-1 (i) of [1]. Is Len ₁ (i)), (x ₁ ¹ (i), y ₁ ² (i)) and (x ₁ ² (i), y ₁ ² (i)). likewise the [3] white spots (〇) and black points (●) as shown as the converted two points to the second image Pn (t) above ^{_{(x n 1L (i),}} y n 2L (i) ) and (x _n ^2R (i), determine the y _n ^2R (i)).

Note that the conversion is, as is well known as mathematics in the field of computer graphics, etc., as a multiplication of a 3 × 3 matrix H _{n, 1} to a size 3 column vector expressing the coordinates in homogeneous coordinates as follows: The conversion can be performed as in equations (2L) and (2R). The superscript T is transpose, and represents a column vector of size 3 in homogeneous coordinates.
(x _n ^1L (i), y _n ^2L (i), 1) ^T = H _{n, 1} * (x ₁ ¹ (i), y ₁ ² (i), 1) ^T … (2L)
(x _n ^2R (i), y _n ^2R (i), 1) ^T = H _{n, 1} * (x ₁ ² (i), y ₁ ² (i), 1) ^T … (2R)

In the second acquisition unit 3, the horizontal width Len _n (i) = | x _n ^2R (i) is obtained by the width in the horizontal direction (x direction) formed by the converted two points obtained by the above equations (2L) and (2R). -x _n ^1L (i) | is defined, as shown in [3], in the second image Pn (t) corresponding to the rectangular region Obj-1 (i) of the first image P1 (t). The rectangular area Obj-n (i) can be obtained. In this manner, information on the second candidate area Rn (t) (n = 2, 3,..., N) can be obtained as information on the rectangular area Obj-n (i) for all detected targets i.

Along with the horizontal width Len _n (i) = | x _n ^2R (i) -x _n ^1L (i) |, information on a rectangular area Obj-n (i) having sides parallel to the coordinate axes x and y of the image Among them, the right end side and the left end side as the x-axis direction range are determined as positions x _n ^2R (i) and x _n ^1L (i) (or vice versa). Here, in order to determine the information of the rectangular area Obj-n (i), it is necessary to further determine the lower end position and the upper end position as the y-axis direction range, as follows. Can be determined.

<Lower position>
First, with respect to the rectangular area Obj-n (i) in the second image Pn (t), the lower end position is the converted two points (x _n ^1L (i), y _n ^2L (i)) and (x _n ^2R ( i), y _n ^2R (i)) may be defined as max (y _n ^2L (i), y _n ^2R (i)) having the larger y coordinate among the y coordinates. In [3] in FIG. 8, max may become ^{_{(y n 2L (i),}} y n 2R (i)) = y n 2R (i) is shown as an example as a result of converted.

<Top position>
Further, regarding the rectangular area Obj-n (i) in the second image Pn (t), the position on the upper end side may be determined as follows. That is, the aspect ratio of the rectangular region Obj-n (i) in the second image Pn (t) is the same as the corresponding aspect ratio of the rectangular region Obj-1 (i) in the first image P1 (t). For example, by determining the vertical width of the rectangular area Obj-n (i), the position on the upper end side of the rectangular area Obj-n (i) can also be determined.

That is, if the vertical width of the rectangular area Obj-1 (i) is Height ₁ (i) and the vertical width of the rectangular area Obj-n (i) is Height _n (i), the aspect ratio (vertical to horizontal length ratio) is The vertical width Height _n (i) of the rectangular area Obj-n (i) can be determined by the following relational expression (3) that is equal.
Len ₁ (i) / Height ₁ (i) = Len _n (i) / Height _n (i)… (3)

Note that, as described above, one embodiment of the determination of the rectangular area Obj-n (i) in the second image Pn (t) by the method of FIG. the homography transformation H _{n, 1} is is obtained by the applicable solid region.

The first premise is that all objects (for example, soccer players) to be detected in the first image P1 (t) and the second image Pn (t) are in a space as a plane to which the homography matrix H _{n, 1} is applied. (For example, the plane of the soccer field F in FIG. 1). In other words, it does not largely deviate from the common plane in the height direction, and exists almost as standing on the plane. The conversion of the two points at the lower end of the rectangle described in FIG. 8 is based on the first assumption. (In the case of a soccer player, the position of the head away from the ground is transformed into a distorted position by the homography conversion, but the position of the foot in contact with the ground is almost in contact with the ground by the homography conversion. The first premise is that the image is converted into a position.) In the first premise, the lower side (+ y direction) of the first image P1 (t) and the second image Pn (t) is in the space. It is assumed that images are taken by a camera arrangement that is on the side closer to a common plane (such as the ground).

  The second premise is that rectangles surrounding the same object when viewed in the first image P1 (t) and the second image Pn (t) have substantially the same aspect ratio. The determination of the upper end side position based on this second premise.

Each homography matrix H _{n, 1} should be calculated in advance by an arbitrary existing method by using a camera calibration marker (for example, a square marker) arranged on a common plane such as the ground. Can be done. For a fixed camera, a matrix H _{n, 1} may be prepared as a fixed parameter. In the case of a moving camera, camera calibration using the marker may be performed at each time.

<Second detector 4>
As described above, the processing of the second detection unit 4 is the same as the processing content of the first detection unit 2. That is, while the data to be processed was the first image P1 (t) and the first candidate region R1 (t) referred to for window limitation in the first detection unit 2, the second detection unit 4 Is different only in that it is a second image Pn (t) and a second candidate region Rn (t) to be referred to for window limitation, and regarding the processing contents, the first detection unit 2 and the second detection unit 4 Are common.

  However, in the second detection unit 4, the following additional embodiment can be performed.

  First, the significance of the additional embodiment will be described. As described above, the second detection unit 4 (and the first detection unit 2) uses a detector constructed by learning in advance by deep learning such as YOLO. In the pre-learning, a large number of pieces of learning data that are labeled with respect to an object photographed at a standard size in an image of a standard size are used. Therefore, the constructed detector has a preferable standard size with respect to the size of the object in the detected window from the viewpoint of securing the detection reliability. That is, if the size is too small, there is little information in the first place, and the detection reliability is lowered.

  However, in the shooting situation described with reference to FIG. 1, the above-mentioned certain size cannot be secured, and it is too small to secure a predetermined detection reliability with a learned detector. is there. Therefore, in the additional embodiment, when it is determined that the size is too small, the processing of the detector in the window is performed after being enlarged in advance.

FIG. 9 is a diagram for explaining the enlargement processing by the second detection unit 4. [1] of FIG. 9 shows the rectangular area Obj-n (i) in the second image Pn (t) shown in [3] of FIG. If it is determined that the rectangular area Obj-n (i) is small, the upper left vertex (x _n ^1L (i), y _n ^1L (i)) indicated by a black point (●) is set to a reference position (fixed position) for enlargement. ), It is possible to obtain an enlarged rectangular area Obj-n ′ (i) by enlarging (similarly enlarging) by a scale factor as shown in [2]. Thus, the window W including the rectangular region Obj-n '(i) as the enlarged target as shown in [2] may be set as the detection target by the detector.

  Note that a predetermined point in the rectangular area Obj-n (i) other than the upper left vertex may be used as a reference position for enlargement. Note that the background BGW other than the rectangular area Obj-n (i) in the window W shown in [1] when the image is not enlarged as shown in FIG. 9 and the Obj-n 'in the window W shown in [2] when the image is enlarged. The background BGW 'other than (i) may be treated as if the texture captured in the corresponding second image Pn (t) does not exist, and the processing of the detector may be performed.

  The determination as to whether or not to enlarge the image and the calculation of the enlargement factor scale in the case of enlargement in the enlargement process as an additional embodiment by the second detection unit 4 may be performed as follows.

First, based on the number of pixels of the rectangular region Obj-1 (i) in the first image P1 (t) shown in [1] of FIG. 8, that is, the area S ₁ (i), [3] in FIG. The area S _n (i) of the rectangular area Obj-n (i) in [1] of FIG. 9 is obtained as follows. Here, since the aspect ratios are equal as described in the second premise, the area is determined by the following equation (4) by multiplying the square of the width ratio Len _n (i) / Len ₁ (i). Can be.
S _n (i) = S ₁ (i) * (Len _n (i) / Len ₁ (i)) ² … (4)

When the obtained area S _n (i) is _equal to or smaller than a predetermined threshold S _TH (a predetermined threshold that can be determined according to a standard size detected by a detector), enlargement processing is performed. The enlargement ratio scale may be calculated by calculating the intermediate value “s _tmp ” of the following expression (5A) and then dividing by the following expressions (5B) and (5C). That is, when the intermediate value s _tmp is less than the threshold value TH, Expression (5B) is adopted, and when the intermediate value s _tmp is equal to or more than the threshold value TH, Expression (5C) is adopted, and the enlargement ratio scale may be obtained. sqrt () is a square root operation, and is an operation for converting a value determined based on an area into an enlargement ratio scale as a length.
s _tmp = S _peak (c) / S _n (i)… (5A)
scale = sqrt (s _tmp ) (if s _tmp <TH)… (5B)
scale = sqrt (TH) (if s _tmp ≧ TH)… (5C)

In the above (5B) and (5C), TH is a threshold value that is set in advance so as not to make the enlargement ratio scale too large. (Note that the threshold TH is different from the threshold S _TH for the area S _n (i).) Excessive enlargement tends to lower the reliability of detection due to deterioration of image quality. Set it. In the above (5A), S _peak (c) corresponds to the type c of the rectangular area Obj-1 (i) detected in the first image P1 (t) (for example, c = soccer player, ball,...) Although it is a predetermined area where the detection reliability is ensured, a fixed value independent of the type c may be used. Information of the type c may be obtained by referring to the detection result of the first detection unit 2.

  As described above, according to the present invention, in a 4K video or the like, a player or the like to be detected in a soccer video as shown in FIG. It is possible. Hereinafter, a supplementary description of additional embodiments and the like in the present invention will be provided.

(1) As described in the above additional embodiment, when it is determined that the area S _n (i) of the region Obj- _n (i) is small, the second detection unit 4 can perform the scale-up process by scale times. . As an alternative and / or additional embodiment to this, the first of the already learned detectors is used so that even if it is determined to be small, it can be appropriately detected by the detector without performing the enlargement processing. Second learning data obtained by reducing the image of the learning data may be newly prepared, the detector may be re-learned by deep learning or the like, and the re-learned detector may be used by the second detection unit 4.

  Specifically, the size of the target is calculated from the target region (bounding box) labeled in the original learning data (first learning data), and compared with the minimum size of the target obtained in advance by statistical processing or the like. Then, re-learning may be performed using the second learning data to which the reduced size is added to the target size of a plurality of levels. For example, when the minimum size of the ball is 100 in the statistical processing, it is assumed that the area (bounding box) of the ball labeled with a certain learning image in the first learning data is 400. This one training image is reduced to 1/4, 1/3, 1/2, and 1 times (the original size is kept in case of 1 times reduction), and 4 images (of which 3 images are A new learning image reduced to a new one. By applying the same processing to other learning images in the first learning data, second learning data can be obtained.

(2) The first detection unit 2 and the second detection unit 4 detect the target region and type in the image using a detector such as YOLO, but output only one of the region and type. You may. At this time, the area and the type may be detected as the internal processing, and only one of the area and the type may be used as the output data to be actually used. However, in order for the second acquisition unit 3 and the second detection unit 4 to operate, even if the second detection unit 2 and the second detection unit 4 do not output to the outside (as data used by a user or the like), the first detection unit 2 outputs It is necessary to output (information of the first area D1 (t)) to the second acquisition unit 3. In addition, in the first detection unit 2 and the second detection unit 4, a SSD or the like disclosed in the above-mentioned Non-Patent Document 3 or the like may be used in addition to YOLO as a detector capable of detecting at high speed on a window basis. .

(3) As described above, in the present invention, the embodiment EA (where both the first operation and the second operation are performed) and the embodiment EB (where only the first operation is performed) are possible. Further, an embodiment EC in which only the second operation in the embodiment EA is extracted is also possible. In the embodiment EC, by completing the first operation in advance, the first region D1 (t) may be used as an input. Further, the embodiment EC can also be as follows. That is, when a second area Dn (t) as a detection result has already been obtained for the second image Pn (t) of a certain camera viewpoint n, another camera viewpoint m (m ≠ n, m ≧ 2) As an input for obtaining the second area Dm (t) as a detection result from the second image Pm (t), the detected second area Dn (t) is used instead of the detected first area D1 (t). It is also possible.

(4) The present invention can also be provided as a program that causes a computer to function as the detection device 10. The computer may have a well-known hardware configuration such as a CPU (Central Processing Unit), a memory, and various I / Fs, and the CPU may execute an instruction corresponding to a function of each unit of the detection device 10. Becomes In addition, the computer further includes a GPU (graphics processing device) capable of performing parallel processing at a higher speed than the CPU, and instead of the CPU, executes the functions of all or any part of the detection device 10 by reading the program on the GPU. You may make it.

  10 detection device, 1 first acquisition unit, 2 first detection unit, 3 second acquisition unit, 4 second detection unit

Claims (8)

A first acquisition unit that acquires a first candidate region as a region with motion from the first image,
Of the windows defined on the first image, for only the window from which the first candidate region is obtained, by detecting the region and / or type of the target, the target of the entirety of the first image is detected. A detection device comprising: a first detection unit that detects an area and / or a type.   The detection device according to claim 1, wherein the first acquisition unit acquires the first candidate region by applying a background subtraction method. The first detection unit detects at least a target area,
From the second image obtained by shooting the same scene as that captured in the first image from another viewpoint, as an area corresponding to the target area detected from the first image by the first detection unit A second acquisition unit for acquiring a second candidate area,
Of the windows defined on the second image, only the window from which the second candidate region is obtained, by detecting the target region and the type, the target region and the target region in the entire second image The detection device according to claim 1, further comprising: a second detection unit configured to detect a type. For the first image, the target area has already been detected,
From a second image obtained by shooting a scene common to the first image from a different viewpoint, a second candidate area is obtained as an area corresponding to a target area detected from the first image A second acquisition unit to
By detecting the target area and / or type only for the window from which the second candidate area is obtained among the windows defined on the second image, the target in the entire second image is detected. A second detection unit that detects an area and / or a type.   The said 2nd acquisition part acquires the said 2nd candidate area | region by applying a homography conversion with respect to the area | region of the target detected from the said 1st image, The Claims 3 or 4 characterized by the above-mentioned. Detection device. The first image and the second image are photographed as an object can move on a predetermined plane,
In the second acquisition unit, of the rectangle surrounding the target area detected from the first image, on the second image obtained by applying homography conversion to the side near the predetermined plane The detection device according to claim 5, wherein the second candidate area is acquired as a rectangle surrounding a side.   The detection of the area and / or type of the target in the second detection unit is performed by applying a detector pre-constructed by deep learning using learning data, and the detection is performed in comparison with the standard size of the target in the detector. When it is determined that the size of the second candidate area is small, the second detection unit expands the second candidate area in advance before detecting the target area and / or type, and then expands the second candidate area. The detection device according to claim 3, wherein a target region and / or a type is detected for the region.   A program for causing a computer to function as the detection device according to claim 1.