JP7195892B2 - Coordinate transformation matrix estimation method and computer program - Google Patents

Description

The present invention relates to a coordinate transformation matrix estimation method and a computer program.

Conventionally, shooting sports such as soccer and rugby and analyzing the shot images to analyze the tactics of the team and the performance of each player leads to the planning of tactics and the recruitment of highly promising players.
On the other hand, the scale of potential sports videos that can be analyzed is extremely large, and it takes a huge amount of time and money to manually analyze all of them. If the analysis of sports videos can be automatically performed by a machine, it will be possible to collect useful knowledge hidden in large-scale data without omission. Therefore, the industrial value of technology that contributes to the realization of automatic analysis of sports videos is extremely high.

Usually, sports videos are often captured by cameras arranged on the side of the sports field while following the athletes. Typical statistical values that can be analyzed from sports videos include the amount of activity of a player (for example, the running distance of a player during a game) and the trajectory of a player's movement. However, simply detecting and tracking players in video is not sufficient to obtain the above statistics. To obtain the above statistics, it must be known how the fields appear in the multiple video frames that make up the video. More specifically, it is necessary to know the coordinate transformation matrix that associates the physically identical points of the imaged field with the image frame at time t, t+ _t0 in the image. It is known that when the field is a two-dimensional plane, the above coordinate transformation matrix is expressed as a matrix of 3 rows and 3 columns. Registration is a technique for estimating a coordinate transformation matrix from information of an input video frame when the parameters of the coordinate transformation matrix are unknown.

The simplest method of performing registration based on video frames is as follows. First, between video frames at times t and t+ _t0 , arbitrary four points capturing the same position in the field are manually associated. Next, the parameters of the coordinate transformation matrix are estimated by DLT (Direct Linear Transform) from the corresponding relationship. With this method, when the camera moves according to the game situation, it is necessary to manually assign corresponding points to all video frames at consecutive times, which is extremely costly and is not suitable for real-time analysis. There is

As a method of estimating the parameters of a coordinate transformation matrix without manually giving corresponding points, techniques such as Non-Patent Documents 1 and 2 have been proposed.
In Non-Patent Document 1, a characteristic area (for example, a line or a circle) in a field captured in each video frame is detected in advance, and the characteristic area is associated with a plurality of video frames to obtain coordinates. You are estimating the parameters of the transformation matrix.
In addition, in Non-Patent Document 2, a neural network that assigns each pixel in an image to a class such as "center circle", "sideline", "grass" is trained in advance, and video frames are generated based on the output. The parameters of the coordinate transformation matrix are estimated in association with each other.

Ankur Gupta, James J. Little, Robert J. Woodham, “Using Line and Ellipse Features for Rectification of Broadcast Hockey Video”, Computer and Robot Vision (CRV), 2011 Canadian Conference on Namdar Homayounfar, Sanja Fidler, Raquel Urtasun, “Sports Field Localization via Deep Structured Models”, In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, p.5212-5220

When a characteristic area is imaged between a plurality of video frames, the parameters of the coordinate transformation matrix can be estimated by the methods of Non-Patent Documents 1 and 2, respectively. However, when the target is a video frame in which characteristic areas in the field are hardly captured, the methods of Non-Patent Documents 1 and 2 degrade the accuracy of estimating the parameters of the coordinate transformation matrix. As a means to solve such problems, there is a method of using time-series information more explicitly.

Specifically, under the assumption that the situation changes smoothly between a plurality of video frames shot at close times, a known optical flow estimation method is used to estimate corresponding points between the video frames, and the coordinates There are methods such as estimating the transformation matrix. However, in this method, when the above assumption is deviated from, for example, when the area captured by the camera changes rapidly between video frames captured continuously, the optical flow estimation accuracy decreases. put away. As a result, it becomes impossible to accurately estimate the parameters of the coordinate transformation matrix. It is very common for video for viewing purposes that the area captured by the camera changes abruptly between successively captured video frames. Therefore, the inability to deal with such a situation may lead to a significant restriction on the types of images that can be obtained as analysis targets.

As described above, in the conventional method, the accuracy of estimating the coordinate transformation matrix decreases when a characteristic area in the imaging target area is not captured or when the area captured by the camera changes rapidly. There was a problem of hoarding.
In view of the above circumstances, an object of the present invention is to provide a technique capable of improving the estimation accuracy of a coordinate transformation matrix.

One aspect of the present invention is a detection step of detecting an object from each of a plurality of images, and association of images captured at predetermined time intervals from the plurality of images or captured at the same time by different imaging devices. an extraction step of extracting a plurality of target images; an object matching step of matching results of object detection between the plurality of images extracted as matching targets; and a matching result in the object matching step. and a transformation matrix estimation step of estimating a coordinate transformation matrix between the images based on the detection result of the object.

One aspect of the present invention is the coordinate transformation matrix estimation method described above, wherein, in the detection step, after detecting the object, the object is classified by determining the classification of the detected object, and the image When the detection result of the object between the images includes information indicating the classification result of the object, in the object matching step, the detection result of the object including the information indicating the same classification result between the images is classified as the same object. Correlate as a detection result.

One aspect of the present invention is the coordinate transformation matrix estimation method described above, wherein the object matching step comprises a provisional matching step of provisionally matching the detection result of the object between the images; a graph construction step of constructing a graph in which each set of object detection results between the temporarily associated images is a node; A correspondence that is assumed to be a correct correspondence by judging whether or not the condition for addition to the cluster containing the nodes of the correspondence that is assumed to be a correct correspondence is satisfied, in order from the node with the highest priority and a selection step of selecting a node of interest.

One aspect of the present invention is the coordinate transformation matrix estimation method described above, wherein, in the detection step, after detecting the object, the object is classified by determining the classification of the detected object, and the image When the detection result of the object between the images includes information indicating the classification result of the object, in the provisional association step, the detection result of the object including the information indicating the same classification result between the images is classified as the same object. Temporarily associated as a detection result.

One aspect of the present invention is the coordinate transformation matrix estimation method described above, wherein in the graph construction step, the presence or absence of geometric consistency of node pairs included in the constructed graph is evaluated, and if the geometric consistency is The node pairs evaluated to be correct are connected by edges, and in the selection step, nodes with a large number of nodes connected by the edges are given high priority, and the nodes of the correspondence assumed to be the correct correspondence are selected. select.

One aspect of the present invention is the coordinate transformation matrix estimation method described above, wherein, in the selection step, the feature points of the nodes that have already been added to the cluster are selected in order from the node with the highest priority. is not used as a node, it is determined that the condition for addition to the cluster is satisfied, and the node satisfying the condition for addition to the cluster is added to the cluster.

One aspect of the present invention is the coordinate transformation matrix estimation method described above, wherein a detection result of an imaged object is input to each of a plurality of images obtained by imaging an imaging target region in opposite directions by a plurality of imaging devices. If so, in the object matching step, the matching is performed after reversing the detection result of one of the objects.

One aspect of the present invention is a computer program for executing the coordinate transformation matrix estimation method described above.

According to the present invention, it is possible to improve the estimation accuracy of the coordinate transformation matrix.

1 is a diagram showing a functional configuration of an image processing system 100 according to a first embodiment; FIG. 4 is a flow chart showing the flow of coordinate transformation matrix estimation processing performed by the image processing apparatus 10 according to the first embodiment. 4A and 4B are diagrams illustrating an example of person detection by the object detection unit 102 in the first embodiment; FIG. FIG. 4 is a schematic block diagram showing the functional configuration of an image processing apparatus 10a according to a second embodiment; FIG. 11 is a schematic block diagram showing the functional configuration of an object associating unit 103a in the second embodiment; FIG. 10 is a flow chart showing the flow of coordinate transformation matrix estimation processing performed by the image processing apparatus 10a according to the second embodiment. FIG. 10 is a diagram showing an example of person detection by an object detection unit 102a in the second embodiment; FIG. 10 is a diagram showing a result of provisional association by a provisional association unit 1031 in the second embodiment; FIG. FIG. 10 is a diagram showing a graph construction result by a graph construction unit 1032 in the second embodiment; FIG. 13 is a diagram showing processing in a graph constructed by a graph construction unit 1032 according to the second embodiment; FIG. 9 is a flowchart showing the flow of cluster extraction processing performed by the image processing apparatus 10a according to the second embodiment; FIG. 10 is a diagram showing cluster extraction results in the second embodiment; FIG. 11 is a schematic block diagram showing the functional configuration of an image processing device 10b according to a third embodiment; FIG. 13 is a diagram showing the functional configuration of an image processing system 100c according to the fourth embodiment; FIG.

An embodiment of the present invention will be described below with reference to the drawings.
(First embodiment)
FIG. 1 is a diagram showing the functional configuration of an image processing system 100 according to the first embodiment. The image processing system 100 includes an imaging device 1 and an image processing device 10 .
The image capturing device 1 captures an image of an image capturing target area 2 . A shooting target area 2 is, for example, a stadium where sports such as soccer and rugby are played. The imaging device 1 is, for example, a camera. The imaging device 1 generates a plurality of video frames in which part or all of the imaging target region 2 is captured by capturing a moving image of the imaging target region 2 , and transmits the generated video frames to the image processing device 10 . Output. A video frame is an example of a still image captured by an imaging device.

The image processing apparatus 10 estimates parameters of a coordinate transformation matrix between video frames based on a plurality of input video frames, and performs image transformation of the video frames using the estimated parameters of the coordinate transformation matrix. For example, the image processing apparatus 10 estimates a coordinate transformation matrix between video frames based on two video frames captured at different times. A plurality of video frames input to the image processing apparatus 10 are images captured at different times t and t+ _t0 in the video. There is a predetermined time interval difference between different times t, t+ _t0 in the image, such as 1 frame, several frames, 1 second, 5 seconds, 10 seconds, hours, years, and so on. The interval between times t and t+ _t0 is preferably as close to 0 as possible. In the following description, the image captured at time ^t is described as video frame It, and the image captured at ^t+t0 _is described as video frame It+t0.

Next, a specific configuration of the image processing apparatus 10 will be described.
The image processing apparatus 10 includes a CPU (Central Processing Unit), a memory, an auxiliary storage device, etc., which are connected via a bus, and executes an image processing program. By executing the image processing program, the image processing device 10 functions as a device including an image acquisition unit 101 , an object detection unit 102 , an object matching unit 103 , a transformation matrix estimation unit 104 and an image processing unit 105 . All or part of each function of the image processing apparatus 10 is implemented by hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field Programmable Gate Array), or a GPU (Graphics Processing Unit). may be implemented using Also, the image processing program may be recorded on a computer-readable recording medium. Computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs and CD-ROMs, and storage devices such as hard disks incorporated in computer systems. Also, the image processing program may be transmitted and received via an electric communication line.

The image acquisition unit 101 acquires video frames output from the imaging device 1 . For example, the image acquisition unit 101 acquires multiple video frames.
The object detection unit 102 detects objects captured in the video frames from each of the plurality of video frames acquired by the image acquisition unit 101 . Here, the object is a moving object such as a person, an animal, a robot, a vehicle, or the like, which can move autonomously or according to an operation. In the following description, an example in which the object is a person will be described.

For human detection, for example, the method disclosed in Reference 1 below may be used. Note that the person detection method is not limited to the method disclosed in Reference 1, and a well-known technique can be used.
(Reference 1: Joseph Redmon, Ali Farhadi, “YOLOv3: An Incremental Improvement”, In arXiv, 2018)

The object detection unit 102 detects the coordinate values of the four corners of a rectangle surrounding each person captured in the video frame, that is, the segmented area of the person, as the person detection result. As for the granularity of human detection, a detector may be used in which a specific person imaged in the video frame is labeled and output along with the coordinate values of the four corners of the rectangle. You may use the detector which detects.

In the following description, a detector that outputs the label of a specific person imaged in the video frame along with the coordinate values of the four corners of the rectangle is referred to as a specific object detector, and detects any person imaged in the video frame. We describe the detector as a general object detector. A specific object detector and a general object detector can be implemented in the same way. However, in general, it is necessary to collect learning data regarding individual target persons in order to detect specific persons. Therefore, a specific object detector tends to be more expensive to construct than a general object detector, which can be constructed by preparing learning data about an arbitrary person. The object detection unit 102 in the first embodiment uses a specific object detector.

The object association unit 103 associates the person detection results between the video frames based on the person detection results of each video frame detected by the object detection unit 102 . Specifically, first, the object association unit 103 extracts a plurality of image frames to be associated, which are imaged at a predetermined time interval difference from the plurality of frames. Next, the object association unit 103 detects an object between the plurality of video frames extracted as the association target based on the detection result of the object detected in each of the plurality of image frames extracted as the association target. Match the results. For example, the object associating unit 103 combines the detection result of the object detected in the video frame It captured at time ^t with the video frame It + _t0 captured at time t ^+t0 , which is a predetermined time away from time t. Based on the detection result of the object detected in , the object detection result between the image frame It and the image frame It ^{+t0 is} associated with each other. The person detection result includes the label of the specific person and the coordinate values of the four corners of the rectangle.
The transformation matrix estimating unit 104 calculates the coordinate transformation matrices Ht ^{, t + of the video frames It and It+t0} based on the person detection result by the object detection unit 102 and the matching result by the object matching unit 103. Estimate ^t0 .

Of the coordinate transformation matrices, those with 8 degrees of freedom are called projective transformation matrices, those with 6 degrees of freedom are called affine transformation matrices, and those with 4 degrees of freedom are called similarity transformation matrices. A transformation matrix may be used. The parameters of the assumed transformation matrix can be estimated by using a known parameter estimation method after calculating the coordinates associated between the video frames from the detection results of the persons forming each correspondence indicated by the correspondence results. can. As a method of calculating corresponding coordinates between video frames from the results of human detection, there are methods of calculating corresponding points between the coordinates of the center of gravity of the rectangle, and focusing on the fact that the lower part of the person is in contact with the ground, the lower left corner of the rectangle. , and ((x ₁ +x ₂ )/2, y ₁ ), where the lower right coordinates are (x ₁ , y ₁ ) and (x ₂ , y ₁ ), respectively. As a known parameter estimation method, DLT or RANSAC disclosed in Reference 2 below may be used depending on the number of obtained corresponding points.
(Reference 2: Martin A. Fischler, Robert C. Bolles, “Random Sample Consensus: A paradigm for model fitting with applications to image analysis and automated cartography”, In Communications of the ACM, 1981)

The image processing unit 105 performs image processing on the input video frame using the coordinate transformation matrix ^Ht,t+t0 estimated by the transformation matrix estimation unit 104 . Specifically, the image processing unit 105 converts the video frame into an image viewed from a specific direction. For example, the image processing unit 105 ^converts the video frame It ^+t0 into an image viewed from the direction in which the video frame It was shot.

Image conversion by the image processing unit 105 makes it possible to visualize information such as the formation of players on the field, the position of each player at a certain time, and the movements of the players.

FIG. 2 is a flow chart showing the flow of coordinate transformation matrix estimation processing performed by the image processing apparatus 10 according to the first embodiment.
The image acquisition unit 101 acquires a plurality of video frames (step S101). The image acquisition unit 101 outputs the acquired plurality of video frames to the object detection unit 102 . The object detection unit 102 detects a person captured in each of the plurality of video frames (step S102). The object detection unit 102 in the first embodiment is a specific object detector as described above, and the object association unit 103 uses the label of the specific person imaged in the video frame as the person detection result together with the coordinate values of the four corners of the rectangle. and output to transformation matrix estimation section 104 .

FIG. 3 is a diagram showing an example of person detection by the object detection unit 102 in the first embodiment. As shown in FIGS. 3A and 3B, FIG. 3A is a diagram showing a video frame 20 captured at time t, and FIG. 3B is a diagram showing a video frame 30 captured at time t+ _t0 .
In the video frame 20, four persons 21-1 to 21-4 are detected. The detected persons 21-1 to 21-4 are given labels for identifying each person by the specific object detector. In FIG. 3A, the object detection unit 102 classifies the person 21-1 as “person 1” and assigns the label “person 1”, and the object detection unit 102 classifies the person 21-2 as “person 2”. Person 21-3 is classified and labeled as “Person 3” by the object detection unit 102, and person 21-4 is classified as “Person 3” by the object detection unit 102. is classified as "Person 4" and labeled as "Person 4".
Object detection unit 102 generates, as results of person detection in image frame 20, labels indicating classification results of persons 21-1 to 21-4 and regions 22-1 to 22- surrounding persons 21-1 to 21-4. Get the coordinate values of the four corners of the rectangle of 4.

In the video frame 30, three persons 31-1 to 31-3 are detected. The detected persons 31-1 to 31-3 are given labels for identifying each person by the specific object detector. In FIG. 3B, person 31-1 is classified as “person 3” by object detection unit 102 and labeled as “person 3”, and person 31-2 is classified as “person 4” by object detection unit 102. It is shown that the person 31-3 has been classified and labeled as "person 4", and that the person 31-3 has been classified as "person 2" by the object detection unit 102 and labeled as "person 2".
Object detection unit 102, as a person detection result of video frame 30, labels indicating the results of classification of persons 31-1 to 31-3 and areas 32-1 to 32- Get the coordinate values of the four corners of the rectangle of 3.

Returning to FIG. 2, the description continues.
The object association unit 103 extracts a plurality of (for example, two) image frames to be associated, which are imaged at a predetermined time interval difference from the plurality of image frames (step S104). Note that the extraction start timing and the predetermined time interval of the video frames extracted by the object association unit 103 are set in advance. The object association unit 103 associates the person detection result in the image frame 20 with the person detection result in the image frame 30 based on the person detection result output from the object detection unit 102 in the extracted image frame (step S104). Specifically, the object associating unit 103 uses the label obtained as the person detection result in each of the plurality of image frames extracted as the object of association, and uses the label obtained as the person detection result in the image frame 20 and the person detection result in the image frame 30. It is associated with the person detection result. For example, the object associating unit 103 associates person detection results to which labels indicating the same information are given as detection results of the same person. Object association section 103 outputs the association result to transformation matrix estimation section 104 .

In the example shown in FIG. 3, the person 21-2 in the video frame 20 and the person 31-3 in the video frame 30 are labeled "person 2". Therefore, the object associating unit 103 associates the person 21-2 in the video frame 20 with the person 31-3 in the video frame 30 as the detection result of the same person.
Also, in the example shown in FIG. 3, the person 21-3 in the video frame 20 and the person 31-1 in the video frame 30 are labeled "person 1". Therefore, the object association unit 103 associates the person 21-3 in the video frame 20 with the person 31-1 in the video frame 30 as the same person detection result.

Also, in the example shown in FIG. 3, the person 21-4 in the video frame 20 and the person 31-2 in the video frame 30 are labeled "person 4". Therefore, the object association unit 103 associates the person 21-4 in the video frame 20 with the person 31-2 in the video frame 30 as the same person detection result.

Note that the object association unit 103 does not have a person detection result with a label indicating the same information between the person detection result in the video frame 20 and the person detection result in the video frame 30. Do not match. In this case, the object association unit 103 does not output the association result for the person detection result for which there is no person detection result with a label indicating the same information, or converts null (no detection) as the association result. Output to matrix estimation section 104 .

The transformation matrix estimating unit 104 calculates coordinate transformation matrices H ^{t and t+} t0 of the video frames It and I ^{t +t0} based on the person detection result by the object detection unit 102 and the matching result by the object matching unit 103 . is estimated (step S105). The transformation matrix estimation unit 104 outputs the estimated coordinate transformation matrix ^Ht,t+t0 to the image processing unit 105 .

In the image processing system 100 according to the first embodiment configured as described above, it is possible to improve the accuracy of estimating the parameters of the coordinate transformation matrix. Specifically, the image processing apparatus 10 detects a person from each of a plurality of input video frames, and associates the detection results of the person between the video frames based on the label. Since the image processing apparatus 10 associates the detection result of a person based on the label, the association can be easily performed without performing complicated calculations. Then, the image processing apparatus 10 estimates a coordinate transformation matrix between video frames based on the association result and the human detection result in each video frame. As described above, in the image processing system 100 according to the first embodiment, the coordinate transformation matrix can be estimated even if the characteristic area in the imaging target area 2 is not imaged as in the conventional art. Also, even if the area captured by the camera changes rapidly, the coordinate transformation matrix can be estimated by detecting and associating the person captured in the video frame. Therefore, the image processing system 100 can estimate the parameters of the coordinate transformation matrix even in a situation where the estimation accuracy of the parameters of the coordinate transformation matrix is degraded by the conventional method. Therefore, it is possible to improve the estimation accuracy of the parameters of the coordinate transformation matrix.

<Modification>
Some functional units included in the image processing apparatus 10 may be configured in separate housings. For example, the image acquisition unit 101, the object detection unit 102, the object association unit 103 and the transformation matrix estimation unit 104, or the object association unit 103 and the transformation matrix estimation unit 104 are installed in another housing as a coordinate transformation matrix estimation device. may be configured. In this configuration, the image processing device 10 acquires the coordinate transformation matrix from the coordinate transformation matrix estimation device and performs image processing on the video frame.

(Second embodiment)
In the second embodiment, an image processing apparatus detects a person from a video frame using a general object detector, and uses the result to estimate parameters of a coordinate transformation matrix. In the second embodiment, the configuration of the entire image processing system 100 is the same as that of the first embodiment, and only the configuration of the image processing apparatus is different.

FIG. 4 is a schematic block diagram showing the functional configuration of an image processing apparatus 10a according to the second embodiment.
The image processing apparatus 10a includes a CPU, a memory, an auxiliary storage device, and the like connected via a bus, and executes an image processing program. By executing the image processing program, the image processing device 10a functions as a device including an image acquisition unit 101, an object detection unit 102a, an object association unit 103a, a transformation matrix estimation unit 104, and an image processing unit 105. All or part of each function of the image processing apparatus 10a may be implemented using hardware such as ASIC, PLD, and FPGA. Also, the image processing program may be recorded on a computer-readable recording medium. Computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs and CD-ROMs, and storage devices such as hard disks incorporated in computer systems. Also, the image processing program may be transmitted and received via an electric communication line.

The image processing apparatus 10a differs in configuration from the image processing apparatus 10 in that it includes an object detection unit 102a and an object association unit 103a instead of the object detection unit 102 and the object association unit 103. FIG. The image processing apparatus 10a has the same configuration as the image processing apparatus 10 in other respects. Therefore, the description of the entire image processing apparatus 10a is omitted, and the object detection unit 102a and the object association unit 103a are described.

The object detection unit 102a detects an object imaged in each of the video frames acquired by the image acquisition unit 101 from each of the plurality of video frames. The object detection unit 102a in the second embodiment detects a person captured in each of the image frames It and It ^{+t0 using} a general object detector.

The object association unit 103a associates the person detection results between the video frames based on the person detection results of the video frames It and It ^{+t0 detected} by the object detection unit 102a.

FIG. 5 is a schematic block diagram showing the functional configuration of the object association unit 103a in the second embodiment. The object association unit 103 a is composed of a temporary association unit 1031 , a graph construction unit 1032 and a selection unit 1033 .

The provisional association unit 1031 provisionally associates the person detection results of the image frames It and It ^{+t0 detected} by the object detection unit 102a. As a method of provisionally matching, there is a method of matching the human detection result detected in the video frame It with the human detection result detected in the video frame It ^{+t0 by} round-robin. For example, when four human detection results are obtained in the video frame It and three human detection results are obtained in the video frame It ^{+t0 ,} the provisional association unit 1031 performs round-robin association to obtain 4×3= The results of the 12 provisional associations are output to the graph constructing unit 1032 .

Note that the provisional association unit 1031 temporarily associates the person detection result of each video frame detected by the object detection unit 102a by using a method of selecting only the plausible association from the round-robin association results. may be attached.
As a first method, the temporary association unit 1031 uses the method disclosed in Reference 2 to combine the image feature amount extracted from the person detection result in the video frame It and the person detection result in the video frame It ^{+t0 .} A degree of similarity with the image feature amount extracted from the result is calculated, and person detection results having a degree of similarity greater than or equal to the first threshold are provisionally associated with each other.

As a second method, the temporary association unit 1031 compares the aspect ratio of the rectangular area obtained from the human detection result in the video frame It and the rectangular area obtained from the human detection result in the video frame It ^{+t0 .} The vertical and horizontal ratios (aspect ratios) of the regions are compared, and human detection results having rectangular regions with the same vertical and horizontal ratios are provisionally associated with each other.

The graph constructing unit 1032 receives the results of the temporary association obtained by the temporary association unit 1031 as input, and constructs a graph having each set of temporarily associated person detection results as nodes. Specifically, the graph constructing unit 1032 constructs a graph in which each set of temporarily associated person detection results is virtually arranged as a node. For example, if the number of temporary association results obtained by the temporary association section 1031 is 12, the graph construction section 1032 constructs a graph in which 12 nodes are virtually arranged.

Based on the graph obtained by the graph construction unit 1032, the selection unit 1033 selects a correspondence that is assumed to be a correct correspondence from among the temporary correspondence results.

FIG. 6 is a flowchart showing the flow of coordinate transformation matrix estimation processing performed by the image processing apparatus 10a according to the second embodiment.
The image acquisition unit 101 acquires a plurality of video frames (step S201). The image acquisition unit 101 outputs the acquired plurality of video frames to the object detection unit 102a. The object detection unit 102a detects a person captured in each of the plurality of video frames (step S202). The object detection unit 102a in the second embodiment is a general object detector, as described above, and uses the coordinate values of the four corners of the rectangle surrounding the detected person as the person detection result. output to

FIG. 7 is a diagram showing an example of person detection by the object detection unit 102a in the second embodiment. As shown in FIGS. 7A and 7B, FIG. 7A is a diagram showing a video frame 20 captured at time t, and FIG. 7B is a diagram showing a video frame 30 captured at time t+ _t0 .
In the video frame 20, four persons 21-1 to 21-4 are detected. As the detection result of the image frame 20, the object detection unit 102a acquires the coordinate values of the four corners of the rectangles 22-1 to 22-4 surrounding the persons 21-1 to 21-4.
In the video frame 30, three persons 31-1 to 31-3 are detected. As the detection result of the image frame 30, the object detection unit 102a acquires the coordinate values of the four corners of the rectangles 32-1 to 32-3 surrounding the persons 31-1 to 31-3.

Returning to FIG. 6, the description continues.
The provisional association unit 1031 extracts a plurality of (for example, two) image frames to be associated that are imaged with a predetermined time interval difference from the plurality of image frames (step S203). Note that the extraction start timing and the predetermined time interval of the video frames extracted by the provisional association unit 1031 are set in advance. Based on the person detection result output from the object detection unit 102a in the extracted image frame, the temporary association unit 1031 temporarily associates the person detection result in the image frame 20 with the person detection result in the image frame 30. Associate (step S204). The provisional association unit 1031 outputs the result of provisional association to the graph construction unit 1032 .

FIG. 8 is a diagram showing the result of provisional association by the provisional association unit 1031 in the second embodiment. The example shown in FIG. 8 shows the result of provisional matching by the plausible matching method, not by round-robin. As a result, the example shown in FIG. 8 shows the results of seven provisional associations. As shown in FIG. 8, since this is a provisional correspondence, there are multiple correspondences for one person such as persons 21-2, 21-3, 21-4, 31-1 to 31-3. There are also places where it is done. In practice, it is correct that no matching is made, or that one matching is made even if a matching is made. Therefore, the image processing device 10a performs more accurate association by the processing described later.

Returning to FIG. 6, the description continues.
The graph constructing unit 1032 virtually arranges each pair of the provisionally associated person detection results obtained by the provisional association unit 1031 as a node. (step S205).

FIG. 9 is a diagram showing a graph construction result by the graph construction unit 1032 in the second embodiment.
The upper part of FIG. 9 shows the result of the provisional matching obtained by the provisional matching unit 1031, and the lower part of FIG. ing. A node 40-21 is a node indicating correspondence between the detection result of the person 21-2 and the detection result of the person 31-1. A node 40-41 is a node indicating correspondence between the detection result of the person 22-2 and the detection result of the person 31-3.

A node 40-33 is a node indicating correspondence between the detection result of the person 21-3 and the detection result of the person 31-3. A node 40-31 is a node indicating correspondence between the detection result of the person 21-3 and the detection result of the person 31-1. Nodes 40-42 are nodes indicating correspondence between the detection result of the person 21-4 and the detection result of the person 31-2. A node 40-12 is a node indicating correspondence between the detection result of the person 21-1 and the detection result of the person 31-2. A node 40-41 is a node indicating correspondence between the detection result of the person 21-4 and the detection result of the person 31-1.

Next, the graph constructing unit 1032 connects the node pairs that are geometrically consistent among the node pairs in the constructed graph with edges. Here, the geometric consistency between node pairs can be evaluated from the attribute information (for example, the coordinate values of the four corners of the rectangle) of the person detection results that form the node pairs. For example, if both of the two nodes of interest have a correct correspondence, the scale of the person imaged in each video frame is considered to be consistent. Therefore, the graph constructing unit 1032 first calculates, for each node, the area ratio of the person detection result area captured in the video frame. Next, the graph constructing unit 1032 calculates the similarities of the calculated area ratios for all node pairs. Then, the graph constructing unit 1032 connects the nodes of only the node pairs whose calculated similarity values are equal to or greater than the first threshold with edges.

Alternatively, if both of the two nodes of interest are correct correspondences, then the distance of the person detection result region captured in the video frame and the scale of the detected person are also consistent. It is believed that there is. Therefore, first, the graph constructing unit 1032 calculates the distance between the two human detection result regions captured in the video frame at time t in the two target nodes, and the distance between the two human detection result regions captured in the video frame at time _t +t and the distance between regions of the person detection result. Next, the graph constructing unit 1032 calculates, for each node, the area ratio of the human detection result area captured in the video frame. Next, the graph constructing unit 1032 calculates the similarity of the calculated area ratios for the two target nodes. Then, the graph constructing unit 1032 connects the nodes with edges only for the node pairs for which the calculated distance and similarity values are equal to or greater than the second threshold.

In the example shown in FIG. 9, it is assumed that the correspondence indicated by nodes 40-42, 40-31, and 40-41 is the correct correspondence. The above evaluation method will be explained with this condition taken into account. Here, the nodes 40-42 and 40-31 are used as the two target nodes. First, the graph constructing unit 1032 calculates the area ratios of the areas of the human detection result imaged in the video frame for each of the nodes 40-42 and 40-31. That is, the graph constructing unit 1032 calculates the ratio of the area of the region 22-4 surrounding the person 21-4 at the node 40-42 to the area of the region 32-2 surrounding the person 31-2 at the node 40-42. . Similarly, the graph constructing unit 1032 calculates the ratio of the area of the region 22-3 surrounding the person 21-3 at the node 40-31 to the area of the region 32-1 surrounding the person 31-1 at the node 40-31. do.

Next, the graph constructing unit 1032 calculates the similarity of the calculated area ratios. Then, when the calculated similarity value is equal to or greater than the first threshold, the graph constructing unit 1032 evaluates that there is geometric consistency between the nodes 40-42 and 40-31. do. The graph constructing unit 1032 performs such processing on all node pairs. Then, the graph constructing unit 1032 connects the node pairs evaluated as being geometrically consistent with edges.

FIG. 10 is a diagram showing processing in the graph constructed by the graph construction unit 1032 according to the second embodiment. As shown in FIG. 10, nodes 40-41, 40-31 and 40-42 are connected to each other by edges 42. In FIG. That is, it is shown that the nodes 40-41, 40-31 and 40-42 connected by the edge 42 were evaluated as geometrically consistent node pairs among the node pairs in the graph.
On the other hand, nodes 40-21, 40-33, 40-12 and 40-41 that are not connected by edge 42 are shown evaluated as geometrically inconsistent among the node pairs in the graph. .

The graph constructing unit 1032 associates, for each node, which person detection result in the video frame 20 (eg, video frame I ^t ) with which person detection result in the video frame 30 (eg, I ^t+t0 ). The graph including the constructed edge is output to the selection unit 1033 along with information indicating whether the edge is formed.
The selection unit 1033 executes cluster extraction processing based on the information output from the graph construction unit 1032 (step S206). Cluster extraction processing is processing for extracting clusters containing nodes that are expected to be correctly associated. Details of the cluster extraction processing will be described with reference to FIG.

The transformation matrix estimation unit 104 determines whether or not clusters have been extracted (step S207). If the output from selection section 1033 includes cluster information, transformation matrix estimation section 104 determines that a cluster has been extracted. On the other hand, when the output from selection section 1033 is Null, transformation matrix estimation section 104 determines that no cluster has been extracted. If no cluster is extracted (step S207-NO), the image processing apparatus 10a terminates the processing of FIG. In other words, the image processing device 10a outputs Null, indicating that the coordinate transformation matrix cannot be obtained.

On the other hand, if a cluster is extracted (step S207-YES), the transformation matrix estimation unit 104, based on the person matching result identified by the node included in the cluster and the person detection result by the object detection unit 102a, A coordinate transformation matrix Ht, ^{t +t0} of the image frames It and ^It+t0 is estimated (step S208). The transformation matrix estimation unit 104 outputs the estimated coordinate transformation matrix ^Ht,t+t0 to the image processing unit 105 .

FIG. 11 is a flow chart showing the flow of cluster extraction processing performed by the image processing apparatus 10a according to the second embodiment.
The selection unit 1033 selects a node with the highest degree in the graph including edges constructed by the graph construction unit 1032 (step S301). Here, the degree represents the number of edges connected to the node. A large number of edges means a large number of connected nodes. That is, the selection unit 1033 selects the node with the largest number of connected nodes in the graph. For example, in FIG. 10, the nodes with the highest number of edges are nodes 40-41, 40-31 and 40-42. Therefore, the selection unit 1033 selects one of the nodes 40-41, 40-31 and 40-42.

The selection unit 1033 determines whether the degree of the selected node is greater than or equal to the third threshold (step S302). If the degree of the selected node is not equal to or greater than the third threshold (step S302-NO), the selection unit 1033 outputs Null as a result of cluster extraction processing (step S303). After that, the selection unit 1033 terminates the cluster extraction process.
On the other hand, if the degree of the selected node is equal to or greater than the third threshold (step S302-YES), the selection unit 1033 initializes the cluster (step S304). Specifically, the selection unit 1033 initializes the cluster with an empty set. This leaves the cluster without any nodes. The selection unit 1033 ranks the peripheral nodes after initializing the cluster (step S305). Peripheral nodes are nodes other than the node selected in the process of step S301.

Specifically, the selection unit 1033 ranks peripheral nodes using the page rank algorithm, with the node selected in the process of step S301 as the starting node. At this time, it is also possible to use the approximate page rank algorithm disclosed in Reference 3 below. In this case, peripheral nodes can be ranked at a calculation cost that does not depend on the size of the graph. In general, the higher the number of edges, the higher the ranking tends to be.
(Reference 3: Reid Andersen, Fan Chung, Kevin Lang, “Local Graph Partitioning using PageRank Vectors”, In Proc. IEEE Annual Symposium on Foundations of Computer Science (FOCS), 2006)

As a result of the ranking, the selection unit 1033 selects a peripheral node ranked N=1 out of the plurality of peripheral nodes (step S306). The selection unit 1033 determines whether the selected peripheral node satisfies the conditions for addition to the cluster (step S307). The condition for adding to a cluster is a condition that must be met in order to add a peripheral node to a cluster. For example, feature points in the selected peripheral node are not used as feature points in nodes that have already been added to the cluster. That is. Here, the feature point in the node is the detection result of the person forming the node.

The condition for adding to a cluster is derived from the restriction that the same person appears only once in one video frame. Therefore, if the detection result of the person configuring the selected peripheral node is used as the detection result of the person configuring the node already included in the cluster, the selecting unit 1033 adds the selected peripheral node to the cluster. It is determined that the conditions for addition to are not met.
On the other hand, if the detection result of the person configuring the selected peripheral node is not used as the detection result of the person configuring the node already included in the cluster, the selecting unit 1033 clusters the selected peripheral node. It is determined that the conditions for addition to are satisfied.

If the selected peripheral node does not satisfy the conditions for addition to the cluster (step S307-NO), the selection unit 1033 does not add the selected peripheral node to the cluster (step S308). After that, the selection unit 1033 adds 1 to the value of the ranking N (step S309). The selection unit 1033 selects the N-th ranked peripheral node (step S310). After that, the selection unit 1033 executes the processes after step S307.

On the other hand, if the selected peripheral node satisfies the conditions for addition to the cluster (step S307-YES), the selection unit 1033 adds the selected peripheral node to the cluster (step S311). When the peripheral node is added to the cluster, the selection unit 1033 calculates an evaluation value score=δS/vol(S) indicating the accuracy of the cluster (step S312). Here, δS represents the number of edges connecting the peripheral nodes not included in the cluster and the nodes within the cluster, and vol(S) represents the sum of degrees of the nodes within the cluster. The selection unit 1033 associates and stores information specifying a node in the cluster with the calculated evaluation value score. In this way, every time the selection unit 1033 calculates the evaluation value score, the selection unit 1033 stores information specifying the nodes included in the cluster in association with the evaluation value score.

The selection unit 1033 determines whether or not the calculated evaluation value score is higher than the previous evaluation value score based on the stored information (step S313). If the evaluation value score is not higher than the previous evaluation value score (step S313-NO), the selection unit 1033 executes the processes after step S309.
On the other hand, when the evaluation value score is higher than the previous evaluation value score (step S313-YES), the selection unit 1033 selects the cluster when the previous evaluation value score was calculated. The selecting unit 1033 then extracts the selected cluster as the optimum cluster (step S314). Note that when the previous evaluation value score is not stored, the selection unit 1033 determines that the evaluation value score is not high because comparison cannot be made.

The evaluation value score becomes a smaller value as the combination forming the cluster is better. Therefore, when the evaluation value score becomes higher than the previous evaluation value score, there is a possibility that the combination of the nodes forming the cluster will deteriorate due to the addition of new peripheral nodes. Therefore, when the calculated evaluation value score is higher than the previous evaluation value score, the selection unit 1033 extracts the cluster when the previous evaluation value score was calculated as the optimum cluster.

The selection unit 1033 determines whether the size of the extracted cluster is equal to or greater than the fourth threshold (step S315). The cluster size is, for example, the number of nodes forming the cluster. If the size of the extracted cluster is less than the fourth threshold (step S315-NO), the selection unit 1033 executes the process of step S303.
On the other hand, if the size of the extracted cluster is equal to or larger than the fourth threshold (step S315-YES), the selection unit 1033 outputs the selected cluster as the result of cluster extraction processing (step S316). The cluster output by the selection unit 1033 satisfies the constraint that the person detection results corresponding to all the nodes included in the cluster are not shared among the nodes included in the cluster.

FIG. 12 is a diagram showing a cluster extraction result in the second embodiment.
As shown in FIG. 12, the cluster 41 selected in the process of FIG. 11 includes nodes 40-41, 40-31 and 40-42. Transformation matrix estimating unit 104 estimates a coordinate transformation matrix using the result of association between video frames indicated by nodes 40-41, 40-31 and 40-42 included in cluster 41 extracted by selecting unit 1033. do.

In the image processing system 100 according to the second embodiment configured as described above, it is possible to improve the accuracy of estimating the parameters of the coordinate transformation matrix. Specifically, first, the image processing device 10a provisionally associates human detection results between video frames. Thereby, one or more correspondences are made in one person. Next, the image processing apparatus 10a constructs a graph in which nodes are virtually arranged with the provisional correspondences as nodes. Next, the image processing device 10a searches for geometrically consistent node pairs in the constructed graph and connects them with edges. Next, the image processing device 10a selects nodes with correspondences that are assumed to be correct correspondences based on the edges, and extracts clusters that include the selected nodes and nodes that satisfy conditions for addition to the clusters. Since the extracted cluster includes nodes that are assumed to have correct association, the image processing device 10a combines the result of association indicated by the nodes included in the cluster with the detection result of the person in each video frame. Estimate the coordinate transformation matrix between video frames based on As described above, in the image processing system 100 of the third embodiment, the coordinate transformation matrix can be estimated even if the characteristic area in the imaging target area 2 is not imaged, as in the conventional art. Also, even if the area captured by the camera changes rapidly, the coordinate transformation matrix can be estimated by detecting and associating the person captured in the video frame. Therefore, the image processing system 100 can estimate the parameters of the coordinate transformation matrix even in a situation where the estimation accuracy of the parameters of the coordinate transformation matrix is degraded by the conventional method. Therefore, it is possible to improve the estimation accuracy of the parameters of the coordinate transformation matrix.

Further, the image processing apparatus 10a determines whether or not the condition for addition to the cluster is satisfied, and adds only nodes that satisfy the condition for addition to the cluster to the class. As a result, the same person detection result is not included in the cluster. Therefore, erroneous association can be reduced. As a result, it is possible to improve the estimation accuracy of the parameters of the coordinate transformation matrix.

<Modification>
Some functional units included in the image processing apparatus 10a may be configured in separate housings. For example, the image acquisition unit 101, the object detection unit 102a, the object association unit 103a and the transformation matrix estimation unit 104, or the object association unit 103a and the transformation matrix estimation unit 104 can be used as a coordinate transformation matrix estimation device in another housing. may be configured. In this configuration, the image processing device 10a acquires the coordinate transformation matrix from the coordinate transformation matrix estimation device and performs image processing on the video frame.
The selection unit 1033 may be configured to determine whether or not all peripheral nodes satisfy the conditions for addition to the cluster. In such a configuration, the selection unit 1033 determines whether or not all the ranked peripheral nodes have been processed instead of the process of step S313 in FIG. Then, if the processing has not been performed for all of the ranked peripheral nodes, the selection unit 1033 executes the processing after step S309. on the other hand. When all the ranked peripheral nodes have been processed, the selection unit 1033 selects a combination of nodes forming a cluster with the smallest evaluation value. Then, the selection unit 1033 extracts a cluster formed by combining the selected nodes as a processing result. If there are a plurality of clusters with the smallest evaluation values, the selection unit 1033 may randomly select clusters from among the clusters with the smallest evaluation values.

(Third embodiment)
As described in the first embodiment, when the specific object person detector is used, there is a label for each specific person, but if the accuracy of learning is low, an incorrect label is assigned, It is also assumed that the same label is given to multiple persons. In such a case, if the person detection results are associated with each other based on the label as in the first embodiment, the accuracy of estimating the parameters of the coordinate transformation matrix may decrease. Therefore, in the third embodiment, after a specific person is detected by the specific object person detector as in the first embodiment, provisional association is performed as in the second embodiment to estimate the coordinate transformation matrix. do.

FIG. 13 is a schematic block diagram showing the functional configuration of an image processing device 10b according to the third embodiment.
The image processing device 10b includes a CPU, a memory, an auxiliary storage device, etc., which are connected via a bus, and executes an image processing program. By executing the image processing program, the image processing device 10a functions as a device including an image acquisition unit 101, an object detection unit 102, an object association unit 103b, a transformation matrix estimation unit 104, and an image processing unit 105. All or part of each function of the image processing device 10b may be implemented using hardware such as ASIC, PLD, FPGA, GPU, or the like. Also, the image processing program may be recorded on a computer-readable recording medium. Computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs and CD-ROMs, and storage devices such as hard disks incorporated in computer systems. Also, the image processing program may be transmitted and received via an electric communication line.

The image processing apparatus 10b differs in configuration from the image processing apparatus 10 in that it includes an object association unit 103b instead of the object association unit 103. FIG. The image processing device 10b has the same configuration as the image processing device 10 in other respects. Therefore, the description of the entire image processing apparatus 10b is omitted, and the object matching unit 103b is described.

The object association unit 103b associates the person detection results between the video frames based on the person detection results of each video frame detected by the object detection unit 102 . Specifically, the object associating unit 103b uses the labels obtained as the person detection results to provisionally associate the person detection results to which labels indicating the same information are assigned as the same person detection results. As described above, if the accuracy of learning is low, an incorrect label may be assigned, or the same label may be assigned to a plurality of persons. Tentatively associate labeled person detection results. After that, the object association unit 103b performs the same processing as in the second embodiment.

Further, the object association unit 103b temporarily associates the person detection results by the same method as in the second embodiment, and sets of person detection results to which labels indicating the same information are assigned indicate the same information. It may be weighted so that it is more likely to be selected than a set of unlabeled person detection results.

In the image processing system 100 according to the third embodiment configured as described above, the person detection results to which the label indicating the same information, which is detected in each video frame, are assigned are associated as the detection result of the same person. Instead, it temporarily associates and selects the final association based on the correspondence with other detection results. As a result, when the accuracy of learning of the specific object person detector is low, even if the wrong label is assigned or the same label is assigned to multiple people, more accurate matching can be performed. It can be performed. Therefore, even when the accuracy of learning of the specific object person detector is low, it is possible to suppress deterioration in accuracy of estimating the parameters of the coordinate transformation matrix.

<Modification>
Some functional units included in the image processing apparatus 10b may be configured in separate housings. For example, the image acquiring unit 101, the object detecting unit 102, the object matching unit 103b and the transformation matrix estimating unit 104, or the object matching unit 103b and the transformation matrix estimating unit 104 are installed in another housing as a coordinate transformation matrix estimating device. may be configured. In this configuration, the image processing device 10b acquires the coordinate transformation matrix from the coordinate transformation matrix estimation device and performs image processing on the video frame.

(Fourth embodiment)
In the fourth embodiment, a case of using a plurality of images shot from different viewpoints (for example, facing directions) using a plurality of cameras will be described as an example.
FIG. 14 is a diagram showing the functional configuration of an image processing system 100c according to the fourth embodiment. The image processing system 100c includes a plurality of imaging devices 1 and 3 and an image processing device 10c. The imaging devices 1 and 3 are installed facing each other with an imaging target region 2 interposed therebetween, for example. Note that the installation positions and shooting directions of the plurality of imaging devices 1 and 3 are known.

The image capturing device 1 captures an image of an image capturing target area 2 . The imaging device 1 is, for example, a camera. The imaging device 1 generates a video frame by capturing an image of the imaging target region 2, and outputs the generated video frame to the image processing device 10c.
The imaging device 3 photographs the imaging target area 2 . The imaging device 3 is, for example, a camera. The imaging device 3 generates a video frame by imaging the imaging target region 2, and outputs the generated video frame to the image processing device 10c.

The image processing device 10c estimates a coordinate transformation matrix between video frames based on a plurality of input video frames, and performs image transformation of the video frames using the estimated coordinate transformation matrix. For example, the image processing device 10c estimates the coordinate transformation matrix between the video frames based on two video frames, the video frame captured by the imaging device 1 and the video frame captured by the imaging device 3. Also, for example, the image processing device 10 c estimates a coordinate transformation matrix between video frames based on a plurality of video frames captured by the imaging device 1 . Also, for example, the image processing device 10 c estimates a coordinate transformation matrix between video frames based on a plurality of video frames captured by the imaging device 3 .

A plurality of video frames input to the image processing device 10c are images captured by different imaging devices or images captured at different times. However, it is preferable that the shooting times of the plurality of video frames input to the image processing device 10c are closer. In the following description, the image captured by the imaging device 1 at time t is referred to as video frame ^It1 , and the image captured by imaging device 3 at time t is referred to as video frame ^It3 .

As shown in FIG. 14, in the fourth embodiment, unlike the first to third embodiments, an image processing device 10c selects one of video frames captured by a plurality of imaging devices 1 and 3. Or use both. The imaging devices 1 and 3 image the imaging target area 2 in opposite directions. Therefore, the moving direction of the object is opposite between the video frame captured by the imaging device 1 and the video frame captured by the imaging device 3 . Therefore, as in the first to third embodiments, if coordinate transformation is performed using the obtained coordinate transformation matrix as it is, a correct image cannot be obtained.

Therefore, first, the image processing device 10c uses the identification information (for example, camera ID) of the imaging devices 1 and 3 to determine which one of the imaging devices 1 and 3 captured the input video frame. judge. Next, when the imaging device is switched as a result of the determination, the image processing device 10c reverses one person detection result based on the relative relationship between the imaging device 1 and the imaging device 3 and associates them.

Next, a specific configuration of the image processing device 10c will be described.
The image processing device 10c includes a CPU, a memory, an auxiliary storage device, etc., which are connected via a bus, and executes an image processing program. By executing the image processing program, the image processing device 10c functions as a device including an image acquisition unit 101c, an object detection unit 102c, an object association unit 103c, a transformation matrix estimation unit 104c, and an image processing unit 105. All or part of each function of the image processing device 10c may be implemented using hardware such as ASIC, PLD, FPGA, GPU, or the like. Also, the image processing program may be recorded on a computer-readable recording medium. Computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs and CD-ROMs, and storage devices such as hard disks incorporated in computer systems. Also, the image processing program may be transmitted and received via an electric communication line.

The image processing device 10c includes an image acquisition unit 101c, an object detection unit 102c, an object association unit 103c, and a transformation matrix estimation unit instead of the image acquisition unit 101, the object detection unit 102, the object association unit 103, and the transformation matrix estimation unit 104. The configuration differs from that of the image processing apparatus 10 in that the image processing apparatus 104c is provided. The image processing device 10c has the same configuration as the image processing device 10 in other respects. Therefore, the overall description of the image processing apparatus 10c is omitted, and the image acquisition unit 101c, the object detection unit 102c, the object association unit 103c, and the transformation matrix estimation unit 104c are described.

The image acquisition unit 101c acquires video frames output from one or each of the imaging devices 1 and 3 . Further, the image acquisition unit 101c acquires the camera ID from the video frame, and detects the change when the acquired camera ID changes. When the image acquisition unit 101c detects a change in the camera ID, the image acquisition unit 101c sends a notification including the camera ID and information indicating which video frame is the video frame acquired when the camera ID changes. Output to the attaching unit 103c.

The object detection unit 102c detects objects captured in the video frames from each of the plurality of video frames acquired by the image acquisition unit 101c. The method of detecting an object by the object detection unit 102c performs the same processing as that of the function unit with the same name in any one of the first to third embodiments. That is, the object detection unit 102c uses either a general object detector or a specific object detector to detect a person from a video frame.

The object association unit 103c associates the person detection results between the video frames based on the person detection results of each video frame detected by the object detection unit 102c. Specifically, first, the object association unit 103c extracts a plurality of image frames to be associated, which are imaged at a predetermined time interval difference from the plurality of frames. Next, the object association unit 103c detects an object between the plurality of video frames extracted as the association target based on the detection result of the object detected in each of the plurality of image frames extracted as the association target. Match the results. A plurality of video frames to be matched by the object matching unit 103c are video frames captured by the same imaging device or different imaging devices with a predetermined time interval difference, and video frames captured by different imaging devices at the same time. It is a video frame. In this way, the object associating unit 103c selects a plurality of image frames captured with a predetermined time interval difference or captured at the same time by different image capturing devices among the image frames detected by the object detection unit 102c. Based on the detection results of the object detected in , the object detection results between a plurality of video frames captured with a predetermined time interval difference or captured by different imaging devices at the same time are associated with each other.

For example, the object associating unit 103c combines the detection result of the object detected in the video frame It captured by the imaging device 1 at time ^t with the imaging device 3 at a time a predetermined time apart from time t or at the same time. Based on the detection results of the objects detected in the imaged video frames, the detection results of the objects between the video frames are associated with each other. Further, for example, the object associating unit 103c combines the detection result of the object detected in the video frame It captured by the imaging device 1 at time ^t with the imaging device 1 at time t+ _t0 , which is a predetermined time away from time t. Based on the detection result of the object detected in the image frame It ^{+t0 picked} up by the image frame It+ ^t0 , the object detection result between the image frame It and the image frame It ^+t0 is associated. Further, for example, the object associating unit 103c combines the detection result of the object detected in the video frame ^It3 captured by the imaging device 3 at time t with the imaging device 3 at time t+ _t0 , which is a predetermined time away from time t. Based on the detection result of the object detected in the image frame ^It3+t0 captured by , the object detection result between the image frame ^It3 and the image frame ^It3+t0 is associated with each other.

It should be noted that the object association unit 103c performs the same processing as the function unit with the same name in any one of the first to third embodiments. For example, when the object detection unit 102c uses a general object detector, the object association unit 103c performs processing similar to that of the same-named function unit in either the first embodiment or the third embodiment. I do. Further, for example, when the object detection unit 102c uses a specific object detector, the object association unit 103c performs the same processing as the functional unit with the same name in the second embodiment. The difference between the object association unit 103c and the first to third embodiments is that the processing of the human detection result between the video frames is different when the notification is acquired from the image acquisition unit 101c.

Specifically, the object associating unit 103c associates the video frame acquired when the camera ID changes with the video frame acquired immediately before or immediately after that, in which the imaging device 1 and the imaging device 3 One of the human detection results is reversed from the relative relationship between the two.

The transformation matrix estimation unit 104c estimates the coordinate transformation matrix of the video frame I based on the person detection result by the object detection unit 102c and the matching result by the object matching unit 103c.

In the image processing system 100c configured as described above, when image frames obtained by photographing the photographing target area 2 from different directions with a plurality of cameras are acquired, the person detection result between the image frames indicates that one person Correlation is performed after reversing the detection result. As a result, even when image frames obtained by photographing the photographing target area 2 from different directions with a plurality of cameras are obtained, the parameters of the coordinate transformation matrix can be estimated.

<Modification>
The imaging devices 1 and 3 may be installed so as to shoot in the same direction, or may be installed so as to shoot in different directions. When the imaging devices 1 and 3 are installed so as to capture images in the same direction, the image acquisition unit 101c does not need to perform determination processing based on the camera ID. When the imaging devices 1 and 3 are installed so as to shoot in different directions, for example, the imaging devices 1 and 3 are installed so as to face orthogonal directions.

<Modification Common to First to Fourth Embodiments>
The image acquisition units 101 and 101c may be configured to acquire video frames from sources other than the imaging device. Specifically, the image acquisition units 101 and 101c may acquire video frames from a storage device that stores video frames in which part or all of the imaging target region 2 is captured, or may acquire video frames from a USB (Universal Serial Bus). ) or a portable recording medium such as an SD card, or from a network.

Although the embodiment of the present invention has been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design and the like are included within the scope of the gist of the present invention.

DESCRIPTION OF SYMBOLS 1, 3... Imaging device 10, 10a, 10b, 10c... Image processing apparatus 101, 101c... Image acquisition part 102, 102a, 102c... Object detection part, 103, 103a, 103b, 103c... Object correspondence part, 1031 Temporary association unit 1032 Graph constructing unit 1033 Selecting unit 104, 104c Transformation matrix estimating unit 105 Image processing unit

Claims (6)

a detection step of detecting an object from each of a plurality of images;
an extracting step of extracting, from the plurality of images, a plurality of images to be associated that have been captured with a predetermined time interval difference or have been captured by different imaging devices at the same time;
an object association step of associating object detection results between the plurality of images extracted as the association target;
a transformation matrix estimation step of estimating a coordinate transformation matrix between the images based on the matching result in the object matching step and the detection result of the object;
has
The object matching step includes:
a provisional matching step of provisionally matching the object detection results between the images;
a graph construction step of constructing a graph whose nodes are each set of object detection results between the images provisionally associated in the provisional association step;
Among the plurality of nodes in the graph, the conditions for addition to the cluster containing the nodes of the correspondence assumed to be correct are satisfied in order from the node with the highest priority assigned in the order of the relationship between the nodes. a selection step of selecting the node of the supposed correct mapping by determining whether
Coordinate transformation matrix estimation method with In the detection step, after detecting the object, classify the object by determining the classification of the detected object;
When information indicating the classification result of the object is included in the detection result of the object between the images, in the provisional association step, the detection result of the object including the information indicating the same classification result is matched between the images to the same detection result. 2. The method of estimating a coordinate transformation matrix according to claim 1 , wherein provisional association is made as a detection result of an object. In the graph construction step, evaluating the presence or absence of geometric consistency of node pairs included in the constructed graph, and connecting the node pairs evaluated as having geometric consistency with edges,
3. The coordinates according to claim 1 or 2 , wherein in the selecting step, the node of the correspondence assumed to be the correct correspondence is selected with a node having a large number of nodes connected by the edge as a node with a high priority. Transformation matrix estimation method. In the selection step, it is determined in order from the node with the highest priority that the conditions for addition to the cluster are satisfied if the feature point of the node is not used as a feature point in a node that has already been added to the cluster. 4. The coordinate transformation matrix estimation method according to any one of claims 1 to 3 , wherein a node that satisfies an addition condition to said cluster is added to said cluster. When a detection result of an object imaged in each of a plurality of images obtained by photographing an imaging target area in opposite directions by a plurality of imaging devices is input,
5. The coordinate transformation matrix estimation method according to any one of claims 1 to 4 , wherein in said object matching step, matching is performed after reversing the detection result of one of the objects. A computer program for executing the coordinate transformation matrix estimation method according to any one of claims 1 to 5 .

Images