JP6659095B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing device, an image processing method, and a program.

  In a surveillance camera system, there has been proposed a method of estimating a three-dimensional position of an object by photographing the object using a plurality of cameras having overlapping visual fields. According to this method, the three-dimensional position of the subject is estimated from the position of the subject on the camera image by photographing the subject with a camera whose position is known according to the principle of stereo vision.

  At this time, there is a problem that it is estimated that an object exists at a three-dimensional position where the object does not actually exist. Hereinafter, a three-dimensional position where an object exists, that is, a correctly estimated three-dimensional position, is called a real image, and a three-dimensional position where no object exists, that is, a three-dimensional position that is incorrectly estimated is called a virtual image. The virtual image occurs due to the positional relationship between the camera and the person. FIG. 2 illustrates a positional relationship between a camera and a person where a virtual image occurs. In FIG. 2, camera 1 captures person B, and camera 2 captures person A and person B, respectively. Since the intersection of the straight line connecting the camera and the person is estimated as the three-dimensional position of the object, a real image is generated at the intersection of the straight line connecting the camera 1 and the person B and the straight line connecting the camera 2 and the person B. On the other hand, a virtual image also occurs at the intersection of a straight line connecting camera 1 and person B and a straight line connecting camera 2 and person A.

  As a solution to this problem, Patent Literature 1 proposes a method of reducing a virtual image based on the interlocking of image movement. Specifically, utilizing the property that many images move in conjunction with a virtual image, images that move in conjunction with a certain image are counted, and if the counted value is large, the image is determined as a virtual image. Patent Document 2 discloses a method of obtaining a three-dimensional movement trajectory of a person, connecting the fragments of the three-dimensional movement trajectory, and calculating a complete movement trajectory of each person. According to the method of Patent Literature 2, a virtual image can be reduced by connecting trajectories over a predetermined time length.

Japanese Patent No. 5454573 JP 2010-063001 A

  The method of Patent Literature 1 has a problem that it is not easy to determine a virtual image based on interlocking, because the virtual image increases when the number of cameras or objects is large. In addition, the method of Patent Document 2 has a problem that it is difficult to use when responsiveness is required because the trajectories having a fixed length of time are connected to each other, and the amount of calculation increases.

  An object of the present invention is to effectively reduce virtual images when estimating a three-dimensional position of an object.

In order to achieve an object of the present invention, for example, an image processing apparatus of the present invention has the following configuration. That is,
Acquisition means for acquiring a group of continuously captured frame images from each of a plurality of imaging units having overlapping visual fields,
Tracking means for tracking an object on the frame image group;
Generating means for generating a candidate for a three-dimensional position of the object at a time of interest based on a position on the frame image of the object and a positional relationship between the plurality of imaging units;
Associating means for associating the candidate of the three-dimensional position at the time of interest with the candidate of the three-dimensional position of the object at a past time before the time of interest;
Based on the local correspondence accuracy indicating the probability that the candidate of the three-dimensional position at the time of interest corresponds to the object, and the integration correspondence accuracy indicating the probability that the candidate of the three-dimensional position at the past time corresponds to the object, Calculation means for calculating an integrated correspondence accuracy indicating a probability that the candidate of the three-dimensional position at the time of interest corresponds to the object,
For each of the candidates for the three-dimensional position at the time of interest, the integration correspondence accuracy recorded corresponding to the object tracked on one frame image group is obtained, and for each of the plurality of frame image groups, Determining the reliability of the candidate of the three-dimensional position according to the acquired integration correspondence accuracy, determining means for determining the three-dimensional position of the object at the time of interest based on the reliability ,
Is provided.

  According to the present invention, it is possible to effectively reduce virtual images when estimating the three-dimensional position of an object.

FIG. 1 is a block diagram illustrating a functional configuration example of an image processing apparatus according to an embodiment. The figure explaining a virtual image. FIG. 1 is a diagram illustrating an example of a hardware configuration of an image processing apparatus according to an embodiment. 9 is a flowchart of a process according to an embodiment. 9 is a flowchart of a process for generating a candidate for a three-dimensional position. The figure explaining the process which produces | generates the candidate of a three-dimensional position. The figure explaining the process which produces | generates the candidate of a three-dimensional position. 9 is a flowchart of a process for calculating a correspondence accuracy. The figure explaining the distance of the candidate of the three-dimensional position projected on the image and the object. 9 is a flowchart of a process of associating a three-dimensional position candidate with an object. FIG. 4 is a diagram illustrating an example of a display image according to the embodiment.

  Hereinafter, embodiments of the present invention will be described. Hereinafter, a case will be described in which a person is a detection target, but the detection target may be another object.

  FIG. 3 illustrates a hardware configuration example of the image processing apparatus 100 according to the present embodiment. The imaging device 301 corresponds to an imaging unit, and converts an optical subject image into an electric signal. As the imaging element 301, for example, a CCD or a CMOS can be used. The signal processing circuit 302 converts a time-series electric signal obtained from the image sensor 301 into a digital signal. An image including a subject image can be generated by the imaging element 301 and the signal processing circuit 302. In the following description, it is assumed that the image processing apparatus 100 has a plurality of cameras, and each of the cameras has an image sensor 301.

  The CPU 303 controls the entire image processing apparatus 100 by executing a control program stored in the ROM 304. More specifically, each function shown in FIG. 1 is realized by the CPU 303 executing the control program, and each processing shown in a flowchart described later is performed. The ROM 304 is a storage medium that stores a control program executed by the CPU 303 and various parameter data. The RAM 305 stores images and various information. The RAM 305 functions as a work area for the CPU 303 or a temporary save area for data. The display 306 displays an image.

  The configuration of the image processing apparatus 100 is not limited to that shown in FIG. For example, the image processing apparatus 100 can be realized using a general-purpose PC. In this case, the image processing device 100 does not need to include the image sensor 301, the signal processing circuit 302, and the display 306. In one embodiment, the imaging device 301 and the signal processing circuit 302 are included in an imaging device such as a camera, and the imaging device can communicate with the image processing device 100. In such an embodiment, the image processing apparatus 100 can acquire an electric signal obtained by each of the two or more imaging elements 301.

  In a processing device such as a computer, a processor such as a CPU 303 executes software (program) acquired via a network or a storage medium to execute the functions shown in FIG. 1 and each process shown in a flowchart described later. It may be realized. On the other hand, one or more of these functions and processes can be realized using dedicated hardware such as an electronic circuit. For example, the image processing device 100 may be a device dedicated to object detection configured by hardware dedicated to object detection.

  FIG. 1 shows a functional configuration of the image processing apparatus 100. The image processing apparatus 100 includes an information storage unit 103, an object tracking unit 102, a position storage unit 123, a probability calculation unit 109, and a position determination unit 115. The image processing apparatus 100 further includes an image acquisition unit 101, a position generation unit 104, a position merging unit 124, a correspondence storage unit 125, a position update unit 119, a position deletion unit 122, and a display control unit 126.

  In the following description, it is assumed that the image processing apparatus 100 has two cameras having overlapping visual fields. As described above, each camera includes the image sensor 301. However, the number of cameras is not particularly limited as long as it is two or more.

  The information holding unit 103 holds information on the internal parameters and the position and orientation of each camera obtained by performing the calibration. A known method can be used as the calibration method. For example, an internal parameter of each camera can be obtained using an image obtained by imaging a calibration board installed in the environment with a camera. As a specific method, there is a method described in Zhengyou Zhang. "A flexible New Technique for Camera Calibration", IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 (11): 1330-1334, 2000. Thereafter, the position and orientation of each camera can be estimated using an image obtained by imaging the calibration marker, whose position and orientation are set in the environment, with a known camera.

  As another method, a method of extracting a feature point such as a corner or a SIFT feature from an image captured by a camera and associating the feature points extracted from each image with a method of estimating the position and orientation of the camera is used. Can also. As a specific method, Pierre Moulon, Pascal Monasse, and Renaud Marlet. "Adaptive structure from motion with a contrario model estimation", Computer Vision-ACCV 2012: 11th Asian Conference on Computer Vision, Part IV, pp. 257-270 , 2013. Further, the camera internal parameters and the position and orientation may be obtained at the same time.

  The image acquisition unit 101 acquires a group of continuously captured frame images from each of the plurality of imaging units. For example, the image acquisition unit 101 can acquire, from each camera, a video composed of a group of frame images shot continuously. In this embodiment, since two cameras are used, the image acquisition unit 101 acquires two videos. Each unit included in the image processing apparatus performs processing with reference to a plurality of frame images captured by each camera almost simultaneously.

  The object tracking unit 102 tracks an object on the frame image group acquired by the image acquiring unit 101. For example, the object tracking unit 102 can perform a tracking process on each frame image acquired by the image acquisition unit 101. As a result, the image area of the tracking target object is detected from each frame image. In this embodiment, since the tracking of the person is performed, the object tracking unit 102 extracts the person region from the image, and the person region extracted from the image at the time of interest and the person extracted from the past image with respect to the same person. The area is associated with the area. Thus, the same person can be tracked over a plurality of times. A known method can be used as the tracking processing method. For example, a person tracking process can be performed by performing a person detection process on each of the continuous images and associating a person detected from each image based on the matching score. As a specific method, a method described in MD Breitenstein et al. "Robust tracking-by-detection using a detector confidence particle filter", 2009 IEEE 12th International Conference on Computer Vision (ICCV), pp. 1515-1522. No.

  As a result of the tracking processing by the object tracking unit 102, a set of the two-dimensional position of the tracking target object included in each image and the identifier of the tracking target object is obtained. Hereinafter, the image area of the tracking target object is referred to as a tracking result or simply an object, and the identifier of the tracking target object is referred to as a tracking result identifier. In the present embodiment, a set of the two-dimensional position of the person included in each image and the identifier of the person is obtained. More specifically, a set of the two-dimensional position of the person region included in each image and the identifier of the person region is obtained. In the present embodiment, the person area is a rectangular area, and the representative point coordinates (x, y) of the rectangular area, the height h of the rectangular area, and the width w of the rectangular area are obtained as the two-dimensional position of the human area. The representative point coordinates may be, for example, the center coordinates of the detected face area or the center coordinates of the rectangular area. Also, a 2D tracking label i is obtained as an identifier of the tracking result. The 2D tracking label i is a code for identifying the person tracked in the image for each camera. The 2D tracking label i is determined so that different persons are given different 2D tracking labels i in images obtained by the same camera.

  The position storage unit 123 stores information on a three-dimensional position. Details will be described later.

  The position generation unit 104 generates a candidate for the three-dimensional position of the object at the time of interest based on the position of the object on the frame image and the positional relationship between the plurality of imaging units. For example, the position generation unit 104 can newly generate a three-dimensional position candidate based on the two-dimensional position of the person obtained by the tracking processing by the object tracking unit 102. Specifically, the position generation unit 104 creates three-dimensional position candidates by associating the two-dimensional positions of the persons detected from the respective images. The three-dimensional position indicates a position of a person in a three-dimensional space, and is represented by coordinates (x, y, z). In addition, the position generation unit 104 assigns an identifier and attribute information to the three-dimensional position candidate. Hereinafter, the three-dimensional position candidate may be simply referred to as a three-dimensional position.

  The identifier is a code for identifying the generated three-dimensional position candidate, and for example, a 3D tracking label j can be used. The 3D tracking label j is determined so that different 3D tracking labels j are assigned to different three-dimensional positions. The attribute information is a value or a code representing the property of the three-dimensional position. In the present embodiment, the attribute of the three-dimensional position is determined using the attribute of the tracking result (the attribute of the object) corresponding to the three-dimensional position. For example, the color (color attribute) of a person can be used as an attribute. In another embodiment, instead of the color attribute, the age, gender, height, direction, or size of the person, a feature amount extracted from the tracking result, or the like can be used as the attribute. As a specific example, the position generation unit 104 extracts a feature amount from a tracking result on each frame image referred to when generating a three-dimensional position candidate, and extracts a feature amount of each frame image as an attribute of the three-dimensional position. It is possible to calculate a statistic of the extracted feature quantity, for example, an average value.

  The position generation unit 104 includes an association unit 105, a position estimation unit 106, and an attribute extraction unit 107. The associating unit 105 associates a person tracked by the object tracking unit 102 between images. The position estimating unit 106 generates candidates for the three-dimensional position of the person associated by the associating unit 105. The attribute extraction unit 107 acquires the attribute of the person associated by the association unit 105. Details of these will be described later.

  The position storage unit 123 stores information on a three-dimensional position. For example, the position storage unit 123 stores a set of the three-dimensional position candidate generated by the position generation unit 104, a 3D tracking label j for the three-dimensional position, and an attribute for the three-dimensional position. Can be.

  The position merging unit 124 associates the candidate of the three-dimensional position at the time of interest with the candidate of the three-dimensional position of the object at a past time before the time of interest. In the present embodiment, the position merging unit 124 includes a three-dimensional position candidate at the time of interest generated based on the position of the object on the frame image and the positional relationship between the plurality of imaging units, and a three-dimensional position at the past time. And the position candidates are merged. Thus, the position merging unit 124 associates the candidate of the three-dimensional position at the time of interest with the candidate of the three-dimensional position at the past time.

  For example, the position merging unit 124 can merge a plurality of three-dimensional position candidates generated by the position generating unit 104 into one. By this processing, three-dimensional positions determined to correspond to the same person are merged. In addition, as a result of this processing, the correspondence between the candidate of the three-dimensional position at the time of interest and the candidate of the three-dimensional position of the object at the past time before the time of interest can be realized. However, it is not always necessary to perform the association by merging the three-dimensional positions, and it is determined that the candidate of the three-dimensional position at the time of interest and the candidate of the three-dimensional position at the past time correspond to the same person. The information shown can also be recorded.

  As a result of the processing by the position generation unit 104, a plurality of three-dimensional positions of the same person may be generated. In addition, a plurality of three-dimensional positions of the same person may exist due to an estimation error caused by the entire process illustrated in FIG. Therefore, the position merging unit 124 searches for a plurality of three-dimensional positions that can be regarded as the same person and merges them into one in order to limit the three-dimensional position of the same person to one. In addition, the position merging unit 124 is generated using the three-dimensional position candidate generated using the frame image at the time of interest that is currently being processed and the frame image at the past time before the time of interest. The three-dimensional position candidates are also merged. The correspondence storage unit 125 stores information indicating that the three-dimensional positions merged by the position merging unit 124 are the same person. Details of these will be described later.

  The accuracy calculation unit 109 obtains the integration accuracy of the three-dimensional position candidate at the time of interest corresponding to the object. This processing is performed based on the local correspondence accuracy indicating the probability that the candidate of the three-dimensional position at the time of interest determined based on the frame image at the time of interest corresponds to the object, and the accuracy of the candidate of the three-dimensional position at the past time corresponding to the object. This is performed based on both the integration correspondence accuracy and In the present embodiment, the accuracy calculation unit 109 calculates the correspondence accuracy (local correspondence accuracy) between the tracking result i and the three-dimensional position j considering only the time of interest t, and the three-dimensional position j based on the past frame image. The correspondence accuracy is calculated based on the correspondence accuracy (accumulation correspondence accuracy) with the tracking result i. The method of calculating the correspondence accuracy will be described later in detail.

  The correspondence accuracy is an index indicating the possibility that the object indicated by the candidate of the three-dimensional position is the same as the object being tracked, and indicates the degree to which the three-dimensional position and the tracking result correspond to the same person. Things. Thus, the accuracy calculation unit 109 calculates the accuracy of the correspondence between the three-dimensional position and the tracking result. Here, the three-dimensional position refers to a three-dimensional position obtained by the merging process by the position merging unit 124, and each three-dimensional position is referred to as a three-dimensional position j using a 3D position label j. The tracking result indicates a person being tracked in each image by the processing by the object tracking unit 102, and each tracking result is referred to as a tracking result i using a 2D position label i.

  The certainty calculating unit 109 includes a distance calculating unit 110, a difference calculating unit 111, a local certainty calculating unit 112, an integrated certainty calculating unit 113, and an integrated certainty storing unit 114. The distance calculation unit 110 calculates the object distance from the three-dimensional position, the result of the object tracking unit, and the positional relationship of the camera. Specifically, the distance between the three-dimensional position projected on the image and the tracking result on the image is calculated. The difference calculation unit 111 calculates the difference between the color attribute of the three-dimensional position and the color attribute extracted from the tracking result of the person detection area. The local accuracy calculation unit 112 calculates the accuracy (local accuracy) between the three-dimensional position and the tracking result in consideration of only the time of interest t, and will be described in detail later. The integrated accuracy calculation unit 113 updates the integrated correspondence accuracy stored in the integrated accuracy storage unit 114 with the local correspondence accuracy acquired by the local accuracy calculation unit 112, and will be described later in detail. The integrated accuracy storage unit 114 records the obtained integrated corresponding accuracy as the integrated corresponding accuracy indicating the accuracy of the candidate of the three-dimensional position at the time of interest corresponding to the object. The recorded integration correspondence accuracy corresponds to a value obtained by integrating the local correspondence accuracy.

  The position determination unit 115 determines the three-dimensional position of the object at the previous eye time based on the integration correspondence accuracy. In the present embodiment, the position determination unit 115 determines that the candidate of the three-dimensional position at the time of interest that has the higher accumulated integration correspondence accuracy corresponds to the three-dimensional position of the object at the time of interest. In this way, the position determination unit 115 determines the three-dimensional position from the correspondence accuracy. In the present embodiment, the real image reliability indicating the real image likelihood of the three-dimensional position is calculated, and the real image reliability is determined using the calculated reliability. Further, the correspondence between the three-dimensional position and the tracking result is obtained. The position determination unit 115 includes a reliability calculation unit 116, a position selection unit 117, and a result selection unit 118. The reliability calculation unit 116 calculates a real image reliability indicating a real image likelihood of a three-dimensional position. The position selection unit 117 obtains a three-dimensional position with a high real image reliability and determines it as a real image. The result selection unit 118 determines a tracking result corresponding to the three-dimensional position. Details of these processes will be described later.

  The position updating unit 119 updates the three-dimensional position and the attribute using the correspondence between the three-dimensional position obtained by the position determining unit 115 and the tracking result. The position updating unit 119 includes a coordinate updating unit 120 and an attribute updating unit 121. The coordinate updating unit 120 and the attribute updating unit 121 update the three-dimensional position and the attribute of the three-dimensional position. The position deleting unit 122 deletes unnecessary three-dimensional positions. The display control unit 126 displays the detection result and the three-dimensional position of the object on the display together with the camera image.

  Hereinafter, the operation of the image processing apparatus 100 will be described with reference to the flowchart of FIG. In step S401, the internal parameters and the position and orientation of each camera are estimated by performing the calibration as described above, and the information is held in the information holding unit 103. In step S402, the image acquisition unit 101 acquires a frame image from each camera as described above. Hereinafter, it is assumed that the image acquisition unit 101 acquires a frame image at time t. In step S403, the object tracking unit performs the person tracking process as described above, and assigns the representative point coordinates (x, y), the height h, the width w, and the 2D tracking label i to each tracking result.

  In step S404, the position generation unit 104 generates the candidates for the three-dimensional position of the person as described above, and stores the information of the three-dimensional position in the position storage unit 123. Details of the processing in step S404 will be described with reference to FIG. In step S501, the position estimating unit 106 searches for a tracking result detected from a certain image corresponding to a tracking result of another image. The method of associating the tracking results between the images is not particularly limited, and includes, for example, a method using epipolar geometry. Hereinafter, a method for associating the tracking result using the epipolar geometry will be described. For the sake of explanation, it is assumed below that two cameras, camera 1 and camera 2, are used, and an image captured by one camera is referred to as camera image 1, and an image captured by the other camera is referred to as camera image 2.

  FIG. 6A shows a camera image 1. The camera image 1 shows the person A. Since the information holding unit 103 has information indicating the relative position and orientation of each camera and internal parameters, the person A shown at a predetermined position in the camera image 1 is referred to by referring to this information. It is possible to know at which position the image 2 appears. Specifically, when the person A is shown in the camera image 2, the position is somewhere on a straight line called an epipolar line.

Let F be a fundamental matrix which is a matrix containing information on the positional relationship between the camera image 1 and the camera image 2 obtained from the position and orientation of the cameras 1 and 2 and the internal parameters. Further, a vector representing the dimensional coordinates of the person A in the camera image 1 is defined as x. Then, as conventionally known, the epipolar line 1 is expressed by the following equation.

  The associating unit 105 determines a person whose distance between the representative point and the epipolar line is equal to or smaller than the threshold in the camera image 2 as a person corresponding to the person A. For example, in FIG. 6, a person A corresponds to a person B and a person C. However, since there is at most one person identical to person A in camera image 2, at least one of person B and person C is erroneously detected as a person corresponding to person A.

  In step S502, the position estimating unit 106 estimates the three-dimensional position of the person based on the set of persons associated in step S501. This estimation can be made according to the principle of triangulation. For example, in a three-dimensional space, an intersection of a straight line from the optical center of the camera A toward the representative point of the person A and a straight line from the optical center of the camera B toward the representative point of the person B, It can be obtained as estimated three-dimensional positions of persons A and B estimated to be the same person. As a specific example, as shown in FIG. 7, the position estimating unit 106 acquires a straight line in the three-dimensional space connecting the coordinates of the representative point of the person in each camera image and the center coordinates of the camera. Then, the position estimating unit 106 acquires coordinates at which these straight lines obtained from a plurality of cameras intersect. Strictly speaking, these straight lines rarely intersect. Therefore, the position estimating unit 106 obtains, as the estimated three-dimensional position of the person, the midpoint coordinates of the common perpendicular to these straight lines instead of the coordinates of the intersection. Can be.

  In step S503, the attribute extracting unit 107 acquires the attribute of the set of persons associated in step S501. In the present embodiment, the attribute extracting unit 107 acquires the attribute of the tracking result included in each camera image, and acquires the attribute of the set of persons based on the obtained attribute value. As a specific example, the attribute extracting unit 107 can calculate the color attribute of the tracking result of the person A in the camera image 1 and the color attribute of the tracking result of the person B in the camera image 2. This color attribute can be, for example, the average of the colors of the tracking result. Then, the attribute extracting unit 107 can use the average of the color attributes of the tracking results as the color attributes of the persons A and B.

  In another embodiment, the associating unit 105 can associate a person with reference to the attribute of the tracking result instead of or in addition to the epipolar geometry. As a specific example, when the error of the attribute of each tracking result is equal to or smaller than a threshold, each person can be associated. In this case, before step S501, the attribute extracting unit 107 can acquire an attribute (for example, a color attribute) of each tracking result. Also, in step S503, the attribute extraction unit 107 can acquire the color attribute of the person by averaging the color attributes of the tracking result included in the corresponding camera image, which have already been obtained.

  Step 404 has described the association between the two cameras. However, when three or more cameras are used, the association can be similarly performed. For example, for each combination of two cameras selected from a plurality of cameras, association and generation of three-dimensional position candidates can be performed. Also, correspondence and generation of three-dimensional position candidates can be performed using geometric constraints of multiple viewpoints.

  According to step S404, a new three-dimensional position is created based on the tracking result detected in step S403. By this process, a new real image candidate that has not been detected in the previous frame image can be found.

  In step S405, the position merging unit 124 merges three-dimensional positions determined to correspond to the same person into one, as described above. In the present embodiment, when the distance between the three-dimensional positions is short, the three-dimensional positions are regarded as representing the same person, and the position merging unit 124 merges the three-dimensional positions. As a specific example, the position merging unit 124 searches for a three-dimensional position in which the distance between the three-dimensional positions is equal to or less than a certain value, and creates a group of three-dimensional positions in which the distance is short. Thus, the three-dimensional positions included in one group are merged. The position merging unit 124 then obtains the average coordinates of the three-dimensional positions for each group, and sets the average coordinates of the three-dimensional positions after the merging.

  In the present embodiment, the distance between the three-dimensional position generated by the position generation unit 104 based on the frame image at the time of interest and the three-dimensional position generated by the position generation unit 104 based on the past frame image is short. Also, the three-dimensional positions are merged. As a specific example, the position merging unit 124 generates the three-dimensional position after merging obtained by referring to the past frame image in the previous step S405 and the frame image at the time of interest in the current step S404. Can be merged with the three-dimensional position candidates. In another embodiment, the candidate of the three-dimensional position generated in the past step S404 based on the past frame image and the candidate of the three-dimensional position generated in the current step S404 based on the frame image of the time of interest are provided. And can also be merged.

  As the attribute of the three-dimensional position after merging and the 3D tracking label j, any one of the three-dimensional positions before merging is adopted. In the present embodiment, the attribute of the three-dimensional position existing from the oldest time and the 3D tracking label j are adopted. Such a configuration can be realized by adding information indicating the time generated at the three-dimensional position. However, other recruitment criteria can be used. The position merging unit 124 further includes information indicating that two or more three-dimensional positions correspond to the same object when merging two or more three-dimensional position candidates at the past time determined to correspond to different objects. Can be recorded. For example, the position merging unit 124 can record information indicating that the three-dimensional position having the adopted 3D tracking label and the three-dimensional position having the adopted 3D tracking label represent the same person. . As a specific example, the position merging unit 124 can record information indicating that these 3D tracking labels are the same in the correspondence storage unit 125.

  In the present embodiment, the three-dimensional positions with a short distance are merged. However, the merging method is not limited to this method, and the three-dimensional position to be merged can be determined according to other criteria. For example, the distance between the candidate of the three-dimensional position at the time of interest and the candidate of the three-dimensional position at the past time can be used. Further, the degree of difference between the attribute of the object corresponding to the candidate of the three-dimensional position at the time of interest and the attribute of the object corresponding to the candidate of the three-dimensional position at the past time can also be used. Based on at least one of these, the candidate of the three-dimensional position at the time of interest and the candidate of the three-dimensional position at the past time can be merged. For example, three-dimensional positions with similar attributes can be merged, or the three-dimensional positions to be merged can be determined based on the distance between the three-dimensional positions and the difference in attributes. Here, the object corresponding to the candidate of the three-dimensional position at the current time or the past time may be an object (tracking result) on each frame image referred to when generating the candidate of the three-dimensional position. On the other hand, the object corresponding to the candidate of the three-dimensional position at the past time is selected for each frame image as the object whose integration corresponding accuracy corresponding to the candidate of the three-dimensional position is equal to or more than the predetermined value and whose integration corresponding accuracy is the highest. It may be an object (tracking result).

  According to step S405, a plurality of real image candidates that may be the same person can be narrowed down to one. This processing is expected to improve the accuracy of specifying the real image. Further, by removing redundant three-dimensional positions, an effect of reducing the amount of calculation is expected.

  In step S406, the accuracy calculation unit 109 calculates the accuracy of the correspondence between the three-dimensional position and the tracking result as described above. Details of step S406 will be described with reference to the flowchart of FIG. In the present embodiment, the correspondence accuracy is calculated for each of the three-dimensional positions j obtained by merging the three-dimensional position candidates at the time of interest with the three-dimensional position candidates at the past time. However, the corresponding accuracy may be calculated for each of the three-dimensional position candidates at the time of interest or the three-dimensional position candidates obtained by merging a plurality of three-dimensional position candidates at the time of interest.

  In step S801, the distance calculation unit 110 projects each of the three-dimensional positions j onto each camera image as shown in FIG. 9A. Next, in step S802, the distance calculation unit 110 calculates the distance on the image between the three-dimensional position j projected on the camera image and the tracking result i, as shown in FIG. 9B. For example, consider a case where persons A and B are tracked on camera image 1 and persons C and D are tracked on camera image 2. In this case, first, the distance between the three-dimensional position j projected on the camera image 1 and the tracking result of the person A and the distance between the three-dimensional position j projected on the camera image 1 and the tracking result of the person B are calculated. Is done. The distance between the three-dimensional position j projected on the camera image 2 and the tracking result of the person C, and the distance between the three-dimensional position j projected on the camera image 2 and the tracking result of the person D are also calculated. Is done. Further, these calculations are performed for each three-dimensional position j.

  In step S803, the difference calculation unit 111 calculates the difference between the attribute of the three-dimensional position j and the attribute of the tracking result i. Here, the attribute of the tracking result i indicates an attribute extracted from the person detection area corresponding to the tracking result i. In the present embodiment, the attribute of the three-dimensional position j and the attribute of the tracking result i indicate color attributes. Further, in the present embodiment, the difference between the color attribute of the three-dimensional position j and the color attribute of the tracking result i is defined as the RGB value of the color attribute of the three-dimensional position j and the RGB value of the color attribute of the tracking result i. Is used. In the above case where the persons A and B are tracked on the camera image 1 and the persons C and D are tracked on the camera image 2, the three-dimensional position j and each of the persons A, B, C and D The difference between is calculated. Further, these calculations are performed for each three-dimensional position j.

  In step S804, the local likelihood calculation unit 112 obtains a local correspondence likelihood indicating the likelihood that the three-dimensional position j corresponds to the tracking result i. At this time, the distance between the position of the object corresponding to the candidate of the three-dimensional position at the time of interest on the frame image at the time of interest and the position of the object, the attribute of the object corresponding to the candidate of the three-dimensional position at the time of interest, and the object And / or the degree of difference from the attribute. As a specific example, the local likelihood calculation unit 112 can calculate the local correspondence accuracy based on at least one of the distance calculated in step S802 and the attribute difference calculated in step S803. Here, the object corresponding to the three-dimensional position candidate may be an object on each frame image referred to when generating the three-dimensional position candidate.

For example, the local probability calculation unit 112 calculates the local corresponding likelihood s ^t _{i, j} on the basis of the distance calculated in step S802, the in the dissimilarity of the computed attributes in step S803. The local likelihood calculation unit 112 can calculate the local correspondence accuracy such that the smaller the distance and the difference, the larger the local correspondence accuracy. As a specific example, the local accuracy calculation unit 112 obtains a weighted average of the distance calculated in step S802 and the difference calculated in step S803 using an arbitrary weight, and is further obtained as the local correspondence accuracy. The reciprocal of the weighted average can be determined. In the present embodiment, the local likelihood calculation unit 112 obtains the local correspondence accuracy from both the distance and the difference, but the local correspondence accuracy may be obtained using only one of them. In step S804, the local correspondence accuracy is calculated between the three-dimensional position j and each of the persons A, B, C, and D. Further, these calculations are performed for each three-dimensional position j. As described above, the local correspondence accuracy is obtained based on the frame image at the time of interest.

Integrating accuracy calculating section 113 in step S805, the current integration corresponding accuracy of the three-dimensional position j the tracking result i, topical corresponding likelihood s ^t _i, and _j calculated in step S804, the integrated response probability a determined previously ^{It is calculated from t-1} _{i, j} . The current integration correspondence accuracy a ^t _{i, j} thus obtained corresponds to the integration correspondence accuracy calculated using the frame image at the time t of interest. Further, the integration-related accuracy a ^t-1 _{i, j} obtained in the past indicates the integration-related accuracy calculated using the frame image at the time t-1.

上式においてｗは更新重みを表し、通常０＜ｗ＜１である。 Integrating accuracy calculating section 113, the local response accuracy s ^t _{i, j,} and the more so the integration corresponding accuracy of the current increases past accumulated corresponding accuracy a ^t-1 _{i, j} is large, the present integration corresponding accuracy a ^t _{i , j} can be obtained. For example, the integrated probability calculation unit 113, the present integration corresponding accuracy a ^t _i, as _j, using any of the weights of the local corresponding likelihood s ^t _{i, j} and past accumulated corresponding accuracy a ^t-1 _{i, j} A weighted average can be determined. As a specific example, there is a method using the following formula.

In the above equation, w represents the update weight, and is usually 0 <w <1.

In step S _< b> 805, the integration accuracy calculation unit 113 further records the current integration correspondence accuracy a ^t _{i, j} having a value equal to or greater than the threshold value in the integration accuracy storage unit 114. Integrating corresponding accuracy by using a frame image at time t + 1 in the next step S407 a ^{t + 1} _i, when calculating the _j is thus recorded accumulated corresponding accuracy a ^t _{i, j} is a past cumulative corresponding Accuracy Referenced. On the other hand, when the current integration correspondence accuracy is less than the predetermined value, the integration accuracy calculation unit 113 does not record the integration correspondence accuracy. Integrating corresponding accuracy a ^t _i between the three-dimensional position j the tracking result _i, if _j is less than the threshold, the accumulated probability calculation unit 113 and those thin relation to the three-dimensional position j and tracking result i Judge and discard without adding the integration correspondence accuracy a ^t _{i, j} .

In step S407, for each three-dimensional position j, the position determination unit 115 uses the integrated correspondence accuracy a ^t _{i, j} between the three-dimensional position j and the tracking result i to obtain a real image reliability L _j representing the likelihood of a real image. Ask for. The position determination unit 115, based on the real image reliability L _j, to determine the three-dimensional position j as a three-dimensional position representing the real image. Specifically, the position determination unit 115 determines some of the three-dimensional positions j as three-dimensional positions representing the real image so that the real image reliability of the three-dimensional position j determined as the real image is increased. Then, the position determination unit 115 allocates a corresponding tracking result to the three-dimensional position determined as the real image, and records the correspondence between the three-dimensional position j determined as the real image and the tracking result i. FIG. 10 shows the details of step S407. In the process of FIG. 10, steps S1001 to S1004 are repeatedly executed until it is determined in step S1004 that the process is not continued.

Step reliability calculation unit 116 in S1001 calculates the real reliability L _j representing the real image likeness of the three-dimensional position j. Reliability calculation unit 116, integrating corresponding accuracy a ^t _i, so _j becomes higher real reliability L _j as have higher tracking result i, calculate the real reliability L _j of the three-dimensional position j I do. Specific examples include the following method. The following processing is performed using all three-dimensional positions j not determined as a real image in step S1003 and all tracking results i not determined as tracking results corresponding to the real image in step S1003.

First, the reliability calculating unit 116 obtains, for each of the three-dimensional position candidates at the time of interest, the integrated correspondence accuracy recorded corresponding to the object tracked on one frame image group. For example, the reliability calculation unit 116 searches for a tracking result i with the highest integrated correspondence accuracy a ^t _{i, j} for each camera image.

  Here, it is assumed that the tracking result i on each camera image corresponding to one three-dimensional position j is one. For example, in the above case where the persons A and B are tracked on the camera image 1 and the persons C and D are tracked on the camera image 2, the three-dimensional position j has not yet been determined as a real image. The case will be described. In this case, the reliability calculation unit 116 selects an object having the highest integrated correspondence accuracy among one or more objects tracked on the frame image group. Then, as the integration correspondence accuracy for the frame image group, the integration correspondence accuracy recorded corresponding to the selected object is selected. For example, of the person A and the person B, the tracking result of the person whose integration correspondence accuracy with the three-dimensional position j is larger is selected as the tracking result on the camera image 1. In addition, a tracking result of a person having a larger integration correspondence accuracy between the three-dimensional position j and the person C and the person D is selected as a tracking result on the camera image 2.

Then, the reliability calculation unit 116 obtains the real image reliability of the candidate of the three-dimensional position according to the integration correspondence accuracy acquired for each of the plurality of frame image groups. For example, the reliability calculation unit 116, a real image reliability L _j of the three-dimensional position j, summing the integration corresponding accuracy between the tracking result selected for each of the camera images and the three-dimensional position j. Specifically, the real image reliability L _{j at} the three-dimensional position j can be expressed by the following equation.

In the above equation, A _j is a set of tracking results corresponding to the three-dimensional position j. Maximum Since tracking result i on each camera image is one, the tracking result contained in A _j is the most one per one camera 1, the A _j number of elements corresponding to one of the three-dimensional position j The value is the number of cameras. As described above, the real image reliability _Lj is obtained by temporarily associating the tracking result i with each of the three-dimensional positions j such that the real image reliability _Lj becomes higher. As described above, the reliability calculation unit 116 obtains the real image reliability L _j for each of the three-dimensional position j.

In step S1002, the position selection unit 117 determines that the candidate of the three-dimensional position having higher real image reliability corresponds to the three-dimensional position of the object at the time of interest. For example, the position selecting unit 117, real reliability L _j obtained in step S1001 to select the three-dimensional position j with the maximum. The selected three-dimensional position j is determined as a three-dimensional position representing a real image. The three-dimensional position thus determined is not selected in the subsequent step S1001.

  The position selection unit 117 selects a candidate of the three-dimensional position having the highest real image reliability, and a candidate of the three-dimensional position in which the object corresponding to the candidate of the three-dimensional position exists in a predetermined number or more, and selects the object at the time of interest. Can be determined to correspond to the three-dimensional position. Here, the predetermined number is not particularly limited as long as it is two or more. If there is only one object corresponding to the candidate for the three-dimensional position, that is, if there is only one frame image showing the object corresponding to the candidate for the three-dimensional position, the three-dimensional position may not be calculated accurately. High in nature.

  In step S1003, for the three-dimensional position j selected in step S1002, the result selecting unit 118 determines the tracking result selected for each camera image in step S1001 as the tracking result corresponding to the three-dimensional position j. However, in another embodiment, the tracking result used to generate the three-dimensional position j may be determined as the tracking result corresponding to the three-dimensional position j. As described above, the object corresponding to the three-dimensional position candidate may be an object on each frame image referred to when generating the three-dimensional position candidate. Further, the object corresponding to the candidate of the three-dimensional position is an object whose integration corresponding accuracy corresponding to the candidate of the three-dimensional position is equal to or more than a predetermined value and which is selected for each frame image as an object having the highest integration corresponding accuracy. Is also good.

  In step S1003, exclusion processing is further performed, and the selected three-dimensional position candidate and the object (tracking result) corresponding to the three-dimensional position candidate are excluded from selection in the selection processing in step S1002. . For example, the tracking result thus determined is not selected in the subsequent step S1002.

  In step S1004, the reliability calculation unit 116 determines whether or not there is an undetermined three-dimensional position and a tracking result that can be associated. If there is, the process returns to step S1001. If not, the process in FIG. 10 ends.

  For example, in step S805, a combination of a three-dimensional position having a high integration correspondence accuracy and a tracking result is recorded. On the other hand, the three-dimensional position can be obtained with high accuracy because the tracking result with high correspondence accuracy with the three-dimensional position exists in two or more camera images. From such a viewpoint, when the number of camera images for which a tracking result with high correspondence accuracy exists for a three-dimensional position is 1 or less, it is determined that the tracking result cannot be associated with the three-dimensional position. Can be. If the tracking results cannot be associated with all the three-dimensional positions, it can be determined that there is no unidentified three-dimensional position and tracking result that can be associated.

  In the above steps S1001 to S1004, a selection process for selecting and confirming the three-dimensional position at which the real image reliability is maximum, and for determining the tracking result corresponding to the three-dimensional position, the selected three-dimensional position and the tracking result And an exclusion process for excluding are repeated. As described above, by searching for a combination of tracking results assigned to the three-dimensional position, the reliability of the real image of the three-dimensional position determined as the real image can be increased, and thus the accuracy of matching between the three-dimensional position and the tracking result is improved. The effect is expected. Performing such repetitive processing is advantageous in that a three-dimensional position representing a real image can be selected with high reliability even if the integration correspondence accuracy at the past time is not considered. On the other hand, the method of comparing the three-dimensional position with the tracking result is not particularly limited as long as the real image reliability of the three-dimensional position determined as the real image increases. In this embodiment, a combination is searched for by a greedy method, but another combination search method may be used. Further, a combination of tracking results assigned to the three-dimensional position may be searched by a brute force or another search method so that the sum of the real image reliability of each three-dimensional position is increased.

  As described above, in step S406, by comparing the three-dimensional position with the tracking result, the integrated correspondence accuracy, which is an index indicating that these are the same object, is calculated. Further, in step S407, it is determined whether or not the three-dimensional position is a real image using the integration correspondence accuracy. By these processes, the three-dimensional position, which is a virtual image, can be effectively reduced, so that improvement in the accuracy of three-dimensional position estimation can be expected. In particular, when calculating the integrated correspondence accuracy, reference is made to the past correspondence accuracy of a plurality of frames in addition to the local correspondence accuracy obtained based on the frame image at the time of interest t. For this reason, since the three-dimensional position based on the past long-term frame image can be compared with the tracking result, the effect of effectively reducing the virtual image is expected.

  Further, according to the present embodiment, the integration correspondence accuracy is recorded in association with the combination of the three-dimensional position and the tracking result, and the three-dimensional position and the tracking result are collated using the recorded integration correspondence accuracy. Therefore, it is not essential to refer to the past camera image itself, and it is not essential to store the position of the tracking result obtained in the past object tracking, the 2D tracking label, and the like, and refer to each frame. According to such a configuration, an effect of reducing the storage area size and an effect of reducing the amount of calculation are expected.

  Further, in step S406, the three-dimensional position is compared with the tracking result using both the distance between the three-dimensional position projected on the camera image and the tracking result on the image and the degree of difference between the attributes. Was. According to such a configuration, even when it is difficult to perform matching only by using either the distance or the difference between the attributes, it is possible to perform effective matching by using both. However, it is not essential to use both the distance and the difference between the attributes, and the matching may be performed using one of them, or another additional information may be used.

In step S407, the real image reliability L _j is calculated as the sum of the corresponding probabilities a ^t _{i, j} . Therefore, real reliability as the number of tracking results corresponding to the three-dimensional position L _j becomes larger. As described above, in one embodiment, the real image reliability of the candidate of the three-dimensional position is obtained such that the larger the number of frame images in which the integration correspondence accuracy is acquired, the larger the real image reliability. In a multi-view geometry, there is a property that the ambiguity of the association is reduced as more objects are reflected in more cameras. In the method of the present embodiment, since the real image reliability _Lj is calculated so as to increase as the number of tracking results corresponding to the three-dimensional position increases, the third order associated with the tracking results on more camera images The original position is easily determined as a real image, and an effect of improving accuracy is expected.

  In step S408, the position update unit 119 determines the position of each object corresponding to the candidate of the three-dimensional position determined to correspond to the three-dimensional position of the object at the time of interest on each frame image, and the positional relationship between the plurality of imaging units. , The three-dimensional position of the object is obtained. For example, the position updating unit 119 can obtain the coordinates of the three-dimensional position using the correspondence between the three-dimensional position and the tracking result obtained in step S407. The position updating unit 119 can further update the candidate of the three-dimensional position at the time of interest with the three-dimensional position of the object determined to correspond. The updated three-dimensional position candidates can be used in the merging process in the next step S405.

  For example, the coordinate updating unit 120 can calculate the coordinates of the three-dimensional position again using the tracking result on each camera image corresponding to the three-dimensional position. Here, the coordinates of the three-dimensional position are calculated using the coordinates of the tracking result detected from the frame image at the time point of interest t. Specifically, similarly to step S502, the coordinates can be obtained by the principle of triangulation based on the coordinates of the representative point of the tracking result on the camera image and the coordinates of the camera center. When calculating the coordinates of the three-dimensional position using the tracking results on three or more camera images, a straight line in the three-dimensional space connecting the coordinates of the camera center and the coordinates of the representative point of the tracking results on the camera images And the coordinates closest to these straight lines can be obtained. As a specific example, the coordinates at which the sum of the squares of the distances from these straight lines is the smallest can be obtained as the coordinates of the three-dimensional position corresponding to the tracking result on the camera image.

  Further, the attribute updating unit 121 can update the attribute of the three-dimensional position based on the attribute of the tracking result. For example, an attribute of a three-dimensional position can be obtained based on a pixel value of a tracking result on each camera image. As a specific example, the average value of the pixel values of the tracking result on each camera image can be obtained as the attribute of the three-dimensional position.

  As described above, the coordinates and attributes of the three-dimensional position candidates recorded in the position storage unit 123 are updated by the processing in step S408. The updated coordinates and attributes of the three-dimensional position candidate can be used in the merging process in step S405. However, it is not always necessary to update the coordinates and attributes of the three-dimensional position candidates recorded in the position storage unit 123.

  In step S409, the position deletion unit 122 deletes an unnecessary three-dimensional position. For example, the position deletion unit 122 deletes a candidate for a three-dimensional position generated at a past time that has not been determined to correspond to an object for a predetermined period or more and has not been merged with a candidate for a three-dimensional position for a predetermined period or more. be able to. As a specific example, a three-dimensional position that has not been determined as a real image in step S <b> 407 during a period equal to or more than a certain number of frames can be a deletion target. By deleting unnecessary three-dimensional positions, the effect of reducing the storage area and the amount of calculation in steps S405 to S408 can be expected.

  In step S410, the display control unit 126 causes the display device to display the person tracking result and the three-dimensional position estimation result obtained by the above processing. For example, the display control unit 126 can cause the display device to display the three-dimensional position determined as the real image in step S407 and the position of the tracking result on each camera image corresponding to the three-dimensional position determined as the real image. .

  FIG. 11 shows a configuration example of a screen displayed on the display device. The screen shown in FIG. 11 includes one or more camera images 1101 and a three-dimensional map 1104. FIG. 11 shows a display example when four cameras are used, and four camera images 1101 are images captured by the respective cameras.

  A symbol (frame) indicating the tracking result in step 403 is superimposed on the camera image 1101. In the present embodiment, a frame indicating a tracking result on each camera image corresponding to the three-dimensional position determined as a real image in step S407 is superimposed on each camera image 1101. Frames indicating the same person are displayed in the same color in different camera images so that the user can easily identify whether or not the same person appears in different cameras.

  In the three-dimensional map 1104, a symbol 1103 indicating the three-dimensional position of the person acquired from the position storage unit 123 and a symbol 1102 indicating the position and orientation of the camera are displayed as a three-dimensional image together with the floor surface. In the present embodiment, the three-dimensional position determined as the real image in step S407 is displayed on the three-dimensional map 1104. A frame indicating a tracking result on the camera image 1101 and a three-dimensional map on the three-dimensional map 1104 so that the user can easily identify whether the person on the camera image 1101 and the person on the three-dimensional map 1104 are the same. The symbol indicating the position is displayed in the same color.

  In step S405, information indicating that the merged three-dimensional position represents the same person is stored in the correspondence storage unit 125. As described above, when information indicating that some three-dimensional positions represent the same person is stored in the correspondence storage unit 125, the three-dimensional position is determined to represent that the three-dimensional positions represent the same person. The original map 1104 can be displayed. For example, when some three-dimensional positions are merged in step S405, a person corresponding to the three-dimensional position before merging is placed near a symbol representing a person corresponding to the three-dimensional position merged in step 405. A mark of the same color as the symbol to be displayed can be displayed. Thus, by providing the correspondence storage unit 125, it is possible to indicate that different three-dimensional positions represent the same person. Such a configuration can be effectively applied to, for example, a case where a long-term three-dimensional trajectory of a person is calculated by video analysis processing or a case where a person is identified, in addition to a case where a tracking result is simply displayed.

  In the example of FIG. 11, each person is displayed in a different color to express whether or not the person is the same. In another example, a number, a character, a symbol, or the like unique to a person may be superimposed on the camera image 1101 or the three-dimensional map 1104 in order to enhance visibility. Further, in the example of FIG. 11, the three-dimensional position is displayed in the three-dimensional space using the three-dimensional map 1104, but the three-dimensional position can be displayed on the two-dimensional map. When the three-dimensional position is represented on the three-dimensional map, the position of the person in the height direction can be known from the three-dimensional map in addition to the position of the person on the floor surface. In particular, when the position of a person is determined based on the head, such as when object tracking is performed based on face recognition, the height of the person can be known from the three-dimensional map. On the other hand, when the three-dimensional position is represented on the two-dimensional map, there is an effect that the position of the person on the floor is easily recognized.

  In addition, the viewpoint of the three-dimensional map 1104 may be fixed, or a mechanism that allows the user to change the viewpoint may be provided. What can be displayed on the three-dimensional map 1104 is not limited to the above. For example, a floor plan showing the arrangement of furniture and the like may be superimposed on the floor, or another object may be superimposed. By doing so, the positional relationship between the person and the object around the person can be presented in an easily understandable manner.

  In step S411, the image acquisition unit 101 determines whether to continue the processing. If a new frame image can be acquired from the camera, the process returns to step S402, and the process is repeated using the newly acquired frame image. On the other hand, when a new frame image cannot be obtained from the camera, or when there is a user instruction indicating the end, the processing in FIG. 4 ends.

上式において、αは定数である。このような構成によれば、実像を示す三次元位置をより正確に選択し、虚像を削減する効果が見込まれる。 When obtaining the real reliability L _j is accumulated corresponding accuracy a ^t _i, can refer to parameters other than _j. For example, in the present embodiment, a three-dimensional position that has not been determined as a real image for a long time is deleted. Therefore, a longer existence time of the three-dimensional position means that the tracking result is frequently associated, that is, the likelihood of being a real image is high. Therefore, as the three-dimensional position longer real reliability L _j is the presence time of j is increased, it may be obtained real image reliability L _j. In one embodiment, the real image reliability of the three-dimensional position candidate is determined so that the more the three-dimensional position candidate generated at an earlier time is merged, the higher the real image reliability becomes. The fact that the three-dimensional position j is merged with the candidate of the three-dimensional position generated at an earlier time means that the existence time of the three-dimensional position j is long. For example, assuming that the existence time of the three-dimensional position j is T _j , the real image reliability L _j can be obtained as in the following equation.

In the above equation, α is a constant. According to such a configuration, an effect of more accurately selecting a three-dimensional position indicating a real image and reducing a virtual image is expected.

  In the present embodiment, the integrated correspondence accuracy is calculated as the sum of the local correspondence accuracy and the past integrated correspondence accuracy, and the real image reliability is calculated as the sum of the integrated correspondence accuracy. On the other hand, the integrated correspondence accuracy may be calculated as the product of the local correspondence accuracy and the past integrated correspondence accuracy, and the real image reliability may be calculated as the product of the integrated correspondence accuracy. In this case, the integration correspondence accuracy and the real image reliability can be calculated as probabilities instead of likelihoods.

(Other Examples)
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions.

101: image acquisition unit, 102: object tracking unit, 104: position generation unit, 109: accuracy calculation unit, 115: position determination unit, 123: position storage unit, 124: position merging unit

Claims (18)

Acquisition means for acquiring a group of continuously captured frame images from each of a plurality of imaging units having overlapping visual fields,
Tracking means for tracking an object on the frame image group;
Generating means for generating a candidate for a three-dimensional position of the object at a time of interest based on a position on the frame image of the object and a positional relationship between the plurality of imaging units;
Associating means for associating the candidate of the three-dimensional position at the time of interest with the candidate of the three-dimensional position of the object at a past time before the time of interest;
Based on the local correspondence accuracy indicating the probability that the candidate of the three-dimensional position at the time of interest corresponds to the object, and the integration correspondence accuracy indicating the probability that the candidate of the three-dimensional position at the past time corresponds to the object, Calculation means for calculating an integrated correspondence accuracy indicating a probability that the candidate of the three-dimensional position at the time of interest corresponds to the object,
For each of the three-dimensional position candidates at the time of interest, the integration correspondence accuracy recorded corresponding to the object tracked on one frame image group is obtained, and for each of the plurality of frame image groups, Determining the reliability of the candidate of the three-dimensional position according to the acquired integration correspondence accuracy, determining means for determining the three-dimensional position of the object at the time of interest based on the reliability ,
An image processing apparatus comprising: The determining unit selects an object having the highest integration correspondence accuracy among one or more objects tracked on the frame image group, and determines the selected object as the integration correspondence accuracy for the frame image group. The image processing apparatus according to claim 1 , wherein the integration correspondence accuracy recorded in correspondence with ( 1 ) is selected. The calculating means, the integration if the corresponding probability is less than a predetermined value, characterized in that to suppress the recording of the integrated response accuracy, the image processing apparatus according to claim 1 or 2. The method according to claim 1, wherein the determining unit obtains the reliability of the candidate of the three-dimensional position so that the reliability increases as the number of the frame image groups from which the integration correspondence accuracy is acquired increases. 4. The image processing device according to 3 . The associating unit merges the candidate for the three-dimensional position at the time of interest and the candidate for the three-dimensional position at the past time, so that the candidate for the three-dimensional position at the time of interest and the three-dimensional position at the past time are combined. characterized in that associating the position of the candidate, the image processing apparatus according to any one of claims 1 to 4. The said determination means calculates | requires the reliability of the said three-dimensional position candidate so that the said reliability may become large, so that it may be merged with the candidate of the three-dimensional position produced | generated at the older time. Item 6. The image processing device according to Item 5 . The associating means includes: a distance between the candidate of the three-dimensional position at the time of interest and the candidate of the three-dimensional position at the past time; an attribute of the object corresponding to the candidate of the three-dimensional position at the time of interest; Merging the three-dimensional position candidate at the time of interest and the three-dimensional position candidate at the past time based on at least one of the degree of difference with the attribute of the object corresponding to the three-dimensional position candidate at the time. wherein the image processing apparatus according to claim 5 or 6. The associating unit, when merging two or more three-dimensional position candidates at the past time determined to correspond to different objects, respectively, generates information indicating that the two or more three-dimensional positions correspond to the same object. wherein the recording, the image processing apparatus according to any one of claims 5 to 7. The associating unit deletes the candidate of the three-dimensional position generated at a past time that is not determined to correspond to the object for a predetermined period or more and is not merged with the candidate of the three-dimensional position for a predetermined period or more. characterized by, the image processing apparatus according to any one of claims 5 to 8. The determining means includes:
A candidate for the three-dimensional position having the highest reliability, and a candidate for a three-dimensional position where an object corresponding to the candidate for the three-dimensional position is present for a predetermined number or more is selected. A selection process for determining that it corresponds to the position;
The selected three-dimensional position candidate and an object corresponding to the three-dimensional position candidate, an exclusion process of excluding from the target in the selection process,
And repeating the image processing apparatus according to any one of claims 1 to 9. The calculating means, the distance between the position of the object being tracked on the frame image in the position and the time of interest on the frame images corresponding to the candidate of the three-dimensional position of the object in the time of interest the local and obtaining the corresponding accuracy, the image processing apparatus according to any one of claims 1 to 10 based on. The local correspondence accuracy is calculated based on a difference between an attribute of an object corresponding to a candidate of the three-dimensional position of the object at the time of interest and an attribute of an object tracked on a frame image at the time of interest. and obtaining the image processing apparatus according to any one of claims 1 to 11. The object corresponding to the candidate of the three-dimensional position is an object on each frame image referred to when generating the candidate of the three-dimensional position, and the integrated correspondence accuracy corresponding to the candidate of the three-dimensional position is a predetermined value or more. 13. The image processing according to any one of claims 7 to 12 , wherein the object is selected for each frame image as the object having the highest integration correspondence accuracy. apparatus. Based on the position on each frame image of the object corresponding to the candidate for the three-dimensional position determined to correspond to the three-dimensional position of the object at the time of interest, based on a positional relationship between the plurality of imaging units, and further comprising means for determining the three-dimensional position of an object, an image processing apparatus according to any one of claims 1 to 13. 15. The image processing apparatus according to claim 14 , further comprising: means for updating a candidate of a three-dimensional position at the time of interest with a three-dimensional position of the object determined to correspond. The said integration correspondence accuracy is the parameter | index which shows the possibility that the object which the said candidate of the three-dimensional position shows is the same as the said tracked object, The said any one of Claim 1 thru | or 15 characterized by the above-mentioned. An image processing apparatus according to claim 1. An image processing method performed by the image processing apparatus,
An acquisition step of acquiring a group of continuously captured frame images from each of a plurality of imaging units having overlapping visual fields,
A tracking step of tracking an object on the frame image group;
A generation step of generating a candidate for a three-dimensional position of the object at a time of interest based on a position on the frame image of the object and a positional relationship between the plurality of imaging units;
An associating step of associating the candidate of the three-dimensional position at the time of interest with the candidate of the three-dimensional position of the object at a past time before the time of interest;
Based on the local correspondence accuracy indicating the probability that the candidate of the three-dimensional position at the time of interest corresponds to the object, and the integration correspondence accuracy indicating the probability that the candidate of the three-dimensional position at the past time corresponds to the object, A calculation step of obtaining an integrated correspondence accuracy indicating a probability that the three-dimensional position candidate at the time of interest corresponds to the object,
For each of the candidates for the three-dimensional position at the time of interest, the integration correspondence accuracy recorded corresponding to the object tracked on one frame image group is obtained, and for each of the plurality of frame image groups, Determining a reliability of the candidate of the three-dimensional position according to the acquired integration correspondence accuracy, a determining step of determining a three-dimensional position of the object at the time of interest based on the reliability ,
An image processing method comprising: Program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 16.

Images