JP2007233523A - Person location estimation method using asynchronous camera image and system therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To construct a system for estimating a location of a person in a space at a low cost by using an image photographed by a camera. <P>SOLUTION: The same person is photographed by two or more asynchronous cameras, and continuous adjacent frames photographed by a reference camera are selected, and a base camera and base frame photographed between the adjacent frames are selected, and the person region in each frame image is estimated, and the top of the head part of the person is estimated from each person region, and the three-dimensional top of the head position of the person is estimated by using the position information of each of the top of the head parts of the adjacent frames and the base frame. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for estimating the position of a person using a plurality of asynchronous camera videos.

  In recent years, there has been an increasing need for surveillance systems due to increased awareness of safety and lower prices for cameras. In addition to the purpose of monitoring, there is an increasing need to improve services for customers and expand sales by collecting position information of people using cameras.

  For such a purpose, it is necessary to acquire a wide range of information. However, in order to acquire a wide range of information, it is impossible to cover with only one camera, and three-dimensional In order to obtain accurate position information, it is necessary to shoot with a multi-camera. In order to acquire position information of people using multiple cameras, it is generally necessary to synchronize each camera. However, exact synchronization is difficult or the system becomes expensive. There is a problem of being fooled.

  Therefore, as a method for estimating the position of a person from an image of one camera without using a plurality of cameras in order to eliminate the need for camera synchronization, the possibility of the assumed person's height as in Patent Document 1 is assumed. A technique for estimating a three-dimensional position of a person from one camera using a distribution is disclosed.

  Non-Patent Document 1 describes a technique for geometrically obtaining three-dimensional position information by interpolating position information captured by two cameras without using synchronized camera images. ing.

JP 2004-302700 A S. Shimizu, et al, “A Pseudo Stereo Vision Method for Unsynchronized Cameras”, ACCV2004, pp.363-368 (2004).

  First, in the method described in Patent Document 1, the accuracy of the possibility distribution of height can be expected to some extent at a specific place and time, but it is considered that the accuracy is lowered in a general environment. In addition, when a three-dimensional position is to be obtained from information on a person captured by a plurality of cameras, two cameras that have been photographed synchronously are used in a conventional matching method using characteristic information such as color and shape. When identifying the same person in one camera and the same person in another camera, it is not possible to recognize that the person is the same person when the method of shooting differs depending on the camera direction. Exists.

  In addition, as described above, it is costly to take a completely synchronized image with a plurality of cameras, and it is desired to use a plurality of asynchronous cameras. However, in the technique described in Non-Patent Document 1, There is one that is reflected in the camera, and it is premised that the correspondence is clear, and there is a problem that it is difficult to determine the same person in the asynchronous camera.

  The present invention is characterized by a technique for estimating the position of a person in a three-dimensional space using a plurality of asynchronous cameras. More specifically, the present invention inputs a frame image taken by each camera and its shooting time into a memory, and specifies two consecutive frames taken by the first camera as a front frame and a rear frame. A frame image taken by a second camera different from the first camera and taken at a time between the previous frame and the rear frame and closest to the rear frame is taken as a reference frame. The area in which the person is reflected is estimated from the images of the previous frame, the rear frame, and the reference frame, the head of the person is estimated from each of the estimated person areas, and the front frame, the rear frame, and the reference frame are estimated. Is used to estimate the three-dimensional top position of the person and display the position of the person on the floor based on the position information.

  Further, according to one embodiment of the present invention, the moving speed is obtained from the difference in the position of the person who has moved during the shooting time interval, and the three-dimensional top position determined for the reference frame immediately before the reference frame. And an error between the three-dimensional top position corresponding to the current reference frame obtained from the moving speed and the currently estimated three-dimensional top position, and using the error, the current three-dimensional position is calculated. The possibility that a person exists in the vicinity of the top position is obtained.

  According to the present invention, even when a wide range of shooting is performed using a plurality of cameras, an inexpensive asynchronous camera can be used, and the person position can be estimated at a low cost. In addition, according to the present invention, there is an effect that an approximate position of a person can be estimated and confirmed although there is an error. Furthermore, by using the position of the top of the head, since it is less likely that an image cannot be acquired with an obstacle such as a table or a shelf, compared to the case of using the information at the foot, it is possible to obtain the person position stably.

  Assuming that position information of a person such as a customer is acquired at a retail store or the like, a method for estimating the position of a person in a building using a plurality of asynchronous cameras will be described with reference to the following embodiments.

  FIG. 1 is a system configuration diagram of an embodiment for realizing the present invention. Images taken by the wired camera group 1 of the wired camera 1-m from the wired camera 1-1 as a plurality of cameras that capture the space are sent to the recording device 9 via the network 7 and stored in the recording device 9. Also, the camera may be a wireless camera, and the video captured by the wireless camera group 3 of the wireless camera 3-n from the wireless camera 3-1 is sent to the recording device 9 via the network 7 via the wireless access point 5 for recording. Accumulate in device 9. A plurality of wireless access points 5 may exist. The image processing device 11 performs person position estimation using the captured image stored in the recording device 9 and displays the result on the display device 13. The recording device 9, the image processing device 11, and the display device 13 may be integrated as one computer.

  The internal configuration of the image processing apparatus 11 will be further described with reference to FIG. The image processing apparatus 11 is a computer including a processor and a memory, and includes an image memory 15, a camera parameter storage unit 17, a control unit 19, and an input / output unit 26. The image memory 15 and the camera parameter storage unit 17 are provided on the memory. The control unit 19 includes programs for the reference / reference camera selection unit 20, the human region estimation unit 21, the top position estimation unit 22, the three-dimensional top position estimation unit 23, the movement amount estimation unit 24, and the existence possibility estimation unit 25. Have The control unit 19 executes these programs stored in the memory by the processor. The memory of the control unit 19 temporarily stores input data and processing results for these programs. The image processing device 11 is connected to the recording device 9, the display device 13 and the input device 27. The input / output unit 26 controls input / output of data between these devices and the memory of the image processing device 11.

  The image processing device 11 acquires the image data recorded in the recording device 9 via the input / output unit 26 and stores it in the image memory 15. The camera parameter storage unit 17 stores internal parameters such as the angle of view and the number of pixels obtained in advance and external parameters such as the position of the camera in space. The field angle is a viewing angle that can be captured by the camera, and includes a horizontal field angle and a vertical field angle. The control unit 19 executes the stored program to input each frame image accumulated in the recording device 9 and its shooting time into the image memory 15 to temporarily store the frame image and store it in the camera parameter storage unit 17. The person image in the frame is analyzed using information or the like, and the person position in the three-dimensional space is estimated. Each frame image is appended with the identifier of the camera that captured the image.

  Next, with respect to an example in which the position of a person is estimated using the configuration of FIG. 1, assuming an environment in which the room is photographed with three cameras as shown in FIG. It explains using. FIG. 4 is a plan view showing the camera position and shooting range when the room is viewed from above. A space not photographed by camera A (30A in the figure), camera B (30B in the figure), and camera C (30C in the figure) is area 33, and a space photographed by only one camera is area 34 and two cameras. A space to be photographed is a region 35, and a space photographed by three cameras is a region 36. It is assumed that the position, direction, and angle of view of each camera have been examined in advance, and information about the overlap of the shooting areas when three cameras are combined is known in advance. It is also assumed that information such as the shape and size of the space to be photographed is also known. Here, when it is assumed that the room is to be photographed, it is desirable to arrange the cameras so that the space near the entrance and the exit is photographed by at least two cameras.

  Assume that the cameras A to C use commercially available web cameras. Although these cameras cannot be synchronized, the time of the internal clock can be set from the outside, and the internal clocks of the cameras A to C are assumed to be the same. And the image image | photographed with each camera is recorded on the video recording apparatus 9 with imaging | photography time. In addition, this camera compresses the captured image and sends the captured image to the recording device 9 via the network 7 and stores it as shown in FIG. 1, but the time required for compression differs depending on the subject to be captured. It is assumed that the shooting time of each frame image of each camera is different. For example, as shown in FIG. 5, the shooting time of the frame image of camera A is 40, 41, the shooting time of camera B is 42, 43, 44, and the shooting time of camera C is 45, 46. To do.

  Based on this assumption, the flow of processing will be described along the flowchart of FIG. First, the control part 19 inputs the command which starts the display of the person position estimation result input from the input device 27 to memory (step S100). The control unit 19 repeats the following person position estimation process until a display end command is input, and ends the process if an end command is input (step S104). The reference / reference camera selection unit 20 refers to the identifier of the camera corresponding to the frame image input to the image memory 15, and two frames of images are continuously recorded in the recording device 9 at the beginning of the person position estimation process. The selected camera is set as a reference camera (step S105). For example, in the case of FIG. 5, if a display start command is input at the time of shooting time t-τ1 (42 in the figure), refer to the camera B in which images for two frames are first stored in the recording device 9. A camera. Then, the reference / reference camera selection unit 20 uses the two frames as the preceding and following frames, and the camera is taken at time t closest to time t + τ2 between time t-τ1 and t + τ2 (43 in the figure). Is a reference camera, and that frame is a reference frame (step S106). For example, in the case of FIG. 5, the camera A is the reference camera, and the frame at time t (40 in the figure) is the reference frame.

  Next, the person area estimation unit 21 refers to the reference frame image and the previous and next frame images in the image memory 15, and estimates an area in which a person is captured from these images (step S112). As an estimation method, an image when a person is not captured may be stored as a background, and a person area may be estimated based on a background difference. However, other methods such as using thermosensor information may be used. Good. Next, the parietal position estimation unit 22 estimates the parietal position from the person area (step S114). The top of the head may be estimated as the highest position in the person area.

  Next, the three-dimensional head position estimation unit 23 estimates a three-dimensional head position from the head position estimated in each frame image (step S116). The method will be described with reference to FIG. The optical center of camera A is C1, the optical center of camera B is C2, and the head of the person in the frame image 51 of camera A taken at time t is taken at point p, and camera B takes the picture at time t-τ1. It is assumed that the top of the person shown in the frame image 52 is the point q and the top of the person shown in the frame image 53 taken at time t + τ2 is the point r. In FIG. 6, each frame image is arranged as a projection plane in a three-dimensional space from the position, direction, and angle of view of the camera. At this time, assuming that the shooting interval is not so long, it is possible to approximate that the height of the person's top position does not change in the meantime, and from the optical center C2 of the camera B, the point q, and the point r It is possible to estimate that the intersection P between the plane and the straight line connecting the optical center C1 of the camera A and the point p is the three-dimensional top position of the person. Here, from the estimated three-dimensional position, the height of the person and the position of the person on the floor (the position where the perpendicular from the top position intersects the floor) are obtained, and the height of the person assumed in advance is, for example, an adult If it is assumed that it is in the range of 150-200cm, it is judged whether it is an area where people can not exist from the information of the shooting space, that is, the size of the room, desk, shelf, etc. If the result seems to be correct, it is adopted as the position of the person, and if the result does not seem correct, it is not adopted.

  From the camera position and angle, it is possible to calculate a 3D principal axis formula that passes through the optical center of the camera, and this principal axis coincides with a straight line in the 3D space that passes through the origin (0,0) that is the center of the frame. Should do. Further, the intersection of the X axis and the end of the frame and the intersection of the Y axis and the end of the frame respectively correspond to a straight line indicating the horizontal field angle and a straight line indicating the vertical field angle. Therefore, the straight line in the three-dimensional space corresponding to an arbitrary point p (X, Y) on the two-dimensional frame photographed by the camera A is an equation of a straight line passing through the optical center C1 according to the proportional distribution of (X, Y). It is expressed by Similarly, the points q and r on the two-dimensional frame photographed by the camera B are also expressed by two straight lines on the three-dimensional space passing through the optical center C2. The intersection of the plane in the three-dimensional space formed by the optical center C2 and these two straight lines and the straight line passing through the optical center C1 is P (x, y, z).

  The three-dimensional head position estimation unit 23 can calculate a formula of a straight line in the three-dimensional space corresponding to the point p (X, Y) using the camera parameters stored in the camera parameter storage unit 17. The three-dimensional head position estimation unit 23 obtains two-dimensional coordinate values of each of the points p, q, and r from the number of pixels of the frame image, and uses each two-dimensional coordinate value using information on the camera position, angle, and angle of view. From the two straight line equations passing through C2, the three-dimensional space equation is obtained, and the coordinate value of the point P is obtained from the straight line equation passing through C1 and this plane equation. . The three-dimensional head position estimation unit 23 advances the calculation while temporarily storing the calculation result in the memory in this manner, and temporarily stores the calculated coordinate value of the point P in the memory.

  If the person does not move and the points q and r on the two-dimensional frame are almost at the same position, the straight line in the three-dimensional space is one, so the straight line from the camera A is closest to the straight line from the camera B. The point may be the position of the point P.

  In the case of the first embodiment, the process proceeds from S116 to S120, and the control unit 19 displays the position of the person on the floor estimated by the above method on the display device 13 (step S120). If there is no person position display end command, the process returns to step S105, and after the time t, two frames of images are recorded in the recording device 9 by the same camera, and the shooting frames of another camera are within the shooting interval of the two frames. Is checked, a reference camera is determined, and then a reference camera and a reference frame are determined in step S106. For example, in the case of FIG. 5, when the frame image at time 41 is acquired, camera A becomes the reference camera, and camera B photographed at time t + τ2 (43 in the figure) closest to time 41 becomes the reference camera. The frame becomes the reference frame.

  According to the first embodiment, the position of a person can be estimated even if a plurality of cameras that are not synchronized are used, and the system can be constructed at low cost.

  In the case of the first embodiment, it is possible to estimate the three-dimensional top position of the person, but this position information includes various errors. For example, there are errors in the optical center position of the camera, the shooting time, the person area estimated from the image, the position of the top of the head, and the like. Therefore, in the second embodiment, in the flowchart of FIG. 3, the step of estimating the moving speed of the person (S117) and the step of estimating the spatial existence possibility of the person using the relationship between the person position, the moving speed, and the time. (S118) is added.

A method of estimating the existence possibility will be described with reference to FIG. First, the three-dimensional top position P of the person at time t is obtained by the method of the first embodiment. Then, the height of the person is obtained, and as described above, assuming that the height does not change because the shooting interval is sufficiently short, the three-dimensional top positions Q and R at time t-τ1, t + τ2 It can be obtained as a point where the distance from the floor surface coincides with the height on each straight line connecting the optical centers C2 and q and C2 and r. Using this result, assuming that constant velocity linear motion is assumed because the interval between the photographing times is sufficiently short, the movement amount estimating unit 24 moves the movement speed (movement direction and speed) from Q to P and from P to R. As an average of the moving speeds, a vector ν _p indicating the moving speed of the point P is obtained (step S117). ν _p has an x component and a y component of velocity, and does not have a z component. The obtained position and movement speed are stored in the memory until the estimation of the presence possibility of the person at the next time is completed.

Next, the existence possibility estimation unit 25 estimates a person's two-dimensional spatial existence possibility (existence probability) in step S118. Assuming that the human position estimated as described above has a Gaussian distribution with the possibility that it exists in a two-dimensional space centered on the point P, the variance σ of the Gaussian distribution is calculated from the human position and the moving speed at the previous time. It shall be proportional to the error from the estimated position. For example, assuming that an evaluation value of the possibility that a person exists at the position P _f (x, y) is F (P _f ) and the neighborhood space of P _f is S, the following is obtained.

ここで点Ｐは、位置座標値のうちｘ成分とｙ成分のみをとるものとし、Ｐ_ｆとＰとの差の絶対値の２乗は、Ｐ_ｆとＰ間の距離の２乗を表すものとする。また時刻tの直前のフレーム時刻t'における人物位置の位置Ｐ’と移動速度としてのベクトルν_ｐ’がメモリに記憶されている場合、数１におけるσは以下に示す数２のように求めるとし、Ｐ’とベクトルν_ｐ’が記憶されていない場合には、デフォルトのσとしてある範囲の値をとる所定値を用いるとする。

Here the point P is assumed to take only the x and y components of the position coordinates, the square of the absolute value of the difference between P _f and P is represents the square of the distance between P _f and P And Further, when the position P ′ of the person position at the frame time t ′ immediately before the time t and the vector ν _{p ′} as the moving speed are stored in the memory, σ in Equation 1 is obtained as shown in Equation 2 below. , P ′ and vector ν _{p ′} are not stored, a predetermined value taking a certain range of values is used as the default σ.

ここでＡは比例定数であり、２つの‖は距離を算出することを示す。σは、点Ｐと前時刻の位置Ｐ’およびその移動速度から推定した位置との距離の差に比例することを示す。

Here, A is a proportionality constant, and two wrinkles indicate that the distance is calculated. σ indicates that it is proportional to the difference in distance between the point P and the position P ′ at the previous time and the position estimated from the moving speed.

In step S120, the existence possibility estimation unit 25 displays a person's existence possibility F (P _f ).

  According to the above method, it is possible to display the human position information including the error by allowing the ambiguity by the spatial existence possibility distribution.

  In the examples so far, it was assumed that a single person was captured in the camera image, so it is limited to use in environments where there are few people or where the angle of view of the camera is narrow. Another embodiment in the case where a plurality of persons who can be applied to is shown will be described with reference to FIGS.

  As shown in FIG. 7, it is assumed that two persons are shown in the frame image 51 of the camera A, and the positions of the tops of the heads are p1 and p2, respectively. Similarly, it is assumed that the frame image 52 of the camera A includes the head top portions q1 and q2 of the person, and the frame image 53 includes the head top portions r1 and r2 of the person. At this time, assuming that it is unknown which individual person is the same person, the possibility of the same top position as p1 is (q1, r1), (q1, r2), (q2 , r1), (q2, r2). Similarly, there are four possibilities for the same top as p2.

  At this time, the three-dimensional top positions P1 and P2 of two persons are obtained for all eight patterns of combinations. As a result, the height and the position on the floor are determined, and a plausible result is adopted. For example, as shown in FIG. 8, the three-dimensional top position P1 (1) of the person obtained when q1 and r2 are adopted as the positions corresponding to p1 is not adopted because it is smaller than the assumed height range. As shown in FIG. 9, the three-dimensional top position P1 (2) when q1 and r1 are used as the position corresponding to p1 is adopted within the assumed height range and the assumed space.

  In S116, the three-dimensional top position estimating unit 23 obtains a position P1 (1) in the three-dimensional space by the above-described processing using the two-dimensional coordinate values of the points p1, q1, and r1. Similarly, the positions P1 (2), P1 (3), and P1 (4) are obtained for the other three patterns. The three-dimensional head position estimation unit 23 obtains each z component as the height for the positions P1 (1) to P1 (4), and stores the coordinate value as a point P1 at the position closest to the height range among these heights. Save to. Similarly, the three-dimensional top position estimating unit 23 obtains positions P2 (1), P2 (2), P2 (3), and P2 (4) in the three-dimensional space for the point P2, and the z component is the tallest. The coordinate value is stored in the memory with a position close to the range as a point P2.

  There is a possibility that there are a plurality of results in the assumed height range and in the assumed space. In such a case, a person who is likely to exist in the shooting space is assumed from the shooting location and time conditions, and the assumed height is defined as a distribution. For example, if it is an exhibition for business, define a distribution that maximizes the average height of adults, and if it is a toy counter, define a distribution that peaks the average height of children and the average height of parents. Good. Then, the most appropriate result may be adopted as the person position from the height information obtained from the three-dimensional top position of the person obtained as described above.

  As described above, according to the present embodiment, it is possible to estimate a person position even if a plurality of persons are captured in a camera image.

  In the third embodiment, when the number of persons appearing in the image increases, the number of combinations to be examined increases exponentially. Therefore, a method for reducing the number of combinations to be examined will be described using the example of FIG.

  First, the top position q1, q2, r1, r2 is obtained in the front and rear frame images of the reference camera. Next, q1 is selected from the images of the previous frame 52 of the reference camera. Here, the maximum moving speed of a person is assumed according to the characteristics of the shooting space, and for example, about 1.7 m / s, which is about a fast pace, is assumed. Then, the movement amount becomes the largest in the photographed image when the person moves in parallel with the projection plane of the camera, and the movement amount L within the frame corresponding to the photographing interval with the frame image 53 can be obtained. . Therefore, for the frame image 53, a combination may be considered only for the top of the head existing within the radius L with the point q1 as the center.

  The three-dimensional head position estimation unit 23 calculates the distance on the frame from the point q1 to each of the points r1, r2,... For the frame image 53, and the head position ri, rj,. Ask all. For other points such as the point q2 in the frame image 52, all the top position rk, rl,. Then, the possibility of the same top position as the point p1 is (q1, ri), (q1, rj), (q1, ...), (q2, rk), (q2, rl), (q2, ... ), ... combination pattern. The three-dimensional head position estimation unit 23 temporarily stores the position information of the person in the memory and applies the third embodiment to the positions P1, P2,. . . And the coordinate value of the calculation result is stored in the memory.

  By using this embodiment, it is possible to reduce the amount of calculation for estimating the three-dimensional position of a person, and it is possible to shorten the time until estimation.

  Next, as in the second embodiment, an embodiment in which a person position at time t is estimated when a plurality of person positions and speeds at the previous time t ′ are estimated will be described with reference to FIG. FIG. 10 shows details of steps S116 and S117 of FIG. 3 according to the present embodiment.

First, as in the second embodiment, in step S114, the positions of the tops of all persons in the base frame image of the base camera and the frame images before and after the reference camera are estimated. When the number of persons estimated at the previous time t ′ is m, the positions estimated in the three-dimensional space are P ′ (1) to P ′ (m), and the velocity vectors are ν _{p ′ (1) to} ν. _{Let p ′ (m)} . Using this result, the three-dimensional top position estimating unit 23 assumes the human positions Pa (1) to Pa (m) at the current time t as follows, Estimation is performed using Equation 3 (step S200).

ただし、Ｐａ（ｋ）およびＰ’（ｋ）は、３次元空間の座標値のうち、ｘ成分とｙ成分のみとるものとする。

However, Pa (k) and P ′ (k) are assumed to be only the x component and the y component among the coordinate values in the three-dimensional space.

  Next, as in the third embodiment, the three-dimensional top position estimation unit 23 assumes all combinations from the reference frame image and the top position in the front and rear frame images obtained in step S114. A temporary head position is calculated (step S202). Next, the three-dimensional parietal position estimation unit 23 checks the validity based on the relationship between the assumed height range and the floor surface, deletes inappropriate combinations, and provisional three-dimensional parietal position P (1) ˜ P (n) is obtained (step S204). Here, n is the remaining number excluding an inappropriate combination of m Pa, and m ≧ n.

  Next, the three-dimensional top position estimation unit 23 determines the person position at time t using the person position information in S200 and the position information in S204 (step S206). Here, m of the temporary three-dimensional top positions P (1) to P (n) obtained from the information of the person positions Pa (1) to Pa (m) estimated from the previous time t ′ and the time t. Xn pairs are created, the distance between two positions is obtained for each pair, and pairs that are equal to or greater than a predetermined threshold are excluded. Subsequently, among the remaining pairs, the combination having the closest distance is adopted, and the provisional three-dimensional top position information P (i) of the person at the time t at that time is determined as i = 1 to n as the person position. . Then, the determined P (i) is deleted from the temporary three-dimensional top position P (1) to P (n). The remaining temporary 3D top position information and the combination of the person positions estimated from the previous time t ′ are adopted, and the process of determining the 3D top position information is repeated. This process is repeated until there is no 3D head position P (i) that is still tentative and not yet determined, or the person position Pa (j) estimated from the previous time t ′.

  Here, if the provisional three-dimensional top position remains, it is considered that a person has newly entered the imaging space at time t, and provisional three-dimensional top position information is determined. Further, if the person position Pa (j) estimated from the previous time t ′ is left, it is determined whether Pa (j) is the shooting range of the reference camera. It is assumed that the estimation at t 'was incorrect, and the person position at time t is not registered. If it is outside the shooting range, it is considered that the person has moved out of the shooting range of the reference camera, and Pa (j) is registered as the person position at time t.

  Subsequently, the movement amount estimation unit 24 obtains the movement speed of the person at time t by using the same method as in the second embodiment for each point determined as the person position at time t in S206 (step S208). However, when Pa (j) is registered as a person position, the speed information uses information at time t ′. Also, the speed of the newly registered person position is assumed to be zero.

  Then, the process proceeds to step S118 in FIG. 3, and the estimation result of the person position is displayed as in the third embodiment.

  As described above, according to the present embodiment, temporal position information is used even if matching is not performed based on the color or shape of an image when a plurality of persons are captured in the base camera and the reference camera. This makes it possible to improve the accuracy of position information in the actual environment.

  Next, an embodiment in which the number of combinations to be calculated is reduced using information at the previous time t ′ in step S202 for obtaining a temporary three-dimensional top position in the fifth embodiment will be described.

As in S200 of the fifth embodiment, the number of persons estimated at the previous time t ′ is m, the estimated positions are P ′ (1) to P ′ (m), and the velocity vector is ν _{p ′ (1)} . _Let ν _{p ′ (m)} . Further, the shooting time of the front frame image of the camera is s1, and the shooting time of the rear frame image is s2. Then, from the estimated information at time t ′, the estimated three-dimensional head position Qa (1) to Qa (m) at time s1 and the estimated three-dimensional head position Ra (1) to Ra (m) at time s2 are as follows: It is possible to obtain by Equation (4).

次に、３次元頭頂部位置推定部２３は、推定３次元頭頂部位置Ｑａ（１）〜Ｑａ（ｍ）を参照カメラで撮影したと想定して、カメラの位置、方向、画角の情報より、フレーム画像上に投影した位置qa(1)〜qa(m)を求める。同様に、３次元頭頂部位置推定部２３は、Ｒａ（１）〜Ｒａ（ｍ）の参照カメラ画像上での位置ra(1)〜ra(m)を求める。さらに同様に、３次元頭頂部位置推定部２３は、数３により求められたＰａ（１）〜Ｐａ（ｍ）の基準フレーム画像上での時刻t'の情報からの推定位置pa(1)〜pa(m)を求める。

Next, the three-dimensional head position estimation unit 23 assumes that the estimated three-dimensional head positions Qa (1) to Qa (m) are captured by the reference camera, and uses information on the position, direction, and angle of view of the camera. Then, the positions qa (1) to qa (m) projected on the frame image are obtained. Similarly, the three-dimensional vertex position estimation unit 23 obtains positions ra (1) to ra (m) on the reference camera image of Ra (1) to Ra (m). Further, similarly, the three-dimensional top position estimation unit 23 calculates the estimated positions pa (1) to Pa (1) to Pa (1) to Pa (m) from the information of the time t ′ on the reference frame images. Find pa (m).

  Next, the three-dimensional head position estimation unit 23 obtains all head positions in the reference frame image and the preceding and following frame images. The top position in the reference frame image is p (1) -p (a), the top position in the previous frame image is q (1) -q (b), and the top position in the rear frame image is r (1 ) To r (c).

  Then, with regard to the top position p (1) to p (a), it is examined whether the estimated positions pa (1) to pa (m) from the information at time t ′ exist within the range of the radius L1 specified in advance. . When the position existing within the range is pa (u) (may be plural), for the previous frame image, the top position q (multiple existing within the peripheral radius L2 of qa (u) And the top frame position r (which may be plural) existing in the peripheral radius L3 of ra (u) is a combination for obtaining a three-dimensional position.

  If there is no combination of q and r corresponding to the top position p (1) to p (a), it is assumed that there is no person within the stereo viewable range, and the three-dimensional top position is calculated. Shall not be used.

  According to the present embodiment described above, when the number of persons captured in an image is large, it is possible to reduce the combination calculation, and it is possible to speed up the person position estimation.

  Here, an embodiment will be described in the case where a person is present in a region other than the image capturing region where both the base camera and the reference camera are capturing images.

  For example, assume a situation in which the room 38 is photographed with the camera arrangement as shown in FIG. It is assumed that there is a result of obtaining the person position using the camera A and the camera B at the time t ′. At the next time t, it is assumed that the person position information is obtained with the camera B as the reference camera and the camera C as the reference camera. Then, at time t, it is possible to obtain only information on the position of a person existing within the shooting range of the cameras B and C. Here, it is assumed that the position of the person outside the shooting range of the cameras B and C is estimated by using the information of the person position and speed at the time t ′. At this time, it is estimated that the person performs a uniform linear motion. In addition, spatial constraints are taken into account when moving a person, reflecting motion is assumed at the boundary with a non-movable area, the speed is reduced according to the distance from the boundary area, and the moving direction is not moved. It may be assumed that the direction is changed away from the possible area.

  Here, an embodiment of a method for displaying a three-dimensional position of a person's 3D vertex and a spatial existence possibility distribution estimated by the above embodiments will be described with reference to FIGS. 11a to 11c.

  FIG. 11a is an example in which a person's existence possibility distribution at time t1 is displayed, and the left side is displayed as a block (60 in the figure) obtained by dividing the floor surface of a space (for example, a room) in a grid pattern, In addition, the possibility distribution of the person is expressed in shades (61 in the figure) around the point where the perpendicular of the obtained three-dimensional top position of the person and the floor intersect. The figure on the right is a profile 63 of a human existence possibility distribution at 62 dotted line positions (peak positions). The existence possibility distribution on the left side is displayed darker as the possibility of existence of a person increases.

  Similarly, FIG. 11b shows the existence possibility distribution (64 and 67 in the figure) of the person at time t2 and FIG. 11c, and the existence possibility distribution profiles at dotted lines 65 and 68 are 66 and 69, respectively.

  Note that the existence possibility distribution may be expressed by a difference in color according to the possibility, and a high possibility may be expressed as red, and a low possibility may be expressed as blue.

  By the display method as described above, it is possible to display the movement of a person including a position estimation error.

  Next, an example of another display method of the person's three-dimensional top position and the spatial existence possibility distribution will be described with reference to FIGS. 12a to 12c. It is assumed that the same three-dimensional top position and spatial existence possibility distribution of a person as those in FIG. 11 are used.

  In the present embodiment, the existence possibility is displayed as the luminance value of the block in proportion to the value obtained by integrating the evaluation value of the existence possibility in each block obtained by dividing the floor surface in a grid pattern. For example, at time t1, the existence possibility profile at the position of the dotted line 71 is as shown in the right diagram of FIG. At this time, the luminance of the block 70 is displayed as a value proportional to the integral value 72 of the existence possibility evaluation value in this block. This figure is displayed so that the density of oblique lines increases as the possibility of existence increases. Similarly, at time t2, as shown in FIG. 12b, the luminance values of the blocks 74 and 75 are proportional to the integrated values 76 and 77 in the block shown in the existence possibility profile on the dotted line 73, and at time t3, As shown in FIG. 12 c, the luminance value of the block 78 is displayed by being proportional to the value of the integrated value 80 in the block shown in the existence possibility profile in the dotted line 79.

  According to the present embodiment described above, it is possible to visualize the possibility of existence of a person at a place of interest, and the possibility of existence according to the number of persons even when there are a plurality of persons in one block. Can be displayed.

  Furthermore, another embodiment of a method for displaying the three-dimensional top position of a person and the spatial existence possibility distribution will be described with reference to FIGS. 13a to 13c. It is assumed that the same three-dimensional top position and spatial existence possibility distribution of a person as those in FIG. 11 are used.

  In the present embodiment, as in the ninth embodiment, the possibility of existence of a person is displayed as the luminance in each block obtained by dividing the floor surface in a grid pattern, which is proportional to the maximum value of the existence possibility evaluation value in the block. Display as luminance. For example, at time t1, as shown in FIG. 13a, the luminance of the block 81 is determined so as to be proportional to the maximum value 83 in the block as shown on the existence possibility profile of the dotted line 82. Similarly, at time t2, as shown in FIG. 13b, the brightness of the blocks 84 and 85 is determined by the values 87 and 88 shown on the existence possibility profile on the dotted line 86, and can exist on the dotted line 90 at time t3. The luminance value of the block 89 is determined so as to be proportional to the evaluation value 91 shown on the sex profile.

  The advantage of the present embodiment compared to the ninth embodiment is that the block that is most likely to have a person is always clearly displayed. For example, looking at FIG. 12, it is clear which block is likely to be present in the diagrams of FIGS. 12a and 12b at times t1 and t3, but the entire diagram is shown in FIG. 12b at time t2. The display is too low in brightness. On the other hand, in this embodiment, as shown in FIG. 13, it can be clearly displayed at any time that a person exists.

  However, when this embodiment is used, it is difficult to express that there are a plurality of persons in one block, so the block size should be small enough to have one person. .

  Next, an embodiment using the display result of a person's spatial possibility distribution according to the above embodiment will be described with reference to FIGS.

  FIG. 14 shows an example in which a room such as a supermarket is assumed to be a shooting space. The spatial presence possibility distribution 95 of the person created by the method of the tenth embodiment is displayed as a person presence possibility distribution 97 in the store as being superimposed on an arrangement 96 such as a physical shelf or table in the space.

  FIG. 15 shows how to use the person presence possibility distribution 97 in the store. The in-store person presence possibility distribution 97 is displayed on the display device 13, and the store staff 100 observes the temporal change and uses it for service creation as shown in 101.

  Such a display method can intuitively grasp how many people exist in which space at which time, and can automatically collect data. For example, by monitoring the possibility of the presence of a person in the space in front of the cash register (either automatically or by a person), when the number of persons increases, a staff is arranged (103 in the figure). It is also possible to reduce the congestion by opening a new cash register (104 in the figure).

It is the figure which showed the structural example of the system in this invention. It is an internal block diagram of an image processing apparatus. It is a flowchart which shows the process sequence of an image processing apparatus. It is the figure which showed the example of arrangement | positioning of imaging | photography space and a camera. It is the figure which showed the example of the imaging | photography time of a camera. It is a figure explaining the method of calculating | requiring the three-dimensional top part of a person from the picked-up image of two cameras. It is a figure which shows the positional relationship when several persons are reflected in the picked-up image of a camera. It is a figure which shows the example which estimates the 3D top position of an incorrect person. It is a figure which shows the example which estimated the person's three-dimensional top part position correctly. FIG. 4 is a processing procedure in which a part of the flowchart of FIG. 3 is detailed. It is an example which displayed the possibility distribution of a person. It is an example (continuation) which displayed the possibility distribution of a person. It is an example (continuation) which displayed the possibility distribution of a person. This is an example in which the existence possibility distribution of a person is displayed by blocking the floor using integration. This is an example (continuation) in which integration is used to block the floor and display the possibility distribution of people. This is an example (continuation) in which integration is used to block the floor and display the possibility distribution of people. This is another example in which the maximum value is used to block the floor and display the possibility distribution of persons. This is another example (continuation) in which the maximum value is used to block the floor and display the possibility distribution of the person. This is another example (continuation) in which the maximum value is used to block the floor and display the possibility distribution of the person. It is an example which displays the person presence possibility distribution in a shop. This is an example in which a service is created by using the displayed distribution of person presence possibility.

Explanation of symbols

1: wired camera group, 3: wireless camera group, 9: recording device, 11: image processing device, 13: display device, 30: camera

Claims (15)

A method by an image processing apparatus for estimating a position of a person in a three-dimensional space using images taken by two or more asynchronous cameras of the same person,
Inputting a frame image captured by each camera and its shooting time into a memory;
Identifying two consecutive frames of images taken by the first camera as a previous frame and a subsequent frame;
A frame image taken by a second camera different from the first camera and taken at the time between the previous frame and the rear frame and closest to the time of the rear frame is used as a reference. Identifying as a frame,
Estimating a region in which a person is shown from the images of the previous frame, the rear frame, and the reference frame;
Estimating the head of the person from each of the estimated person regions;
A first straight line in a three-dimensional space corresponding to the top of the head on the front frame and passing through the optical center of the first camera and a top of the head on the back frame corresponding to the top of the head on the first frame. A plane in the three-dimensional space or the same straight line formed by the second straight line in the three-dimensional space, and a straight line in the three-dimensional space corresponding to the top of the head on the reference frame and passing through the optical center of the second camera Calculating the intersection or closest point of and estimating the 3D head position of the person;
And a step of displaying the position of the person on the floor based on the three-dimensional information on the position of the top of the head. And determining the height of the person from the three-dimensional top position;
Assuming that the height does not change, a three-dimensional first position on the first straight line and a three-dimensional second position on the second straight line are obtained, and the reference frame is obtained from an interval of photographing times. Obtaining a moving speed of the person at the shooting time of
The three-dimensional head position determined for the reference frame immediately before the reference frame and the three-dimensional head position corresponding to the current reference frame determined from the moving speed, and the currently estimated three-dimensional Determining an error from a typical top position;
2. The person position estimating method according to claim 1, further comprising a step of obtaining a possibility that a person exists in the vicinity of the current three-dimensional top position using the error. When a plurality of persons are shown in each of the front frame, the rear frame, and the reference frame,
The three-dimensional top position is estimated for each combination of one person on the reference frame and any corresponding person on the previous frame and any person on the rear frame. Steps,
Obtaining height from the three-dimensional top position for each of the combinations;
The person position estimating method according to claim 1, further comprising: estimating the three-dimensional top position of the combination having the height closest to the height range as the position of the person on the reference frame. Calculating the distance on the frame of the position of the top of any person on the front frame and the position of the top of any person on the back frame;
And a step of excluding the combination having a person on a rear frame exceeding a maximum movable distance in a photographing time interval of the person on the previous frame from a three-dimensional head position estimation candidate. Item 4. The person position estimation method according to Item 3. Estimating the three-dimensional top position for each combination where the height falls within a predetermined height range; and
Assuming that the height does not change, a three-dimensional first position on the first straight line and a three-dimensional second position on the second straight line are obtained, and the reference frame is obtained from an interval of photographing times. Obtaining a moving speed of the person at the shooting time of
Obtaining for each person a second three-dimensional head position corresponding to the current reference frame from the three-dimensional head position determined for the reference frame immediately before the reference frame and the moving speed;
4. A step of determining a position having a minimum distance from the second three-dimensional top position among the estimated three-dimensional top positions as a person position. The person position estimation method described.   3. The person position estimating method according to claim 2, wherein when the person position on the floor is displayed, the possibility of existence of the person is displayed by shading according to the magnitude of the possibility that the person exists.   When displaying the position of a person on the floor, the floor is divided into a plurality of blocks each having the same size, and the possibility of being present in each block is integrated among the possibility distribution of the person 3. The person position estimating method according to claim 2, wherein a value is obtained, and the possibility of existence of a person in the block is displayed by shading according to the magnitude of the integrated evaluation value.   When displaying the position of a person on the floor, the floor is divided into a plurality of blocks each having the same size, and the maximum possibility of being present in each block is obtained from the possibility distribution of the person. 3. The person position estimation method according to claim 2, wherein the possibility of existence of a person in the block is displayed by shading according to the maximum possibility.   3. The method for using person position estimation according to claim 2, wherein a service provided to the person in the photographing space is changed according to the possibility of the person's spatial existence. A group of cameras arranged so that the same person can be photographed by two or more asynchronous cameras, a frame image photographed by each camera and a recording device for recording the photographing time, and a person using the frame image An image processing apparatus for estimating a position in a dimensional space,
The image processing apparatus includes:
Means for inputting a frame image stored in the recording device and a photographing time thereof into a memory;
Means for specifying two consecutive frames of images taken by the first camera as a previous frame and a subsequent frame;
A frame image taken by a second camera different from the first camera and taken at the time between the previous frame and the rear frame and closest to the time of the rear frame is used as a reference. Means to identify as a frame;
Means for estimating a region in which a person is shown from the images of the front frame, the rear frame, and the reference frame;
Means for estimating the head of the person from each of the estimated person regions;
A first straight line in a three-dimensional space corresponding to the top of the head on the front frame and passing through the optical center of the first camera and a top of the head on the back frame corresponding to the top of the head on the first frame. A plane in the three-dimensional space or the same straight line formed by the second straight line in the three-dimensional space, and a straight line in the three-dimensional space corresponding to the top of the head on the reference frame and passing through the optical center of the second camera Means for calculating the intersection or closest point of the object and estimating the three-dimensional head position of the person,
And a means for displaying a person position on the floor based on the three-dimensional information on the position of the top of the head. The image processing apparatus further includes:
Means for determining the height of the person from the three-dimensional top position;
Assuming that the height does not change, a three-dimensional first position on the first straight line and a three-dimensional second position on the second straight line are obtained, and the reference frame is obtained from an interval of photographing times. Means for determining the moving speed of the person at the shooting time;
The three-dimensional head position determined for the reference frame immediately before the reference frame and the three-dimensional head position corresponding to the current reference frame determined from the moving speed, and the currently estimated three-dimensional A means for obtaining an error from a typical top position;
11. The person position estimation system according to claim 10, further comprising means for determining a possibility that a person exists in the vicinity of the current three-dimensional top position using the error. In the case where a plurality of persons are captured in each of the front frame, the rear frame, and the reference frame, the image processing apparatus further includes:
The three-dimensional top position is estimated for each combination of one person on the reference frame and any corresponding person on the previous frame and any person on the rear frame. Means,
Means for determining height from the three-dimensional top position for each of the combinations;
11. The person position estimating system according to claim 10, further comprising means for estimating the position of the person on the reference frame with the three-dimensional top position of the combination having the height closest to the height range. The image processing apparatus further includes:
Means for calculating the distance on the frame of the position of the top of any person on the front frame and the position of the top of any person on the back frame;
And a means for excluding the combination having a person on a rear frame exceeding a maximum movable distance in a photographing time interval of the person on the previous frame from a three-dimensional top position estimation candidate. Item 13. The person position estimation system according to Item 12.   12. The image processing apparatus according to claim 11, wherein when the position of the person on the floor is displayed, the possibility of existence of the person is displayed by shading according to the magnitude of the possibility that the person exists. Human position estimation system. The image processing apparatus divides the floor surface into a plurality of blocks each having the same size when displaying the position of the person on the floor surface, and exists in each block of the possibility distribution in which the person exists. 12. The person position estimation system according to claim 11, wherein an evaluation value obtained by integrating the possibility is obtained, and the possibility of existence of a person in the block is displayed by shading according to the magnitude of the integrated evaluation value. .

Images