JP6588413B2 - Monitoring device and monitoring method - Google Patents

Description

  The present invention relates to a monitoring apparatus and a monitoring method.

  A technique for tracking a person's trajectory from a single image is known. For example, a person appearing in the video is detected, and the pixel position of the human video and the identification information of the human video are output in time series (see Non-Patent Document 1).

  In addition, a technique is known that uses images taken from a plurality of directions and identifies a person who has been photographed only in a part of the images by using a relative relationship between the images as a clue. In addition, there is known a technique for detecting a person in each frame image of a video taken by a plurality of cameras and calculating a movement trajectory of each person (see Patent Document 1).

  In addition, a technique for detecting a suspicious behavior of a person using a stereo camera image has been proposed. This technique acquires a movement trajectory of a person to be monitored, identifies a behavior state of the person based on the movement trajectory, and detects suspicious behavior (see Patent Document 2).

  A person monitoring technique using a plurality of cameras has been proposed. For example, a technique is proposed in which moving objects are extracted from video captured by multiple cameras, the images are collated between cameras, the moving path of the moving object is obtained, and an alarm is displayed to a supervisor when entering a predetermined area. Has been. In this technique, by using a personal identification means such as card authentication, it is determined whether or not a mobile object, that is, a person has access authority to a predetermined area. Also, by analyzing the movement path and movement trajectory of the person, it is determined whether or not the person in the video is a suspicious person, and an alarm is displayed.

JP 2010-063001 A JP 2012-128877 A

“Image Sensing Technology OKAO Vision Finding and Recognizing People”, [online], December 2015, OMRON Corporation, [April 12, 2016 search], Internet <URL: http: //plus-sensing.omron .co.jp / technology / ＞

  However, the conventional technology for detecting a person in an image involves processing for determining the identity by comparing features such as a face of a person image, and stereo matching for associating images from multiple viewpoints with a three-dimensional space. It was. This complicates the process and increases the cost for setting and operating the camera calibration.

  Further, the technology for detecting a suspicious behavior of a person using a video from a stereo camera needs to register a behavior pattern of a suspicious person (outsider) in advance, which is troublesome.

  In addition, when monitoring a person in a video taken by a plurality of cameras, it is premised on the combined use of a personal identification means such as a card authentication means that clearly identifies the person. Therefore, for example, in the case of using only a technique that is ambiguous in identifying a person, such as a video tracking system and a beacon receiving system, it is possible to determine whether or not the person has access authority to a predetermined area detected by the video tracking system. It is difficult to determine whether or not a person is a suspicious person. For example, the position detection accuracy of the beacon receiving system is about several tens of meters, and there are many errors. Therefore, it is also possible to determine whether the person detected by the video tracking system has the authority to access the predetermined area by using the beacon receiving system together, or determine whether the person is a suspicious person. Have difficulty.

  The present invention has been made in view of the above, and it is an object of the present invention to easily identify a person in a video taken from multiple viewpoints and monitor an action in combination with a sensor with ambiguous person identification. To do.

  In order to solve the above-described problems and achieve the object, the monitoring apparatus according to the present invention identifies a plurality of videos taken from different locations within a predetermined monitoring range and persons within the monitoring range at the same time. Generating the training data that combines the acquisition unit that acquires the sensor data detected in this way, the position of the same person appearing in each of the acquired images, and the sensor data that is detected at the same time An estimation unit that calculates an estimation value of the sensor data at the position of the person shown in each of the videos acquired in a continuous time series based on the training data, and the calculated estimation value A specifying unit for specifying identification information for identifying a person in the video based on a time-series change in similarity with the sensor data acquired simultaneously with the acquired video; and the plurality of videos As the identification information of the same person to be superimposed when the position of the person shown in each is converted into a position within the monitoring range, there are many overlapping numbers among the identification information of the person in each video specified before conversion And a determination unit for determining a thing.

  According to the present invention, it is possible to easily identify a person in a video taken from multiple viewpoints and monitor an action in combination with a sensor in which the person is not clearly identified.

FIG. 1 is a schematic diagram showing a schematic configuration of a system including a monitoring device according to an embodiment of the present invention. FIG. 2 is an explanatory diagram for explaining a method of installing the video sensor. FIG. 3 is an explanatory diagram for explaining a method of installing the video sensor. FIG. 4 is an explanatory diagram for explaining a method of installing the video sensor. FIG. 5 is an explanatory diagram for explaining the position of the person video. FIG. 6 is an explanatory diagram for explaining generation of training data. FIG. 7 is an explanatory diagram for explaining global pixel coordinates and local pixel coordinates. FIG. 8 is an explanatory diagram for explaining processing of the estimation unit. FIG. 9 is an explanatory diagram for explaining the processing of the determination unit. FIG. 10 is an explanatory diagram for explaining processing of the determination unit. FIG. 11 is a flowchart showing the monitoring processing procedure. FIG. 12 is a diagram illustrating a computer that executes a monitoring program.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In addition, this invention is not limited by this embodiment. Moreover, in description of drawing, the same code | symbol is attached | subjected and shown to the same part.

[System configuration]
FIG. 1 is a schematic diagram illustrating a schematic configuration of a system including a monitoring device according to the present embodiment. As illustrated in FIG. 1, a plurality of installed video sensors C each capture a predetermined monitoring range R and generate the video. In addition, the wireless signal transmitter B is installed in the monitoring range R, and the mobile terminal m possessed by the person h in the monitoring range R is detected. The monitoring device 1 monitors the behavior of the person h in the monitoring range R by a monitoring process described later.

  The radio signal transmitter B is realized by, for example, a beacon that transmits a BLE (Bluetooth (registered trademark) Low Energy) signal, and transmits a radio signal toward the monitoring range R. When the mobile terminal m having the wireless signal receiving function receives a wireless signal within the monitoring range R, the mobile terminal m includes a terminal ID of the terminal m, identification information of the wireless signal transmitter B, and a reception strength of the wireless signal. Data is transmitted to the server S.

  Here, the terminal ID of the mobile terminal m and the identification information for identifying the person h carrying the mobile terminal m are associated in advance. In that case, the server S detects, based on the sensor data, the radio signal transmitted from which radio signal transmitter B installed at a plurality of locations within the monitoring range R and received with a certain reception strength. The identification information and position of the person h holding the mobile terminal m are detected. Note that this radio signal may be a sound wave in addition to an electromagnetic wave.

  The image sensor C is realized by an optical camera, an infrared camera, or the like, and is installed so that a person h in the monitoring range R can be photographed from a plurality of places with different photographing directions and distances to the person h. Specifically, the installation method of the image sensor C will be described with reference to FIGS.

  For example, the imaging direction with respect to the monitoring range R of each video sensor C is changed so that the range where the imaging ranges overlap is provided. As a result, as illustrated in FIG. 2, it is possible to shoot a person h that has entered the overlapping range (a, b) of the shooting range while moving with a plurality of video sensors C (C1 to C3). For example, a person h in the overlapping range a can be photographed by the video sensor C1 and the video sensor C2.

  In addition, the image sensor C is installed by changing the shooting direction with respect to the person h within the monitoring range R. As a result, as illustrated in FIG. 3, even when there are obstacles between some of the video sensors C1 and the person h, this person h is another video sensor C2 having a different shooting direction with respect to the person h. It becomes possible to shoot with.

  In addition, each video sensor C is installed such that the distance to the person h in the monitoring range R is different. As a result, as illustrated in FIG. 4, the distance d from some of the image sensors C1 is too close, and the moving speed of the person h on the image increases, and the detection accuracy of the person h on the image decreases. In this case, it is possible to shoot with another video sensor C2 having a different distance to the person h. On the other hand, even when the distance d from some of the image sensors C1 is too far and the moving speed of the person h on the image is reduced, and the detection accuracy of the person h on the image is lowered, the person h is also reached. It is possible to shoot with other video sensors C2 having different distances. Note that each position of the image sensor C set here is fixed in the subsequent processing.

[Configuration of monitoring device]
Returning to the description of FIG. The monitoring device 1 is realized by a general-purpose computer such as a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15.

  The input unit 11 is realized by using an input device such as a keyboard or a mouse, and inputs various instruction information such as processing start to the control unit 15 in response to an input operation by the operator. The output unit 12 is realized by a display device such as a liquid crystal display or a printing device such as a printer.

  The communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and is connected between an external device such as a video sensor C or a server S and a control unit 15 via a telecommunication line such as a LAN (Local Area Network) or the Internet. Control communication.

  The storage unit 14 is realized by a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk, and stores training data 14a generated by monitoring processing to be described later. Is done. The storage unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13.

  The controller 15 is implemented using a CPU (Central Processing Unit) or the like, and executes a processing program stored in a memory. Thereby, the control part 15 functions as the acquisition part 15a, the training part 15b, the estimation part 15c, the specific | specification part 15d, and the determination part 15e so that it may illustrate in FIG.

  The acquisition unit 15a acquires, at the same time, a plurality of videos taken from different locations within a predetermined monitoring range, and sensor data that is detected by identifying a person within the monitoring range. Specifically, the acquisition unit 15a uses the communication control unit 13 to display a plurality of videos captured by the plurality of video sensors C in the monitoring range R at the same time and sensor data detected by the server S at the same time. Get continuously in time through.

  Further, the acquisition unit 15a detects a human video portion (hereinafter referred to as a human video) p by detecting, for example, the shape of the head to the shoulder from each of the acquired multiple videos at the same time. . Further, the acquisition unit 15a extracts, as the position xp of each person video p, for example, the position on the pixel coordinates of a representative point such as the center of gravity of the head area (hereinafter also referred to as a local pixel position). At that time, the acquisition unit 15a identifies the person image p of the same person and assigns the same identification value to the images taken from the same place at different consecutive times.

  Here, the local pixel position of the person video will be described with reference to FIG. In FIG. 5, a person hi at a position within the monitoring range R (hereinafter also referred to as a global pixel position) xhi (i = 1, 2, 3) was photographed by a plurality of video sensors Cn (n = 1, 2). A local pixel position xin of a person video in in a plurality of videos n is illustrated. For example, in the video 1 shot by the video sensor C1, the person h1 at the global pixel position xh1 and the person h2 at the global pixel position xh2 are the person video 11 at the local pixel position x11 or the person video at the local pixel position x21. It is reflected as 21. On the other hand, in the video 2 captured by the video sensor C2, in addition to the people h1 and h2, the person h3 at the global pixel position xh3 includes a person video 22 at the local pixel position x22, a person video 12 at the local pixel position x12, Alternatively, the image is shown as a person image 32 at the local pixel position x32. That is, the person h3 at the global pixel position xh3 is not reflected in the video 1 but only in the video 2.

  Note that when the acquisition unit 15a acquires a plurality of videos n and extracts the local pixel position xin of the person video in, the correspondence between each person video in and the person hi at the global pixel position xhi is unknown. Therefore, the monitoring device 1 of the present embodiment specifies identification information of the person video in in each video n by a monitoring process described later. In addition, when the local pixel position xin of the person video in in each video n is converted into the global pixel position xhi, the identification information of the person video in having the same global pixel position xhi has a large number of duplicates. It is determined as identification information of the person hi. Thereby, the monitoring apparatus 1 of this embodiment identifies each person hi, detects the movement locus | trajectory, and monitors action.

  The acquisition unit 15a also captures images of the person h in the monitoring range R taken from a plurality of places with different shooting directions and distances to the person h by the video sensor C installed as described above. To get. Thereby, the range which the imaging | photography range of each image sensor C overlaps is made, and the some image | video which image | photographed the person h in the overlap range with the some image | video sensor C can be acquired (refer FIG. 2). Further, even when there is an obstacle between some of the video sensors C and the person h and it is difficult to track the person h on the video, the person h can be detected using the video captured by the other video sensors C. The decrease in accuracy can be suppressed (see FIG. 3). In addition, for example, when the distance from some of the video sensors C is too short and the moving speed of the person h on the video increases and the detection accuracy on the video of the person h decreases, A decrease in the accuracy of identification of the person h can be suppressed by using the video imaged by the sensor C. On the other hand, even when the distance from some of the image sensors C is too far and the movement speed of the person h on the image becomes small and the detection accuracy on the image of the person h is reduced, other image sensors A reduction in the accuracy of the identification of the person h can be suppressed using the video imaged by C (see FIG. 4).

  Returning to the description of FIG. The training unit 15b generates training data that combines the position of the same person h shown in each of the acquired videos and the sensor data that has been detected at the same time. Specifically, generation of training data 14a will be described with reference to FIG. The training unit 15b, at the same time, the local pixel position of the person video n in each video n obtained by photographing the person h at the global pixel position xh in the monitoring range R by the plurality of video sensors Cn (n = 1, 2). Training data 14a is generated by combining xn and sensor data that detects the same person h.

  Here, the sensor data is represented by, for example, the reception intensity of the radio signal received from the radio signal transmitter B by the mobile terminal m possessed by the person h. That is, the sensor data is represented by a combination of the reception strengths at the positions of the persons h of the radio signals transmitted from the plurality of radio signal transmitters B installed in the monitoring range R. The training data 14a generated in the example shown in FIG. 6 includes the local pixel position xn in each video n of the person h at the global pixel position xh and sensor data.

  Further, the training unit 15b changes the global pixel position xh of the person h, and generates training data 14a at each global pixel position xh in the same manner as the above processing. Thereafter, the training unit 15 b stores the generated training data 14 a in the storage unit 14.

  In this way, the training unit 15b combines the local pixel position xn of the person video n in each video n corresponding to the global pixel position xh of the person h and the sensor data detected at that position to combine the training data 14a. Is generated. Note that, as will be described later, the monitoring device 1 of the present embodiment uses each local image position xn in each image n corresponding to the same global pixel position xh and sensor data detected at that position, for each image. The identification information of the person shown in n is specified. Therefore, it is desirable that training data 14a corresponding to as many global pixel positions xh as possible within the monitoring range R is generated.

  Moreover, FIG. 7 is explanatory drawing explaining the local pixel coordinate and global pixel coordinate of each image | video n. As illustrated in FIG. 7, the training unit 15 b uses a conversion formula or conversion that indicates a correspondence between an arbitrary local pixel position on the local pixel coordinate of each image n by the image sensor Cn and a global pixel position on the global pixel coordinate. A table is generated and stored in the storage unit 14. As will be described later, the determination unit 15e converts the local pixel position xn of the person image in the image n into the global pixel position xh with reference to the conversion formula or conversion table.

  Returning to the description of FIG. Based on the training data 14a, the estimation unit 15c calculates an estimated value of the sensor data at the position of the person shown in each of the images acquired in continuous time series. Specifically, the estimation unit 15c uses the correspondence between the local pixel position in each video of the training data 14a and the sensor data, and the local pixel position of the person video in each video acquired continuously in time. The estimated value of the sensor data at is calculated.

  For example, the estimation unit 15c divides the local pixel coordinates in each video into a plurality of cells, and estimates the sensor data average value in the training data 14a belonging to the same cell as the sensor data in the local pixel position belonging to the cell. Calculate as a value. In other words, the estimation unit 15c calculates an estimated value of sensor data in the vicinity of each local pixel position. The estimation unit 15c calculates an estimated value of sensor data in a cell to which an arbitrary local pixel position in each video belongs in advance and stores the estimated value in the storage unit 14 prior to the processing of the specifying unit 15d described below. It is desirable to keep it.

  The specifying unit 15d specifies identification information for identifying a person in the video based on a time-series change in similarity between the calculated estimated value and the sensor data acquired simultaneously with the acquired video. Specifically, the specifying unit 15d determines the sensor data of the person k actually measured and acquired at the same time as the estimated value calculated by the estimation unit 15c for the local pixel position of a certain person j acquired continuously in time. The similarity is calculated.

  The specifying unit 15d calculates, for example, the distance between the measured value and the estimated value of the sensor data in the l-dimensional space as the similarity with respect to the sensor data for the one radio signal transmitter B installed in the monitoring range R. To do.

  Then, the specifying unit 15d determines whether the person j and the person k are the same person based on the calculated temporal transition of the similarity, and when determining that the person j is the same person, It is specified as identification information for j. For example, the specifying unit 15d determines that the person is the same person when the average value of the similarity in a predetermined period is higher than a predetermined threshold.

  Here, the processing of the specifying unit 15d will be described in detail with reference to FIG. In the example shown in FIG. 8, the temporal transition of the estimated value of the sensor data of the person j is more similar to the temporal transition of the measured value of the sensor data of the person k1 than the temporal transition of the measured value of the sensor data of the person k2. Yes. In the period shown, the average value of the similarity between the estimated value of the sensor data of the person j and the measured value of the sensor data of the person k1 is larger than the average value of the similarity with the measured value of the sensor data of the person k2. Assume that a predetermined threshold value is exceeded. In this case, it is determined that the person j and the person k1 are the same person, and the identification information of the person j is specified to be the same as the identification information of the person k1.

  The specifying unit 15d performs the above process for all combinations of the person j and the person k, and specifies the identification information of the person in the video. When there is no person k determined to be the same person for the person j, the person j is determined to be an outsider. Further, the specifying unit 15d performs the above-described processing for all the videos n to specify the identification information of the person in each video n.

  Returning to the description of FIG. The determination unit 15e identifies the person in each video specified before conversion as identification information of the same person to be superimposed when the position of the person shown in each of the plurality of videos is converted into a position within the monitoring range R. The information having a large number of duplicates is determined.

  Specifically, the process of the determination unit 15e will be described with reference to FIGS. The determination unit 15e refers to the conversion formula or conversion table in the storage unit 14 and determines the local pixel position xin on the local pixel coordinate of the person video in in each video n as the global pixel on the global pixel coordinate in the monitoring range R. Convert to position hi.

  At that time, the determination unit 15e does not necessarily have to convert all the local pixel coordinate points within the angle of view of each video to the global pixel coordinates. It is sufficient to convert the local pixel position of the person video, and other coordinate points may be complemented later.

  In addition, since the local pixel position of the person video in each video is a representative point such as the center of gravity of the head area of each human video as described above, the converted global pixel even if it is a human video of the same person The positions are not necessarily the same. Therefore, person images of the same person are superimposed at positions where the converted global pixel positions are the same or in the vicinity of a distance within a predetermined threshold.

  In the example shown in FIG. 9, the local pixel position x11 of the video 1 of the video sensor C1 and the local pixel position x12 of the video 2 of the video sensor C2 are converted to the vicinity of the global pixel position xh1. In addition, the local pixel position x21 of the video 1, the local pixel position x22 of the video 2, and the local pixel position x23 of the video 3 of the video sensor C3 are converted to the vicinity of the global pixel position xh2.

  Next, the determination unit 15e determines, as the identification information of the person at the global pixel position xhi, the identification information of the person image specified by each of the images n by the specifying unit 15d that has a large number of duplicates. In the example shown in FIG. 9, the person h1 at the global pixel position xh1 is specified as the person of the identification information ID1 possessing the mobile terminal m1 in the video 1 and the video 2. In this case, as illustrated in FIG. 10, the determination unit 15e determines that the person h1 at the global pixel position xh1 is the person with the identification information ID1 that possesses the mobile terminal m1.

  On the other hand, the person h2 at the global pixel position xh2 is identified as the person of the identification information ID2 possessing the mobile terminal m2 in the images 1 and 2, but in the image 3, the person h2 of the identification information ID3 possessing the mobile terminal m3 is identified. Identified as a person. In this case, as illustrated in FIG. 10, the determination unit 15e determines that the person h2 at the global pixel position xh2 is the person with the identification information ID2 that possesses the mobile terminal m2. In this way, the determination unit 15e can identify the person identification information and track the movement locus of the person.

  Further, the determination unit 15 e outputs the determined person identification information to the output unit 12. For example, the determination unit 15e controls to display the identification information of the person superimposed on the local pixel position of the identified person on the video by each video sensor C. Thereby, for example, the convenience of a user such as an administrator who monitors an area where entry is prohibited except for related persons is improved.

[Monitoring process]
Next, with reference to FIG. 11, the monitoring process by the monitoring apparatus 1 according to the present embodiment will be described. FIG. 11 is a flowchart showing the monitoring processing procedure. The flowchart in FIG. 11 is started, for example, at a timing when there is an operation input instructing start of the monitoring process.

  The acquisition unit 15a acquires a plurality of videos taken from different locations in the predetermined monitoring range R and sensor data that is detected by identifying a person in the monitoring range R at the same time (step S1).

  Next, the training unit 15b generates training data 14a that combines the position of the same person shown in each of the acquired videos and the sensor data that has been acquired at the same time (step S2).

  After that, the acquisition unit 15a continuously generates a plurality of videos taken from different locations within the predetermined monitoring range R and sensor data that is detected by identifying a person within the monitoring range R at the same time. (Step S3).

  Moreover, the estimation part 15c calculates the estimated value of the sensor data in the position of the person shown in each of the acquired image | video based on the training data 14a. Further, the specifying unit 15d specifies identification information of a person in the video based on a time-series change in similarity between the calculated estimated value and the sensor data acquired at the same time as the acquired video (Step S15). S4).

  Further, when the determination unit 15e converts the position of the person shown in each of the plurality of videos to the global pixel position in the monitoring range R, the identification information of the same person to be superimposed is included in each video specified before the conversion. The identification information having a large number of duplicates is identified (step S5). Further, the determination unit 15e superimposes the identification information of the determined person on each video and outputs it to the output unit 12, identifies a person within the monitoring range R, and monitors the action. Thereby, a series of monitoring processes are completed.

  As described above, in the monitoring apparatus 1 according to the present embodiment, the acquisition unit 15a captures a plurality of videos taken from different locations in the predetermined monitoring range R and persons in the monitoring range R at the same time. The sensor data identified and detected is acquired. In addition, the training unit 15b generates training data 14a that combines the position of the same person shown in each of the acquired videos and the sensor data that has been acquired at the same time.

  Moreover, the estimation part 15c calculates the estimated value of the sensor data in the position of the person shown in each of the image | video acquired in the continuous time series based on the training data 14a. Further, the specifying unit 15d specifies identification information of a person in the video based on a time-series change in similarity between the calculated estimated value and the sensor data acquired simultaneously with the acquired video. Further, when the determination unit 15e converts the position of the person shown in each of the plurality of videos to the global pixel position in the monitoring range R, the identification information of the same person to be superimposed is included in each video specified before the conversion. The person identification information having a large number of duplicates is determined.

  This makes it possible to easily identify a person in a video taken from multiple viewpoints and monitor the behavior in combination with a sensor in which the person is not clearly identified. For example, when specifying the same person for each person based on the combination of positions in the images of multiple cameras, it is necessary to calculate the order of the “number of detected cameras” to the power of “number of cameras”. Become. That is, as the number of cameras increases, the amount of calculation increases exponentially. On the other hand, according to the monitoring process of the present embodiment, since a person is specified in each video, it is possible to specify a person with a calculation amount in the order of the product of “number of detected persons” and “number of cameras”.

  In addition, the acquisition unit 15a acquires videos obtained by shooting a person in the monitoring range R from a plurality of places having different shooting directions and distances to the person. Thereby, the range which the imaging | photography range of each image sensor C overlaps is made, and the some image | video which image | photographed the person in the overlap range with the some image sensor C can be acquired. In addition, even when there are obstacles between some of the video sensors C and the person and it is difficult to track the person on the video, the detection accuracy of the person is reduced using the video taken by the other video sensors C. Can be suppressed. In addition, for example, even when the distance from some of the video sensors C is too short and the movement speed of this person increases on the video, and the detection accuracy on the video of this person decreases, the other video sensors C It is possible to suppress a decrease in the accuracy of person identification by using the video taken by. On the other hand, even when the distance from some of the image sensors C is too far and the movement speed of the person on the image becomes small and the detection accuracy on the image of the person is lowered, It is possible to suppress a decrease in the accuracy of person identification using the captured video. Thus, it becomes possible to identify a person with high accuracy.

  Further, the determination unit 15 e outputs identification information of the identified person to the output unit 12. For example, control is performed so that the identification information of the person is displayed superimposed on the position of the identified person. Thereby, for example, the convenience of a user such as an administrator who monitors an area where entry is prohibited except for related persons is improved.

[program]
It is also possible to create a program in which processing executed by the monitoring apparatus 1 according to the above embodiment is described in a language that can be executed by a computer. As one embodiment, the monitoring device 1 can be implemented by installing a monitoring program for executing the above-described monitoring processing as package software or online software on a desired computer. For example, the information processing apparatus can function as the monitoring apparatus 1 by causing the information processing apparatus to execute the above monitoring program. The information processing apparatus referred to here includes a desktop or notebook personal computer. In addition, the information processing apparatus includes mobile communication terminals such as smartphones, mobile phones and PHS (Personal Handyphone System), and slate terminals such as PDA (Personal Digital Assistants). In addition, a terminal device used by a user can be a client, and the client can be implemented as a server device that provides services related to the above-described monitoring processing to the client. For example, the monitoring device 1 provides a monitoring processing service that receives a plurality of videos captured within a predetermined monitoring range and sensor data received by the beacon receiving system and outputs the position of the identified person and identification information. Implemented as a server device. In this case, the monitoring device 1 may be implemented as a Web server, or may be implemented as a cloud that provides services related to the above-described monitoring processing by outsourcing. Hereinafter, an example of a computer that executes a monitoring program that realizes the same function as the monitoring device 1 will be described.

  As shown in FIG. 12, a computer 1000 that executes a monitoring program includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface. 1070. These units are connected by a bus 1080.

  The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1031. The disk drive interface 1040 is connected to the disk drive 1041. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1041. For example, a mouse 1051 and a keyboard 1052 are connected to the serial port interface 1050. For example, a display 1061 is connected to the video adapter 1060.

  Here, as shown in FIG. 12, the hard disk drive 1031 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. Each table described in the above embodiment is stored in the hard disk drive 1031 or the memory 1010, for example.

  Further, the monitoring program is stored in the hard disk drive 1031 as a program module 1093 in which a command executed by the computer 1000 is described, for example. Specifically, a program module 1093 describing each process executed by the monitoring apparatus 1 described in the above embodiment is stored in the hard disk drive 1031.

  Further, data used for information processing by the monitoring program is stored in the hard disk drive 1031 as the program data 1094, for example. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the hard disk drive 1031 to the RAM 1012 as necessary, and executes the above-described procedures.

  Note that the program module 1093 and the program data 1094 related to the monitoring program are not limited to being stored in the hard disk drive 1031, but are stored in a removable storage medium, for example, and read out by the CPU 1020 via the disk drive 1041 or the like. May be. Alternatively, the program module 1093 and the program data 1094 related to the monitoring program are stored in another computer connected via a network such as a LAN or a WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070. May be.

  As mentioned above, although embodiment which applied the invention made | formed by this inventor was described, this invention is not limited with the description and drawing which make a part of indication of this invention by this embodiment. That is, other embodiments, examples, operational techniques, and the like made by those skilled in the art based on this embodiment are all included in the scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Monitoring apparatus 11 Input part 12 Output part 13 Communication control part 14 Storage part 14a Training data 15 Control part 15a Acquisition part 15b Training part 15c Estimation part 15d Identification part 15e Judgment part B Radio signal transmitter C Video sensor R Monitoring range h Person

Claims (6)

An acquisition unit that acquires a plurality of videos taken from different locations within a predetermined monitoring range at the same time, and sensor data that is detected by identifying a person within the monitoring range;
A training unit that generates training data that combines the position of the same person shown in each of the acquired images and the sensor data that has been detected at the same time;
Based on the training data, an estimation unit that calculates an estimated value of the sensor data at the position of the person shown in each of the videos acquired in a continuous time series;
A specifying unit that specifies identification information for identifying a person in the video based on a time-series change in similarity between the calculated estimated value and the sensor data acquired simultaneously with the acquired video;
Among the identification information of the person in each video specified before the conversion as the identification information of the same person to be superimposed when the position of the person shown in each of the plurality of videos is converted into a position within the monitoring range A determination unit that determines a large number of duplicates;
A monitoring device comprising:   The monitoring apparatus according to claim 1, wherein the acquisition unit acquires videos obtained by shooting a person within the monitoring range from a plurality of places having different shooting directions and distances to the person.   The monitoring apparatus according to claim 1, further comprising an output unit that presents identification information of the person determined by the determination unit.   The monitoring device according to claim 1, wherein the sensor data indicates a strength of a radio signal received from a radio signal transmitter by a mobile terminal possessed by a person within the monitoring range. .   The estimation unit divides a position in the video into a plurality of cells, and calculates an average value of the sensor data of the training data at a position belonging to the same cell as an estimated value of the sensor data at a position belonging to the cell. The monitoring device according to any one of claims 1 to 4, wherein A monitoring method executed by a monitoring device,
An acquisition step of acquiring a plurality of videos taken from different locations within a predetermined monitoring range at the same time and sensor data detected by identifying a person within the monitoring range in time series,
A training step for generating training data combining the position of the same person shown in each of the acquired images and the sensor data obtained by detecting the person simultaneously;
Based on the training data, an estimation step of calculating an estimated value of the sensor data at the position of the person shown in each of the acquired images,
A specifying step of identifying identification information for identifying a person in the video based on a time-series change in similarity between the calculated estimated value and the sensor data acquired simultaneously with the acquired video;
Among the identification information of the person in each video specified before the conversion as the identification information of the same person to be superimposed when the position of the person shown in each of the plurality of videos is converted into a position within the monitoring range A determination step of determining what has a large number of duplicates;
The monitoring method characterized by including.

Images