JP2020102135A - Tracking system, tracking processing method, and program - Google Patents

Abstract

To enable an object in a facility to be tracked while taking privacy into account.SOLUTION: A tracking system is constituted by comprising: a first object detection unit for generating first object tracking information based on the result of detection of an object based on the detection information of an object detection sensor provided in correspondence to a divided area of type 1 among divided areas in a facility and detection of the position of the detected object; a second object detection unit for detecting an object in a divided area of type 2 among the divided areas in the facility on the basis of an image recognition process that utilizes an imaged image obtained by an imaging device provided in correspondence to the divided area of type 2, and generating second object tracking information based on the detection result; and an object management unit for associating the first and second object tracking information of the same object having moved a boundary between the divided areas of type 1 and type 2 with each other.SELECTED DRAWING: Figure 11

Description

The present invention relates to a tracking system, a tracking processing method, and a program.

There is known a technique capable of monitoring a person staying in a facility in real time by using captured images of a plurality of cameras (for example, refer to Patent Document 1).

JP, 2018-85597, A

In operating a facility for various purposes, there is a demand to track the movement of the facility user to grasp the flow line in the facility, acquire position information, and grasp the whereabouts. For example, examples include grasping the current position of an employee in an office building, grasping the flow line, and grasping the flow line for security and marketing in a commercial facility such as a shopping mall where customers move widely. Such movement tracking of facility users can be realized by installing a large number of imaging devices such as surveillance cameras.
However, depending on the facility, the area is divided into a private part and a common part, such as a tenant office building or a shopping mall in which a tenant store is located, and the management and operation of privacy protection and security etc. are different between the private part and the common part. There are cases. For example, it is not preferable to install a surveillance camera in the private part because the privacy of the facility user is important, while in the common part, the surveillance camera (that is, the imaging device) should be installed from the viewpoint of ensuring the safety of the facility user. We can mention the case of being liked. In such a case, it is not preferable to track the facility users in the entire facility by the image capturing device, because it is possible to identify an individual, and it is not preferable especially in the exclusive section.

The present invention has been made in view of such circumstances, and an object thereof is to enable tracking of an object in a facility while considering privacy.

One aspect of the present invention for solving the above-described problem is detection of an object based on detection information of an object detection sensor provided corresponding to a first-type divided area among divided areas in a facility and the detected object. The first object detection unit that generates the first object tracking information based on the result of the detection of the position and the image pickup device that is provided corresponding to the second type of divided area among the divided areas in the facility. A second object detection unit that detects an object in the second type of segmented area based on image recognition processing using the captured image, and generates second object tracking information based on the detection result; An object management unit that associates the first object tracking information and the second object tracking information of the same object that has moved the boundary between the first type of segmented area and the second type of segmented area. It is a tracking system.

Further, according to one embodiment of the present invention, detection of an object and detection of a position of the detected object based on detection information of an object detection sensor which is provided corresponding to a first-type divided area among divided areas in a facility. A first object detection step of generating first object tracking information based on the result of the above, and a captured image obtained by an imaging device provided corresponding to a second type of divided area among the divided areas in the facility. A second target object detection step of detecting a target object in the second type of segmented area based on image recognition processing utilizing, and generating second target object tracking information based on the detection result; A tracking processing method comprising: an object management step of associating the first object tracking information and the second object tracking information of the same object that has moved a boundary between the divided area and the second type of divided area. is there.

Further, one aspect of the present invention is a program for causing a computer provided in a tracking system to function as each of the above units.

As described above, according to the present invention, it is possible to obtain the effect that the target can be traced in the facility while considering privacy.

It is a figure which shows the example of the whole structure of the tracking system in this embodiment. It is a figure which shows the structural example of the imaging correspondence tracking server in this embodiment. It is a figure which shows an example of the 2nd target object tracking information in this embodiment. It is a figure which shows the structural example of the tracking server corresponding to an object detection sensor in this embodiment. It is a figure which shows an example of the 1st target object tracking information in this embodiment. It is a figure which shows the structural example of the integrated tracking server in this embodiment. It is a figure which shows an example of the common target object information in this embodiment. It is a figure which shows the structural example of the facility management server in this embodiment. It is a flow chart which shows an example of a processing procedure which an image pick-up correspondence pursuit server in this embodiment performs corresponding to pursuit of a facility user corresponding to one common part. It is a flow chart which shows an example of a processing procedure which a tracking server corresponding to an object detection sensor in this embodiment performs. It is a flowchart which shows the example of a processing procedure which the integrated tracking server in this embodiment performs.

FIG. 1 shows an example of the overall configuration of the tracking system of this embodiment. The tracking system shown in the figure tracks the facility users who individually move in the facility FAC.
In the figure, among the divided areas in the facility FAC, there are two common areas PAR-1, PAR-2 (an example of the second type of divided areas) and a certain exclusive area TAR (of the first type). An example of a section area) is shown. The exclusive part TAR is adjacent to each of the common parts PAR-1 and PAR-2.
In the following description, common parts PAR-1 and PAR-2 will be referred to as common parts PAR unless otherwise specified.
In the following, a case where there are two shared parts PAR and one dedicated part TAR will be described as an example, but the number of shared parts PAR and dedicated parts TAR in the facility FAC is not particularly limited, and is appropriately set. The required number may be provided.

Further, in the figure, an example is shown in which the image capturing device 10-1 is provided in the shared portion PAR-1 and the image capturing device 10-2 is provided in the shared portion PAR-2.
In the following description, the imaging devices provided in the common part PAR, including the imaging device 10-1 and the imaging device 10-2, will be referred to as the imaging device 10 unless otherwise specified.
As the image pickup device, for example, a surveillance camera or a WEB camera can be used.

The image pickup apparatus 10-1 is provided so as to pick up an image of a place within a predetermined range in the shared portion PAR-1, and thus picks up an image of a facility user existing in the predetermined range. Although one imaging device 10-1 is shown in the shared portion PAR-1 in the figure, the shared portion PAR-1 includes a plurality of imaging devices 10-1 so as to capture images at different places. The imaging device 10 may be provided. As a specific example, when the shared portion PAR-1 is a passage, a plurality of image pickup devices 10 are provided so as to pick up images of a predetermined range of space in the passage at certain intervals along the extending direction of the passage. Good. Further, the number of the plurality of image pickup devices 10, the respective installation locations, and the image pickup ranges may be appropriately changed according to the size of the common part PAR, the installation state of the shield, and the like.
In addition, the imaging device 10-2 is provided so as to image a place within a predetermined range in the shared portion PAR-2, and thus images the facility user existing in the predetermined range. Also for the shared portion PAR-2, a plurality of image pickup devices 10 may be provided so as to pick up images of different places including the image pickup device 10-2.
Information on the captured image obtained by each of the image capturing apparatuses 10 is transmitted to the image capturing support tracking server 100. That is, the information of the captured images obtained by the plurality of image capturing devices is collected in the image capturing correspondence tracking server 100.

An object detection sensor 20 is provided in the exclusive portion TAR. The object detection sensor 20 may be, for example, a sensor using a LiDAR (light detection and ranging) technique. LiDAR is a technique for detecting an object based on the time until the laser light emitted from the sensor is reflected by the object and returns.

It should be noted that, in the same figure, an example in which two object detection sensors 20 are provided in the exclusive portion TAR is shown, but the number of object detection sensors 20 provided in the exclusive portion TAR and the arrangement position of the object detection sensors 20 are shown. Etc. may be determined according to, for example, the size of the exclusive part TAR and the installation status of the shield.
Object detection information obtained by the object detection sensor 20 performing object detection is transmitted to the object detection correspondence tracking server 200. That is, the object detection correspondence tracking server 200 collects the object detection information obtained by the plurality of object detection sensors 20.

As described above, the picked-up image obtained by picking up the image of the shared portion PAR by the image pickup apparatus 10 is transmitted to the image pickup-capable tracking server 100. The image capturing correspondence tracking server 100 performs image recognition processing on the captured image transmitted from the image capturing apparatus 10 to extract a person from the captured image and perform personal recognition based on the extracted feature amount of the face, clothing, etc. of the person. I do. In such personal recognition, even when a person is moving, the common part PAR can detect that the person is the same before and after the movement based on the feature amount. The imaging-compatible tracking server 100 generates tracking information (second object tracking information) with the facility user as a tracking target based on the result of individual recognition of the facility user in this way. Based on the second object tracking information, the imaging-compatible tracking server 100 can individually track the facility users detected in the shared part PAR.
Further, in the present embodiment, the position of the facility user to be tracked can be recognized with accuracy corresponding to the imaging range based on the fact that the imaging ranges of the imaging devices 10 are different. Alternatively, by referring to a table in which the coordinates (coordinates) in the location of the shared portion PAR to be imaged are associated with each other in the imaged image obtained by imaging with the imaging device 10, the facility user of the tracking target is identified. The position coordinates in the shared portion PAR may be recognized. As will be described later, such position coordinate information may be included in the second object tracking information.
The imaging-compatible tracking server 100 stores the generated second object tracking information. Further, the imaging-compatible tracking server 100 transmits the stored second object tracking information to the integrated tracking server 300.
The imaging-capable tracking server 100 may also store the captured image transmitted from the imaging device 10.

Further, the object detection information obtained by the object detection sensor 20 is transmitted to the object detection correspondence tracking server 200. The object detection correspondence tracking server 200 performs shape recognition on the object detected by the object detection information transmitted from the object detection sensor 20. The object detection correspondence tracking server 200 discriminates and detects a person from the objects existing in the exclusive portion TAR based on the shape recognition result, and detects each of the detected persons according to the characteristic of the recognized shape. Can be individually recognized. The object detection correspondence tracking server 200 treats the person recognized in this way as a facility user in the exclusive section TAR.
Further, the object detection correspondence tracking server 200 detects, based on the object detection information, the direction in which the object existing in the exclusive portion TAR starting from the object detection sensor 20 is located and the distance from the object detection sensor 20 to the object. it can. Based on such a direction and position of the object, the position of the object can be specified as coordinates. That is, the object detection correspondence tracking server 200 detects the position coordinates of the object in the exclusive portion TAR. Further, when the object detected by the object detection information is moving, the object detection correspondence tracking server 200 detects that the feature amount of the shape of the object before and after the movement matches or before and after the movement. The object can be tracked based on the correlation of the position coordinates of the object. That is, the object detection correspondence tracking server 200 can track each facility user based on the position of each facility user detected by the exclusive unit TAR.
The object detection correspondence tracking server 200 stores the generated first object tracking information. Further, the object detection correspondence tracking server 200 transmits the stored first object tracking information to the integrated tracking server 300. The object detection-corresponding tracking server 200 may also store a sensor detection image formed based on the object detection sensor 20.

The integrated tracking server 300 uses the second object tracking information received from the imaging compatible tracking server 100 and the first object tracking information received from the object detection compatible tracking server 200 and shared in the facility FAC. Information management is performed so that it can be tracked for each facility user in response to the situation of moving between the department PAR and the exclusive department TAR.

As can be understood from the above description, in the present embodiment, the facility user is tracked by the individual recognition based on the captured image output by the imaging device 10 in the common area PAR, so that the common area PAR is detected. Individuals will be identified for each facility user.
On the other hand, in the exclusive unit TAR, a person is tracked based on the object detection information by the optical technique. In the detection of a person based on the object detection information, for example, the shape of a person is recognized, but feature amounts such as a person's face and clothes are not recognized. For this reason, in the exclusive department TAR, the detected facility users can be individually discriminated, but they are not identified.
For example, as can be seen in the case of a tenant office building or a tenant store, generally, in the private part TAR, the privacy of the facility user tends to be more important than in the common part PAR. Therefore, with respect to the exclusive department TAR, it is possible to take the idea that it is not preferable to identify an individual when tracing the facility user.
On the other hand, the common part PAR can be considered to be more public than the exclusive part TAR, and the importance of privacy may be low in consideration of crime prevention and the like. Therefore, in the common area PAR, when tracing the facility users, it is possible to allow the individual to be specified.
According to the configuration of the present embodiment, in accordance with the above concept, it is possible to identify the individual facility user in the common area PAR, and the individual information in the exclusive portion TAR so that the individual person is not identified. By managing and operating, the privacy of the facility user in the facility FAC can be appropriately managed.

Further, FIG. 1 shows a state in which the facility user PS-1 tries to move from the common area PAR-1 to the exclusive area TAR. Also, in the figure, a state is shown in which the facility user PS-2 is about to move from the exclusive section TAR to the common section PAR-2. That is, the facility users PS-1 and PS-2 are facility users who are going to move along the boundary between the shared part PAR and the proprietary part TAR. In the following description, facility users PS-1 and PS-2 will be referred to as facility users PS unless otherwise specified.

As the facility user PS-1 moves from the shared portion PAR-1 to the exclusive portion TAR, the individual as the facility user PS-1 is no longer detected in the shared portion PAR-1. On the other hand, the facility user PS-1 is detected in the exclusive section TAR, and tracking is started.
The shared part PAR and the exclusive part TAR track the facility users by different methods. In particular, in tracking the facility user in the exclusive department TAR, the individual as the facility user is not specified. Therefore, if no special measures are taken, at the timing when the facility user PS moves the boundary between the shared part PAR and the exclusive part TAR, the facility user PS detected before moving the boundary and the boundary It is not possible to confirm the agreement with the facility user PS detected after moving to.
Therefore, in the present embodiment, in response to the case where the facility user PS moves along the boundary between the shared part PAR and the exclusive part TAR, the integrated tracking server 300 tracks the facility user PS before the move. The tracking information indicating the status and the tracking information indicating the tracking status of the facility user PS after the movement are associated with each other. As a result, it is possible to track the facility user in the facility FAC in response to the movement of the facility user between the shared part PAR and the exclusive part TAR.

Further, the facility management server 400 is shown in FIG. The facility management server 400 is a server that performs management corresponding to the facility FAC. The facility management server 400 controls the operation of various facilities 500 provided in the facility FAC. The specific example of the equipment 500 shown in the figure is not particularly limited. For example, the facility 500 may include an air conditioner, a lighting device, a display device as a digital signage, an elevator or an escalator facility, and the like.
Further, the facility management server 400 can control the facility 500 in the facility FAC based on the current tracking status of the facility user, the history of the tracking status of the facility user, and the like. This point will be described later.

FIG. 2 shows an example of the configuration of the imaging-compatible tracking server 100. The imaging-compatible tracking server 100 in the figure includes a communication unit 101, a control unit 102, and a storage unit 103.
The communication unit 101 communicates with the imaging device 10 and the integrated tracking server 300.
The control unit 102 executes control in the imaging-compatible tracking server 100. The function as the control unit 102 is realized by a CPU (Central Processing Unit) included in the imaging-compatible tracking server 100 executing a program. The control unit 102 includes a second object detection unit 121.

The second object detection unit 121 detects the facility user in the common area PAR based on the image recognition process using the captured image obtained from the imaging device 10 provided corresponding to the common area PAR. Depending on the image recognition processing, individual recognition is performed for each facility user.
In addition, the second object detection unit 121 can estimate (detect) the position coordinates of the recognized individual in the shared portion PAR. For example, the second object detection unit 121 can estimate the position coordinate of the individual in the shared portion PAR based on the position of the individual shown in the image by the image recognition processing. Alternatively, since the coordinates in the shared part PAR in which the image pickup apparatus 10 is installed are known, the position coordinates of the individual can be estimated as the vicinity of the coordinates of the image pickup apparatus 10.
Therefore, the 2nd target object detection part 121 can track for every facility user in the common part PAR based on the individual recognition result for every facility user.

When the same facility user that was detected at the previous time is also detected at the next time, the second target object detection unit 121 and the second target object identifier indicating the same facility user and the next Second object tracking information storing a time stamp indicating a time is generated. Then, the second object detection unit 121 stores the generated second object tracking information in the second object tracking information storage unit 131. According to the second object tracking information stored in this way, the facility user moving in the common area PAR is recognized as an individual. That is, the facility user in the common area PAR can be tracked by recognizing the individual.

The storage unit 103 stores various kinds of information corresponding to the imaging correspondence tracking server 100. The storage unit 103 includes a second object tracking information storage unit 131.
The second target tracking information storage unit 131 stores the second target tracking information. The second object tracking information is information about the facility user detected in the common area PAR.
FIG. 3 shows an example of the second object tracking information stored in the second object tracking information storage unit 131 in a database format, for example. The second object tracking information of one record in the figure is information corresponding to a certain time of one facility user.

One piece of second object tracking information includes a region of the image pickup device identifier, the second object identifier, the time stamp, the estimated position coordinate, and the feature amount data.
The area of the imaging device identifier stores the imaging device identifier indicating the imaging device 10 that has detected the detected facility user. In the figure, an example is shown in which "10-1" is stored as the imaging device identifier of the imaging device 10-1 (FIG. 1).
As described above, the plurality of image pickup devices 10 are provided so as to pick up images at different predetermined locations in the common part PAR. Therefore, depending on the image pickup device identifier that uniquely indicates the image pickup device 10, it is possible to specify a predetermined place in the corresponding shared portion PAR. That is, the image pickup device identifier functions as position information indicating a predetermined location of the shared portion PAR corresponding to the image pickup range of the corresponding image pickup device 10.
The area of the second target object identifier stores the second target object identifier given to the facility user as the detection target detected by the shared portion PAR. An individual is recognized when the facility user is detected by the imaging device 10. Therefore, the second object identifier may be assigned to each facility user so as to be unique to each recognized individual.
The time stamp area stores a time stamp indicating the time when the facility user is detected.
In the area of the estimated position coordinate, the position coordinate in the shared part PAR estimated for the facility user (recognized individual) based on the captured image as described above is stored. In the figure, an example is shown in which the estimated position coordinates are expressed as three-dimensional coordinates in the x-axis direction, the y-axis direction, and the z-axis direction.
The area of the feature amount data stores data (feature amount data) indicating the feature amount extracted by image recognition for the facility user as the detection target. Individuals can be identified depending on the feature amount data.

In the above example, the individual recognition of the facility user in the common area PAR is performed by analyzing the captured image. However, for example, the employee ID etc. provided at the entrance/exit of the entrance/exit facility provided in the common area PAR It may be performed based on the information about the individual facility user, which is read by a card reader compatible with an IC card. In this case, the estimated position coordinates may be estimated based on the position where the card reader is installed in the shared part PAR. For example, the position of the facility user in this case can be treated as being approximately the same as the position where the card reader is installed. Therefore, the estimated position coordinates may be, for example, the coordinates of a predetermined position based on the position where the card reader is installed. In this case, the estimated position coordinate may indicate the coordinate as one point, or may indicate the coordinate range having a certain extent of space.

FIG. 4 shows an example of the configuration of the object detection compatible tracking server 200. The object detection correspondence tracking server 200 in the figure includes a communication unit 201, a control unit 202, and a storage unit 203.
The communication unit 201 communicates with the object detection sensor 20 and the integrated tracking server 300.
The control unit 202 executes control in the object detection correspondence tracking server 200. The function as the control unit 202 is realized by the CPU included in the object detection compatible tracking server 200 executing a program. The control unit 202 includes a first object detection unit 221.
The first object detection unit 221 uses the object detection result obtained by the object detection sensor 20 provided corresponding to the exclusive unit TAR to detect a facility user (an example of an object) in the exclusive unit TAR, Coordinates as the detected position of the facility user are detected. That is, the first target detection unit 221 can perform tracking in accordance with the movement of each facility user in the exclusive unit TAR.
In addition, the first object detection unit 221 may determine the attribute of the detected facility user based on the recognized shape of the detected facility user.
For example, the first target object detection unit 221 estimates the height based on the shape recognized for the detected facility user. Depending on the estimated height, the first object detection unit 221 may determine that the detected facility user has the attribute of “adult” or “child”.
In addition, the first object detection unit 221 determines the attributes such as “in a stroller”, “in a wheelchair”, and “falling down” for the facility user based on the recognized shape. You may In addition, the first object detection unit 221 may recognize a group of facility users who are acting together by a plurality of people, and may include the recognition result of the group in the attribute information. The determination of such an attribute may be realized by machine learning, for example.

The storage unit 203 stores various kinds of information corresponding to the object detection correspondence tracking server 200. The storage unit 203 includes a first object tracking information storage unit 231.
The first target object tracking information storage unit 231 stores the first target object tracking information. The first target object tracking information is information about the facility user of the tracking target detected by the exclusive unit TAR.

FIG. 5 shows an example of the first object tracking information stored in the first object tracking information storage unit 231 in a database format, for example. The first object tracking information of one record in the figure is information corresponding to a certain time for one facility user.
One piece of first object tracking information includes a detection device identifier, a first object identifier, a time stamp, object position information, and an area of attribute information.
The detection device identifier area stores a detection device identifier indicating the object detection sensor 20 that has detected the detected facility user. In the figure, an example in which “20-1” is stored as the detection device identifier of the object detection sensor 20-1 (FIG. 1) is shown.
For example, the corresponding exclusive unit TAR can be identified by the detection device identifier. When a plurality of object detection sensors 20 are provided in the exclusive portion TAR, the corresponding area portion in the exclusive portion TAR can be specified by the detection device identifier.
The area of the first target object identifier stores the first target object identifier given to the facility user as a detection target detected by the exclusive unit TAR. Since the individual is not recognized in the detection of the facility user by the object detection sensor 20, the first object identifier may be, for example, a serial number or the like according to the order in which the detection by the object detection tracking server 200 is started.
The time stamp area stores the time when the facility user was detected (time stamp).
The area of the object position information stores the object position information, which is information indicating the position of the facility user detected at the time indicated by the corresponding time stamp by coordinates. In the figure, an example is shown in which the coordinates are expressed as three-dimensional coordinates in the x-axis direction, the y-axis direction, and the z-axis direction.
The attribute information area stores attribute information indicating the attribute determined by the first object detection unit 221 for the detected facility user.

For example, when the facility user with the first object identifier “23869” shown in the figure is detected at the next time as well, the facility user with the same first object identifier “23869” is newly added. One target tracking information is generated and stored in the first target tracking information storage unit 231. In the first object tracking information stored in this way, the value of "time stamp" is updated corresponding to the current time. In addition, the first object detection unit 221 may change the determination result that the facility user has a different attribute from the previous one in response to a change in the detected facility user. In such a case, the first target object detection unit 221 may update the attribute information so that the changed attribute is indicated. Even when the attribute information is updated, the acquired original data can be saved and the history can be reviewed later.
As described above, the first object tracking information is generated and stored in the first object tracking information storage unit 231 every time the facility user of the exclusive unit TAR is detected corresponding to each time. Among the first object tracking information stored in this way, depending on the plurality of first object tracking information storing the same first object identifier, the facility user in the exclusive section TAR for each time indicated by the time stamp Position will be indicated. That is, the first object tracking information indicates the movement trajectory of the facility user in the exclusive unit TAR, and the tracking of the facility user in the exclusive unit TAR is realized.

A configuration example of the integrated tracking server 300 will be described with reference to FIG. The integrated tracking server 300 shown in the figure includes a communication unit 301, a control unit 302, and a storage unit 303.
The communication unit 201 communicates with the object detection sensor 20 and the integrated tracking server 300.
The control unit 202 executes control in the object detection correspondence tracking server 200. The function as the control unit 202 is realized by the CPU included in the object detection compatible tracking server 200 executing a program. The control unit 202 includes an object management unit 321.
The target object management unit 321 associates the first target object tracking information and the second target object tracking information regarding the common facility user who has moved the boundary between the exclusive unit TAR and the common unit PAR as described below.

The storage unit 303 stores various kinds of information corresponding to the integrated tracking server 300. In the storage unit 303, the common object information storage unit 331 stores the common object information in a database format, for example. The common object information is information in which the first object identifier and the second object identifier for the same facility user are associated with each other.
FIG. 7 shows an example of common object information stored in the common object information storage unit 331. One record in the figure is common object information corresponding to one movement between the common part PAR and the exclusive part TAR of the facility user. In the common object information of the figure, the facility user of the first object identifier “23869” shown in FIG. 4 and the facility user of the second object identifier “U123456” shown in FIG. 5 are the same (common ) Is shown as an example.
The common object information in the figure includes areas of a common object identifier, a first object identifier, a second object identifier, and a correspondence confirmation time stamp.
The area of the common object identifier stores the common object identifier assigned as corresponding to both the first object identifier and the second object identifier given to the corresponding one facility user.
The area of the first object identifier stores the first object identifier assigned to the corresponding one facility user.
The area of the second object identifier stores the second object identifier given to the corresponding one facility user.
The area of the correspondence confirmation time stamp stores the time when the corresponding common object information is generated. In the present embodiment, the common object information corresponding to one facility user is the object management unit of the integrated tracking server 300 every time the corresponding facility user moves between the shared section PAR and the exclusive section TAR. It is newly generated by 321 and is stored in the common object information storage unit 331. The correspondence confirmation time stamp indicates the time when the corresponding common object information is generated in response to the corresponding facility user moving between the shared portion PAR and the exclusive portion TAR.

A configuration example of the facility management server 400 will be described with reference to FIG. 8. The facility management server 400 includes a communication unit 401, a control unit 402, and a storage unit 403.
The communication unit 401 communicates with the integrated tracking server 300 and the facility 500 to be controlled in the facility FAC.
The control unit 402 executes various controls in the facility management server 400. The function as the control unit 402 is realized by the CPU of the facility management server 400 executing a program. The control unit 402 includes an equipment control unit 421. The equipment control unit 421 controls the equipment 500 provided in the facility FAC.
The storage unit 403 stores various information related to the facility management server 400.

With reference to the flowchart of FIG. 9, an example of a processing procedure executed by the imaging-compatible tracking server 100 according to the present embodiment in association with the tracking of the facility user based on the imaged image of one imaging device 10 in a certain shared part PAR. Will be described.
Step S101: In the imaging-compatible tracking server 100, the second object detection unit 121 detects an individual (facility user) by performing image recognition on the imaged image of the imaging device 10 provided in the shared portion PAR. There is. Then, the second target object detection unit 121 determines whether or not the detection of the facility user has started. For example, when a certain facility user moves from another place to a place in the corresponding shared portion PAR, the second object detection unit 121 starts detecting the facility user.

Step S102: In response to the start of the detection of the facility user in step S101, the feature amount data is obtained by the image recognition process for the facility user whose detection is started. Therefore, the second object detection unit 121 stores the second object tracking information (the second object tracking information of the same facility user (object)) that stores the same feature amount data obtained this time as the second object tracking information. In the object tracking information storage unit 131, it is determined whether or not it has already been stored. Therefore, the second target object detection unit 121 searches the second target object tracking information storage unit 131 for the second target object tracking information of the same facility user.

Step S103: When it is determined in step S102 that the second object tracking information of the same facility user is stored in the second object tracking information storage unit 131, the facility user detected this time is detected this time. A facility user who has existed at a place in the corresponding shared area PAR before being performed.
In this case, the second target object detection unit 121 acquires the second target object identifier stored in the second target object tracking information of the same facility user searched in the determination process in step S102. The acquired second target object identifier is the second target object identifier that should be stored in the second target object tracking information generated in response to the current start of detection.

Step S104: When it is determined in step S102 that the second object tracking information of the same facility user is not yet stored in the second object tracking information storage unit 131, the facility user who has started the detection is This is the first time we have moved to the corresponding shared area PAR.
Therefore, the second target object detection unit 121 newly issues a second target object identifier indicating the facility user whose detection has been started.

Step S105: The second object detection unit 121 starts generation of second object tracking information corresponding to the facility user whose detection has been started. After that, the second object detection unit 121 generates the second object tracking information at regular time intervals and stores the second object tracking information in the second object tracking information storage unit 131.
When executing the processing of step S105 after the processing of step S103, the second object detection unit 121 stores the second object identifier acquired in step S103 in the second object tracking information generated thereafter. To do. In addition, when the second target object detection unit 121 executes the process of step S105 after the process of step S104, the second target object identifier generated in step S104 is added to the second target object tracking information generated thereafter. Store.

Step S106: The second object detection unit 121 performs integrated tracking of the second object tracking information stored in the second object tracking information storage unit 131 after the detection is started in Step S101 at predetermined timings. It is transmitted to the server 300. The second object detection unit 121 may transmit, for example, untransmitted second object tracking information at a timing of every predetermined time, or a predetermined number of untransmitted second object tracking information are accumulated (buffering). ), the untransmitted second object tracking information may be transmitted.

Step S107: After the process of step S105, the second object detection unit 121 detects that the facility user whose detection has been started in step S101 can be detected at a place in the corresponding shared part PAR, Wait for transition to the undetectable state.
Step S108: When the facility user whose detection is started in step S201 becomes incapable of detection at a place in the corresponding shared portion PAR, the second object detection unit 121 causes the second object detection unit 121 to start from step S105. 2 Stop generating the object tracking information.

With reference to the flowchart of FIG. 10, a processing procedure executed by the object detection correspondence tracking server 200 of the present embodiment in correspondence with the tracking of the facility user based on the object detection information of one object detection sensor 20 in the exclusive portion TAR. An example will be described.
Step S201: In the object detection correspondence tracking server 200, the first target object detection unit 221 performs shape recognition by using the object detection information of the object detection sensor 20 provided in the exclusive unit TAR, and thus the person (facility utilization) Person) is detected. Then, the first target object detection unit 221 determines whether or not the detection of the facility user has started. For example, when a certain facility user moves from another place to the location of the corresponding exclusive unit TAR, the first target object detection unit 221 starts to detect the target object as the facility user. Become.

Step S202: When the detection of the facility user is started in step S201, the first object detection unit 221 newly issues a first object identifier as a serial number.
Step S203: The first object detection unit 221 starts generation of the first object tracking information corresponding to the facility user whose detection has been started. After that, the first target object detection unit 221 generates the first target object detection unit 221 at regular time intervals and stores the first target object tracking information storage unit 231. The second object detection unit 121 stores the first object identifier issued in step S202 in the generated first object tracking information.
Step S204: The first target object detection unit 221 performs integrated tracking of the first target object tracking information stored in the first target object tracking information storage unit 231 after the detection is started in Step S201, at predetermined timings. It is transmitted to the server 300.
Step S205: After the processing of step S204, the first object detection unit 221 detects that the facility user whose detection has been started in step S201 can be detected at a location within the corresponding exclusive section TAR, Wait for transition to the undetectable state.
Step S206: When the facility user whose detection is started in step S201 becomes incapable of detection at a place in the corresponding exclusive section TAR, the first target object detection unit 221 starts the first object detection unit 221 from step S203. 1 Stop the generation of target tracking information.

An example of the processing procedure executed by the integrated tracking server 300 will be described with reference to FIG. 11.
Step S301: In the integrated tracking server 300, the object management unit 321 is waiting for the object tracking information to be received. That is, the object management unit 321 waits for reception of the second object tracking information transmitted from the imaging-compatible tracking server 100. Further, the object management unit 321 waits for reception of the first object tracking information transmitted from the object detection correspondence tracking server 200.

Step S302: The object management unit 321 stores the received object tracking information in the storage unit 303 in response to the reception of the object tracking information.

Step S303: The object management unit 321 regards the object tracking information stored in step S302 as a pair of the first object tracking information and the second object tracking information corresponding to the movement of the same facility user (object). It is determined whether or not there is object tracking information.
The target object management unit 321 determines the first target object tracking information and the second target object tracking information, which satisfy the following pair establishment conditions, as a pair corresponding to the movement of the same facility user (target object).

The pair establishment condition is to satisfy all of the following three condition elements 1, 2, and 3.
Condition element 1: As a combination of the first target tracking information and the second target tracking information, one target tracking information is first generated in response to the start of detection by the facility user, and the other target tracking information. The object tracking information is the last one generated when the generation is stopped in response to the determination that detection is impossible.
Conditional element 2: The difference between the times indicated by the time stamps stored in the first object tracking information and the second object tracking information is within a certain range that can be regarded as the same time.
Condition element 3: a location corresponding to the imaging range of the imaging device 10 indicated by the imaging device identifier stored in the first object tracking information, and a position (coordinates) indicated by the object position information stored in the first object tracking information. ) Corresponds to the flow line when moving the boundary between the exclusive part TAR and the common part PAR.
In order to determine whether or not the condition element 3 is satisfied, for example, the storage unit 103 of the imaging-compatible tracking server 100 stores location information indicating a location corresponding to the imaging range of each imaging device 10. Every time, the object management unit 321 may refer to the stored location information.
Even if there are a plurality of places (doorways) where facility users can move between the shared part PAR and the exclusive part TAR, by adding the conditional element 3 relating to such a positional relationship, Corresponding to the movement of the facility user at the entrance/exit, it is possible to appropriately determine the first object tracking information and the second object tracking information of the pair corresponding to the movement of the same facility user.
As an example, when the facility user moves from the exclusive department TAR to the common area PAR at a certain entrance/exit, the facility user is generated lastly in response to having exited from the exclusive department TAR. One object tracking information and the second object tracking information initially generated in response to entering the common area PAR are obtained. The first object tracking information generated last and the second object tracking information finally generated store time stamps indicating the time differences that can be regarded as the same time. Further, the coordinates indicated by the object position information stored in the first object tracking information and the location information of the imaging device corresponding to the second object tracking information sandwich the boundary between the exclusive portion TAR and the common portion PAR. Then, the relationship is in the same flow line corresponding to the position of the doorway.
Therefore, regarding the first object tracking information and the second object tracking information satisfying the above-mentioned pair formation condition, the facility user moves from the exclusive section TAR to the common section PAR at a specific entrance/exit, Alternatively, it can be obtained only by moving from the shared part PAR to the exclusive part TAR.

Step S304: If the object management unit 321 determines in step S303 that there is a pair of the first object tracking information and the second object tracking information that satisfies the pair establishment condition, the object managing unit 321 further performs the following determination.
That is, the target object management unit 321 stores the same second target object identifier that is stored in the second target object tracking information, which is one of the pair objects determined in step S303, as the same facility user (target object). Common object information of object) is already stored in the common object information storage unit 331.
Step S305: When it is determined in step S304 that the common object information of the same facility user (object) is not yet stored in the common object information storage unit 331, the object management unit 321 determines that the common object information is common. Newly issue target information.
Step S306: On the other hand, when it is determined in step S304 that the common target object information of the same facility user (target object) is already stored in the common target object information storage unit 331, the target object management unit 321 , The common object identifier of the common object information of the same facility user (object) already stored in the common object information storage unit 331 is acquired.
Step S307: After the processing of step S305 or step S306, the object management unit 321 generates common object information. In the generated common object information, the first object identifier and the second object stored in each of the first object tracking information and the second object tracking information determined to be a pair in step S303. And an object identifier. Further, the common object information obtained in step S305 or step S306 is stored in the generated common object information. Further, in the generated common object information, for example, a correspondence confirmation time stamp indicating the time corresponding to the generation timing of the current common object information is stored. The correspondence confirmation time stamp is, for example, a time based on the time indicated by the time stamps stored in each of the first target object tracking information and the second target object tracking information determined to be a pair in step S303. May be
Step S308: The target object management unit 321 causes the common target object information storage unit 331 to store the common target object information generated in step S307.

In this way, by executing the processing of FIGS. 9 to 11, for the same one facility user, the second object tracking information indicating the detection result in the common part PAR and the detection result in the exclusive part TAR. Correspondence with the first target object tracking information indicating is indicated by the common target object information. That is, in the facility FAC, it is possible to track the facility user who moves between the shared part PAR and the exclusive part.

For example, the integrated tracking server 300 or the facility management server 400 performs tracking for each facility user in the facility FAC based on the first target object tracking information, the second target object tracking information, and the common target object information obtained as described above. Information indicating the result (in-house user tracking information) can be obtained. Based on such in-facility user tracking information, it is possible to grasp the movement trajectory of each facility user in the facility FAC and the distribution of the facility users in the current and past facility FACs in the facility FAC.

Therefore, for example, the facility control unit 421 of the facility management server 400 can control a predetermined facility in the facility FAC, for example, based on the in-facility user tracking information.
As a specific example, the facility control unit 421 can determine the number of people in each area of the facility FAC, and can control the air conditioning equipment for each area according to the determined number of people.
In addition, the facility control unit 421 determines that the facility user is present in the vicinity of the display device for digital signage in the facility FAC, and then sends an advertisement or other content to the facility user. The equipment 500 as a display device can be controlled to display. In controlling the facility 500 in this way, for example, when the attribute information (FIG. 5) about the corresponding facility user is acquired, the display device displays the content corresponding to the attribute indicated by the attribute information. It may be controlled to occur. In addition, the display position of the content on the display device is changed according to the height of the facility user's eyes specified according to the attribute of the facility user near the display device so that the facility user can easily see the contents. Good. Specifically, when the attribute information indicates "adult", the equipment control unit 421 displays the attribute information as "child" or "child" when the attribute information indicates "adult". In the case of indicating that "you are in a wheelchair", you may control so that it may be displayed in line with an average lower line of sight.
In addition, the equipment control unit 421, in response to the acquisition of the attribute information such as "falling down", includes the information about the occurrence of the sudden illness and the information about the area where the sudden illness occurs in a predetermined notification destination. You may be notified.
Further, the facility control unit 421 classifies a facility user whose face cannot be individually recognized as a result of image recognition by the imaging device 10 in the common unit PAR, for example, into a suspicious person and, for example, a suspicious terminal device for security. The presence of the person may be notified.
Further, the in-facility user tracking information can be referred to by an operator of the institution FAC at a predetermined opportunity, and can be used for studying various measures in the institution FAC.

In the above embodiment, in associating the object tracking information before and after the movement for the facility user who has moved between the shared portion PAR and the exclusive portion TAR, condition elements 1, 2, The case where all 3 are satisfied is taken as an example. That is, in this case, the object tracking information corresponding to the last detection of the facility user in the section area before the movement and the object tracking information corresponding to the start of the detection of the facility user in the section area after the movement And the relationship between the positions indicated by and, and the relationship between the times.
However, for example, in a situation where the common area PAR and the exclusive area TAR can be moved to only one place and only one facility user can go in and out at a time in the movable place. However, the position is naturally limited. Therefore, in such a case, the condition element 3 relating to the positional relationship is omitted, and the object tracking information before the movement and the object tracking information after the movement are associated with each other in accordance with the satisfaction of the condition elements 1 and 2. May be done.
Further, the processing of the first object tracking information by the object detection compatible tracking server 200, the processing of the second object tracking information by the imaging compatible tracking server 100, and the generation of the common object information by the integrated tracking server 300 are almost real time. It can be executed at a good timing. Considering this, the object management unit 321 of the integrated tracking server 300 determines that the condition elements 1 and 2 are satisfied at any time without checking the correspondence relationship (condition element 2) of the time indicated by the time stamp. The common target object information may be generated by associating the first target object tracking information with the second target object tracking information.

The configuration of the tracking system of the present embodiment can be applied without any particular limitation on the use of the building. The configuration of the tracking system of the present embodiment can be applied to, for example, office buildings, shopping malls, facilities for other purposes such as hospital facilities, and the like.

The object to be tracked by the tracking system of this embodiment may be a person other than a person such as a specific animal or luggage.

The tracking of one facility user in the present embodiment is performed by dividing the object into predetermined time segments, for example, every day, every predetermined time zone in one business day. It may be implemented to manage information. In this case, the imaging-compatible tracking server 100, the object detection-compatible tracking server 200, and the integrated tracking server 300, for example, also correspond to the facility user who has been detected in the previous time zone division in response to the start of a predetermined time zone division. Alternatively, the first object tracking information, the second object tracking information, and the common object information may be generated by processing as a newly detected facility user.

In the above embodiment, the imaging compatible tracking server 100, the object detection compatible tracking server 200, and the integrated tracking server 300 are configured as independent servers. However, at least a part of these servers may be configured as one integrated server.

It should be noted that a program for realizing the functions of the above-described imaging compatible tracking server 100, object detection compatible tracking server 200, integrated tracking server 300, facility management server 400, etc. is recorded in a computer-readable recording medium, and this recording is performed. The program recorded in the medium may be read by a computer system and executed to perform the processing as the above-described imaging-compatible tracking server 100, object detection-compatible tracking server 200, integrated tracking server 300, facility management server 400, and the like. .. Here, “reading and executing a program recorded in a recording medium on a computer system” includes installing the program in the computer system. The "computer system" here includes an OS and hardware such as peripheral devices. The “computer system” may include a plurality of computer devices connected via a network including communication lines. The recording medium also includes a recording medium provided inside or outside accessible from the distribution server for distributing the program.

10 (10-1, 10-2) imaging device, 20 object detection sensor, 100 imaging compatible tracking server, 121 second object detection unit, 131 second object tracking information storage unit, 200 object detection compatible tracking server, 221 First object detection unit, 231 First object tracking information storage unit, 300 Integrated tracking server, 321 Object management unit, 331 Common object information storage unit, 400 Facility management server, 500 facilities

Claims (7)

A first object based on the result of detection of the object based on the detection information of the object detection sensor provided corresponding to the first type of area in the facility and detection of the position of the detected object A first object detection unit that generates tracking information,
Based on the image recognition processing using the imaged image obtained by the imaging device provided corresponding to the second type of divided area in the facility, the target object in the second type of divided area is detected. A second object detection unit that generates second object tracking information based on the detection result,
An object management unit that associates the first object tracking information and the second object tracking information of the same object that has moved the boundary between the first type of segmented area and the second type of segmented area, Tracking system equipped. The object management unit stores an object identifier stored as information for identifying an object in each of the first object tracking information and the second object tracking information of the same object having a corresponding relationship. The tracking system according to claim 1, wherein the common object information is associated with each other. The object management unit, within a certain range of time, the object tracking information corresponding to the one type of divided area where the object was finally detected in the one type of divided area before the movement, and the movement. The tracking system according to claim 1, wherein the tracking system is associated with target object tracking information corresponding to the other type of segmented area where detection of an object has started in the other type of segmented area. The object management unit, the position of the object shown in the object tracking information generated in response to the last detection of the object in one type of divided area before movement, and the other of the after movement If the relationship with the position of the object indicated in the object tracking information generated in response to the start of detection of the object in the classification area of the type satisfies a predetermined condition, the classification area of the one type The tracking system according to any one of claims 1 to 3, wherein the object tracking information corresponding to the item is associated with the object tracking information corresponding to the other type of segmented area. The first object detection unit determines an attribute of the detected object based on the detection result, and associates attribute information indicating the determined attribute with the detected object. The tracking system according to 1 above. A first object based on the result of detection of the object based on the detection information of the object detection sensor provided corresponding to the first type of area in the facility and detection of the position of the detected object A first object detection step of generating tracking information,
Based on the image recognition processing using the imaged image obtained by the imaging device provided corresponding to the second type of divided area in the facility, the target object in the second type of divided area is detected. A second object detection step of generating second object tracking information based on the detection result,
An object management step of associating the first object tracking information and the second object tracking information of the same object that has moved the boundary between the first type of segmented area and the second type of segmented area. Tracking method provided. A computer equipped in the tracking system,
A first object based on the result of detection of the object based on the detection information of the object detection sensor provided corresponding to the first type of area in the facility and detection of the position of the detected object A first object detection unit that generates tracking information,
Based on the image recognition processing using the imaged image obtained by the imaging device provided corresponding to the second type of divided area in the facility, the target object in the second type of divided area is detected. A second object detection unit that generates second object tracking information based on the detection result,
Functioning as an object management unit that associates the first object tracking information and the second object tracking information of the same object that has moved along the boundary between the first type section area and the second type section area A program to let you.

Images