JP2023018811A - Terminal tracking device, program, and method for coping with occlusion using communication state information - Google Patents

Abstract

To provide a terminal tracking device capable of continuing, even when occlusion occurs in a terminal being tracked, tracking of the terminal.SOLUTION: This terminal tracking device is a device that detects, from environmental information acquired by a sensor, a predetermined object including a terminal and tracks the terminal. This device includes means for determining terminal communication state information which is information relating to the state of communication with the terminal, based on communication information acquired from communication means, means for determining, from the environmental information, object communication state information, which is communication state information that would be embodied if the predetermined object includes the terminal, and means for calculating, when occlusion occurs in the predetermined object, the degree of correspondence between the determined terminal communication state information and the determined object communication state information, and determining, based on this degree, whether there is a corresponding relationship between the terminal being tracked and the predetermined object, and further determining the predetermined object determined to have the corresponding relationship as the object relating to the terminal being tracked.SELECTED DRAWING: Figure 1

Description

The present invention relates to technology for acquiring information about a terminal as a communication target, and particularly to technology for tracking the terminal.

The 5th generation mobile communication system (5G), which is currently being introduced, uses radio waves in the millimeter wave band (millimeter waves) and can realize high-performance communication such as high speed, large capacity, low delay, and multi-terminal connection. do. Here, millimeter waves have high rectilinearity and are less likely to be diffracted, and there is a high possibility that their propagation will be blocked or attenuated by an object existing between the terminal and the base station. Therefore, in 5G, the presence of such an object causes a sudden drop in received power, which can cause a situation in which, for example, the communication connection itself is interrupted, which is a serious problem.

Therefore, if it is possible to predict a decrease in received power due to such an object, it becomes possible to maintain communication connection without interruption by controlling/securing the communication path of the terminal. As a technique for predicting the received power, for example, in Non-Patent Document 1, time-series 3D environmental information acquired by an RGB-D camera capable of generating not only RGB images but also depth images is used, and machine learning is performed in communication devices. Techniques for predicting received power are disclosed.

Specifically, this technology corrects the time-series information of the received power measured (at the same time as the image is taken) at the communication terminal that received the radio waves from the transmitting station for the time-series environmental information that is the result of imaging the communication environment. A machine learning model is constructed using the training data linked as data, and using this constructed machine learning model, the predicted value of the received power at the communication terminal is calculated from the time-series environmental information acquired by the RGB-D camera. have decided.

Here, in such millimeter-wave reception power prediction, it is important to continuously grasp the position of the communication terminal. However, if the position is determined using the positioning results obtained by the sensor on the communication terminal side, a mechanism for notifying the positioning results from the communication terminal to the base station is required. load becomes a problem.

On the other hand, as a technology that does not depend on the positioning result of the communication terminal side, Patent Document 1, which includes the inventors of the present application, discloses time-series information of millimeter-wave received power and distance between objects estimated from, for example, image information of the environment. A technique for estimating the position of a communication terminal using shielding-related time-series information is disclosed. Furthermore, Non-Patent Document 2 discloses a technique for continuously capturing the position of an object on an image by using the consistency of the detected position of the object and the features of its appearance obtained from the image. .

JP 2021-078022 A

Masashi Nishio, "Prediction and Control of Wireless Communication Quality by Machine Learning", IEICE Technical Report, Vol. 118, No. 57, SR2018-1, pp.1-8, 2018 Nicolai Wojke et al., "Simple Online and Realtime Tracking with a Deep Association Metric", 2017 IEEE International Conference on Image Processing, pp. 3645-3649, 2017. <https://doi.org/10.1109/ICIP.2017.8296962 ＞

Here, if it is possible to continue tracking the communication terminal using such a conventional technology, even if the received power suddenly drops due to the intervention of an object, the communication connection can be continued or temporarily suspended. It is also possible to resume immediately with

However, in the object tracking technology based on image information as disclosed in Non-Patent Document 2, when an object on the front side covers at least a part of an object on the back side, a so-called occlusion phenomenon occurs, tracking There is a possibility that the tracking of the target will fail. In other words, when occlusion occurs, conventional object tracking technology based on image information suffers from “lost,” in which the object to be tracked is hidden behind other objects and becomes undetectable, and tracking is interrupted. crossing, the other object is mistaken as the object to be tracked, and an "ID switch" occurs in which the tracking that should be performed is deviated.

Therefore, even if the object tracking technology disclosed in Non-Patent Document 2, for example, is applied to tracking a communication terminal, it is difficult to accurately grasp the position of the communication terminal when such a problem related to occlusion occurs. I can't. Also, the technique of estimating the position of a communication terminal described in Patent Document 1 is based on the premise that no loss or ID switch has occurred, and does not propose any countermeasures against such problems. not

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a terminal tracking device, a program, and a method capable of more appropriately continuing tracking of a terminal being tracked even when occlusion occurs.

According to the present invention, a predetermined object including the terminal is detected from environmental information about the environment, which is obtained by a sensor that senses the environment in which the terminal may exist, and which may include the predetermined object. , a terminal tracking device for tracking the terminal,
terminal communication state determination means for determining terminal communication state information, which is information regarding the state of communication with the terminal, based on information regarding communication acquired from communication means that communicates with the terminal;
target communication state determining means for determining, from the environment information, target communication state information, which is information relating to a communication state to be realized with the predetermined target if the predetermined target includes a terminal;
The degree of communication compatibility, which is the degree of correspondence between the terminal communication state information determined for the terminal being tracked and the target communication state information determined for the predetermined target when occlusion occurs for the predetermined target. and determines whether or not the tracked terminal and the predetermined target have a corresponding relationship based on the degree of communication compatibility, and the predetermined target determined to have a corresponding relationship and tracked object determination means for determining an object associated with a terminal that is tracking.

In the terminal tracking device according to the present invention, the target communication state determination means interpolates using the detection positions determined before and after the occlusion in the time interval during which the occlusion for which the detection position is not determined for the predetermined target occurs. It is also preferable to determine the target communication state information on the assumption that the predetermined target exists at the position where the target is located.

Further, as an embodiment of the terminal tracking device according to the present invention, from the environment information, the target position related information, which is information related to the detection position of the predetermined target, and/or the detected appearance of the predetermined target. further comprising object detection information determining means for determining object appearance information, which is such information;
The tracked object determining means determines the degree of position-related matching, which is the degree of matching between the determined target position-related information and the predicted target position-related information predicted from the past target position-related information, and/or the determined target position-related information. It is also preferable to determine whether or not the tracked terminal and the predetermined target have a corresponding relationship based on the degree of appearance matching, which is the degree of matching between the target appearance information and the past target appearance information. .

Furthermore, as another embodiment of the terminal tracking device according to the present invention, from the environment information, object position related information, which is information related to the detected position of the predetermined object, and/or the detected appearance of the predetermined object further comprising object detection information determining means for determining object appearance information, which is information relating to
The tracked object determining means at least matches the determined target position-related information with the predicted target position-related information predicted from the past target position-related information in a time interval in which occlusion does not occur for the predetermined object. and/or the degree of appearance matching, which is the degree of matching between the determined target appearance information and the past target appearance information. It is also preferable to determine whether or not there is a correspondence relationship between .

Further, as still another embodiment of the terminal tracking device according to the present invention, the tracked object determination means includes a position-related cost, which is a monotonically decreasing function of the degree of position matching, and/or the degree of appearance matching, in the predetermined target. and the communication state cost, which is a monotonically decreasing function of the degree of communication, are weighted by a weighting factor that can take a zero value, and the tracking cost is a predetermined condition. It is also preferable to determine a predetermined target that is small enough to satisfy , to be the target associated with the tracking terminal.

Furthermore, in the embodiment for calculating the tracking cost described above, the tracking target determination means, when an occlusion in which the detection position is not determined for the predetermined target occurs and is resolved, the weight applied to the communication state cost for the predetermined target It is also preferable to calculate a tracking cost with a larger factor and a smaller or zero weighting factor on the location-related cost and/or the appearance cost.

In addition, in the embodiment for calculating the tracking cost described above, the tracking target determining means adds a weighting factor to the communication state cost of the predetermined target as time elapses from the point at which occlusion occurs and disappears for the predetermined target. It is also preferable to calculate a tracking cost in which the weighting factors for the position-related cost and/or the appearance cost are sequentially increased from a smaller value or from zero.

Furthermore, in the embodiment for calculating the tracking cost described above, the tracking target determination means determines that the communication in the predetermined target is performed during the period from the time when occlusion occurs and disappears for the predetermined target until the time when the predetermined time has passed. It is also preferable to calculate a tracking cost with a higher weighting factor on the state cost and a lower or zero weighting factor on the location-related cost and/or the appearance cost.

In addition, in the embodiment for calculating the tracking cost described above, the tracking target determining means
when occlusion occurs between the predetermined targets, determining whether each of the predetermined targets related to the occlusion is on the front side or the back side based on the past target position-related information;
When the occlusion is resolved and it is determined that the terminal communication state information has been affected by the shielding of communication radio waves, the communication state cost of the predetermined target determined to be on the near side is set to a larger value. , the communication state cost of a predetermined target determined to be on the back side is set to a smaller value or zero, and if it is determined that there is no effect of shielding of communication radio waves in the terminal communication state information, it is on the front side It is also preferable to set the communication state cost of the predetermined target determined to be a smaller value or zero, and set the communication state cost of the predetermined target determined to be on the far side to a larger value.

Furthermore, in the embodiment of calculating the tracking cost described above, the tracked object determining means
when occlusion occurs and disappears for a plurality of predetermined targets, each of the plurality of predetermined targets continues the tracking process as a target candidate for the terminal being tracked;
Ending the tracking process for a predetermined target whose tracking cost or communication state cost has increased to the extent that satisfies a predetermined condition in the continuous tracking process as not being a target related to the terminal being tracked. is also preferred.

According to the present invention, a predetermined target including the terminal is detected from environmental information that is information related to the environment and that may include the predetermined target, which is acquired by a sensor that senses the environment in which the terminal may exist. and a terminal tracking program that causes a computer that tracks the terminal to function,
terminal communication state determination means for determining terminal communication state information, which is information regarding the state of communication with the terminal, based on information regarding communication acquired from communication means that communicates with the terminal;
target communication state determining means for determining, from the environment information, target communication state information, which is information relating to a communication state to be realized with the predetermined target if the predetermined target includes a terminal;
The degree of communication compatibility, which is the degree of correspondence between the terminal communication state information determined for the terminal being tracked and the target communication state information determined for the predetermined target when occlusion occurs for the predetermined target. and determines whether or not the tracked terminal and the predetermined target have a corresponding relationship based on the degree of communication compatibility, and the predetermined target determined to have a corresponding relationship A terminal tracking program is provided that causes a computer to function as a tracked object determination means for determining a target associated with a terminal that is running.

According to the present invention, a predetermined target including the terminal is further detected from environmental information that is information related to the environment and that may include the predetermined target, which is acquired by a sensor that senses the environment in which the terminal may exist. and a terminal tracking method implemented by a computer that tracks the terminal,
determining terminal communication status information, which is information relating to the status of communication with the terminal, based on information relating to communication acquired from a communication means that communicates with the terminal;
determining, from the environment information, target communication state information, which is information relating to a communication state to be realized with the predetermined object if the predetermined object includes a terminal;
The degree of communication compatibility, which is the degree of correspondence between the terminal communication state information determined for the terminal being tracked and the target communication state information determined for the predetermined target when occlusion occurs for the predetermined target. and determines whether or not the tracked terminal and the predetermined target have a corresponding relationship based on the degree of communication compatibility, and the predetermined target determined to have a corresponding relationship and determining a target for a terminal that is performing a terminal tracking method.

According to the terminal tracking device, program, and method of the present invention, even when an occlusion occurs in a terminal being tracked, it is possible to continue tracking the terminal in question more appropriately.

It is a functional block diagram showing a functional configuration in one embodiment of the terminal tracking device according to the present invention. FIG. 4 is a schematic diagram for explaining two modes of occlusion determination processing in an occlusion determination unit according to the present invention; FIG. 5 is a schematic diagram for explaining a specific example of communication state cost calculation processing according to the present invention; FIG. 5 is a schematic diagram for explaining a specific example of calculation processing of position-related costs and appearance costs according to the present invention; FIG. 4 is a schematic diagram for explaining various embodiments of tracking target determination processing in a tracking target determination/management unit according to the present invention; FIG. 10 is a schematic diagram for explaining still another embodiment of the tracking target determination process in the tracking target determination/management unit according to the present invention; FIG. 10 is a schematic diagram for explaining still another embodiment of the tracking target determination process in the tracking target determination/management unit according to the present invention;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Terminal tracking device]
FIG. 1 is a functional block diagram showing a functional configuration in one embodiment of a terminal tracking device according to the present invention.

The base station 1 as one embodiment of the terminal tracking device according to the present invention shown in FIG. 1 is a communication relay device compatible with 5G (fifth generation mobile communication system) in this embodiment, It is a device capable of communicating with a plurality of terminals 2, which are communication terminals.

In addition, the base station 1, as an embodiment of the terminal tracking device, is information related to the environment acquired by the camera 103 as a sensor that senses the environment in which the terminal 2 may exist, and may include a predetermined target. It is also a device capable of detecting a predetermined target including the terminal 2 from image data as "environmental information" and tracking the terminal 2.

Here, in this embodiment, the camera 103 is an RGB camera capable of photographing the situation of the communication area with a predetermined angle of view. Image data (video data) generated by the camera 103 includes:
(a) a person who possesses or carries the terminal 2,
(b) Mobile objects such as automobiles, motorcycles, railroad vehicles, robots, drones, etc. equipped with, including or boarded by the terminal 2, and (c) equipment, facilities, buildings, fixed objects where the terminal 2 is installed There is a possibility that such "object" is included (imaged). Furthermore, "objects" such as people, mobile objects, equipment, facilities, buildings, and fixed objects (including plants such as trees) that are not related to the terminal 2 can also be included.

By the way, such a "target" can be the one whose correspondence relationship with the tracked terminal 2 should be determined (in other words, the one which should determine whether or not it is the tracked target to be tracked), while the terminal 2 and the base station 1 It can also become an obstacle for communication radio waves.

Here, in particular, 5G, which is the communication method of the present embodiment, uses radio waves in the millimeter wave band (millimeter waves) for communication. , its propagation is likely to be blocked or attenuated by "objects" that exist between the terminal 2 and the base station 1. Therefore, in 5G, there is a serious problem that the power of the received signal at the base station 1 is suddenly reduced due to the intervention of such a "target", and for example, the communication connection itself may be interrupted. On the other hand, even if "occlusion" occurs in the terminal 2 being tracked, if the tracking of this terminal 2 can be appropriately continued, the communication connection with the terminal 2 can be continued or temporarily suspended. It is also possible to restart immediately after doing so.

Incidentally, the above-mentioned "occlusion" is hereinafter referred to as a phenomenon in which an "object" on the near side as viewed from the base station 1 obscures at least a part of an "object" on the far side. For example, a phenomenon in which a plurality of "objects" intersect or at least one "object" is occluded by at least one other "object" is also treated as "occlusion" below. When such "occlusion" occurs, there is a possibility that a predetermined target (e.g., "person") cannot be detected from "environmental information" (image data in this embodiment) or can only be detected inaccurately. It increases.

Therefore, specifically, the base station 1, in order to solve the serious problem when such occlusion occurs,
(A) Based on information (for example, communication history information) related to communication acquired from communication means (communication interface 101, communication control unit 116, and communication history information accumulation unit 102 provided by itself in this embodiment) for communicating with the terminal , a terminal communication state determination unit 111 that determines “terminal communication state information” that is information related to the communication state with the terminal 2;
(B) If the terminal 2 is included in the predetermined target (for example, "person") detected from the "environmental information" acquired by the sensor (camera 103 in this embodiment), the predetermined target ("person "), a target communication state determination unit 112 that determines "target communication state information" that is information related to the communication state to be embodied between
(C) When occlusion occurs for a predetermined target (for example, "person"), "terminal communication state information" determined for the terminal 2 being tracked and "terminal communication state information" determined for the predetermined target ("person") A "communication correspondence degree" is calculated, which is the degree of correspondence between the "target communication state information" and the "communication correspondence degree". A tracking target determination/management unit 115 that determines whether or not there is a correspondence relationship, and determines a predetermined target ("person") determined to have a correspondence relationship as a target related to the terminal 2 being tracked; have.

Here, the ``predetermined object'' detected from the ``environmental information'' in the above (B) is a ``person'' or a ``moving body'' who may carry, mount, or include the terminal 2, as will be explained later. " can be. Further, depending on the case, "equipment", "facility", "building", "fixed object", etc., which may include the terminal 2, may also be the "predetermined object".

In addition, the above (A) "terminal communication state information" is, as a preferred embodiment, (α) information related to the strength of the radio wave received from the terminal 2 (for example, received signal power, RSSI (Received Signal Strength Indicator)) and can do. Alternatively, (β) information relating to the presence/absence of communication connection with the terminal 2 may be used.

On the other hand, the "target communication state information" in (B) above corresponds to the above "terminal communication state information", and if (α') the detected predetermined target (for example, "person") includes the terminal 2, It can be information (for example, RSSI) related to the intensity of radio waves to be received from the predetermined target (“person”). Alternatively, (β′) it may be information regarding whether or not communication connection with a predetermined target (“person”) is possible, determined using the detected position of the predetermined target (for example, “person”). It is possible.

Furthermore, the "degree of communication support" in (C) above is specifically, for example, time-series data of the amount (e.g., RSSI value) related to the strength of radio waves as "terminal communication state information" and "target communication state information" as can be the degree of matching with the time-series data of the quantity (for example, the RSSI value) related to the radio wave intensity. Alternatively, it can be the degree of matching between time-series data relating to the presence or absence of communication connection as "terminal communication state information" and time-series data relating to the availability of communication connection as "target communication state information".

Thus, according to the base station 1 as the terminal tracking device according to the present invention, when occlusion occurs, the terminal 2 being tracked and the detected predetermined object (for example, "person") correspond to each other. The communication state of the terminal 2 and the estimated communication state of the predetermined target without depending on whether the predetermined target is detected from the "environmental information" (for example, image data). It can be implemented using the "communication correspondence degree" with. As a result, even if occlusion occurs for the terminal 2 being tracked, it is possible to continue tracking this terminal 2 more appropriately.

Further, in this embodiment, the base station 1 continues the communication connection with the terminal 2 even if an object is interposed between the base station 1 and the terminal 2 and the reception power drops sharply. or resume immediately.

By the way, the base station 1 carries out the terminal tracking process using the above-described "communication compatibility level" when occlusion occurs and is resolved, for example. In addition to this processing, object tracking processing by image recognition using the consistency of the detection position of the object and the appearance feature obtained from the image, as described in Non-Patent Document 2, is also performed together. is also preferred. Such preferred embodiments will be described in detail later.

Furthermore, in the non-occlusion period separated from the occlusion time interval,
(a) Terminal tracking processing using the "communication compatibility level" described above,
(b) object tracking by image recognition using the consistency of object detection positions and appearance features obtained from images, as described in Non-Patent Document 2; Terminal/target tracking processing, for example, terminal location information determined based on communication history information and detection of a predetermined target detected from environmental information (image data), which is described as an embodiment in Patent Document 1 Calculate the degree of correspondence between the target position information related to the position (position correspondence degree), and also based on this position correspondence degree, determine whether or not the terminal and the predetermined target have a correspondence relationship, and perform tracking. It is also preferable to implement at least one of the techniques, or two or more together.

In addition, although 5G is adopted as the communication method between the base station 1 and the terminal 2 in this embodiment, other communication methods such as LTE may be used as a matter of course. Communication between the device and the communication terminal may be based on various other wireless communication standards. For example, even in communication systems where the problem of shielding by objects is not as pronounced as in 5G, there are quite a few situations in which it is desirable to accurately identify and track communication terminals under control using targets detected from "environmental information." For example, there are various needs for terminal identification and tracking, such as determining the relationship between a user's browsing page determined to have a corresponding relationship with a certain communication terminal and the user's flow line and utilizing it for marketing and management. there is In contrast, according to the terminal tracking device according to the present invention, it is possible to perform such terminal identification/tracking processing with high accuracy.

Further, the terminal tracking device according to the present invention is, of course, not limited to a communication relay device such as a base station. For example, as a dedicated device for terminal tracking processing, it may be provided in the form of being connected to a communication relay device/equipment such as a base station. Further, as the terminal tracking device according to the present invention, it is possible to use a cloud server, a non-cloud server device, a personal computer (PC), a notebook or tablet computer, or the like, which is equipped with the terminal tracking program according to the present invention. be.

Furthermore, at least one of the above (A) to (C), which are components of the terminal tracking device (base station 1) according to the present invention, is a device different from the other components. It is not impossible to take. For example, it is possible to implement the above functions (A) to (C) by a plurality of servers as a whole. Even in such a case, the entirety of these can be regarded as a terminal tracking device or system that implements the terminal tracking method according to the present invention.

[Functional configuration of terminal tracking device, terminal tracking program]
Similarly, in the functional block diagram of FIG. 1, the base station 1 includes a communication interface 101, a communication history information accumulation unit 102, a camera 103, an environment information It has a storage unit 104, a tracking information storage unit 105, and a processor memory. Here, the processor memory stores an embodiment of a communication relay program including a terminal tracking program according to the present invention, and has computer functionality, and by executing this communication relay program, Perform terminal tracking processing and communication relay processing.

In addition, the processor memory includes, as functional components, a terminal communication state determination unit 111, a target communication state determination unit 112 including a position interpolation unit 112a, a target detection information determination unit 113, an occlusion determination unit 114, and a communication state determination unit 111. It has a tracking target determination/management unit 115 including a cost calculation unit 115a, a position-related cost calculation unit 115b, and an appearance cost calculation unit 115c, and a communication control unit . These functional configuration units can be regarded as the functions of the terminal tracking program and the communication relay program stored in the processor memory. can be understood as an embodiment of the terminal tracking method and communication relay method according to the present invention.

Similarly, in the functional block diagram of FIG. is acquired and recorded, and communication history information is generated by organizing the information in chronological order, and the communication history information storage unit 102 stores and manages the information.

Similarly in the functional block diagram of FIG. It can also be a -D camera, a stereo camera, or an omnidirectional (omnidirectional) camera. Of course, instead of the camera 103, it is also possible to adopt a sensor capable of sensing the environment in which the terminal 2 may exist and generating environmental information, such as LiDAR, laser/infrared positioning device, TOF camera, and thermography device. be.

Here, in this embodiment, the camera 103 is an RGB camera capable of generating RGB image data (RGB video data) as environmental information, and the generated RGB image data is stored and managed by the environmental information storage unit 104. be done. Note that the camera 103 is installed inside the base station 1 in the present embodiment, but is installed at a position distant from the base station 1, for example, a surveillance camera in town, and communicates with the base station 1. It may be connected.

Moreover, a plurality of cameras 103 may be provided at mutually different positions regardless of whether they are inside or outside the base station 1 . For example, the base station 1 receives and acquires the "target communication state information" generated by the camera 103 installed in the base station 1 from each of a plurality of base stations 1 existing in its surroundings, and then details it later. You may implement the tracking object determination process demonstrated to. In this case, it becomes possible for a plurality of base stations 1 to work together to more preferably implement the terminal tracking processing according to the present invention.

<Target detection information determination processing>
Similarly, in the functional block diagram of FIG.
At least one of (a) target position-related information, which is information relating to the detection position of the "predetermined target", and (b) target appearance information, which is information relating to the detected appearance of the "predetermined target" On the other hand, preferably both are determined.

As already mentioned here, the "predetermined target" is, for example,
(a) a person (user) possessing the terminal 2,
(b) A vehicle equipped with a terminal 2 having a drive recorder function, a CAN information transfer function, an interface function for automatic driving control by a server, etc., and
(c) "Person" who may include the terminal 2 as the communication destination related to the communication history information, such as an autonomous mobile robot or an autonomous flying drone equipped with a terminal 2 having an interface function for autonomous movement control by a server It can be "or" mobile. In some cases, "facility", "facility", "building", "fixed object", etc., which may include the terminal 2, may also be the "predetermined object".

Of course, the target detection information determining unit 113 of the present embodiment can also detect multiple types of targets including the "predetermined target" at the same time. Here, when one "predetermined target" as a tracked target is detected from one image data, other targets (including other predetermined targets) detected from this image data are detected from this "predetermined target". It will be treated as a target that can cause occlusion (can cause disability), as opposed to "the target of".

Here, the processing for determining the target position-related information (a) and the target appearance information (b) will be described. In this embodiment, the object detection information determination unit 113 sets a plurality of image regions specified in the acquired image data, and calculates a score representing a predetermined object-likeness in each image region using the corresponding object detector. , an image region having a score high enough to satisfy a predetermined condition is detected as a detection image region of a predetermined object. then
(a) The position (detection position) of the detection image area related to the detected predetermined object, the range of the detection image area (for example, the coordinate values of the four corners), and the change in the detection position between consecutive image data The detected speed of is used as the target position-related information of (a) above,
(b) The image feature amount of the detected image region of the predetermined target calculated using a known image feature extractor (e.g., CNN (Convolutional Neural Networks) image feature extractor) Information.

Regarding the above object detector, for example, in Non-Patent Document: Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao, “YOLOv4: Optimal Speed and Accuracy of Object Detection”, arXiv:2004.10934v1, 2020 What is described can be adopted. Furthermore, using various other object detectors and image feature extractors known in the field of image recognition technology, the above (a) target position-related information and the above (b) target appearance information are determined. is also possible.

When a stereo camera is used as the camera 103, for example, non-patent literature: Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang It is also possible to apply the object detector described in Fu, Alexander C. Berg, “SSD: single shot multibox detector”, European Conference on Computer Vision, Computer Vision-ECCV 2016, pp.21-37, 2016. It's becoming

As still another embodiment, when LiDAR is used instead of the camera 103 and three-dimensional point cloud data is used as environmental information, an object detector for predetermined target detection may be used, for example, by non-patent document: Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas, “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation”, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 77-85, 2017 can be adopted.

Furthermore, in this embodiment using image data as environment information, instead of the above image feature extractor, for example, non-patent literature: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, "Deep Residual Learning for Image Recognition ”, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1, pp.770-778, 2016.

Incidentally, in this embodiment, the target detection information determination unit 113 stores the determined "target position-related information" and "target appearance information" for each predetermined detected target for a past predetermined period. It can be provided to the decision/management unit 115 as appropriate.

<Terminal communication state information determination processing>
Similarly, in the functional block diagram of FIG. A certain "terminal communication state information" is determined. Here, in this embodiment, this "terminal communication state information" is
Either or both of (α) time-series data of received signal power (RSSI) of the radio signal received from the terminal 2 and (β) time-series data of information regarding the presence or absence of communication connection with the terminal 2 ing.

Regarding (α) above, the terminal communication state determining unit 111 may generate time-series data of the received signal power variable E_t that indicates the signal power value at time t of the radio signal received from the terminal 2 . In addition, the terminal communication state determination unit 111 determines whether or not the communication connection with the terminal 2 is established at each point in time regarding (β) above. Time-series data of a binary communication connection presence/absence variable B_t (ε{-1, 1}) that takes a value of 1 when communication is not established and -1 when communication is not established may be generated. However, considering the accuracy of the degree of matching (degree of communication compatibility) with the target communication state information calculated later, the "terminal communication state information" may include the time-series data of the received signal power variable E_t in (α) above. preferable.

Incidentally, in this embodiment, the terminal communication state determination unit 111 stores the “terminal communication state information” determined for each terminal 2 being tracked for a predetermined period in the past. can be provided as appropriate. In addition, the terminal communication state determination unit 111 acquires, for example, an IMSI (International Mobile Subscriber Identity) from each terminal 2 under the control of the base station 1, and determines the determined "terminal communication state" for each terminal 2 identified by the IMSI. It is also preferable to compile "information" and have the subsequent processing performed separately for each terminal 2 concerned.

<Target communication state information determination processing>
Similarly, in the functional block diagram of FIG. is included, 'target communication state information', which is information relating to the communication state to be embodied with this 'predetermined target', is determined.

(When using received signal strength data as "target communication status information")
The "target communication state information" determined by the target communication state determination unit 112 is
(α′) Information (received radio wave intensity data) relating to the strength of the radio wave to be received from the “predetermined target” if the “predetermined target” includes the terminal 2
can be In this case, the target communication state determination unit 112 determines the detection position of the “predetermined target” detected from the image data (by the target detection information determination unit 113) and the detection position of the “other target” (which may cause a problem). Based on and, as shown on the right side of FIG. 3 used for explanation later,
(a) determining a straight line segment connecting the "predetermined target" and (the antenna of) the base station 1;
(b) calculating shielding degree information relating to the degree to which the "other target" shields a predetermined area including the straight line segment;
(c) From the calculated degree of shielding information, calculate the received radio field intensity data that will be received from the "predetermined target" if the "predetermined target" includes the terminal 2 as the "target communication state information". of.

Specifically, the shielding degree information (b) above is calculated as follows. First, as shown on the right side of FIG. After setting in advance and further setting a rotating body region of a predetermined size including the above-mentioned straight line segment as a rotating axis, at a certain time t (= t0-M, t0-M+1, ..., t0- 1, t0),
(b1) A part of the cross-sectional area of any of the "other targets" is not included in the above-mentioned rotating body area at the detection position (the overlapping part of the cross-sectional area of the other target and the rotating body area does not have a value of 1, whereas
(b2) In one of the "other targets", a part of its cross-sectional area is included in the above-mentioned rotating body area at the detection position (the cross-sectional area of the other target and the rotating body area overlap (1) R = (S _a ∩ S _b ^- )/S _a
The time-series data of the variable E′_t, which takes the R value calculated with , can be used as the shielding degree information.

Here, in the above formula (1), S _a is a set representing a cross-sectional area (perpendicular to the axis of rotation) at the detection position of the "other object" in the rotating body area (for example, a unit forming the area A set of point areas), and S _b is a set (for example, a set of unit point areas forming the area) representing a cross-sectional area at the detection position of the “other object”. In addition, S _b ⁻ is the complement of S _b , and “/” indicates the area value of the cross-sectional area related to the set of molecules (eg, the number of unit point areas), and the area value of the cross-sectional area related to the denominator (eg, the unit This is an operator indicating division by the number of point areas).

Next, the target communication state determining unit 112 regards the time-series data of this variable E′_t (or the result obtained by multiplying the variable E′_t by a predetermined constant) as “target communication state information” as received radio wave intensity data. I do. Hereinafter, the variable E'_t will be referred to as a variable equivalent to received power in order to distinguish it from the above-described received signal power variable E_t.

If there are two or more “other objects” that take the above R value, S _b is the area formed by the entire cross-sectional area of these “other objects”, that is, the sum of these cross-sectional areas It may be a region represented as a set. Further, the body of revolution region may be a cylindrical region, or may be formed by rotating a predetermined curve around an axis of rotation, such as an ellipsoid.

Furthermore, as a modification, the target communication state determination unit 112 uses a learned "target radio wave information estimation model" that outputs information related to the strength of the received radio wave according to the input image data, and the acquired image If the "predetermined target" includes the terminal 2, information on the strength of the radio wave that will be received from the "predetermined target" is derived from the data, and the information on the strength of the radio wave is used as the "target communication status information”.

Here, a known machine learning model as disclosed in Non-Patent Document 1 can be adopted as the above-mentioned "target radio wave information estimation model". Specifically, image data at a certain point in time t (= t0-M, t0-M+1, ..., t0-1, t0) is input to this model, and an estimated value of the received signal power is output. The estimated value is used as the received power equivalent variable E'_t value, and the time-series data of this received power equivalent variable E'_t is used as the "target communication state information".

(When communication connection availability data is used as "target communication status information")
Also, the "target communication state information" determined by the target communication state determination unit 112 is
(β') Information regarding whether or not communication connection is possible with the "predetermined target" (communication connection permission/inhibition data)
It is also possible to In this case, the target communication state determination unit 112, based on the detected position of the "predetermined target" detected from the acquired image data and the detected position of the "other target" that may cause a failure,
(a) determining a straight line segment connecting the "predetermined target" and (the antenna of) the base station 1;
(b) calculating communication connection availability data indicating whether or not the "other object" is positioned on this straight line segment or within a predetermined area including this straight line segment, or whether or not this straight line segment is blocked;
(c) It is also preferable to determine the calculated communication connectability data as the "target communication state information".

More specifically, as the communication connection propriety data in (b) above, the detection position of one of the "other targets" is at a certain point in time t (== t0-M, t0-M+1, ..., t0- 1, t0) is -1 if it is located on the above straight line segment or within a rotating body region of a predetermined size that includes this straight line segment as the rotation axis, and takes a value of 1 otherwise. Binary communication connection availability variable:
(2) B'_t∈{－1, 1}
may be generated and used as the "target communication state information". Here, the body of revolution region may be a cylindrical region, or may be formed by rotating a predetermined curve around a rotation axis, such as an ellipsoid.

As a modification mode, the target communication state determination unit 112 presets a predetermined size (cross-sectional area at the detection position) according to the type (class) of the "other target". When the "other target" blocks the above straight line segment with its size (cross-sectional area) at the detection position (the above straight line segment penetrates the cross section of the "other target") B′_t in (2) above may be defined as a communication connection enable/disable variable that takes a value of −1, otherwise 1.

As described above, the "target communication state information" is
(α') Time-series data of received power equivalent variable E'_t (received radio wave intensity data), and (β') Time-series data of communication connection availability variable B'_t (communication connection availability data) explained the process of Here, considering the accuracy of the matching degree (communication compatibility degree) with the terminal communication state information calculated later, the "target communication state information" is the time-series data of the received power equivalent variable E'_t in (α') It is also preferable that

Incidentally, in the present embodiment, the target communication state determination unit 112 stores the determined "target communication state information" for each predetermined target detected for a predetermined period in the past. It can be provided to the management unit 115 as appropriate.

<Occlusion determination processing>
Similarly, in the functional block diagram of FIG. is occurring (or is predicted to occur), and the result of this determination is output to the tracking target determining/managing unit 115, which will be described later in detail.

FIG. 2 is a schematic diagram for explaining two modes of occlusion determination processing in the occlusion determination unit 114. FIG.

First, according to FIG. 2A, the occlusion determination unit 114 receives from the object detection information determination unit 113 a predetermined time related to the terminal 2 being tracked at a time point (for example, time point t0-1) past time point t0. The detected position and detected speed (target position-related information) of an object (for example, "person") are acquired, and from these information, the predicted position of this predetermined object ("person") at time t0 is determined.

Next, the occlusion determination unit 114 detects another predetermined object at time t0 within a predetermined range including the determined predicted position (for example, an image region corresponding to a real-space circular range with a predetermined size centered on the predicted position). When a person (for example, "person") is detected, it can be determined that occlusion is occurring at this time t0.

Next, in the mode shown in FIG. 2B, the occlusion determination unit 114 first determines a predetermined target (for example, a "person") related to the terminal 2 being tracked in the same way as in the mode shown in FIG. 2A described above. ) at time t0.

Next, the occlusion determination unit 114 detects the predetermined object at time t0 within a predetermined range including the determined predicted position (for example, an image region corresponding to a real-space circular range with a predetermined size centered on the predicted position). is detected, it can be determined that occlusion is occurring at this time t0. It should be noted that in this case, there is a high possibility that an occlusion occurs such that the predetermined target being tracked is completely occluded.

Further, as another aspect of the occlusion determination process, the occlusion determination unit 114 uses the object detection information determination unit 113 to determine the object position relation of a predetermined object being tracked (for example, a “person”) from the image data at a certain time t0. If one or both of the information and the object appearance information could not be determined, it is possible to receive a report to that effect and determine that occlusion is occurring at this time t0.

Furthermore, as yet another aspect of the occlusion determination unit 114, the object detection information determination unit 113 determines a predetermined target being tracked (for example, a “person”) and another predetermined target from image data at a certain time t0. When the target (for example, "person") is detected, if the distance between the detection positions (for example, the distance on the image space corresponding to the predetermined distance on the real space) is less than a predetermined threshold, It is also possible to determine that occlusion is occurring at this time t0. In this case, there is a high possibility that partial occlusion occurs to the extent that the detection position can be determined.

<Tracking target determination processing/terminal tracking processing>
Returning to the functional block diagram of FIG. 1, the tracking target determining/managing unit 115 of the present embodiment determines that no occlusion has occurred for a predetermined target (for example, a “person”) and the target detection information determining unit 113 determines the target position-related information. and the time interval in which the target appearance information is determined, at least
(a) "position-related matching degree", which is the degree of matching between the determined target-position-related information and the predicted target-position-related information predicted from the past target-position-related information; and (b) determined target appearance information. "Appearance matching degree", which is the degree of matching between the target appearance information and the past
Based on either one or both (both in this embodiment), it is determined whether or not the terminal 2 being tracked and a predetermined target ("person") have a corresponding relationship, and terminal tracking processing is performed.

Here, in the present embodiment, the processing using the "communication correspondence degree" described so far is further performed.
(c) "Communication correspondence degree", which is the degree of correspondence between the terminal communication state information determined for the terminal 2 being tracked and the target communication state information determined for a predetermined target ("person").
Also based on , is determined as a target related to the terminal 2 being tracked, and the terminal tracking process is performed.

Furthermore, the tracking target determining/managing unit 115 of the present embodiment, when an occlusion in which the detection position is not determined for a predetermined target occurs and is resolved, the above (c) "communication correspondence degree" described so far is always Using, for example, based only on this "degree of communication compatibility", correspondence determination processing and terminal tracking processing between the terminal 2 and a predetermined target ("person") are performed.

This is because the calculation of the above (a) "position-related matching degree" and the above (b) "appearance matching degree" at the point in time when occlusion occurs and is resolved, for which the detection position cannot be determined, is based on the time period during which this occlusion occurred. Where "(past) target position-related information" and "(past) target appearance information" in the section are required, these information cannot be obtained precisely because such occlusion has occurred. By being there. In other words, in this case, neither the above (a) "position-related matching degree" nor the above (b) "appearance matching degree" cannot be calculated.

Specific correspondence determination/terminal tracking processing using the above-described "position-related matching degree", "appearance matching degree", and "communication compatibility degree" will be described below. Similarly, in the functional block diagram of FIG. After calculating the degree of related matching, the degree of appearance matching, and the degree of communication correspondence,
(a) a "location-related cost" that is a monotonically decreasing function (e.g., inverse) of the location-related degree of matching;
(b) "appearance cost", which is a monotonically decreasing function (eg, reciprocal) of the degree of appearance matching; and (c) "communication state cost," which is a monotonically decreasing function (eg, reciprocal) of the degree of communication compatibility.
Calculate Each of these costs is a value such that the smaller the cost, the higher the possibility of having a corresponding relationship (between the terminal 2 and the predetermined target).

Next, the tracking target determining/managing unit 115 of the present embodiment calculates (a) position-related cost Cgeo, (b) appearance cost Capp, and ( c) Calculate the tracking cost Cost, which is a weighted sum of the communication state cost Cradio with the weight λ that can take a zero value, and calculate the tracking cost Cost that is small enough to satisfy a predetermined condition. ("person") is determined to be the object associated with the terminal 2 being tracked. Here, such a chasing cost Cost is, for example, the following formula (3) Cost=(1−λ)·(Cgeo+Capp)+λ·Cradio
can be calculated by In the above equation (3), λ is a weight of 0 or more and 1 or less. Incidentally, as a modification mode, the weighting factors for the position-related cost Cgeo and the appearance cost Capp in the above equation (3) (both 1-λ in the above equation (3)) can be set to different values. Although there are still accuracy issues, it is also possible to modify the above equation (3) to use only one of the location-related cost Cgeo and the appearance cost Capp.

(Communication state cost calculation)
FIG. 3 is a schematic diagram for explaining a specific example of communication state cost calculation processing according to the present invention.

In the specific example of calculating the communication state cost Cradio shown in FIG. 3, the communication state cost calculation unit 115a first:
(a) time-series data of the received signal power variable E_t of the terminal 2 being tracked (as terminal communication state information) determined by the terminal communication state determination unit 111;
(b) The degree of matching (communication correspondence degree) Cr of the received power equivalent variable E′_t related to the predetermined target (as the target communication state information) determined by the target communication state determination unit 112 with the time-series data is set to a predetermined It is calculated as the cross-correlation (Σ _t E_t*E'_t) of both in the time interval.

Incidentally, when the time-series data of the communication connection presence/absence variable B_t and the communication connection availability variable B'_t are adopted as the terminal communication state information and the target communication state information, respectively, the matching degree Cr is the cross-correlation ( Σ _t B_t*B'_t).

More specifically, the communication state cost calculation unit 115a determines that no occlusion has occurred such that the detection position cannot be determined (the target detection information determination unit 113 determines that the predetermined target detection position (target position related information) is stored) is calculated from the received signal power variable E_t and the received power As the cross-correlation with the equivalent variable E'_t, the following formula (4) Cr＝C0・Σ _t=t0-M ^t0-N E_t*E'_t
Calculated using Here, in the above equation (4), Σ _t=t0-N ^t0-N is an operator indicating the summation for time t (= t0-M, t0-M+1, ..., t0-N) is. Also, C0 is a coefficient for adjusting the Cr value to be 1 upon perfect matching. Furthermore, M is an integer set such that the time t0-M is sufficiently far in the past (for example, 10 seconds ago) for cross-correlation.

Also, N is the latest (closest from time t0) at which the detection position (target position-related information) of the predetermined target is stored when time t0-N is viewed from time t0 (in target detection information determination unit 113). An integer greater than or equal to 1 that is a point in time. Therefore, when N=1 and the detection position of the predetermined target at time t0−1 immediately before time t0 is stored in the target detection information determination unit 113, that is, when the occlusion whose detection position is not determined is the latest , the degree of coincidence Cr at time t0 is given by the following formula (4') Cr=C0・Σ _t=t0-M ^t0-1 E_t*E'_t
will be calculated using

On the other hand, when N>1, the detection position of the predetermined target in the time interval from time t0-N+1 to time t0-1 is not determined by the occurrence of occlusion to the extent that detection processing cannot be performed, As a result, the received power equivalent variable E'_t is not calculated in the time interval in which such occlusion occurred. Therefore, in this case, the degree of coincidence Cr represented by the above equation (4) cannot be calculated as it is.

Therefore, the position interpolation unit 112a (FIG. 1) of the target communication state determination unit 112 does not determine and store the detection position in the time interval from time t0-N+1 to time t0-1 (received power equivalent variable E'_t is not calculated), the position of the predetermined target at each time point (time point t0-N+1, time point t0-N+2, ..., time point t0-1) of the time interval where such occlusion occurred is determined by interpolation. Specifically, the detected position at the acquired time t0-N (or the acquired detected position before the acquired time t0-N) and the acquired detected position at the time t0 are interpolated by a known interpolation method such as linear By interpolating by interpolation or spline interpolation, the positions at times t0-N+1, t0-N+2, . . . and t0-1 can be determined.

Next, the target communication state determining unit 112 (FIG. 1) places the target communication state at each time (time t0-N+1, time t0-N+2, . . . , time t0-1) determined as the interpolation position. exists, the received power equivalent variable E'_t (target communication state information) at each time is determined. As a result, the communication state cost calculation unit 115a can calculate the degree of coincidence Cr using the above equation (4) even at time t0 when the occlusion that has occurred is resolved.

Furthermore, the communication state cost calculator 115a obtains the reciprocal of the matching degree Cr calculated as described above, and determines this reciprocal as the communication state cost Cradio. Here, since the degree of coincidence Cr takes a numerical value that satisfies 0<Cr≦1, the communication state cost Cradio takes a value of 1 or more. There is a high possibility that it will not correspond to 2.

Incidentally, in FIG. 3, the communication state cost Cradio of target A is smaller than the communication state cost Cradio of target B because the waveform of the received power equivalent variable E'_t is more similar to the waveform of the received signal power variable E_t. Therefore, looking only at the communication state cost Cradio, it can be said that there is a higher possibility that the target A corresponds to the terminal 2 being tracked.

(Calculation of location-related cost, calculation of appearance cost)
FIG. 4 is a schematic diagram for explaining a specific example of the position-related cost and appearance cost calculation processing according to the present invention.

First, in the specific example of position-related cost Cgeo calculation shown in FIG. 4A, the position-related cost calculation unit 115b
(a) the determined detection image area range (as the target position-related information) (for example, the coordinate values of the four corners);
(b) Calculate the degree of matching (position-related matching degree) Cg with the range of the detected image area (as prediction target position-related information) predicted from the past detection position (as target position-related information).

More specifically, the position-related cost calculation unit 115b determines that no occlusion has occurred such that the detection position is not determined (the detection position (target position-related information) of the predetermined target is determined by the target detection information determination unit 113). is stored) at time t0,
(a) the range S1 of the detected image area for the given object at this time t0;
(b) At the latest (closest to time t0) time t0-N where the detection position (target position-related information) of the predetermined target is stored (in the target detection information determination unit 113) from this time t0 Let S1∩S2 be the area of the superimposed portion of the predicted detection image area range S2 at time t0, predicted using the detection speed at time t0-N from the detection position, and (a) the detection image area range Assuming that the area occupied by S1 and (b) the range S2 of the predicted detection image region as a whole is S1∪S2, the following equation (5) Cg = (S1∩S2)/(S1∪S2)
Then, the degree of coincidence Cg at time t0 is calculated.

According to the processing for calculating the degree of coincidence Cg as described above, even if there is a point in time when occlusion occurs at which the detection position is not determined, that is, when N>1, the degree of coincidence Cg at time t0 is It can be calculated with certainty.

Furthermore, the position-related cost calculation unit 115b obtains the reciprocal of the degree of matching Cg calculated as described above, and determines this reciprocal as the position-related cost Cgeo. Here, since the degree of matching Cg takes a numerical value that satisfies 0<Cg≦1, the location-related cost Cgeo takes a value of 1 or more, and the greater the location-related cost Cgeo, the more likely the predetermined target is being tracked. There is a high possibility that the terminal 2 will not be supported.

Next, in the specific example of appearance cost Capp calculation shown in FIG. 4B, the appearance cost calculation unit 115c calculates
(a) the image feature amount of the determined detection image area (as target appearance information);
(b) Calculate the matching degree (appearance matching degree) Ca with the image feature amount of the detected image area in the past (as the target appearance information).

More specifically, the appearance cost calculation unit 115c determines that no occlusion that does not determine the detection position has occurred (the target appearance information of the predetermined target is stored in the target detection information determination unit 113). At time t0,
(a) an image feature derived using a known CNN image feature extractor for the detected image region of the given target at this time t;
(b) The predetermined target at the latest (closest to the time t0) time t0-N at which the target appearance information of the predetermined target is stored (in the target detection information determination unit 113) viewed from this time t0 For the detected image region, the similarity with the image feature amount derived using the same CNN image feature extractor as in (a) above is calculated by a known similarity calculation method between feature amounts, for example, described in Non-Patent Document 2 The degree of similarity calculated using the method described above is used as the degree of coincidence Ca at time t0.

According to the processing for calculating the degree of coincidence Ca as described above, even if there is a point in time when occlusion occurs at which the detection position is not determined, that is, when N>1, the degree of coincidence Ca at time t0 is It can be calculated with certainty.

Furthermore, the appearance cost calculation unit 115c obtains the reciprocal of the degree of matching Ca calculated as described above, and determines the reciprocal as the appearance cost Capp. Here, since the degree of coincidence Ca takes a numerical value that satisfies 0<Ca≦1, the appearance cost Ca takes a value of 1 or more. There is a high possibility that it will not correspond to

<Various Embodiments of Tracking Target Determination Processing>
FIG. 5 is a schematic diagram for explaining various embodiments of the tracking target determination process in the tracking target determination/management unit 115. As shown in FIG.

According to FIG. 5, the tracking target determination/management unit 115 determines the position-related costs Cgeo_A and Cgeo_B, the appearance costs Capp_A and Capp_B, and the communication State costs Cradio_A and Cradio_B are calculated, and the tracking costs Cost_A and Cost_B at time t0 are calculated by the following equation (6) Cost_A = (1 - λ) (Cgeo_A + Capp_A) + λ Cradio_A
(7) Cost_B=(1−λ)・(Cgeo_B＋Capp_B)＋λ・Cradio_B
is determined using

Next, in this embodiment, the tracking target determining/managing unit 115 tracks target A ("person") with the lowest tracking cost (in the case of FIG. 5, the target with the lowest tracking cost). A target related to the terminal 2 is determined, and the tracking of the terminal 2 is continued. Incidentally, in FIG. 5, although there is no difference in appearance cost Capp between the two, both the location-related cost Cgeo and the communication state cost Cradio are clearly smaller for object A ("person").

Here, the tracking target determination/management unit 115 uses, for example, the content of the determination result from the occlusion determination unit 114 to determine the value of the common weight λ (0≤λ≤1) in the above equations (6) and (7). By appropriately setting the value, the accuracy of the determined tracking cost Cost value is increased, and more appropriate tracking target determination/terminal tracking processing is performed.

For example, as a preferred embodiment, the tracking target determining/managing unit 115 determines whether occlusion occurs and disappears for a predetermined target (each of target A and target B in FIG. 5) at time t0, or for a predetermined time from time t0. It is also preferable to increase the weight λ (beyond the value in the situation where no occlusion occurred) until the time t0+Δt after Δt has passed. For example, if λ=0 in a situation where occlusion does not occur, the weight λ may be set to a non-zero finite value, such as 1, at time t0 or for a predetermined time Δt thereafter.

By setting the weight λ in this manner, the weighting factor (λ) applied to the communication state cost Cradio for the predetermined target becomes larger, while the weighting factor (1−λ) applied to the location-related cost Cgeo and the appearance cost Capp becomes smaller or zero, and eventually the tracking cost Cost is calculated with an emphasis on the communication state cost Cradio.

In this way, we reduce or eliminate the contribution of the location-related cost Cgeo and the appearance cost Capp to the tracking cost Cost, which are likely to be less accurate because of occlusion, while at the same time being less affected by occlusion. By more strongly reflecting the communication state cost Cradio that does not exist, it becomes possible to determine the tracking cost Cost that is expected to have considerable accuracy despite occlusion.

As another preferred embodiment, the tracking target determining/managing unit 115 sets the weight λ as time elapses from the time t0 when occlusion occurs and disappears for a predetermined target (targets A and B in FIG. 5). It is also preferable to gradually decrease the value from a larger value (than the value in the situation where occlusion did not occur). For example, when λ=1 at time t0, the value of weight λ may be decreased from 1 to 0 as time elapses.

For example, the attenuation coefficient of the weight λ is δ, the weight λ at time t0 is λ0, and the weight λ is expressed by the following equation (8) λ=λ0・exp(t0−t)
As a result, the tracking cost Cost(t) at time t is given by the following equation (9) Cost(t)=(1−λ0・exp(t0−t))・(Cgeo+Capp)+λ0・exp(t0－t)・Cradio
It is also preferable to calculate using

By setting the weight λ in this way, the weight coefficient (λ) applied to the communication state cost Cradio for the predetermined target decreases sequentially (from 1, for example) toward 0 from the time t0 when the occlusion is resolved. , the weighting factor (1-λ) on the location-related cost Cgeo and the appearance cost Capp gradually increases (for example, from 0) toward 1, and as time passes, the location-related cost Cgeo and the appearance cost Capp become more important. Conversely, the tracking cost Cost is calculated by decreasing the contribution of the communication state cost Cradio.

In this way, as time passes after the occlusion is resolved, the contribution of the position-related cost Cgeo and the appearance cost Capp to the tracking cost Cost, which is expected to have higher accuracy (in the situation where occlusion has not occurred), is reduced. By increasing it sequentially, it becomes possible to determine the tracking cost Cost with higher accuracy.

FIG. 6 is a schematic diagram for explaining still another embodiment of the tracking target determination process in the tracking target determination/management unit 115. As shown in FIG.

In the embodiment shown in FIG. 6, the tracking target determining/managing unit 115 calculates position-related costs Cgeo_A and Cgeo_B and appearance costs Capp_A and Capp_B of target A ("person") and target B ("person") at time t0, respectively. , and communication state costs Cradio_A and Cradio_B are calculated, the tracking costs Cost_A and Cost_B at each time t0 are determined, the target related to the terminal 2 being tracked is determined, and the terminal 2 is tracked.

Here, the tracking target determining/managing unit 115 uses, for example, the content of the determination result from the occlusion determination unit 114 to determine the target A (“person”) and the target B (“ If it is determined that occlusion has occurred between
(a) First, based on the past detection position (object position related information), for example, the detection position at the latest time point tN when occlusion has not yet occurred such that the detection position is not determined, the object related to this occlusion It is determined whether each of A (“person”) and object B (“person”) is on the front side or the back side.

Next, when the occlusion is resolved, that is, at time t0, the tracking target determination/management unit 115
(b1) As shown in FIG. 6A, when it is determined that the time-series data (terminal communication state information) of the received signal power variable E_t is affected by the shielding of communication radio waves, a predetermined (object B ("person") in FIG. 6) is set to a larger predetermined value, and the predetermined object (object A (" Set the communication state cost Cradio (Cradio_A) of person ")) to a smaller value or zero,
(b2) On the other hand, as shown in FIG. 6B, when it is determined that the time-series data (terminal communication state information) of the received signal power variable E_t is not affected by shielding of communication radio waves, The communication state cost Cradio (Cradio_B) of the determined predetermined object (object B ("person") in FIG. 6) is set to a smaller value or zero, and the predetermined object (in FIG. 6 The communication state cost Cradio (Cradio_A) of the target A ("person") is set to a larger predetermined value.

Here, the front side/back side determination in the above (a) is based on the premise that the front side/back side positional relationship at the detection position at the latest time point tN is maintained thereafter. The detected position at time tN is closer to the camera 103 (base station 1) (target B (“person”) in FIG. ) may be the back side. In addition, the determination of the presence or absence of the "effect of shielding of communication radio waves" in the above (b1) and (b2) is performed by determining whether or not there is a drop in received radio wave power or disconnection of wireless connection. This can be done by confirming whether or not there is a time interval in which the value of the variable E_t is less than a predetermined threshold in the E_t time-series data.

As described above, according to the communication state cost setting process of the present embodiment,
(c1) If the terminal communication state information of the terminal 2 being tracked includes "effect of shielding of communication radio waves," it is highly likely that the predetermined target corresponding to the terminal 2 is on the far side. By setting the communication state cost Cradio (Cradio_A) of a predetermined object (object A (“person”) in FIG. 6) determined to be on the same side to be a small value or zero, “this predetermined object (object A (“person”) )) is an object corresponding to the terminal 2 being tracked."

(c2) On the other hand, if the terminal communication status information does not indicate the "effect of shielding of communication radio waves" in the terminal 2 being tracked, there is a high possibility that the predetermined target corresponding to the terminal 2 is on the near side. , the communication state cost Cradio (Cradio_B) of a predetermined object (object B (“person”) in FIG. 6) determined to be on the near side is set to a small value or zero, and “this predetermined object (object B (“ This makes it easier to determine that the person ")) is an object corresponding to the terminal 2 being tracked."

FIG. 7 is a schematic diagram for explaining still another embodiment of the tracking target determination process in the tracking target determination/management unit 115. As shown in FIG.

In the embodiment shown in FIG. 7, the tracking target determination/management unit 115
(a) When occlusion (intersection) occurs and disappears for a plurality of predetermined objects (in FIG. 7, object A (“person”) and object B (“person”)), that is, at time t0, this occlusion is involved Continuing the tracking process for each of these plurality of predetermined targets (target A (“person”) and target B (“person”)) as candidates for the target related to the terminal 2 being tracked,
(b) For a predetermined target whose communication state cost Cradio or tracking cost Cost has increased by satisfying a predetermined condition (for example, has exceeded a predetermined threshold value Cth) during the subsequent tracking process, The tracking process is terminated as it is not a target related to the terminal 2 being tracked.

Here, in FIG. 7, among the object A (“person”) and the object B (“person”), which are candidates for the object related to the terminal 2 being tracked at time t0, the object A (“person”) , at time t0+P (P is a positive integer), the communication state cost Cradio_A exceeds the predetermined threshold value Cth due to the occurrence of shielding by some object. Therefore, at this time t0+P, the tracking target determining/managing unit 115 determines that "the target A ("person") is not a target related to the terminal 2 being tracked." ") to terminate the tracking process.

As described above, according to the process of setting a plurality of candidates according to the present embodiment, by continuing the tracking process as all predetermined targets involved in occlusion as candidates for the tracking target for the time being, it is possible to prevent an ID switch from tracking an erroneous target. Occurrence can be prevented more reliably. In addition, by removing unsuitable candidates based on the behavior of the subsequent communication state cost Cradio and tracking cost Cost, it is also possible to avoid situations where the number of candidates to be tracked increases and the computational load of tracking processing becomes excessive. It becomes possible.

Returning to the functional block diagram of FIG. 1, the tracking target determination/management unit 115 identifies and tracks the results of the tracking process of each terminal 2 that has been described in detail with various embodiments and specific examples above ( For example, time-series data in which the location and image features at each point in time are associated with each terminal ID of the terminal 2) is stored and managed in the tracking information storage unit 105, and is sent to the communication control unit 116 at all times and periodically. , or report as appropriate.

The communication control unit 116 uses the identification/tracking result of the more appropriate terminal 2 (avoiding loss or ID switch) to grasp the situation including the current position and communication route of the terminal 2 connected to communication. However, even if the received power from the terminal 2 with which communication is connected suddenly drops, the communication connection with this terminal 2 can be continued or immediately restarted.

As described in detail above, according to the present invention, when occlusion occurs, it is determined whether or not there is a corresponding relationship between a terminal being tracked and a predetermined detected target based on environmental information ( For example, it can be implemented using the degree of correspondence between the communication state of the terminal and the estimated communication state of the predetermined target, regardless of whether the predetermined target is detected from the image data). As a result, even if occlusion occurs for the terminal being tracked, it is possible to continue tracking this terminal 2 more appropriately.

In addition, such terminal tracking processing according to the present invention solves the problem of communication failure due to obstructions in the communication path in 5G, which is currently being introduced, and prevents people or other vehicles from approaching a vehicle equipped with a terminal. In addition, various situations such as determining the relationship between the page viewed by the user determined to be in correspondence with a certain terminal and the flow line of the user and utilizing it for marketing and management・It is possible to apply it in various fields.

Furthermore, by popularizing the information communication system including the terminal tracking device according to the present invention, which can stably realize a huge number of large-capacity communication connections, mobile and mobile communication services that increase connections between people will be widely developed. It is also possible to aim for community empowerment. Moreover, it is believed that the information communication system including the terminal tracking device according to the present invention will exert its power particularly in urban areas where the population is increasing worldwide. That is, according to the present invention, the United Nations-led Sustainable Development Goals (SDGs) Goal 9 "Develop resilient infrastructure, promote sustainable industrialization, and expand innovation" and Goal 11 " It will also be possible to contribute to the achievement of “making cities inclusive, safe, resilient and sustainable.”

In the various embodiments of the present invention described above, various changes, modifications and omissions within the scope of the technical ideas and aspects of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to limit the present invention in any way. The invention is to be limited only as limited by the claims and the equivalents thereof.

1 base station (terminal tracking device, communication relay device)
101 communication interface 102 communication history information accumulation unit 103 camera (sensor)
104 Environmental information accumulation unit 105 Tracking information storage unit 111 Terminal communication state determination unit 112 Target communication state determination unit 112a Position interpolation unit 113 Target detection information determination unit 114 Occlusion determination unit 115 Tracked target determination/management unit 115a Communication state cost calculation unit 115b Location-related cost calculator 115c Appearance cost calculator 116 Communication controller 2 Terminal

Claims (12)

Detecting a predetermined target including the terminal from environmental information obtained by a sensor that senses the environment in which the terminal may exist and which may include the predetermined target, and tracking the terminal. A terminal tracking device,
terminal communication state determination means for determining terminal communication state information, which is information regarding the state of communication with the terminal, based on information regarding communication acquired from communication means that communicates with the terminal;
target communication state determining means for determining, from the environment information, target communication state information, which is information relating to a communication state to be realized with the predetermined target if the predetermined target includes a terminal;
The degree of communication compatibility, which is the degree of correspondence between the terminal communication state information determined for the terminal being tracked and the target communication state information determined for the predetermined target when occlusion occurs for the predetermined target. and determines whether or not the tracked terminal and the predetermined target have a corresponding relationship based on the degree of communication compatibility, and the predetermined target determined to have a corresponding relationship a tracking target determination means for determining a target related to a terminal that is tracking a terminal. The target communication state determination means determines that the predetermined target is located at a position interpolated using the detection positions determined before and after the occlusion in a time interval during which occlusion occurs for which the detection position of the predetermined target is not determined. 2. The terminal tracking device according to claim 1, wherein the target communication state information is determined as existing. Determining target detection information for determining target position-related information, which is information relating to the detection position of the predetermined target, and/or target appearance information, which is information relating to the detected appearance of the predetermined target, from the environment information further comprising means;
The tracking target determination means is a degree of position-related matching, which is a degree of matching between the determined target position-related information and predicted target position-related information predicted from past target position-related information, and/or the determined Determining whether or not the tracked terminal and the predetermined target have a corresponding relationship based on the degree of appearance matching, which is the degree of matching between the target appearance information and the past target appearance information. 3. The terminal tracking device according to claim 1 or 2. Determining target detection information for determining target position-related information, which is information relating to the detection position of the predetermined target, and/or target appearance information, which is information relating to the detected appearance of the predetermined target, from the environment information further comprising means;
The tracked object determining means causes at least the determined object position-related information and the predicted object position-related information predicted from the past object position-related information to match in a time interval in which occlusion does not occur for the predetermined object. and/or based on the degree of appearance matching, which is the degree of matching between the determined target appearance information and the past target appearance information, the tracking terminal and the predetermined 4. The terminal tracking device according to any one of claims 1 to 3, wherein it is determined whether or not there is a corresponding relationship with an object. The tracked object determining means may determine, for the predetermined object, a location-related cost that is a monotonically decreasing function of the degree of location matching, and/or an appearance cost that is a monotonically decreasing function of the degree of appearance matching, and a monotonically decreasing degree of communication compatibility. A tracking cost that is a weighted sum of a communication state cost that is a decreasing function and a weighting factor that can take a zero value is calculated, and a predetermined target whose tracking cost is small enough to satisfy a predetermined condition is sent to the tracking terminal. 5. The terminal tracking device according to any one of claims 1 to 4, wherein the target is determined. The tracked object determining means increases a weighting factor applied to the communication state cost in the predetermined object when occlusion in which the detected position is not determined for the predetermined object occurs and is resolved, and the position-related cost and/or 6. The terminal tracking device according to claim 5, wherein the tracking cost is calculated with a smaller or zero weighting factor applied to the appearance cost. The tracked object determination means sequentially decreases the weighting factor for the communication state cost of the predetermined object from a larger value as time elapses from the time when the occlusion occurs and disappears for the predetermined object, and the position-related 7. The terminal tracking device according to claim 6, wherein the tracking cost is calculated by increasing a weighting factor for the cost and/or the appearance cost from a smaller value or from zero. The tracked object determining means increases a weighting factor applied to the communication state cost for the predetermined object for a period of time from the time when the occlusion occurs and disappears for the predetermined object to the time when the predetermined time elapses, 8. The terminal tracking device according to any one of claims 5 to 7, wherein a tracking cost is calculated with a smaller or zero weighting factor applied to the location-related cost and/or the appearance cost. The tracked object determining means is
when occlusion occurs between the predetermined targets, determining whether each of the predetermined targets related to the occlusion is on the front side or the back side based on the past target position-related information;
When the occlusion is resolved and it is determined that the terminal communication state information has been affected by the shielding of communication radio waves, the communication state cost of the predetermined target determined to be on the near side is set to a larger value. , the communication state cost of a predetermined target determined to be on the back side is set to a smaller value or zero, and if it is determined that there is no effect of shielding of communication radio waves in the terminal communication state information, it is on the front side A claim characterized in that the communication state cost of a predetermined target determined to be on the back side is set to a smaller value or zero, and the communication state cost of a predetermined target determined to be on the back side is set to a larger value 9. The terminal tracking device according to any one of items 5 to 8. The tracked object determining means is
when occlusion occurs and disappears for a plurality of predetermined targets, continuing the tracking process for each of the plurality of predetermined targets as a target candidate for the terminal being tracked;
Ending the tracking process for a predetermined target whose tracking cost or communication state cost has increased to the extent that satisfies a predetermined condition in the continuous tracking process as not being a target related to the terminal being tracked. The terminal tracking device according to any one of claims 5 to 9, characterized by: Detecting a predetermined target including the terminal from environmental information obtained by a sensor that senses the environment in which the terminal may exist and which may include the predetermined target, and tracking the terminal. A terminal tracking program operating a computer, comprising:
terminal communication state determination means for determining terminal communication state information, which is information regarding the state of communication with the terminal, based on information regarding communication acquired from communication means that communicates with the terminal;
target communication state determining means for determining, from the environment information, target communication state information, which is information relating to a communication state to be realized with the predetermined target if the predetermined target includes a terminal;
The degree of communication compatibility, which is the degree of correspondence between the terminal communication state information determined for the terminal being tracked and the target communication state information determined for the predetermined target when occlusion occurs for the predetermined target. and determines whether or not the tracked terminal and the predetermined target have a corresponding relationship based on the degree of communication compatibility, and the predetermined target determined to have a corresponding relationship A terminal tracking program characterized by causing a computer to function as a tracking target determination means for determining a target related to a terminal that is running. Detecting a predetermined target including the terminal from environmental information obtained by a sensor that senses the environment in which the terminal may exist and which may include the predetermined target, and tracking the terminal. A computer-implemented terminal tracking method comprising:
determining terminal communication status information, which is information relating to the status of communication with the terminal, based on information relating to communication acquired from a communication means that communicates with the terminal;
determining, from the environment information, target communication state information, which is information relating to a communication state to be realized with the predetermined object if the predetermined object includes a terminal;
The degree of communication compatibility, which is the degree of correspondence between the terminal communication state information determined for the terminal being tracked and the target communication state information determined for the predetermined target when occlusion occurs for the predetermined target. and determines whether or not the tracked terminal and the predetermined target have a corresponding relationship based on the degree of communication compatibility, and the predetermined target determined to have a corresponding relationship and determining a target associated with a terminal that is tracking a terminal.

Images