JP2023116958A - Terminal tracking device, program, and method for performing tracking based on degree of communication compatibility of detection target - Google Patents

Abstract

To provide a technique for continuing tracking a tracked terminal even if occlusion occurs in the tracked terminal.SOLUTION: A terminal tracking device includes: a target detection information determination unit that detects a predetermined target and determines target position information; a terminal communication information determination unit that determines terminal communication information based on information from a communication interface; means that determines target communication information related to a communication state to be embodied if a target includes a terminal; and a tracking target determination and management unit that determines whether or not there is a correspondence relation between the terminal and the target based on the degree of accordance between the target position information and predicted target position information and/or the degree of correspondence between the terminal communication information and the target communication information. When occlusion is eliminated for at least the target, the tracking target determination and management unit makes a determination based on the degree of communication compatibility when the degree of communication compatibility is larger than a predetermined level or makes a determination by increasing a degree based on the degree of communication compatibility as the degree of communication compatibility is larger or when it is larger than the predetermined level.SELECTED DRAWING: Figure 1

Description

The present invention relates to technology for acquiring information about a terminal as a communication target, and particularly to technology for tracking the terminal.

The 5th generation mobile communication system (5G), which is currently being introduced, uses radio waves in the millimeter wave band (millimeter waves) and can realize high-performance communication such as high speed, large capacity, low delay, and multi-terminal connection. do. Here, millimeter waves have high rectilinearity and are less likely to be diffracted, and there is a high possibility that their propagation will be blocked or attenuated by an object existing between the terminal and the base station. Therefore, in 5G, the presence of such an object causes a sudden drop in received power, which can cause a situation in which, for example, the communication connection itself is interrupted, which is a serious problem.

Therefore, if it is possible to predict a decrease in received power due to such an object, it becomes possible to maintain communication connection without interruption by controlling/securing the communication path of the terminal. As a technique for predicting the received power, Non-Patent Document 1 describes the use of time-series 3D environmental information acquired by an RGB-D camera capable of generating not only RGB images but also depth images, and using machine learning to perform reception in a communication device. Techniques for predicting power are disclosed.

Specifically, this technology corrects the time-series information of the received power measured (at the same time as the image is taken) by the communication terminal that received the radio waves from the transmitting station for the time-series environmental information that is the result of imaging the communication environment. A machine learning model is constructed using the training data linked as data, and using this constructed machine learning model, the predicted value of the received power at the communication terminal is calculated from the time-series environmental information acquired by the RGB-D camera. have decided.

Here, in such millimeter-wave reception power prediction, it is important to continuously grasp the position of the communication terminal. However, if the position is determined using the positioning results obtained by the sensor on the communication terminal side, a mechanism for notifying the positioning results from the communication terminal to the base station is required. load becomes a problem.

On the other hand, as a technology that does not depend on the positioning result of the communication terminal side, Patent Document 1, which includes the inventors of the present application, discloses time-series information of millimeter-wave received power and distance between objects estimated from, for example, image information of the environment. A technique for estimating the position of a communication terminal using shielding-related time-series information is disclosed. Furthermore, Non-Patent Document 2 discloses a technique for continuously capturing the position of an object on an image by using the consistency of the detected position of the object and the features of its appearance obtained from the image. .

JP 2021-078022 A

Masashi Nishio, "Prediction and Control of Wireless Communication Quality by Machine Learning", IEICE Technical Report, Vol. 118, No. 57, SR2018-1, pp.1-8, 2018 Nicolai Wojke et al., "Simple Online and Realtime Tracking with a Deep Association Metric", 2017 IEEE International Conference on Image Processing, pp. 3645-3649, 2017. <https://doi.org/10.1109/ICIP.2017.8296962 ＞

Here, if it is possible to continue tracking the communication terminal using such a conventional technology, even if the received power suddenly drops due to the intervention of an object, the communication connection can be continued or temporarily suspended. It is also possible to resume immediately with

However, in the object tracking technology based on image information as disclosed in Non-Patent Document 2, when an object on the front side covers at least a part of an object on the back side, a so-called occlusion phenomenon occurs, tracking There is a possibility that the tracking of the target will fail. In other words, when occlusion occurs, conventional object tracking technology based on image information suffers from “lost,” in which the tracked object is hidden by other objects and becomes undetectable, and tracking is interrupted. crossing, the other object is misidentified as the object to be tracked, and an "ID switch" occurs in which the object is not tracked as it should be.

Therefore, even if the object tracking technique disclosed in Non-Patent Document 2, for example, is applied to tracking a communication terminal, it is difficult to accurately grasp the position of the communication terminal when such a problem related to occlusion occurs. I can't. Also, the technique of estimating the position of a communication terminal described in Patent Document 1 is based on the premise that no loss or ID switch has occurred, and does not propose any countermeasures against such problems. not

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a terminal tracking device, a program, and a method capable of more appropriately continuing tracking of a terminal being tracked even when occlusion occurs.

According to the invention,
Target detection that detects a predetermined target that may include the terminal from environmental information generated by a sensor and is related to the environment in which the terminal may exist, and determines target position information that is information related to the detection position of the detected target. an information determining means;
terminal communication information determination means for determining terminal communication information, which is information regarding a communication state with the terminal, based on information regarding communication acquired from a communication means that communicates with the terminal;
a target communication information determining means for determining, from the environment information, target communication information, which is information relating to a communication state to be realized with a detected target if the target includes a terminal;
The degree of position matching, which is the degree of matching between the target position information and the predicted target position information predicted from the target position information determined for the tracking terminal, and the tracking terminal Determining whether or not the tracked terminal and the detected target are in correspondence based on at least one of the degree of correspondence between the terminal communication information and the target communication information. a tracked object determining means for making a determination and determining the object determined to have a corresponding relationship as an object related to the terminal being tracked;
The tracked object determination means makes the determination based on the degree of communication compatibility, not based on the degree of location matching when the degree of communication compatibility is large enough to satisfy a predetermined condition, or the higher the degree of communication compatibility, the more Provided is a terminal tracking device characterized by making the determination by increasing the degree based on the degree of communication compatibility when the degree of correspondence is large enough to satisfy a predetermined condition.

As an embodiment of the terminal tracking device according to the present invention, the tracked object determining means, when the degree of communication compatibility is small enough to satisfy a predetermined condition, and the degree of position matching is small enough to satisfy the predetermined condition, the communication concerned. It is also preferable to make the determination without being based on the degree of compatibility, or to make the determination with a smaller degree based on the degree of communication compatibility as the degree of position matching is smaller.

Further, as another embodiment of the terminal tracking device according to the present invention, the object detection information determination means also determines object appearance information, which is information related to the detected appearance of the detected object, from the environment information,
When the tracked object determination means makes the determination based on at least the degree of position matching, the degree of matching between the object appearance information and the past object appearance information of the object determined to be related to the terminal being tracked. It is also preferable to perform the determination based on the degree of matching of appearance.

Furthermore, in the embodiment related to the target appearance information described above, the tracked target determination means includes:
When the degree of position matching exceeds a predetermined condition and is small, the determination is made based on at least the degree of appearance matching and the degree of position matching, and the degree based on the degree of appearance matching is increased at that time, or
It is also preferable to perform the determination based on at least the degree of appearance matching and the degree of position matching, and to increase the degree based on the degree of appearance matching as the degree of position matching decreases.

Further, as an embodiment related to the target communication information determination means, the target communication information determination means uses the detection positions determined before and after the occlusion in the time period during which the occlusion for which the detection position is not determined for the target occurs. It is also preferable to determine the target communication information assuming that the target exists at the position interpolated by

Furthermore, as an embodiment relating to the degree of position matching, the environment information is information relating to an image of the environment, and the target position information is the position and including the range,
The degree of position matching is the degree of overlap between the range of the detected object and the expected range in the image of the determined object associated with the tracking terminal, or It is also preferably the degree of closeness between the location of the object obtained and the predicted location in the image of the determined object associated with the tracking terminal.

Further, as a preferred embodiment of the tracking target determination process according to the present invention, the tracking target determination means includes:
Determining a weighting factor that can take a zero value, which is a monotonically increasing function of the degree of communication compatibility,
A weighted sum of an environmental information cost including a location cost that is a monotonically decreasing function of the degree of location matching and a communication cost that is a monotonically decreasing function of the degree of communication compatibility, wherein the weighting factor is applied to the communication cost, A tracking cost of the target is calculated, which is a weighted sum in which another weighting factor that can take a zero value is applied to the environmental cost, and the target whose tracking cost is small until it satisfies a predetermined condition is tracked. It is also preferable to determine the target related to the terminal.

Furthermore, in the above-described embodiment of determining the weighting factor, the tracked object determination means preferably determines the weighting factor as a monotonically increasing function for both the degree of communication correspondence and the degree of position matching.

According to the present invention, also
Detecting a predetermined target that may include the terminal from environmental information generated by a sensor and relating to the environment in which the terminal may exist, determining target position information that is information relating to the detection position of the detected target, and , object detection information determining means for also determining object appearance information, which is information relating to the detected appearance of the detected object, from the environment information;
terminal communication information determination means for determining terminal communication information, which is information regarding a communication state with the terminal, based on information regarding communication acquired from a communication means that communicates with the terminal;
a target communication information determining means for determining, from the environment information, target communication information, which is information relating to a communication state to be realized with a detected target if the target includes a terminal;
The degree of position matching, which is the degree of matching between the target position information and the predicted target position information predicted from the target position information determined for the terminal being tracked, the target appearance information and the tracked target The degree of appearance matching, which is the degree of matching with the past target appearance information of the target determined to be related to the terminal being tracked, and the degree of correspondence between the terminal communication information of the terminal being tracked and the target communication information Based on at least one of the degree of communication compatibility, it is determined whether or not the tracked terminal and the detected target have a corresponding relationship, and the target determined to have a corresponding relationship as a target related to the terminal being tracked, and
The tracked object determination means makes the determination based on the degree of communication compatibility, not based on the degree of location matching, when the degree of communication compatibility is large enough to satisfy a predetermined condition at least when occlusion is eliminated for the detected target. or, when the degree of communication correspondence is large or is large until a predetermined condition is satisfied, the degree based on the degree of communication correspondence is increased and the determination is performed.

According to the present invention, furthermore,
Target detection that detects a predetermined target that may include the terminal from environmental information generated by a sensor and is related to the environment in which the terminal may exist, and determines target position information that is information related to the detection position of the detected target. an information determining means;
terminal communication information determination means for determining terminal communication information, which is information regarding a communication state with the terminal, based on information regarding communication acquired from a communication means that communicates with the terminal;
a target communication information determining means for determining, from the environment information, target communication information, which is information relating to a communication state to be realized with a detected target if the target includes a terminal;
The degree of position matching, which is the degree of matching between the target position information and the predicted target position information predicted from the target position information determined for the tracking terminal, and the tracking terminal Determining whether or not the tracked terminal and the detected target are in correspondence based on at least one of the degree of correspondence between the terminal communication information and the target communication information. causing the computer to function as a tracked object determination means for determining the object determined to be in a corresponding relationship as the object related to the terminal being tracked;
The tracked object determination means makes the determination based on the degree of communication compatibility, not based on the degree of location matching, when the degree of communication compatibility is large enough to satisfy a predetermined condition at least when occlusion is eliminated for the detected target. or making the judgment with a larger degree based on the degree of communication compatibility as the degree of communication compatibility increases or when the degree of communication compatibility is large until a predetermined condition is satisfied.

According to the present invention, still further,
Detecting a predetermined target that may include the terminal from environmental information generated by a sensor and relating to the environment in which the terminal may exist, determining target position information that is information relating to the detection position of the detected target, and , terminal communication information, which is information relating to the state of communication with the terminal, is determined based on information relating to communication acquired from a communication means that communicates with the terminal; a first step of determining target communication information, which is information relating to a communication state to be embodied with a target if the target includes a terminal;
The degree of position matching, which is the degree of matching between the target position information and the predicted target position information predicted from the target position information determined for the tracking terminal, and the tracking terminal Determining whether or not the tracked terminal and the detected target are in correspondence based on at least one of the degree of correspondence between the terminal communication information and the target communication information. a second step of determining and determining the object determined to be in correspondence with the object associated with the tracking terminal;
In the second step, when occlusion is eliminated at least for the detected object, the determination is based on the degree of communication compatibility, not based on the degree of position matching when the degree of communication compatibility is large enough to satisfy a predetermined condition. or, when the degree of communication compatibility is large or is large until a predetermined condition is satisfied, the determination is made by increasing the degree based on the degree of communication compatibility. A method is provided.

According to the terminal tracking device, program, and method of the present invention, even when an occlusion occurs in a terminal being tracked, it is possible to continue tracking the terminal in question more appropriately.

It is a functional block diagram showing a functional configuration in one embodiment of the terminal tracking device according to the present invention. FIG. 4 is a schematic diagram for explaining two modes of occlusion determination processing in an occlusion determination unit according to the present invention; FIG. 4 is a schematic diagram for explaining a specific example of communication cost calculation processing according to the present invention; FIG. 10 is a schematic diagram for explaining a specific example of the position cost and appearance cost calculation processing according to the present invention; FIG. 4 is a schematic diagram for explaining various embodiments of tracking target determination processing in a tracking target determination/management unit according to the present invention;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Terminal tracking device]
FIG. 1 is a functional block diagram showing a functional configuration in one embodiment of a terminal tracking device according to the present invention.

The base station 1 as one embodiment of the terminal tracking device according to the present invention shown in FIG. 1 is a communication relay device compatible with 5G (fifth generation mobile communication system) in this embodiment, It is a device capable of communicating with a plurality of terminals 2, which are communication terminals.

In addition, the base station 1, as an embodiment of the terminal tracking device, is information related to the environment acquired by the camera 103 as a sensor that senses the environment in which the terminal 2 may exist, and may include a predetermined target. It is also a device capable of detecting a predetermined target including the terminal 2 from image data as "environmental information" and tracking the terminal 2.

Here, in this embodiment, the camera 103 is an RGB camera capable of photographing the situation of the communication area with a predetermined angle of view. Image data (video data) generated by the camera 103 includes:
(a) a person who possesses or carries the terminal 2,
(b) Mobile objects such as automobiles, motorcycles, railroad vehicles, robots, drones, etc. equipped with, including or boarded by the terminal 2, and (c) equipment, facilities, buildings, fixed objects where the terminal 2 is installed There is a possibility that such "object" is included (imaged). Furthermore, "objects" such as people, mobile objects, equipment, facilities, buildings, and fixed objects (including plants such as trees) that are not related to the terminal 2 can also be included.

By the way, such a “target” can be the one that should determine the correspondence with the tracked terminal 2 (in other words, the one that should determine whether or not it is the tracked target that should be tracked), while the terminal 2 and the base station 1 It can also become an obstacle for communication radio waves.

Here, in particular, 5G, which is the communication method of the present embodiment, uses radio waves in the millimeter wave band (millimeter waves) for communication. , its propagation is likely to be blocked or attenuated by "objects" that exist between the terminal 2 and the base station 1. Therefore, in 5G, there is a serious problem that the power of the received signal at the base station 1 is suddenly reduced due to the intervention of such a "target", and for example, the communication connection itself may be interrupted.

On the other hand, even if "occlusion" occurs in the terminal 2 being tracked, if the tracking of this terminal 2 can be appropriately continued, the communication connection with the terminal 2 can be continued or temporarily suspended. It is also possible to restart immediately after doing so.

Incidentally, the above-mentioned "occlusion" is hereinafter referred to as a phenomenon in which an "object" on the near side as viewed from the base station 1 obscures at least a part of an "object" on the far side. For example, a phenomenon in which a plurality of "objects" intersect or at least one "object" is occluded by at least one other "object" is also treated as "occlusion" below. When such "occlusion" occurs, there is a possibility that a predetermined target (e.g., "person") cannot be detected from "environmental information" (image data in this embodiment) or can only be detected inaccurately. It increases.

In order to solve the serious problem when occlusion as described above occurs, the base station 1 specifically:
(A) Detecting a predetermined target (for example, a "person") that may include the terminal 2 from "environmental information" (image data in this embodiment) generated by the sensor (camera 103 in this embodiment), a target detection information determination unit 113 that determines “target position information” that is information related to the detection position of the detected target (“person”);
(B) Based on information (for example, communication history information) related to communication acquired from communication means (communication interface 101, communication control unit 116, and communication history information storage unit 102 provided by itself in this embodiment) that communicates with terminal 2 a terminal communication information determination unit 111 that determines "terminal communication information" that is information related to the communication state with the terminal 2;
(C) From the "environmental information", if the detected target (for example, "person") includes the terminal 2, "target communication information", which is information relating to the communication state that will be realized with this target, is selected. a target communication information determining unit 112 to determine;
(D) (d1) Matching the "target location information" with the predicted target location information predicted from the "target location information" of the object (e.g., "person") determined to be related to the terminal 2 being tracked and (d2) a "communication correspondence degree" that is a degree of correspondence between the "terminal communication information" of the terminal 2 being tracked and the "target communication information". Based on one of them, it is determined whether or not there is a corresponding relationship between the tracked terminal 2 and the detected target ("person"), and the target ("person") determined to have a corresponding relationship is determined. , and a tracking target determining/managing unit 115 that determines a target related to the terminal 2 being tracked.

Furthermore, the tracking target determining/managing unit 115 in (D) above, when occlusion occurs and is resolved at least for the detected target (for example, a “person”),
(D1) When the "degree of communication readiness" is large enough to satisfy a predetermined condition (in this embodiment, the representative value (for example, the maximum value) of the degree of communication readiness of each detected target exceeds a predetermined threshold value. the above determination is made based on the "degree of communication compatibility" instead of the "degree of location matching", or
(D2) The larger the "communication compatibility level" (in this embodiment, the larger the representative value (for example, the maximum value) of the communication compatibility level of each detected target), or the larger it meets a predetermined condition (this embodiment In the form, when the representative value (for example, the maximum value) exceeds a predetermined threshold value), the degree based on the "communication correspondence degree" is increased and the above determination is performed. be.

Here, the ``predetermined object'' detected from the ``environmental information'' in (A) above is a ``person'' or a ``moving body'' who carries, is equipped with, or may include the terminal 2, as will be described in detail later. " can be. Furthermore, depending on the case, it is possible that "equipment", "facility", "building", "fixed object", etc., which may include the terminal 2, may be the "predetermined object". The "target position information" in (A) above is information related to the target detection position, and is position-related information that can include the position, speed, etc. of the detected target (for example, "person").

Furthermore, the above (B) "terminal communication information" is preferably (α) information related to the strength of the radio wave received from the terminal 2 (for example, received signal power, RSSI (Received Signal Strength Indicator)). be able to. Alternatively, (β) information relating to the presence/absence of communication connection with the terminal 2 may be used. In addition, the above (C) "target communication information" is combined with the above "terminal communication information", and (α') if the detected target (for example, "person") includes the terminal 2, the predetermined target It can be information (eg RSSI) about the strength of the radio waves that will be received from (“person”). Alternatively, (β′) can be information regarding whether or not communication connection with a detected target (for example, a “person”) is determined using the detection position of the target (“person”).

Further, the above (D) "degree of communication support" is specifically, for example, time-series data of the amount (for example, RSSI value) related to the strength of radio waves as "terminal communication information" and radio waves as "target communication information" can be the degree of matching with the time-series data of the quantity (for example, the RSSI value) related to the intensity of . Alternatively, it is also possible to use the degree of matching between time-series data relating to the presence or absence of communication connection as "terminal communication information" and time-series data relating to the availability of communication connection as "target communication information".

Under such settings of the "degree of communication support" and the "degree of position matching", the base station 1 as the terminal tracking device according to the present invention, at least when the occlusion is resolved, is shown in (D1) and (D2) above. , (after confirming the magnitude of the degree of communication readiness of each detected target), the above determination is made by increasing the degree based on the "degree of communication readiness", or only the "degree of communication readiness" is It is used as a criterion in the determination of

Here, the "degree of communication support" is determined based on the communication state of the terminal 2 and the "environmental information", regardless of whether or not the object to be determined (for example, "person") is detected midway. It can be determined from the estimated communication state of the target. In other words, there is a high possibility that the "degree of communication compatibility" can be used as a criterion with higher reliability even when occlusion occurs. Therefore, according to the terminal tracking device (base station 1) according to the present invention, even when occlusion occurs in the terminal 2 being tracked, it is possible to continue tracking the terminal 2 more appropriately.

Further, in this embodiment, even if an object such as a person intervenes between the base station 1 and the terminal 2 and the reception power drops sharply, the base station 1 can still communicate with the terminal 2. It is possible to continue the connection or to resume it immediately.

Incidentally, in this embodiment, the base station 1 (terminal tracking device), except for a predetermined period immediately after the occlusion is resolved, during a period in which no occlusion occurs, at least (a) the above-described "degree of position matching",
(b) based on either or both of the "appearance matching degree" determined from the "environmental information" (for example, image data), which will be described in detail later, the terminal 2 being tracked and the detected object (for example, It is also preferable to determine whether or not there is a corresponding relationship with "person"), and further,
(c) "Communication correspondence degree" mentioned above
It is also preferable to make the determination based on

In addition, although 5G is adopted as the communication method between the base station 1 and the terminal 2 in this embodiment, it is natural that other mobile communication methods such as LTE and future mobile communication methods that use radio waves with shorter wavelengths will be used. Any communication scheme may be used, and furthermore, the communication between the terminal tracking device according to the invention and the communication terminal may be based on various other wireless communication standards. For example, even in communication systems where the problem of shielding by objects is not as prominent as in 5G, there are more than a few situations in which it is desirable to accurately identify and track subordinate communication terminals using targets detected from camera image data. In addition, the need for terminal identification and tracking is applied in various fields, such as determining the relationship between the user's browsing page determined to be in correspondence with a certain communication terminal and the user's flow line and utilizing it for marketing and management. Existing. In contrast, according to the terminal tracking device according to the present invention, it is possible to perform such terminal identification/tracking processing with high accuracy.

Further, the terminal tracking device according to the present invention is of course not limited to a communication relay device such as a base station. For example, as a dedicated device for terminal tracking processing, it may be provided in the form of being connected to a communication relay device/equipment such as a base station. Also, as the terminal tracking device according to the present invention, a cloud server, a non-cloud server device, a personal computer (PC), a notebook or tablet computer, or a smartphone, etc., which is equipped with the terminal tracking program according to the present invention, can be used. It is possible.

Furthermore, at least one of the above (A) to (D), which are components of the terminal tracking device (base station 1) according to the present invention, is a device different from the other components. It is not impossible to take. For example, it is possible to implement the above functions (A) to (D) by a plurality of servers as a whole. Even in such a case, the entirety of these can be regarded as a terminal tracking device or system that implements the terminal tracking method according to the present invention and that is operated by the terminal tracking program according to the present invention.

[Functional configuration of terminal tracking device, terminal tracking program]
Similarly, in the functional block diagram of FIG. 1, the base station 1 includes a communication interface 101, a communication history information accumulation unit 102, a camera 103, an environment information It has a storage unit 104, a tracking information storage unit 105, and a processor memory (arithmetic processing system having a memory function). Here, the processor memory stores an embodiment of a communication relay program including a terminal tracking program according to the present invention, and has computer functionality, and by executing this communication relay program, Perform terminal tracking processing and communication relay processing.

The processor memory includes, as functional components, a terminal communication information determination unit 111, a target communication information determination unit 112 including a position interpolation unit 112a, a target detection information determination unit 113, an occlusion determination unit 114, a communication cost It has a tracking target determination/management unit 115 including a calculation unit 115a, a location cost calculation unit 115b, an appearance cost calculation unit 115c, and a weight determination unit 115d, and a communication control unit . These functional configuration units can be regarded as the functions of the terminal tracking program and the communication relay program stored in the processor memory. can be understood as an embodiment of the terminal tracking method and communication relay method according to the present invention.

Similarly, in the functional block diagram of FIG. is acquired and recorded, and communication history information is generated by organizing the information in chronological order, and the communication history information storage unit 102 stores and manages the information.

Further, the camera 103 may be, for example, a visible light, near-infrared, or infrared imaging device equipped with a solid-state imaging device such as a CCD image sensor or a CMOS image sensor. An omnidirectional (omnidirectional) camera or the like can also be used. Of course, instead of the camera 103, it is also possible to adopt a sensor capable of sensing the environment in which the terminal 2 may exist and generating environmental information, such as LiDAR, laser/infrared positioning device, TOF camera, and thermography device. be.

Here, in this embodiment, the camera 103 is an RGB camera capable of generating RGB image data (RGB video data) as environmental information, and the generated RGB image data is stored and managed by the environmental information storage unit 104. be done. Note that the camera 103 is installed inside the base station 1 in the present embodiment, but is installed at a position distant from the base station 1, for example, a surveillance camera in town, and communicates with the base station 1. It may be connected.

Moreover, a plurality of cameras 103 may be provided at mutually different positions regardless of whether they are inside or outside the base station 1 . For example, the base station 1 receives "target location information" and "target communication information" generated using the camera 103 installed in the base station 1 from each of a plurality of base stations 1 existing around itself. - You may acquire and carry out a tracking object determination process. In this case, it becomes possible for a plurality of base stations 1 to work together to more preferably implement the terminal tracking processing according to the present invention.

<Target detection information determination processing>
Similarly, in the functional block diagram of FIG.
At least one of (a) target position information, which is information relating to the detection position of the "predetermined target", and (b) target appearance information, which is information relating to the detected appearance of the "predetermined target" , preferably both.

As already mentioned here, the "predetermined target" is, for example,
(a) a person (user) who possesses or carries the terminal 2,
(b) A vehicle equipped with a terminal 2 having a drive recorder function, a CAN information transfer function, an interface function for automatic driving control by a server, etc.,
(c) "Person" who may include the terminal 2 as the communication destination related to the communication history information, such as an autonomous mobile robot or an autonomous flying drone equipped with a terminal 2 having an interface function for autonomous movement control by a server It can be "or" mobile. In some cases, "facility", "facility", "building", "fixed object", etc., which may include the terminal 2, may also be the "predetermined object".

Of course, the target detection information determining unit 113 of the present embodiment can also detect multiple types of targets including the "predetermined target" at the same time. Here, when one "predetermined target" as a tracked target is detected from one image data, other targets (including other "predetermined targets") detected from this image data are In contrast to the "predetermined object", it is treated as an object that can cause occlusion, ie, an object that can cause obstruction.

Here, the determination processing of the above (a) target position information and the above (b) target appearance information will be described. In this embodiment, the object detection information determination unit 113 sets a plurality of image regions in the acquired image data, calculates a score representing a predetermined object likelihood in each image region using the corresponding object detector, and calculates a predetermined score. The image regions with scores high enough to satisfy the condition are determined to be the detected detected image regions of the given object. then
(a) The position (detection position) of the detection image area related to the detected object, the range of the detection image area (for example, the coordinate values of the four corners), and the change in the detection position between consecutive image data Let the detected speed be the target position information in (a) above,
(b) The image feature amount of the detection image region related to the detected target, calculated using a known image feature extractor (e.g., CNN (Convolutional Neural Networks) image feature extractor), is calculated using the above (b) target Appearance information.

Regarding the above object detector, for example, in Non-Patent Document: Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao, “YOLOv4: Optimal Speed and Accuracy of Object Detection”, arXiv:2004.10934v1, 2020 What is described can be adopted. Furthermore, the above (a) target position information and the above (b) target appearance information may be determined using various other object detectors and image feature extractors known in the field of image recognition technology. It is possible.

When a stereo camera is used as the camera 103, for example, non-patent literature: Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang The object detector described in Fu, Alexander C. Berg, “SSD: single shot multibox detector”, European Conference on Computer Vision, Computer Vision-ECCV 2016, pp.21-37, 2016 may be applied.

As still another embodiment, when LiDAR is used instead of the camera 103 and three-dimensional point cloud data is used as environmental information, an object detector for predetermined target detection may be used, for example, by non-patent document: Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas, “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation”, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 77-85, 2017 can be adopted.

Furthermore, in the present embodiment using image data as environment information, instead of the above image feature extractor, for example, non-patent literature: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep Residual Learning for Image Recognition ”, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1, pp.770-778, 2016.

Incidentally, in this embodiment, the target detection information determination unit 113 stores the determined "target position information" and "target appearance information" for each predetermined detected target for a past predetermined period. It can be provided to the decision/management unit 115 as appropriate.

<Terminal communication information determination processing>
Similarly, in the functional block diagram of FIG. A certain "terminal communication information" is determined. Here, in this embodiment, this "terminal communication information" is
Either or both of (α) time-series data of received signal power (RSSI) of the radio signal received from the terminal 2 and (β) time-series data of information regarding the presence or absence of communication connection with the terminal 2 ing.

Regarding (α) above, the terminal communication information determining unit 111 may generate time-series data of the received signal power variable E_t indicating the signal power value at time t of the radio signal received from the terminal 2 . In addition, the terminal communication information determination unit 111 determines whether or not the communication connection with the terminal 2 is established at each point in time for the above (β). Time-series data of a binary communication connection presence/absence variable B_t (ε{-1, 1}) that takes a value of 1 when communication is not established and -1 when communication is not established may be generated. Here, considering the accuracy of the matching degree (communication correspondence degree) with the "target communication information" to be calculated later, the "terminal communication information" includes at least the time-series data of the received signal power variable E_t of (α) above. is preferred.

Incidentally, in this embodiment, the terminal communication information determination unit 111 stores the “terminal communication information” determined for each tracked terminal 2 for a predetermined period in the past. It is available as appropriate. In addition, the terminal communication information determination unit 111 acquires, for example, an IMSI (International Mobile Subscriber Identity) from each terminal 2 under the control of the base station 1, and for each terminal 2 identified by the IMSI, the determined "terminal communication information , and the subsequent processing is preferably performed separately for each terminal 2 concerned.

<Target communication information determination processing>
Similarly, in the functional block diagram of FIG. The "target communication information", which is information related to the communication state to be embodied with the target of the communication, is determined.

(When using received signal strength data as "target communication information")
The "target communication information" determined by the target communication information determination unit 112 is
(α') information on the intensity of the radio wave that will be received from the detected object if the object includes the terminal 2 (received radio wave intensity data)
can be In this case, the target communication information determination unit 112, based on the detection position of the target detected from the image data (by the target detection information determination unit 113) and the detection position of the "other target" (which may cause an obstacle), As shown on the right side of FIG. 3 used for explanation later,
(a) determining a straight line segment connecting the detected object and (the antenna of) the base station 1;
(b) calculating shielding degree information relating to the degree to which the "other target" shields a predetermined area including the straight line segment;
(c) From the calculated degree of shielding information, as "target communication information", if the detected target includes the terminal 2, the reception radio wave intensity data that will be received from this target is calculated.

Specifically, the shielding degree information (b) above is calculated as follows. First, as shown on the right side of FIG. 3 (to be used for explanation later), a detection position having a predetermined size (predetermined area) according to the type (class) of "other objects" that can cause obstacles is detected. cross-sectional area at) is set in advance, and further, after setting a rotating body area of a predetermined size including the above-mentioned straight line segment as a rotating axis, at a certain time point t (= t0-M, t0-M+1,・・・, t0－1, t0),
(b1) Part of the cross-sectional area of any of the "other targets" at the detection position is not included in the above-mentioned rotating body area (the cross-sectional area of the other target and the rotating body area overlap not have) takes the value of 1, whereas
(b2) In any of the "other targets", a part of its cross-sectional area at the detection position is included in the above-mentioned rotating body area (the cross-sectional area of the other target and the rotating body area overlap (1) R = (S _a ∩ S _b ^- )/S _a
The time-series data of the variable E′_t, which takes the R value calculated with , can be used as the shielding degree information.

Here, in the above formula (1), S _a is a set representing a cross-sectional area (perpendicular to the rotation axis) at the detection position of the "other object" in the above-mentioned rotating body area (for example, a unit area forming the area ), and S _b is a set representing the cross-sectional area at the detection position of the “other object” (for example, a set of unit areas forming the area). In addition, S _b ⁻ is the complement of S _b , and “/” indicates the area value (eg, the number of unit regions) of the cross-sectional region related to the set of molecules, the area value of the cross-sectional region related to the denominator (eg, the number of unit regions) It is an operator that indicates division by a number).

Next, the target communication information determining unit 112 sets the time-series data of this variable E'_t (or the variable E'_t multiplied by a predetermined constant) as "target communication information" as received radio wave intensity data. of. Hereinafter, the variable E'_t will be referred to as a variable equivalent to received power in order to distinguish it from the above-described received signal power variable E_t.

If there are two or more “other objects” that take the above R value, S _b is the area formed by the entire cross-sectional area of these “other objects”, that is, the sum of these cross-sectional areas It may be a region represented as a set. Further, the body of revolution region may be a cylindrical region, or may be formed by rotating a predetermined curve around an axis of rotation, such as an ellipsoid.

Furthermore, as a modified aspect, the target communication information determination unit 112 uses a learned "target radio wave information estimation model" that outputs information related to the strength of the received radio wave according to the input image data, and uses the acquired image From the data, information on the strength of the radio wave that would be received from the detected target including the terminal 2 is derived, and the information on the strength of the radio wave is determined as "target communication information." may

Here, as the "target radio wave information estimation model", a known trained model as disclosed in Non-Patent Document 1 can be adopted. Specifically, the image data at a certain time t (= t0-M, t0-M+1, ..., t0-1, t0) is input to this trained model, and the estimated value of the received signal power is output. , this estimated value is used as the received power equivalent variable E′_t value, and the time-series data of this received power equivalent variable E′_t is used as the “target communication information”.

(When communication connection availability data is used as "target communication information")
Also, the "target communication information" determined by the target communication information determination unit 112 is
(β′) Information regarding availability of communication connection with the detected target (communication connection availability data)
It is also possible to In this case, the target communication information determining unit 112, based on the detected position of the target detected from the acquired image data and the detected position of the "other target" that may cause a failure,
(a) determining a straight line segment connecting the detected object and (the antenna of) the base station 1;
(b) calculating communication connection availability data indicating whether or not the "other object" is positioned on this straight line segment or within a predetermined area including this straight line segment, or whether or not this straight line segment is blocked;
(c) It is also preferable to determine the calculated communication connectability data as the "target communication information".

More specifically, as the communication connection propriety data in (b) above, the detection position of one of the "other targets" is at a certain point in time t (== t0-M, t0-M+1, ..., t0- 1, t0) is -1 if it is located on the above straight line segment or within a rotating body region of a predetermined size that includes this straight line segment as the rotation axis, and takes a value of 1 otherwise. Binary communication connection availability variable:
(2) B'_t∈{－1, 1}
may be generated and used as the "target communication information". Here, the body of revolution region may be a cylindrical region, or may be formed by rotating a predetermined curve around a rotation axis, such as an ellipsoid.

As a modification mode, the target communication information determination unit 112 presets a predetermined size (cross-sectional area at the detection position) according to the type (class) of the "other target". When the "other target" blocks the above straight line segment with its size (cross-sectional area) at the detection position (the above straight line segment penetrates the cross section of the "other target") B′_t in (2) above may be defined as a communication connection enable/disable variable that takes a value of −1, otherwise 1.

As described above, the "target communication information" is
(α') Time-series data of received power equivalent variable E'_t (received radio wave intensity data), or (β') Time-series data of communication connection availability variable B'_t (communication connection availability data)
I explained the case where . Here, the "target communication information" preferably includes both (α') and (β') when the "terminal communication information" includes both (α) and (β). Furthermore, considering the accuracy of the degree of matching (degree of communication compatibility) with the "terminal communication information" calculated later, the "target communication information" is the time-series data of the above (α') received power equivalent variable E'_t preferably included.

Incidentally, in this embodiment, the target communication information determination unit 112 stores the determined "target communication information" for each predetermined detected target for a past predetermined period. It can be provided to the unit 115 as appropriate.

<Occlusion determination processing>
Similarly in the functional block diagram of FIG. It is determined whether or not there is (or is predicted to occur), and the result of this determination is output to the tracking target determining/managing unit 115, which will be described later in detail.

FIG. 2 is a schematic diagram for explaining two modes of occlusion determination processing in the occlusion determination unit 114. FIG.

First, according to the determination mode shown in FIG. 2(A), the occlusion determination unit 114 receives from the target detection information determination unit 113 the tracked terminal at a time point (for example, time point t0-1) past time point t0. The detected position and detected speed (target position information) of the target (for example, "person") related to 2 are acquired, and from these information, the predicted position of this target ("person") at time t0 is determined.

Next, the occlusion determination unit 114 detects another predetermined object at time t0 within a predetermined range including the determined predicted position (for example, an image region corresponding to a real-space circular range with a predetermined size centered on the predicted position). When a person (for example, "person") is detected, it can be determined that occlusion is occurring at this time t0.

Next, in the determination mode shown in FIG. 2(B), the occlusion determination unit 114 first, in the same manner as in the determination mode of FIG. Determine the predicted position of the person") at time t0.

Next, the occlusion determination unit 114 includes the target at time t0 within a predetermined range including the determined predicted position (for example, an image region corresponding to a real-space circular range with a predetermined size centered on the predicted position). If no object is detected, it can be determined that occlusion is occurring at this time t0. It should be noted that in this case there is a high probability that an occlusion has occurred such that the object being tracked is completely occluded.

In addition, as yet another mode of occlusion determination, the occlusion determination unit 114 uses the object detection information determination unit 113 to obtain target position information and If one or both of the target appearance information could not be determined, it is possible to receive a report to that effect and determine that occlusion is occurring at this time t0.

Furthermore, the occlusion determination unit 114 uses the object detection information determination unit 113 to determine the object being tracked (for example, a “person”) and another predetermined When an object (for example, a "person") is detected, if the distance between the detection positions (for example, the distance in the image space corresponding to the predetermined distance in the real space) is less than a predetermined threshold, this It is also possible to determine that occlusion is occurring at time t0. In this case, there is a high possibility that partial occlusion occurs to the extent that the detection position can be determined.

<Tracking target determination processing/terminal tracking processing>
Returning to the functional block diagram of FIG. 1, the tracking target determination/management unit 115 of the present embodiment determines that no occlusion has occurred in a detected target (for example, a “person”) related to tracking, and that target position information and target appearance are detected. In the time interval over which the information is determined, at least
(A) "Position matching degree", which is the degree of matching between the determined target position information and the predicted target position information predicted from the past target position information, and (B) the determined target appearance information and the past target "Appearance matching degree" that is the degree of matching with appearance information
(both in this embodiment), it is determined whether or not there is a corresponding relationship between the terminal 2 being tracked and the detected object ("person"), and if there is a corresponding relationship The determined target (“person”) is determined as the target related to the terminal 2 being tracked, and the terminal tracking process is performed.

Here, in this embodiment, as will be described later, although the weight (λ) is adjusted,
(c) "Communication correspondence degree", which is the degree of correspondence between the terminal communication information determined for the terminal 2 being tracked and the target communication information determined for the detected target ("person").
Correspondence determination processing may also be performed based on

In addition, in this embodiment, the tracking target determination/management unit 115 always uses the above (c) "communication correspondence degree" when occlusion in which the detection position is not determined for the detected target related to tracking occurs and is resolved. For example, based only on this "degree of communication compatibility", correspondence determination processing (tracking target determination processing) between the terminal 2 and the target ("person") is performed.

This is because the calculation of the above (a) "position matching degree" and the above (b) "appearance matching degree" at the time when occlusion where the detection position is not determined occurs and is resolved, is based on the time interval in which this occlusion occurred. Where "(past) target position information" and "(past) target appearance information" are required, these information cannot be obtained precisely because such occlusion occurred. by. That is, in this case, neither the "degree of position matching" in (a) nor the "degree of appearance matching" in (b) above can be calculated.

Hereinafter, specific correspondence determination processing (tracking target determination processing) using the above-described "position matching degree", "appearance matching degree", and "communication compatibility degree" will be described.

Similarly, in the functional block diagram of FIG. 1, the communication cost calculation unit 115a, the location cost calculation unit 115b, and the appearance cost calculation unit 115c of the tracking target determination/management unit 115 each perform the above-described position matching for the detected target. After calculating the degree, the degree of appearance matching, and the degree of communication compatibility,
・"Location cost" Cgeo, which is a monotonically decreasing function of the degree of location matching,
・“Appearance cost” Capp, which is a monotonically decreasing function of the degree of appearance matching; and ・“Communication cost,” Cradio, which is a monotonically decreasing function of the degree of communication compatibility.
Calculate

Each of these costs is a value such that the smaller the cost, the higher the possibility that the terminal 2 and the detected target are in a corresponding relationship. Here, as the monotonically decreasing function of each degree, for example, the sign of the degree can be reversed (the degree is multiplied by -1), or the reciprocal of the degree can be used. . Note that the location cost Cgeo and appearance cost Capp may be collectively referred to as "environmental information cost" hereinafter.

Next, the tracked object determination/management unit 115 of this embodiment will be described in detail later.
(a) A weighted sum of the calculated location cost Cgeo, appearance cost Capp, and communication cost Cradio for each of a plurality of detected objects (for example, "people") by a "weighting factor λ" that can take a zero value. and calculating the tracking cost Cost, which is a weighted sum of the communication cost Cradio and the weighting factor λ,
(b) The target (“person”) whose tracking cost is small enough to satisfy a predetermined condition, for example, the lowest tracking cost, is determined as the target related to the terminal 2 being tracked.

A specific example of calculating the communication cost Cradio, the location cost Cgeo, and the appearance cost Capp, which are necessary for calculating the tracking cost Cost, will be described below.

(Communication cost calculation)
FIG. 3 is a schematic diagram for explaining a specific example of communication cost calculation processing according to the present invention.

In the specific example of calculating the communication cost Cradio shown in FIG. 3, the communication cost calculation unit 115a first
(a) time-series data of the received signal power variable E_t of the tracking terminal 2 (as terminal communication information) determined by the terminal communication information determination unit 111;
(b) The matching degree Cr (degree of communication compatibility) with the time-series data of the received power equivalent variable E′_t related to the detected target (as the target communication information) determined by the target communication information determination unit 112 is set to a predetermined It is calculated as the cross-correlation (Σ _t E_t*E'_t) of both in the time interval.

Incidentally, when time-series data of the communication connection presence/absence variable B_t and the communication connection availability variable B'_t are adopted as the terminal communication information and the target communication information, respectively, the matching degree Cr (communication correspondence degree) It will be calculated as the cross-correlation (Σ _t B_t*B'_t). Also, the weighted sum of the cross-correlation (Σ _t E_t*E'_t) and the cross-correlation (Σ _t B_t*B'_t) can be used as the degree of coincidence Cr.

In this specific example, the communication cost calculation unit 115a determines that no occlusion has occurred such that the detection position is not determined (the detection position (target position information) of the target detected by the target detection information determination unit 113 is stored). The degree of coincidence Cr at time t0 is calculated from the received signal power variable E_t and the variable As the cross-correlation with E'_t, the following formula (3) Cr＝C0・Σ _t=t0-M ^t0-N E_t*E'_t
Calculated using Here, in the above equation (3), Σ _t=t0-N ^t0-N is an operator indicating the summation for time t (= t0-M, t0-M+1, ..., t0-N) is. Also, C0 is a coefficient for adjusting the Cr value to be 1 upon perfect matching. Furthermore, M is an integer set such that the time t0-M is sufficiently far in the past (for example, 10 seconds ago) for cross-correlation.

N is the latest (closest to time t0) time point (closest to time t0) at which the detection position (target position information) of the detected target (in the target detection information determination unit 113) is stored when time t0-N is viewed from time t0. An integer greater than or equal to 1 such that Therefore, when N=1 and the detection position of the object detected at time t0-1 immediately before time t0 is stored in the target detection information determination unit 113, that is, when the occlusion whose detection position is not determined is the latest If it does not occur at time t0, the degree of coincidence Cr at time t0 is given by the following formula (3') Cr=C0・Σ _t=t0-M ^t0-1 E_t*E'_t
will be calculated using

On the other hand, when N>1, the detection position of the detected object in the time interval from time (t0-N+1) to time (t0-1) is determined by the occurrence of occlusion that cannot be detected. As a result, the received power equivalent variable E′_t is not calculated in the time interval where such occlusion occurred. Therefore, in this case, the degree of coincidence Cr represented by the above equation (3) cannot be calculated as it is.

Therefore, the position interpolation unit 112a (FIG. 1) of the target communication information determination unit 112 determines and stores the detection position in the time interval from time (t0-N+1) to time (t0-1) (equivalent to received power). variable E'_t is not calculated), the detected occlusion at each time point (time t0-N+1, time t0-N+2, ..., time t0-1) of the time interval where such occlusion occurred The position of the target is determined by an interpolation process. Specifically, the detected position at the acquired time (t0-N) (or the detected position before the acquired time t0-N) and the detected position at the acquired time t0 are interpolated by a known interpolation method, For example, the positions at times t0-N+1, t0-N+2, .

Next, the target communication information determining unit 112 (FIG. 1) places the detected target exists, the received power equivalent variable E'_t (target communication information) at each time is determined. As a result, the communication cost calculation unit 115a can calculate the degree of coincidence Cr using the above equation (3) even at time t0 when the occlusion that has occurred is resolved.

Furthermore, in this specific example, the communication cost calculation unit 115a inverts the sign of the degree of coincidence Cr calculated as described above (multiplies the degree of coincidence Cr by -1), and the sign inversion result (-Cr) to the communication cost Cradio. Therefore, the higher the communication cost Cradio, the higher the possibility that the detected object does not correspond to the terminal 2 being tracked.

Incidentally, in FIG. 3, the communication cost Cradio for target A is smaller than the communication cost Cradio for target B because the waveform of the received power equivalent variable E'_t is more similar to the waveform of the received signal power variable E_t. Therefore, looking at only the communication cost Cradio, it can be said that there is a higher possibility that the target A corresponds to the terminal 2 being tracked.

(position cost calculation, appearance cost calculation)
FIG. 4 is a schematic diagram for explaining a specific example of position cost and appearance cost calculation processing according to the present invention.

First, in the specific example of position cost Cgeo calculation shown in FIG. 4A, the position cost calculation unit 115b
(a) the determined detection image area range (as the target position information) (for example, the coordinate values of the four corners);
(b) Calculate the degree of matching (degree of position matching) Cg with the range of the detection image area (as prediction target position information) predicted from the past detection position (as target position information).

More specifically, the position cost calculation unit 115b determines that no occlusion that does not determine the detection position has occurred (when the target detection position (target position information) detected by the target detection information determination unit 113 is stored). ) at time t0,
(a) the extent S1 of the detected image area for the detected object at this time t0;
(b) The latest (closest to time t0) time (t0-N) at which the detection position (target position information) of the detected target (in the target detection information determination unit 113) is stored from this time t0 Let S1∩S2 be the area of the superimposed portion of the predicted detection image region range S2 at time t0, which is predicted using the detection speed at time (t0-N) from the detection position at time t0, and (a) the detection image Assuming that the area occupied by the range S1 of the region and the range S2 of the predicted detection image region (b) is S1∪S2, the following formula (4) Cg = (S1∩S2)/(S1∪S2)
Then, the matching degree (position matching degree) Cg at time t0 is calculated.

As a modification, the position cost calculation unit 115b predicts (using the detected speed at time t0-N) from the detected position of the detected target at time t0 and the detected position at the latest time (t0-N). It is also possible to set the degree of closeness to the predicted position, for example, the reciprocal of the distance between the two positions, as the degree of matching (degree of position matching) Cg at time t0.

In any case, according to the calculation process of the degree of coincidence Cg as described above, even if there is a point in time when occlusion occurs at which the detection position is not determined, that is, when N>1, The degree of matching Cg can be calculated.

Here, in this embodiment, the position cost calculation unit 115b inverts the sign of the degree of matching Cg calculated in this way (multiplies the degree of matching Cg by -1), and converts the result of this sign inversion (-Cg) to the position cost. Decide on Cgeo. Therefore, the higher this location cost Cgeo, the higher the probability that the detected object does not correspond to the terminal 2 being tracked.

Next, in the specific example of appearance cost Capp calculation shown in FIG. 4B, the appearance cost calculation unit 115c
(a) the image feature amount of the determined detection image area (as target appearance information);
(b) Calculate the matching degree (appearance matching degree) Ca with the image feature amount of the detected image area in the past (as the target appearance information).

More specifically, the appearance cost calculation unit 115c determines that no occlusion has occurred for which the detection position cannot be determined (the target appearance information of the target detected by the target detection information determination unit 113 is stored). At time t0,
(a) an image feature derived using a known CNN image feature extractor for the detected image region of the detected object at this time t;
(b) The detected object at the latest (closest to the time t0) time t0-N at which the object appearance information of the detected object (in the object detection information determination unit 113) is stored from this time t0 For the detected image region, the similarity with the image feature amount derived using the same CNN image feature extractor as in (a) above is calculated by a known similarity calculation method between feature amounts, for example, described in Non-Patent Document 2 The degree of similarity calculated using the method described above is used as the degree of coincidence Ca at time t0.

According to such processing for calculating the degree of coincidence Ca, even when there is a point in time when occlusion occurs at which the detection position is not determined, that is, when N>1, the degree of coincidence Ca at time t0 can be calculated. You can.

Here, in this embodiment, the appearance cost calculation unit 115c inverts the sign of the degree of matching Ca calculated as described above (multiplies the degree of matching Ca by -1), and the result of this sign inversion (-Ca) to be the appearance cost Capp. Therefore, the higher the appearance cost Ca, the higher the possibility that the detected object does not correspond to the terminal 2 being tracked.

The processing for calculating the communication cost Cradio, the location cost Cgeo, and the appearance cost Capp has been described above. The determination process (tracking target determination process) will be described.

(Tracking cost calculation, tracking target determination processing)
FIG. 5 is a schematic diagram for explaining a specific example of tracking cost calculation processing and tracking target determination processing according to the present invention.

As shown in FIG. 5, the tracking target determination/management unit 115 in this embodiment
(a) For each of a plurality of detected targets (target A and target B in FIG. 5), the calculated positional cost Cgeo, appearance cost Capp, and communication cost Cradio are calculated according to the "weighting factor λ" that can take a zero value. Calculating the tracking cost Cost, which is a weighted sum and is a weighted sum related to the communication cost Cradio and the "weighting factor λ",
(b) The target (“person”) whose tracking cost is small enough to satisfy a predetermined condition, for example, the lowest tracking cost, is determined as the target related to the terminal 2 being tracked.

Here, in this embodiment, such a tracking cost Cost is expressed by the following formula (5) Cost=(1−λ)·(α·Cgeo+β·Capp)+λ·Cradio
can be calculated by In the above equation (5), the weighting factor λ takes a value of 0 or more and 1 or less. Also, α (≧0) and β (≧0) satisfy α+β=1 and are weighting coefficients that determine the relative weight of the location cost Cgeo and the appearance cost Capp. may be

As a modification, the weighting factor (1-λ) of the first term (environmental information cost) on the right side of the above equation (5) can be set to another weighting factor, for example, a constant. Furthermore, it is also possible to use only one of the location cost Cgeo and the appearance cost Capp in the above equation (5), although there is an accuracy problem. For example, it is possible to use only the location cost Cgeo as the environmental information cost. Incidentally, when the tracking cost Cost described above is greater than the predetermined threshold value C_th, the correspondence determination process may be set not to be performed.

Here, correspondence determination processing (tracking target determination processing) using the tracking cost Cost of the above equation (5) when occlusion occurs in the detected target will be described. When such occlusion is resolved, the weight determination unit 115d of the tracking target determination/management unit 115 in this embodiment determines the maximum value (maxCr) of the degree of matching Cr (degree of communication compatibility) calculated for each detected target. Determine a weighting factor λ that is a monotonically increasing function of . That is, the value of the weighting factor λ preset as such a monotonically increasing function is determined by substituting the maximum value (maxCr) of the calculated degree of coincidence Cr. For example, when the following formula (6) maxCr>K, λ = λ ₁
λ = λ ₀ (λ ₀ < λ ₁ ) when maxCr≤K
The weighting factor λ may be determined by

Here, K is a predetermined threshold, and defines a boundary as to whether or not the tracking cost Cost reflects the communication cost Cradio to a greater extent. For example, in the above formula (6), if λ ₁ = 1 and λ ₀ = 0, the tracking cost Cost is It becomes the communication cost Cradio itself, or the environmental information cost (α · Cgeo + β · Capp) itself.

As another aspect, the weight determining unit 115d uses the following equation (7) λ=1/(1+e ^{−a(maxCr+k)} ) using a sigmoid function.
The weighting factor λ may be determined by where a is the gain of the sigmoid function. Also, k is an offset value with respect to the maximum value (maxCr) of the degree of coincidence Cr, and is set in advance. Incidentally, when using the above formula (7), the communication cost calculation unit 115a preferably uses a cross-correlation (eg, Σ _t E_t*E′_t) value less than 0 rounded up to 0 as the degree of coincidence Cr.

Here, the matching degree Cr (communication correspondence degree) is determined by the communication state of the terminal 2 and the environment information ( For example, it can be determined from the estimated communication state of the object (object A or object B) to be determined obtained from the image data). That is, there is a high possibility that the degree of coincidence Cr (degree of communication correspondence) can be used as a criterion with higher reliability even when occlusion occurs. As a result, as described above, in this embodiment, by determining the weighting factor λ, which is a monotonically increasing function of the maximum value (maxCr) of the degree of matching Cr (degree of communication compatibility), even when occlusion occurs, It is possible to calculate/determine a more appropriate (conforming to the actual situation) tracking cost. It also makes it possible to improve the accuracy of terminal tracking.

In this respect, of course, despite the fact that the degree of matching Cr (degree of communication compatibility) is considerably low, if the tracking target determination processing that emphasizes the communication cost Cradio or considers only the communication cost Cradio is performed, the terminal tracking accuracy There is concern about a decline in On the other hand, the tracking target determining/managing unit 115 of the present embodiment can determine a smaller weighting factor λ in such a situation to avoid or minimize deterioration of terminal tracking accuracy. On the other hand, when the degree of coincidence Cr (degree of communication compatibility) is higher, conversely, when occlusion occurs by determining a larger weighting factor λ and executing tracking target determination processing with a greater degree based on the communication cost Cradio. Even so, it is possible to maintain and improve the accuracy of terminal tracking.

Here, the weighting factor λ, which is a monotonically increasing function of the maximum value (maxCr) of the degree of matching Cr (degree of communication correspondence) in each detected object, such as the above equations (6) and (7), is , may be set to be employed for a predetermined period immediately after the occlusion is resolved. In this case, outside of this predetermined period, the weighting factor λ can be set to a predetermined value (a small value such as 0.1) or zero. For example, if the weighting factor λ is set to zero, the correspondence relationship determination process (tracking target determination process) is performed based on the environmental information cost (Cgeo, Capp) that is judged to be highly reliable in a situation where occlusion does not occur. It is done. It should be noted that it is not limited to when the occlusion is resolved, and it is also possible to always set the weighting factor λ as in the above equations (6) and (7).

Next, an embodiment in which the degree of matching Cg (degree of positional matching) is also considered in determining the weighting factor λ will be described. In this case, when the occlusion is resolved, the weight determination unit 115d in the present embodiment also determines the maximum value (maxCr) of the matching degree Cr (communication correspondence degree) of each detected target. Also for the maximum value (maxCg) of (degree of position matching), a weighting factor λ that is a monotonically increasing function (if one is a constant, the other is a monotonically increasing function) is determined. for example,
(a) If the maximum value (maxCg) of the matching degree Cg (positional matching degree) of each detected object is larger than the predetermined threshold Kg (maxCg>Kg), the above formula (6) or the above formula (7) is followed. ,on the other hand,
(b) The maximum value (maxCg) of the degree of matching Cg (degree of position matching) of each detected target is equal to or less than a predetermined threshold value Kg (maxCg≦Kg) and the degree of matching Cr (degree of communication compatibility) of each detected target ) is also less than or equal to a predetermined threshold value K'(maxCr≤K'(>K)), it takes a zero value or a small value less than a predetermined value (here, even if maxCg≤Kg, maxCr>K ' follow the above formula (6) or above formula (7))
A weighting factor λ can be determined. As another aspect, according to the above equation (7), both the maximum value (maxCr) of the degree of coincidence Cr and the maximum value (maxCg) of the degree of coincidence Cg are sigmoid functions (if one is a constant, It is also possible to determine the weighting factor λ, which is the other sigmoidal function.

By determining such a weighting factor λ, the ratio of the communication cost Cradio to the tracking cost Cost can be made zero or reduced in a situation where the matching degree Cg (position matching degree) is considerably low. Here, in such a situation, even if the degree of matching Cr (degree of communication compatibility) is high, since the target detection position is used to calculate the degree of matching Cr in the first place, It is judged that the reliability of communication cost Cradio is declining. Therefore, by determining the weighting factor λ as described above and setting the ratio of the communication cost Cradio in the tracking cost to zero or reducing it, it is possible to maintain and improve the accuracy of terminal tracking.

To summarize the correspondence relationship determination process described above, the tracking target determination/management unit 115 performs the tracking target determination/management unit 115 when the degree of matching Cr (degree of communication correspondence) is small enough to satisfy a predetermined condition (in this embodiment, each detected target). is less than or equal to a predetermined threshold), when the matching degree Cg (position matching degree) is small enough to satisfy a predetermined condition (in this embodiment, the matching degree of each detected object When the maximum value maxCg of Cg is equal to or less than a predetermined threshold), the correspondence relationship determination process is performed regardless of the degree of matching Cr (degree of communication correspondence), or the smaller the degree of matching Cg (degree of position matching), the more In the embodiment, the smaller the maximum value maxCg of the degree of matching Cg of each detected target, the smaller the degree based on the degree of matching Cr (degree of communication correspondence), and the correspondence determination process is to be executed. .

Also, as a preferred modification, the tracking target determination/management unit 115
(a) When the degree of matching Cg (degree of position matching) exceeds a predetermined condition and is small (for example, when the maximum value maxCg of the degree of matching Cg of each detected object is equal to or less than a predetermined threshold), at least the degree of matching Cg ( Position matching degree) and matching degree Ca (appearance matching degree) are used to determine the correspondence relationship. ), it is also preferable to increase the value of the weighting factor β (that is, to decrease the value of the weighting factor α (=β-1)), or
(b) Correspondence determination processing is performed based on at least the matching degree Cg (positional matching degree) and the matching degree Ca (appearance matching degree), and the smaller the matching degree Cg (positional matching degree) (for example, the It is also preferable to increase the degree based on the degree of matching Ca (degree of appearance matching), specifically, to increase the value of the weighting factor β as the maximum value maxCg of the degree of matching Cg decreases.

It should be noted that such adjustment of the weighting factor β based on the matching degree Cg (positional matching degree) includes at least the matching degree Cg (positional matching degree) and the matching degree Ca It can be implemented when the correspondence relationship determination process is performed based on the (degree of matching in appearance).

In any case, in a situation where the matching degree Cg (position matching degree) is considerably low, not only the communication cost Cradio obtained from the matching degree Cr (communication correspondence degree) but also the location cost Cgeo obtained from this matching degree Cg It is judged that the reliability of the value itself is also declining. Therefore, as described above, by increasing the value of the weighting factor β and decreasing the relative weight of the location cost Cgeo (increasing the relative weight of the appearance cost Capp obtained from the degree of matching Ca), the accuracy of terminal tracking can be maintained and improved. It is possible.

Similarly, according to FIG. 5, the tracking target determining/managing unit 115 determines the location costs Cgeo_A and Cgeo_B, the appearance costs Capp_A and Capp_B, and the communication Costs Cradio_A and Cradio_B are calculated, and the tracking costs Cost_A and Cost_B at time t0 are calculated by the following formula (8) Cost_A = (1-λ) (α · Cgeo_A + β · Capp_A) + λ · Cradio_A
(9) Cost_B=(1−λ)・(α・Cgeo_B＋β・Capp_B)＋λ・Cradio_B
is determined using Here, the weighting factor λ and the weighting factors α and β are determined by the determination process described above.

Next, in this embodiment, the tracking target determining/managing unit 115 tracks target A ("person") with the lowest tracking cost (in the case of FIG. 5, the target with the lowest tracking cost). A target related to the terminal 2 is determined, and the tracking of the terminal 2 is continued. Incidentally, in FIG. 5, although there is no difference in the appearance cost Capp between the two, both the location cost Cgeo and the communication cost Cradio are clearly smaller for object A ("person").

Here, as described above, the weighting factor λ and the weighting factors α and β are used even when occlusion occurs with respect to the detected object for which the correspondence relationship with the terminal 2 should be determined. degree), it is set to a more suitable value. As a result, even if occlusion occurs in the tracked terminal 2, the tracking of the terminal 2 is continued more appropriately (in FIG. to be tracked).

Returning to the functional block diagram of FIG. 1, the tracking target determination/management unit 115 performs the tracking target determination processing (correspondence relationship The result of determination processing) is stored and managed in the tracking information storage unit 105, and is output to the communication control unit 116 at all times, periodically, or as appropriate upon request. Here, the result of this tracking object determination process (correspondence determination process) can be time-series data in which the location and image features at each point in time are associated with each terminal ID of the terminal 2, for example. adverse effects (for example, lost, ID switch, etc.) are avoided or reduced.

The communication control unit 116 uses the result of the tracked object determination process (correspondence determination process) of such a suitable terminal 2 to more accurately determine the current position of the terminal 2 connected to communication and the situation including the communication route. can grasp. Further, as a result, even if a situation occurs in which the received power from the terminal 2 connected to this communication connection suddenly drops, the communication connection with this terminal 2 can be continued or immediately restarted. be.

As described in detail above, according to the present invention, when an occlusion occurs, the process of determining whether or not the terminal being tracked and the detected object have a corresponding relationship is performed by detecting the terminal. It can be implemented based on the degree of communication correspondence with the target. As a result, even if an occlusion occurs for the terminal being tracked, it is possible to continue tracking this terminal more appropriately.

In addition, such terminal tracking processing according to the present invention can be applied to various fields and situations related to 5G, which is currently being introduced, and future mobile communication systems using radio waves with shorter wavelengths. . For example, it can solve the problem of communication failure due to obstructions in the communication path, notify or warn a car equipped with a terminal of the approach of a person or other vehicle, or determine that there is a corresponding relationship with a certain terminal. It is also possible to determine the relationship between the browsed page of the user and the line of flow of the user, and utilize it for marketing and management.

In addition, mobile and mobile communication services that strengthen people's "connections" by popularizing information communication systems including the terminal tracking device according to the present invention, which can stably realize a huge number of large-capacity communication connections. It will also be possible to widely deploy the system and aim for the empowerment of the community. Moreover, it is believed that the information communication system including the terminal tracking device according to the present invention will exert its power particularly in urban areas where the population is increasing worldwide. That is, according to the present invention, the United Nations-led Sustainable Development Goals (SDGs) Goal 9 "Develop resilient infrastructure, promote sustainable industrialization, and promote innovation" and Goal 11 It will also be possible to contribute to the achievement of “making cities inclusive, safe, resilient and sustainable.”

In the various embodiments of the present invention described above, various changes, modifications and omissions within the scope of the technical ideas and aspects of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to limit the present invention in any way. The invention is to be limited only as limited by the claims and the equivalents thereof.

1 base station (terminal tracking device, communication relay device)
101 communication interface 102 communication history information accumulation unit 103 camera (sensor)
104 Environment information accumulation unit 105 Tracking information storage unit 111 Terminal communication information determination unit 112 Target communication information determination unit 112a Position interpolation unit 113 Target detection information determination unit 114 Occlusion determination unit 115 Tracked target determination/management unit 115a Communication cost calculation unit 115b Position Cost calculation unit 115c Appearance cost calculation unit 115d Weight determination unit 116 Communication control unit 2 Terminal

Claims (11)

Target detection that detects a predetermined target that may include the terminal from environmental information generated by a sensor and is related to the environment in which the terminal may exist, and determines target position information that is information related to the detection position of the detected target. an information determining means;
terminal communication information determination means for determining terminal communication information, which is information regarding a communication state with the terminal, based on information regarding communication acquired from a communication means that communicates with the terminal;
a target communication information determining means for determining, from the environment information, target communication information, which is information relating to a communication state to be realized with a detected target if the target includes a terminal;
The degree of position matching, which is the degree of matching between the target position information and the predicted target position information predicted from the target position information determined for the tracking terminal, and the tracking terminal Determining whether or not the tracked terminal and the detected target are in correspondence based on at least one of the degree of correspondence between the terminal communication information and the target communication information. a tracked object determining means for making a determination and determining the object determined to have a corresponding relationship as an object related to the terminal being tracked;
The tracked object determining means, at least when occlusion is resolved for the detected object, when the degree of communication compatibility is large enough to satisfy a predetermined condition, the tracking object determination means is based on the degree of communication compatibility, not based on the degree of location matching. A terminal tracking device, characterized in that it makes a judgment, or makes the judgment by increasing the degree based on the degree of communication compatibility as the degree of communication compatibility increases or when it is large until a predetermined condition is satisfied. When the degree of communication compatibility is small enough to satisfy a predetermined condition, the tracked object determining means makes the determination without being based on the degree of communication compatibility when the degree of position matching is small enough to satisfy the predetermined condition. Alternatively, the terminal tracking device according to claim 1, wherein the smaller the degree of position matching, the smaller the degree based on the degree of communication compatibility, and performing the determination. The target detection information determining means also determines target appearance information, which is information related to the detected appearance of the detected target, from the environment information,
When the tracked object determination means makes the determination based on at least the degree of position matching, the object appearance information matches the past object appearance information of the object determined to be related to the tracked terminal. 3. The terminal tracking device according to claim 1 or 2, wherein said determination is made also based on the degree of appearance matching, which is a degree. The tracked object determining means is
When the degree of position matching exceeds a predetermined condition and is small, the determination is made based on at least the degree of appearance matching and the degree of position matching, and the degree based on the degree of appearance matching is increased at that time, or
4. The terminal tracking device according to claim 3, wherein the determination is made based on at least the degree of appearance matching and the degree of position matching, and the smaller the degree of position matching, the greater the degree based on the degree of appearance matching. . The target communication information determination means assumes that the target exists at a position interpolated using the detection positions determined before and after the occlusion in a time interval during which occlusion occurs for which the detection position of the target is not determined, 5. The terminal tracking device according to claim 1, wherein said target communication information is determined. the environment information is information relating to an image of the environment, and the target position information includes the position and range of the image area of the detected target within the image;
The degree of position matching is the degree of overlap between the range of the detected object and the expected range in the image of the determined object associated with the tracking terminal, or 6. The degree of closeness between the position of the object obtained and the predicted position in the image of the object determined for the tracking terminal. or the terminal tracking device according to item 1. The tracked object determining means is
Determining a weighting factor that can take a zero value, which is a monotonically increasing function of the degree of communication compatibility,
A weighted sum of an environmental information cost including a location cost that is a monotonically decreasing function of the degree of location matching and a communication cost that is a monotonically decreasing function of the degree of communication compatibility, wherein the weighting factor is applied to the communication cost, A tracking cost of the target is calculated, which is a weighted sum in which another weighting factor that can take a zero value is applied to the environmental cost, and the target whose tracking cost is small until it satisfies a predetermined condition is tracked. 7. The terminal tracking device according to any one of claims 1 to 6, wherein the terminal tracking device determines a target related to the terminal. 8. The terminal tracking device according to claim 7, wherein said tracking target determining means determines said weighting factor which is a monotonically increasing function for both said degree of communication compatibility and said degree of position matching. Detecting a predetermined target that may include the terminal from environmental information generated by a sensor and relating to the environment in which the terminal may exist, determining target position information that is information relating to the detection position of the detected target, and , object detection information determining means for also determining object appearance information, which is information relating to the detected appearance of the detected object, from the environment information;
terminal communication information determination means for determining terminal communication information, which is information regarding a communication state with the terminal, based on information regarding communication acquired from a communication means that communicates with the terminal;
a target communication information determining means for determining, from the environment information, target communication information, which is information relating to a communication state to be realized with a detected target if the target includes a terminal;
The degree of position matching, which is the degree of matching between the target position information and the predicted target position information predicted from the target position information determined for the terminal being tracked, the target appearance information and the tracked target The degree of appearance matching, which is the degree of matching with the past target appearance information of the target determined to be related to the terminal being tracked, and the degree of correspondence between the terminal communication information of the terminal being tracked and the target communication information Based on at least one of the degree of communication compatibility, it is determined whether or not the tracked terminal and the detected target have a corresponding relationship, and the target determined to have a corresponding relationship as a target related to the terminal being tracked, and
The tracked object determining means, at least when occlusion is resolved for the detected object, when the degree of communication compatibility is large enough to satisfy a predetermined condition, the tracking object determination means is based on the degree of communication compatibility, not based on the degree of location matching. A terminal tracking device, characterized in that it makes a judgment, or makes the judgment by increasing the degree based on the degree of communication compatibility as the degree of communication compatibility increases or when it is large until a predetermined condition is satisfied. Target detection that detects a predetermined target that may include the terminal from environmental information generated by a sensor and is related to the environment in which the terminal may exist, and determines target position information that is information related to the detection position of the detected target. an information determining means;
terminal communication information determination means for determining terminal communication information, which is information regarding a communication state with the terminal, based on information regarding communication acquired from a communication means that communicates with the terminal;
a target communication information determining means for determining, from the environment information, target communication information, which is information relating to a communication state to be realized with a detected target if the target includes a terminal;
The degree of position matching, which is the degree of matching between the target position information and the predicted target position information predicted from the target position information determined for the tracking terminal, and the tracking terminal Determining whether or not the tracked terminal and the detected target are in correspondence based on at least one of the degree of correspondence between the terminal communication information and the target communication information. causing the computer to function as a tracked object determination means for determining the object determined to be in a corresponding relationship as the object related to the terminal being tracked;
The tracked object determining means, at least when occlusion is resolved for the detected object, when the degree of communication compatibility is large enough to satisfy a predetermined condition, the tracking object determination means is based on the degree of communication compatibility, not based on the degree of location matching. A terminal tracking program, characterized in that it makes a judgment, or makes the judgment by increasing the degree based on the degree of communication compatibility as the degree of communication compatibility increases or when it is large until a predetermined condition is satisfied. Detecting a predetermined target that may include the terminal from environmental information generated by a sensor and relating to the environment in which the terminal may exist, determining target position information that is information relating to the detection position of the detected target, and , terminal communication information, which is information relating to the state of communication with the terminal, is determined based on information relating to communication acquired from a communication means that communicates with the terminal; a first step of determining target communication information, which is information relating to a communication state to be embodied with a target if the target includes a terminal;
The degree of position matching, which is the degree of matching between the target position information and the predicted target position information predicted from the target position information determined for the tracking terminal, and the tracking terminal Determining whether or not the tracked terminal and the detected target are in correspondence based on at least one of the degree of correspondence between the terminal communication information and the target communication information. a second step of determining and determining the object determined to be in correspondence with the object associated with the tracking terminal;
In the second step, when occlusion is eliminated at least for the detected object, the determination is based on the degree of communication compatibility, not based on the degree of position matching when the degree of communication compatibility is large enough to satisfy a predetermined condition. or, when the degree of communication compatibility is large or is large until a predetermined condition is satisfied, the determination is made by increasing the degree based on the degree of communication compatibility. Method.

Images