JP7514212B2 - Device, program, and method for tracking terminals using communication status information to handle occlusion - Google Patents

Description

This relates to technology for obtaining information about a terminal as a communication target, in particular technology for tracking that terminal.

The fifth-generation mobile communications system (5G), which is currently being introduced, uses millimeter-wave radio waves (mm waves) to enable high-performance communications such as high speed, large capacity, low latency, and multi-terminal connection. Here, mmWaves have a high degree of directivity and are not prone to diffraction, making it highly likely that their propagation will be blocked or attenuated by objects between the terminal and the base station. For this reason, with 5G, a major problem is that the presence of such objects can cause a sudden drop in received power, which can result in, for example, the communication connection itself being cut off.

If it were possible to predict the drop in received power due to such objects, it would be possible to maintain communication connections without interruption by controlling and securing the communication paths of the terminals. For example, Non-Patent Document 1 discloses a technique for predicting received power in a communication device through machine learning, using time-series three-dimensional environmental information acquired by an RGB-D camera that can generate depth images in addition to RGB images.

Specifically, in this technology, a machine learning model is constructed using training data that links time-series environmental information resulting from capturing an image of the communication environment with time-series information on the received power measured (at the same time as capturing the image) at the communication terminal that receives radio waves from the transmitting station as correct answer data, and this constructed machine learning model is used to determine a predicted value for the received power at the communication terminal from the time-series environmental information captured by the RGB-D camera.

Here, in predicting the received power of millimeter waves, it is important to continuously grasp the position of the communication terminal. However, if the position is grasped using the positioning results from a sensor on the communication terminal side, a mechanism is required for the communication terminal to notify the base station of the positioning results, and furthermore, the large power consumption and calculation load on the communication terminal side become problems.

In contrast to this, Patent Document 1, which includes the present inventor as one of its inventors, discloses a technology that does not depend on the positioning results from the communication terminal, and that estimates the position of a communication terminal using time-series information on the received power of millimeter waves and time-series information on the shielding relationship between objects estimated from, for example, image information of the environment. Furthermore, Non-Patent Document 2 discloses a technology that continuously captures the position of an object in an image by utilizing the consistency of the detected position and appearance features of the object obtained from the image.

JP 2021-078022 A

Nishio, R., "Prediction and control of wireless communication quality using machine learning", IEICE Technical Report, Vol. 118, No. 57, SR2018-1, pp. 1-8, 2018. Nicolai Wojke et al., "Simple Online and Realtime Tracking with a Deep Association Metric", 2017 IEEE International Conference on Image Processing, pp. 3645-3649, 2017. ＜https://doi.org/10.1109/ICIP.2017.8296962＞

If it were possible to continue tracking a communications terminal using such conventional technology, it would be possible to continue the communications connection or to temporarily suspend and then immediately resume it, even if the reception power suddenly drops due to the presence of an object.

However, object tracking technology based on image information such as that disclosed in Non-Patent Document 2 may fail to track a target when a phenomenon known as occlusion occurs, in which an object in the foreground at least partially covers an object in the background. In other words, conventional object tracking technology based on image information has problems when occlusion occurs, such as the target being "lost" because it is hidden by another object and cannot be detected, causing tracking to cease, or an "ID switch" occurring when the target being tracked crosses another object and the other object is mistaken for the target being tracked, causing the tracking to stray from what should have been done.

Therefore, even if the target tracking technology disclosed in Non-Patent Document 2 is applied to tracking communication terminals, if such an occlusion problem occurs, it will not be possible to accurately grasp the position of the communication terminal. Furthermore, the communication terminal position estimation technology described in Patent Document 1 also assumes that the terminal has not been lost or an ID switch has not occurred, and does not propose any measures to be taken when such problems occur.

The present invention aims to provide a device tracking device, program, and method that can continue tracking a device more appropriately even if occlusion occurs for the device being tracked.

According to the present invention, there is provided a terminal tracking device that detects a predetermined target including a terminal from environmental information related to the environment, the environmental information being acquired by a sensor that senses an environment in which a terminal may exist, and that may include a predetermined target, and tracks the terminal, comprising:
a terminal communication state determination means for determining terminal communication state information, which is information relating to a communication state between the terminal and the communication means, based on information relating to communication acquired from a communication means for communicating with the terminal;
a target communication state determination means for determining, from the environmental information, target communication state information which is information related to a communication state to be realized between the predetermined target and the terminal if the predetermined target includes a terminal;
an object detection information determination means for determining object position related information, which is information related to the detected position of the predetermined object, and/or object appearance information, which is information related to the detected appearance of the predetermined object, from the environmental information;
a tracking target determination means for calculating a communication compatibility level, which is a level of correspondence between terminal communication state information determined for a tracked terminal and target communication state information determined for the predetermined target when occlusion occurs for the predetermined target, determining whether or not the tracked terminal and the predetermined target are in a corresponding relationship based on the communication compatibility level, and determining the predetermined target determined to be in a corresponding relationship as a target related to the tracked terminal;
The tracking target determination means determines whether or not the tracking terminal and the predetermined target are in a corresponding relationship based on a position-related matching degree, which is a degree of matching between the determined target position-related information and predicted target position-related information predicted from past target position-related information, and/or an appearance matching degree, which is a degree of matching between the determined target appearance information and past target appearance information.
The present invention provides a terminal tracking device.
According to the present invention, there is also provided a terminal tracking device that detects a predetermined target including a terminal from environmental information related to the environment, the environmental information being acquired by a sensor that senses an environment in which a terminal may exist, and that may include the predetermined target, and tracks the terminal, comprising:
a terminal communication state determination means for determining terminal communication state information, which is information relating to a communication state between the terminal and the communication means, based on information relating to communication acquired from a communication means for communicating with the terminal;
a target communication state determination means for determining, from the environmental information, target communication state information which is information related to a communication state to be realized between the predetermined target and the terminal if the predetermined target includes a terminal;
an object detection information determination means for determining object position related information, which is information related to the detected position of the predetermined object, and/or object appearance information, which is information related to the detected appearance of the predetermined object, from the environmental information;
a tracking target determination means for calculating a communication compatibility level, which is a level of correspondence between the terminal communication state information determined for the tracked terminal and the target communication state information determined for the predetermined target, when occlusion occurs for the predetermined target, determining whether or not the tracked terminal and the predetermined target are in a corresponding relationship based on the communication compatibility level, and determining the predetermined target determined to be in a corresponding relationship as a target related to the tracked terminal;
having
The tracking target determination means determines whether or not the tracked terminal and the predetermined target are in a corresponding relationship based on at least a position-related matching degree, which is a degree of matching between the determined target position-related information and predicted target position-related information predicted from past target position-related information, and/or an appearance matching degree, which is a degree of matching between the determined target appearance information and past target appearance information, during a time period in which no occlusion occurs for the predetermined target.
The present invention provides a terminal tracking device.

In the terminal tracking device according to the present invention, it is also preferable that the target communication state determination means determines the target communication state information by assuming that the target is present at a position interpolated using the detection positions determined before and after the occlusion during a time period during which an occlusion occurs and the detection position of the target is not determined.

In yet another embodiment of the terminal tracking device according to the present invention, the tracking target determination means calculates a tracking cost which is a weighted sum of a position-related cost which is a monotonically decreasing function of the position -related match degree and/or an appearance cost which is a monotonically decreasing function of the appearance match degree for the specified target, and a communication state cost which is a monotonically decreasing function of the communication compatibility degree, using a weighting coefficient which may take on a zero value, and preferably determines a specified target for which the tracking cost is small enough to satisfy a specified condition as the target related to the tracked terminal.

Furthermore, in the embodiment in which the tracking cost is calculated as described above, it is also preferable that the tracking target determination means, when an occlusion occurs for the specified target, in which the detection position cannot be determined, and then resolves, calculates a tracking cost in which the weighting factor applied to the communication state cost for the specified target is increased and the weighting factor applied to the position-related cost and/or the appearance cost is decreased or set to zero.

In the embodiment in which the tracking cost is calculated as described above, it is also preferable that the tracking target determination means calculates a tracking cost in which the weighting coefficient applied to the communication state cost for the specified target is successively decreased from a larger value over time from the point in time when occlusion occurs and is resolved for the specified target, and the weighting coefficient applied to the position-related cost and/or the appearance cost is successively increased from a smaller value or zero.

Furthermore, in the embodiment in which the tracking cost is calculated as described above, it is also preferable that the tracking target determination means calculates the tracking cost by increasing the weighting factor applied to the communication state cost for the specified target and decreasing or setting to zero the weighting factor applied to the position-related cost and/or the appearance cost for the specified target during the period from the time when occlusion occurs and is resolved for the specified target until a specified time has elapsed.

In the embodiment for calculating the tracking cost, the tracking target determination means
When an occlusion occurs between the predetermined objects, it is determined whether each of the predetermined objects related to the occlusion is located in front or behind based on past information related to the object positions;
When the occlusion is resolved, if it is determined in the terminal communication status information that there was an effect of blocking of communication radio waves, it is preferable to set the communication state cost of the specified target determined to be in the foreground to a larger value and set the communication state cost of the specified target determined to be in the background to a smaller value or zero, and if it is determined in the terminal communication status information that there is no effect of blocking of communication radio waves, it is preferable to set the communication state cost of the specified target determined to be in the foreground to a smaller value or zero and set the communication state cost of the specified target determined to be in the background to a larger value.

Furthermore, in the embodiment for calculating the tracking cost, the tracking target determination means
When an occlusion occurs and is resolved for a plurality of predetermined objects, a tracking process is continued for each of the plurality of predetermined objects as a candidate object for the tracked terminal;
During the continued tracking process, when the tracking cost or the communication state cost of a specific target becomes large enough to satisfy a predetermined condition, it is also preferable to terminate the tracking process by determining that the target is not related to the terminal being tracked.

According to the present invention, there is also provided a terminal tracking program that causes a computer to function to detect a predetermined target including a terminal from environmental information that is information related to the environment and may include a predetermined target, the information being acquired by a sensor that senses an environment in which a terminal may be present, and to track the terminal, the terminal tracking program comprising:
a terminal communication state determination means for determining terminal communication state information, which is information relating to a communication state between the terminal and the communication means, based on information relating to communication acquired from a communication means for communicating with the terminal;
a target communication state determination means for determining, from the environmental information, target communication state information which is information related to a communication state to be realized between the predetermined target and the terminal if the predetermined target includes a terminal;
an object detection information determination means for determining object position related information, which is information related to the detected position of the predetermined object, and/or object appearance information, which is information related to the detected appearance of the predetermined object, from the environmental information;
a computer is made to function as a tracking target determination means for, when occlusion occurs with respect to the predetermined target, calculating a communication compatibility degree which is a degree of correspondence between terminal communication state information determined for the tracked terminal and target communication state information determined for the predetermined target, determining whether or not the tracked terminal and the predetermined target are in a corresponding relationship based on the communication compatibility degree, and determining the predetermined target determined to be in a corresponding relationship as a target related to the tracked terminal ;
The tracking target determination means determines whether or not the tracking terminal and the predetermined target are in a corresponding relationship based on a position-related matching degree, which is a degree of matching between the determined target position-related information and predicted target position-related information predicted from past target position-related information, and/or an appearance matching degree, which is a degree of matching between the determined target appearance information and past target appearance information.
A terminal tracking program is provided.

According to the present invention, there is further provided a terminal tracking method implemented by a computer, which detects a predetermined target including the terminal from environmental information related to the environment, which may include the predetermined target, acquired by a sensor that senses an environment in which the terminal may be present, and tracks the terminal, comprising:
determining terminal communication state information, which is information related to a communication state between the terminal and the communication means, based on information related to communication acquired from a communication means that communicates with the terminal;
determining, from the environment information, object communication state information which is information related to a communication state to be realized between the predetermined object and the terminal if the predetermined object includes a terminal;
determining object position related information, which is information related to a detected position of the predetermined object, and/or object appearance information, which is information related to a detected appearance of the predetermined object, from the environmental information;
a step of calculating a communication compatibility degree, which is a degree of correspondence between terminal communication state information determined for the tracked terminal and target communication state information determined for the predetermined target when occlusion occurs for the predetermined target, determining whether or not the tracked terminal and the predetermined target are in a correspondence relationship based on the communication compatibility degree, and determining the predetermined target determined to be in a correspondence relationship as a target related to the tracked terminal;
In the step of determining the target related to the tracked terminal, whether or not the tracked terminal and the predetermined target are in a corresponding relationship is determined based on a position-related matching degree, which is a degree of matching between the determined target position-related information and predicted target position-related information predicted from past target position-related information, and/or an appearance matching degree, which is a degree of matching between the determined target appearance information and past target appearance information.
The present invention provides a terminal tracking method.

The device, program, and method of the present invention make it possible to continue tracking a terminal more appropriately even if occlusion occurs for the terminal being tracked.

1 is a functional block diagram showing a functional configuration of an embodiment of a terminal tracking device according to the present invention; 5A to 5C are schematic diagrams for explaining two aspects of occlusion determination processing in the occlusion determination unit according to the present invention. FIG. 11 is a schematic diagram for explaining a specific example of a calculation process of a communication state cost according to the present invention. 10A and 10B are schematic diagrams for explaining a specific example of a calculation process of a position-related cost and an appearance cost according to the present invention; 1A to 1C are schematic diagrams for explaining various embodiments of a tracking target determination process in a tracking target determination and management unit according to the present invention. FIG. 13 is a schematic diagram for explaining still another embodiment of the tracking target determination process in the tracking target determination and management unit according to the present invention. FIG. 13 is a schematic diagram for explaining still another embodiment of the tracking target determination process in the tracking target determination and management unit according to the present invention.

The following describes in detail an embodiment of the present invention with reference to the drawings.

[Terminal tracking device]
FIG. 1 is a functional block diagram showing the functional configuration of an embodiment of a terminal tracking device according to the present invention.

The base station 1 shown in FIG. 1 as one embodiment of a terminal tracking device according to the present invention is a communication relay device compatible with 5G (fifth generation mobile communication system) in this embodiment, and is a device capable of communicating with multiple terminals 2, which are also communication terminals compatible with 5G.

In addition, as one embodiment of a terminal tracking device, the base station 1 is also a device capable of detecting specific targets, including the terminal 2, from image data serving as "environmental information" acquired by the camera 103, which acts as a sensor that senses the environment in which the terminal 2 may be present, and which is information relating to the environment and which may include specific targets, and tracking the terminal 2.

In this embodiment, the camera 103 is an RGB camera capable of capturing images of the communication area with a predetermined angle of view. The image data (video data) generated by the camera 103 includes the following:
(a) A person who possesses or carries the terminal 2,
(b) Moving objects such as automobiles, motorcycles, railroad cars, robots, drones, etc. that have terminal 2 mounted, included, or carried on board, and (c) "objects" such as equipment, facilities, buildings, and fixed objects on which terminal 2 is installed, may be included (images may be captured). Furthermore, "objects" such as people, moving objects, equipment, facilities, buildings, and fixed objects (including plants such as trees) that are not related to terminal 2 may also be included.

Incidentally, while such an "object" can be something for which a correspondence with the terminal 2 being tracked needs to be determined (in other words, something for which it needs to be determined whether or not it is a tracking target that should be tracked), its presence between the terminal 2 and the base station 1 can also become an obstacle to communication radio waves.

Here, in particular, 5G, which is the communication method of this embodiment, uses millimeter wave radio waves (millimeter waves) for communication. These millimeter waves have a high degree of linearity and are not easily diffracted, so their propagation is likely to be blocked or attenuated by "objects" that exist between the terminal 2 and the base station 1. For this reason, in 5G, a serious problem occurs in that the presence of such "objects" can cause a sudden drop in the received signal power at the base station 1, leading to, for example, a situation in which the communication connection itself is cut off. In contrast, even if "occlusion" occurs with respect to the terminal 2 being tracked, if tracking of this terminal 2 can be continued appropriately, it is possible to continue the communication connection with the terminal 2, or to immediately resume it after temporarily suspending it.

Incidentally, the above-mentioned "occlusion" hereinafter refers to a phenomenon in which an "object" in front of the base station 1 covers at least a part of an "object" in the back. For example, a phenomenon in which multiple "objects" intersect, or at least one "object" is blocked by at least one other "object", will also be treated as "occlusion" hereinafter. When such "occlusion" occurs, there is an increased possibility that a specific object (e.g. a "person") cannot be detected from "environmental information" (image data in this embodiment), or can only be detected inaccurately.

Specifically, in order to solve the serious problem when such an occlusion occurs, the base station 1:
(A) a terminal communication state determination unit 111 that determines "terminal communication state information" which is information related to the communication state with the terminal 2 based on information related to communication (e.g., communication history information) acquired from a communication means (in this embodiment, a communication interface 101, a communication control unit 116, and a communication history information storage unit 102 provided in the terminal itself) that communicates with the terminal;
(B) a target communication state determination unit 112 that determines, if a predetermined target (e.g., a “person”) detected from the “environmental information” acquired by the sensor (the camera 103 in this embodiment) includes the terminal 2, “target communication state information” that is information related to the communication state to be realized with the predetermined target (the “person”);
(C) A tracking target determination and management unit 115 is provided which, when occlusion occurs with respect to a specified target (e.g., a "person"), calculates a "communication compatibility degree" which is the degree of correspondence between the "terminal communication status information" determined for the tracked terminal 2 and the "target communication status information" determined for the specified target (e.g., a "person"), determines whether or not the tracked terminal 2 and the specified target (e.g., a "person") correspond to each other based on this "communication compatibility degree", and determines the specified target (e.g., a "person") that is determined to correspond to each other as the target related to the tracked terminal 2.

The "predetermined target" detected from the "environmental information" in (B) above can be a "person" or a "mobile object" that carries, is equipped with, or may contain the terminal 2, as will be explained later. In some cases, the "predetermined target" may also be "equipment," "facilities," "buildings," "fixed objects," etc. that may contain the terminal 2.

In addition, the "terminal communication state information" in (A) above can be, in a preferred embodiment, (α) information related to the strength of the radio waves received from terminal 2 (e.g., received signal power, RSSI (Received Signal Strength Indicator)). Or, (β) information related to the presence or absence of a communication connection with terminal 2.

On the other hand, the "target communication status information" in (B) above can be, in accordance with the "terminal communication status information" above, (α') information (e.g., RSSI) relating to the strength of radio waves received from a detected predetermined target (e.g., a "person") if the detected predetermined target (e.g., a "person") includes terminal 2. Or, (β') information relating to the possibility of a communication connection with the detected predetermined target (e.g., a "person"), which is determined using the detection position of the detected predetermined target (e.g., a "person").

Furthermore, the "degree of communication compatibility" in (C) above can be specifically, for example, the degree of agreement between time series data on the quantity related to radio wave strength (e.g., RSSI value) as "terminal communication state information" and time series data on the quantity related to radio wave strength (e.g., RSSI value) as "target communication state information". Alternatively, it can be the degree of agreement between time series data on the presence or absence of a communication connection as "terminal communication state information" and time series data on the possibility of a communication connection as "target communication state information".

In this way, according to the base station 1 serving as a terminal tracking device of the present invention, when occlusion occurs, it is possible to determine whether or not there is a correspondence between the tracked terminal 2 and a detected predetermined target (e.g., a "person") using the "degree of communication compatibility" between the communication state of the terminal 2 and the estimated communication state of the predetermined target, without depending on whether or not the predetermined target is detected from "environmental information" (e.g., image data). As a result, even when occlusion occurs for the tracked terminal 2, it is possible to continue tracking of this terminal 2 more appropriately.

As a result, in this embodiment, even if an object comes between the base station 1 and the terminal 2 and the reception power drops suddenly, the base station 1 can continue or immediately resume communication with the terminal 2.

Incidentally, when an occlusion occurs, for example when an occlusion occurs and is resolved, the base station 1 performs terminal tracking processing using the "degree of communication compatibility" described above. Here, in conjunction with this processing, it is also preferable to perform object tracking processing by image recognition that utilizes the consistency of the detection position and appearance features of the object obtained from the image, as described in Non-Patent Document 2. Such a preferred embodiment will be described in detail later.

Furthermore, in a period in which no occlusion occurs that is separate from the time interval in which occlusion occurs,
(a) Terminal tracking process using the "communication compatibility level" described above;
(b) a target tracking process using image recognition that utilizes the consistency between the detection position and the appearance features of a target obtained from an image, as described in Non-Patent Document 2, and (c) other publicly known terminal/target tracking processes, such as a technology described in Patent Document 1 as one embodiment, which calculates the degree of correspondence (positional correspondence) between terminal position information determined based on communication history information and target position information related to the detection position of a specific target detected from environmental information (image data), and determines whether or not there is a correspondence between the terminal and the specific target based on this positional correspondence, and performs tracking. It is also preferable to implement at least one or more of the following technologies in combination.

In addition, although 5G is adopted as the communication method between the base station 1 and the terminal 2 in this embodiment, other communication methods such as LTE may be used, and further, the communication between the terminal tracking device according to the present invention and the communication terminal may be based on various other wireless communication standards. For example, even in a communication method in which the problem of obstruction by objects is not as significant as in 5G, there are many situations in which it is necessary to accurately identify and track subordinate communication terminals using targets detected from "environmental information". For example, there are various needs for terminal identification and tracking, such as determining the relationship between the viewed pages of a user determined to be in correspondence with a certain communication terminal and the user's movement line, and using this for marketing and management. In contrast, the terminal tracking device according to the present invention makes it possible to perform such terminal identification and tracking processing with high accuracy.

The terminal tracking device according to the present invention is not limited to a communication relay device such as a base station, of course. For example, it may be provided as a dedicated device for terminal tracking processing, connected to a communication relay device or facility such as a base station. Also, as the terminal tracking device according to the present invention, it is possible to use a cloud server, a non-cloud server device, a personal computer (PC), or a notebook or tablet computer, etc., equipped with the terminal tracking program according to the present invention.

Moreover, it is not impossible for at least one of the components (A) to (C) of the terminal tracking device (base station 1) according to the present invention to be a separate device from the other components. For example, it is also possible for the functions of (A) to (C) to be realized by a set of multiple servers. Even in such a case, the set of all of these can be considered to be a terminal tracking device or system that implements the terminal tracking method according to the present invention.

[Functional configuration of terminal tracking device, terminal tracking program]
1, the base station 1 has a communication interface 101, a communication history information storage unit 102, a camera 103, an environment information storage unit 104, a tracking information storage unit 105, and a processor memory as an embodiment of a terminal tracking device and a communication relay device according to the present invention. Here, the processor memory stores an embodiment of a communication relay program including a terminal tracking program according to the present invention, and has a computer function, and executes this communication relay program to perform terminal tracking processing and communication relay processing.

The processor memory also has, as functional components, a terminal communication state determination unit 111, a target communication state determination unit 112 including a position interpolation unit 112a, a target detection information determination unit 113, an occlusion determination unit 114, a tracking target determination and management unit 115 including a communication state cost calculation unit 115a, a position-related cost calculation unit 115b, and an appearance cost calculation unit 115c, and a communication control unit 116. These functional components can be considered as functions of a terminal tracking program and a communication relay program stored in the processor memory, and the process flow shown by connecting the functional components of the base station 1 in the functional block diagram of Figure 1 with arrows can also be understood as one embodiment of the terminal tracking method and communication relay method according to the present invention.

Also in the functional block diagram of FIG. 1, the communication control unit 116 functions as a base station by controlling wireless communication operations between the communication interface 101 and each terminal 2, and further acquires and records various information related to communication with each terminal 2, generates communication history information by organizing the information in chronological order, and stores and manages the information in the communication history information storage unit 102.

Also in the functional block diagram of FIG. 1, the camera 103 may be a visible light, near-infrared or infrared compatible imaging device equipped with a solid-state imaging element such as a CCD image sensor or a CMOS image sensor, and may also be an RGB camera, an RGB-D camera, a stereo camera or an all-sphere (omnidirectional) camera. Of course, instead of the camera 103, it is also possible to employ a sensor capable of sensing the environment in which the terminal 2 may be present and generating environmental information, such as a LiDAR, a laser/infrared positioning device, a TOF camera or a thermography device.

In this embodiment, the camera 103 is an RGB camera capable of generating RGB image data (RGB video data) as environmental information, and the generated RGB image data is stored and managed in the environmental information storage unit 104. Note that, although the camera 103 is installed within the base station 1 in this embodiment, it may also be, for example, a camera installed at a location distant from the base station 1, such as a surveillance camera in a city, which is connected to the base station 1 for communication.

In addition, multiple cameras 103 may be installed at different locations, whether inside or outside the base station 1. For example, the base station 1 may receive and acquire "target communication state information" generated by the cameras 103 installed at each of multiple base stations 1 present around the base station 1, and perform a tracking target determination process, which will be described in detail later. In this case, multiple base stations 1 work together to more appropriately perform the terminal tracking process according to the present invention.

<Target Detection Information Determination Process>
Also in the functional block diagram of FIG. 1, the object detection information determination unit 113 determines, from image data (environmental information) that may include a “predetermined object” acquired from the environmental information accumulation unit 104,
At least one, and preferably both, of (a) target position related information, which is information relating to the detected position of a "specified target," and (b) target appearance information, which is information relating to the detected appearance of a "specified target," is determined.

As already mentioned, the "prescribed subject" is, for example,
(a) A person (user) who owns the terminal 2,
(b) An automobile equipped with a terminal 2 having a drive recorder function, a CAN information transfer function, an interface function for automatic driving control by a server, etc.,
(c) It can be a "person" or "moving object" that may include the terminal 2 that is the communication destination related to the communication history information, such as an autonomous mobile robot or an autonomous flying drone equipped with a terminal 2 that has an interface function for autonomous movement control by a server. In some cases, "equipment,""facility,""building,""fixedobject," etc. that may include the terminal 2 may also be a "predetermined target."

Note that the target detection information determination unit 113 of this embodiment can of course also simultaneously detect multiple types of targets, including a "predetermined target." Here, when one "predetermined target" is detected as a tracking target from one piece of image data, other objects (including other predetermined targets) detected from this image data are treated as targets that may cause occlusion (may cause interference) with the detected "predetermined target."

Here, the process of determining the target position related information (a) and the target appearance information (b) will be described. In this embodiment, the target detection information determination unit 113 sets a plurality of image regions specified in the acquired image data, calculates a score representing the likelihood of each image region being a predetermined target using a corresponding object detector, and detects an image region having a score high enough to satisfy a predetermined condition as a detected image region related to the predetermined target. Next,
(a) The position (detection position) of a detected image area relating to a detected predetermined object, the range of the detected image area (e.g., coordinate values of the four corners), and further, the detection speed as a change in the detection position between successive image data are set as the object position related information of (a) above,
(b) The image feature amount of the detected image area relating to the specified target, calculated using a known image feature extractor (e.g., a CNN (Convolutional Neural Networks) image feature extractor), is used as the target appearance information described above in (a).

The object detector may be, for example, the one described in non-patent document: Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao, “YOLOv4: Optimal Speed and Accuracy of Object Detection”, arXiv:2004.10934v1, 2020. Furthermore, it is also possible to determine the target position-related information (a) above and the target appearance information (b) above using various other object detectors and image feature extractors known in the field of image recognition technology.

When a stereo camera is used as the camera 103, it is also possible to apply an object detector to the stereo camera image data, which is the generated environmental information, for example, as described in non-patent literature: Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg, “SSD: single shot multibox detector”, European Conference on Computer Vision, Computer Vision-ECCV 2016, pp.21-37, 2016.

In yet another embodiment, when LiDAR is used instead of the camera 103 and three-dimensional point cloud data is used as environmental information, an object detector for detecting a specific target can be used, for example, as disclosed in non-patent document: Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas, “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation”, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp.77-85, 2017.

Furthermore, in this embodiment in which image data is used as environmental information, instead of the above image feature extractor, a feature extractor described in, for example, non-patent document: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep Residual Learning for Image Recognition”, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1, pp.770-778, 2016 may be used.

Incidentally, in this embodiment, the target detection information determination unit 113 stores the determined "target position related information" and "target appearance information" for each detected target for a specified period of time in the past, and can provide them to the tracking target determination and management unit 115 as appropriate.

<Terminal communication status information determination process>
1, the terminal communication state determination unit 111 acquires communication history information from the communication history information storage unit 102, and determines "terminal communication state information" related to communication with the terminal 2 based on the communication history information. In this embodiment, the "terminal communication state information" is
The data is either one or both of (α) time series data of the received signal power (RSSI) of the radio signal received from terminal 2 and (β) time series data of information regarding the presence or absence of a communication connection with terminal 2.

For the above (α), the terminal communication state determination unit 111 may generate time series data of the received signal power variable E_t indicating the signal power value at time t of the radio signal received from the terminal 2. For the above (β), the terminal communication state determination unit 111 may also generate time series data of time series information on whether or not a communication connection with the terminal 2 is established at each time point, that is, for example, time series data of a binary communication connection presence/absence variable B_t (∈{-1, 1}) that takes a value of 1 if a communication connection is established at time t and -1 if it is not established. However, taking into consideration the accuracy of the degree of agreement (degree of communication compatibility) with the target communication state information calculated later, it is preferable that the "terminal communication state information" includes the time series data of the received signal power variable E_t of the above (α).

Incidentally, in this embodiment, the terminal communication state determination unit 111 stores the "terminal communication state information" determined for each terminal 2 being tracked for a predetermined period of time in the past, and can provide it to the tracking target determination and management unit 115 as appropriate. In addition, it is also preferable that the terminal communication state determination unit 111 acquires, for example, an IMSI (International Mobile Subscriber Identity) from each terminal 2 present under the control of the base station 1, compiles the determined "terminal communication state information" for each terminal 2 identified by the IMSI, and performs subsequent processing separately for each terminal 2.

<Target communication status information determination process>
Also in the functional block diagram of Figure 1, the target communication state determination unit 112 determines, from image data (environmental information) that may include a "specified target" obtained from the environmental information accumulation unit 104, "target communication state information," which is information related to the communication state that will be realized between the "specified target" and the terminal 2 if the "specified target" includes this terminal 2.

(When using received signal strength data as "target communication status information")
The “target communication state information” determined by the target communication state determination unit 112 is
(α') Information on the strength of radio waves received from the "predetermined target" if the "predetermined target" includes terminal 2 (received radio wave strength data)
In this case, the target communication state determining unit 112 determines, based on the detection position of the "predetermined target" detected from the image data (by the target detection information determining unit 113) and the detection positions of "other targets" (which may cause interference), as shown on the right side of FIG. 3, which will be used in the description later.
(a) determining a straight line segment connecting the “predetermined object” and the base station 1 (antenna);
(b) calculating occlusion degree information relating to the degree to which the “other object” occludes a predetermined area including the line segment;
(c) From the calculated degree of obstruction information, received radio wave intensity data that would be received from the "predetermined target" if the "predetermined target" includes terminal 2 is calculated as "target communication state information."

Specifically, the above-mentioned (b) information on the degree of occlusion is calculated as follows: First, as shown on the right side of Fig. 3, a predetermined size (a cross-sectional area at the detection position having a predetermined area) is set in advance for the "other objects" that may cause an obstacle according to their type (class), and then a rotating body area of a predetermined size that includes the above-mentioned straight line segment as the rotation axis is set. Then, at a certain point in time t (= t0 - M, t0 - M + 1, ..., t0 - 1, t0),
(b1) When any of the "other objects" has a cross-sectional area that is not included in the above-mentioned rotating body area at its detection position (the cross-sectional area of the other object does not overlap with the rotating body area), the value is 1.
(b2) In the case where a part of the cross-sectional area of any of the "other objects" is included in the above-mentioned rotating body area at the detection position (the cross-sectional area of the other object and the rotating body area have an overlapping part), the following formula (1) R = (S _a ∩ S _b ^- ) / S _a
The time series data of the variable E'_t having an R value calculated by the above formula can be used as the occlusion degree information.

Here, in the above formula (1), S _a is a set (e.g., a set of unit point areas forming the area) representing the cross-sectional area (perpendicular to the axis of rotation) at the detection position of the "other object" in the above rotating body area, and S _b is a set (e.g., a set of unit point areas forming the area) representing the cross-sectional area at the detection position of the "other object". Also, S _b ^- is the complement of S _b , and "/" is an operator indicating that the area value of the cross-sectional area related to the numerator set (e.g., the number of unit point areas) is divided by the area value of the cross-sectional area related to the denominator (e.g., the number of unit point areas).

Next, the target communication state determination unit 112 determines the time series data of this variable E'_t (or the variable E'_t multiplied by a predetermined constant) as "target communication state information" as received radio wave intensity data. Hereinafter, the variable E'_t will be referred to as a received power equivalent variable to distinguish it from the above-mentioned received signal power variable E_t.

In addition, when there are two or more "other objects" that have the above R value, _Sb may be the area formed by the entire cross-sectional areas of these "other objects," that is, the area expressed as the union of these cross-sectional areas. Furthermore, the above-mentioned revolution body area may be a cylindrical area, or may be one that can be formed by rotating a specified curve around a rotation axis, such as an ellipsoid.

As a further modification, the target communication state determination unit 112 may use a trained "target radio wave information estimation model" that outputs information related to the strength of received radio waves in response to input image data to derive information related to the strength of radio waves that will be received from a "specified target" if the "specified target" includes terminal 2 from the acquired image data, and determine the information related to the strength of radio waves as "target communication state information."

Here, as the above-mentioned "target radio wave information estimation model", a known machine learning model such as that disclosed in Non-Patent Document 1 can be adopted. Specifically, image data at a certain time t (= t0-M, t0-M+1, ..., t0-1, t0) is input to this model, and an estimated value of the received signal power is output. This estimated value is set as the received power equivalent variable E'_t value, and the time series data of this received power equivalent variable E'_t is set as the "target communication state information".

(When using communication connection availability data as "target communication status information")
The “target communication state information” determined by the target communication state determination unit 112 is
(β') Information regarding the availability of a communication connection with a "predetermined target" (communication connection availability data)
In this case, the target communication state determination unit 112 determines, based on the detected position of the "predetermined target" detected from the acquired image data and the detected positions of the "other targets" that may cause a fault,
(a) determining a straight line segment connecting the “predetermined object” and the base station 1 (antenna);
(b) calculating communication connection availability data indicating whether or not the “other object” is located on the line segment or within a predetermined area including the line segment, or whether or not the other object is blocking the line segment;
(c) It is also preferable to determine the calculated communication connection availability data as the "target communication status information."

More specifically, as the communication connection availability data of (b) above, a binary communication connection availability variable that takes a value of -1 if the detected position of any of the "other targets" is located on the above line segment or within a rotating body area of a predetermined size that includes this line segment as the rotation axis at a certain time point t (==t0-M, t0-M+1, ..., t0-1, t0), and takes a value of 1 otherwise:
(2) B'_t∈{－1, 1}
The time series data may be generated and used as the "target communication state information." Here, the above-mentioned rotating body region may be a cylindrical region, or may be one that can be formed by rotating a predetermined curve around a rotation axis, such as an ellipsoid.

As a modification, the target communication status determination unit 112 may preset a predetermined size (cross-sectional area at the detection position) for the "other target" according to its type (class), and define B'_t in (2) above as a communication connection feasibility variable that takes a value of -1 if any of the "other targets" is blocking the above line segment with that size (cross-sectional area) at its detection position (the above line segment penetrates the cross section of the "other target"), and a value of 1 if this is not the case.

The above is the "target communication status information"
The processing has been described for the case where (α') the time series data of the received power equivalent variable E'_t (received radio wave intensity data) is used, and for the case where (β') the time series data of the communication connection availability variable B'_t (communication connection availability data) is used. Here, taking into consideration the accuracy of the degree of agreement with the terminal communication state information calculated later (degree of communication compatibility), it is also preferable that the "target communication state information" is the time series data of the received power equivalent variable E'_t described above in (α').

Incidentally, in this embodiment, the target communication state determination unit 112 stores the determined "target communication state information" for each detected target for a specified period of time in the past, and can provide it to the tracking target determination and management unit 115 as appropriate.

<Occlusion Determination Processing>
Also in the functional block diagram of Figure 1, the occlusion determination unit 114 determines whether or not occlusion has occurred (or is predicted to occur) at each point in time for a specific target (e.g., a "person") related to the terminal 2 being tracked, based on information from the target detection information determination unit 113, and outputs this determination result to the tracking target determination and management unit 115, which will be described in detail later.

Figure 2 is a schematic diagram illustrating two aspects of the occlusion determination process in the occlusion determination unit 114.

First, according to FIG. 2(A), the occlusion determination unit 114 obtains from the target detection information determination unit 113 the detection position and detection speed (target position related information) of a specific target (e.g., a "person") related to the tracked device 2 at a time point earlier than time point t0 (e.g., time point t0-1), and determines the predicted position of this specific target ("person") at time point t0 from this information.

Next, if the occlusion determination unit 114 detects another specified object (e.g., a "person") at time t0 within a specified range including the determined predicted position (e.g., an image area corresponding to a real-space circular range of a specified size centered on the predicted position), it can determine that occlusion has occurred at this time t0.

Next, in the aspect shown in FIG. 2(B), the occlusion determination unit 114 first determines the predicted position at time t0 of a predetermined target (e.g., a "person") related to the tracked device 2, in the same manner as in the aspect shown in FIG. 2(A) described above.

Next, the occlusion determination unit 114 can determine that occlusion has occurred at time t0 if no object, including the specified object, is detected at time t0 within a specified range including the determined predicted position (for example, an image area corresponding to a real-space circular range of a specified size centered on the predicted position). In this case, it is highly likely that occlusion has occurred in which the specified object being tracked is completely blocked.

As another aspect of the occlusion determination process, when the target detection information determination unit 113 is unable to determine one or both of the target position-related information and the target appearance information for a specific target being tracked (e.g., a "person") from image data at a certain time t0, the occlusion determination unit 114 can receive a report to that effect and determine that occlusion has occurred at this time t0.

Furthermore, as a further aspect, the occlusion determination unit 114 can determine that occlusion has occurred at time t0 if the target detection information determination unit 113 detects a specific target (e.g., a "person") being tracked and another specific target (e.g., a "person") from image data at a certain time t0 and the distance between their detected positions (e.g., the distance in image space equivalent to a specific distance in real space) is less than a specific threshold. In this case, it is highly likely that partial occlusion has occurred to the extent that the detection position can be determined.

<Tracking target determination process/terminal tracking process>
Returning to the functional block diagram of FIG. 1 , in a time period in which no occlusion occurs for a specific target (e.g., a “person”) and the target detection information determination unit 113 determines target position related information and target appearance information, the tracking target determination/management unit 115 of this embodiment performs at least the following:
(a) a "position-related matching degree" which is the degree to which the determined target position-related information matches the predicted target position-related information predicted from the past target position-related information, and (b) an "appearance matching degree" which is the degree to which the determined target appearance information matches the past target appearance information.
Based on either one or both of the above (both in this embodiment), it is determined whether or not there is a corresponding relationship between the terminal 2 being tracked and a predetermined target (a "person"), and terminal tracking processing is performed.

Here, in this embodiment, the process uses the "communication compatibility level" as described above.
(c) A "communication correspondence degree" which is the degree to which the terminal communication state information determined for the tracked terminal 2 corresponds to the target communication state information determined for a given target ("person").
Based on this, it is determined whether or not there is a correspondence between the tracked terminal 2 and a specified target (a "person"), and the specified target (a "person") that is determined to have a correspondence is determined to be the target associated with the tracked terminal 2, and terminal tracking processing is performed.

Furthermore, when an occlusion occurs for a specific target, in which the detection position cannot be determined, and is then resolved, the tracking target determination and management unit 115 of this embodiment always uses the "degree of communication compatibility" described above in (c) above, and performs a correspondence relationship determination process between the terminal 2 and the specific target ("person") and a terminal tracking process, for example, based only on this "degree of communication compatibility."

This is because, in order to calculate the "position-related degree of agreement" in (A) above or the "appearance degree of agreement" in (B) above at the time when an occlusion with an undetermined detection position occurs and is resolved, "(past) target position-related information" and "(past) target appearance information" for the time period in which this occlusion occurred are required, but this information cannot be obtained precisely because such an occlusion occurred. In other words, in this case, neither the "position-related degree of agreement" in (A) above nor the "appearance degree of agreement" in (B) above can be calculated in the first place.

Hereinafter, a specific correspondence determination and terminal tracking process using the above-mentioned "position-related match degree", "appearance match degree", and "communication compatibility degree" will be described. In the functional block diagram of FIG. 1, the communication state cost calculation unit 115a, the position-related cost calculation unit 115b, and the appearance cost calculation unit 115c of the tracking target determination and management unit 115 each calculate the above-mentioned position-related match degree, appearance match degree, and communication compatibility degree for a predetermined target, and then:
(a) a “location-related cost” that is a monotonically decreasing function (e.g., the inverse) of the location-related degree of match;
(b) an “appearance cost” which is a monotonically decreasing function (e.g., the inverse) of the appearance match degree, and (c) a “communication status cost” which is a monotonically decreasing function (e.g., the inverse) of the communication compatibility degree.
In each of these costs, the smaller the cost, the higher the possibility that there is a corresponding relationship (between terminal 2 and a predetermined target).

Next, the tracking target determination and management unit 115 of this embodiment calculates a tracking cost Cost, which is a weighted sum of the calculated (a) position-related cost Cgeo, (b) appearance cost Capp, and (c) communication state cost Cradio for a plurality of detected predetermined targets (e.g., "people"), using a weight λ, which may be zero, and determines the predetermined target ("person") for which this tracking cost Cost is small enough to satisfy a predetermined condition, for example, the minimum tracking cost Cost, as the target related to the tracked terminal 2. Here, such a tracking cost Cost can be calculated, for example, by the following formula (3): Cost = (1 - λ) (Cgeo + Capp) + λ Cradio
In the above formula (3), λ is a weighting factor that is equal to or greater than 0 and equal to or less than 1. As a modification, the weighting factors for the position-related cost Cgeo and the appearance cost Capp in the above formula (3) (both are 1-λ in the above formula (3)) can be set to different values. Furthermore, although this will result in a problem of accuracy, it is also possible to use only one of the position-related cost Cgeo and the appearance cost Capp in the above formula (3).

(Communication state cost calculation)
FIG. 3 is a schematic diagram for explaining a specific example of the calculation process of the communication state cost according to the present invention.

In the specific example of calculation of the communication state cost Cradio shown in FIG. 3, the communication state cost calculation unit 115a first calculates:
(a) time series data of the received signal power variable E_t of the tracked terminal 2 (as terminal communication state information) determined by the terminal communication state determination unit 111;
(b) The degree of agreement (degree of communication compatibility) Cr with the time series data of the received power equivalent variable E'_t relating to a specific target (as target communication state information) determined by the target communication state determination unit 112 is calculated as the cross-correlation between the two in a specific time interval (Σ _t E_t * E'_t).

Incidentally, when time series data of the communication connection presence/absence variable B_t and the communication connection feasibility variable B'_t are adopted as the terminal communication state information and the target communication state information, respectively, the degree of coincidence Cr is calculated as the cross-correlation (Σ _t B_t * B'_t) between them in a specified time interval.

More specifically, the communication state cost calculation unit 115a calculates the degree of match Cr at time t0 when it is determined that no occlusion has occurred such that the detection position cannot be determined (when the detection position (object position related information) of the predetermined object is stored in the object detection information determination unit 113), as a cross-correlation between the received signal power variable E_t and the received power equivalent variable E'_t in a predetermined past time interval (time interval of t=t0-M, t0-M+1, ..., t0-N), using the following equation (4): Cr=C0·Σ _t=t0-M ^t0-N E_t*E'_t
Here, in the above formula (4), Σ _t=t0-N ^t0-N is an operator indicating the summation for time t (=t0-M, t0-M+1, ..., t0-N). Furthermore, C0 is a coefficient for adjusting the Cr value to 1 in the case of a perfect match. Furthermore, M is an integer set so that time t0-M is a time in the past that is sufficient to obtain a cross-correlation (for example, a time 10 seconds ago).

Furthermore, N is an integer equal to or greater than 1 such that time t0-N is the latest (closest to time t0) time point at which the detected position (object position related information) of the predetermined object is stored (in the object detection information determination unit 113) from time t0. Therefore, when N=1 and the detected position of the predetermined object at time t0-1, which is immediately before time t0, is stored in the object detection information determination unit 113, that is, when no occlusion with an undetermined detected position has occurred recently, the degree of match Cr at time t0 is expressed by the following formula (4'): Cr=C0·Σ _t=t0-M ^t0-1 E_t*E'_t
It will be calculated using:

In contrast, if N>1, the detection position of the specified target in the time interval from time t0-N+1 to time t0-1 is not determined due to the occurrence of occlusion to an extent that detection processing is not possible, and as a result, the received power equivalent variable E'_t is not calculated in the time interval in which such occlusion occurred. Therefore, in this case, the degree of match Cr expressed by the above formula (4) cannot be calculated as it is.

Then, the position interpolation unit 112a (FIG. 1) of the target communication state determination unit 112, in response to the fact that the detection position in the time interval from time t0-N+1 to time t0-1 has not been determined and stored (the received power equivalent variable E'_t has not been calculated), determines the position of the specified target at each time point (time t0-N+1, time t0-N+2, ..., time t0-1) in the time interval in which such occlusion occurred by an interpolation process. Specifically, the detection position at the acquired time t0-N (or the detection position acquired before time t0-N) and the detection position at the acquired time t0 are complemented by a known interpolation method, such as linear interpolation or spline interpolation, to determine the positions at time t0-N+1, time t0-N+2, ..., and time t0-1.

Then, the target communication state determination unit 112 (Figure 1) determines that the specified target is present at the position at each time point determined as the interpolation position (time point t0-N+1, time point t0-N+2, ..., time point t0-1), and determines the received power equivalent variable E'_t (target communication state information) at each time point. This enables the communication state cost calculation unit 115a to calculate the degree of match Cr using the above formula (4) even at time point t0 when the occlusion that occurred is resolved.

Furthermore, the communication state cost calculation unit 115a calculates the reciprocal of the degree of match Cr calculated as described above, and determines this reciprocal as the communication state cost Cradio. Here, since the degree of match Cr is a value that satisfies 0<Cr≦1, the communication state cost Cradio is a value of 1 or more, and the larger this communication state cost Cradio is, the more likely it is that the specified target does not correspond to the terminal 2 being tracked.

Incidentally, in Figure 3, the communication state cost Cradio of target A is smaller than the communication state cost Cradio of target B because the waveform of the received power equivalent variable E'_t is more similar to the waveform of the received signal power variable E_t. Therefore, looking only at the communication state cost Cradio, it can be said that target A is more likely to correspond to the tracked terminal 2.

(Location-related cost calculation, appearance cost calculation)
FIG. 4 is a schematic diagram for explaining a specific example of the calculation process of the position-related cost and the appearance cost according to the present invention.

First, in the specific example of calculating the location-related cost Cgeo shown in FIG. 4A, the location-related cost calculation unit 115b calculates the location-related cost Cgeo for the given target as follows:
(a) The range (e.g., coordinate values of the four corners) of the determined detected image area (as the target position related information),
(b) A degree of match (position-related degree of match) Cg between the range of the detected image area predicted (as predicted target position-related information) from the past detected position (as target position-related information) is calculated.

More specifically, at time t0 when it is determined that no occlusion has occurred such that the detection position cannot be determined (the detection position (target position related information) of the predetermined target is stored in the target detection information determination unit 113), the position related cost calculation unit 115b
(a) the range S1 of the detected image area for the given object at this time t0;
(b) From the detection position at time t0-N, the latest (closest to time t0) at which the detection position (object position related information) of the predetermined object is stored (in the object detection information determination unit 113), the area of the overlapping portion with the range S2 of the predicted detection image area at time t0 predicted using the detection speed at time t0-N is taken as S1∩S2, and further, the area occupied by the entire range S1 of the detected image area (a) and the range S2 of the predicted detection image area (b) is taken as S1∪S2, and the following formula (5) Cg=(S1∩S2)/(S1∪S2) is obtained.
The degree of match Cg at time t0 is calculated by the above.

In addition, according to the calculation process of the degree of coincidence Cg described above, even if there are times when occlusion occurs and the detection position cannot be determined, i.e., when N>1, it is possible to reliably calculate the degree of coincidence Cg at time t0.

Furthermore, the location-related cost calculation unit 115b calculates the reciprocal of the degree of match Cg calculated as described above, and determines this reciprocal as the location-related cost Cgeo. Here, since the degree of match Cg is a value that satisfies 0<Cg≦1, the location-related cost Cgeo is a value of 1 or more, and the larger this location-related cost Cgeo is, the more likely it is that the specified target does not correspond to the terminal 2 being tracked.

Next, in the specific example of the appearance cost Capp calculation shown in FIG. 4B, the appearance cost calculation unit 115c calculates the appearance cost Capp for the given target as follows:
(a) a determined image feature amount of the detected image region (as object appearance information);
(b) The degree of match (degree of appearance match) Ca with the image feature amount of the past detected image area (as target appearance information) is calculated.

More specifically, at time t0 when it is determined that no occlusion has occurred such that the detection position cannot be determined (when the object appearance information of the predetermined object is stored in the object detection information determination unit 113), the appearance cost calculation unit 115c
(a) image features derived using a known CNN image feature extractor for the detected image region of the given object at this time t0;
(b) As viewed from this time t0 (in the object detection information determination unit 113), at the latest time t0-N (closest to time t0) for which object appearance information of the specified object is saved, the similarity between the detected image area of the specified object and the image feature derived using the same CNN image feature extractor as in (a) is calculated using a known method for calculating similarity between features, for example the method described in Non-Patent Document 2, and the calculated similarity is regarded as the matching degree Ca at time t0.

In addition, according to the calculation process of the degree of coincidence Ca as described above, even when there are times when occlusion occurs and the detection position cannot be determined, i.e., when N>1, it is possible to reliably calculate the degree of coincidence Ca at time t0.

Furthermore, the appearance cost calculation unit 115c calculates the reciprocal of the degree of coincidence Ca calculated as described above, and determines this reciprocal as the appearance cost Capp. Here, since the degree of coincidence Ca is a value that satisfies 0<Ca≦1, the appearance cost Ca takes a value of 1 or more, and the larger this appearance cost Ca is, the more likely it is that the specified target does not correspond to the device 2 being tracked.

<Various embodiments of tracking target determination process>
FIG. 5 is a schematic diagram for explaining various embodiments of the tracking target determination process in the tracking target determination and management unit 115. In FIG.

According to FIG. 5, the tracking target determination and management unit 115 calculates the position-related costs Cgeo_A and Cgeo_B, the appearance costs Capp_A and Capp_B, and the communication state costs Cradio_A and Cradio_B of each of the target A ("person") and the target B ("person") at the time t0, and calculates the tracking costs Cost_A and Cost_B at the time t0 according to the following equation (6): Cost_A=(1-λ)·(Cgeo_A+Capp_A)+λ·Cradio_A
(7) Cost_B＝(1－λ)・(Cgeo_B＋Capp_B)＋λ・Cradio_B
It is determined using.

Next, in this embodiment, the tracking target determination and management unit 115 determines target A ("person") with the smallest tracking cost (the smaller tracking cost in the case of Figure 5) as the target related to the terminal 2 being tracked, and continues tracking that terminal 2. Incidentally, in Figure 5, there is no difference in appearance cost Capp between the two, but both the location-related cost Cgeo and communication status cost Cradio are clearly smaller for target A ("person").

Here, the tracking target determination and management unit 115 appropriately sets the value of the common weight λ (0≦λ≦1) in the above equations (6) and (7) using, for example, the contents of the determination result from the occlusion determination unit 114, thereby improving the accuracy of the determined tracking cost value and performing more appropriate tracking target determination and terminal tracking processing.

For example, in one preferred embodiment, the tracking target determination and management unit 115 preferably increases the weight λ (compared to the value in a situation where no occlusion has occurred) at time t0 when occlusion occurs and is resolved for a specific target (each of targets A and B in FIG. 5), or during the period from time t0 to time t0+Δt when a specific time Δt has elapsed. For example, if λ=0 has been set in a situation where no occlusion has occurred, the weight λ may be set to a non-zero finite value, such as 1, at time t0 or during the specific time Δt thereafter.

By setting the weight λ in this way, the weighting coefficient (λ) applied to the communication state cost Cradio for the specified target becomes larger, while the weighting coefficient (1-λ) applied to the location-related cost Cgeo and the appearance cost Capp becomes smaller or zero, ultimately resulting in the calculation of the tracking cost Cost with an emphasis on the communication state cost Cradio.

In this way, by reducing or eliminating the contribution to the tracking cost Cost of the position-related cost Cgeo and the appearance cost Capp, which are likely to be less accurate due to the occurrence of occlusion, while more strongly reflecting the communication state cost Cradio, which is not as affected by occlusion, it is possible to determine a tracking cost Cost that is expected to be reasonably accurate despite the occurrence of occlusion.

As another preferred embodiment, it is also preferable that the tracking target determination and management unit 115 gradually decreases the weight λ from a larger value (compared to the value in a situation where no occlusion has occurred) over time from time t0 when occlusion occurs and is resolved for a specific target (target A and target B in FIG. 5). For example, if λ = 1 at time t0, the value of the weight λ may be decreased from 1 to 0 over time.

For example, the attenuation coefficient of the weight λ is δ, and the weight λ at time t0 is λ0. The weight λ is expressed by the following equation (8): λ = λ0 · exp(t0 - t)
As a result, the tracking cost Cost(t) at time t is calculated as follows: (9) Cost(t) = (1 - λ0 exp(t0 - t)) (Cgeo + Capp) + λ0 exp(t0 - t) Cradio
It is also preferable to calculate using the following formula:

By setting the weight λ in this way, the weighting coefficient (λ) applied to the communication state cost Cradio for the specified target gradually decreases (e.g., from 1) toward 0 from the time t0 when the occlusion is resolved, while the weighting coefficient (1-λ) applied to the position-related cost Cgeo and appearance cost Capp gradually increases (e.g., from 0) toward 1. As a result, the tracking cost Cost is calculated with greater emphasis on the position-related cost Cgeo and appearance cost Capp over time (conversely, with a reduced contribution of the communication state cost Cradio).

In this way, as time passes after the occlusion is resolved, the contributions of the position-related cost Cgeo and appearance cost Capp, which would be expected to be more accurate (in a situation where no occlusion has occurred), to the tracking cost are gradually increased, making it possible to determine a more accurate tracking cost.

Figure 6 is a schematic diagram for explaining yet another embodiment of the tracking target determination process in the tracking target determination and management unit 115.

In the embodiment shown in FIG. 6, the tracking target determination and management unit 115 calculates the position-related costs Cgeo_A and Cgeo_B, the appearance costs Capp_A and Capp_B, and the communication state costs Cradio_A and Cradio_B for each of target A ("person") and target B ("person") at time t0, determines the tracking costs Cost_A and Cost_B for each of the target A and B at time t0, determines the target related to the terminal 2 being tracked, and tracks the terminal 2.

Here, when the tracking target determination/management unit 115 determines that an occlusion occurred between the target A ("person") and the target B ("person") at a time point before (in the past) time point t0 using, for example, the contents of the determination result from the occlusion determination unit 114,
(a) First, based on past detection positions (information related to object position), for example, the detection positions at the most recent time t-N when no occlusion had occurred such that a detection position could not be determined, it is determined whether object A ("person") and object B ("person") related to this occlusion are in the foreground or background.

Next, when the occlusion is resolved, that is, at time t0, the tracking target determination and management unit 115
(b1) As shown in FIG. 6A, when it is determined that the time series data of the received signal power variable E_t (terminal communication state information) is affected by the shielding of communication radio waves, the communication state cost Cradio (Cradio_B) of a predetermined target determined to be in the foreground (target B ("person") in FIG. 6) is set to a larger predetermined value, and the communication state cost Cradio (Cradio_A) of a predetermined target determined to be in the background (target A ("person") in FIG. 6) is set to a smaller value or to zero,
(b2) On the other hand, as shown in FIG. 6 (B), when it is determined that the time series data of the received signal power variable E_t (terminal communication state information) is not affected by the obstruction of communication radio waves, the communication state cost Cradio (Cradio_B) of a specified target determined to be in the foreground (target B ("person") in FIG. 6) is set to a smaller value or to zero, and the communication state cost Cradio (Cradio_A) of a specified target determined to be in the background (target A ("person") in FIG. 6) is set to a larger specified value.

Here, the determination of the near side/far side in (a) above may be made on the assumption that the near side/far side positional relationship at the detection position at the latest time t-N is maintained thereafter, with the detection position closer to the camera 103 (base station 1) at the latest time t-N (subject B ("person") in FIG. 6) being the near side, and the detection position farther away (subject A ("person") in FIG. 6) being the far side. In addition, the determination of the presence or absence of the "effect of communication radio wave shielding" in (b1) and (b2) above can be made by checking whether or not there is a drop in radio wave reception power or a wireless connection disconnection, specifically, by checking whether or not there is a time period in which the value of the variable E_t is less than a predetermined threshold value in the time series data of the received signal power variable E_t.

As described above, according to the communication state cost setting process of the present embodiment,
(c1) If the terminal communication status information of the tracked terminal 2 indicates the “effect of communication radio wave shielding,” it is highly likely that a specific target corresponding to the terminal 2 is located at the back. Therefore, the communication status cost Cradio (Cradio_A) of the specific target determined to be at the back (target A ("person") in FIG. 6 ) can be set to a small value or to zero, making it easier to determine that “this specific target (target A ("person")) is the target corresponding to the tracked terminal 2.”

(c2) On the other hand, if there is no "effect of communication radio wave shielding" in the terminal communication state information of the tracked terminal 2, it is highly likely that the specified target corresponding to the terminal 2 is in the foreground, so the communication state cost Cradio (Cradio_B) of the specified target determined to be in the foreground (target B ("person") in FIG. 6) is set to a small value or to zero, making it easier to determine that "this specified target (target B ("person")) is the target corresponding to the tracked terminal 2."

Figure 7 is a schematic diagram for explaining yet another embodiment of the tracking target determination process in the tracking target determination and management unit 115.

In the embodiment shown in FIG. 7, the tracking target determination and management unit 115
(a) when an occlusion (intersection) occurs for a plurality of predetermined objects (in FIG. 7 , object A ("person") and object B ("person")) and is resolved, i.e., at time t0, continue tracking processing for each of the plurality of predetermined objects involved in the occlusion (object A ("person") and object B ("person")) as a candidate object related to the terminal 2 being tracked;
(b) In the course of the subsequent continuing tracking process, if the communication state cost Cradio or the tracking cost Cost becomes large enough to satisfy a predetermined condition (for example, exceeds a predetermined threshold value Cth), the tracking process is terminated as the target is not related to the terminal 2 being tracked.

In FIG. 7, of targets A ("person") and B ("person") which are candidates for targets related to the tracked terminal 2 at time t0, at time t0+P (P is a positive integer), target A ("person") is occluded by some object, causing the communication state cost Cradio_A to exceed the predetermined threshold Cth. Therefore, at this time t0+P, the tracked target determination and management unit 115 determines that "target A ("person") is not a target related to the tracked terminal 2," and terminates the tracking process of target A ("person") for that terminal 2.

As described above, according to the process of setting multiple candidates in this embodiment, by continuing the tracking process by temporarily setting all predetermined targets involved in the occlusion as candidates for the tracking target, it is possible to more reliably prevent the occurrence of an ID switch that would result in tracking of the wrong target. Furthermore, by removing inappropriate candidates based on the subsequent behavior of the communication state cost Cradio and the tracking cost Cost, it is possible to avoid a situation in which the number of candidates for tracking targets increases and the computational load of the tracking process becomes excessive.

Returning to the functional block diagram of FIG. 1, the tracking target determination and management unit 115 causes the tracking information storage unit 105 to store and manage the results of the identification and tracking process of each tracked terminal 2 (for example, time-series data in which the location and image features at each point in time are linked to each terminal ID of the terminal 2), which has been described in detail above with various embodiments and specific examples, and reports them to the communication control unit 116 continuously, periodically, or as required.

By using the identification and tracking results of this more appropriate terminal 2 (which has avoided being lost or having its ID switched), the communication control unit 116 is able to grasp the situation, including the current location and communication path, of the connected terminal 2, and can continue or immediately resume the communication connection with this terminal 2 even if a sudden drop in the reception power from the connected terminal 2 occurs.

As described above in detail, according to the present invention, when occlusion occurs, it is possible to determine whether or not there is a correspondence between the tracked terminal and a detected predetermined target, using the degree of communication correspondence between the communication state of the terminal and the estimated communication state of the predetermined target, without depending on whether or not the predetermined target is detected from environmental information (e.g., image data). As a result, even when occlusion occurs for the tracked terminal, it is possible to continue tracking the terminal 2 more appropriately.

In addition, the terminal tracking process of the present invention can be applied in a variety of situations and fields, such as solving the problem of communication failures caused by obstacles to the communication path in 5G, which is currently being introduced, notifying and warning automobiles equipped with terminals of approaching people or other vehicles, and even determining the relationship between the pages viewed by a user determined to be in correspondence with a certain terminal and the movement of that user, which can be used for marketing and management.

Furthermore, by popularizing information and communication systems including the terminal tracking device of the present invention, which can stably realize a huge number of high-capacity communication connections, it will be possible to widely deploy mobile and cell phone communication services that increase connections between people and empower communities. It is also believed that information and communication systems including the terminal tracking device of the present invention will be particularly effective in urban areas where the global population is growing. In other words, the present invention can contribute to the achievement of Goal 9 "Develop resilient infrastructure, promote sustainable industrialization and foster innovation" and Goal 11 "Make cities inclusive, safe, resilient and sustainable" of the Sustainable Development Goals (SDGs) led by the United Nations.

In the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical ideas and viewpoints of the present invention can be easily made by a person skilled in the art. The above description is merely an example and is not intended to restrict the present invention in any way. The present invention is limited only by the scope of the claims and their equivalents.

1. Base station (terminal tracking device, communication relay device)
101 Communication interface 102 Communication history information storage unit 103 Camera (sensor)
104 Environmental information accumulation unit 105 Tracking information storage unit 111 Terminal communication state determination unit 112 Target communication state determination unit 112a Position interpolation unit 113 Target detection information determination unit 114 Occlusion determination unit 115 Tracking target determination and management unit 115a Communication state cost calculation unit 115b Position-related cost calculation unit 115c Appearance cost calculation unit 116 Communication control unit 2 Terminal

Claims (11)

A terminal tracking device that detects a predetermined target including a terminal from environmental information related to the environment, the environmental information being acquired by a sensor that senses an environment in which a terminal may be present, and may include the predetermined target, and tracks the terminal,
a terminal communication state determination means for determining terminal communication state information, which is information relating to a communication state between the terminal and the communication means, based on information relating to communication acquired from a communication means for communicating with the terminal;
a target communication state determination means for determining, from the environmental information, target communication state information which is information related to a communication state to be realized between the predetermined target and the terminal if the predetermined target includes a terminal;
an object detection information determination means for determining object position related information, which is information related to the detected position of the predetermined object, and/or object appearance information, which is information related to the detected appearance of the predetermined object, from the environmental information;
a tracking target determination means for calculating a communication compatibility level, which is a level of correspondence between terminal communication state information determined for a tracked terminal and target communication state information determined for the predetermined target when occlusion occurs for the predetermined target, determining whether or not the tracked terminal and the predetermined target are in a corresponding relationship based on the communication compatibility level, and determining the predetermined target determined to be in a corresponding relationship as a target related to the tracked terminal;
The tracking target determination means determines whether or not the tracking terminal and the predetermined target are in a corresponding relationship based on a position-related matching degree, which is a degree of matching between the determined target position-related information and predicted target position-related information predicted from past target position-related information, and/or an appearance matching degree, which is a degree of matching between the determined target appearance information and past target appearance information.
A terminal tracking device comprising: A terminal tracking device that detects a predetermined target including a terminal from environmental information related to the environment, the environmental information being acquired by a sensor that senses an environment in which a terminal may be present, and may include the predetermined target, and tracks the terminal,
a terminal communication state determination means for determining terminal communication state information, which is information relating to a communication state between the terminal and the communication means, based on information relating to communication acquired from a communication means for communicating with the terminal;
a target communication state determination means for determining, from the environmental information, target communication state information which is information related to a communication state to be realized between the predetermined target and the terminal if the predetermined target includes a terminal;
an object detection information determination means for determining object position related information, which is information related to the detected position of the predetermined object, and/or object appearance information, which is information related to the detected appearance of the predetermined object, from the environmental information ;
a tracking target determination means for calculating a communication compatibility level, which is a level of correspondence between the terminal communication state information determined for the tracked terminal and the target communication state information determined for the predetermined target, when occlusion occurs for the predetermined target, determining whether or not the tracked terminal and the predetermined target are in a corresponding relationship based on the communication compatibility level, and determining the predetermined target determined to be in a corresponding relationship as a target related to the tracked terminal;
having
The tracking target determination means determines whether or not the tracked terminal and the specified target are in a corresponding relationship based on at least a position-related consistency degree, which is the degree to which the determined target position-related information matches predicted target position-related information predicted from past target position-related information, and/or an appearance consistency degree, which is the degree to which the determined target appearance information matches past target appearance information, during a time period in which no occlusion occurs for the specified target. The terminal tracking device according to claim 1 or 2, characterized in that the target communication state determination means determines the target communication state information by assuming that the specified target is present at a position interpolated using detection positions determined before and after the occlusion during a time period during which an occlusion occurs and the detection position of the specified target is not determined. The tracking target determination means calculates a tracking cost which is a weighted sum of a position-related cost which is a monotonically decreasing function of the position -related match degree and/or an appearance cost which is a monotonically decreasing function of the appearance match degree for the specified target, and a communication state cost which is a monotonically decreasing function of the communication compatibility degree, using a weighting coefficient which can take on a zero value, and determines a specified target for which the tracking cost is small enough to satisfy a specified condition as the target related to the tracked terminal. The terminal tracking device described in claim 4, characterized in that when an occlusion occurs for the specified target, in which the detection position cannot be determined, and is then resolved, the tracking target determination means calculates a tracking cost for the specified target by increasing the weighting coefficient applied to the communication state cost and decreasing or setting to zero the weighting coefficient applied to the position-related cost and/or the appearance cost. The terminal tracking device according to claim 5, characterized in that the tracking target determination means calculates a tracking cost by successively decreasing the weighting coefficient applied to the communication state cost for the specified target from a larger value over time from the point at which occlusion occurs and is resolved for the specified target, and successively increasing the weighting coefficient applied to the position-related cost and/or the appearance cost from a smaller value or zero . The terminal tracking device according to any one of claims 4 to 6, characterized in that the tracking target determination means calculates a tracking cost for the specified target by increasing the weighting coefficient applied to the communication state cost and decreasing or setting to zero the weighting coefficient applied to the position-related cost and/or the appearance cost for the specified target during the period from the time when occlusion occurs and is resolved for the specified target to the time when a specified time has elapsed. The tracking target determination means
When an occlusion occurs between the predetermined objects, it is determined whether each of the predetermined objects related to the occlusion is located in front or behind based on past information relating to the object positions;
A terminal tracking device as described in any one of claims 4 to 7, characterized in that when the occlusion is resolved, if it is determined in the terminal communication state information that there is an effect of blocking of communication radio waves, the communication state cost of a specific target determined to be in the foreground is set to a larger value, and the communication state cost of a specific target determined to be in the background is set to a smaller value or to zero, and if it is determined in the terminal communication state information that there is no effect of blocking of communication radio waves, the communication state cost of a specific target determined to be in the foreground is set to a smaller value or to zero , and the communication state cost of a specific target determined to be in the background is set to a larger value. The tracking target determination means
When an occlusion occurs and is resolved for a plurality of predetermined targets, continuing a tracking process for each of the plurality of predetermined targets as a candidate for a target related to the tracked terminal;
A terminal tracking device as described in any one of claims 4 to 8, characterized in that, during continued tracking processing, for a specified target whose tracking cost or communication state cost has become large enough to satisfy a specified condition, the tracking processing is terminated as the target is not related to the terminal being tracked. A terminal tracking program that causes a computer to function to detect a predetermined target including a terminal from environmental information related to the environment, the environmental information being acquired by a sensor that senses an environment in which a terminal may be present and may include the predetermined target, and to track the terminal,
a terminal communication state determination means for determining terminal communication state information, which is information relating to a communication state between the terminal and the communication means, based on information relating to communication acquired from a communication means for communicating with the terminal;
a target communication state determination means for determining, from the environmental information, target communication state information which is information related to a communication state to be realized between the predetermined target and the terminal if the predetermined target includes a terminal;
an object detection information determination means for determining object position related information, which is information related to the detected position of the predetermined object, and/or object appearance information, which is information related to the detected appearance of the predetermined object, from the environmental information;
a computer is made to function as a tracking target determination means for, when occlusion occurs with respect to the predetermined target, calculating a communication compatibility degree which is a degree of correspondence between terminal communication state information determined for the tracked terminal and target communication state information determined for the predetermined target, determining whether or not the tracked terminal and the predetermined target are in a corresponding relationship based on the communication compatibility degree, and determining the predetermined target determined to be in a corresponding relationship as a target related to the tracked terminal ;
The tracking target determination means determines whether or not the tracking terminal and the predetermined target are in a corresponding relationship based on a position-related matching degree, which is a degree of matching between the determined target position-related information and predicted target position-related information predicted from past target position-related information, and/or an appearance matching degree, which is a degree of matching between the determined target appearance information and past target appearance information.
A terminal tracking program comprising : A terminal tracking method implemented by a computer, which detects a predetermined target including a terminal from environmental information related to the environment, the environmental information being acquired by a sensor that senses an environment in which a terminal may be present, and which may include the predetermined target, and tracks the terminal, comprising:
determining terminal communication state information, which is information related to a communication state between the terminal and the communication means, based on information related to communication acquired from a communication means that communicates with the terminal;
determining, from the environment information, object communication state information which is information related to a communication state to be realized between the predetermined object and the terminal if the predetermined object includes a terminal;
determining object position related information, which is information related to a detected position of the predetermined object, and/or object appearance information, which is information related to a detected appearance of the predetermined object, from the environmental information;
a step of calculating a communication compatibility degree, which is a degree of correspondence between terminal communication state information determined for the tracked terminal and target communication state information determined for the predetermined target when occlusion occurs for the predetermined target, determining whether or not the tracked terminal and the predetermined target are in a correspondence relationship based on the communication compatibility degree, and determining the predetermined target determined to be in a correspondence relationship as a target related to the tracked terminal;
In the step of determining the target related to the tracked terminal, whether or not the tracked terminal and the predetermined target are in a corresponding relationship is determined based on a position-related matching degree, which is a degree of matching between the determined target position-related information and predicted target position-related information predicted from past target position-related information, and/or an appearance matching degree, which is a degree of matching between the determined target appearance information and past target appearance information.
A terminal tracking method comprising :

Images