JP7492756B2 - Radio wave source terminal position detection system and radio wave source terminal position detection method - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act March 1, 2022 Announced at the Proceedings of the 2022 National Conference of the Institute of Electrical Engineers of Japan

The present invention relates to a radio wave source terminal position detection system and a radio wave source terminal position detection method for detecting the position of a radio wave source terminal that transmits radio waves to perform communication in a specific space.

Conventionally, known technologies for controlling the wireless communication state in a specific space where a radio wave source terminal that transmits and receives radio waves is located include a technology that improves reception performance even when the reception environment of a mobile body changes by controlling the antenna directivity of a variable directivity antenna equipped on the mobile body based on the current position and topographical information of the mobile body (see, for example, Patent Document 1), and a technology that improves communication state by having a base station estimate the direction in which radio waves from a mobile station are coming and point the peak of the directivity of an array antenna in that direction (see, for example, Patent Document 2).

JP 2005-295365 A Japanese Patent Application Laid-Open No. 11-215049

In these technologies for controlling the wireless communication state in specific spaces, in order to actively control radio wave propagation in the specific space, it is necessary to identify the positions of wireless stations that can move within the specific space. For this reason, radio wave sensors that sense the direction of the radio wave source are used to identify the positions of these wireless stations. However, when using these radio wave sensors, it is only possible to roughly identify the two-dimensional position of the wireless station in the specific space, and there is a problem that the positioning accuracy is insufficient to actively control radio wave propagation in the specific space.

The present invention was made to address these problems, and aims to provide a radio wave source terminal position detection system and radio wave source terminal position detection method with high position accuracy.

The radio wave source terminal position detection system according to claim 1,
A radio wave source terminal position detection system that detects the position of a radio wave source terminal (e.g., a desktop computer 1, a laptop computer 2, or a mobile terminal 3, which are wireless LAN terminals) that transmit radio waves for communication in a specific space (e.g., a room R),
Imaging means (e.g., RGB-D1 camera 15, RGB-D2 camera 16, RGB-D3 camera 21) for capturing images of the specific space during operation ;
A three-dimensional map generating means for generating a three-dimensional map of the entire specific space in advance before the start of operation (for example, a part in which the processing device (A) generates a 3D map using a map generating module program before operation);
a three-dimensional map storage means capable of storing the three-dimensional map generated by the three-dimensional map generating means;
a three -dimensional position identification means for identifying the three-dimensional position of the radio wave source terminal in the specific space from the three-dimensional map stored in the three-dimensional map storage means and the image captured by the imaging means (for example, as shown in FIG. 10, a part in which a processing device (A) 25 or a processing device (B) 30 detects the terminal position using a terminal position detection module program during operation);
a three-dimensional position output means (e.g., a part that transmits terminal information in step S8 or step S13 in the terminal DB update process shown in FIG. 12) that outputs the three-dimensional position identified by the three-dimensional position identification means to a radio wave propagation control means (e.g., a processing device (C)) that can control radio wave propagation in the specific space;
Equipped with
It is characterized by the following.
According to this feature, by using a three-dimensional map and an image, the three-dimensional position of the radio wave source terminal can be specified with high accuracy.

The radio wave source terminal position detection system of claim 2 is the radio wave source terminal position detection system according to claim 1,
The 3D map generating means includes LiDAR-SLAM using a laser scanner (e.g., LiDAR24);
It is characterized by the following.
This feature makes it possible to improve the accuracy of the three-dimensional map.

The radio wave source terminal position detection system according to claim 3 is the radio wave source terminal position detection system according to claim 1 or 2,
The three-dimensional position identification means includes a deep learning function (e.g., a function based on a DNN module program stored in the processing device (A) 25 or the processing device (B) 30) that enables the radio wave source terminal to be detected with high accuracy from the image captured by the imaging means by providing learning data,
It is characterized by the following.
According to this feature, the radio wave source terminal can be detected with high accuracy.

The radio wave source terminal position detection method according to claim 4,
A method for detecting the position of a radio wave source terminal (e.g., a desktop computer 1, a laptop computer 2, or a mobile terminal 3, which are wireless LAN terminals) that transmit radio waves for communication in a specific space (e.g., a room R), comprising:
An imaging stage in which an image of the specific space is captured during operation (for example, a stage in which the RGB-D1 camera 15, the RGB-D2 camera 16, and the RGB-D3 camera 21 capture images at a frequency of 5 times per second during operation);
A three-dimensional map generation step of generating a three-dimensional map of the entire specific space in advance before the start of operation (for example, a step in which the processing device (A) generates a 3D map using a map generation module program before operation);
a 3D map storage step of storing the 3D map generated in the 3D map generation step;
A three-dimensional position determination step of determining the three-dimensional position of at least the radio wave source terminal in the specific space from the three-dimensional map stored in the three-dimensional map storage step and the image captured in the imaging step (for example, as shown in FIG. 10, a step in which the processing device (A) 25 or the processing device (B) 30 detects the terminal position using a terminal position detection module program during operation);
a three-dimensional position output step (e.g., a step of transmitting terminal information in step S8 or step S13 in the terminal DB update process shown in FIG. 12 ) of outputting the three-dimensional position identified in the three-dimensional position identification step to a radio wave propagation control means (e.g., a processing device (C)) capable of controlling radio wave propagation in the specific space;
including,
It is characterized by the following.
According to this feature, by using a three-dimensional map and an image, the three-dimensional position of the radio wave source terminal can be specified with high accuracy.

The radio wave source terminal position detection method according to claim 5 is the radio wave source terminal position detection method according to claim 4,
In the 3D map generation step, the 3D map is generated by LiDAR-SLAM using a laser scanner (e.g., LiDAR24);
It is characterized by the following.
This feature makes it possible to improve the accuracy of the three-dimensional map.

A radio wave source terminal position detection method according to claim 6 is the radio wave source terminal position detection method according to claim 4 or 5, further comprising the steps of:
The three-dimensional position identification step includes a step of detecting the radio wave source terminal from the image captured in the imaging step by using a deep learning function (e.g., a function by a DNN module program stored in the processing device (A) 25 or the processing device (B) 30) that has been machine-learned by providing learning data (e.g., a step in which the DNN module program detects and outputs the terminal relative position in FIG. 10 ).
It is characterized by the following.
According to this feature, the radio wave source terminal can be detected with high accuracy.

The present invention may have only the invention-specific matters described in the claims of the present invention, or may have the invention-specific matters described in the claims of the present invention as well as configurations other than the invention-specific matters.

1 is a diagram showing a room as an example of a specific space to which a radio wave propagation control system including a radio wave source terminal position detection system of the present invention is applied; 2 is a diagram showing a situation after a person carrying a mobile terminal has moved from the room shown in FIG. 1 . 1 is a block diagram showing a configuration of a radio wave propagation control system including a radio wave source terminal position detection system of the present invention. 1 is an external perspective view showing a radio wave sensor mounted on a mobile robot that constitutes a radio wave propagation control system including a radio wave source terminal position detection system of the present invention; FIG. 5 is a block diagram showing a configuration of the radio wave sensor shown in FIG. 4 . FIG. 1 is an external perspective view showing an intelligent reflecting surface (IRS) used in the present invention. FIG. 7 is a block diagram showing the configuration of the IRS shown in FIG. 6 . FIG. 11 is an explanatory diagram showing the processing operation of creating a 3D map before the processing device A is put into operation; 11 is an explanatory diagram showing the processing operation of creating position and attitude data of the RGB-D camera during initial setting before operation in the processing device (B). FIG. 1 is an explanatory diagram showing a flow of generating each data to be stored in a terminal database (DB) in a processing device (A) and a processing device (B). FIG. 1 is a diagram showing an example of a terminal DB in the processing device (A) and the processing device (C). FIG. FIG. 11 is a flow diagram showing a terminal DB update process in the processing device (A). FIG. 11 is a flow chart showing a propagation path control process in the processing device (C).

The following describes an embodiment of a radio wave propagation control system that includes a radio wave source terminal position detection system of the present invention.

Figure 1 shows a room R, which is a specific space to which a radio wave propagation control system including the radio wave source terminal position detection system of the present invention is applied, and Figure 3 is a block diagram showing the configuration of a radio wave propagation control system including the radio wave source terminal position detection system of the present invention.

As shown in FIG. 1, the specific space in this embodiment, room R, is rectangular and has a door for entering room R. Room R is divided into a large space where desk D1 is placed, on which desktop PC 1 and laptop PC 2, which are wireless LAN (Wi-Fi) terminals that serve as radio wave transmission sources, are placed, and a small space where desk D2 is placed, by a partition PT that is L-shaped when viewed from above and extends from the wall opposite the wall on which the door is located. The small space can be used for meetings, for example.

As shown in Figure 1, in room R, wireless LAN (Wi-Fi) access point terminals (AP) 11, 12 are placed on each wall on either side of the door, and an IRS (Intelligent Reflecting Surface) 13 is placed on the wall facing the entrance to the small space that will become the conference room, which can reflect radio waves in the frequency band used for wireless LAN (Wi-Fi) at different angles with high efficiency.

As shown in FIG. 1, these access point terminals (AP) 11, 12 are connected to an external network such as the Internet via a switching hub 2, and a desktop computer 1 or laptop computer 2 placed on desk D1, and even a mobile terminal 3 such as a mobile phone carried by a user who has entered room R, can communicate data with the external network such as the Internet via access point terminals (AP) 11, 12. Note that FIG. 1 omits devices such as routers for connecting to the external network, but it goes without saying that routers for connecting to the external network may also be provided.

These access point terminals (AP) 11, 12, like known wireless LAN (Wi-Fi) access point terminals, can perform two-way wireless communication with wireless LAN terminals using radio waves in the 2.4 GHz and 5 GHz bands, and in particular have a function for transmitting data on the session status (communication status) with the wireless LAN terminal to a management device C described below, and a three-dimensional beamforming function for concentrating radio waves of a specified frequency (channel) toward a three-dimensional position specified by instructions from the management device C.

These three-dimensional beamforming techniques can be implemented using known methods, such as three-dimensional beamforming technology using planar antenna arrays, which is being considered for use in 5G mobile phones.

In addition, RGB-D1 camera 15 and RGB-D2 camera 16, which can capture color images and measure the distance to a subject, are placed on the ceiling at two corners of room R, facing approximately diagonally across from room R as shown in FIG. 1. This allows RGB-D1 camera 15 to capture images of almost the entire area of room R except for the area close to RGB-D1 camera 15 in the small space of room R, and RGB-D2 camera 16 to capture images of and measure the area of the large space of the room except for the area close to RGB-D2 camera 16 and the area hidden by partition PT.

The RGB-D1 camera 15 and the RGB-D2 camera 16, as well as the RGB-D3 camera 21 mounted on the mobile robot described later, are all combinations of an RGB camera with a visible light color image sensor and a depth camera capable of measuring the distance to the subject (subject depth). These RGB cameras have sufficient resolution to enable detection of wireless LAN terminals in the room R, particularly small mobile terminals as subjects, by images, and any known RGB camera can be used as long as it can capture images at time intervals that allow image processing. In this embodiment, these RGB cameras are used that can be connected to a wired LAN via a LAN cable, taking into account connections to the processing device (A) 25 and processing device (B) 30 described later, but the present invention is not limited to this, and the communication interface (I/F) for transmitting image data from these RGB cameras can be appropriately selected taking into account the conditions of the communication interface (I/F) of the receiving side, etc.

In addition, in this embodiment, an infrared ToF camera is used as the depth camera because the size of the device is relatively small, the measurement accuracy is relatively high, and the time required for measurement is short, but the present invention is not limited to this. These depth cameras may be appropriately selected from known cameras in terms of compatibility with the image of the RGB camera used, the light conditions (light and dark) of the space to be imaged, the required measurement accuracy, measurement time, etc. Specifically, instead of a ToF camera, a depth camera of a method other than ToF may be used, for example, a camera that measures the subject depth from the parallax of multiple camera images, or a camera that measures the subject depth from the size and shape of the radiation pattern. In this embodiment, these depth cameras are used that can be connected to a wired LAN via a LAN cable, taking into consideration the connection with the processing device (A) 25 and the processing device (B) 30, etc., but the present invention is not limited to this, and the communication I/F for transmitting the data of the depth camera may be appropriately selected in consideration of the situation of the communication I/F of the receiving side.

In addition, in this embodiment, the RGB camera and the depth camera are individually connected to a wired LAN, but the present invention is not limited to this. For example, if the RGB camera and the depth camera are integrated into a camera, it goes without saying that the image data and depth data may be transmitted from a single communication I/F.

As shown in FIG. 1, the RGB-D1 camera 15 and the RGB-D2 camera 16 are connected to a processing device (B) 30 located in the corner of the room R via a LAN cable and a switching hub 1 via a wired LAN, and image data captured by the RGB-D1 camera 15 and the RGB-D2 camera 16 and distance data to the subject (subject depth data) are input to the processing device (B) 30. The RGB-D1 camera 15 and the RGB-D2 camera 16 capture and measure images five times per second and send the image data and distance data to the processing device (B) 30, but the number of times the images are captured can be determined appropriately taking into account the capabilities of the cameras used, the processing capabilities of the processing device (B) 30, the communication speed of the communication I/F, etc.

The processing device (B) 30 is a highly reliable hardware device with relatively excellent processing capabilities and low susceptibility to malfunctions. A well-known personal computer (PC) having a LAN communication interface can be suitably used as the processing device (B) 30. These personal computers (PCs) are used with various programs installed to provide the processing device (B) 30 with the functions required, but the present invention is not limited to this. The processing device (B) 30 can be any computer that can provide the functions of the processing device (B) 30 described below.

The processing device (B) 30 stores module programs for providing various functions as shown in FIG. 3, as well as various data used by these module programs. In FIG. 3, the thick dashed rectangular lines indicate module programs, and the thin dashed cylindrical lines indicate data.

The module programs that can be operated by the operating system (OS) of the processing device (B) 30 include an RGB-D1 and RGB-D2 position/attitude estimation module program for estimating the position and attitude of the RGB-D1 camera 15 and the RGB-D2 camera 16 at the time of initialization before operation, as shown in FIG. 9, an image processing module program for performing corrections, etc., on image data and depth data transmitted from the RGB-D1 camera 15 and the RGB-D2 camera 16 by calculations using image processing data consisting of determinants, etc., a deep learning (Deep Neural Network; DNN) module program that can identify the terminal and the type of terminal from the captured image by providing learning data, and a terminal position detection module program that detects the position of the wireless LAN terminal from data from the deep learning module program and image data from the image processing module program. The terminal position detection module programs are stored individually so as to correspond to each of the RGB-D1 camera 15 and the RGB-D2 camera 16.

In this embodiment, as described below, the processing device (B) 30 uses the 3D map generated by the processing device (A) 25 to estimate the position and orientation of the RGB-D1 camera 15 and the RGB-D2 camera 16. Therefore, in consideration of convenience when using the 3D map, a SLAM program identical to the SLAM program of the processing device (A) 25 that creates these 3D maps is also installed in the processing device (B) 30, and the self-position estimation function of the SLAM program is used to estimate the position and orientation of the RGB-D1 camera 15 and the RGB-D2 camera 16. However, the present invention is not limited to this, and the module programs for estimating the position and orientation of the RGB-D1 camera 15 and the RGB-D2 camera 16 may be unrelated to the SLAM program of the processing device (A) 25.

In addition, the data stored includes the position and orientation data of the RGB-D1 camera 15 and the RGB-D2 camera 16 estimated by the position and orientation estimation module program at the time of initialization before operation, as well as image processing data including determinants used by the image processing module program, and 3D map data generated by the processing device (A) 25 at the time of initialization before operation, as described below.

In addition, a processing device (C) 40 is located near the processing device (B) 30 and is connected to the processing device (B) 30 via a LAN cable and a switching HUB 2. The IRS 13 is locally connected to the processing device (C) 40, and the access point terminals (AP) 15 and 16 are also connected via a LAN cable and a switching HUB 2, so that the processing device (C) 40 can control the radio wave reflection direction of the locally connected IRS 13 and can also control three-dimensional beamforming of the access point terminals (AP) 15 and 16, and the processing device (C) 40 actively controls radio wave propagation in room R, which is a specific space.

The processing device (C) 40, like the processing device (B) 30, is a highly reliable piece of hardware that has relatively excellent processing capabilities and is unlikely to malfunction. A known personal computer (PC) having a LAN communication interface and an interface for locally connecting the IRS 13 can be suitably used. These personal computers (PCs) are used with various programs installed to provide the functions required by the processing device (C) 40. However, the present invention is not limited to this, and the processing device (C) 40 can be any computer that can provide the functions of the processing device (C) 40 described below.

The processing device (C) 40 stores module programs for providing various functions as shown in FIG. 3, as well as various data used by these module programs. Specifically, as module programs operable on the operation system (OS) of the processing device (C) 40, a propagation path calculation module program that calculates the propagation path between each access point terminal (AP) 11, 12 and the wireless LAN terminal in the room R, and an AP/IRS control module program that performs directional control of three-dimensional beamforming optimized for the propagation path calculated by the propagation path calculation module program and reflection angle control of the optimized IRS 13 are stored.

The data used by these module programs includes 3D map data generated by the processing device (A) 25 at the time of initialization before operation, a communication status database (DB) in which session information including the three-dimensional position coordinates of each access point terminal (AP) 11, 12 in room R, the local IP addresses, MAC addresses, radio frequencies (channels) used for transmission and reception, and communication levels of the wireless LAN terminals in room R with which each access point terminal (AP) 11, 12 is communicating, a terminal database (DB) (see FIG. 11) in which the three-dimensional positions of the wireless LAN terminals in room R estimated by the processing device (A) and information on the radio frequencies transmitted from the wireless LAN terminals are updated sequentially, and terminal correspondence table data in which the correspondence between the terminal IDs used in the terminal DB and the MAC addresses that serve as terminal identification information in the communication status database is stored.

In this embodiment, the three-dimensional position coordinates of each access point terminal (AP) 11, 12 are set by directly inputting them into the processing device (C) 40 at a pre-operational stage when the 3D map data is stored, but the present invention is not limited to this. The three-dimensional positions of each of these access point terminals (AP) 11, 12 may also be detected by the processing device (B) 30 or processing device (A) 25 and stored in the processing device (C) 40, in the same way as the position detection of the RGB-D1 camera 15 and the RGB-D2 camera 16.

Here, the IRS 13, which is locally connected to the processing device (C) 40, will be briefly explained using Figures 6 and 7. As shown in Figure 6, the IRS 13 is mainly composed of a rectangular box-shaped main body 132 that is attached to a wall, a radio wave reflector 131 that protrudes from the main body 132 in the forward direction of the wall surface, and a vertical drive unit 133 that is provided on the back of the radio wave reflector 131 and can change the vertical angle of the radio wave reflector 131.

The radio wave reflector 131 is capable of reflecting radio waves in the 2.4 GHz and 5 GHz bands used in wireless LANs with high efficiency, and an example of such a reflector is that disclosed in JP 2021-141359 A.

The connecting shaft connecting the vertical drive unit 133 and the main body 132 can be swung horizontally by the horizontal drive unit 134 built into the main body 132. Because the connecting shaft can be swung horizontally by the horizontal drive unit 134 and the vertical angle can be changed by the vertical drive unit 133, it is possible to change the angle of the radio wave reflector 131 up, down, left, and right. The angle of the radio wave reflector 131 is controlled by the drive control unit 135 built into the main body 132 in response to the reception of drive information including vertical and horizontal angle information, which is information transmitted from the processing device (C) 40, as shown in FIG. 7. This makes it possible to reflect radio waves from, for example, the access point terminal (AP12) toward the inside of the small space (see FIG. 2).

In addition, a mobile robot 20 capable of moving within room R is placed within room R. The mobile robot 20 is mounted on an automated guided vehicle (AGV) 23 for enabling movement, and is made of a material capable of storing a radio wave sensor 22 and capable of transmitting radio waves detected by the radio wave sensor 22 with high efficiency. A housing is mounted on the top of the housing in which a relatively small RGB-D3 camera 21 having the same functions as the RGB-D1 camera 15 and the RGB-D2 camera 16 and a laser scanner (hereinafter referred to as LiDAR) 24 for performing LiDAR-SLAM (Simultaneous Localization and Mapping) are arranged. A processing device (A) 25 is also mounted, which is connected to the RGB-D3 camera 21, radio wave sensor 22, automated guided vehicle (AGV) 23, and LiDAR 24 to create a three-dimensional map (hereinafter abbreviated as 3D map) of the room R and to detect and estimate the position of wireless LAN terminals based on data from the radio wave sensor 22 and the RGB-D3 camera 21.

The radio wave sensor 22 used in the mobile robot 20 of this embodiment will be briefly described with reference to Figures 4 and 5. The radio wave sensor 22 has an appearance as shown in Figure 4, and has a rectangular plate-shaped planar antenna array 221 for receiving radio waves in the 2.4 GHz and 5 GHz bands, which are frequency bands used in wireless LANs, a horizontal drive unit 222 that can rotate the planar antenna array 221 in the horizontal direction by rotating the connecting shaft to which the planar antenna array 221 is connected, and a vertical drive unit 223 that is provided between the connecting shaft and the back of the planar antenna array 221 and can rotate the planar antenna array 221 in the vertical direction around a rotation shaft not shown.

Inside the housing of the radio wave sensor 22 is a signal processing unit 224 to which RF signals from the individual antennas formed in the planar antenna array 221 are input. Based on the multiple RF signals from the individual antennas that make up the planar antenna array 221, the signal processing unit 224 identifies the direction of arrival of the radio waves, which is the direction of the radio wave source, the frequency of the arriving radio waves (measurement frequency), and the strength of the arriving radio waves (electric field strength), and outputs this together with data on the measurement time.

The signal processing unit 224 starts radio wave measurement (scanning) in response to an external measurement start command. This radio wave measurement (scanning) is performed by rotating the horizontal direction in which the planar antenna array 221 faces by a predetermined angle (for example, 10 degrees) at each rotation position of 10 degrees, scanning radio wave channels of the 2.4 GHz band and the 5 GHz band, and when one rotation is completed, the vertical angle (depression angle or elevation angle) is changed by the amount of the detectable vertical angle range (for example, an angle range of 10 degrees) that the planar antenna array 221 can detect, and at the changed vertical angle, the horizontal direction is rotated by a predetermined angle at each rotation to scan the radio wave channel for one rotation, and then the vertical angle is further changed by the amount of the detectable vertical angle range. In other words, one rotation of rotation measurement → vertical angle change → one rotation of rotation measurement → vertical angle change ... is repeated, and the radio wave measurement (scanning) ends when one rotation of rotation measurement is completed at an angle where the vertical angle corresponds to an elevation angle of 90 degrees.

Therefore, if the horizontal angle for scanning the radio channel is made fine (for example, 2 degrees, which is 180 divisions of 360), the directional accuracy (resolution) will be high, but the time required for radio wave measurement (scanning) will be very long, making it difficult to accommodate the movement of the wireless LAN terminal. Therefore, it is preferable to shorten the time required for radio wave measurement (scanning) as much as possible by setting the minimum necessary directional accuracy (resolution) in consideration of the characteristics of the planar antenna array 221 used. Specifically, in this embodiment, the specified horizontal angle is set to 10 degrees, which is 36 divisions of 360, and the detectable vertical angle range is also set to approximately 10 degrees, so that the time required for radio wave measurement (scanning) is approximately 60 seconds.

The horizontal rotation control and vertical angle control in these radio wave measurements (scans) are realized by the signal processing unit 224 outputting instructions to the horizontal drive unit 222 and the vertical drive unit 223. Furthermore, since these radio wave measurements (scans) must be performed when the mobile robot 20 is not moving, when the mobile robot 20 stops, it starts the radio wave measurement (scan) upon receiving a measurement start instruction sent from the processing device (A) 25 connected via the LAN, and sends a measurement end notification to the processing device (A) 25 when the radio wave measurement (scan) is completed.

Next, the automated guided vehicle (AGV) 23 constituting the mobile robot 20 will be described. Any known automated guided vehicle (AGV) 23 can be suitably used as long as it has multiple drive wheels and can move forward, backward, left and right and change direction. The automated guided vehicle (AGV) 23 has a drive mechanism such as a motor and a reducer, as well as a power source such as a battery for operating these drive mechanisms.

The automated guided vehicle (AGV) 23 is connected to the processing device (A) 25 via a USB, which is a specified interface (I/F), and moves according to instructions from the processing device (A) 25.

In this embodiment, an example is shown in which an automated guided vehicle (AGV) 23 is used, taking into consideration the size and weight of the load to be carried, but the present invention is not limited to this, and any object capable of moving within the room R, such as an automatic cleaning robot, may be used.

Next, the LiDAR 24 mounted on the mobile robot 20 will be described. As the LiDAR 24, a known laser scanner that can be used for SLAM can be suitably used, but if the LiDAR 24 is large, the size of the mobile robot 20 will also be large, and the area that the mobile robot 20 cannot enter will expand, so a small LiDAR that consumes little power is preferable.

As shown in FIG. 3, the LiDAR 24 is connected to the processing device (A) 25 via a LAN, and outputs information such as distance data to objects in each direction scanned by the laser to the processing device (A) 25. This information (scan data) is used by the processing device (A) 25 to create a 3D map.

As described above, the RGB-D3 camera 21 mounted on the mobile robot 20 is a combination of an RGB camera and a depth camera, similar to the RGB-D1 camera 15 and the RGB-D2 camera 16, and these depth cameras are infrared ToF cameras. The RGB camera and the depth camera are connected to the processing device (A) via a LAN cable and a switching HUB 3, as shown in FIG. 3.

The processing device (A) 25 mounted on the mobile robot 20, like the processing device (B) 30 and processing device (C) 40, is a highly reliable hardware that is relatively superior in processing capabilities and unlikely to break down due to vibrations or the like. A well-known personal computer (PC) having a LAN communication I/F or USB-I/F can be suitably used for the processing device (A) 25. Various programs for providing the functions required for the processing device (A) 25 are installed on these personal computers (PCs), but the present invention is not limited to this. The processing device (A) 25 can be any computer that can provide the functions of the processing device (A) 25 described below.

The processing device (A) 25 stores module programs for providing various functions as shown in FIG. 3, as well as various data used by these module programs.

The module programs that can be operated by the operation system (OS) of the processing device (A) 25 include an AGV guidance control module program having a navigation function for controlling the movement of the automatic guided vehicle (AGV) 23, a DB search engine module program for searching data stored in a terminal database (hereinafter abbreviated as terminal DB), an image processing module program for performing corrections, etc., on image data and depth data transmitted from the RGB-D3 camera 21 by calculations using image processing data consisting of determinants, etc., a deep learning (Deep Neural Network; DNN) module program that can identify the terminal and the type of terminal from the captured image by providing learning data, and a SLAM program. The SLAM program includes a map generation module program for generating a 3D map, a self-position estimation module program for estimating the position of the mobile robot 20, and a terminal position detection module program for detecting the position of a wireless LAN terminal from data from the deep learning module program and image data from the image processing module program.

The data used by these module programs includes 3D map data of room R generated by the map generation module program during initialization before operation, image processing data including determinants used by the image processing module program, and the terminal DB shown in Figure 11.

Here, the terminal DB of this embodiment will be briefly described with reference to FIG. 11. In the terminal DB of this embodiment, as shown in FIG. 11, the following data is stored in association with a terminal ID uniquely assigned to a wireless LAN terminal present in room R: detection time in an image, terminal position (three-dimensional), terminal type, measurement time by a radio wave sensor, measurement frequency, radio wave strength, number of simultaneous detections, and distance from the direction of arrival.

As described above, the measurement period by the radio wave sensor 22 does not match the period for detecting the terminal position by the image, and the measurement period by the radio wave sensor 22 is longer than the period for detecting the terminal position by the image. Therefore, as shown in FIG. 11, the data for detecting the terminal position by the image does not store data on the measurement time, measurement frequency, radio wave strength, number of simultaneous detections, and distance from the direction of arrival, while the data for measurement by the radio wave sensor 22 stores data on the measurement time, measurement frequency, radio wave strength, number of simultaneous detections, and distance from the direction of arrival, but does not store other data such as the detection time by the image, terminal position (three dimensions), and terminal type.

In the example shown in FIG. 11, three types of terminals are shown as examples: a desktop PC, a laptop PC, and a mobile phone. However, the present invention is not limited to these types, and these types may be set appropriately depending on the learning content of the DNN module program, etc.

The measurement frequency may be determined using a wireless LAN channel number or the like. As described above, in this embodiment, wireless LAN communication is performed using three-dimensional beamforming, so the same channel (frequency) may be used by different wireless LAN terminals.

As shown in Figs. 1 and 3, the processing device (A) 25 is connected to the processing device (B) 30 and the processing device (C) 40 for data communication via a specific low-power data communication modem that uses the 920 MHz band, which has a different frequency from the 2.4 GHz and 5 GHz bands used for wireless LANs and has a lower frequency than the frequencies used for these wireless LANs, making it more effective at getting around obstacles.

The reason for using wireless communication in the 920 MHz band for data communication with the processing device (B) 30 and the processing device (C) 40 is that if the processing device (B) 30 and the processing device (C) 40 were connected by wired communication cables, there is a risk that the communication cables would become tangled as the mobile robot 20 moves, causing problems for the mobile robot 20's movement, and the communication cables would get in the way of people's movement and there is a risk of people tripping over them, so wireless data communication is preferable, and the mobile robot 20 is equipped with a radio wave sensor 22 that detects radio waves used for wireless LANs with high sensitivity. If radio waves with a frequency close to those used for wireless LANs are used, the performance of the radio wave sensor 22 will decrease. Therefore, data communication (50 Kbps) is performed using a frequency in the 920 MHz band that is different from the radio waves used for wireless LANs detected by the radio wave sensor 22 and has high versatility around obstacles.

In this embodiment, a specific low-power data communication modem using the 920 MHz band is used as a data communication means between the processing device (A) 25 and the processing device (B) 30 and the processing device (C) 40, but the present invention is not limited to this. As described above, it is preferable that these data communication means do not impede the movement of the mobile robot 20, do not adversely affect the radio wave detection of the radio wave sensor 22 mounted on the mobile robot 20, and satisfy the conditions such as obtaining the required data communication speed and having high turnaround ability around obstacles. If such conditions are satisfied, wireless data communication using a frequency other than the above-mentioned 920 MHz band may be used. In addition, the wireless communication is not limited to radio waves, and optical data communication using light such as visible light and infrared light may be used. In addition, when infrared communication is used, there is a possibility that it may adversely affect the measurement of the infrared ToF camera used as the depth camera of the RGB-D1 camera 15, the RGB-D2 camera 16, and the RGB-D3 camera 21, so it is preferable to use a type of infrared light that does not adversely affect the infrared ToF camera. Also, instead of using infrared communication, the depth cameras provided in the RGB-D1 camera 15, RGB-D2 camera 16, and RGB-D3 camera 21 may be depth cameras that do not use infrared light.

In addition, the data communication means between the processing device (A) 25 and the processing device (B) 30 and the processing device (C) 40 are not always the same, and different communication means may be used before and after operation. In other words, as described below, before operation, 3D map data with a large data volume generated by the processing device (A) 25 is transmitted to the processing device (B) 30 and the processing device (C) 40, so before operation when the communication data volume is large, data communication connection with the processing device (B) 30 and the processing device (C) 40 may be made via a wired LAN, and during actual operation, data communication may be made via the above-mentioned 920 MHz band wireless or optical communication.

As described above, the radio wave propagation control system of the present invention is a system that identifies and outputs the three-dimensional position of a wireless LAN terminal in a specific space, room R, from radio waves and images. It is composed of a radio wave source terminal position detection system consisting of an RGB-D1 camera 15, an RGB-D2 camera 16, a processing device (B) 30, and a mobile robot 20, and a processing device (C) 40 that serves as radio wave propagation control means that controls each access point terminal (AP) 15, 16 and IRS 13 installed in room R based on the three-dimensional position of the wireless LAN terminal in room R output from the radio wave source terminal position detection system, thereby actively controlling the propagation of radio waves in room R.

Next, the processing operations of the radio wave propagation control system of this embodiment will be described with reference to Figures 8 to 13. The processing operations of the radio wave propagation control system are broadly divided into processing operations performed during initialization before operation and processing operations performed after operation.

Specific processing operations performed during initialization before operation include a processing operation performed by the processing device (A) 25 shown in FIG. 8, which generates and stores a 3D map and transmits the generated 3D map to the processing device (B) 30 and the processing device (C) 40, and a processing operation performed by the processing device (B) 30 shown in FIG. 9, which estimates the camera positions and orientations of the RGB-D1 camera 15 and the RGB-D2 camera 16 and stores the estimated camera positions and orientations as data.

The processing operation of FIG. 8 performed by the processing device (A) 25 will be explained. As described above, the RGB-D3 camera 21 mounted on the mobile robot 20 is connected to the processing device (A) 25. Image data captured by the RGB camera of the RGB-D3 camera 21 and depth data captured by the depth camera are output (transmitted) from the RGB-D3 camera 21, and these image data and depth data are processed using image processing data by the image processing module program stored in the processing device (A) 25. The image processing contents by these image processing module programs include nonlinear image conversion processing such as distortion correction of the camera lens, position correction and integration processing of image data and depth data, and the processed data that has undergone image processing is output to the map generation module program included in the SLAM program. The processed data includes, for example, the shooting time, the corrected RGB image + associated stacked depth image, or three-dimensional point cloud data, as shown in FIG. 10.

In addition, the map generation module program included in the SLAM program sequentially receives movement data output (transmitted) from an automated guided vehicle (AGV) 23 connected to the processing device (A) 25 and scan data output (transmitted) from the LiDAR 24 in accordance with the movement of the mobile robot 20.

The map generation module program then uses these various input data to generate a 3D map of room R. When generating the 3D map, the AGV guidance control module program outputs instructions to the automated guided vehicle (AGV) 23 to control the direction and distance of movement, thereby moving sequentially inside room R while avoiding obstacles in room R, thereby sequentially creating a 3D map of room R. Then, when the 3D map is completed and stored, the AGV guidance control module program ends the movement of the mobile robot 20.

The completed and stored 3D map data is then transmitted to processing device (B) 30 and processing device (C) 40, where it is stored in each processing device and used for the processing of camera position and attitude estimation in processing device (B) 30 and for propagation path calculation in processing device (C) 40, etc.

In this embodiment, the 3D map data is generated only before operation and is not updated during operation, but the present invention is not limited to this. Even during operation, some or all of the 3D map data may be updated at appropriate times, such as at preset intervals.

Next, the processing operation of FIG. 9 performed by the processing device (B) 30 will be described. It is assumed that the 3D map data transmitted from the processing device (A) 25 described above has already been stored in the processing device (B) 30. Therefore, the processing operation of FIG. 9 is a processing operation that is executed after the processing operation of FIG. 8 is completed.

As mentioned above, the RGB-D1 camera 15 and the RGB-D2 camera 16 are connected to the processing device (B) 30 via a wired LAN, and operation processing is performed for each of these cameras individually. Note that since the operation processing for the RGB-D1 camera 15 is the same as the processing operation for the RGB-D2 camera 16, only the operation processing for the RGB-D1 camera 15 will be explained, and the processing operation for the RGB-D2 camera 16 will be omitted. Note that the reference characters in parentheses in FIG. 9 indicate that the RGB-D2 camera 16 is also similar.

Image data captured by the RGB camera of the RGB-D1 camera 15 and depth data captured by the depth camera are sent to the processing device (B) 30 via HUB 1. The image data and depth data are then processed using image processing data by an image processing module program stored in the processing device (B) 30. The image processing content by these image processing module programs is the same as that of the RGB-D3 camera 21 described above, so a description thereof will be omitted here. The data processed by the image processing module program is output to the RGB-D1 position and orientation estimation module program.

The RGB-D1 position/attitude estimation module program uses the stored 3D map and processed data output from the image processing module program to estimate the three-dimensional position of the RGB-D1 camera 15 and the attitude, such as the direction (orientation) of the RGB-D1 camera 15, and stores the estimated three-dimensional position data and attitude data. This estimated three-dimensional position data and attitude data are used to detect the position of the wireless LAN terminal during operation, as shown in Figure 10.

Next, using FIG. 10, the flow of updating the terminal DB shown in FIG. 11 by detecting the terminal position by an image while the system is in operation will be described. Note that the terminal DB may also be updated by data input from the radio wave sensor 22 while the system is in operation, as shown in FIG. 12.

As shown in FIG. 10, there are two main cases in which the terminal DB is updated by detecting a wireless LAN terminal using a captured image, etc.: the terminal DB is updated by detecting a wireless LAN terminal by the processing device (A) 25, and the terminal DB is updated by detecting a wireless LAN terminal by the processing device (B) 30.

First, the flow when the terminal DB is updated by the detection of a wireless LAN terminal by the processing device (A) 25 will be explained. First, in the processing device (A) 25, as in the case described in FIG. 9, the image data and depth data output (transmitted) from the RGB-D3 camera 21 are processed by the image processing module program using the image processing data, and the processed data is output to the self-position estimation module program and the DNN module program. Note that this processed data includes, for example, the shooting time, the corrected RGB image + associated stacked depth image, or three-dimensional point cloud data, as shown in FIG. 10.

The self-position estimation module program that outputs these processed data uses the processed data, as well as 3D map data generated and stored before operation, movement data output (transmitted) from the automated guided vehicle (AGV) 23, and scan data output (transmitted) from the LiDAR 24, to estimate the position (self-position coordinates, attitude) of the mobile robot 20 within the room R.

The estimated self-location coordinates, attitude, and estimated time data are then output to the terminal device detection module program, which constitutes the terminal position detection module program.

Meanwhile, the DNN module program extracts various wireless LAN terminals from the above-mentioned processed data input, identifies the type of the terminal, and outputs the relative position of the extracted wireless LAN terminal to the terminal device detection module program. The DNN module program outputs data on the identified terminal type and data on the detection time to the terminal DB along with data on the terminal position (three-dimensional) described below.

Here, the DNN module program of this embodiment will be described. As the DNN module program, for example, a module program of a known deep learning model having a function of extracting objects of a specific shape from an image, such as M2Det, an object detection model as shown in the technical paper Zhao Qijie et al., "M2det: A single-shot object detector based on multi-level feature pyramid network", Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, can be suitably used.

Then, for each wireless LAN terminal whose type can be identified, machine learning is performed by providing these publicly known deep learning models with images of various scenes from multiple directions according to the usage status using RGB-D cameras actually used in operation, and the images and depth data (after image processing) captured by these RGB-D cameras, as well as data in which the position and type of the wireless LAN terminal to be identified on the images and depth data are labeled as learning data.

By performing machine learning in this way, the image captured by the RGB-D camera and the depth data (image processed) are input to the machine-learned DNN module program, and the position and type of the wireless LAN terminal in the image contained in the input image and depth data (image processed) are estimated and output.

In this embodiment, a DNN module program that uses image and depth data as described above is exemplified, but the present invention is not limited to this, and these DNN module programs may be DNN module programs that use three-dimensional point cloud data created from the image and depth data, rather than the image and depth data described above.

In addition, the terminal device detection module program also receives processed data from the image processing module program, in which the detection area of the terminal is set in the coordinate system of the RGB-D3 camera 21 by the detection area setting module program that constitutes the terminal position detection module program.

Then, the terminal device detection module program outputs the terminal position (three-dimensional) by, for example, converting the RGB-D3 origin coordinates into 3D map coordinates based on the self-position coordinates and attitude input as described above, the processed data with the detection area set, and the data on the relative position of the wireless LAN terminal. Note that in this embodiment, the data on the terminal detection area is not stored in the terminal DB, so the configuration is such that the data on the terminal detection area is not output, but the present invention is not limited to this, and the terminal device detection module program may output the data on the terminal detection area and store (store) it in the terminal DB.

Then, the terminal position (three-dimensional), terminal type, and detection time data output from the terminal device detection module program and the DNN module program as described above are stored in association with the terminal ID. The association with the terminal ID will be described later with reference to FIG. 12.

Next, we will explain the flow when the terminal DB is updated upon detection of a wireless LAN terminal by the processing device (B) 30. As shown in FIG. 10, the basic flow and processing operations are the same as those of the processing device (A) 25, but the difference is that the processing device (A) 25 uses images from the RGB-D3 camera 21 that moves as the mobile robot 20, so it is necessary to perform self-position estimation sequentially, whereas in the processing device (B) 30, the RGB-D1 camera 15 and the RGB-D2 camera 16 are fixed and their positions and orientations do not change, so the position and orientation data of the RGB-D1 camera 15 and the position and orientation data of the RGB-D2 camera 16 that were estimated and stored before operation are input to the terminal device detection module program instead of the self-position coordinates and orientation data. Therefore, an explanation of the processing operations that are the same as those of the processing device (A) 25 will be omitted, but the terminal position (three-dimensional), terminal type, and detection time data output from the terminal device detection module program and the DNN module program of the processing device (B) 30 are stored in association with the terminal ID.

Here, the specific processing contents for updating these terminal DBs in the processing device (A) 25, in particular the update data stored in the terminal DB as update data for which terminal ID, will be explained using the processing flow diagram of the terminal DB update processing shown in FIG. 12.

In the terminal DB update process executed by the processing device (A) 25, first, it is determined whether or not there is data input from the radio wave sensor 22 (step S1). If there is no data input from the radio wave sensor 22 (No in step S1), it is determined whether or not there is input of update data for the terminal DB, that is, whether or not there is input (reception) of each data of the terminal position (three-dimensional), terminal type, and detection time output from the terminal device detection module program and DNN module program of the processing device (A) 25 or processing device (B) 30 (step S2).

If no update data has been input to the terminal DB (No in step S2), the process returns to step S1. On the other hand, if update data has been input to the terminal DB (Yes in step S2), the DB search engine is used to identify the terminal ID of the wireless LAN terminal whose latest terminal location is closest to the terminal location in the update data (step S3).

Then, the latest terminal location of the terminal ID of the identified wireless LAN terminal is compared with the terminal location in the update data (step S4), and then it is determined whether the difference (change) in the terminal location is equal to or less than a preset threshold (step S5). Note that these thresholds are set to different values depending on the wireless LAN terminal, with the lowest value being set for a desktop PC, a larger value being set for a laptop PC than for a desktop PC, and the highest value being set for a mobile phone. Note that these thresholds may be set to appropriate values depending on the update period of the terminal DB, etc.

If the difference (change) in the terminal position is equal to or less than the threshold (Yes in step S5), the update data is stored (stored) in association with the terminal ID of the wireless LAN terminal identified in step S3 as the latest data for the terminal ID (step S6), and then the process proceeds to step S8.

On the other hand, if the difference (change) in the terminal position is not below the threshold (No in step S5), it is recognized as a newly detected wireless LAN terminal, and a terminal ID to be assigned to the wireless LAN terminal is identified from terminal IDs not yet stored in the terminal DB, and the terminal ID is stored in the terminal DB, and update data is stored in the terminal DB as the latest data for the terminal ID (step S7).

Then, proceed to step S8, and transmit terminal information including the terminal ID and update data to the processing device (C) 40, and then return to step S1. In this way, in step S8, the terminal information including the terminal ID and update data is transmitted to the processing device (C) 40, and the terminal ID and update data are updated and stored in the terminal DB stored in the processing device (C) 40, so that the same data is stored in the terminal DB of the processing device (C) 40 and the terminal DB of the processing device (A) 25.

On the other hand, if the answer is Yes in step S1, that is, if data has been input from the radio wave sensor 22, the terminal DB is searched using a DB search engine (step S10), and it is determined whether or not there is a wireless LAN terminal that corresponds to the direction of radio waves included in the input data from the radio wave sensor 22 (step S11).

If there is no wireless LAN terminal that corresponds to the radio wave arrival direction included in the input data from the radio wave sensor 22 (No in step S11), the process returns to step S1. On the other hand, if there is a wireless LAN terminal that corresponds to the radio wave arrival direction included in the input data from the radio wave sensor 22 (Yes in step S11), the measurement time, measurement frequency, radio wave intensity, number of simultaneous detections, and distance from the arrival direction, which are data from the radio wave sensor, are memorized (stored) as the latest data for the terminal ID of the corresponding wireless LAN terminal (step S12).

Then, the terminal ID of the corresponding wireless LAN terminal and the data of the measurement time, measurement frequency, radio wave strength, number of simultaneous detections, and distance from the direction of arrival stored in the terminal DB are transmitted as terminal information to the processing device (C) 40 (step S13), and the process returns to step S1. In this way, in step S13, the terminal ID and the data of the measurement time, measurement frequency, radio wave strength, number of simultaneous detections, and distance from the direction of arrival are transmitted as terminal information to the processing device (C) 40, and the terminal ID and each data are updated and stored in the terminal DB stored in the processing device (C) 40, so that the same data is stored in the terminal DB of the processing device (C) 40 and the terminal DB of the processing device (A) 25.

Next, the propagation path control process executed by the processing device (C) 40 to actively control the propagation path during operation, in particular the process of associating the device ID, which is the terminal identification information of the wireless LAN terminal used in the radio wave source terminal position detection system, with the MAC address, which is the terminal identification information used in each access point terminal (AP) 15, 16 that constitutes the radio wave propagation control system, will be described with reference to FIG. 13.

In the propagation path control process, first, a communication status DB update process is performed (step S101). In this update process, it is determined whether or not session information has been received from each access point terminal (AP) 11, 12, and if session information has been received, information such as the local IP address, MAC address, each radio frequency (channel) used for transmission and reception, and communication level of the wireless LAN terminal in room R with which each access point terminal (AP) 11, 12 is communicating is updated and stored in the communication status DB.

Next, it is determined whether terminal information has been received from the processing device (A) 25 (step S102). If terminal information has been received from the processing device (A) 25 (Yes in step S102), each data included in the received terminal information, such as the terminal ID, is stored in the terminal DB (step S103).

Then, it is determined whether the terminal ID included in the received terminal information is registered in the stored terminal correspondence table data, that is, whether the terminal information is of a wireless LAN terminal for which the terminal ID and MAC address have already been associated (step S104).

If the terminal ID is already registered in the terminal correspondence table data (Yes in step S104), it is further determined whether the received terminal information includes terminal location information, i.e., whether the terminal information is the terminal information sent by the processing in step S8 described above (step S105).

If the received terminal information does not include terminal location information (No in step S105), the process returns to step S101. On the other hand, if the received terminal information includes terminal location information (Yes in step S105), the data in the communication status DB is used to identify whether the access point terminal communicating with the MAC address registered in the terminal correspondence table data in correspondence with the terminal ID is access point terminal (AP) 11 or access point terminal (AP) 12 (step S106).

Next, the propagation path between the identified access point terminal and the terminal position included in the received terminal information (the latest terminal position stored in the terminal DB) is calculated using a propagation path calculation module program (step S107), and the contents of the access point (AP) control and IRS control of IRS13 corresponding to the calculated propagation path are identified (step S108). Specifically, as the access point (AP) control contents, when there is no obstacle such as a partition PT on the calculated propagation path, the control contents for performing three-dimensional beamforming that can increase the radio wave intensity toward the terminal position (three dimensions) included in the received terminal information are identified. Also, when there is an obstacle such as a partition PT on the calculated propagation path, the control contents for performing three-dimensional beamforming toward IRS13 are identified along with the control contents for using a reflection path utilizing the reflection of IRS13.

Then, by outputting the access point (AP) control contents and IRS control contents identified in step S108 to the AP/IRS control module program, control information corresponding to these identified control contents, such as information on the horizontal and vertical angles at which the three-dimensional beamforming radio waves are emitted and information on the horizontal and vertical angles at which the IRS 13 reflects the radio waves, is output (transmitted) to the access point terminal (AP) 11 or the access point terminal (AP) 12 and the IRS 13 (step S109), and then the process returns to step S101.

Also, if the answer is No in step S104, that is, if the terminal ID has not yet been registered in the terminal correspondence table data, it is further determined whether or not the received terminal information includes frequency information, that is, whether or not it is the terminal information transmitted by the processing of step S13 described above (step S111).

If the received terminal information does not include frequency information (No in step S111), the process returns to step S101. On the other hand, if the received terminal information includes frequency information (Yes in step S111), the communication status DB is searched to determine whether or not there is an access point terminal using the frequency included in the received terminal information (step S113).

If there is no access point terminal using the frequency included in the received terminal information (No in step S113), the process returns to step S101. On the other hand, if there is an access point terminal using the frequency included in the received terminal information (Yes in step S113), the process identifies the MAC address of the wireless LAN terminal with which the access point terminal is communicating at that frequency from the communication status DB, associates the identified MAC address with the terminal ID included in the received terminal information, and stores it in the terminal association table data (step S114), and then returns to step S101.

In this way, the processing device (C) 40 executes the propagation path control process shown in FIG. 13 based on the accurate three-dimensional position of the wireless LAN terminal in room R detected by performing SLAM using a 3D map or DNN extraction using an image in the radio wave source terminal position detection system. For example, as shown in FIG. 1, when an access point terminal (AP) 11 and a laptop computer (Laptop PC) 2 are communicating, three-dimensional beamforming is performed toward the three-dimensional position of the laptop computer (Laptop PC) 2 based on the control information transmitted in the above-mentioned step S109, thereby concentrating and increasing the radio wave strength on the propagation path between the access point terminal (AP) 11 and the laptop computer (Laptop PC) 2, and significantly reducing the radio wave strength in the space other than the propagation path.

Similarly, when the access point terminal (AP) 12 and the desktop personal computer (Desktop PC) 1 are communicating, three-dimensional beamforming is performed toward the three-dimensional position of the desktop personal computer (Desktop PC) 1 based on the control information transmitted in step S109 described above. This makes it possible to concentrate and increase the radio wave strength on the propagation path between the access point terminal (AP) 12 and the desktop personal computer (Desktop PC) 1 and to significantly reduce the radio wave strength in spaces other than the propagation path. As a result, as shown in FIG. 1, the access point terminal (AP) 11 and the access point terminal (AP) 12 can use radio waves of the same frequency (channel), making it possible to effectively utilize the radio wave resources in the specific space, room R.

In addition, even if the mobile terminal 3 shown in FIG. 1 moves along with the movement of the person carrying it, the three-dimensional position after the movement is identified, and appropriate three-dimensional beamforming can be performed toward the identified three-dimensional position after the movement, so it is possible to prevent a decrease in communication quality, such as a decrease in the communication speed of wireless communication, caused by such movements.

Furthermore, even if the mobile terminals 3 move into a small space surrounded by partitions PT as shown in FIG. 2, making communication between the access point terminal (AP) 12 and the mobile terminals 3 difficult due to the presence of the partitions PT on the propagation path, wireless LAN communication between the access point terminal (AP) 12 and the mobile terminals 3 is possible by controlling the three-dimensional beamforming of the access point terminal (AP) 12 and the radio wave reflection angle by the IRS 13 with the processing device (C) 40. Therefore, it is possible to reduce the space where communication is not possible in the specific space, room R.

Although the embodiments of the present invention have been described above with reference to the drawings, the specific configuration is not limited to these embodiments, and the present invention also includes modifications and additions that do not deviate from the gist of the present invention.

For example, in the above embodiment, room R, which is an indoor space, is given as an example of a specific space, but the present invention is not limited to this, and these specific spaces may be outdoor spaces. Needless to say, the shape and size of room R are not limited to those in the above embodiment, and the room may be circular or pentagonal, or may be a space that spans multiple floors. In other words, a specific space may not be a space with clearly defined boundaries like room R, but rather a space with unclear boundaries.

In addition, in the above embodiment, an example is shown in which there are two access point terminals (AP), but the present invention is not limited to this, and there may be one access point terminal (AP) or three or more access point terminals (AP). The number of access point terminals (AP) may be appropriately selected depending on the size of the specific space, the number of users, etc.

Similarly, in the above embodiment, a configuration in which only one IRS 13 is provided is exemplified, but the present invention is not limited to this, and a configuration in which multiple IRS 13 are provided may also be used.

In the above embodiment, the mobile robot 20 is provided with a radio wave sensor 22, which minimizes the number of radio wave sensors 22 while reducing the spatial area that cannot be detected by the radio wave sensor 22. However, the present invention is not limited to this. The radio wave sensor may be fixed together with the RGB-D1 camera 15 and the RGB-D2 camera 16, for example. When the radio wave sensor is fixed in a specific space in this way, a 3D map of the specific space may be created in advance, and the mobile robot 20 may not be used during operation.

In addition, in the above embodiment, the processing device (A) 25 and the processing device (B) 30 are separate, but the present invention is not limited to this. For example, if the mobile robot 20 and the processing device (B) 30 are connected by high-speed data communication to ensure sufficient communication capacity for transmitting radio sensor data, laser scan data, image data, and depth data, the processing device (A) 25 may not be provided on the mobile robot 20, and the functions of the processing device (A) 25 may be integrated into the processing device (B) 30.

In addition, in the above embodiment, an example is shown in which two cameras, an RGB-D1 camera 15 and an RGB-D2 camera 16, are placed in room R, which is a specific space, but the present invention is not limited to this, and the number and placement positions of these RGB-D cameras can be determined appropriately depending on the shape and size of the specific space and the layout of the objects to be placed.

In addition, in the above embodiment, the radio waves used for communication in a specific space are exemplified as radio waves in the 2.4 GHz band or 5 GHz band, which are radio waves for wireless LAN (Wi-Fi) communication, but the present invention is not limited to this. These radio waves may be radio waves in the 60 GHz band, which is expected to be used as the next standard for wireless LAN (Wi-Fi), or radio waves other than these wireless LAN (Wi-Fi) communications, such as radio waves in the 1.9 GHz band that have traditionally been used for communication with PHS terminals, or radio waves in the frequency band used for 5G communication for mobile phone communication.

In addition, in the above embodiment, the processing device (C) 40 is provided separately from the access point terminals (AP) 11 and 12, but the present invention is not limited to this. The functions of the processing device (C) 40 may be incorporated into either of the access point terminals (AP) 11 and 12, or the functions of the processing device (C) 40 may be incorporated into the processing device (B).

In addition, the above embodiment illustrates a LiDAR-SLAM configuration in which the LiDAR 24 is used as the SLAM, but the present invention is not limited to this, and the SLAM may be a SLAM that uses an RGB-D camera without using the LiDAR 24.

In the above embodiment, as shown in FIG. 3, the processing device (B) 30 is provided with a separate terminal position detection module program for the RGB-D1 camera 15 and a terminal position detection module program for the RGB-D2 camera 16, which eliminates the need to synchronize the image capture timing of the RGB-D1 camera 15 and the RGB-D2 camera 16, simplifying the system configuration while ensuring redundancy in the event of a malfunction. However, the present invention is not limited to this, and the terminal position detection module program for the RGB-D1 camera 15 and the terminal position detection module program for the RGB-D2 camera 16 may be combined into a single terminal position detection module program.

In addition, in the above embodiment, the mobile robot 20 moves freely within a specific space, the room R, while avoiding obstacles, etc., but the present invention is not limited to this. The movement control of these mobile robots 20 may be controlled in terms of the movement direction, orientation, etc., based on information such as the direction of arrival of radio waves detected by the radio wave sensor 22, or the terminal position detected by the terminal position detection module program.

In addition, in the above embodiment, as shown in FIG. 12, whether the wireless LAN terminals are the same or not is determined based on whether the magnitude of the change in the terminal's position is equal to or less than a predetermined threshold value. However, the present invention is not limited to this, and in addition to this, additional processing may be performed to confirm whether the determination of whether the terminals are the same or not is correct.

In the above embodiment, the control of radio wave propagation in room R, which is a specific space, is performed by controlling the access point terminals (AP) 11, 12 and IRS 13, which are the higher-level stations. However, the present invention is not limited to this. For example, if the desktop computer 1, laptop computer 2, or mobile terminal 3, which are wireless LAN terminals (radio wave source terminals), have a radio wave propagation control function such as beamforming, information such as the beamforming direction (coordinates) can be transmitted to the wireless LAN terminal via the access point terminals (AP) 11, 12, so that the radio wave propagation in room R, which is a specific space, can be controlled by both the access point terminals (AP) 11, 12, which are the higher-level stations, and the wireless LAN terminal.

1. Desktop PC
2. Laptop PC
3. Mobile devices (mobile phones)
11, 12 Access point terminal (AP)
13. IRS
15 RGB-D1 camera 16 RGB-D2 camera 20 Mobile robot 21 RGB-D3 camera 22 Radio wave sensor 23 Automated guided vehicle (AGV)
24 Laser scanner (LiDAR)
25 Processing device (A)
30 Processing device (B)
40 Processing device (C)

Claims (6)

A radio wave source terminal position detection system for detecting a position in a specific space of a radio wave source terminal that transmits radio waves for communication, comprising:
An imaging means for capturing an image of the specific space during operation ;
A three-dimensional map generating means for generating a three-dimensional map of the entire specific space before the start of operation;
a three-dimensional map storage means capable of storing the three-dimensional map generated by the three-dimensional map generating means;
a three-dimensional position specifying means for specifying a three-dimensional position of the radio wave source terminal in the specific space based on the three-dimensional map stored in the three-dimensional map storage means and an image captured by the imaging means;
a three-dimensional position output means for outputting the three-dimensional position identified by the three-dimensional position identification means to a radio wave propagation control means capable of controlling radio wave propagation in the specific space;
Equipped with
A radio wave source terminal location detection system comprising: The three-dimensional map generating means includes LiDAR-SLAM using a laser scanner.
2. The radio wave source terminal position detection system according to claim 1. The three-dimensional position identification means includes a deep learning function that can detect the radio wave source terminal with high accuracy from the image captured by the imaging means by providing learning data,
3. The radio wave source terminal position detection system according to claim 1 or 2. A method for detecting the position of a radio wave source terminal that transmits radio waves for communication in a specific space, comprising:
An imaging stage of capturing an image of the specific space during operation ;
A 3D map generating step of generating a 3D map of the entire specific space before the start of operation;
a 3D map storage step of storing the 3D map generated in the 3D map generation step;
a three-dimensional position specifying step of specifying a three-dimensional position of at least the radio wave source terminal in the specific space from the three-dimensional map stored in the three-dimensional map storing step and the image captured in the imaging step;
a three-dimensional position output step of outputting the three-dimensional position identified in the three-dimensional position identification step to a radio wave propagation control means capable of controlling radio wave propagation in the specific space;
including,
A radio wave source terminal position detection method comprising: In the 3D map generation step, the 3D map is generated by LiDAR-SLAM using a laser scanner.
5. The method for detecting a radio wave source terminal position according to claim 4. The three-dimensional position identification step includes a step of detecting the radio wave source terminal from the image captured in the imaging step by a deep learning machine function that has been machine-learned by providing learning data.
6. The method for detecting a radio wave source terminal position according to claim 4 or 5.

Images