JP2015219745A - Base station device, video information processing device, three-dimensional map generation system, and three-dimensional map generation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make the time required for generating a three-dimensional map shorter than before, when using an SFM system.SOLUTION: A position estimation part 206 estimates the position of a terminal device by using a parameter outputted from a radio reception part 204. An additional information generation part 208 outputs image data with positional information in which the positional information of the terminal device to which image data is transmitted is coupled to the image data to an image interface part 209. The image interface part 209 transmits the image data with positional information to a video information processing device 300. The video information processing device 300 generates the three-dimensional map by using the image data with positional information. The position of the terminal device has already been estimated by each base station device 200, so that it is unnecessary that the video information processing device 300 performs processing for calculating a camera position (the position of the terminal device) by using a feature point.

Description

  The present invention relates to a base station apparatus, a video information processing apparatus, a three-dimensional map generation system, and a three-dimensional map generation method related to generation of a three-dimensional map.

  An SFM (Structure from Motion) method is known as one of methods for generating a three-dimensional map from a captured image. In the SFM method, a plurality of images taken from different positions by a camera (terminal device) are analyzed to extract characteristic portions (feature points) of an object, and the correspondence between feature points between images is used. Thus, the three-dimensional position and the camera position of each feature point of the object are calculated (see Patent Document 1).

JP 2009-237845 A

"Experimental verification of indoor and outdoor radio station location estimation using array antenna" IEICE Transactions Vol. J90-B No. 9 pp. 784-796

  However, since the conventional SFM method for generating a three-dimensional map requires calculation of the camera position using image analysis, there is a problem that it takes a considerable time to generate the three-dimensional map.

  An object of the present invention is to provide a base station device, a video information processing device, a three-dimensional map generation system, and a three-dimensional map generation that can shorten the time required to generate a three-dimensional map when using the SFM method. Is to provide a method.

  A base station apparatus according to the present invention is a base station apparatus that performs wireless communication with one or a plurality of terminal apparatuses existing in its own cell and is connected to a video information processing apparatus that generates a three-dimensional map, An image data acquisition unit that acquires image data from a terminal device, a position estimation unit that estimates the position of the terminal device, and with position information obtained by combining the image data with the position information of the terminal device that transmitted the image data An additional information generation unit that generates image data and an image interface unit that transmits the image data with position information to the video information processing apparatus are employed.

  The video information processing apparatus according to the present invention is a video information processing apparatus that is connected to a plurality of base station apparatuses that perform wireless communication with one or a plurality of terminal apparatuses existing in the own cell, and is an image captured by the terminal apparatus BS interface unit that receives image data with position information obtained by combining the position information of the terminal device that transmitted the image data with the data from each base station device, and a plurality of image data taken from different positions A feature point detection unit that analyzes and detects a feature point, and a three-dimensional map generation unit that generates a three-dimensional map using the position information of the terminal device and the three-dimensional position of the feature point are adopted. .

  A three-dimensional map generation system according to the present invention includes a plurality of terminal devices that convert digital images of images taken by a camera, and a plurality of bases that each perform wireless communication with one or a plurality of the terminal devices existing in the own cell. A three-dimensional map generation system comprising a station apparatus and a video information processing apparatus that is connected to the plurality of base station apparatuses and generates a three-dimensional map, wherein each base station apparatus receives image data from the terminal apparatus With position information obtained by combining the position information of the terminal device that transmitted the image data to the image data, the position estimation unit that estimates the position of the terminal device that transmitted the image data, An additional information generation unit that generates image data; and an image interface unit that transmits the image data with position information to the video information processing apparatus, and the video information The physical device includes a BS interface unit that receives the image data with position information from the base station device, a feature point detection unit that analyzes a plurality of pieces of image data taken from different positions, and detects a feature point; And a three-dimensional map generation unit that generates a three-dimensional map using the position information of the terminal device and the three-dimensional position of the feature point.

  A three-dimensional map generation method according to the present invention includes a plurality of terminal devices that convert an image captured by a camera into digital data, and a plurality of bases that each perform wireless communication with one or a plurality of the terminal devices existing in the own cell. A three-dimensional map generation method of a three-dimensional map generation system comprising a station apparatus and a video information processing apparatus that is connected to the plurality of base station apparatuses and generates a three-dimensional map, wherein each of the base station apparatuses is Image data with position information obtained by receiving image data from the terminal apparatus, estimating a position of the terminal apparatus that transmitted the image data, and combining position information of the terminal apparatus that transmitted the image data with the image data. Generating and transmitting the image data with position information to the video information processing apparatus, the video information processing apparatus receiving the image data with position information from the base station apparatus, Detecting a feature point by analyzing a plurality image data taken from different positions to have, to generate a three-dimensional map using the three-dimensional position of the position information and the feature points of the terminal device.

  According to the present invention, since the calculation process of the camera position is not required in the process of generating the three-dimensional map, the time until the three-dimensional map is generated can be shortened compared to the conventional case.

The figure which shows the structure of the three-dimensional map production | generation system which concerns on Embodiment 1 of this invention. The block diagram which shows the structure of the base station apparatus which concerns on Embodiment 1 of this invention. 1 is a block diagram showing a configuration of a video information processing apparatus according to Embodiment 1 of the present invention. The sequence diagram which shows the flow of a process of Embodiment 1 of this invention The block diagram which shows the structure of the base station apparatus which concerns on Embodiment 2 of this invention. The sequence diagram which shows the flow of the process of Embodiment 2 of this invention

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate.

(Embodiment 1)
<System configuration>
First, the configuration of the three-dimensional map generation system according to Embodiment 1 of the present invention will be described with reference to FIG.

  The three-dimensional map generation system mainly includes a terminal device 100, a base station device 200, and a video information processing device 300. Base station apparatus 200 is wirelessly connected to one or a plurality of terminal apparatuses 100 existing in its own cell. Video information processing apparatus 300 is connected to a plurality of base station apparatuses 200 by wire or wirelessly.

  Each terminal device 100 receives a signal from the base station device 200 of the cell to which the terminal device 100 belongs, and transmits a signal to the base station device 200. Each terminal apparatus 100 is a smartphone with a camera or the like, and converts an image captured by the camera into digital data and transmits the digital data to the base station apparatus 200.

  Each base station apparatus 200 transmits a signal to each terminal apparatus 100 existing in the own cell and receives a signal from the terminal apparatus 100. Further, each base station apparatus 200 estimates the position of each terminal apparatus 100 existing in its own cell, and sends the image data received from each terminal apparatus 100 to the video information processing apparatus 300 together with the position information of the terminal apparatus 100. Send.

  The video information processing device 300 generates a three-dimensional map by the SFM method using the image data transmitted from each base station device 200 and the position information of the terminal device 100.

<Configuration of base station device>
Next, the internal configuration of base station apparatus 200 according to the present embodiment will be described with reference to FIG. As illustrated in FIG. 2, the base station apparatus 200 includes a network interface unit 201, a MAC unit 202, a wireless transmission unit 203, a wireless reception unit 204, an image data acquisition unit 205, a position estimation unit 206, and a storage. The unit 207, the additional information generation unit 208, and the image interface unit 209 are mainly configured.

  The network interface unit 201 outputs a signal addressed to the terminal device 100 transmitted from a radio network controller (RNC: Radio Network Controller) (not shown) to the MAC unit 202. Further, the network interface unit 201 transmits the signal output from the MAC unit 202 to the radio network control station.

  The MAC unit 202 performs predetermined MAC processing such as adding header information to the signal output from the network interface unit 201, and outputs the signal after the MAC processing to the wireless transmission unit 203. The MAC unit 202 performs predetermined MAC processing such as removing header information on the signal output from the wireless reception unit 204 and outputs the signal after MAC processing to the network interface unit 201 and the image data acquisition unit 205. .

  The wireless transmission unit 203 performs wireless processing such as modulation, amplification, and up-conversion on the signal output from the MAC unit 202, and wirelessly transmits a radio frequency signal (radio wave) to each terminal device 100 via an antenna. Send.

  The radio reception unit 204 performs radio processing such as down-conversion, amplification, and demodulation on the radio frequency signal (radio wave) received from each terminal device 100 from the antenna 100 and outputs the signal to the MAC unit 202. Radio reception section 204 calculates a parameter necessary for position estimation from the received signal, and outputs the parameter to position estimation section 206.

  The image data acquisition unit 205 analyzes whether or not the reception signal output from the MAC unit 202 includes image data, and outputs the image data to the additional information generation unit 208.

  The position estimation unit 206 estimates the position of the terminal device 100 using the parameters output from the wireless reception unit 204, and stores the position information of the terminal device 100, which is the estimation result, in the storage unit 207. As the position estimation method in the position estimation unit 206, an existing one such as “4. Radio station position estimation by array antenna” described in Non-Patent Document 1 is used.

  The storage unit 207 is a RAM (Random Access Memory), and temporarily stores position information of the terminal device 100.

  When the image data is input from the image data acquisition unit 205, the additional information generation unit 208 acquires the position information of the terminal device 100 that has transmitted the image data from the storage unit 207 and combines it with the image data. Then, the additional information generating unit 208 outputs the image data with position information obtained by combining the position information to the image data to the image interface unit 209.

  The image interface unit 209 transmits the image data with position information output from the additional information generation unit 208 to the video information processing apparatus 300.

<Configuration of video information processing device>
Next, the internal configuration of the video information processing apparatus 300 according to the present embodiment will be described with reference to FIG. As shown in FIG. 3, the video information processing apparatus 300 includes a BS interface unit 301, a feature point detection unit 302, a frame stitch processing unit 303, a bundle adjustment processing unit 304, and a 3D map generation unit 305. It is mainly composed.

  The BS interface unit 301 receives the image data with position information transmitted from each base station apparatus 200 and outputs it to the feature point detection unit 302.

  The feature point detection unit 302 detects feature points by analyzing a plurality of pieces of image data taken from different positions. In the present embodiment, since each base station apparatus 200 has already estimated the position of the terminal apparatus 100, the feature point detection unit 302 uses a detected feature point to perform a camera using the detected feature point. There is no need to calculate the position (the position of the terminal device 100).

  The frame stitch processing unit 303 estimates the three-dimensional position of the feature point of each image detected by the feature point detection unit 302 using the position information of the terminal device 100, and sets it to a plurality of feature point positions in a plurality of image data. Based on this, frame stitch processing (Frame Stitch) is performed in which image frames are joined to generate data reflecting the three-dimensional position of each feature point.

  The bundle adjustment processing unit 304 sets the three-dimensional positions of feature points included in a plurality of pieces of image data captured from different positions for each feature point whose three-dimensional position is estimated by the frame stitch processing unit 303 as one position. Bundle adjustment processing (Bundle Adjustment) is performed for obtaining the final three-dimensional position of the feature points by converging to.

  The three-dimensional map generation unit 305 generates a three-dimensional map using the position information of the terminal device 100 and the three-dimensional position of the feature point obtained by the bundle adjustment processing unit 304.

<Description of sequence>
Next, the processing flow of the present embodiment will be described with reference to FIG. In FIG. 4, it is assumed that the terminal device 100 is in a standby state in the initial stage.

  In the standby state, terminal apparatus 100 repeats transmission / reception of a radio signal including a reference signal with base station apparatus 200 (ST401, ST402).

  For the received signal from terminal apparatus 100, base station apparatus 200 estimates the position of terminal apparatus 100 using, for example, a reference signal included in the received signal (ST403), and stores position information indicating the estimated position (ST403). ST404).

  Next, it is assumed that terminal apparatus 100 transmits image data captured by the camera to base station apparatus 200 (ST405). Receiving the image data, base station apparatus 200 combines position information with the image data (ST406), and transmits image data with position information to video information processing apparatus 300 (ST407).

  Video information processing apparatus 300 generates a three-dimensional map using position information and image data of terminal apparatus 100 (ST408).

<Effect>
As described above, according to the present embodiment, the base station apparatus 200 estimates the position of the terminal apparatus 100 that captures an image, and transmits the position information of the terminal apparatus 100 together with the image data to the video information processing apparatus 300. The video information processing apparatus 300 does not need to calculate the camera position (position of the terminal device 100) in the process of generating the three-dimensional map, and can shorten the time until generating the three-dimensional map.

  In the present embodiment, the case where base station apparatus 200 estimates the position of terminal apparatus 100 using a radio signal that does not include image data has been described. However, the present invention is not limited to this, and base station apparatus 200 may You may estimate the position of the terminal device 100 using the radio signal at the time of receiving image data.

  Further, in the present embodiment, the case where the base station apparatus 200 estimates the position of the terminal apparatus 100 before receiving the image data has been described, but the present invention is not limited to this, and the base station apparatus 200 receives the image data. The position of the terminal apparatus 100 may be estimated using the received radio signal, or the position of the terminal apparatus 100 using the radio signal received within a predetermined time before and after the base station apparatus 200 receives the image data. May be estimated.

(Embodiment 2)
In Embodiment 1, the case has been described in which each base station apparatus 200 estimates the position of the terminal apparatus 100 existing in the own cell from only the signal received in the own station. On the other hand, in the second embodiment, each base station apparatus 200A uses the signal received at its own station and the signal received at another base station apparatus 200B for the position of the terminal apparatus 100 existing in its own cell. The case of estimation will be described. The system configuration and the video information processing apparatus according to the second embodiment are the same as those described in the first embodiment.

<Configuration of base station device>
Next, the internal configuration of base station apparatus 200A according to the present embodiment will be described with reference to FIG. In the base station apparatus 200A shown in FIG. 5, the same components as those in the base station apparatus 200 shown in FIG. The base station apparatus 200A illustrated in FIG. 5 is different from the base station apparatus 200 illustrated in FIG. 2 in the functions of the network interface unit 201A and the position estimation unit 206A from those of the network interface unit 201 and the position estimation unit 206. .

  The network interface unit 201A has all the functions of the network interface unit 201. Furthermore, the network interface unit 201A inputs parameters necessary for position estimation calculated in the other base station apparatus 200B via the radio network control station, and outputs them to the position estimation unit 206A.

  The position estimation unit 206A uses the parameters output from the radio reception unit 204 and the parameters necessary for position estimation calculated in the other base station device 200B input from the network interface unit 201A, to determine the position of the terminal device 100. And the location information of the terminal device 100, which is the estimation result, is stored in the storage unit 207. As the position estimation method in the position estimation unit 206A, existing methods such as “2.1 Position estimation by RSS” and “2.2 Position estimation by AOA” described in Non-Patent Document 1 are used. Further, the position estimation unit 206A outputs the parameters necessary for position estimation input from the wireless reception unit 204 to the network interface unit 201A.

<Description of sequence>
Next, the processing flow of the present embodiment will be described with reference to FIG. In FIG. 6, it is assumed that the terminal device 100 is in a standby state in the initial stage.

  In the standby state, terminal apparatus 100 repeats transmission / reception of a radio signal including a reference signal with base station apparatus 200A (ST601, ST602). A signal transmitted from terminal apparatus 100 to base station apparatus 200A is also received by other base station apparatus 200B.

  The other base station apparatus 200B calculates a parameter necessary for position estimation for the received signal from the terminal apparatus 100 using, for example, a reference signal included in the received signal (ST603), and transmits the parameter to the base station apparatus 200A. (ST604).

  Base station apparatus 200A estimates the position of terminal apparatus 100 using the received signal from terminal apparatus 100 and the parameters necessary for position estimation received from other base station apparatus 200B (ST605), and the estimated position Is stored (ST606).

  Next, it is assumed that terminal apparatus 100 transmits image data captured by the camera to base station apparatus 200A (ST607). Receiving the image data, base station apparatus 200A combines position information with the image data (ST608), and transmits image data with position information to video information processing apparatus 300 (ST609).

  Video information processing apparatus 300 generates a three-dimensional map using position information and image data of terminal apparatus 100 (ST610).

<Effect>
Thus, according to the present embodiment, as in the first embodiment, the base station apparatus 200A estimates the position of the terminal apparatus 100 that captures an image, and the position information of the terminal apparatus 100 together with the image data is displayed as video information. Since it is transmitted to the processing device 300, the video information processing device 300 does not need to calculate the camera position (the position of the terminal device 100) in the process of generating the three-dimensional map, so that the time until the three-dimensional map is generated is conventional. Can be shortened.

  In the present embodiment, a case has been described in which base station apparatus 200A inputs parameters necessary for position estimation calculated in other base station apparatus 200B via a radio network control station. However, the present invention is not limited to this, and the base station apparatus 200A may directly input parameters necessary for position estimation calculated in the other base station apparatus 200B from the other base station apparatus 200B.

  Further, in the present embodiment, a case has been described in which another base station apparatus 200B periodically calculates parameters necessary for position estimation. However, the present invention is not limited to this, and the base station apparatus 200B When there is a request from the apparatus 200A, parameters necessary for position estimation may be calculated.

  Further, in the present embodiment, the case where only one base station apparatus 200B exists as another base station apparatus for base station apparatus 200A has been described, but the present invention is not limited to this, and base station apparatus 200A includes a plurality of base station apparatuses 200A. Parameters necessary for position estimation calculated in the base station apparatus may be input from a plurality of base station apparatuses.

  Moreover, although this Embodiment demonstrated the case where 200 A of base station apparatuses estimate the position of the terminal device 100 using the parameter required for position estimation input from other base station apparatuses 200B, this invention is However, the present invention is not limited thereto, and the base station apparatus 200A may input the position information of the terminal apparatus 100 calculated in the other base station apparatus 200B as a parameter necessary for position estimation from the other base station apparatus 200B. The apparatus 200A may combine the position information with the image data and transmit it to the video information processing apparatus 300.

  The present invention is suitable for use in generating a three-dimensional map from a captured image.

DESCRIPTION OF SYMBOLS 100 Terminal device 200, 200A, 200B Base station apparatus 201, 201A Network interface part 202 MAC part 203 Wireless transmission part 204 Wireless reception part 205 Image data acquisition part 206, 206A Position estimation part 207 Storage part 208 Additional information generation part 209 Image interface Unit 300 video information processing apparatus 301 BS interface unit 302 feature point detection unit 303 frame stitch processing unit 304 bundle adjustment processing unit 305 3D map generation unit

Claims (6)

A base station apparatus that performs wireless communication with one or a plurality of terminal apparatuses existing in its own cell and is connected to a video information processing apparatus that generates a three-dimensional map,
An image data acquisition unit for acquiring image data from the terminal device;
A position estimation unit that estimates the position of the terminal device;
An additional information generating unit that generates image data with position information obtained by combining the image data with the position information of the terminal device that transmitted the image data;
An image interface unit for transmitting the image data with position information to the video information processing apparatus;
A base station apparatus comprising: A receiving unit for calculating a parameter necessary for position estimation from a received signal from the terminal device;
The position estimation unit estimates the position of the terminal device using the parameter output from the reception unit,
The base station apparatus according to claim 1. The position estimation unit estimates the position of the terminal device using parameters output from the reception unit and parameters necessary for position estimation calculated in another base station device,
The base station apparatus according to claim 2. A video information processing apparatus connected to a plurality of base station apparatuses that perform radio communication with one or a plurality of terminal apparatuses existing in the own cell,
A BS interface unit that receives, from each base station device, image data with position information obtained by combining the image data captured by the terminal device with the position information of the terminal device that transmitted the image data;
A feature point detection unit that detects a feature point by analyzing a plurality of pieces of image data taken from different positions;
A three-dimensional map generation unit that generates a three-dimensional map using the position information of the terminal device and the three-dimensional position of the feature points;
A video information processing apparatus. A plurality of terminal devices that convert digital images of images taken by a camera, a plurality of base station devices that each perform wireless communication with one or a plurality of the terminal devices existing in its own cell, and the plurality of base station devices, A 3D map generation system comprising: a video information processing apparatus connected to generate a 3D map;
Each of the base station devices
An image data acquisition unit for acquiring image data from the terminal device;
A position estimation unit that estimates the position of the terminal device that transmitted the image data;
An additional information generating unit that generates image data with position information obtained by combining the image data with the position information of the terminal device that transmitted the image data;
An image interface unit for transmitting the image data with position information to the video information processing apparatus;
Have
The video information processing apparatus
A BS interface unit that receives the image data with position information from the base station device;
A feature point detection unit that detects a feature point by analyzing a plurality of pieces of image data taken from different positions;
A three-dimensional map generation unit that generates a three-dimensional map using the position information of the terminal device and the three-dimensional position of the feature points;
Having
3D map generation system. A plurality of terminal devices that convert digital images of images taken by a camera, a plurality of base station devices that each perform wireless communication with one or a plurality of the terminal devices existing in its own cell, and the plurality of base station devices, A 3D map generation method of a 3D map generation system comprising: a video information processing apparatus connected to generate a 3D map;
Each of the base station devices
Receiving image data from the terminal device;
Estimating the position of the terminal device that transmitted the image data,
Generating image data with position information obtained by combining the image data with the position information of the terminal device that transmitted the image data;
Transmitting the image data with position information to the video information processing apparatus;
The video information processing apparatus is
Receiving the image data with position information from the base station device;
Analyzing multiple pieces of image data taken from different positions to detect feature points,
Generating a three-dimensional map using the position information of the terminal device and the three-dimensional position of the feature point;
3D map generation method.

Images