WO2022091165A1 - Information processing device, information processing method, information processing system, and computer-readable storage medium - Google Patents

Abstract

Provided are an information processing device, etc., that are capable of appropriately associating a moving body and information pertaining to radio waves, even if a plurality of moving bodies are present. This information processing device comprises: a control means that, if a plurality of moving bodies appear in an image captured by an imaging device, sets the position of a prescribed moving body among the plurality of moving bodies as the detection range for a radio wave sensor and performs control whereby the radio wave sensor is caused to detect radio waves; and an acquisition means that acquires information pertaining to the radio waves detected by the radio wave sensor.

Description

Information processing equipment, information processing methods, information processing systems, and computer-readable storage media

This disclosure relates to the technology for acquiring information on radio waves.

There is a technology that detects radio waves transmitted from communication devices such as smartphones and associates the detected radio waves with a person who owns the communication device or a moving object such as a vehicle equipped with the communication device.

For example, in Patent Document 1, a radio wave transmitted from a car equipped with a radio is detected, and if the radio wave is an illegal radio wave, the car is photographed with a camera to identify the car that transmitted the illegal radio wave. The technology to be used will be disclosed.

Japanese Unexamined Patent Publication No. 08-254560

In the technique of Patent Document 1, it is premised that there is one automobile in the range where radio waves can be detected and the range where radio waves can be photographed. Therefore, for example, when there are a plurality of automobiles in a range in which radio waves can be detected or in a range in which radio waves can be photographed, it may be difficult to appropriately associate the detected radio waves with the automobiles.

The present disclosure has been made in view of the above problems, and provides an information processing device and the like capable of appropriately associating information on radio waves with a mobile body even when a plurality of mobile bodies exist. One of the purposes is to do.

In the information processing apparatus of one aspect of the present disclosure, when a plurality of moving objects are reflected in an image captured by the photographing device, the position of a predetermined moving object among the plurality of moving objects is set as a detection range of the radio wave sensor. A control means for controlling the radio wave sensor to detect a radio wave and an acquisition means for acquiring information on the radio wave detected by the radio wave sensor are provided.

In the information processing method of one aspect of the present disclosure, when a plurality of moving objects are reflected in an image captured by an imaging device, the position of a predetermined moving object among the plurality of moving objects is set as a detection range of a radio wave sensor. Then, the radio wave sensor is controlled to detect the radio wave, and the information about the radio wave detected by the radio wave sensor is acquired.

The computer-readable storage medium of one aspect of the present disclosure detects the position of a predetermined moving body among the plurality of moving bodies when a plurality of moving bodies are reflected in the captured image taken by the photographing device by a radio wave sensor. It stores a program that is set as a range and causes a computer to execute a process of controlling the radio wave sensor to detect a radio wave and a process of acquiring information about the radio wave detected by the radio wave sensor.

According to the present disclosure, even when there are a plurality of mobile bodies, it is possible to appropriately associate the information on radio waves with the mobile body.

It is a figure which shows typically an example of the structure of the information processing system of 1st Embodiment of this disclosure. It is a block diagram which includes an example of the functional structure of the information processing apparatus of 1st Embodiment of this disclosure. It is a flowchart which shows an example of the operation of the information processing apparatus of 1st Embodiment of this disclosure. It is a block diagram which includes an example of the functional structure of the information processing system of the 2nd Embodiment of this disclosure. It is a figure which shows an example of the mobile database of the 2nd Embodiment of this disclosure. It is a figure which shows an example of the radio wave characteristic database of the 2nd Embodiment of this disclosure. It is a flowchart which shows an example of the operation of the information processing apparatus of the 2nd Embodiment of this disclosure. It is a block diagram which includes an example of the functional structure of the information processing system of the 3rd Embodiment of this disclosure. It is a flowchart which shows an example of the operation of the information processing apparatus of the 3rd Embodiment of this disclosure. It is a block diagram which includes an example of the functional structure of the information processing system of 4th Embodiment of this disclosure. It is a block diagram which shows an example of the hardware composition of the computer apparatus which realizes the information processing apparatus of 1st, 2nd, 3rd, and 4th Embodiment of this disclosure.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<First Embodiment>
An information processing system including the information processing apparatus of the first embodiment will be described.

FIG. 1 is a diagram schematically showing an example of the configuration of the information processing system 1000 of the first embodiment. As shown in FIG. 1, the information processing system 1000 includes an information processing device 100, a photographing device 200, and a radio wave sensor 300. The information processing device 100 is connected to the photographing device 200 and the radio wave sensor 300 so as to be able to communicate with each other by wire or wirelessly. The photographing device 200 and the radio wave sensor 300 may be an integrated device. Further, the information processing device 100 may be a device integrated with either or both of the photographing device 200 and the radio wave sensor 300. The information processing device 100 is also connected to the storage device 400 so as to be able to communicate by wire or wirelessly. The storage device 400 may be mounted on the information processing device 100.

In the situation shown in FIG. 1, the photographing device 200 is photographing a moving object. The radio wave sensor 300 detects radio waves in a range including at least one mobile body. The photographing device 200 and the radio wave sensor 300 may be installed at a plurality of places.

The moving body is, for example, a person, but the example of the moving body is not limited to this. The moving body may be a vehicle or an aircraft such as a multicopter (also referred to as a multirotor helicopter or a multirotor). A multicopter that a person does not board may be called a UAV (Unmanned Aerial Vehicle), a small unmanned aerial vehicle, or a drone. In addition, the moving body may be a robot that operates autonomously (autonomous robot) or a robot that is remotely controlled (remotely controlled robot).

The photographing device 200 photographs a predetermined range and generates photographing data. In the present specification, the range in the real space taken by the photographing apparatus 200 is also referred to as an imaging range, and the image captured by the photographing apparatus 200 is also referred to as a captured image. The captured image may be each of a plurality of frames constituting the moving image, or may be each of the still images captured at regular time intervals. If there is a moving object in the shooting range, the moving object appears in the captured image. The shooting data indicates information constituting the shot image. The photographing device 200 transmits the photographing data to, for example, the information processing apparatus 100. The information processing apparatus 100 acquires a photographed image based on the photographed data.

The radio wave sensor 300 detects radio waves output from communication devices existing in a predetermined range and generates detection data. In the present specification, the range in real space where the radio wave sensor 300 detects radio waves is also referred to as a detection range. The radio wave sensor 300 can detect radio waves in a range including, for example, a shooting range. The radio wave sensor 300 detects radio waves output from, for example, a communication device possessed by a person, a communication device possessed by a person in a vehicle, or a communication device mounted on a vehicle, an aircraft, or the like. The detection data is information related to the radio wave detected by the radio wave sensor 300. The radio wave sensor 300 transmits the detected data to the information processing device 100.

As shown in FIG. 2, the information processing apparatus 100 includes a control unit 110 and an acquisition unit 120. When a plurality of moving objects are reflected in the captured image captured by the photographing device 200, the control unit 110 sets the position of a predetermined moving object among the plurality of moving objects as the detection range of the radio wave sensor 300, and the radio wave sensor 300. Controls to detect radio waves. The control unit 110 is an example of control means.

For example, the control unit 110 acquires in advance the feature amount to be extracted based on the image data of another image in which a predetermined moving object is reflected, and the feature amount and the feature extracted from a plurality of moving objects appearing in the captured image. Compare with quantity. The control unit 110 detects a specific moving object having a feature amount similar to the feature amount extracted based on the image data of another image among the compared feature amounts. Then, for example, the control unit 110 acquires a position (position information) in the real space from the position on the captured image of the detected moving object, and sets the acquired position in the real space as the detection range of the radio wave sensor 300. Set. Here, the position on the captured image and the position on the real space are associated in advance. The control unit 110 may calculate the position in the real space from the position of the moving body on the captured image and the size of the moving object reflected in the captured image. It should be noted that at least one of the other image in which the predetermined moving body is reflected and the feature amount extracted based on the image data of the other image may be stored in the storage device 400 in advance.

Alternatively, for example, when the control unit 110 detects that a plurality of moving objects are reflected in the captured image, the control unit 110 may acquire the position of each of the moving objects in the real space. Then, the control unit 110 may set the position of each moving object in the real space in order as the detection range of the radio wave sensor 300, and control the radio wave sensor 300 to detect the radio wave in each detection range.

The acquisition unit 120 acquires information on the radio wave detected by the radio wave sensor 300. For example, the acquisition unit 120 may acquire the detection data as information related to radio waves from the radio wave sensor 300 controlled by the control unit 110, or may acquire the processed data of the detection data as information related to radio waves. good. The acquisition unit 120 is an example of acquisition means.

Next, the operation of the information processing apparatus 100 will be described with reference to FIG. In this specification, each step of the sequence diagram and the flowchart is expressed by using the number assigned to each step as in "S101". In this operation example, the information processing apparatus 100 has acquired a captured image.

FIG. 3 is a flowchart illustrating an example of the operation of the information processing apparatus 100. First, when a plurality of moving objects appear in a captured image, the control unit 110 sets a predetermined position of the moving object as a detection range of the radio wave sensor, and causes the radio wave sensor to detect the radio wave (S101). Next, the acquisition unit 120 acquires information regarding the radio wave detected by the radio wave sensor (S102).

As described above, in the information processing apparatus 100 of the first embodiment, when a plurality of moving objects are reflected in the captured image captured by the photographing device, the position of a predetermined moving body among the plurality of moving objects is determined by the radio wave sensor. Set it as the detection range and let the radio wave sensor detect the radio wave. Then, the information processing apparatus 100 acquires information about the radio wave detected by the radio wave sensor. As a result, the information processing apparatus 100 can set the detection range of the radio wave sensor according to the position of the moving body, so that information on the radio wave output by the communication device of the predetermined moving body among the plurality of moving bodies can be obtained. Can be obtained. That is, the information processing apparatus 100 of the first embodiment has an effect that even when a plurality of mobile bodies exist, the information about the radio wave and the mobile body can be associated with each other.

<Second embodiment>
Next, an information processing system including the information processing apparatus of the second embodiment will be described.

FIG. 4 is a block diagram showing an example of the configuration of the information processing system 1001 according to the second embodiment. As shown in FIG. 4, the information processing system 1001 includes an information processing device 101 instead of the information processing device 100 in the first embodiment, and other than that, the information processing system 1000 described in the first embodiment. The same is true. That is, the information processing system 1001 includes an information processing device 101, a photographing device 200, and a radio wave sensor 300. The description of the contents in which the configuration and operation of the information processing system 1001 shown in FIG. 4 overlaps with the description of the first embodiment will be omitted.

[Details of the photographing apparatus 200]
The photographing device 200 photographs an imaging range and generates imaging data. The shooting data may be data in which the shot image is encoded. At this time, the information processing apparatus 101 acquires a captured image by decoding the captured data received from the imaging device 200. The shooting data may include information on the shooting time, information indicating the shooting location, and information for identifying the shooting device 200.

Further, the photographing device 200 may be controlled by the information processing device 101. For example, the photographing apparatus 200 may change the detection range according to the instruction from the information processing apparatus 101.

[Details of radio wave sensor 300]
The radio wave sensor 300 detects radio waves output from communication devices existing in the detection range and generates detection data. The detection data includes information indicating the signal waveform of the detected radio wave. Further, the detection data may include information indicating the radio wave intensity of the detected radio wave. The communication device that outputs radio waves is, for example, a mobile phone, a smartphone, a tablet terminal, or the like, but is not limited to this example. The communication device may be a device that outputs radio waves of any communication method. The radio wave detected by the radio wave sensor 300 may be, for example, a radio wave such as LTE (Long Term Evolution), wireless LAN (Local Area Network), or short-range radio wave. When the communication device is a smartphone or the like, the timing at which the radio wave is transmitted is, for example, a voice call, communication by operating an application, polling communication between the communication device and the base station, and the like.

The radio wave sensor 300 transmits the detected data to the information processing device 101. At this time, the detection data may include information indicating the time when the radio wave was detected, information indicating the location where the radio wave sensor 300 is installed, and information for identifying the radio wave sensor 300.

Further, the radio wave sensor 300 may be controlled by the information processing device 101. For example, the radio wave sensor 300 may change the detection range according to an instruction from the information processing device 101. The radio wave sensor 300 can detect radio waves in a range including a shooting range. The range in which the radio wave sensor 300 can detect radio waves and the shooting range are associated in advance. The detection range may be set in advance to include the shooting range.

[Details of storage device 400]
The storage device 400 includes a mobile database 410 and a radio wave feature database 420. In the mobile body database 410, data in which information for identifying a mobile body and information about the mobile body are associated with each other are registered. The information about the moving body is information indicating the characteristics of the moving body, such as an image showing the moving body and a feature amount extracted from the image showing the moving body. In the present specification, information about a mobile body is also referred to as "mobile body information". FIG. 5 is a diagram showing an example of a mobile database. In the mobile database shown in FIG. 5, the mobile information of "data X" is associated with the mobile of the identification information "001". Here, the mobile information is not limited to the above example. The mobile information may further include personal information such as, for example, when the mobile is a person, the gender, age and address of the mobile. Further, for example, when the moving body is a vehicle, the information on the vehicle type and the license plate may be further included in the moving body information. Further, when the moving body is an aircraft or other robot, the serial number may be further included in the moving body information. In the present specification, the mobile body registered in the mobile body database 410 is also referred to as a registered mobile body.

Data associated with mobile information and radio wave feature information is registered in the radio wave feature database 420. Details of the radio wave feature database 420 and radio wave feature information will be described later.

[Details of information processing device 101]
As shown in FIG. 4, the information processing apparatus 101 includes a control unit 111, an acquisition unit 121, and a registration unit 130.

The control unit 111 has a mobile body detection unit 141 and a sensor control unit 142. The moving object detection unit 141 detects a moving object reflected in the captured image. For the detection of the moving object, for example, a method of detecting the moving object based on the difference between a plurality of captured images, that is, a background subtraction method can be considered, but the present invention is not limited to this example. The method for detecting the moving body may be various known methods such as a method for detecting the shape of the moving body. Further, the moving body detection unit 141 detects the position information indicating the spatial position in the photographing range with respect to the detected moving body. The moving body detection unit 141 notifies the sensor control unit 142 of the position information corresponding to the predetermined moving body among the detected moving bodies. For example, when the registered mobile body registered in the mobile body database 410 is detected from the captured image, the mobile body detection unit 141 detects the position information of the detected registered mobile body. At this time, the moving object detection unit 141 acquires a feature amount related to the registered moving object from the moving object information registered in the moving object database, and collates the feature amount with the information extracted based on the captured image. By doing so, the registered moving object is detected from the captured image. Further, the mobile body detection unit 141 may detect the position information corresponding to the moving body having the characteristics when the moving body having the preset characteristics is detected. For example, the mobile body detection unit 141 may detect the position information of each of the moving bodies (persons) whose gender is male, or detect the position information of each moving body whose age is a predetermined value or more. May be good. The moving object detection unit 141 may detect the position information corresponding to all the moving objects detected from the captured image without performing the collation as described above. Further, the moving body detection unit 141 may calculate the number of detected moving bodies. In this way, the moving body detection unit 141 detects a predetermined moving body from the captured image. The mobile body detection unit 141 is an example of the mobile body detection means.

The sensor control unit 142 controls the radio wave sensor 300 based on the detection result of the mobile body detection unit 141. Specifically, the sensor control unit 142 sets the detection range to include the position indicated by the position information based on the position information detected by the mobile body detection unit 141. For example, when the detection range is set to include the photographing range, the sensor control unit 142 narrows the detection range and sets the detection range to be the position indicated by the position information and the range in the vicinity thereof. Then, the sensor control unit 142 controls the radio wave sensor 300 to detect the radio wave within the set detection range. If the moving object detection unit 141 detects only one moving object and the detection range is set to include the photographing range, the sensor control unit 142 does not have to set the detection range again. good. In this way, when a predetermined moving body is detected, the sensor control unit 142 sets the position of the predetermined moving body as a detection range and controls the radio wave sensor to detect the radio wave. The sensor control unit 142 is an example of the sensor control means.

The acquisition unit 121 acquires the imaging data from the imaging device 200 and acquires the detection data from the radio wave sensor 300. Further, the acquisition unit 121 acquires the radio wave feature information extracted from the detection data. The radio wave characteristic information is information showing the characteristics of the signal waveform of the radio wave, which differs depending on the device that outputs the radio wave.

Devices that output radio waves are manufactured so as to meet predetermined communication standards, but there are slight individual differences due to variations in analog circuits and the like. Therefore, the radio wave has a feature amount that can be identified based on the individual difference of the device that is the source. The radio wave feature information is, for example, a feature amount based on a modulation error by modulation analysis, a feature amount based on a time waveform of a radio wave, or the like. Further, for example, the radio wave characteristic information includes the transient (rising and falling) of the received signal, the power spectral density of the preamble portion, the error vector amplitude (EVM), the IQ (In-phase Quadrature-phase) phase error, and the like. It may be a feature quantity indicating one or more of the IQ imbalance amount, the frequency offset, and the symbol clock error, but is not limited thereto.

Since the radio wave feature information depends on the hardware of the communication device, it is difficult to rewrite it manually. Therefore, the radio wave feature information is information that is unlikely to be intentionally disguised. Further, the radio wave feature information is information that can identify an individual communication device regardless of the type of information transmitted by the radio wave, for example, a mobile phone operator.

The process of extracting radio wave feature information from the detected data is performed by, for example, the acquisition unit 121 according to a predetermined rule. Here, the information processing apparatus 101 can use machine learning when extracting radio wave feature information from information indicating a radio wave signal waveform. Specifically, the information processing apparatus 101 can extract radio wave feature information from the information indicating the signal waveform of the radio wave by using the extraction model generated based on the learning data. For example, the learning data includes information indicating radio wave signal waveforms output from various communication devices and radio wave feature information corresponding to each of the information indicating radio wave signal waveforms. By using such learning data, it is possible to generate an extraction model in which information indicating a signal waveform of a radio wave is input and radio wave feature information is output. The training data and the extraction model may be stored in the storage device 400 in advance. Further, the acquisition unit 121 may generate the learning model.

The registration unit 130 associates the acquired radio wave feature information with the information about a predetermined mobile object and registers it in the radio wave feature database 420. The information about the predetermined moving body may be, for example, a photographed image in which the predetermined moving body is reflected, or an image obtained by cutting out a region in which the predetermined moving body is reflected from the photographed image. Further, the information about the predetermined moving body may be information indicating the appearance of the predetermined moving body extracted from the captured image in which the predetermined moving body is reflected. Further, when the registered mobile body is detected as a predetermined mobile body by the mobile body detection unit 141, the registration unit 130 associates the mobile body information registered in the mobile body database 410 with the acquired radio wave feature information. It may be registered in the radio wave feature database.

The registration unit 130 registers in the radio wave feature database 420 data in which the acquired radio wave feature information is associated with information about a predetermined mobile object. FIG. 6 is a diagram showing an example of the radio wave feature database 420. Here, the mobile body information shown in FIG. 6 indicates information regarding a predetermined mobile body. In the record of the first line of FIG. 6, "data 1" which is radio wave characteristic information and "data A" which is mobile information are associated with each other. Further, as shown in FIG. 6, the registration unit 130 may further associate the time and place information with the data in which the radio wave feature information and the mobile body information are associated with each other. The time information may be, for example, either the time taken by the photographing device 200 or the time when the radio wave sensor 300 detects the radio wave. The location information may be, for example, information on the installation location of the photographing device 200 or the radio wave sensor 300.

In this way, the registration unit 130 registers the acquired information on the radio wave and the information on the predetermined mobile object in the database in association with each other. The registration unit 130 is an example of registration means.

Note that the detection data may include multiple radio wave feature information. For example, when there are a plurality of communication devices in the detection range, the detection data may include a plurality of radio wave feature information. Further, even when one communication device outputs a plurality of radio waves, the detection data may include a plurality of radio wave feature information. In such a case, the acquisition unit 121 acquires a plurality of radio wave feature information. Then, the registration unit 130 registers the data in which the mobile body information and the plurality of radio wave feature information are associated with each other in the radio wave feature database 420. When the detection data includes a plurality of radio wave feature information as described above, the radio wave sensor 300 may also detect radio waves of a communication device that the predetermined mobile body does not have. Even in such a case, since it is possible to search whether or not the same radio wave feature information has been acquired among the records having the same mobile body information in the radio wave feature database 420, there is a predetermined mobile body. It is possible to identify the radio wave of the communication device that is being used. That is, the information processing apparatus 101 can more accurately associate the mobile body with the radio wave information by registering the data in which the mobile body information and the plurality of radio wave feature information are associated with each other in the radio wave feature database 420. It becomes.

[Operation of information processing device 101]
Next, the operation of the information processing apparatus 101 will be described with reference to FIG. 7. FIG. 7 is a flowchart illustrating an example of the operation of the information processing apparatus 101. In this operation example, an example in which the information processing apparatus 101 controls the radio wave sensor when the registered mobile body is detected from the captured image will be described. At this time, it is assumed that the detection range is set to include the shooting range.

First, the acquisition unit 121 acquires shooting data from the shooting device 200 (S201). At this time, for example, the acquisition unit 121 acquires the captured image by decoding the captured data. The moving object detection unit 141 of the control unit 111 detects a moving object from the captured image (S202). When the moving body detection unit 141 detects a plurality of moving bodies from the captured image (“Yes” in S203) and further detects a registered moving body (“Yes” in S204), it detects the position information of the registered moving body. (S205). Then, the sensor control unit 142 sets the detection range of the radio wave sensor 300 based on the detected position information (S206). For example, the sensor control unit 142 narrows the detection range and sets a position indicating the detected position information as the detection range. Then, the sensor control unit 142 causes the radio wave sensor 300 to detect the radio wave (S208). The acquisition unit 121 acquires the detection data from the radio wave sensor 300, and acquires the radio wave feature information from the detection data (S209). At this time, the acquisition unit 121 acquires the radio wave feature information by extracting the radio wave feature information from the detection data using, for example, a learning model stored in the storage device 400 in advance. Then, the registration unit 130 registers the data in which the acquired radio wave feature information and the mobile body information are associated with each other in the radio wave feature database 420 (S210). At this time, the registration unit 130 registers, for example, the mobile body information of the registered mobile body registered in the mobile body database 410 in association with the acquired radio wave feature information.

When a plurality of moving objects are not detected in the process of S203 (“No” in S203), that is, when only one moving object is displayed in the captured image, the moving object detection unit 141 registers the moving object to be displayed in the captured image. Determine if it is a mobile body. When the mobile body detection unit 141 detects the target mobile body (“Yes” in S207), the information processing apparatus 101 proceeds to the process of S208. When the mobile object detection unit 141 does not detect the target mobile object (“No” in S207), the information processing apparatus 101 returns to the process of S201. Even when the registered mobile body is not detected in the process of S204 (“No” in S204), the information processing apparatus 101 returns to the process of S201.

As described above, when the information processing apparatus 101 of the second embodiment shows a plurality of moving objects in the captured image captured by the photographing device, the position of the predetermined moving object among the plurality of moving objects is determined by the radio wave sensor. It is set as a detection range, controls the radio wave sensor to detect radio waves, and acquires information on radio waves detected by the radio wave sensor. Further, the information processing apparatus 101 associates the acquired information on the radio wave with the information on the predetermined mobile body and registers the information in the database. As a result, the information processing apparatus 101 of the second embodiment has the same effect as the information processing apparatus 100 of the first embodiment. That is, the information processing apparatus 101 has an effect that the information about the radio wave and the mobile body can be associated with each other even when a plurality of mobile bodies exist.

[Modification 1]
In the above-described embodiment, an example is described in which the acquisition unit 121 performs a process of extracting radio wave feature information from the detection data. The radio wave sensor 300 may perform the process of extracting the radio wave feature information from the detection data. For example, when the radio wave sensor 300 detects a radio wave in the detection range, the radio wave sensor 300 extracts radio wave feature information from the signal waveform of the detected radio wave. The radio wave sensor 300 transmits the extracted radio wave feature information as detection data to the information processing apparatus 101. The acquisition unit 121 of the information processing apparatus 101 acquires radio wave feature information as detection data. At this time, the extraction model may be stored in advance in a storage device (not shown) included in the radio wave sensor 300.

<Third embodiment>
Next, an information processing system including the information processing apparatus of the third embodiment will be described.

FIG. 8 is a block diagram showing an example of the configuration of the information processing system 1002 according to the third embodiment. As shown in FIG. 8, the information processing system 1002 includes an information processing device 102 instead of the information processing device 101 in the second embodiment, and other than that, the information processing system 1001 described in the second embodiment is used. The same is true. That is, the information processing system 1002 includes an information processing device 102, a photographing device 200, and a radio wave sensor 300. The description of the contents in which the configuration and operation of the information processing system 1002 shown in FIG. 8 overlaps with the description of the first and second embodiments will be omitted.

[Details of Information Processing Device 102]
As shown in FIG. 8, the information processing apparatus 102 includes a control unit 112, an acquisition unit 121, a registration unit 131, a collation unit 150, and a display control unit 160. The control unit 112 includes a mobile body detection unit 143 and a sensor control unit 142. The registration unit 131 and the mobile body detection unit 143 perform the operations described below in addition to the operations of the registration unit 130 and the mobile body detection unit 141 described in the second embodiment, respectively.

The collation unit 150 collates the information regarding the radio wave acquired by the acquisition unit 121 with the information regarding the radio wave registered in the database. For example, when the radio wave feature information is acquired by the acquisition unit 121, the collation unit 150 collates the acquired radio wave feature information with the radio wave feature information registered in the radio wave feature database 420.

Here, the collation unit 150 can use machine learning when collating the radio wave feature information. Specifically, the collation unit 150 uses a collation model generated based on the training data to determine whether or not the collation matches. The collation model is, for example, a model for collating the acquired radio wave feature information with the registered radio wave feature information and determining whether or not they match. By inputting the acquired radio wave feature information into the collation model, the degree of similarity with the registered radio wave feature information is calculated. The collation unit 150 identifies the mobile body information associated with the radio wave feature information having a similarity of a predetermined value or more in the radio wave feature database 420. If the collation does not match, the collation unit 150 outputs information indicating that the collation did not match.

In this way, the collation unit 150 collates the acquired information on the radio wave with the information on the radio wave registered in the database. The collating unit 150 is an example of collating means.

The display control unit 160 causes the display device (not shown) to display the captured image captured by the photographing device 200. Here, the display device is a device that is connected to the information processing device 102 so as to be able to communicate by wire or wirelessly. Further, the display device may be, for example, a display included in a device communicably connected to the information processing device 102. The device communicably connected to the information processing device 102 may be, for example, a personal computer, a smartphone, a tablet terminal, or the like.

The display control unit 160 causes the display device to display the captured image when the collation by the collation unit 150 matches. For example, the display control unit 160 causes the display device to display a captured image in which at least a part of the detection range is displayed. The detection range at this time is the range in which the radio wave including the radio wave feature information used for the collation is detected. The display control unit 160 may display the processed image of the captured image on the display device.

In this way, the display control unit 160 causes the display device to display the captured image in which the range including the detection range is captured when the collation matches. The display control unit 160 is an example of the display control means.

The mobile body detection unit 143 determines whether or not there is a mobile body having the mobile body information in the captured image based on the mobile body information specified by the collating unit 150. For example, when the collation matches, the mobile body detection unit 143 acquires a captured image in which a range in which the radio wave including the radio wave feature information used for the collation is detected is captured. Then, the moving body detection unit 143 detects a moving body having the moving body information specified by the collating unit 150 from the captured image. When a moving body having the specified moving body information is detected by the collating unit 150, the moving body detecting unit 143 further detects the position information on the captured image of the detected moving body. At this time, the display control unit 160 adds information indicating the detected position of the moving body on the captured image to the captured image and displays it on the display device based on the position information. For example, the display control unit 160 causes the display device to display a captured image in which a rectangle is superimposed on the area where the detected moving object is reflected.

When the collation matches, the registration unit 131 registers the data associated with the radio wave feature information acquired by the acquisition unit 121 with the time and place information in the radio wave feature database 420. At this time, the registration unit 131 may associate the information of the time when the radio wave sensor 300 detects the radio wave and the information of the installation location of the radio wave sensor 300 with the acquired radio wave feature information. The registration unit 131 may further associate the acquired radio wave feature information with a captured image showing a range in which the radio wave including the acquired radio wave feature information is detected. Further, the registration unit 131 may further associate the specified mobile object information with the acquired radio wave feature information.

Further, when the collation by the collation unit 150 matches, the information processing apparatus 102 may perform the following processing. For example, the sensor control unit 142 may control the radio wave sensor 300 to detect the radio wave around the range in which the radio wave including the radio wave feature information used for the collation is detected. Then, the acquisition unit 121 acquires the radio wave feature information extracted from the detected radio wave. The registration unit 131 is the radio wave feature information acquired at this time and the information registered in the radio wave feature database 420, and the matching radio wave feature information or the moving body information associated with the matching radio wave feature information. And may be associated.

Further, when there are a plurality of radio wave feature information acquired by the acquisition unit 121, and at least one of the plurality of radio wave feature information matches the radio wave feature information registered in the radio wave feature database 420, The registration unit 131 may perform the following processing. For example, the registration unit 131 may associate the acquired plurality of radio wave feature information with the mobile body information associated with the matched radio wave feature information in the radio wave feature database 420. Further, the registration unit 131 may associate the acquired plurality of radio wave feature information with the radio wave feature information of the radio wave feature database 420 that matches the matching.

In this way, the following effects can be achieved by associating the registered mobile body with the radio wave feature information different from the radio wave feature information of the registered mobile body. For example, it is assumed that the radio wave feature information of the moving object registered in the radio wave feature database 420 is detected at a plurality of places. In this case, assuming that radio wave feature information different from the radio wave feature information of the registered mobile is also detected at a plurality of locations, the registered mobile has or is registered with a plurality of communication devices. It can be estimated that there are many cases where a specific mobile body exists in the vicinity of the mobile body.

[Operation of information processing device 102]
Next, the operation of the information processing apparatus 102 will be described with reference to FIG. FIG. 9 is a flowchart illustrating an example of the operation of the information processing apparatus 102. In this operation example, it is assumed that at least one or more records are registered in the radio wave feature database 420 by the operation shown in FIG. 7.

First, the acquisition unit 121 acquires radio wave feature information from the detection data (S301). The collation unit 150 collates the acquired radio wave feature information with the radio wave feature information registered in the radio wave feature database 420 (S302). When the matching is matched (Yes in S303), the matching unit 150 identifies the mobile body information associated with the matching radio wave feature information (radio wave feature information registered in the radio wave feature database 420) (S304). .. The registration unit 131 registers the data in which the acquired radio wave feature information is associated with the time and place information in the radio wave feature database 420 (S305). If the collation does not match (No in S303), the process returns to the process of S301.

When the moving body detection unit 143 detects a moving body having the specified moving body information from the captured image (Yes in S306), the display control unit 160 indicates the position of the detected moving object on the captured image. The captured image is displayed on the display device (S307). When the moving body detection unit 143 does not detect the moving body having the specified moving body information from the captured image (No of S306), the captured image obtained by capturing the detection range of the radio wave sensor 300 is displayed. The detection range of the radio wave sensor 300 at this time is a range including the range in which the radio wave including the acquired radio wave feature information is detected.

As described above, the information processing apparatus 102 of the third embodiment collates the acquired information on the radio wave with the information on the radio wave registered in the database. As a result, the information processing apparatus 102 can detect a specific moving object by using the information about the radio wave.

Further, the information processing apparatus 102 of the third embodiment causes the display device to display the captured image in which the range including the detection range is captured when the collation matches. Thereby, when the information processing apparatus 102 detects a specific mobile body by using the information about the radio wave, for example, the information processing apparatus 102 can inform the user where the detected mobile body exists.

[Modification 2]
The photographing apparatus 200 may be controlled according to the collation result of the collation unit 150. For example, the information processing apparatus 102 sets the photographing range so as to include the range in which the radio wave including the radio wave feature information used for the matching is detected when the matching by the collating unit 150 is matched, and causes the photographing apparatus 200 to take an image. You may. At this time, the information processing apparatus 102 may set the photographing range so as to include the entire detection range of the radio wave sensor 300, or may include a radio wave feature information which is a part of the detection range and used for collation. The shooting range may be set so as to include the range where the radio wave is detected. When the shooting range is set so as to include a part of the detection range, the information processing apparatus 102 uses the radio wave intensity information including the radio wave feature information used for the collation to detect the radio wave in the detection range. You may narrow down the range in which the transmitted communication device exists. The sensor control unit 142 may perform such a process of controlling the photographing apparatus 200, for example.

<Fourth Embodiment>
Next, an information processing system including the information processing apparatus of the fourth embodiment will be described.

FIG. 10 is a block diagram showing an example of the configuration of the information processing system 1003 according to the fourth embodiment. As shown in FIG. 10, the information processing system 1003 includes an information processing device 103 instead of the information processing device 102 in the third embodiment, and other than that, the information processing system 1002 described in the third embodiment is used. The same is true. That is, the information processing system 1003 includes an information processing device 103, a photographing device 200, and a radio wave sensor 300. The description of the contents in which the configuration and operation of the information processing system 1003 shown in FIG. 10 overlaps with the description of the first, second, and third embodiments will be omitted.

[Details of information processing device 103]
As shown in FIG. 10, the information processing apparatus 103 includes a control unit 112, an acquisition unit 121, a registration unit 131, a collation unit 150, and an alarm generation unit 170.

The alarm generation unit 170 generates an alarm when the collation by the collation unit 150 matches. Here, the alarm is, for example, a sound generated from a speaker included in a device communicably connected to the information processing device 102, but is not limited to this example. For example, the alarm generation unit 170 may display information indicating that the collation has been matched on the display device included in the device communicably connected to the information processing device 102. At this time, the alarm generation unit 170 may display information on the time and place where the radio wave including the radio wave feature information that matches the matching is detected, in addition to the information indicating that the matching is matched. In this way, the alarm generation unit 170 generates an alarm when the collation matches. The alarm generation unit 170 is an example of an alarm generation means.

The alarm generation unit 170 performs, for example, a process of generating an alarm after the process of S305 in the flowchart of FIG. The information processing device 103 may include a display control unit 160. That is, the information processing apparatus 103 may perform the processing after S306 in FIG. 9 after the processing for generating the alarm is performed.

As described above, the information processing apparatus 103 of the fourth embodiment generates an alarm when the collation matches. As a result, the information processing apparatus 103 can more quickly notify the user that the collation has been matched.

<Example of hardware configuration of information processing device>
The hardware constituting the information processing apparatus of the first, second, third, and fourth embodiments described above will be described. FIG. 11 is a block diagram showing an example of a hardware configuration of a computer device that realizes the information processing device in each embodiment. Each block shown in FIG. 11 can be realized by a combination of software and a computer device 10 that realizes an information processing device and an information processing method in each embodiment.

As shown in FIG. 11, the computer device 10 includes a processor 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, a storage device 14, an input / output interface 15, a bus 16, and a drive device 17. The information processing device may be realized by a plurality of electric circuits.

The storage device 14 stores a program (computer program) 18. The processor 11 executes the program 18 of the information processing apparatus using the RAM 12. Specifically, for example, the program 18 includes a program that causes a computer to execute the processes shown in FIGS. 3, 7, and 9. Depending on the processor 11 executing the program 18, each component of the information processing apparatus ( control units 110, 111, 112, acquisition units 120, 121, registration units 130, 131, collation unit 150, described above). The functions of the display control unit 160, the alarm generation unit 170, etc.) are realized. The program 18 may be stored in the ROM 13. Further, the program 18 may be recorded on the storage medium 20 and read out using the drive device 17, or may be transmitted from an external device (not shown) to the computer device 10 via a network (not shown).

The input / output interface 15 exchanges data with peripheral devices (keyboard, mouse, display device, etc.) 19. The input / output interface 15 functions as a means for acquiring or outputting data. The bus 16 connects each component.

There are various variations in the method of realizing the information processing device. For example, the information processing device can be realized as a dedicated device. Further, the information processing device can be realized based on a combination of a plurality of devices.

A processing method in which a program for realizing each component in the function of each embodiment is recorded in a storage medium, the program recorded in the storage medium is read out as a code, and the program is executed in a computer is also included in the category of each embodiment. .. That is, a computer-readable storage medium is also included in the scope of each embodiment. Further, the storage medium in which the above-mentioned program is recorded and the program itself are also included in each embodiment.

The storage medium is, for example, a floppy (registered trademark) disk, a hard disk, an optical disk, a magneto-optical disk, a CD (Compact Disc) -ROM, a magnetic tape, a non-volatile memory card, or a ROM, but the storage medium is not limited to this example. The program recorded on the storage medium is not limited to a program that executes processing by itself, but operates on an OS (Operating System) in cooperation with other software and expansion board functions to execute processing. Programs to be implemented are also included in the category of each embodiment.

Although the invention of the present application has been described above with reference to the embodiment, the invention of the present application is not limited to the above embodiment. Various changes that can be understood by those skilled in the art can be made within the scope of the present invention in terms of the configuration and details of the present invention.

A part or all of the above embodiment may be described as in the following appendix, but is not limited to the following.

<Additional Notes>
[Appendix 1]
When a plurality of moving objects appear in the captured image captured by the photographing device, the position of a predetermined moving object among the plurality of moving objects is set as the detection range of the radio wave sensor, and the control for causing the radio wave sensor to detect the radio wave. And the control means to do
A means for acquiring information about the radio wave detected by the radio wave sensor, and the like.
Information processing equipment.

[Appendix 2]
A registration means for associating the acquired information on the radio wave with the information on the predetermined mobile object and registering the information in the database is provided.
The information processing apparatus according to Appendix 1.

[Appendix 3]
When a plurality of the radio waves are detected by the radio wave sensor,
The acquisition means acquires information about the radio wave corresponding to each of the plurality of radio waves, and obtains information about the radio wave.
The registration means associates the acquired information on the radio waves with the information on the predetermined mobile body and registers the information in the database.
The information processing device according to Appendix 2.

[Appendix 4]
A collation means for collating the acquired information on the radio wave with the information on the radio wave registered in the database is provided.
The information processing apparatus according to Appendix 2 or 3.

[Appendix 5]
When the collation matches, the display control means for displaying the captured image captured in the range including the detection range on the display device is provided.
The information processing apparatus according to Appendix 4.

[Appendix 6]
An alarm generating means for generating an alarm when the collation is matched is provided.
The information processing apparatus according to Appendix 4 or 5.

[Appendix 7]
The control means is
A moving body detection means for detecting the predetermined moving body from the captured image,
When the predetermined moving body is detected, the position of the predetermined moving body is set as the detection range, and the sensor control means for controlling the radio wave sensor to detect the radio wave is provided.
The information processing apparatus according to any one of Supplementary note 1 to 6.

[Appendix 8]
The information regarding the radio wave is radio wave characteristic information indicating the characteristics of the signal waveform of the radio wave.
The information processing apparatus according to any one of Supplementary Provisions 1 to 7.

[Appendix 9]
The mobile is at least one of a person, a vehicle, an aircraft, an autonomous robot, and a remote-controlled robot.
The information processing apparatus according to any one of Supplementary Provisions 1 to 8.

[Appendix 10]
With the above-mentioned imaging device
With the radio wave sensor
The information processing apparatus according to any one of Supplementary note 1 to 9 is provided.
Information processing system.

[Appendix 11]
When a plurality of moving objects appear in the captured image captured by the photographing device, the position of a predetermined moving object among the plurality of moving objects is set as the detection range of the radio wave sensor, and the control for causing the radio wave sensor to detect the radio wave. And
Acquires information about the radio wave detected by the radio wave sensor.
Information processing method.

[Appendix 12]
The acquired information on the radio wave and the information on the predetermined mobile object are associated and registered in the database.
The information processing method according to Appendix 11.

[Appendix 13]
When a plurality of the radio waves are detected by the radio wave sensor,
Obtain information about the radio wave corresponding to each of the plurality of radio waves,
The acquired information on the radio waves and the information on the predetermined mobile object are associated and registered in the database.
The information processing method according to Appendix 12.

[Appendix 14]
The acquired information on the radio wave and the information on the radio wave registered in the database are collated.
The information processing method according to Appendix 12 or 13.

[Appendix 15]
When the collation is matched, the captured image in which the range including the detection range is captured is displayed on the display device.
The information processing method according to Appendix 14.

[Appendix 16]
If the above matching is matched, an alarm is generated.
The information processing method according to Appendix 14 or 15.

[Appendix 17]
In the process to be controlled
The predetermined moving body is detected from the captured image,
When the predetermined moving body is detected, the position of the predetermined moving body is set as the detection range, and the radio wave sensor is controlled to detect the radio wave.
The information processing method according to any one of Supplementary note 11 to 16.

[Appendix 18]
The information regarding the radio wave is radio wave characteristic information indicating the characteristics of the signal waveform of the radio wave.
The information processing method according to any one of Supplementary note 11 to 17.

[Appendix 19]
The mobile is at least one of a person, a vehicle, an aircraft, an autonomous robot, and a remote-controlled robot.
The information processing method according to any one of Supplementary note 11 to 18.

[Appendix 20]
When a plurality of moving objects appear in the captured image captured by the photographing device, the position of a predetermined moving object among the plurality of moving objects is set as the detection range of the radio wave sensor, and the control for causing the radio wave sensor to detect the radio wave. And the process of doing
Stores a program that causes a computer to execute a process of acquiring information on the radio wave detected by the radio wave sensor.
Computer-readable storage medium.

[Appendix 21]
Stores a program that causes a computer to further execute a process of associating the acquired information on the radio wave with the information on the predetermined mobile object and registering the information in the database.
The computer-readable storage medium according to Appendix 20.

[Appendix 22]
When a plurality of the radio waves are detected by the radio wave sensor,
In the acquisition process, information about the radio wave corresponding to each of the plurality of radio waves is acquired.
In the registration process, the acquired information on the radio waves and the information on the predetermined mobile body are associated and registered in the database.
The computer-readable storage medium according to Appendix 21.

[Appendix 23]
A program for further executing a process of collating the acquired information on the radio wave with the information on the radio wave registered in the database is stored.
A computer-readable storage medium according to Appendix 21 or 22.

[Appendix 24]
When the collation matches, a program for causing the computer to further execute a process of displaying the captured image captured in the range including the detection range on the display device is stored.
The computer-readable storage medium according to Appendix 23.

[Appendix 25]
Stores a program that causes the computer to further execute the process of generating an alarm when the matching is matched.
A computer-readable storage medium according to Appendix 23 or 24.

[Appendix 26]
In the process to be controlled
The predetermined moving body is detected from the captured image,
When the predetermined moving body is detected, the position of the predetermined moving body is set as the detection range, and the radio wave sensor is controlled to detect the radio wave.
The computer-readable storage medium according to any one of Supplementary Provisions 20 to 25.

[Appendix 27]
The information regarding the radio wave is radio wave characteristic information indicating the characteristics of the signal waveform of the radio wave.
The computer-readable storage medium according to any one of Supplementary Provisions 20 to 26.

[Appendix 28]
The mobile is at least one of a person, a vehicle, an aircraft, an autonomous robot, and a remote-controlled robot.
The computer-readable storage medium according to any one of Supplementary Provisions 20 to 27.

100, 101, 102, 103 Information processing device 110, 111, 112 Control unit 120, 121 Acquisition unit 130, 131 Registration unit 141 Mobile detection unit 142 Sensor control unit 150 Matching unit 160 Display control unit 170 Alarm generation unit 200 Imaging device 300 radio wave sensor 400 storage device

Claims (28)

1. When a plurality of moving objects appear in the captured image captured by the photographing device, the position of a predetermined moving object among the plurality of moving objects is set as the detection range of the radio wave sensor, and the control for causing the radio wave sensor to detect the radio wave. And the control means to do
   A means for acquiring information about the radio wave detected by the radio wave sensor, and the like.
   Information processing equipment.
2. A registration means for associating the acquired information on the radio wave with the information on the predetermined mobile object and registering the information in the database is provided.
   The information processing apparatus according to claim 1.
3. When a plurality of the radio waves are detected by the radio wave sensor,
   The acquisition means acquires information about the radio wave corresponding to each of the plurality of radio waves, and obtains information about the radio wave.
   The registration means associates the acquired information on the radio waves with the information on the predetermined mobile body and registers the information in the database.
   The information processing apparatus according to claim 2.
4. A collation means for collating the acquired information on the radio wave with the information on the radio wave registered in the database is provided.
   The information processing apparatus according to claim 2 or 3.
5. When the collation matches, the display control means for displaying the captured image captured in the range including the detection range on the display device is provided.
   The information processing apparatus according to claim 4.
6. An alarm generating means for generating an alarm when the collation is matched is provided.
   The information processing apparatus according to claim 4 or 5.
7. The control means is
   A moving body detection means for detecting the predetermined moving body from the captured image,
   When the predetermined moving body is detected, the position of the predetermined moving body is set as the detection range, and the sensor control means for controlling the radio wave sensor to detect the radio wave is provided.
   The information processing apparatus according to any one of claims 1 to 6.
8. The information regarding the radio wave is radio wave characteristic information indicating the characteristics of the signal waveform of the radio wave.
   The information processing apparatus according to any one of claims 1 to 7.
9. The mobile is at least one of a person, a vehicle, an aircraft, an autonomous robot, and a remote-controlled robot.
   The information processing apparatus according to any one of claims 1 to 8.
10. With the above-mentioned imaging device
    With the radio wave sensor
    The information processing apparatus according to any one of claims 1 to 9 is provided.
    Information processing system.
11. When a plurality of moving objects appear in the captured image captured by the photographing device, the position of a predetermined moving object among the plurality of moving objects is set as the detection range of the radio wave sensor, and the control for causing the radio wave sensor to detect the radio wave. And
    Acquires information about the radio wave detected by the radio wave sensor.
    Information processing method.
12. The acquired information on the radio wave and the information on the predetermined mobile object are associated and registered in the database.
    The information processing method according to claim 11.
13. When a plurality of the radio waves are detected by the radio wave sensor,
    Obtain information about the radio wave corresponding to each of the plurality of radio waves,
    The acquired information on the radio waves and the information on the predetermined mobile object are associated and registered in the database.
    The information processing method according to claim 12.
14. The acquired information on the radio wave and the information on the radio wave registered in the database are collated.
    The information processing method according to claim 12 or 13.
15. When the collation is matched, the captured image in which the range including the detection range is captured is displayed on the display device.
    The information processing method according to claim 14.
16. If the above matching is matched, an alarm is generated.
    The information processing method according to claim 14 or 15.
17. In the process to be controlled
    The predetermined moving body is detected from the captured image,
    When the predetermined moving body is detected, the position of the predetermined moving body is set as the detection range, and the radio wave sensor is controlled to detect the radio wave.
    The information processing method according to any one of claims 11 to 16.
18. The information regarding the radio wave is radio wave characteristic information indicating the characteristics of the signal waveform of the radio wave.
    The information processing method according to any one of claims 11 to 17.
19. The mobile is at least one of a person, a vehicle, an aircraft, an autonomous robot, and a remote-controlled robot.
    The information processing method according to any one of claims 11 to 18.
20. When a plurality of moving objects appear in the captured image captured by the photographing device, the position of a predetermined moving object among the plurality of moving objects is set as the detection range of the radio wave sensor, and the control for causing the radio wave sensor to detect the radio wave. And the process of doing
    Stores a program that causes a computer to execute a process of acquiring information on the radio wave detected by the radio wave sensor.
    Computer-readable storage medium.
21. Stores a program that causes a computer to further execute a process of associating the acquired information on the radio wave with the information on the predetermined mobile object and registering the information in the database.
    The computer-readable storage medium of claim 20.
22. When a plurality of the radio waves are detected by the radio wave sensor,
    In the acquisition process, information about the radio wave corresponding to each of the plurality of radio waves is acquired.
    In the registration process, the acquired information on the radio waves and the information on the predetermined mobile body are associated and registered in the database.
    The computer-readable storage medium of claim 21.
23. A program for further executing a process of collating the acquired information on the radio wave with the information on the radio wave registered in the database is stored.
    The computer-readable storage medium of claim 21 or 22.
24. When the collation matches, a program for causing the computer to further execute a process of displaying the captured image captured in the range including the detection range on the display device is stored.
    The computer-readable storage medium of claim 23.
25. Stores a program that causes the computer to further execute the process of generating an alarm when the matching is matched.
    The computer-readable storage medium of claim 23 or 24.
26. In the process to be controlled
    The predetermined moving body is detected from the captured image,
    When the predetermined moving body is detected, the position of the predetermined moving body is set as the detection range, and the radio wave sensor is controlled to detect the radio wave.
    The computer-readable storage medium according to any one of claims 20 to 25.
27. The information regarding the radio wave is radio wave characteristic information indicating the characteristics of the signal waveform of the radio wave.
    The computer-readable storage medium according to any one of claims 20 to 26.
28. The mobile is at least one of a person, a vehicle, an aircraft, an autonomous robot, and a remote-controlled robot.
    The computer-readable storage medium according to any one of claims 20 to 27.

Images